zjw1111 commented on code in PR #222: URL: https://github.com/apache/paimon-cpp/pull/222#discussion_r3852450860
########## docs/source/user_guide/format_table.rst: ########## @@ -0,0 +1,272 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. Ported from the Paimon documentation: +.. https://github.com/apache/paimon/blob/master/docs/docs/concepts/rest/tables.mdx + +.. default-domain:: cpp +.. highlight:: cpp + +Format Table +============ +A format table is a directory that holds multiple files of the same format. It carries no +snapshots and no manifests: the files in the directory are the table, so reading it lists +directories and writing to it adds files. A table is a format table when its ``type`` option is +``format-table``; ``file.format`` then names the format of every file in it, which here is +``parquet`` or ``orc``. + +A partitioned format table uses the standard Hive directory layout, and its partitions are +discovered from that layout rather than from metadata. By default a partition directory is named +``key=value``; setting ``format-table.partition-path-only-value`` names it by the value alone. + +Because a directory of plain files records no row identity, a format table only accepts inserts. +Reads still carry the leading ``_VALUE_KIND`` field every ``BatchReader`` promises, so an engine +that reads a batch by field index sees the same layout it does for a managed table; every row of a +format table is an insert. + +Reading and writing +------------------- +A format table is not served through :cpp:func:`Catalog::GetTable`, which describes a managed +table; use :cpp:func:`Catalog::GetFormatTable` instead. + +Reading and writing go through the entry points every other table uses: ``TableScan::Create``, +``TableRead::Create``, ``FileStoreWrite::Create`` and ``FileStoreCommit::Create``, each from its +usual context builder. Each reads the table's schema - from under the table path, or from the one +the context carries - and dispatches to a format table when its ``type`` says so, which is what +Java Paimon does through ``FormatTable.newReadBuilder()`` and ``newBatchWriteBuilder()``. + +``FormatTable`` is the only format-table type in the public API. The classes behind it - +``FormatTableScan``, ``FormatTableRead``, ``FormatTableWrite``, ``FormatTableCommit``, +``FormatDataSplit`` and ``FormatCommitMessage`` - are implementation details under ``src`` and are +named here only to describe what happens. A caller never needs them: a plan comes back as +``Plan``, a split as ``Split``, and a commit message as ``CommitMessage``. + +A catalog that can load a format table itself overrides ``Catalog::LoadFormatTable()``, the +protected hook :cpp:func:`Catalog::GetFormatTable` calls. The file system catalog uses it to say +its metadata lives under the table location, and the REST catalog to take the location and the +schema from one response instead of two that could disagree. A catalog that does not override it +still serves format tables, by reading the location and the schema through the virtuals every +catalog has. + +``TableScan::ListPartitions()`` lists the partitions a scan can see. A format table answers it by +listing directories; every other table type returns ``NotImplemented`` for now. + +Not all of the generic interfaces fit. ``FileStoreCommit`` is mostly about snapshots and manifests - +expiring them, rolling back to one, filtering by a commit identifier recorded in one - and a format +table keeps none of that state, so those calls are refused rather than quietly doing nothing. +``FileStoreWrite::Compact()`` is refused for the same reason, and both write and commit take batch +writes only, since there is no snapshot to record a commit identifier or a watermark in. +``TableScan`` takes a partition filter and a limit; a predicate or a bucket filter is refused. Java +refuses a predicate from ``FormatTableScan.withFilter`` too, but ``FormatReadBuilder.newScan()`` +splits one first and hands the partition half to the scan, so there a predicate over partition +columns still prunes directories. See the limits below. + +Options given at the call win over the ones the schema stored, as they do for every other table - +except ``type``, which is structural and is read from the schema alone, so one read or write +cannot decide what kind of table this is. The merged result is validated, not the schema's own +options, so an option a format table refuses - ``metastore.partitioned-table``, a file format +nothing here can read - is refused wherever it comes from rather than dropped in silence. + +A setting the format path cannot act on is refused by name rather than quietly dropped: + +* ``ReadContextBuilder::SetReadSchema()``. A projected read schema can rename a column, prune a + nested one and give it metadata of its own; a format table's projection is a list of top-level + names, so name the columns instead. +* ``WithStreamingMode()`` on a scan or a write, a global index result on a scan, and a real-time + context on a scan, a read or a write: a format table has no snapshots, no real-time store and no + index. +* ``WriteContextBuilder::WithWriteSchema()``, which names a subset of the columns to write. +* ``WriteContextBuilder::WithWriteId()``, which prefixes a postpone-bucket writer's files so one + compaction reader can put them back in order; a format table has no buckets. +* ``CommitContextBuilder::IgnoreEmptyCommit(false)``, ``UseRESTCatalogCommit(true)`` and + ``AppendCommitCheckConflict(true)``. Keeping an empty commit means writing a snapshot that adds + no files, a rest-catalog commit sends that snapshot to a catalog, and the conflict check reads + the manifests of concurrent commits - none of which exist here. Each is refused only when set + away from its default, so an ordinary commit is unaffected. +* A scan predicate or bucket filter, as above, and more than one partition filter: a scan descends + one directory layout, so it takes the values of a single partition rather than a set of them. + +What a data file is opened with is not one of the refusals. ``EnablePrefetch()``, the read-ahead +cache and its ``CacheConfig``, and the ``Cache`` a read carries all apply, because a format table +opens its files through the same component the managed table path opens its own with. What differs +between the two paths is which files there are and how a row is put back together, not how a file +is read. + +Some settings are not refused because they describe machinery the format path never reaches, and +refusing them would refuse the defaults: ``EnableMultiThreadRowToBatch()`` on a read, a write's +temporary directory and spill configuration, and ``WithIgnoreNumBucketCheck()`` and +``WithIgnorePreviousFiles()`` on a write. They have no effect here: a format read hands out the +batches parquet or orc already produced rather than assembling them from rows, a format write +buffers in memory and never spills, and a table with no buckets has no bucket count to check and +no previous files to read back. + +A write is two-phase, since a directory has no metadata to switch atomically: a file is written +into a ``_temporary`` directory beside where it will end up, under a hidden name of its own, and +only the commit renames it into place. That is the layout Java Paimon's +``RenamingTwoPhaseOutputStream`` stages under. The directory and the name are both hidden, the +convention a Hive-style directory uses for output that is not committed table data, and what a scan +of this table skips. The ``_temporary`` directory is shared with every other writer +of the same table and is left behind after a commit. + +A plan is in-memory only. ``FormatDataSplit`` has no serialized form - ``Split::Serialize()`` +refuses it - and neither has ``FormatCommitMessage``: a format table's plan has no cross-runtime +encoding, so plan, read and commit within one process. + +A ``FormatTableWrite`` and a ``FormatTableCommit`` are each driven by one thread, but separate +ones may fill and add to a table at once: each write stages its files under a uuid of its own, and +each commit publishes only the files its own messages name. Two *overwriting* commits over the +same directory race, since an overwrite clears what is committed there before publishing anything. +A ``FormatTableScan`` may be shared, since planning leaves it as it was. + +``TableRead::CreateCountReader()`` is not implemented for a format table, so counting its rows +means reading them. That is a gap here rather than something the layout forces: ``parquet`` and +``orc`` both record a row count in their own footer. + +A writer starts a new file once the one it is filling reaches ``target-file-row-num`` rows or +``target-file-size`` bytes. Both are checked between batches rather than between rows, because a +batch is the unit this API writes in, so a file may pass either target by up to one batch. Java +checks the row count on every row and the size every thousand rows, and its files therefore sit +closer to the target. + +Aborting a write +---------------- +``FormatTableWrite::Abort()`` removes the files the write staged. It is the one call still allowed +after ``PrepareCommit()``, and that is what it is for: a write dropped *before* preparing clears +its staged files from its own destructor, so only a commit that is prepared and then abandoned +needs it. + +Path containment is checked on the path text, which stops a ``..`` from leaving the table but not +a symbolic link pointing out of it - the same as Java's own local file system behaviour. + +``FormatTableCommit::Abort()`` does the same for the messages a commit was given. **Neither undoes +a commit that succeeded**: once a file has been renamed into place it is no longer staged, and +nothing here will take it back. Java's committer removes the published path as well as the staged +one, which matters there because a commit publishes file by file with nothing watching; here a +commit that fails part way takes its own published files back before it returns, so an abort is +left with the staged files alone. Both are best effort and never fail, so a warning in the log is +the only signal that a file could not be removed. + +Give ``FormatTableCommit`` only the messages this job's own writers produced. A message names a +staged file by path, and a commit can tell that the path belongs to this table, sits in the +partition the message declares, and is staged rather than already published - not whose staged file +it is. A well-formed message from somewhere else is published, or discarded by ``Abort()``, like +any other. + +Relationship to Java Paimon +--------------------------- +Java serves format tables from a Hive or REST catalog, which holds the schema. This implementation +also serves them from a file system catalog, which keeps the schema under the table directory - an +extension Java does not have. Only for such a table are the ``schema`` and ``branch`` directories +below the location treated as metadata rather than as data. + +A file system catalog keeps a table's schema in ``schema`` and its branches in ``branch`` below +the table location, so under ``format-table.partition-path-only-value`` the first partition value +may not be ``schema`` or ``branch``: the directory a write would use is the one holding the +table's own metadata. Such a write is refused, as is an overwrite naming that partition - which +would otherwise delete the schema. A table served from a REST or Hive catalog keeps its schema +elsewhere, so there these are ordinary partition values and are read and written like any other. + +Under that same layout a partition value may not start with ``_`` or ``.`` either, whichever +catalog serves the table: the value is the whole directory name, and a scan skips every hidden +name. Java writes such a directory and then cannot read it back; here the write is refused +instead. The one exception is the value standing for a null partition, ``partition.default-name``, +which the scan reads at a partition level by design. Under the ``key=value`` layout the question +does not arise, since the key in front of the value keeps the directory name visible. + +A few smaller differences come from this library's own conventions: + +* a write takes one partition per batch: the batch declares it through + ``RecordBatch::SetPartition()``, every row is checked against that declaration, and a batch + mixing partitions is refused. Java routes row by row, so one write call there may land in any + number of partitions; +* a write takes its partition from ``RecordBatch::SetPartition()`` rather than from the rows, so Review Comment: Small naming fix: `SetPartition()` is declared on `RecordBatchBuilder`, not on `RecordBatch` - `include/paimon/record_batch.h:126` has ```cpp RecordBatchBuilder& SetPartition(const std::map<std::string, std::string>& data); ``` inside `class PAIMON_EXPORT RecordBatchBuilder` (line 97), and `class PAIMON_EXPORT RecordBatch` (line 39) has no such method. Could you change both mentions here to ``RecordBatchBuilder::SetPartition()``? A reader following the first bullet would otherwise look for it on the wrong type. Every other symbol this page names checks out, for what it is worth - `ReadContextBuilder::SetReadSchema`, `WriteContextBuilder::WithWriteSchema` / `WithWriteId`, `CommitContextBuilder::IgnoreEmptyCommit`, `TableRead::CreateCountReader` and `TableScan::ListPartitions` all match their declarations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
