SteNicholas commented on code in PR #222: URL: https://github.com/apache/paimon-cpp/pull/222#discussion_r3840607377
########## docs/source/user_guide/format_table.rst: ########## @@ -0,0 +1,162 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. Ported from the Paimon documentation: +.. https://github.com/apache/paimon/blob/master/docs/docs/concepts/rest/tables.mdx + +.. default-domain:: cpp +.. highlight:: cpp + +Format Table +============ +A format table is a directory that holds multiple files of the same format. It carries no +snapshots and no manifests: the files in the directory are the table, so reading it lists +directories and writing to it adds files. A table is a format table when its ``type`` option is +``format-table``; ``file.format`` then names the format of every file in it, which here is +``parquet`` or ``orc``. + +A partitioned format table uses the standard Hive directory layout, and its partitions are +discovered from that layout rather than from metadata. By default a partition directory is named +``key=value``; setting ``format-table.partition-path-only-value`` names it by the value alone. + +Because a directory of plain files records no row identity, a format table only accepts inserts, +and reads return the table's own columns with no ``_VALUE_KIND`` field. + +Reading and writing +------------------- +A format table is not served through :cpp:func:`Catalog::GetTable`, which describes a managed +table; use :cpp:func:`Catalog::GetFormatTable` instead. It is then read and written through +``FormatTableScan``, ``FormatTableRead``, ``FormatTableWrite`` and ``FormatTableCommit`` rather +than through the managed table path. + +A write is two-phase, since a directory has no metadata to switch atomically: written files are +staged under hidden names that a scan skips, and only the commit renames them into place. + +A ``FormatTableWrite`` and a ``FormatTableCommit`` are each driven by one thread, but separate +ones may fill and add to a table at once: each write stages its files under a uuid of its own, and +each commit publishes only the files its own messages name. Two *overwriting* commits over the +same directory race, since an overwrite clears what is committed there before publishing anything. +A ``FormatTableScan`` may be shared, since planning leaves it as it was. + +``TableRead::CreateCountReader()`` is not implemented for a format table, so counting its rows +means reading them. That is a gap here rather than something the layout forces: ``parquet`` and +``orc`` both record a row count in their own footer. + +A writer starts a new file once the one it is filling reaches ``target-file-row-num`` rows or +``target-file-size`` bytes. Both are checked between batches rather than between rows, because a +batch is the unit this API writes in, so a file may pass either target by up to one batch. Java +checks the row count on every row and the size every thousand rows, and its files therefore sit +closer to the target. + +Aborting a write +---------------- +``FormatTableWrite::Abort()`` removes the files the write staged. It is the one call still allowed +after ``PrepareCommit()``, so a commit that is prepared and then abandoned can still be cleaned up. + +Path containment is checked on the path text, which stops a ``..`` from leaving the table but not +a symbolic link pointing out of it - the same as Java's own local file system behaviour. + +``FormatTableCommit::Abort()`` does the same for the messages a commit was given. **Neither undoes +a commit that succeeded**: once a file has been renamed into place it is no longer staged, and +nothing here will take it back. Both are best effort and never fail, so a warning in the log is the +only signal that a file could not be removed. + +Give ``FormatTableCommit`` only the messages this job's own writers produced. A message names a +staged file by path, and a commit can tell that the path belongs to this table, sits in the +partition the message declares, and is staged rather than already published - not whose staged file +it is. A well-formed message from somewhere else is published, or discarded by ``Abort()``, like +any other. + +Relationship to Java Paimon +--------------------------- +Java serves format tables from a Hive or REST catalog, which holds the schema. This implementation +also serves them from a file system catalog, which keeps the schema under the table directory - an +extension Java does not have. Only for such a table are the ``schema`` and ``branch`` directories +below the location treated as metadata rather than as data. + +A file system catalog keeps a table's schema in ``schema`` and its branches in ``branch`` below +the table location, so under ``format-table.partition-path-only-value`` the first partition value +may not be ``schema`` or ``branch``: the directory a write would use is the one holding the +table's own metadata. Such a write is refused, as is an overwrite naming that partition - which +would otherwise delete the schema. A table served from a REST or Hive catalog keeps its schema +elsewhere, so there these are ordinary partition values and are read and written like any other. + +Under that same layout a partition value may not start with ``_`` or ``.`` either, whichever +catalog serves the table: the value is the whole directory name, and a scan skips every hidden +name. Java writes such a directory and then cannot read it back; here the write is refused +instead. The one exception is the value standing for a null partition, ``partition.default-name``, +which the scan reads at a partition level by design. Under the ``key=value`` layout the question +does not arise, since the key in front of the value keeps the directory name visible. + +Two smaller differences come from this library's own conventions: + +* a write takes one partition per batch: the batch declares it through + ``RecordBatch::SetPartition()``, every row is checked against that declaration, and a batch + mixing partitions is refused. Java routes row by row, so one write call there may land in any + number of partitions; +* a projection that names the same column twice is rejected when the read is built. Java reads + such a column once per entry. + +Current limits +-------------- +Compared with Java Paimon, this implementation does not yet support: + +* the ``csv``, ``json``, ``text`` and ``mosaic`` file formats, leaving ``parquet`` and ``orc``. + All four are line-delimited text in Java, which shares one line-reading layer between them; + this library has no text file format at all, so the first of them to be added has to bring that + layer with it; +* cutting one large data file into byte ranges so that several readers share it. Java does this + only for its line-delimited text formats, which are the ones missing here; ``parquet`` and + ``orc`` each record where their own row groups and stripes begin, and a reader handed a byte + range of one would have to find that out for itself; +* ``metastore.partitioned-table``, which moves partition visibility into the catalog, and the + Hive partition sync that goes with it; +* partition filters beyond equality on partition values, where Java accepts a full predicate. + Partition discovery here also lists one directory level at a time and applies the filter to each + name, while Java turns a leading run of equality constraints into a path and starts listing + below it; a table with many partitions therefore costs more listings here than in Java; +* ``scan.ignore-corrupt-file`` and ``scan.ignore-lost-file``, which are not implemented: a Review Comment: Fixed, thank you: the page now says `scan.ignore-corrupt-files` and `scan.ignore-lost-files`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
