SteNicholas opened a new pull request, #222:
URL: https://github.com/apache/paimon-cpp/pull/222
### Purpose
Linked issue: close #170
A format table is a directory of same-format data files with no snapshots
and no manifests, laid
out like a standard Hive table. It is loaded through
`Catalog::GetFormatTable()` and read and
written through `FormatTableScan`, `FormatTableRead`, `FormatTableWrite` and
`FormatTableCommit`
rather than through the managed table path.
This adds:
- `FormatTable` and its scan, read, write and commit, with partition
discovery from the directory
layout, split merging, projection, predicate pushdown, limit, and
two-phase writes that stage
their output under hidden names until the commit renames it into place.
- A csv file format plugin - parser, reader, writer, options and compression
- usable on its own,
with byte-range reads of an uncompressed file so that one large file can
be read by several
readers.
- `FormatTableCatalog`, a capability interface a catalog implements when it
can load a format
table itself, so that `Catalog` gains no vtable slot; the rest catalog
takes the location and
the schema from one response rather than from two that could disagree.
- `TextReaderOptions`, the same shape on the format side, so a text format
can be told the file
schema, path and byte range without changing `ReaderBuilder`.
- Create-time and open-time schema validation through one shared entry
point, so a table one
catalog accepts is one every catalog can open.
- Path containment, visibility and partition-binding checks on every public
split and commit
message, since both are documented as decodable from untrusted bytes.
### Tests
**These tests have not been run.** The change was authored under an explicit
constraint not to
compile, build or run anything, so every case below is new or updated source
that still needs a
first execution. Please treat CI as the first real signal.
New test files:
- `src/paimon/core/table/format/format_table_test.cpp` - scan, read, write,
commit, overwrite,
partition layouts, path and partition validation, serialization
round-trips.
- `src/paimon/core/table/format/format_file_listing_test.cpp`
- `src/paimon/core/table/format/format_file_naming_test.cpp`
- `src/paimon/core/table/format/lazy_concat_batch_reader_test.cpp`
- `src/paimon/core/table/format/limit_batch_reader_test.cpp`
- `src/paimon/core/table/format/partition_completing_batch_reader_test.cpp`
- `src/paimon/format/csv/` - `csv_parser_test.cpp`, `csv_options_test.cpp`,
`csv_format_writer_test.cpp`, `csv_file_batch_reader_test.cpp`,
`csv_compression_test.cpp`,
`csv_value_converter_test.cpp`, `csv_file_format_factory_test.cpp`.
Updated:
- `src/paimon/rest/rest_catalog_test.cpp` - format table loaded in one
request, create-time
validation refusing a schema this client could not open, a response
without a path.
- `src/paimon/core/catalog/file_system_catalog_test.cpp` - `GetTable`
refusing a format table.
Commands to run:
```bash
cmake --build build --target unittest -j "$(nproc)"
./build/debug/unittest
--gtest_filter='FormatTable*:Csv*:LazyConcat*:Limit*:PartitionCompleting*'
./build/debug/csv_format_test
pre-commit run --files $(git diff --name-only HEAD~1 HEAD)
```
### API and Format
Yes, this adds public API. New headers under `include/paimon/`:
- `paimon/table/format/format_table.h`, `format_table_scan.h`,
`format_table_read.h`,
`format_table_write.h`, `format_table_commit.h`, `format_data_split.h`,
`format_commit_message.h`
- `paimon/catalog/format_table_catalog.h`
- `paimon/format/text_reader_options.h`
`Catalog` gains `GetFormatTable()` as a **non-virtual** member: it
dispatches through a
`dynamic_cast` to the new `FormatTableCatalog` capability interface, so no
vtable slot is added
and a catalog compiled against an older header keeps working.
`ReaderBuilder` is unchanged for the
same reason - the text-format hooks live in `TextReaderOptions` instead.
New storage format: none. `FormatDataSplit` and `FormatCommitMessage` do
define a serialization,
but java paimon's cross-runtime split protocol has no format table split, so
that encoding is this
library's own and round-trips only here. It is documented as such in
`include/paimon/table/source/split.h`.
`cmake_modules/ThirdpartyToolchain.cmake` turns on `ARROW_WITH_BZ2`, needed
for bzip2 csv files.
### Documentation
Yes:
- `docs/source/user_guide/format_table.rst` - the layout, reading and
writing, aborting, the
relationship to java paimon, and a "Current limits" section listing every
difference that
remains (json/text/mosaic formats, snappy/lz4,
`metastore.partitioned-table`, partition filters
beyond equality, `scan.ignore-corrupt-file`, the csv `TIME` type,
all-partition-column tables,
`TIMESTAMP`/`DECIMAL`/`FLOAT`/`DOUBLE` partition columns, duplicate
projection, case-insensitive
`type`, and compressed csv held in memory).
- `docs/source/api/format_table.rst` - the API reference, with
`FormatTableCatalog` and
`TextReaderOptions` added to `api/catalog.rst` and `api/file_format.rst`.
- `docs/source/build_system.rst` - the new
`Paimon::paimon_csv_file_format_shared` target.
The accurate summary is a **documented subset** of java paimon's format
table, not full behavioural
compatibility.
### Generative AI tooling
Generated-by: Claude Code (Claude Opus 5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
