SteNicholas opened a new issue, #170: URL: https://github.com/apache/paimon-cpp/issues/170
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Motivation Java Paimon supports format tables via [`org.apache.paimon.table.FormatTable`](https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/table/FormatTable.java): a table provided as a directory of files in a given file format, organized like a standard Hive table — partitions are discovered from the directory structure, and there is no snapshot or manifest metadata. It is identified by the table option `type` = `format-table` (`Options::TYPE`), and its file format is given by `file.format` (`Options::FILE_FORMAT`). paimon-cpp has no equivalent today, so it cannot read or write these plain Hive-style directory tables. Users that keep data as directories of CSV/Parquet/ORC files (e.g. produced by external engines) cannot access them natively through paimon-cpp, which diverges from the Java table set. ### Solution Introduce native support for format tables, aligned with the Java `FormatTable` semantics. Unlike managed paimon tables, format table reads return plain columns without the `_VALUE_KIND` field, and only INSERT rows can be written. Public API under `include/paimon/table/format/`: - `FormatTable` — table handle: `Create(...)` (load from a path or from a schema), format parsing, and option accessors (`Location`, `GetFormat`, `PartitionKeys`, `FileCompression`, …). - `FormatTableScan` / `FormatDataSplit` / `FormatTableRead` — read path: partition discovery from the directory layout, equality partition filtering, projection, and limit. - `FormatTableWrite` / `FormatCommitMessage` / `FormatTableCommit` — write path: INSERT-only, two-phase commit (data written to hidden `.{name}.tmp` files, renamed to `data-{uuid}-{n}.{ext}` on commit). Supported file formats: `PARQUET`, `ORC`, `CSV`, `TEXT`, `JSON`. This also adds a new **CSV file format module** (`src/paimon/format/csv/`) implementing the `FileFormat` / `FileFormatFactory` interfaces (reader, writer, options, and factory). Format tables can be created and loaded through `FileSystemCatalog`. Intentional deviations from Java: - `FileSystemCatalog` can create format tables. In Java this is only possible via `HiveCatalog`, which paimon-cpp does not have; disallowing creation would make the feature unusable. - `SchemaManager::CreateTable` skips managed-table schema validation for format tables (bucket/manifest options do not apply) but still enforces generic structural invariants (partition keys exist, no duplicates, primitive key types, reserved names). - No offset-based splitting of large files (whole-file splits only). - Read path supports equality partition filtering + projection + limit; no row-level predicate pushdown. - CSV: no `BINARY` type (Java uses Base64), `csv.mode` = `FAILFAST` only, compression `none` only. ### Anything else? <details> <summary><b>Review findings & follow-ups</b> (from the multi-angle review of the initial implementation)</summary> Each finding below was independently verified against the actual code. Fixes for the P1 items and most P2/cleanup items are being prepared. #### P1 — Correctness **1. CSV column pruning can silently read wrong columns.** `CsvFileBatchReader::SetReadSchema` re-maps an arbitrary projection to positions `0..N-1` when the file has no header and the requested field names do not match the generated names `f0..fn`. Since format table data files are written without a header by default (`csv.include-header=false`), projecting a non-prefix / reordered subset reads the wrong file columns. Fix: the read path passes the full non-partition schema to positional formats (CSV/TEXT/JSON) and prunes/reorders after decode, keeping name-based pushdown for self-describing formats (Parquet/ORC). **2. CSV writer never escapes the escape character.** `CsvFormatWriter::EscapeField` only doubles the quote character; a value containing the escape character (default `\`) is written raw, and `CsvParser` consumes a lone escape char and emits nothing. Writing `a\b` reads back as `ab`. Fix: escape the escape character on write (symmetric with the parser). **3. CSV line splitting is not quote-aware.** The writer quotes fields containing the line delimiter, but `SplitLines` splits on the delimiter unconditionally, so a quoted field with an embedded newline is broken into two malformed rows on read. Fix: quote/escape-aware line scanning. **4. Format tables skip all generic schema validation.** `SchemaManager::CreateTable` skips `SchemaValidation::ValidateTableSchema` entirely for format tables, which also drops generic structural invariants (partition keys exist, no duplicates, primitive key types, reserved names). Fix: split generic structural validation from managed-table-specific validation and run the generic part for format tables. **5. Format table write does not validate field nullability.** `FormatTableWrite::Impl::Write` imports the Arrow batch without `ArrowUtils::CheckNullabilityMatch`, so a null in a `nullable=false` field can be silently serialized. Java `FormatTableWrite` performs this validation. Fix: apply the same check. **6. CSV format library missing from test/benchmark link lists.** `TEST_PLUGIN_LINK_LIBS` and `PAIMON_BENCHMARK_STATIC_LINK_LIBS` omit `paimon_csv_file_format`, so the CSV `FileFormatFactory` is not registered in `core_test`/integration/benchmark binaries. #### P2 **7. CSV reader loads the whole file before the first batch; no range splits.** `LoadContent` materializes the entire file into memory; `FormatDataSplit` has no offset/length, so a single multi-GB CSV cannot be split and may OOM. Follow-up: add offset/length to split metadata and a streaming line reader. **8. Reader metrics lost after each file completes.** `FormatTableBatchReader::GetReaderMetrics` only returns the current file reader's metrics; finished readers' metrics are not aggregated and the method returns null after EOF. Fix: aggregate finished + current reader metrics. #### Conventions / reuse / efficiency - `FormatTableUtils::ParseIntOption` hand-rolls `std::stoi` in `try/catch` instead of reusing exception-free `OptionsUtils::GetValueFromMap<int32_t>`. - `CsvOptions` getters use snake_case, violating the PascalCase method naming convention. - Write hot path exports the full Arrow schema per batch; read path re-parses constant partition scalars per batch — both cacheable. - Redundant `Exists()` round-trip per discovered partition during scans. - Duplicated temp-file cleanup loops in `FormatTableWrite::Abort` / `FormatTableCommit::Abort`; dead members in the CSV reader; `SplitPartitionLevels` duplicates `StringUtils::Split`. - The "batches must be released before the reader that produced them" lifetime contract belongs on the `BatchReader` base contract, not only on `FormatTableRead`. #### Follow-ups (not in the first fix pass) - Streaming CSV line reader + offset/length range splits (P2 item 7). - CSV read path: parse fields directly into typed builders instead of building an intermediate `StringArray` then casting. - A shared home for text-format option keys (`csv.field-delimiter`, …). - Reusable data-file naming convention shared between the format table writer and `DataFilePathFactory`. </details> ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
