Zouxxyy opened a new pull request, #8317: URL: https://github.com/apache/paimon/pull/8317
### Purpose Format tables (`format-table.implementation = paimon`) currently generate one split per data file, which produces a large number of splits/tasks when a directory contains many small files. This change lets a single split carry multiple files, packed in the core scan by `source.split.target-size` (with `source.split.open-file-cost` as a per-file weight floor, mirroring `AppendOnlySplitGenerator`). Because packing happens in `FormatTableScan`, it is engine-agnostic; the reader concatenates a split's files via `ConcatRecordReader`. - `FormatDataSplit` now holds a `List<FileMeta>` (path/size/offset/length) instead of a single file, with `files()/partition()/totalSize()/fileCount()`. - `FormatTableScan#createSplits` sorts files by path for deterministic packing and bin-packs segments with `max(readSize, openFileCost)` as the weight; large CSV/JSON offset-slicing is preserved. - `FormatReadBuilder#createReader` builds one reader per file and chains them with `ConcatRecordReader`. - On Spark, `SplitUtils` reports size/file-count over the multi-file split, and each `FormatDataSplit` maps to one input partition (no extra re-packing). ### Tests - New e2e in `PaimonFormatTableTest`: with the default target size three files pack into one split; with target size = one file size each file becomes its own split; query results are unchanged in both cases. - Updated `FormatDataSplitTest` / `FormatReadBuilderTest` / `FormatTableScanTest` for the multi-file API. All core format tests pass (61) and the Spark format-table suite passes (16). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
