XiaoHongbo-Hope opened a new pull request, #8845: URL: https://github.com/apache/paimon/pull/8845
### Purpose Format table split planning lists every partition's files serially in `FormatTableScan.splits()` (`fileIO.listFiles(path, true)` per partition). For internal (catalog-managed) tables with many partitions this is the dominant cost of planning — thousands of sequential object-store LIST calls on the coordinator (e.g. a ~5600-partition table spends minutes in `count(*)` planning). Parallelize the per-partition listing for internal tables only, keeping the external (filesystem-discovered) path serial and unchanged. Extract the per-partition "list files and build splits" step into a `PartitionSplitPlanner` strategy, selected by the existing internal/external discriminator (`partitionManager != null`), so `splits()` stays neutral with a single selection point: - `SequentialSplitPlanner` (external): unchanged serial path; a missing partition directory is rethrown. - `ParallelSplitPlanner` (internal): bounded, order-preserving fan-out via `ManifestReadThreadPool.randomlyExecuteSequentialReturn`; a registered partition without a directory reads as empty (matching Hive), and any other listing failure fails the whole scan (no truncated results). The per-partition `Arrays.sort` and `BinPacking.packForOrdered` are unchanged and the fan-out is order-preserving, so the produced splits are identical to the serial path. Parallelism is bounded by a new option `format-table.scan.list-parallelism` (default: number of processors); external tables ignore it and always list serially. The executor is a shared static pool, so `FormatTable` / `FormatTablePartitionManager` stay serializable and no `FileIO` is pushed into the manager. ### Tests `CatalogManagedPartitionScanTest`: - internal parallel (8) vs serial (1) listing produce identical, correctly ordered splits and match the expected files; - an internal registered partition without a directory is treated as empty (warn); - a non-`FileNotFound` listing error on any internal partition fails the whole scan; - external tables still rethrow on a missing partition directory. Full `FormatTableScanTest` / `CatalogFormatTablePartitionManagerTest` / `CatalogManagedPartitionScanTest` pass with no regression (124 tests). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
