zhengxiang378928908-code commented on issue #19051: URL: https://github.com/apache/iceberg/issues/19051#issuecomment-7044016635
I reproduced this locally. The same symptom also shows up with Flink's built-in filesystem connector once source statistics reporting is turned off, so it may not be specific to Iceberg. **Setup:** official `flink:2.1.0-scala_2.12-java17` and `flink:2.3.0-scala_2.12-java17` images. The Iceberg runs use `iceberg-flink-runtime-2.1-1.11.0.jar` with a Hadoop catalog on the local file system (no REST catalog or S3). All runs use the same 4 rows and the same query without the `snapshot-id` hint, in batch mode, with `EXPLAIN PLAN FOR` only. | Source | Flink | `table.optimizer.source.report-statistics-enabled` | Result | |---|---|---|---| | filesystem (CSV) | 2.1.0 | `true` (default) | plan in ~25s, including client startup | | Iceberg 1.11.0 | 2.1.0 | `true` (default) | still planning after 240s | | filesystem (CSV) | 2.1.0 | `false` | still planning after 240s | | filesystem (CSV) | 2.3.0 | `true` (default) | plan in ~14s, including client startup | | filesystem (CSV) | 2.3.0 | `false` | still planning after 240s | `FileSystemTableSource` implements `SupportsStatisticReport`, and the CSV format can report an estimated row count. `IcebergTableSource` does not report statistics through that interface, either in [1.11.0 for Flink 2.1](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/source/IcebergTableSource.java) or on `main` (1.20, 2.2, 2.3). In the Iceberg 1.11.0 / Flink 2.1.0 run, a thread dump taken after ~200s showed the planning thread CPU-bound in batch physical optimization, with about 200s of CPU time: ``` VolcanoPlanner.findBestExp IterativeRuleDriver.drive FlinkExpandConversionRule.onMatch / satisfyTraitsByInput VolcanoRuleCall.transformTo -> VolcanoPlanner.register -> propagateCostImprovements BatchPhysicalHashAggregateBase.computeSelfCost -> RelMetadataQuery.getAverageRowSize ``` I haven't confirmed that the CSV runs with statistics disabled go through the same planner path, and I haven't tested whether an Iceberg row estimate makes the Iceberg case finish. Still, the results point to Flink's planner struggling with this query shape when the source provides no statistics. Possible follow-ups: 1. Flink side: report the long planning time, using the filesystem connector with `report-statistics-enabled = false` as a reproducer that doesn't need Iceberg. 2. Iceberg side: implement `SupportsStatisticReport` in `IcebergTableSource`. A first version could return a row-count estimate from the selected snapshot's `total-records` summary for batch scans, and `TableStats.UNKNOWN` when the value is missing. That is a snapshot-level estimate: row-level deletes and pushed-down filters can make the actual scan smaller. Whether this avoids the hang would need testing, but it would give the planner better estimates in general. I'm happy to try (2) and check it against this reproducer if that direction sounds reasonable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
