zhengxiang378928908-code commented on issue #19051:
URL: https://github.com/apache/iceberg/issues/19051#issuecomment-7044016635

   I reproduced this locally. The same symptom also shows up with Flink's 
built-in filesystem connector once source statistics reporting is turned off, 
so it may not be specific to Iceberg.
   
   **Setup:** official `flink:2.1.0-scala_2.12-java17` and 
`flink:2.3.0-scala_2.12-java17` images. The Iceberg runs use 
`iceberg-flink-runtime-2.1-1.11.0.jar` with a Hadoop catalog on the local file 
system (no REST catalog or S3). All runs use the same 4 rows and the same query 
without the `snapshot-id` hint, in batch mode, with `EXPLAIN PLAN FOR` only.
   
   | Source | Flink | `table.optimizer.source.report-statistics-enabled` | 
Result |
   |---|---|---|---|
   | filesystem (CSV) | 2.1.0 | `true` (default) | plan in ~25s, including 
client startup |
   | Iceberg 1.11.0 | 2.1.0 | `true` (default) | still planning after 240s |
   | filesystem (CSV) | 2.1.0 | `false` | still planning after 240s |
   | filesystem (CSV) | 2.3.0 | `true` (default) | plan in ~14s, including 
client startup |
   | filesystem (CSV) | 2.3.0 | `false` | still planning after 240s |
   
   `FileSystemTableSource` implements `SupportsStatisticReport`, and the CSV 
format can report an estimated row count. `IcebergTableSource` does not report 
statistics through that interface, either in [1.11.0 for Flink 
2.1](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/source/IcebergTableSource.java)
 or on `main` (1.20, 2.2, 2.3).
   
   In the Iceberg 1.11.0 / Flink 2.1.0 run, a thread dump taken after ~200s 
showed the planning thread CPU-bound in batch physical optimization, with about 
200s of CPU time:
   
   ```
   VolcanoPlanner.findBestExp
     IterativeRuleDriver.drive
       FlinkExpandConversionRule.onMatch / satisfyTraitsByInput
         VolcanoRuleCall.transformTo -> VolcanoPlanner.register -> 
propagateCostImprovements
           BatchPhysicalHashAggregateBase.computeSelfCost -> 
RelMetadataQuery.getAverageRowSize
   ```
   
   I haven't confirmed that the CSV runs with statistics disabled go through 
the same planner path, and I haven't tested whether an Iceberg row estimate 
makes the Iceberg case finish. Still, the results point to Flink's planner 
struggling with this query shape when the source provides no statistics.
   
   Possible follow-ups:
   
   1. Flink side: report the long planning time, using the filesystem connector 
with `report-statistics-enabled = false` as a reproducer that doesn't need 
Iceberg.
   2. Iceberg side: implement `SupportsStatisticReport` in 
`IcebergTableSource`. A first version could return a row-count estimate from 
the selected snapshot's `total-records` summary for batch scans, and 
`TableStats.UNKNOWN` when the value is missing. That is a snapshot-level 
estimate: row-level deletes and pushed-down filters can make the actual scan 
smaller. Whether this avoids the hang would need testing, but it would give the 
planner better estimates in general.
   
   I'm happy to try (2) and check it against this reproducer if that direction 
sounds reasonable.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

  • [I] Flink: EXPLAIN / quer... 2162174202-sudo (via GitHub) via issues
    • Re: [I] Flink: EXPLA... zhengxiang378928908-code (via GitHub) via issues

Reply via email to