Revanth14 opened a new issue, #2429: URL: https://github.com/apache/iceberg-go/issues/2429
### Problem `ScanReport` counts manifests (total / scanned / skipped) and the result (files, bytes, deletes), but not the live entries inside scanned manifests that the partition evaluator or the inclusive metrics evaluator rejected. That is the file-level pruning step, and without it a report can't show how much a filter actually pruned. The omission is a documented follow-up from #1501 (comments on `scanMetricsAccumulator` and `buildScanReport` in `table/scan_metrics.go`). The wire fields (`metrics.ScanMetricsResult.SkippedDataFiles` / `SkippedDeleteFiles`) and the OTel mapping (`iceberg.scan.data_files.skipped`) already exist; only the counting is missing. Example: planning `wikipedia.pageviews_2026` from Google's Lakehouse public datasets (3,001 files, 8 data manifests) with a partition + column filter reports `scanned-data-manifests=1`, `result-data-files=147`, and nothing about how many of the scanned manifest's entries were pruned. ### Proposal Count rejected live entries in the per-manifest goroutines with goroutine-local counters and merge them into the accumulator after the errgroup barrier, the same no-atomics pattern #2149 uses for task finalization. Data manifests feed `skipped-data-files`, delete manifests feed `skipped-delete-files`. Deleted entries are not counted and pruned manifests stay in `skipped-data-manifests` only, matching Java (`ManifestReader.entries()`, `ManifestGroup`). Covers `PlanFiles` and the incremental append/changelog scans; remote planning emits no report and is unchanged. Emit `0` rather than omitting the field when nothing is pruned, as Java does. Open question: Java also counts entries dropped by `ignoreExisting` in incremental scans. I'd include the equivalent `discardExisting` skip for parity unless there's a reason not to. ### Context Part of #485, follow-up to #1501 (proposal #1236). `indexed-delete-files` is deferred by the same comment and would be a separate follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
