hankivstmb opened a new issue, #1536:
URL: https://github.com/apache/iceberg-go/issues/1536
### Apache Iceberg version
main (also reproduced on v0.5.0 and v0.6.0)
### Please describe the bug
A filtered table scan can silently return zero rows (no error) when a scan
task earlier in plan order produces no record batches — for example when every
row group of its file is pruned by parquet row-group statistics.
**Root cause.** `createIterator` (table/arrow_scanner.go) sequences per-task
records and only releases task N+1's records after task N emitted a record with
`Last=true`. The `dropFile` path emits an empty sentinel batch for statically
skipped files, but `processRecords` only sends records when `prev != nil` — a
task whose reader yields no batches emits nothing at all. The first such task
stalls the sequenced channel; when the channel closes, the iterator's deferred
drain releases every queued record from the tasks behind it without yielding
them.
**Trigger conditions.** ≥2 planned files where an earlier-ordered file (a)
survives file-level pruning (e.g. its manifest entry has no column bounds —
common for files registered via `NewDataFileBuilder`/`AddDataFiles` by external
committers) and (b) has all of its row groups eliminated by row-group
statistics under the scan filter.
**Real-world impact.** We hit this in production reading a table populated
by a 2-minute micro-batch committer plus AWS Glue compaction: any time-windowed
filtered read returned 0 rows while Athena/Trino returned millions from the
same snapshot. Nothing fails — the data is just missing.
**Reproduction.** The linked PR adds `TestScanSurvivesFullyPrunedTask`: two
statless data files registered in one commit, a filter that fully prunes the
first file's row groups; before the fix the second file's rows are silently
dropped (`expected: []int32{1000, 1001, 1002}, actual: nil`), and a checked
allocator additionally reports the stalled record leak.
**Fix.** Emit the same empty sentinel batch the `dropFile` path uses when a
task's reader produced no records. PR incoming.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
