hankivstmb opened a new pull request, #1537:
URL: https://github.com/apache/iceberg-go/pull/1537
Closes #1536.
## What
A filtered scan silently returns zero rows (no error) when a scan task
earlier in plan order produces no record batches — e.g. when every row group of
its file is pruned by parquet row-group statistics.
`createIterator` sequences per-task records and only releases task N+1's
records after task N emitted a record with `Last=true`. The `dropFile` path
already emits an empty sentinel batch for statically skipped files, but a task
whose reader yields no batches emitted nothing: the first such task stalls the
sequenced channel, and on close the deferred drain releases every queued record
from later tasks without yielding them.
## Fix
Emit the same empty sentinel batch the `dropFile` path uses when
`processRecords` finishes with no records produced.
## Test
`TestScanSurvivesFullyPrunedTask`: two statless data files (registered via
`NewDataFileBuilder`/`AddDataFiles` in one commit, so plan order is
deterministic and file-level pruning cannot eliminate either), scanned with a
filter that fully prunes the first file's row groups.
- Before: `expected: []int32{1000, 1001, 1002}, actual: nil` — the second
file's rows are silently dropped; a checked allocator additionally reports the
stalled record leak.
- After: the second file's rows are returned; allocator clean.
Found in production: a table populated by a micro-batch committer plus AWS
Glue compaction returned 0 rows for every windowed filtered read in iceberg-go
while Athena/Trino returned millions from the same snapshot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
