mbutrovich commented on PR #4982:
URL:
https://github.com/apache/datafusion-comet/pull/4982#issuecomment-5026167940
Thanks @sandugood, that log is exactly what we needed. The instrumentation
pins it, and it is not the delete files:
```
IcebergScan: 13695 tasks, 9 pools (81.6% avg dedup)
Common pools (unique/bytes): schema=1/1868, ..., partition_data=1395/12555,
delete_file=0/0, delete_files_set=0/0, residual=13695/3432661835;
total=3432676470 bytes
```
The `residual` pool is 3.43 GB: 13,695 residuals, one per task, none
deduplicated, averaging ~250 KB each. `delete_file=0/0`, so the delete-file
change in this PR is orthogonal to your failure (it is still a valid fix for
merge-on-read tables, just not the cause here). This is the same overflow, a
different pool.
Two things stand out. A single residual is normally a small reduced
predicate, so ~250 KB each means the scan's filter is very large, and 13,695
distinct residuals means each task carries its own copy. The residual is only
used for row-group pruning on the native side; the post-scan filter still
enforces correctness, so we can safely bound or skip it. I will add a
residual-pool size cap that falls back to no residual pushdown when it would
overflow, as a follow-up to this PR.
To make sure the fix targets the real shape: what does the pushed-down
filter on the source scan look like? The residual comes from Iceberg's per-file
`ResidualEvaluator` against the scan's static predicate, so ~250 KB per
residual points to a large static filter, for example a `WHERE col IN (...)`
with many literals, or a large or complex predicate on a non-partition column.
Is there something like that on the table being read?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
