LuciferYang commented on PR #55579: URL: https://github.com/apache/spark/pull/55579#issuecomment-6602663552
@pan3793 thanks for the review. Addressed in 7158934 and e721ae7: 1. The memo now holds the file listing, and `FileScan.partitions` is split into a `filePartitions(selectedPartitions)` step. The runtime path filters the listed directories first, then splits and packs the survivors for their own size, as V1 does. "runtime pruning splits the surviving files for their own size" uses your `minPartitionNum=20` setup on small files and checks that a scalar-subquery scan and a DPP scan get the same splits as selecting those partitions at compile time. 2. A runtime filter that fails to evaluate on a directory now keeps it. After dongjoon's review the file scans no longer declare their partition columns fully pushed, so a scalar-subquery filter keeps its `FilterExec` and is evaluated again like a DPP filter. One test covers both kinds with ANSI on. 3. An empty array now returns the partitions `planInputPartitions()` planned, without packing the files again. 4. `replanWithRuntimeFilters` now calls `planInputPartitionsWithRuntimeFilters` with an empty array when all of a node's runtime filters are screened out, so a group-based scan sees every node that has runtime filters. I updated the interface doc and the fixture comment, and added a test where a DPP filter degraded to `true` still reaches the scan. 5. Split out to SPARK-60840 (#60053), with the in-memory table repro, for master and branch-4.x. This PR keeps the same change until that one merges, then I'll merge master here. 6. I'll rewrite the description once CI is green. 7. I shortened the comments in the three files and removed the history wording. The `holdsLock` test is replaced by one where another thread reads the scan while its files are listed. That thread blocks if the listing holds the scan's monitor. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
