peterxcli opened a new pull request, #5733: URL: https://github.com/apache/datafusion-comet/pull/5733
## Which issue does this PR close? Closes #5245. ## Rationale for this change AQE does not recognize `CometInMemoryTableScanExec` as a table-cache stage. On first access to a cached join, this prevents AQE from using the materialized cache's partitioning to remove redundant shuffles and sorts. Recognizing the cache stage also requires Comet's native input traversal to stop at that stage. ## What changes are included in this PR? - Implement `InMemoryTableScanLike` on Spark 3.5+ through version-specific inheritance shims, with shared implementations exposing cache materialization, the underlying cache RDD, and runtime statistics. Retain a `LeafExecNode` shim for Spark 3.4, which has no table-cache stage API. - Treat `QueryStageExec` as a native input boundary so table-cache stages are handled alongside shuffle and broadcast stages. - Port three AQE regressions from Spark's `AdaptiveQueryExecSuite`, with source links: cold/warm cache materialization and shuffle/sort elimination, partition preservation beside a table-cache stage, and cache statistics used for join selection. ## How are these changes tested? - Spark 4.1: all 36 tests in `CometInMemoryCacheSuite` passed; the three new AQE regressions also passed after consolidating the implementation into the shared class. - Spark 3.5: all three new AQE regressions passed. - Spark 3.4 / JDK 17: the cache-statistics regression passed; the two table-cache-stage tests are explicitly skipped because Spark 3.4 lacks that API. - `spotless:check`, `scalastyle:check`, and `git diff --check`. - An exploratory before/after run on Spark 3.5.9 (250,000 rows per cached join input, 10 measured samples per variant and phase) produced correct results and reduced cold outer-query shuffles from two to one. Competing builds and a debug native library make the latency comparison inconclusive; this PR makes no measured speedup claim. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
