andygrove opened a new pull request, #7238: URL: https://github.com/apache/datafusion-comet/pull/7238
## Which issue does this PR close? Part of #6834. ## Rationale for this change Spark builds a cache from the batches of its plan only when the plan supports columnar output. Before Spark 4.0 a cached `AdaptiveSparkPlanExec` does not, so on Spark 3.4 and 3.5 a relation cached under AQE (on by default) goes through the plan's native operators, a columnar-to-row conversion and `UnsafeRow`, and is written into Arrow again before it is compressed. Spark 4.0 and later, and any version without AQE, write the plan's Arrow batches as they are. Comet cannot change this without a hook in Spark. The user guide's build timings (944 ms against Spark's 3285 ms for Comet operators) were measured on Spark 4.1 and did not say so for the cache they describe, and its Limitations section did not mention the 3.x behaviour. #6834 lists this as the one performance item where the right fix is to document the limit. ## What changes are included in this PR? - `in-memory-cache.md` says in its Limitations section that on Spark 3.4 and 3.5 a relation cached under AQE is built from rows, why, and that the build timings in the guide are from Spark 4.1 and do not apply to that case. The paragraph that introduces the build timings points to it. The guide does not give a build timing for 3.x, because none was measured. - A new `CometInMemoryCacheSuite` test pins the condition the note rests on. It caches a Comet aggregate with AQE on and off and asserts that the cached plan is an `AdaptiveSparkPlanExec` exactly when AQE is on, and that it reports columnar output unless AQE is on before Spark 4.0. ## How are these changes tested? The new test passes on Spark 3.4, 3.5, 4.0 and 4.1. The strict-warnings build and Prettier on the guide pass. I did not time a 3.x build, so the guide makes no claim about how much slower it is. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
