andygrove commented on PR #5634: URL: https://github.com/apache/datafusion-comet/pull/5634#issuecomment-6018219037
On Spark 3.4, Comet's cache format now stays opt-in (88258a736). `spark.comet.exec.inMemoryCache.enabled` defaults to `true` from Spark 3.5 and to `false` on 3.4, where Spark has no hook for `CometCoalesceShufflePartitions`. The plugin follows the entry's default, so on 3.4 a cached relation keeps Spark's format unless the application sets the config, and a union over it stays Spark's, as on `main`. The Spark 3.4 diff goes back to the one on `main`, since its sessions install Comet's serializer only where the plugin would, and the cache guide and upgrade guide say why 3.4 keeps the old default. A new test pins the default for each version. The cache, Kryo, pruning and plugin suites pass locally on Spark 3.4 and 4.1. `main` is merged in again too. Only #4565 changed the Spark diffs, and each one was merged at the Spark source level and applies to its tag. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
