andygrove commented on PR #5543: URL: https://github.com/apache/datafusion-comet/pull/5543#issuecomment-5796403613
Thanks @mbutrovich, all three are addressed in 20dfd0838, with details on the threads. The codec table is worth pulling up here, because the answer changed the docs and not just the benchmark. The old numbers had zstd winning on speed as well as size. With the codec now varied by the benchmark, `none` is faster at everything but footprint. zstd stays the default, but on footprint grounds, and the docs and the config doc now say so. Regenerating the numbers also turned up a build break this PR introduced. Any build without `clean` in which no class has changed, such as a second `make release`, fails in shade with a duplicate `META-INF/services` entry. The jar plugin leaves the already-shaded jar in place, and the services transformer then writes the relocated name twice. The fix is an exclude in `spark/pom.xml`. I reproduced the failure without it, and both clean and non-clean builds pass with it. Verified on the default profile: `CometInMemoryCacheSuite` 49/49, with `CometInMemoryCacheKryoSuite` and `UtilsSuite` green. It compiles against Spark 3.4, 3.5 and 4.0 as well as 4.1, and scalafix, scalastyle and spotless are clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
