parthchandra opened a new pull request, #3366: URL: https://github.com/apache/datafusion-comet/pull/3366
## Which issue does this PR close? ## Rationale for this change While running an Iceberg benchmark on a graviton cluster, @hsiang-c found the following calls taking ~12% of the execution time ``` _dl_tlsdesc_dynamic __aarch64_cas8_acq_rel __aarch64_swp8_acq_rel __aarch64_ldadd8_rel ``` The `__aarch64_cas8_acq_rel, __aarch64_swp8_acq_rel, __aarch64_ldadd8_rel` calls can be optimized on graviton V2 and newer ARM processors into a single instruction. The `_dl_tlsdesc_dynamic` call can be optimized to reduce the overhead of thread local storage resolution by the using a static linking model (https://maskray.me/blog/2021-02-14-all-about-thread-local-storage) ## What changes are included in this PR? Adds a separate build for a graviton binary. Loading of the native binary checks for graviton processors and then loads the graviton version if necessary. This can be overridden by specifying the environment variable - `NATIVE_VARIANT_ENV` The build for graviton uses `nightly` in the build container. This is for the `-Z tls-model=initial-exec` flag which is only available in nightly. The build also adds explicit dependencies on `serde` and `serde_urlencoded` which are depended on transitively by `reqwest` and are somehow not resolved while building on graviton. The build adds to the size of the uber jar ``` -rwxr-xr-x@ 1 parth staff 90455632 Feb 2 10:32 ./common/target/classes/org/apache/comet/linux/aarch64/libcomet.so -rwxr-xr-x@ 1 parth staff 98510816 Feb 2 10:20 ./common/target/classes/org/apache/comet/linux/amd64/libcomet.so -rwxr-xr-x@ 1 parth staff 95560232 Feb 2 10:43 ./common/target/classes/org/apache/comet/linux/graviton/libcomet.so ``` The `build-comet-release.sh` script is also updated and makes it a little easier to specify which platforms to build the binary for. Building for graviton is _optional_ ## How are these changes tested? Manually by running the `build-release-comet.sh` script. Script and binary independently verified by @hsiang-c Contributor note: @hsiang-c contributed to this PR. Also Claude Code contributed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
