andygrove opened a new pull request, #7635:
URL: https://github.com/apache/datafusion-comet/pull/7635

   ## Which issue does this PR close?
   
   Part of #7344.
   
   ## Rationale for this change
   
   #7356 added a `jemalloc` Maven profile, but the published jars still use the 
system allocator, which is glibc malloc on Linux. Users who want to try 
jemalloc in an executor, and anyone gathering the data that #7344 asks for, 
have to build Comet from source. Publishing a jemalloc jar with each release 
lets them swap the jar and compare. The jar with the system allocator stays the 
artifact, so nothing changes for existing users.
   
   ## What changes are included in this PR?
   
   - `dev/release/comet-rm/build-comet-native-libs.sh` builds `libcomet.so` 
twice in the same container, with the system allocator and with the `jemalloc` 
Cargo feature, and fails if the two builds are identical. The arm64 jemalloc 
library is built with `JEMALLOC_SYS_WITH_LG_PAGE=16`, because jemalloc built 
for small pages fails every allocation on the arm64 kernels that use 64 KiB 
pages. That also works on kernels with smaller pages, but can raise memory use 
there.
   - `dev/release/build-release-comet.sh` copies both sets of libraries out of 
the containers and builds every Spark and Scala combination twice. The jars 
with the system allocator are installed as before. The script then runs 
`clean`, packages the jemalloc libraries with `-P jemalloc`, and copies each 
jar into the staging repository with the `jemalloc` classifier, for example 
`comet-spark-spark4.1_2.13-1.2.0-jemalloc.jar`. These builds run `package` 
rather than `install`, because installing would replace the jars with the 
system allocator. The jemalloc jar has no pom or sources jar of its own.
   - The script checks that each jar contains the native libraries built for 
its variant, which catches a jar that kept the libraries of an earlier 
incremental build.
   - `publish-to-maven.sh` is unchanged: it already uploads and signs every 
`.jar` and `.pom` under the staging repository, which includes the classifier 
jar.
   - `release_process.md`, the script usage text and `source.md` describe the 
two jars.
   
   `publish_snapshot.yml` is not changed, so snapshots still publish only the 
jar with the system allocator.
   
   ## How are these changes tested?
   
   The Docker half of the release script takes hours and has not been run. The 
shell scripts pass `bash -n`. For the Maven half, I built `spark-4.1` with 
stand-in libraries on a throwaway checkout: a default `package`, then `clean`, 
then a `-P jemalloc` `package`. The jar name matches what the script copies, 
`clean` removes the stale libraries, and the jemalloc jar holds the jemalloc 
libraries. I also ran `check_native_libs` against that jar. It passes for the 
matching variant and fails for the other one. The first release candidate built 
with these scripts is the end-to-end test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to