andygrove opened a new pull request, #2861: URL: https://github.com/apache/datafusion-ballista/pull/2861
# Which issue does this PR close? No issue. # Rationale for this change The executor binary can only use mimalloc (the default, through `build-binary`) or the system allocator. Allocators differ in how much freed memory they keep, which shows up in an executor's resident memory under a container limit, and in speed. jemalloc is a common choice for long-running data processing services, and Comet can already run with it, so being able to build Ballista's executor with jemalloc makes it possible to measure the difference on the same workload. # What changes are included in this PR? - A new opt-in `jemalloc` feature on `ballista-executor` makes `tikv_jemallocator::Jemalloc` the global allocator of the `ballista-executor` binary. It is still wrapped in `AccountingAllocator`, so the allocator-balance metrics keep working. - `jemalloc` takes precedence over `mimalloc`. Because `build-binary` (a default feature) enables `mimalloc`, `cargo install ballista-executor --features jemalloc` is enough to switch, and `--all-features` (which CI's clippy job uses) still builds. When both are enabled the `mimalloc` dependency is marked as used, to keep Cargo's unused-dependency lint quiet. - `tikv-jemallocator` 0.7 is added as a workspace dependency, optional in the executor. - The benchmark image (`benchmarks/cluster/docker/ballista.Dockerfile`) takes a `FEATURES` build arg, so `--build-arg FEATURES=ballista-executor/jemalloc` builds a jemalloc executor. The cluster benchmark README mentions it. - The tuning guide has a short section on choosing the executor's allocator. Tested by building the executor with and without the feature and checking the binary's symbols (jemalloc's symbols and none of mimalloc's with `--features jemalloc`, mimalloc's and none of jemalloc's by default), and by running the `remote-sql` example against a scheduler and a jemalloc executor. # Are there any user-facing changes? A new `jemalloc` Cargo feature on `ballista-executor`, documented in the tuning guide. The default allocator is unchanged. jemalloc is not supported on Windows (MSVC) targets. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
