This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new f3d7e26aafa6 [SPARK-57700][K8S] Support opt-in `jemalloc` in Spark
Docker images
f3d7e26aafa6 is described below
commit f3d7e26aafa6c3694adb35308a9893c6b76c93e1
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Jun 25 12:40:32 2026 -0700
[SPARK-57700][K8S] Support opt-in `jemalloc` in Spark Docker images
### What changes were proposed in this pull request?
This PR aims to install `libjemalloc2` in the Spark Docker image while the
default allocator is unchanged — the library is not preloaded unless a user
opts in:
```
spark.executorEnv.LD_PRELOAD=libjemalloc.so.2
spark.kubernetes.driverEnv.LD_PRELOAD=libjemalloc.so.2
```
### Why are the changes needed?
[jemalloc](http://jemalloc.net) reduces native/off-heap memory
fragmentation, especially `off-heap`, `PySpark`, `Apache Arrow`, and
`Spark-accellerator` use-cases. Shipping the package lets operators enable it
with one env var instead of building a custom image, with no regression for
anyone else.
### Does this PR introduce _any_ user-facing change?
No behavior change because the default allocator is unchanged; the image
just additionally ships `libjemalloc2`.
```
$ ls -al /usr/lib/aarch64-linux-gnu/libjemalloc.so.2
-rw-r--r-- 1 root root 527176 Apr 18 2024
/usr/lib/aarch64-linux-gnu/libjemalloc.so.2
```
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
Closes #56782 from dongjoon-hyun/SPARK-57700.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit d1bcb02a73a1d354b8555fda57827f1dc71f7ae2)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../kubernetes/docker/src/main/dockerfiles/spark/Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
index f7c7908aeab9..c7ec49cebbc1 100644
--- a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
+++ b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
@@ -36,7 +36,7 @@ ARG spark_uid=185
RUN set -ex && \
apt-get update && \
ln -s /lib /lib64 && \
- apt install -y --no-install-recommends bash tini libc6 libpam-modules
krb5-user libnss3 procps net-tools logrotate libssl-dev && \
+ apt install -y --no-install-recommends bash tini libc6 libjemalloc2
libpam-modules krb5-user libnss3 procps net-tools logrotate libssl-dev && \
mkdir -p /opt/spark && \
mkdir -p /opt/spark/examples && \
mkdir -p /opt/spark/work-dir && \
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
