Yikun Jiang created SPARK-43367:
-----------------------------------
Summary: Recover sh support in Dockerfiles
Key: SPARK-43367
URL: https://issues.apache.org/jira/browse/SPARK-43367
Project: Spark
Issue Type: Sub-task
Components: Spark Docker
Affects Versions: 3.5.0
Reporter: Yikun Jiang
rm /bin/sh && ln -sv /bin/bash /bin/sh
Does spark expect /bin/sh to be bash (in posix mode) and not just a generic
posix compliant shell? Is there a bug with using dash? Minimally, this should
do a dpkg --divert to ensure that users in dependent images don't accidentally
revert it with package updates.
env | grep SPARK_JAVA_OPT_ |...
This is susceptible to a few bugs particularly around newlines in values. I see
two ways around it.
ensuring the matching name is at the start, -E '^SPARK_JAVA_OPT_', and running
all the commands with null-terminated input and output: env -0 and -z on the
other commands
or bash variable prefix expansion
for v in "${!SPARK_JAVA_OPT_@}"; do
SPARK_EXECUTOR_JAVA_OPTS+=( "${!v}" )
done
[1]
https://github.com/docker-library/official-images/pull/13089?notification_referrer_id=NT_kwDOABp-orI0MzIwMzMwNzY5OjE3MzYzNTQ#issuecomment-1533540388
[2] related origina change:
https://github.com/apache-spark-on-k8s/spark/pull/444/files
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
