Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r175518159
--- Diff:
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh ---
@@ -53,14 +53,10 @@ fi
case "$SPARK_K8S_CMD" in
driver)
CMD=(
- ${JAVA_HOME}/bin/java
- "${SPARK_JAVA_OPTS[@]}"
- -cp "$SPARK_CLASSPATH"
- -Xms$SPARK_DRIVER_MEMORY
- -Xmx$SPARK_DRIVER_MEMORY
- -Dspark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS
- $SPARK_DRIVER_CLASS
- $SPARK_DRIVER_ARGS
+ "$SPARK_HOME/bin/spark-submit"
+ --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS"
+ --deploy-mode client
+ "$@"
--- End diff --
We've seen some users override the entry point to do a bunch of extra
bootstrapping before calling into the actual command to run the driver in
custom images. I vaguely recall something related to Openshift doing this -
@erikerlandson.
The point is, even if `entrypoint.sh` defines the contract, what arguments
would a custom implementation pass it? Something like this would fail because
we didn't pass the appropriate arguments to the entrypoint:
```
#!/bin/bash
# my-entrypoint.sh
source /opt/my-custom.script.sh
/opt/spark/entrypoint.sh # Notice we don't pass arguments
```
So the contract may be, that whatever arguments the user's entrypoint is
passed, they must eventually call `entrypoint.sh` with all of those arguments
forwarded through. That might be an acceptable contract. With environment
variables we can say that it doesn't matter what arguments the downstream
entrypoint is passed, just that you eventually need to run that code and it
knows what to do if the application submission code set up the container's
environment properly.
Having defined these ideas more precisely, I think either contract is
acceptable here, with a slight preference towards having Spark interpret the
environment rather than the arguments.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
