[
https://issues.apache.org/jira/browse/SPARK-37572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dagang Wei updated SPARK-37572:
-------------------------------
Description:
Currently Spark launches executor processes by constructing and running
commands [1], for example:
{code:java}
/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/jre/bin/java -cp
/opt/spark-3.2.0-bin-hadoop3.2/conf/:/opt/spark-3.2.0-bin-hadoop3.2/jars/*
-Xmx1024M -Dspark.driver.port=35729
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://coarsegrainedschedu...@dagang.svl.corp.google.com:35729 --executor-id 0
--hostname 100.116.124.193 --cores 6 --app-id app-20211207131146-0002
--worker-url spark://Worker@100.116.124.193:45287 {code}
But there are use cases which require more flexible ways of launching
executors. In particular, our use case is that we run Spark in standalone mode,
Spark master and workers are running in VMs. We want to allow Spark app
developers to provide custom container images to customize the job runtime
environment (typically Java and Python dependencies), so drivers and executors
(which run the job code) need to run in Docker containers.
After investigating in the source code, we found that the concept of Spark
Command Runner might be a good solution. Basically, we want to introduce an
optional Spark command runner in Spark, so that instead of running the command
to launch executor directly, it passes the command to the runner, which the
runner then runs the command with its own strategy which could be running in
Docker, or by default running the command directly.
The runner should be customizable through an env variable
`SPARK_COMMAND_RUNNER`, which by default could be a simple script like:
{code:java}
#!/bin/bash
exec "$@" {code}
or in the case of Docker container:
{code:java}
#!/bin/bash
docker run ... – "$@" {code}
I have a patch for that and I've already tested in our environment.
[1]:
[https://github.com/apache/spark/blob/v3.2.0/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala#L52]
was:
Currently Spark launches executor processes by constructing and running
commands [1], for example:
{code:java}
/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/jre/bin/java -cp
/opt/spark-3.2.0-bin-hadoop3.2/conf/:/opt/spark-3.2.0-bin-hadoop3.2/jars/*
-Xmx1024M -Dspark.driver.port=35729
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://coarsegrainedschedu...@dagang.svl.corp.google.com:35729 --executor-id 0
--hostname 100.116.124.193 --cores 6 --app-id app-20211207131146-0002
--worker-url spark://Worker@100.116.124.193:45287 {code}
But there are use cases which require more flexible ways of launching
executors. In particular, our use case is that we run Spark in standalone mode,
Spark master and workers are running in VMs. We want to allow Spark app
developers to provide custom container images to customize the job runtime
environment (typically Java and Python dependencies), so drivers and executors
(which run the job code) need to run in Docker containers.
After investigating in the source code, we found that the concept of Spark
Command Runner might be a good solution. Basically, we want to introduce an
optional Spark command runner in Spark, so that instead of running the command
to launch executor directly, it passes the command to the runner, which the
runner then runs the command with its own strategy which could be running in
Docker, or by default running the command directly.
The runner should be customizable through an env variable
`SPARK_COMMAND_RUNNER`, which by default could be a simple script like:
{code:java}
#!/bin/bash
exec "$@" {code}
or in the case of Docker container:
{code:java}
#!/bin/bash
docker run ... – "$@" {code}
[1]:
[https://github.com/apache/spark/blob/v3.2.0/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala#L52]
> Flexible ways of launching executors
> ------------------------------------
>
> Key: SPARK-37572
> URL: https://issues.apache.org/jira/browse/SPARK-37572
> Project: Spark
> Issue Type: New Feature
> Components: Deploy
> Affects Versions: 3.2.0
> Reporter: Dagang Wei
> Priority: Critical
>
> Currently Spark launches executor processes by constructing and running
> commands [1], for example:
> {code:java}
> /usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/jre/bin/java -cp
> /opt/spark-3.2.0-bin-hadoop3.2/conf/:/opt/spark-3.2.0-bin-hadoop3.2/jars/*
> -Xmx1024M -Dspark.driver.port=35729
> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
> spark://coarsegrainedschedu...@dagang.svl.corp.google.com:35729 --executor-id
> 0 --hostname 100.116.124.193 --cores 6 --app-id app-20211207131146-0002
> --worker-url spark://Worker@100.116.124.193:45287 {code}
> But there are use cases which require more flexible ways of launching
> executors. In particular, our use case is that we run Spark in standalone
> mode, Spark master and workers are running in VMs. We want to allow Spark app
> developers to provide custom container images to customize the job runtime
> environment (typically Java and Python dependencies), so drivers and
> executors (which run the job code) need to run in Docker containers.
> After investigating in the source code, we found that the concept of Spark
> Command Runner might be a good solution. Basically, we want to introduce an
> optional Spark command runner in Spark, so that instead of running the
> command to launch executor directly, it passes the command to the runner,
> which the runner then runs the command with its own strategy which could be
> running in Docker, or by default running the command directly.
> The runner should be customizable through an env variable
> `SPARK_COMMAND_RUNNER`, which by default could be a simple script like:
> {code:java}
> #!/bin/bash
> exec "$@" {code}
> or in the case of Docker container:
> {code:java}
> #!/bin/bash
> docker run ... – "$@" {code}
>
> I have a patch for that and I've already tested in our environment.
>
> [1]:
> [https://github.com/apache/spark/blob/v3.2.0/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala#L52]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
