Benson Muite created SPARK-35317:
------------------------------------
Summary: Job submission in high performance computing environments
Key: SPARK-35317
URL: https://issues.apache.org/jira/browse/SPARK-35317
Project: Spark
Issue Type: Improvement
Components: Spark Submit
Affects Versions: 3.1.2
Reporter: Benson Muite
Spark is often used in high performance computing environments. The default
launcher does not directly support schedulers used such as
[slurm|https://slurm.schedmd.com/overview.html] and [pbs|https://openpbs.org/].
It would be good to support these directly. The repositories
[https://github.com/ekasitk/spark-on-hpc] and
[https://github.com/rokroskar/sparkhpc] contain most of the material necessary
for this, but it may be good to incorporate this in Spark directly so that Java
and Scala are also well supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
