[
https://issues.apache.org/jira/browse/SPARK-27378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841412#comment-16841412
]
Thomas Graves commented on SPARK-27378:
---------------------------------------
Spark 3.0 already added support for requesting any resource from YARN via the
configs: spark.yarn.\{executor/driver/am}.resource, so the changes required for
this Jira are simply to map the new spark configs:
spark.\{executor/driver}.resource.\{fpga/gpu}.count into the corresponding yarn
configs. For other resource types we can't map them though because we don't
know what they are called on the yarn side. So for any other resource they
will have to specify both configs spark.yarn.\{executor/driver/am}.resource and
spark.\{executor/driver}.resource.\{fpga/gpu}. That isn't ideal but the only
other option would be to have some sort of mapping the user would pass in. We
can always add more yarn resource types if it adds them. The main 2 people are
interested in seem to be gpu and fpga anyway, so I think for now this is fine.
> spark-submit requests GPUs in YARN mode
> ---------------------------------------
>
> Key: SPARK-27378
> URL: https://issues.apache.org/jira/browse/SPARK-27378
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Submit, YARN
> Affects Versions: 3.0.0
> Reporter: Xiangrui Meng
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
