[ https://issues.apache.org/jira/browse/SPARK-32429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164440#comment-17164440 ]
Thomas Graves commented on SPARK-32429: --------------------------------------- [~jiangxb1987] [~Ngone51] thoughts on the above proposal? > Standalone Mode allow setting CUDA_VISIBLE_DEVICES on executor launch > --------------------------------------------------------------------- > > Key: SPARK-32429 > URL: https://issues.apache.org/jira/browse/SPARK-32429 > Project: Spark > Issue Type: Improvement > Components: Deploy > Affects Versions: 3.0.0 > Reporter: Thomas Graves > Priority: Major > > It would be nice if standalone mode could allow users to set > CUDA_VISIBLE_DEVICES before launching an executor. This has multiple > benefits. > * kind of an isolation in that the executor can only see the GPUs set there. > * If your GPU application doesn't support explicitly setting the GPU device > id, setting this will make any GPU look like the default (id 0) and things > generally just work without any explicit setting > * New features are being added on newer GPUs that require explicit setting > of CUDA_VISIBLE_DEVICES like MIG > ([https://www.nvidia.com/en-us/technologies/multi-instance-gpu/]) > The code changes to just set this are very small, once we set them we would > also possibly need to change the gpu addresses as it changes them to start > from device id 0 again. > The easiest implementation would just specifically support this and have it > behind a config and set when the config is on and GPU resources are > allocated. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org