[jira] [Commented] (SPARK-27488) Driver interface to support GPU resources
[ https://issues.apache.org/jira/browse/SPARK-27488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839402#comment-16839402 ] Thomas Graves commented on SPARK-27488: --- With this Jira we need to move the function that does the checks for various configs to a common place and we need to address the comment here: [https://github.com/apache/spark/pull/24406/files/b9dacef1d7d47d19df300628d3841b3a13c03547#r283617243] To make it more clear on variables names. > Driver interface to support GPU resources > -- > > Key: SPARK-27488 > URL: https://issues.apache.org/jira/browse/SPARK-27488 > Project: Spark > Issue Type: Story > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Major > > We want to have an interface to allow the users on the driver to get what > resources are allocated to them. This is mostly to handle the case the > cluster manager does not launch the driver in an isolated environment and > where users could be sharing hosts. For instance in standalone mode it > doesn't support container isolation so a host may have 4 gpu's, but only 2 of > them could be assigned to the driver. In this case we need an interface for > the cluster manager to specify what gpu's for the driver to use and an > interface for the user to get the resource information -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27488) Driver interface to support GPU resources
[ https://issues.apache.org/jira/browse/SPARK-27488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820119#comment-16820119 ] Thomas Graves commented on SPARK-27488: --- Note, the api design is here: https://issues.apache.org/jira/browse/SPARK-27364 > Driver interface to support GPU resources > -- > > Key: SPARK-27488 > URL: https://issues.apache.org/jira/browse/SPARK-27488 > Project: Spark > Issue Type: Story > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Major > > We want to have an interface to allow the users on the driver to get what > resources are allocated to them. This is mostly to handle the case the > cluster manager does not launch the driver in an isolated environment and > where users could be sharing hosts. For instance in standalone mode it > doesn't support container isolation so a host may have 4 gpu's, but only 2 of > them could be assigned to the driver. In this case we need an interface for > the cluster manager to specify what gpu's for the driver to use and an > interface for the user to get the resource information -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
