[jira] [Comment Edited] (SPARK-20327) Add CLI support for YARN custom resources, like GPUs
[ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360904#comment-16360904 ] Szilard Nemeth edited comment on SPARK-20327 at 2/13/18 9:08 AM: - Hey [~vanzin]! I see what you said about compatibility. By reflection, you meant a similar solution like this? [https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L268] By the way, isn't it a good way to go forward if users specify any of these new resource configs and the Hadoop version Spark currently depends on is 2.x then we handle this case as an error explicitly and inform the user about this? In general, is there a conventional way to detect Hadoop version in Spark code? Thanks! was (Author: snemeth): Hey [~vanzin]! I see what you said about compatibility. By reflection, you meant a similar solution like this? [https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L268] By the way, isn't it a good way to go forward if users specify any of these new resource configs and the Hadoop version Spark currently depends on is 2.x then we handle this case as an error explicitly and inform the user about this? In general, is that a conventional way to detect Hadoop version in Spark code? Thanks! > Add CLI support for YARN custom resources, like GPUs > > > Key: SPARK-20327 > URL: https://issues.apache.org/jira/browse/SPARK-20327 > Project: Spark > Issue Type: Improvement > Components: Spark Shell, Spark Submit >Affects Versions: 2.1.0 >Reporter: Daniel Templeton >Priority: Major > Labels: newbie > > YARN-3926 adds the ability for administrators to configure custom resources, > like GPUs. This JIRA is to add support to Spark for requesting resources > other than CPU virtual cores and memory. See YARN-3926. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-20327) Add CLI support for YARN custom resources, like GPUs
[ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360904#comment-16360904 ] Szilard Nemeth edited comment on SPARK-20327 at 2/12/18 3:43 PM: - Hey [~vanzin]! I see what you said about compatibility. By reflection, you meant a similar solution like this? [https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L268] By the way, isn't it a good way to go forward if users specify any of these new resource configs and the Hadoop version Spark currently depends on is 2.x then we handle this case as an error explicitly and inform the user about this? In general, is that a conventional way to detect Hadoop version in Spark code? Thanks! was (Author: snemeth): Hey [~vanzin]! I see what you said about compatibility. By reflection, you meant a similar solution like this? [https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L268] By the way, isn't it a good way to go forward if users specify any of these new resource onfigs and the Hadoop version Spark currently depends on is 2.x then we handle this case as an error explicitly and inform the user about this? In general, is that a conventional way to detect Hadoop version in Spark code? Thanks! > Add CLI support for YARN custom resources, like GPUs > > > Key: SPARK-20327 > URL: https://issues.apache.org/jira/browse/SPARK-20327 > Project: Spark > Issue Type: Improvement > Components: Spark Shell, Spark Submit >Affects Versions: 2.1.0 >Reporter: Daniel Templeton >Priority: Major > Labels: newbie > > YARN-3926 adds the ability for administrators to configure custom resources, > like GPUs. This JIRA is to add support to Spark for requesting resources > other than CPU virtual cores and memory. See YARN-3926. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
