[jira] [Commented] (SPARK-26867) Spark Support of YARN Placement Constraint
[ https://issues.apache.org/jira/browse/SPARK-26867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358463#comment-17358463 ] Apache Spark commented on SPARK-26867: -- User 'AngersZh' has created a pull request for this issue: https://github.com/apache/spark/pull/32804 > Spark Support of YARN Placement Constraint > -- > > Key: SPARK-26867 > URL: https://issues.apache.org/jira/browse/SPARK-26867 > Project: Spark > Issue Type: New Feature > Components: Spark Core, YARN >Affects Versions: 3.1.0 >Reporter: Prabhu Joseph >Priority: Major > > YARN provides Placement Constraint Features - where application can request > containers based on affinity / anti-affinity / cardinality to services or > other application containers / node attributes. This is a useful feature for > Spark Jobs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26867) Spark Support of YARN Placement Constraint
[ https://issues.apache.org/jira/browse/SPARK-26867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358461#comment-17358461 ] Apache Spark commented on SPARK-26867: -- User 'AngersZh' has created a pull request for this issue: https://github.com/apache/spark/pull/32804 > Spark Support of YARN Placement Constraint > -- > > Key: SPARK-26867 > URL: https://issues.apache.org/jira/browse/SPARK-26867 > Project: Spark > Issue Type: New Feature > Components: Spark Core, YARN >Affects Versions: 3.1.0 >Reporter: Prabhu Joseph >Priority: Major > > YARN provides Placement Constraint Features - where application can request > containers based on affinity / anti-affinity / cardinality to services or > other application containers / node attributes. This is a useful feature for > Spark Jobs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26867) Spark Support of YARN Placement Constraint
[ https://issues.apache.org/jira/browse/SPARK-26867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17354987#comment-17354987 ] angerszhu commented on SPARK-26867: --- We have such requirement and I am working on this but something not clear about the new API. Is there progress update and what we can discuss about this? > Spark Support of YARN Placement Constraint > -- > > Key: SPARK-26867 > URL: https://issues.apache.org/jira/browse/SPARK-26867 > Project: Spark > Issue Type: New Feature > Components: Spark Core, YARN >Affects Versions: 3.1.0 >Reporter: Prabhu Joseph >Priority: Major > > YARN provides Placement Constraint Features - where application can request > containers based on affinity / anti-affinity / cardinality to services or > other application containers / node attributes. This is a useful feature for > Spark Jobs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26867) Spark Support of YARN Placement Constraint
[ https://issues.apache.org/jira/browse/SPARK-26867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245785#comment-17245785 ] boy commented on SPARK-26867: - [~prabhujoseph] HI, [~prabhujoseph] , is there any progress on this feature? Since Yarn 3.2 release Node Attributes , it nice job and used with Placement Constraint, so hope spark can support it as quickly as possible > Spark Support of YARN Placement Constraint > -- > > Key: SPARK-26867 > URL: https://issues.apache.org/jira/browse/SPARK-26867 > Project: Spark > Issue Type: New Feature > Components: Spark Core, YARN >Affects Versions: 3.1.0 >Reporter: Prabhu Joseph >Priority: Major > > YARN provides Placement Constraint Features - where application can request > containers based on affinity / anti-affinity / cardinality to services or > other application containers / node attributes. This is a useful feature for > Spark Jobs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26867) Spark Support of YARN Placement Constraint
[ https://issues.apache.org/jira/browse/SPARK-26867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245784#comment-17245784 ] boy commented on SPARK-26867: - Hi, Prabhu Joseph ,is there any progress on this feature? Since Yarn 3.2 release Node Attributes , it nice job and used with Placement Constraint, so hope spark can support it as quickly as possible > Spark Support of YARN Placement Constraint > -- > > Key: SPARK-26867 > URL: https://issues.apache.org/jira/browse/SPARK-26867 > Project: Spark > Issue Type: New Feature > Components: Spark Core, YARN >Affects Versions: 3.1.0 >Reporter: Prabhu Joseph >Priority: Major > > YARN provides Placement Constraint Features - where application can request > containers based on affinity / anti-affinity / cardinality to services or > other application containers / node attributes. This is a useful feature for > Spark Jobs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26867) Spark Support of YARN Placement Constraint
[ https://issues.apache.org/jira/browse/SPARK-26867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782270#comment-16782270 ] Prabhu Joseph commented on SPARK-26867: --- Spark can allow users to configure the Placement Constraint so that users will have more control on where the executors will get placed. For example: 1. Spark job wants to be run on machines where Python version is x or Java version is y (Node Attributes) 2. Spark job needs / does not need executors to be placed on machine where Hbase RegionServer / Zookeeper / Or any other Service is running. (Affinity / Anti Affinity) 3. Spark job wants no more than 2 of it's executors on same node (Cardinality) 4. Spark Job A executors wants / does not want to be run on where Spark Job / Any Other Job B containers runs (Application_Tag NameSpace) > Spark Support of YARN Placement Constraint > -- > > Key: SPARK-26867 > URL: https://issues.apache.org/jira/browse/SPARK-26867 > Project: Spark > Issue Type: New Feature > Components: Spark Core, YARN >Affects Versions: 3.0.0 >Reporter: Prabhu Joseph >Priority: Major > > YARN provides Placement Constraint Features - where application can request > containers based on affinity / anti-affinity / cardinality to services or > other application containers / node attributes. This is a useful feature for > Spark Jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26867) Spark Support of YARN Placement Constraint
[ https://issues.apache.org/jira/browse/SPARK-26867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782105#comment-16782105 ] Sean Owen commented on SPARK-26867: --- How would Spark use it? > Spark Support of YARN Placement Constraint > -- > > Key: SPARK-26867 > URL: https://issues.apache.org/jira/browse/SPARK-26867 > Project: Spark > Issue Type: New Feature > Components: Spark Core, YARN >Affects Versions: 3.0.0 >Reporter: Prabhu Joseph >Priority: Major > > YARN provides Placement Constraint Features - where application can request > containers based on affinity / anti-affinity / cardinality to services or > other application containers / node attributes. This is a useful feature for > Spark Jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26867) Spark Support of YARN Placement Constraint
[ https://issues.apache.org/jira/browse/SPARK-26867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767378#comment-16767378 ] Dongjoon Hyun commented on SPARK-26867: --- Hi, [~Prabhu Joseph]. Since this is a new feature, I updated the affected version from 2.4.0 to 3.0.0. > Spark Support of YARN Placement Constraint > -- > > Key: SPARK-26867 > URL: https://issues.apache.org/jira/browse/SPARK-26867 > Project: Spark > Issue Type: New Feature > Components: Spark Core, YARN >Affects Versions: 3.0.0 >Reporter: Prabhu Joseph >Priority: Major > > YARN provides Placement Constraint Features - where application can request > containers based on affinity / anti-affinity / cardinality to services or > other application containers / node attributes. This is a useful feature for > Spark Jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
