[jira] [Comment Edited] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions
[
https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14254309#comment-14254309
]
Josh Rosen edited comment on SPARK-3967 at 12/19/14 11:59 PM:
--
I've merged [~rdub]'s patch (SPARK-4896) into {{master}}, {{branch-1.1}}, and
{{branch-1.2}} and have backported the other patch to {{branch-1.2}}.
It would be great if folks could confirm whether these fixes have resolved this
issue, or whether there's still more work to be done.
was (Author: joshrosen):
I've merged [~rdub]'s patch (SPARK-4896) into {master}, {branch-1.1}, and
{branch-1.2} and have backported the other patch to {{branch-1.2}.
It would be great if folks could confirm whether these fixes have resolved this
issue, or whether there's still more work to be done.
Spark applications fail in yarn-cluster mode when the directories configured
in yarn.nodemanager.local-dirs are located on different disks/partitions
-
Key: SPARK-3967
URL: https://issues.apache.org/jira/browse/SPARK-3967
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.1.0
Reporter: Christophe PRÉAUD
Attachments: spark-1.1.0-utils-fetch.patch,
spark-1.1.0-yarn_cluster_tmpdir.patch
Spark applications fail from time to time in yarn-cluster mode (but not in
yarn-client mode) when yarn.nodemanager.local-dirs (Hadoop YARN config) is
set to a comma-separated list of directories which are located on different
disks/partitions.
Steps to reproduce:
1. Set yarn.nodemanager.local-dirs (in yarn-site.xml) to a list of
directories located on different partitions (the more you set, the more
likely it will be to reproduce the bug):
(...)
property
nameyarn.nodemanager.local-dirs/name
valuefile:/d1/yarn/local/nm-local-dir,file:/d2/yarn/local/nm-local-dir,file:/d3/yarn/local/nm-local-dir,file:/d4/yarn/local/nm-local-dir,file:/d5/yarn/local/nm-local-dir,file:/d6/yarn/local/nm-local-dir,file:/d7/yarn/local/nm-local-dir/value
/property
(...)
2. Launch (several times) an application in yarn-cluster mode, it will fail
(apparently randomly) from time to time
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions
[
https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176782#comment-14176782
]
Christophe PRÉAUD edited comment on SPARK-3967 at 10/20/14 12:20 PM:
-
Hi Ryan,
Thanks for your help. You should probably add the same test on the Files.move
on [line
437|https://github.com/apache/spark/blob/v1.1.0/core/src/main/scala/org/apache/spark/util/Utils.scala#L437]
was (Author: preaudc):
Hi Ryan,
Thanks for your help. You should probably add the same test on the
{code:java}Files.move{code} on [line
437|https://github.com/apache/spark/blob/v1.1.0/core/src/main/scala/org/apache/spark/util/Utils.scala#L437]
Spark applications fail in yarn-cluster mode when the directories configured
in yarn.nodemanager.local-dirs are located on different disks/partitions
-
Key: SPARK-3967
URL: https://issues.apache.org/jira/browse/SPARK-3967
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.1.0
Reporter: Christophe PRÉAUD
Attachments: spark-1.1.0-utils-fetch.patch,
spark-1.1.0-yarn_cluster_tmpdir.patch
Spark applications fail from time to time in yarn-cluster mode (but not in
yarn-client mode) when yarn.nodemanager.local-dirs (Hadoop YARN config) is
set to a comma-separated list of directories which are located on different
disks/partitions.
Steps to reproduce:
1. Set yarn.nodemanager.local-dirs (in yarn-site.xml) to a list of
directories located on different partitions (the more you set, the more
likely it will be to reproduce the bug):
(...)
property
nameyarn.nodemanager.local-dirs/name
valuefile:/d1/yarn/local/nm-local-dir,file:/d2/yarn/local/nm-local-dir,file:/d3/yarn/local/nm-local-dir,file:/d4/yarn/local/nm-local-dir,file:/d5/yarn/local/nm-local-dir,file:/d6/yarn/local/nm-local-dir,file:/d7/yarn/local/nm-local-dir/value
/property
(...)
2. Launch (several times) an application in yarn-cluster mode, it will fail
(apparently randomly) from time to time
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions
[
https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176782#comment-14176782
]
Christophe PRÉAUD edited comment on SPARK-3967 at 10/20/14 12:19 PM:
-
Hi Ryan,
Thanks for your help. You should probably add the same test on the
{code:scala}Files.move{code} on [line
437|https://github.com/apache/spark/blob/v1.1.0/core/src/main/scala/org/apache/spark/util/Utils.scala#L437]
was (Author: preaudc):
Hi Ryan,
Thanks for your help. You should probably add the same test on the Files.move
on line 437:
https://github.com/apache/spark/blob/v1.1.0/core/src/main/scala/org/apache/spark/util/Utils.scala#L437
(sorry, I'm not familiar enough with JIRA syntax to produce a nice link like
you :-) )
Spark applications fail in yarn-cluster mode when the directories configured
in yarn.nodemanager.local-dirs are located on different disks/partitions
-
Key: SPARK-3967
URL: https://issues.apache.org/jira/browse/SPARK-3967
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.1.0
Reporter: Christophe PRÉAUD
Attachments: spark-1.1.0-utils-fetch.patch,
spark-1.1.0-yarn_cluster_tmpdir.patch
Spark applications fail from time to time in yarn-cluster mode (but not in
yarn-client mode) when yarn.nodemanager.local-dirs (Hadoop YARN config) is
set to a comma-separated list of directories which are located on different
disks/partitions.
Steps to reproduce:
1. Set yarn.nodemanager.local-dirs (in yarn-site.xml) to a list of
directories located on different partitions (the more you set, the more
likely it will be to reproduce the bug):
(...)
property
nameyarn.nodemanager.local-dirs/name
valuefile:/d1/yarn/local/nm-local-dir,file:/d2/yarn/local/nm-local-dir,file:/d3/yarn/local/nm-local-dir,file:/d4/yarn/local/nm-local-dir,file:/d5/yarn/local/nm-local-dir,file:/d6/yarn/local/nm-local-dir,file:/d7/yarn/local/nm-local-dir/value
/property
(...)
2. Launch (several times) an application in yarn-cluster mode, it will fail
(apparently randomly) from time to time
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions
[
https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176782#comment-14176782
]
Christophe PRÉAUD edited comment on SPARK-3967 at 10/20/14 12:20 PM:
-
Hi Ryan,
Thanks for your help. You should probably add the same test on the
{code:java}Files.move{code} on [line
437|https://github.com/apache/spark/blob/v1.1.0/core/src/main/scala/org/apache/spark/util/Utils.scala#L437]
was (Author: preaudc):
Hi Ryan,
Thanks for your help. You should probably add the same test on the
{code:scala}Files.move{code} on [line
437|https://github.com/apache/spark/blob/v1.1.0/core/src/main/scala/org/apache/spark/util/Utils.scala#L437]
Spark applications fail in yarn-cluster mode when the directories configured
in yarn.nodemanager.local-dirs are located on different disks/partitions
-
Key: SPARK-3967
URL: https://issues.apache.org/jira/browse/SPARK-3967
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.1.0
Reporter: Christophe PRÉAUD
Attachments: spark-1.1.0-utils-fetch.patch,
spark-1.1.0-yarn_cluster_tmpdir.patch
Spark applications fail from time to time in yarn-cluster mode (but not in
yarn-client mode) when yarn.nodemanager.local-dirs (Hadoop YARN config) is
set to a comma-separated list of directories which are located on different
disks/partitions.
Steps to reproduce:
1. Set yarn.nodemanager.local-dirs (in yarn-site.xml) to a list of
directories located on different partitions (the more you set, the more
likely it will be to reproduce the bug):
(...)
property
nameyarn.nodemanager.local-dirs/name
valuefile:/d1/yarn/local/nm-local-dir,file:/d2/yarn/local/nm-local-dir,file:/d3/yarn/local/nm-local-dir,file:/d4/yarn/local/nm-local-dir,file:/d5/yarn/local/nm-local-dir,file:/d6/yarn/local/nm-local-dir,file:/d7/yarn/local/nm-local-dir/value
/property
(...)
2. Launch (several times) an application in yarn-cluster mode, it will fail
(apparently randomly) from time to time
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions
[
https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176782#comment-14176782
]
Christophe PRÉAUD edited comment on SPARK-3967 at 10/20/14 12:21 PM:
-
Hi Ryan,
Thanks for your help. You should probably add the same test on the
{{Files.move}} on [line
437|https://github.com/apache/spark/blob/v1.1.0/core/src/main/scala/org/apache/spark/util/Utils.scala#L437]
was (Author: preaudc):
Hi Ryan,
Thanks for your help. You should probably add the same test on the Files.move
on [line
437|https://github.com/apache/spark/blob/v1.1.0/core/src/main/scala/org/apache/spark/util/Utils.scala#L437]
Spark applications fail in yarn-cluster mode when the directories configured
in yarn.nodemanager.local-dirs are located on different disks/partitions
-
Key: SPARK-3967
URL: https://issues.apache.org/jira/browse/SPARK-3967
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.1.0
Reporter: Christophe PRÉAUD
Attachments: spark-1.1.0-utils-fetch.patch,
spark-1.1.0-yarn_cluster_tmpdir.patch
Spark applications fail from time to time in yarn-cluster mode (but not in
yarn-client mode) when yarn.nodemanager.local-dirs (Hadoop YARN config) is
set to a comma-separated list of directories which are located on different
disks/partitions.
Steps to reproduce:
1. Set yarn.nodemanager.local-dirs (in yarn-site.xml) to a list of
directories located on different partitions (the more you set, the more
likely it will be to reproduce the bug):
(...)
property
nameyarn.nodemanager.local-dirs/name
valuefile:/d1/yarn/local/nm-local-dir,file:/d2/yarn/local/nm-local-dir,file:/d3/yarn/local/nm-local-dir,file:/d4/yarn/local/nm-local-dir,file:/d5/yarn/local/nm-local-dir,file:/d6/yarn/local/nm-local-dir,file:/d7/yarn/local/nm-local-dir/value
/property
(...)
2. Launch (several times) an application in yarn-cluster mode, it will fail
(apparently randomly) from time to time
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
