[jira] [Commented] (SPARK-4434) spark-submit cluster deploy mode JAR URLs are broken in 1.1.1
[ https://issues.apache.org/jira/browse/SPARK-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215022#comment-14215022 ] Apache Spark commented on SPARK-4434: - User 'davies' has created a pull request for this issue: https://github.com/apache/spark/pull/3320 spark-submit cluster deploy mode JAR URLs are broken in 1.1.1 - Key: SPARK-4434 URL: https://issues.apache.org/jira/browse/SPARK-4434 Project: Spark Issue Type: Bug Components: Deploy, Spark Core Affects Versions: 1.1.1, 1.2.0 Reporter: Josh Rosen Assignee: Andrew Or Priority: Blocker When submitting a driver using {{spark-submit}} in cluster mode, Spark 1.1.0 allowed you to omit the {{file://}} or {{hdfs://}} prefix from the application JAR URL, e.g. {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/old-spark-releases/spark-1.1.0-bin-hadoop1/lib/spark-examples-1.1.0-hadoop1.0.4.jar {code} In Spark 1.1.1 and 1.2.0, this same command now fails with an error: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:/Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) Usage: DriverClient [options] launch active-master jar-url main-class [driver options] Usage: DriverClient kill active-master driver-id {code} I tried changing my URL to conform to the new format, but this either resulted in an error or a job that failed: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) {code} If I omit the extra slash: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Sending launch command to spark://joshs-mbp.att.net:7077 Driver successfully submitted as driver-20141116143235-0002 ... waiting before polling master for driver state ... polling master for driver state State of driver-20141116143235-0002 is ERROR Exception from cluster was: java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329) at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:157) at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:74) {code} This bug effectively prevents users from using {{spark-submit}} in cluster mode to run drivers whose JARs are stored on shared cluster filesystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4434) spark-submit cluster deploy mode JAR URLs are broken in 1.1.1
[ https://issues.apache.org/jira/browse/SPARK-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215061#comment-14215061 ] Apache Spark commented on SPARK-4434: - User 'davies' has created a pull request for this issue: https://github.com/apache/spark/pull/3320 spark-submit cluster deploy mode JAR URLs are broken in 1.1.1 - Key: SPARK-4434 URL: https://issues.apache.org/jira/browse/SPARK-4434 Project: Spark Issue Type: Bug Components: Deploy, Spark Core Affects Versions: 1.1.1, 1.2.0 Reporter: Josh Rosen Assignee: Andrew Or Priority: Blocker When submitting a driver using {{spark-submit}} in cluster mode, Spark 1.1.0 allowed you to omit the {{file://}} or {{hdfs://}} prefix from the application JAR URL, e.g. {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/old-spark-releases/spark-1.1.0-bin-hadoop1/lib/spark-examples-1.1.0-hadoop1.0.4.jar {code} In Spark 1.1.1 and 1.2.0, this same command now fails with an error: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:/Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) Usage: DriverClient [options] launch active-master jar-url main-class [driver options] Usage: DriverClient kill active-master driver-id {code} I tried changing my URL to conform to the new format, but this either resulted in an error or a job that failed: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) {code} If I omit the extra slash: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Sending launch command to spark://joshs-mbp.att.net:7077 Driver successfully submitted as driver-20141116143235-0002 ... waiting before polling master for driver state ... polling master for driver state State of driver-20141116143235-0002 is ERROR Exception from cluster was: java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329) at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:157) at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:74) {code} This bug effectively prevents users from using {{spark-submit}} in cluster mode to run drivers whose JARs are stored on shared cluster filesystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4434) spark-submit cluster deploy mode JAR URLs are broken in 1.1.1
[ https://issues.apache.org/jira/browse/SPARK-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215279#comment-14215279 ] Apache Spark commented on SPARK-4434: - User 'sarutak' has created a pull request for this issue: https://github.com/apache/spark/pull/3326 spark-submit cluster deploy mode JAR URLs are broken in 1.1.1 - Key: SPARK-4434 URL: https://issues.apache.org/jira/browse/SPARK-4434 Project: Spark Issue Type: Bug Components: Deploy, Spark Core Affects Versions: 1.1.1, 1.2.0 Reporter: Josh Rosen Assignee: Andrew Or Priority: Blocker When submitting a driver using {{spark-submit}} in cluster mode, Spark 1.1.0 allowed you to omit the {{file://}} or {{hdfs://}} prefix from the application JAR URL, e.g. {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/old-spark-releases/spark-1.1.0-bin-hadoop1/lib/spark-examples-1.1.0-hadoop1.0.4.jar {code} In Spark 1.1.1 and 1.2.0, this same command now fails with an error: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:/Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) Usage: DriverClient [options] launch active-master jar-url main-class [driver options] Usage: DriverClient kill active-master driver-id {code} I tried changing my URL to conform to the new format, but this either resulted in an error or a job that failed: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) {code} If I omit the extra slash: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Sending launch command to spark://joshs-mbp.att.net:7077 Driver successfully submitted as driver-20141116143235-0002 ... waiting before polling master for driver state ... polling master for driver state State of driver-20141116143235-0002 is ERROR Exception from cluster was: java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329) at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:157) at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:74) {code} This bug effectively prevents users from using {{spark-submit}} in cluster mode to run drivers whose JARs are stored on shared cluster filesystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4434) spark-submit cluster deploy mode JAR URLs are broken in 1.1.1
[ https://issues.apache.org/jira/browse/SPARK-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214123#comment-14214123 ] Josh Rosen commented on SPARK-4434: --- I think that there are only a small number of patches in branch-1.1 that are related to this, so I'm going to see if I can narrow it down to a specific commit. https://github.com/apache/spark/pull/2925 is one potential culprit, but there may be others. I'm not sure whether this affects HDFS URLs; I haven't tried it yet since I don't have a Docker-ized HDFS set up in my integration tests project. spark-submit cluster deploy mode JAR URLs are broken in 1.1.1 - Key: SPARK-4434 URL: https://issues.apache.org/jira/browse/SPARK-4434 Project: Spark Issue Type: Bug Components: Deploy, Spark Core Affects Versions: 1.1.1, 1.2.0 Reporter: Josh Rosen Assignee: Andrew Or Priority: Blocker When submitting a driver using {{spark-submit}} in cluster mode, Spark 1.1.0 allowed you to omit the {{file://}} or {{hdfs://}} prefix from the application JAR URL, e.g. {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/old-spark-releases/spark-1.1.0-bin-hadoop1/lib/spark-examples-1.1.0-hadoop1.0.4.jar {code} In Spark 1.1.1 and 1.2.0, this same command now fails with an error: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:/Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) Usage: DriverClient [options] launch active-master jar-url main-class [driver options] Usage: DriverClient kill active-master driver-id {code} I tried changing my URL to conform to the new format, but this either resulted in an error or a job that failed: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) {code} If I omit the extra slash: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Sending launch command to spark://joshs-mbp.att.net:7077 Driver successfully submitted as driver-20141116143235-0002 ... waiting before polling master for driver state ... polling master for driver state State of driver-20141116143235-0002 is ERROR Exception from cluster was: java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329) at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:157) at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:74) {code} This bug effectively prevents users from using {{spark-submit}} in cluster mode to run drivers whose JARs are stored on shared cluster filesystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4434) spark-submit cluster deploy mode JAR URLs are broken in 1.1.1
[ https://issues.apache.org/jira/browse/SPARK-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214152#comment-14214152 ] Josh Rosen commented on SPARK-4434: --- It looks like this was caused by https://github.com/apache/spark/pull/2925 (SPARK-4075), since reverting that fixes this issue. I'll work on committing my test code to our internal tests repository and open a PR to investigate / revert that commit. spark-submit cluster deploy mode JAR URLs are broken in 1.1.1 - Key: SPARK-4434 URL: https://issues.apache.org/jira/browse/SPARK-4434 Project: Spark Issue Type: Bug Components: Deploy, Spark Core Affects Versions: 1.1.1, 1.2.0 Reporter: Josh Rosen Assignee: Andrew Or Priority: Blocker When submitting a driver using {{spark-submit}} in cluster mode, Spark 1.1.0 allowed you to omit the {{file://}} or {{hdfs://}} prefix from the application JAR URL, e.g. {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/old-spark-releases/spark-1.1.0-bin-hadoop1/lib/spark-examples-1.1.0-hadoop1.0.4.jar {code} In Spark 1.1.1 and 1.2.0, this same command now fails with an error: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:/Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) Usage: DriverClient [options] launch active-master jar-url main-class [driver options] Usage: DriverClient kill active-master driver-id {code} I tried changing my URL to conform to the new format, but this either resulted in an error or a job that failed: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) {code} If I omit the extra slash: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Sending launch command to spark://joshs-mbp.att.net:7077 Driver successfully submitted as driver-20141116143235-0002 ... waiting before polling master for driver state ... polling master for driver state State of driver-20141116143235-0002 is ERROR Exception from cluster was: java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329) at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:157) at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:74) {code} This bug effectively prevents users from using {{spark-submit}} in cluster mode to run drivers whose JARs are stored on shared cluster filesystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4434) spark-submit cluster deploy mode JAR URLs are broken in 1.1.1
[ https://issues.apache.org/jira/browse/SPARK-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214155#comment-14214155 ] Matei Zaharia commented on SPARK-4434: -- [~joshrosen] make sure to revert this on 1.2 and master as well. spark-submit cluster deploy mode JAR URLs are broken in 1.1.1 - Key: SPARK-4434 URL: https://issues.apache.org/jira/browse/SPARK-4434 Project: Spark Issue Type: Bug Components: Deploy, Spark Core Affects Versions: 1.1.1, 1.2.0 Reporter: Josh Rosen Assignee: Andrew Or Priority: Blocker When submitting a driver using {{spark-submit}} in cluster mode, Spark 1.1.0 allowed you to omit the {{file://}} or {{hdfs://}} prefix from the application JAR URL, e.g. {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/old-spark-releases/spark-1.1.0-bin-hadoop1/lib/spark-examples-1.1.0-hadoop1.0.4.jar {code} In Spark 1.1.1 and 1.2.0, this same command now fails with an error: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:/Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) Usage: DriverClient [options] launch active-master jar-url main-class [driver options] Usage: DriverClient kill active-master driver-id {code} I tried changing my URL to conform to the new format, but this either resulted in an error or a job that failed: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) {code} If I omit the extra slash: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Sending launch command to spark://joshs-mbp.att.net:7077 Driver successfully submitted as driver-20141116143235-0002 ... waiting before polling master for driver state ... polling master for driver state State of driver-20141116143235-0002 is ERROR Exception from cluster was: java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329) at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:157) at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:74) {code} This bug effectively prevents users from using {{spark-submit}} in cluster mode to run drivers whose JARs are stored on shared cluster filesystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4434) spark-submit cluster deploy mode JAR URLs are broken in 1.1.1
[ https://issues.apache.org/jira/browse/SPARK-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214164#comment-14214164 ] Apache Spark commented on SPARK-4434: - User 'sarutak' has created a pull request for this issue: https://github.com/apache/spark/pull/3295 spark-submit cluster deploy mode JAR URLs are broken in 1.1.1 - Key: SPARK-4434 URL: https://issues.apache.org/jira/browse/SPARK-4434 Project: Spark Issue Type: Bug Components: Deploy, Spark Core Affects Versions: 1.1.1, 1.2.0 Reporter: Josh Rosen Assignee: Andrew Or Priority: Blocker When submitting a driver using {{spark-submit}} in cluster mode, Spark 1.1.0 allowed you to omit the {{file://}} or {{hdfs://}} prefix from the application JAR URL, e.g. {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/old-spark-releases/spark-1.1.0-bin-hadoop1/lib/spark-examples-1.1.0-hadoop1.0.4.jar {code} In Spark 1.1.1 and 1.2.0, this same command now fails with an error: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:/Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) Usage: DriverClient [options] launch active-master jar-url main-class [driver options] Usage: DriverClient kill active-master driver-id {code} I tried changing my URL to conform to the new format, but this either resulted in an error or a job that failed: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) {code} If I omit the extra slash: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Sending launch command to spark://joshs-mbp.att.net:7077 Driver successfully submitted as driver-20141116143235-0002 ... waiting before polling master for driver state ... polling master for driver state State of driver-20141116143235-0002 is ERROR Exception from cluster was: java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329) at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:157) at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:74) {code} This bug effectively prevents users from using {{spark-submit}} in cluster mode to run drivers whose JARs are stored on shared cluster filesystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4434) spark-submit cluster deploy mode JAR URLs are broken in 1.1.1
[ https://issues.apache.org/jira/browse/SPARK-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214165#comment-14214165 ] Apache Spark commented on SPARK-4434: - User 'sarutak' has created a pull request for this issue: https://github.com/apache/spark/pull/3296 spark-submit cluster deploy mode JAR URLs are broken in 1.1.1 - Key: SPARK-4434 URL: https://issues.apache.org/jira/browse/SPARK-4434 Project: Spark Issue Type: Bug Components: Deploy, Spark Core Affects Versions: 1.1.1, 1.2.0 Reporter: Josh Rosen Assignee: Andrew Or Priority: Blocker When submitting a driver using {{spark-submit}} in cluster mode, Spark 1.1.0 allowed you to omit the {{file://}} or {{hdfs://}} prefix from the application JAR URL, e.g. {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/old-spark-releases/spark-1.1.0-bin-hadoop1/lib/spark-examples-1.1.0-hadoop1.0.4.jar {code} In Spark 1.1.1 and 1.2.0, this same command now fails with an error: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:/Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) Usage: DriverClient [options] launch active-master jar-url main-class [driver options] Usage: DriverClient kill active-master driver-id {code} I tried changing my URL to conform to the new format, but this either resulted in an error or a job that failed: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) {code} If I omit the extra slash: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Sending launch command to spark://joshs-mbp.att.net:7077 Driver successfully submitted as driver-20141116143235-0002 ... waiting before polling master for driver state ... polling master for driver state State of driver-20141116143235-0002 is ERROR Exception from cluster was: java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329) at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:157) at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:74) {code} This bug effectively prevents users from using {{spark-submit}} in cluster mode to run drivers whose JARs are stored on shared cluster filesystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4434) spark-submit cluster deploy mode JAR URLs are broken in 1.1.1
[ https://issues.apache.org/jira/browse/SPARK-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214166#comment-14214166 ] Aaron Davidson commented on SPARK-4434: --- Side note: the error message about file://, which was not introduced in the patch you reverted, is incorrect. A file://XX.jar URI is never valid. One or three slashes must be used; two slashes indicates that a hostname follows. spark-submit cluster deploy mode JAR URLs are broken in 1.1.1 - Key: SPARK-4434 URL: https://issues.apache.org/jira/browse/SPARK-4434 Project: Spark Issue Type: Bug Components: Deploy, Spark Core Affects Versions: 1.1.1, 1.2.0 Reporter: Josh Rosen Assignee: Andrew Or Priority: Blocker When submitting a driver using {{spark-submit}} in cluster mode, Spark 1.1.0 allowed you to omit the {{file://}} or {{hdfs://}} prefix from the application JAR URL, e.g. {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/old-spark-releases/spark-1.1.0-bin-hadoop1/lib/spark-examples-1.1.0-hadoop1.0.4.jar {code} In Spark 1.1.1 and 1.2.0, this same command now fails with an error: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:/Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) Usage: DriverClient [options] launch active-master jar-url main-class [driver options] Usage: DriverClient kill active-master driver-id {code} I tried changing my URL to conform to the new format, but this either resulted in an error or a job that failed: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) {code} If I omit the extra slash: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Sending launch command to spark://joshs-mbp.att.net:7077 Driver successfully submitted as driver-20141116143235-0002 ... waiting before polling master for driver state ... polling master for driver state State of driver-20141116143235-0002 is ERROR Exception from cluster was: java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329) at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:157) at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:74) {code} This bug effectively prevents users from using {{spark-submit}} in cluster mode to run drivers whose JARs are stored on shared cluster filesystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4434) spark-submit cluster deploy mode JAR URLs are broken in 1.1.1
[ https://issues.apache.org/jira/browse/SPARK-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214196#comment-14214196 ] Josh Rosen commented on SPARK-4434: --- Fellow Databricks folks: I've added a regression test for this in https://github.com/databricks/spark-integration-tests/commit/f121f45aecbeafcec21d3bb670737fc9f7d6da0b (I'm sharing this link here so that it's easy to find this test once we open-source that repository). The test is essentially a scripted / automated version of the commands that I've listed in this JIRA. These tests confirm that reverting that earlier PR fixes this issue. [~adav], do you want to open a separate JIRA to fix the file:// error message? spark-submit cluster deploy mode JAR URLs are broken in 1.1.1 - Key: SPARK-4434 URL: https://issues.apache.org/jira/browse/SPARK-4434 Project: Spark Issue Type: Bug Components: Deploy, Spark Core Affects Versions: 1.1.1, 1.2.0 Reporter: Josh Rosen Assignee: Andrew Or Priority: Blocker When submitting a driver using {{spark-submit}} in cluster mode, Spark 1.1.0 allowed you to omit the {{file://}} or {{hdfs://}} prefix from the application JAR URL, e.g. {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/old-spark-releases/spark-1.1.0-bin-hadoop1/lib/spark-examples-1.1.0-hadoop1.0.4.jar {code} In Spark 1.1.1 and 1.2.0, this same command now fails with an error: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:/Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) Usage: DriverClient [options] launch active-master jar-url main-class [driver options] Usage: DriverClient kill active-master driver-id {code} I tried changing my URL to conform to the new format, but this either resulted in an error or a job that failed: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) {code} If I omit the extra slash: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Sending launch command to spark://joshs-mbp.att.net:7077 Driver successfully submitted as driver-20141116143235-0002 ... waiting before polling master for driver state ... polling master for driver state State of driver-20141116143235-0002 is ERROR Exception from cluster was: java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329) at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:157) at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:74) {code} This bug effectively prevents users from using {{spark-submit}} in cluster mode to run drivers whose JARs are stored on shared cluster filesystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4434) spark-submit cluster deploy mode JAR URLs are broken in 1.1.1
[ https://issues.apache.org/jira/browse/SPARK-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214199#comment-14214199 ] Josh Rosen commented on SPARK-4434: --- As a regression test, we should probably add a triple-slash test case to ClientSuite. spark-submit cluster deploy mode JAR URLs are broken in 1.1.1 - Key: SPARK-4434 URL: https://issues.apache.org/jira/browse/SPARK-4434 Project: Spark Issue Type: Bug Components: Deploy, Spark Core Affects Versions: 1.1.1, 1.2.0 Reporter: Josh Rosen Assignee: Andrew Or Priority: Blocker When submitting a driver using {{spark-submit}} in cluster mode, Spark 1.1.0 allowed you to omit the {{file://}} or {{hdfs://}} prefix from the application JAR URL, e.g. {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/old-spark-releases/spark-1.1.0-bin-hadoop1/lib/spark-examples-1.1.0-hadoop1.0.4.jar {code} In Spark 1.1.1 and 1.2.0, this same command now fails with an error: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi /Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:/Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) Usage: DriverClient [options] launch active-master jar-url main-class [driver options] Usage: DriverClient kill active-master driver-id {code} I tried changing my URL to conform to the new format, but this either resulted in an error or a job that failed: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Jar url 'file:///Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar' is not in valid format. Must be a jar file path in URL format (e.g. hdfs://XX.jar, file://XX.jar) {code} If I omit the extra slash: {code} ./bin/spark-submit --deploy-mode cluster --master spark://joshs-mbp.att.net:7077 --class org.apache.spark.examples.SparkPi file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar Sending launch command to spark://joshs-mbp.att.net:7077 Driver successfully submitted as driver-20141116143235-0002 ... waiting before polling master for driver state ... polling master for driver state State of driver-20141116143235-0002 is ERROR Exception from cluster was: java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// java.lang.IllegalArgumentException: Wrong FS: file://Users/joshrosen/Documents/Spark/examples/target/scala-2.10/spark-examples_2.10-1.1.2-SNAPSHOT.jar, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329) at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:157) at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:74) {code} This bug effectively prevents users from using {{spark-submit}} in cluster mode to run drivers whose JARs are stored on shared cluster filesystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org