[GitHub] spark pull request #20320: [SPARK-22962][K8S] Fail fast if submission client...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/20320


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20320: [SPARK-22962][K8S] Fail fast if submission client...

[liyinan926]

Github user liyinan926 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20320#discussion_r162486330
  
--- Diff: 
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigOrchestrator.scala
 ---
@@ -117,6 +117,13 @@ private[spark] class DriverConfigOrchestrator(
   .map(_.split(","))
   .getOrElse(Array.empty[String])
 
+// TODO(SPARK-23153): remote once submission client local dependencies 
are supported.
--- End diff --

Done.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20320: [SPARK-22962][K8S] Fail fast if submission client...

[liyinan926]

Github user liyinan926 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20320#discussion_r162486349
  
--- Diff: 
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigOrchestrator.scala
 ---
@@ -117,6 +117,13 @@ private[spark] class DriverConfigOrchestrator(
   .map(_.split(","))
   .getOrElse(Array.empty[String])
 
+// TODO(SPARK-23153): remote once submission client local dependencies 
are supported.
+if (existSubmissionLocalFiles(sparkJars) || 
existSubmissionLocalFiles(sparkFiles)) {
+  throw new SparkException("The Kubernetes mode does not yet support 
application " +
--- End diff --

Done.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20320: [SPARK-22962][K8S] Fail fast if submission client...

[foxish]

Github user foxish commented on a diff in the pull request:

https://github.com/apache/spark/pull/20320#discussion_r162484585
  
--- Diff: 
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigOrchestrator.scala
 ---
@@ -117,6 +117,13 @@ private[spark] class DriverConfigOrchestrator(
   .map(_.split(","))
   .getOrElse(Array.empty[String])
 
+// TODO(SPARK-23153): remote once submission client local dependencies 
are supported.
--- End diff --

s/remote/remove/


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20320: [SPARK-22962][K8S] Fail fast if submission client...

[foxish]

Github user foxish commented on a diff in the pull request:

https://github.com/apache/spark/pull/20320#discussion_r162485460
  
--- Diff: 
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigOrchestrator.scala
 ---
@@ -117,6 +117,13 @@ private[spark] class DriverConfigOrchestrator(
   .map(_.split(","))
   .getOrElse(Array.empty[String])
 
+// TODO(SPARK-23153): remote once submission client local dependencies 
are supported.
+if (existSubmissionLocalFiles(sparkJars) || 
existSubmissionLocalFiles(sparkFiles)) {
+  throw new SparkException("The Kubernetes mode does not yet support 
application " +
--- End diff --

I'd shorten this to just "Kubernetes mode does not support referencing 
application dependencies in the local file system".


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20320: [SPARK-22962][K8S] Fail fast if submission client...

[liyinan926]

Github user liyinan926 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20320#discussion_r162475825
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -117,7 +117,10 @@ This URI is the location of the example jar that is 
already in the Docker image.
 If your application's dependencies are all hosted in remote locations like 
HDFS or HTTP servers, they may be referred to
 by their appropriate remote URIs. Also, application dependencies can be 
pre-mounted into custom-built Docker images.
 Those dependencies can be added to the classpath by referencing them with 
`local://` URIs and/or setting the
-`SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles.
+`SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles. The 
`local://` scheme is also required when referring to
+dependencies in custom-built Docker images in `spark-submit`. Note that 
using application dependencies local to the submission
+client is currently not yet supported.
--- End diff --

Done.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20320: [SPARK-22962][K8S] Fail fast if submission client...

[liyinan926]

Github user liyinan926 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20320#discussion_r162475552
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -117,7 +117,10 @@ This URI is the location of the example jar that is 
already in the Docker image.
 If your application's dependencies are all hosted in remote locations like 
HDFS or HTTP servers, they may be referred to
 by their appropriate remote URIs. Also, application dependencies can be 
pre-mounted into custom-built Docker images.
 Those dependencies can be added to the classpath by referencing them with 
`local://` URIs and/or setting the
-`SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles.
+`SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles. The 
`local://` scheme is also required when referring to
--- End diff --

Yes, but that's about adding to the classpath. I wanted to make it more 
specific and clearer.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20320: [SPARK-22962][K8S] Fail fast if submission client...

[liyinan926]

Github user liyinan926 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20320#discussion_r162475425
  
--- Diff: 
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigOrchestrator.scala
 ---
@@ -117,6 +117,12 @@ private[spark] class DriverConfigOrchestrator(
   .map(_.split(","))
   .getOrElse(Array.empty[String])
 
+if (existSubmissionLocalFiles(sparkJars) || 
existSubmissionLocalFiles(sparkFiles)) {
+  throw new SparkException("The Kubernetes mode does not yet support 
application " +
+"dependencies local to the submission client. It currently only 
allows application" +
--- End diff --

Done.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20320: [SPARK-22962][K8S] Fail fast if submission client...

[liyinan926]

Github user liyinan926 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20320#discussion_r162475338
  
--- Diff: 
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigOrchestrator.scala
 ---
@@ -117,6 +117,12 @@ private[spark] class DriverConfigOrchestrator(
   .map(_.split(","))
   .getOrElse(Array.empty[String])
 
+if (existSubmissionLocalFiles(sparkJars) || 
existSubmissionLocalFiles(sparkFiles)) {
--- End diff --

Done, and also created https://issues.apache.org/jira/browse/SPARK-23153.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20320: [SPARK-22962][K8S] Fail fast if submission client...

[foxish]

Github user foxish commented on a diff in the pull request:

https://github.com/apache/spark/pull/20320#discussion_r162465264
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -117,7 +117,10 @@ This URI is the location of the example jar that is 
already in the Docker image.
 If your application's dependencies are all hosted in remote locations like 
HDFS or HTTP servers, they may be referred to
 by their appropriate remote URIs. Also, application dependencies can be 
pre-mounted into custom-built Docker images.
 Those dependencies can be added to the classpath by referencing them with 
`local://` URIs and/or setting the
-`SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles.
+`SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles. The 
`local://` scheme is also required when referring to
+dependencies in custom-built Docker images in `spark-submit`. Note that 
using application dependencies local to the submission
+client is currently not yet supported.
--- End diff --

"application dependencies from the local file system"? 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20320: [SPARK-22962][K8S] Fail fast if submission client...

[foxish]

Github user foxish commented on a diff in the pull request:

https://github.com/apache/spark/pull/20320#discussion_r162465377
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -117,7 +117,10 @@ This URI is the location of the example jar that is 
already in the Docker image.
 If your application's dependencies are all hosted in remote locations like 
HDFS or HTTP servers, they may be referred to
 by their appropriate remote URIs. Also, application dependencies can be 
pre-mounted into custom-built Docker images.
 Those dependencies can be added to the classpath by referencing them with 
`local://` URIs and/or setting the
-`SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles.
+`SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles. The 
`local://` scheme is also required when referring to
--- End diff --

The point above already covers that `local://` is needed with custom-built 
images.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20320: [SPARK-22962][K8S] Fail fast if submission client...

[jiangxb1987]

Github user jiangxb1987 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20320#discussion_r162465620
  
--- Diff: 
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigOrchestrator.scala
 ---
@@ -117,6 +117,12 @@ private[spark] class DriverConfigOrchestrator(
   .map(_.split(","))
   .getOrElse(Array.empty[String])
 
+if (existSubmissionLocalFiles(sparkJars) || 
existSubmissionLocalFiles(sparkFiles)) {
+  throw new SparkException("The Kubernetes mode does not yet support 
application " +
+"dependencies local to the submission client. It currently only 
allows application" +
--- End diff --

nit: extra space in the end of line


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20320: [SPARK-22962][K8S] Fail fast if submission client...

[jiangxb1987]

Github user jiangxb1987 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20320#discussion_r162465510
  
--- Diff: 
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigOrchestrator.scala
 ---
@@ -117,6 +117,12 @@ private[spark] class DriverConfigOrchestrator(
   .map(_.split(","))
   .getOrElse(Array.empty[String])
 
+if (existSubmissionLocalFiles(sparkJars) || 
existSubmissionLocalFiles(sparkFiles)) {
--- End diff --

We can add a `TODO` here if this is planned to be supported in the future.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20320: [SPARK-22962][K8S] Fail fast if submission client...

[liyinan926]

GitHub user liyinan926 opened a pull request:

https://github.com/apache/spark/pull/20320

[SPARK-22962][K8S] Fail fast if submission client local files are used

## What changes were proposed in this pull request?

In the Kubernetes mode, fails fast in the submission process if any 
submission client local dependencies are used as the use case is not supported 
yet. 

## How was this patch tested?

Unit tests.

@vanzin @foxish 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/liyinan926/spark-k8s master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20320.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20320


commit df9cdfb4588ba4142c36f209816c45b00bef2f7f
Author: Yinan Li 
Date:   2018-01-18T20:22:51Z

[SPARK-22962][K8S] Fail fast if submission client local files are used




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org