Github user echarles closed the pull request at:
https://github.com/apache/spark/pull/20451
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/20451
See #21748
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
> I'm personally leaning towards doing that for the user.
Especially if the user is a data scientist behind his notebook launching a
paragraph which is supposed to instanciate a Sp
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
> Label spark-app-id is only set if spark-submit goes through the steps to
create the driver pod so doesn't apply in this case.
In that case, the client process could create its
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
Got you points. About labels, right, we could take the road of the code
that creates labels on its own pod. To ensure uniqueness, we could use the
`spark-app-id` as key (if it maps the requirement
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
@mccheah If I compare with yarn-client with all nodes on the same LAN, we
introduce complexity here as the user has to ensure not only configuration, but
also deployment of a particular resource
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
PS: Actually, there would even be no issue with the port assignment as
Spark knows which ports he will be using, so he can create the headless service
with the correct ports for the user
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
> Note that we only invoke any of the feature steps and the entry point of
KubernetesClientApplication if we run in cluster mode. If we run in client
mode, we enter directly into the user's m
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
@mccheah @liyinan926 the code base has largely changed from the fork, but
at that time it was working fine without having to manually create any headless
service. Not sure why... but sure
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
@mccheah @liyinan926 it is now working on my env in Out-Cluster. I was
failing because I forgot to remove the `spark.kubernetes.driver.pod.name`
props. In general, configuration is tedious and we
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
@mccheah thx for information. As a reader, I didn't understand that if I
didn't implement a headless service, I had to implement something else
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
> @mccheah agreed with @echarles that it would be great if the submission
client will still create a headless service for the driver if the driver is
running in a pod in client m
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
> Can you share on how you know that your executor pod has access to your
host set by spark.driver.host and spark.driver.port over the network?
spark.driver.host is set to the hostn
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
@mccheah my pod has access to my host, so there is nothing to do on network
level. In other words, which are the steps to make this PR work in client mode
for Out-Cluster (assuming the network
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
Thx @mccheah and @liyinan926. It now works with an headless service using
`spark.driver.host=spark-driver-service` and
`spark.kubernetes.driver.pod.name=spark-pod`
Two more questions
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
@mccheah Tried this PR in client-mode In-Cluster on minikube v0.25.2:
Exectuors are started but directly removed. As the start/remove is so fast, I
can hardly see logs (and the logs I have seen
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/20451
@liyinan926 Thx for you comments (see my answers). I have merged the last
changes, added a tiny doc section and pushed again
Github user echarles commented on a diff in the pull request:
https://github.com/apache/spark/pull/20451#discussion_r201781927
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala
---
@@ -20,41 +20,65
Github user echarles commented on a diff in the pull request:
https://github.com/apache/spark/pull/20451#discussion_r201778571
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
---
@@ -58,6 +58,8 @@ private[spark] object Config
Github user echarles commented on a diff in the pull request:
https://github.com/apache/spark/pull/20451#discussion_r201778543
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/OptionRequirements.scala
---
@@ -0,0 +1,40 @@
+/*
--- End
Github user echarles commented on a diff in the pull request:
https://github.com/apache/spark/pull/20451#discussion_r201327022
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala
---
@@ -140,13 +140,6
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/20451
@rayburgemeestre Thx for sharing you Toree conf. Are you sure you are
running in client mode? (I don't see something similar to `--conf
spark.submit.deployMode=client`). Maybe it just works
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/20451
@liyinan926 I pushed the merge with master and tested OutCluster (I didn't
test InCluster but as mentioned previously, we would need something similar to
the `DriverServiceBootstrapStep
Github user echarles commented on a diff in the pull request:
https://github.com/apache/spark/pull/20451#discussion_r201215612
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala
---
@@ -18,43 +18,68
Github user echarles commented on a diff in the pull request:
https://github.com/apache/spark/pull/20451#discussion_r201215439
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala
---
@@ -140,13 +140,6
Github user echarles commented on a diff in the pull request:
https://github.com/apache/spark/pull/20451#discussion_r201214812
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
---
@@ -88,6 +103,56 @@ private
Github user echarles commented on a diff in the pull request:
https://github.com/apache/spark/pull/20451#discussion_r201212364
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
---
@@ -88,6 +103,56 @@ private
Github user echarles commented on a diff in the pull request:
https://github.com/apache/spark/pull/20451#discussion_r201211050
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/OptionRequirements.scala
---
@@ -0,0 +1,40
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/20451
@rayburgemeestre This is cool!
Apache Toree configuration for K8S is on my todo list but would be happy to
copycat your conf...
Any gist?
IMHO both client and cluster mode
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/20451
I have updated this branch and tested successfully client mode OutCluster.
Happy to get confirmation of this from anyone here (cc/ @foxish)
For InCluster (was working fine before
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/20451
Now that #20910 has been merged, I will update this PR to take account the
refactoring. @inetfuture Once these changes are pushed, there is the review
process which needs to occur, so difficult
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/20910
@mccheah Before asking a user to rewrite the full `spark-submit`, I still
believe we can propose a middle ground that would make people with specific
requirements happy. You could also answer
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/20910
Is there a way to make the step chain configurable. I am thinking to a list
of class fqn, with the default being the one which is implemented now
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/20451
Current state works for out-cluster client mode.
For in-cluster, we miss the `DriverServiceBootstrapStep` of the fork which
"Allows the driver to be reachable by executor pods th
GitHub user echarles opened a pull request:
https://github.com/apache/spark/pull/20451
[SPARK-23146][WIP] Support client mode for Kubernetes cluster backend
## What changes were proposed in this pull request?
The changes allow to support Kubernetes resource manager
Github user echarles commented on a diff in the pull request:
https://github.com/apache/spark/pull/19717#discussion_r159282180
--- Diff:
resource-managers/kubernetes/docker/src/main/dockerfiles/spark-base/Dockerfile
---
@@ -0,0 +1,47 @@
+#
+# Licensed to the Apache
36 matches
Mail list logo
