Github user ifilonenko commented on the issue:
https://github.com/apache/spark/pull/21748
@mccheah the integration tests did not include the ClientModeTestsSuite.
Can you add `with ClientModeTestsSuite` else, the PRB doesn't actually test the
client mode support accurately.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93551/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93551 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93551/testReport)**
for PR 21748 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
Kubernetes integration test status success
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1317/
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
Kubernetes integration test starting
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1317/
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93551 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93551/testReport)**
for PR 21748 at commit
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
Ok after the next build passes I'm going to merge immediately. Thanks for
the review.
---
-
To unsubscribe, e-mail:
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
Merging in a few hours if no additional comments are raised.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93357/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93357 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93357/testReport)**
for PR 21748 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93360/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93360 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93360/testReport)**
for PR 21748 at commit
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/21748
LGTM for the docs updates.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
Never mind, think it's recovering now.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93361 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93361/testReport)**
for PR 21748 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93361/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
Anyone know what's happening with this:
```
[error]
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93359/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93359 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93359/testReport)**
for PR 21748 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93361 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93361/testReport)**
for PR 21748 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93358/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93358 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93358/testReport)**
for PR 21748 at commit
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93359 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93359/testReport)**
for PR 21748 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93360 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93360/testReport)**
for PR 21748 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
@liyinan926 did some of my own edits on top of your suggestions for docs
wording on the latest patch.
---
-
To unsubscribe,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
Kubernetes integration test status success
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1180/
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93358 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93358/testReport)**
for PR 21748 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
Kubernetes integration test starting
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1180/
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93357 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93357/testReport)**
for PR 21748 at commit
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
We discussed this offline. After some experimentation, we concluded that
it's not actually straightforward to set up the headless service in the
Kubernetes scheduler code in client mode, which
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
> I'm personally leaning towards doing that for the user.
Especially if the user is a data scientist behind his notebook launching a
paragraph which is supposed to instanciate a Spark
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/21748
> That is why I suggested also to remove the driver's knowledge of the
driver pod name and to remove the owner reference concept entirely.
While, not worrying about the driver pod name
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
I think taking a step back, it seems unwise more so to be making any
assumptions about the location in which a driver is running in client mode.
Client mode is simply just saying that the
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/21748
> In that case, the client process could create its own
spark-client-app-id...
Yes, and that's what my point above is about. Regardless of how the driver
pod is created and managed,
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
> Label spark-app-id is only set if spark-submit goes through the steps to
create the driver pod so doesn't apply in this case.
In that case, the client process could create its own
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/21748
> Got you points. About labels, right, we could take the road of the code
that creates labels on its own pod. To ensure uniqueness, we could use the
spark-app-id as key (if it maps the
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
Got you points. About labels, right, we could take the road of the code
that creates labels on its own pod. To ensure uniqueness, we could use the
`spark-app-id` as key (if it maps the requirement
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/21748
> The problem is that the driver's labels might not be unique to that
driver, which therefore would require the user to assign their own unique
labels or for us to patch the driver pod in-place
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
> Yes, the service gets its endpoints by matching its label selector
against labels on the pods so it's critical to have matching labels. Another
tenable solution is for the driver backend code to
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/21748
> I don't think you can back a service with a selector that's a pod's name,
but someone with more knowledge of the Service API might be able to correct me
here. I was under the impression one
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
> About selecting the pod with labels, another approach I have taken is
simply using the name of the driver pod, a bit like I have done with the
following deployment (so no need to ensure labels -
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
@mccheah If I compare with yarn-client with all nodes on the same LAN, we
introduce complexity here as the user has to ensure not only configuration, but
also deployment of a particular resource.
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
Though I suppose you could have the driver patch its own metadata fields to
assign itself a unique label. I could see that being confusing to users when
they observe that their driver pod metadata
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
@echarles I don't think we should be special-casing Kubernetes here as
being any different from the other cluster managers. The main point of client
mode is that the driver is running locally and
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
PS: Actually, there would even be no issue with the port assignment as
Spark knows which ports he will be using, so he can create the headless service
with the correct ports for the user.
---
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
> Note that we only invoke any of the feature steps and the entry point of
KubernetesClientApplication if we run in cluster mode. If we run in client
mode, we enter directly into the user's main
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/21748
> Sounds fine. How does the documentation look now in that regard?
I think we should add the following: 1) be explicit about the
`OwnerReference` when there's a driver pod, and 2)
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
Sounds fine. How does the documentation look now in that regard?
---
-
To unsubscribe, e-mail:
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/21748
> I wonder if we want to have the pod name owner reference still be a
thing, if you will, in client mode. For example what if the pod name that is
given is accidentally one that is assigned to a
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
> there was a change in Spark recently in how the driver self-discovered
its hostname by default, if I am not mistaken. Can't recall the exact patch. I
remember that change specifically prompting
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
@echarles there was a change in Spark recently in how the driver
self-discovered its hostname by default, if I am not mistaken. Can't recall the
exact patch. I remember that change specifically
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
Kubernetes integration test status success
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1102/
---
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
@mccheah @liyinan926 the code base has largely changed from the fork, but
at that time it was working fine without having to manually create any headless
service. Not sure why... but sure it was
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
@liyinan926 I wonder if we want to have the pod name owner reference still
be a thing, if you will, in client mode. For example what if the pod name that
is given is accidentally one that is
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
Kubernetes integration test starting
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1102/
---
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
@mccheah @liyinan926 it is now working on my env in Out-Cluster. I was
failing because I forgot to remove the `spark.kubernetes.driver.pod.name`
props. In general, configuration is tedious and we
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
@mccheah thx for information. As a reader, I didn't understand that if I
didn't implement a headless service, I had to implement something else.
---
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
> Can you point in the fork where the submission client is create the
headless service? (just to help me understand the internals)
> Btw If we stick to this manual approach, the need for
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93244/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93244 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93244/testReport)**
for PR 21748 at commit
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/21748
> Can you point in the fork where the submission client is create the
headless service? (just to help me understand the internals).
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
Kubernetes integration test status success
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1101/
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
> @mccheah agreed with @echarles that it would be great if the submission
client will still create a headless service for the driver if the driver is
running in a pod in client mode.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93243/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93243 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93243/testReport)**
for PR 21748 at commit
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
Also make sure your driver can actually allocate the port specified by
`spark.driver.port`?
---
-
To unsubscribe, e-mail:
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
@echarles then you'd probably want more information such as the logs of the
executors, though I'd imagine that one would have trouble getting those given
that the executor exits so quickly. But
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
> Can you share on how you know that your executor pod has access to your
host set by spark.driver.host and spark.driver.port over the network?
spark.driver.host is set to the hostname of
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/21748
> The goal of this approach that specifically does not create a headless
service is so that the client mode implementation here is identical to the
client mode implementation of the other
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
Kubernetes integration test starting
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1101/
---
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
> agreed with @echarles that it would be great if the submission client
will still create a headless service for the driver if the driver is running in
a pod in client mode.
The goal of
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21748
**[Test build #93244 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93244/testReport)**
for PR 21748 at commit
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/21748
@mccheah agreed with @echarles that it would be great if the submission
client will still create a headless service for the driver if the driver is
running in a pod in client mode.
---
Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
@mccheah my pod has access to my host, so there is nothing to do on network
level. In other words, which are the steps to make this PR work in client mode
for Out-Cluster (assuming the network
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/21748
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
1 - 100 of 322 matches
Mail list logo
