Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153350280
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153350234
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153352450
**[Test build #44919 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44919/consoleFull)**
for PR 8093 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153357269
**[Test build #44921 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44921/consoleFull)**
for PR 8093 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153357278
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153357280
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153366300
**[Test build #44923 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44923/consoleFull)**
for PR 8093 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153365863
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153401701
**[Test build #44923 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44923/consoleFull)**
for PR 8093 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153401863
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153404861
Build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153404862
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153404652
**[Test build #44919 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44919/consoleFull)**
for PR 8093 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153401864
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153355848
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153355818
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153356058
**[Test build #44921 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44921/consoleFull)**
for PR 8093 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153365937
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153414205
Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-153414851
Thanks Marcelo!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/8093
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r43578203
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -189,7 +188,9 @@ class
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-152786751
LGTM but needs a rebase.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-151314454
**[Test build #44383 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44383/consoleFull)**
for PR 8093 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-151313105
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-151313091
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-151337484
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-151337268
**[Test build #44383 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44383/consoleFull)**
for PR 8093 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-151337488
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r43016585
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -367,9 +367,15 @@ private[spark] object JsonProtocol {
("Job ID"
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-151201804
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-151198182
Thanks @vanzin for your feedback. I have made your suggested changes,
build/tested again and updated this pull request. Thanks again!
---
If your project is set up
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-151196859
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-151196890
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-151200883
**[Test build #44362 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44362/consoleFull)**
for PR 8093 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-151201802
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-151201793
**[Test build #44362 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44362/consoleFull)**
for PR 8093 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-150852484
**[Test build #44301 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44301/consoleFull)**
for PR 8093 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-150869715
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-150869716
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-150869698
**[Test build #44301 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44301/consoleFull)**
for PR 8093 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-150852000
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-150852007
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-150851790
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r42916821
--- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala ---
@@ -217,11 +217,14 @@ case class TaskCommitDenied(
* the task crashed the JVM.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-150698699
Looks OK, just a few style nits remaining.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r42915834
--- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala ---
@@ -217,11 +217,14 @@ case class TaskCommitDenied(
* the task crashed the JVM.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r42916428
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -367,9 +367,15 @@ private[spark] object JsonProtocol {
("Job ID" ->
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r42916732
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -367,9 +367,15 @@ private[spark] object JsonProtocol {
("Job ID" ->
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r42915637
--- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala ---
@@ -217,11 +217,14 @@ case class TaskCommitDenied(
* the task crashed the JVM.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r42916134
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -189,7 +188,8 @@ class
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-150401167
Hi all, thanks for your feedback. I have updated this branch now, ran the
unit-tests which are green. The previously much talked about race condition has
been taken
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-132291876
Thanks Marcelo, the code more or less works but GetExecutorLossReason is
getting called twice. So, I need to work that out with @mccheah. Once that's
resolved, I
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-131947059
@markgrover you need to update your branch to fix conflicts, after that we
can trigger tests for this.
---
If your project is set up for it, you can reply to this email
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-131185171
I am have ported (aka stolen) some relevant ideas from #8007 here, just
pushed another commit. I still have some concerns regarding the testing
thereof, so I am
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130865695
Thanks @vanzin. I looked at #8007 and some pieces of it are definitely very
relevant. So, I will use some of the ideas from there.
In full disclosure, I did
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130803328
I haven't looked closely at the code yet, but #8007 probably needs to fix a
similar issue to the race you have here; might be a source of ideas.
---
If your project is
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130385210
I chatted w/ Mark about this a bit. I *think* both calls will get to
[`removeExecutor` in
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130408846
Updated the review incorporating all the review feedback. In the meanwhile,
let me poke a little more into the race condition and get back with what I find.
---
If
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130082220
@markgrover can you post a screenshot?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130084493
@andrewor14
[here](https://issues.apache.org/jira/secure/attachment/12749771/error_showing_in_UI.png)'s
the link from the JIRA.
---
If your project is set up for
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130103072
there are cases where the RemoveExecutor event may never come, right
That can happen, yes. In a normal RPC system, that message not arriving
would probably
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36782307
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -17,6 +17,10 @@
package
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36782853
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -795,7 +796,8 @@ private[spark] object JsonProtocol {
case
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36783358
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -110,6 +114,7 @@ private[spark] abstract class
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130016534
The race part is pretty tricky. The only solution I can think of is to
delay the posting of the `SparkListenerExecutorRemoved` event until an explicit
`RemoveExecutor`
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36782545
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -17,6 +17,10 @@
package
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36783484
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -423,6 +424,7 @@ private[yarn] class YarnAllocator(
for
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36783459
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -423,6 +424,7 @@ private[yarn] class YarnAllocator(
for
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36782151
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -19,6 +19,9 @@ package
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36782349
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -110,6 +114,7 @@ private[spark] abstract class
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36782462
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -19,6 +19,9 @@ package
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36783267
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -795,7 +796,8 @@ private[spark] object JsonProtocol {
case
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36784216
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -110,6 +114,7 @@ private[spark] abstract class
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36792140
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -110,6 +114,7 @@ private[spark] abstract class
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36793126
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -795,7 +796,8 @@ private[spark] object JsonProtocol {
case
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36793927
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -795,7 +796,8 @@ private[spark] object JsonProtocol {
case
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130053497
Thanks for the review, @vanzin! Indeed, the race is tricky. Yup, I like
that solution but there are cases where the RemoveExecutor event may never
come, right (in
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36794461
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -795,7 +796,8 @@ private[spark] object JsonProtocol {
case
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130137795
If you only have one event, then just do the best you can with that one.
It's a little unexpected to receive two removed events for a single
added one. It's
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36825910
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -435,19 +437,24 @@ private[yarn] class YarnAllocator(
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36825911
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -423,6 +424,7 @@ private[yarn] class YarnAllocator(
for
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130159464
Thanks for the review, @squito. I did consider that as well. The UI listens
to whatever is sent as `SparkListenerTaskEnd` message by the
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36819713
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -435,19 +437,24 @@ private[yarn] class YarnAllocator(
//
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36819662
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -423,6 +424,7 @@ private[yarn] class YarnAllocator(
for
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130167519
guess we could do that. My concern is that the race is probably always
going to be won by the executor disconnect message (instead of the explicit
RemoveExecutor
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130133128
Just throwing an idea out on dealing with the race for getting the full
reason (without fully understanding the details yet myself): couldn't this just
be handled at the
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-129706966
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user markgrover opened a pull request:
https://github.com/apache/spark/pull/8093
SPARK-9790: [YARN] Expose in WebUI if NodeManager is the reason why
executors were killed.
You can merge this pull request into a Git repository by running:
$ git pull
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-129709754
This pull request is meant to achieve two goals:
1. Show in driver logs, primarily in yarn client mode, if YARN is killing
containers because of one or more of
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-129710600
Jenkins, please test this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-129712836
There's one thing I'd really appreciate people's thoughts on:
There seems to be a race condition related to displaying the error message
in the UI. When YARN
92 matches
Mail list logo
