Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-64323860
Andrew's got a patch for this: #3447
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-64316156
It looks like this patch may have introduced a race-condition / bug during
multi-master failover: https://issues.apache.org/jira/browse/SPARK-4592. I'm
working on a
Github user mccheah commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59803881
@JoshRosen agreed with @ash211, this is really good.
Are there any actual comments on the PR, or can it be merged? =)
---
If your project is set up for it, you
Github user CodingCat commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59807896
@JoshRosen , this is awesome to test Spark integration with Docker
@mccheah , this PR is LGTM now, except that we exposed too many
should-be-private members in
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59810037
@CodingCat, Worker is private[spark], so what is the nature of your
concern? In fact, I'm wondering whether we really want the changes in this PR
that make some
Github user CodingCat commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59810532
@markhamstra , yeah, my concern is just this, though Worker is marked as
private[spark], is it a good practice to expose every detail in the
implementation to the
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r19101917
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -166,26 +178,47 @@ private[spark] class Worker(
}
}
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59811803
A legitimate concern, and certainly something that could be worked up into
a JIRA issue and separate pull request. But it's not a very pressing issue
since nothing
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59812443
As a general principle, you should use the most private access modifiers
that are sufficient. We can always make methods / fields _more_ visible, but
it's much harder
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r19102299
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -64,8 +66,17 @@ private[spark] class Worker(
// Send a heartbeat
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r19102413
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -166,26 +178,47 @@ private[spark] class Worker(
}
}
Github user CodingCat commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59814026
sure, I created the JIRA: https://issues.apache.org/jira/browse/SPARK-4011
---
If your project is set up for it, you can reply to this email and have your
reply
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59814881
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21926/consoleFull)
for PR 2828 at commit
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59815368
This looks good to me. Thanks! I'm going to merge this into `master`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2828
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59823491
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59823485
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21926/consoleFull)
for PR 2828 at commit
Github user mccheah commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59824518
The PR doesn't seem to be related to the unit tests that failed. How shall
we tackle this issue?
---
If your project is set up for it, you can reply to this email and
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59833561
Don't worry about it. This test is a little flaky and will be fixed
shortly. I highly doubt that the test failure is caused by this PR.
---
If your project is set up
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59602394
**tl;dr**: _this patch looks pretty good to me based on the testing that
I've done so far. For my own interest / fun, I'd like to find a way to extend
my test
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59639085
This is EXCELLENT work @JoshRosen ! Looking forward to future integration
tests that cover these sorts of behaviors.
---
If your project is set up for it, you can reply
Github user ash211 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r19002923
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -362,9 +372,19 @@ private[spark] class Worker(
}
}
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r19011614
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -362,9 +372,19 @@ private[spark] class Worker(
}
}
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59596562
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21873/consoleFull)
for PR 2828 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59598067
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21873/consoleFull)
for PR 2828 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59598071
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
GitHub user mccheah opened a pull request:
https://github.com/apache/spark/pull/2828
[SPARK-3736] Workers reconnect when disassociated from the master.
Before, if the master node is killed and restarted, the worker nodes
would not attempt to reconnect to the Master. Therefore,
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59406399
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user mccheah commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59408043
One remark is that there are no automated tests in this commit for now.
I was unsuccessful in setting up TestKit to emulate a worker and master
sending messages
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18976036
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -94,6 +96,7 @@ private[spark] class Worker(
val finishedExecutors =
Github user ash211 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18976148
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -341,7 +341,11 @@ private[spark] class Master(
case
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18976594
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -365,6 +375,16 @@ private[spark] class Worker(
def
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18976619
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -365,6 +375,16 @@ private[spark] class Worker(
def
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18977018
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -341,7 +341,11 @@ private[spark] class Master(
case
Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18977288
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -341,7 +341,11 @@ private[spark] class Master(
case
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18978412
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -362,9 +372,19 @@ private[spark] class Worker(
}
}
Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18978981
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -362,9 +372,19 @@ private[spark] class Worker(
}
}
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59425292
add to whitelist
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59425990
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21818/consoleFull)
for PR 2828 at commit
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18985861
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -362,9 +372,19 @@ private[spark] class Worker(
}
}
Github user markhamstra commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18985880
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -243,6 +249,10 @@ private[spark] class Worker(
Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18986188
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -362,9 +372,19 @@ private[spark] class Worker(
}
}
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18986488
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -362,9 +372,19 @@ private[spark] class Worker(
}
}
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59430399
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21822/consoleFull)
for PR 2828 at commit
Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18986702
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -362,9 +372,19 @@ private[spark] class Worker(
}
}
Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18986941
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -362,9 +372,19 @@ private[spark] class Worker(
}
}
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18988031
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -362,9 +372,19 @@ private[spark] class Worker(
}
}
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18988140
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -362,9 +372,19 @@ private[spark] class Worker(
}
}
Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/2828#discussion_r18988742
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -362,9 +372,19 @@ private[spark] class Worker(
}
}
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59435763
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21818/consoleFull)
for PR 2828 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59435771
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59439367
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21822/consoleFull)
for PR 2828 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2828#issuecomment-59439376
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
53 matches
Mail list logo
