Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-101463101
I have revised the implementation of this PR in a followup PR #6096
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub a
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-97531627
This PR was reverted because I had used MutableBoolean which does not seem
to work well with Hadoop 1.0.4. I reopened the PR in #5773.
---
If your project is set up for it,
Github user zzcclp commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-96484530
hi, @tdas , why this PR was be reverted?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project do
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/5428
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95678011
Merging this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enab
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95497583
[Test build #697 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/697/consoleFull)
for PR 5428 at commit
[`94db63c`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95474793
[Test build #697 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/697/consoleFull)
for PR 5428 at commit
[`94db63c`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95439240
[Test build #696 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/696/consoleFull)
for PR 5428 at commit
[`94db63c`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95398368
[Test build #695 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/695/consoleFull)
for PR 5428 at commit
[`94db63c`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95383691
[Test build #695 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/695/consoleFull)
for PR 5428 at commit
[`94db63c`](https://github
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95323488
LGTM pending Jenkins.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28910108
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
---
@@ -655,6 +656,7 @@ object JavaStreamingContext {
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95305068
[Test build #694 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/694/consoleFull)
for PR 5428 at commit
[`94db63c`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95206224
[Test build #693 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/693/consoleFull)
for PR 5428 at commit
[`94db63c`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95174821
[Test build #693 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/693/consoleFull)
for PR 5428 at commit
[`94db63c`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95147418
[Test build #30751 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30751/consoleFull)
for PR 5428 at commit
[`94db63c`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95147426
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95128390
[Test build #30751 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30751/consoleFull)
for PR 5428 at commit
[`94db63c`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95074518
[Test build #30743 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30743/consoleFull)
for PR 5428 at commit
[`524f519`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95074534
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95073586
[Test build #30743 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30743/consoleFull)
for PR 5428 at commit
[`524f519`](https://githu
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28850543
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
---
@@ -655,6 +656,7 @@ object JavaStreamingContext {
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28850515
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -621,19 +636,59 @@ object StreamingContext extends Logging {
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28847514
--- Diff:
streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java ---
@@ -1707,6 +1708,71 @@ public Integer call(String s) throws Exception {
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28847303
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
---
@@ -328,6 +330,138 @@ class StreamingContextSuite extends FunSuite
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28847268
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -107,6 +107,15 @@ class StreamingContext private[streaming] (
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28841893
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
---
@@ -328,6 +330,138 @@ class StreamingContextSuite extends Fun
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28841856
--- Diff:
streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java ---
@@ -1707,6 +1708,71 @@ public Integer call(String s) throws Exception {
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28841681
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
---
@@ -655,6 +656,7 @@ object JavaStreamingContext {
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28841661
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -621,19 +636,59 @@ object StreamingContext extends Logging {
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95010237
[Test build #692 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/692/consoleFull)
for PR 5428 at commit
[`eabd092`](https://githu
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28841304
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -107,6 +107,15 @@ class StreamingContext private[streaming] (
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28841272
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala ---
@@ -271,7 +282,10 @@ object CheckpointReader extends Logging {
}
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95002858
Yes, this is not intended to solve SPARK-5206.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your proje
Github user zzcclp commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-95000892
@tdas , I tested streaming recovering from checkpoint with this PR, it
failed if it use accumulators, so this assuredly can't solve [issue
SPARK-5206](https://issues.apach
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-94996743
[Test build #692 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/692/consoleFull)
for PR 5428 at commit
[`eabd092`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-94978575
[Test build #691 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/691/consoleFull)
for PR 5428 at commit
[`eabd092`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-94967558
[Test build #691 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/691/consoleFull)
for PR 5428 at commit
[`eabd092`](https://github
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-94586606
Jenkins, test this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this featur
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-93671163
Jenkins, test this again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-93620037
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-93620032
[Test build #30389 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30389/consoleFull)
for PR 5428 at commit
[`eabd092`](https://gith
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28478175
--- Diff:
streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java ---
@@ -987,12 +988,12 @@ public void testPairMap2() { // Maps pair -> single
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-93611520
@JoshRosen Please take a quick look at the Function.
@jerryshao @harishreedharan I have updated the patch with Java API and unit
tests. I think I am going to create a se
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-93610784
[Test build #30389 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30389/consoleFull)
for PR 5428 at commit
[`eabd092`](https://githu
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28388452
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala ---
@@ -77,7 +77,8 @@ object Checkpoint extends Logging {
}
/*
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-93122446
@all This is still a WIP. Adding the equivalent Java API requires
refactoring the existing `JavaStreamingContext.getOrCreate` to not use
`JavaStreamingContextFactory` and us
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28384609
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -114,11 +123,15 @@ class StreamingContext private[streaming] (
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28384469
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala ---
@@ -77,7 +77,8 @@ object Checkpoint extends Logging {
}
/*
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-93120421
@zzcclp I dont think it will solve this issue directly. But it may allow
the SparkContext to be re-initialized properly before the StreamingContext is
recreated from checkpo
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28384336
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala ---
@@ -271,7 +282,10 @@ object CheckpointReader extends Logging {
})
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28086500
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -114,11 +123,15 @@ class StreamingContext private[streamin
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28085485
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala ---
@@ -77,7 +77,8 @@ object Checkpoint extends Logging {
}
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-91232170
It looks good to me. Simply curious about the scenarios of this usage, is
there any situation where streaming context is failed but spark context is
still existed when
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28060398
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala ---
@@ -271,7 +282,10 @@ object CheckpointReader extends Logging {
}
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-91214515
@tdas Yeah, will do.
@zzcclp I'm not sure, maybe you can take a try, from my guess, this could
possibly work, since accumulator is registered in SparkContext, w
Github user zzcclp commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-91180116
@tdas , can this RP resolve [this
issue](https://issues.apache.org/jira/browse/SPARK-5206)?
Restart a streaming app from checkpoint incorrectly if using accumulators
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-91176687
@jerryshao Mind taking a look at this? Its still WIP as unit tests are
commented out.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-91073266
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-91073251
[Test build #29898 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29898/consoleFull)
for PR 5428 at commit
[`36a7823`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5428#issuecomment-91056649
[Test build #29898 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29898/consoleFull)
for PR 5428 at commit
[`36a7823`](https://githu
61 matches
Mail list logo
