Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-72949130
I can merge this for now and we can focus on that issue later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3655
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-72959021
Agreed. I will file a jira. We should discuss the issue there. This one was
more of a break-fix, but that would be a more elaborate fix.
---
If your project is
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-72784543
ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-72784728
@harishreedharan This begs a higher level questions of whether the write
ahead log (which is the probably component to fail) should have its own retries
independent of the
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-72789875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-72784691
[Test build #26710 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26710/consoleFull)
for PR 3655 at commit
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-70015327
No, this does prevent data loss - basically if the store fails multiple
times, we shutdown the receiver completely. So the new receiver which gets
started starts
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-69684611
Hi @harishreedharan , After carefully looking at the code, I think data
will not be lost even in such failure situation. For example, if we meet
exception in
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-69684946
So I think the aim of this patch is to fix the recoverable problems of data
store with retries, not prevent data loss. That's my thought, sorry for my
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-69384366
@tdas Any comments on this one, or is this one ready to go in?
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/3655#discussion_r21766800
--- Diff:
external/kafka/src/main/scala/org/apache/spark/streaming/kafka/ReliableKafkaReceiver.scala
---
@@ -201,12 +201,31 @@ class
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-66393421
I messed up the jira number in the commit. Please fix it when merging.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-66398284
Thanks Hari, seems this is a simple solution. BTW should we make `count =
3` as a configurable parameter? For others LGTM.
Original thoughts of introducing
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-66399105
[Test build #24282 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24282/consoleFull)
for PR 3655 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-66399109
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3655#discussion_r21586197
--- Diff:
external/kafka/src/main/scala/org/apache/spark/streaming/kafka/ReliableKafkaReceiver.scala
---
@@ -201,12 +201,31 @@ class ReliableKafkaReceiver[
17 matches
Mail list logo
