[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2017-08-02 Thread BiyuHuang
Github user BiyuHuang commented on the issue: https://github.com/apache/spark/pull/11863 I'm wondering that why the setting "enable.auto.commit" existed, but it was set to false by default and I could't modify it . Anyway, how do I use it ? --- If your project is set up for it, you

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2017-08-02 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 You won't get any reasonable semantics out of auto commit, because it will commit on the driver without regard to what the executors have done. On Aug 2, 2017 21:46, "Wallace Huang"

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2017-08-02 Thread BiyuHuang
Github user BiyuHuang commented on the issue: https://github.com/apache/spark/pull/11863 hey,I have an question about the setting "auto.commit.enable", It could be changed Because I wanna save the offsets information to zookeeper cluster. --- If your project is set up for

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-30 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 I played around with the API and I found a few issues 1. I mentioned above, case classes lead to problems in the public API. The API could be simpler, and same for both Java and Scala users (dont

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-30 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 LGTM. Merging this to master and 2.0. Thank you very much @koeninger for this awesome effort. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61513/ Test PASSed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61513 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61513/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #3150 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3150/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #3151 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3151/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61506/ Test FAILed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61506/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61513 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61513/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 sounds good. thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 I'll do the scaladoc fix and the package rename. I think the package rename is fine even if it did work with docs, just to disambiguate things. Will start a separate ticket for

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #3151 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3151/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #3150 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3150/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 Well.. after the tests pass. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 Aah, right. My bad. In that case, there arent major issues as far as i can see, let me merge this, and test how the docs look like. I am pretty sure its going to cause trouble with two KafkaUtils. And

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61495/ Test FAILed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61495 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61495/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 You do need CanCommitOffsets because DirectKafkaInputDstream is now private, so otherwise you have nothing to cast to to access that method. On Jun 29, 2016 7:11 PM, "Tathagata Das"

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 Overall, this is looking good. Two high level points. 1. Now we have two subprojects both creating org.apache.spark.streaming.kafka.KafkaUtils. I think this is going to cause problems

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 @zsxwing Thanks for the fixes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61506/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/11863 Finished my round of reviewing. Some some nits and one question about `commitAsync` left. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61495 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61495/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 @koeninger Sorry, I did not get a chance to review your changes because I was too busy with the programming guide PR (as you probably have guessed). I will jump back into your PR once that is merged,

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61436/ Test PASSed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61436 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61436/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61436/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61359/ Test PASSed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61359 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61359/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61359 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61359/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-27 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 I'm already logging at error level a message about serializability / suggestion to use map first before persist, which seems like the most common case that will come up. I can do something

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-27 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 Yeah, I guessed that its not a good idea, but I know some people do that. So I wanted to understand what actually would happen. --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-27 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 If they're using the same group id, that's at least plausible, but it's in general not a good idea to run two different applications with the same group id at the same time. On

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-27 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 what happens in the case where a partition was being processed by some other application, then gets assigned to this direct kafka stream. will it pick up the offsets that was last consumed by the

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-27 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 I was going to leave some code comments once I'm not at work & have time to make another patch, but the quick answer is that should be determined in the consumer by auto.offset.reset --- If

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-27 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 And I think you missed answering one of the comments in the code. Let me ask it once again. Can you explain what happens in the subscribe case when the consumer gets subscribed to a new partition?

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-27 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 3. Okay, no strong feelings on this one. 4b. I dont think there is a need for Java tests for methods which need class tags. Those are probably least likely to be used. --- If your project

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-27 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 Regarding your general comments 1 + 2 sure, will add 3. The reason ConsumerStrategy is last is because it's much more natural for inline one-off subclasses of it to customize

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-27 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 I took a quick glance, and I think the following. 1. I like the LocationStrategy. But needs Java tests for each of the strategies. 2. Regarding ConsumerStrategy, I think I am trying to

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61272/ Test PASSed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61272 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61272/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-26 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 Moved the prefered location and consumer creation strategies to an explicit interface, let me know if that's more usable from your point of view. Scala api looks like

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61272 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61272/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61259/ Test FAILed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61259 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61259/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61259/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61252/ Test FAILed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61252 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61252/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #61252 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61252/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-25 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 Sorry for the delayed reply, I had travel plans that had to be canceled due to a family emergency (everyone's mostly ok). 1 + 2, I understand that preferred locations is not a

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-24 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 @koeninger Ping! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-23 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 *1+2. Architecture* I had to double check the code to understand the use of sorted list of executors to get consistent topic-to-preferred locations. PLEASE WRITE MORE SCALA DOCS on how the logic

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-23 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 1. When the user has no preferences, the system already does figure out preferred locations, and not in a random way as you claimed. 2. So lets talk concretely, not hypothetically.

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-23 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 1. I didnt quite get it when you meant "But your description of what the code is currently doing is not accurate, and your recommendation does not meet the use cases." I just collapsed the three

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-22 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 Regarding the possible arguments to preferredHosts, I'm pretty sure you're misunderstanding what happens. There are 3 uses cases here: - I don't care where things run. I pass

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-22 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 @koeninger Okay, I think I understand the code at a high level. And I have a number of architecture and API concerns. Let me start with the highest level architectural concern to keep the discussion

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-22 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/11863 @koeninger I am reading the whole code and understand how it works. I am just marking some style stuff along the way, dont worry about them for now. Let me understand the whole thing and then i will

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-16 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 @rxin I'm pretty confident that fix addressed the test issue. It's passed twice now, the prior failure was unrelated. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60622/ Test PASSed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #60622 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60622/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #60622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60622/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 Jenkins, retest this please On Jun 15, 2016 8:43 PM, "UCB AMPLab" wrote: > Merged build finished. Test FAILed. > > — > You are receiving this

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60606/ Test FAILed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #60606 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60606/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #60606 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60606/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60575/ Test FAILed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #60575 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60575/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60572/ Test FAILed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #60572 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60572/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #60575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60575/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #60572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60572/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60557/ Test FAILed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #60557 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60557/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #60557 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60557/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-15 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60550/ Test FAILed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #60550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60550/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #60550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60550/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11863 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60521/ Test FAILed. ---

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #60521 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60521/consoleFull)** for PR 11863 at commit

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11863 **[Test build #60521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60521/consoleFull)** for PR 11863 at commit

  1   2   >