[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-05 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15102 > Merged this to master, @zsxwing do you have another PR for 2.0. See #15367 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-05 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/15102 Merged this to master, @zsxwing do you have another PR for 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66398/ Test PASSed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66398 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66398/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66397/ Test PASSed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66397 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66397/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66396/ Test PASSed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66396 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66396/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66398/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66397/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66396/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66345/ Test PASSed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66345 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66345/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66338/ Test PASSed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66338 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66338/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66338 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66338/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #3294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3294/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #3294 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3294/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66289/ Test FAILed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66289 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66289/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66288/ Test FAILed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66288 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66288/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66285/ Test PASSed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66285/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66289/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66283/ Test PASSed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66283 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66283/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66288/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66287/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66287/ Test FAILed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66287/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66285/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66283 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66283/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66274/ Test PASSed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66274 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66274/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66265/ Test PASSed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66265 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66265/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-10-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66274/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-29 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 > It would be nice to be able to do something other than earliest/latest. That's what Assign and the starting offset arguments to the Subscribe strategies are for. The

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-29 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15102 I spent a while playing around with this today on a real cluster, and overall it is pretty cool! I have a few suggestions we should implement in the long run, but these can probably be done in

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66047/ Test PASSed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66047/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66047/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-27 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15102 FYI: #15274 adds support for parsing JSON from the key/value into a Spark SQL `StructType` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66005/ Test FAILed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66005 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66005/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #66005 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66005/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-27 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15102 Just pushed to verify if the workaround for https://issues.apache.org/jira/browse/KAFKA-1894 does work on Jenkins. Not ready for another round of review. --- If your project is set up for it, you

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-27 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 Ok, so this kind of thing is why I was concerned about the copy, paste, randomly change things approach to developing this module. > (5) Topics are deleted when a Spark job is runinng,

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-27 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15102 We can add an option to allow the user failing the query instead of just logging the warning. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-27 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15102 Right now since there are some arguments about how to handle various failures, I'm listing what I found via stress test to discuss: (1) Kafka APIs fail because we cannot connect to Kafka

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-26 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 Ok, finished a line-by-line compare + comment. The biggest thing I'm having trouble reconciling is the stated emphasis on limiting user options in order to give guarantees, yet throwing

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65937/ Test FAILed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #65937 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65937/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #65937 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65937/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65926/ Test PASSed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #65926 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65926/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #65926 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65926/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65845/ Test FAILed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #65845 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65845/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #65845 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65845/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-23 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 Source.getOffset returns an sql Offset, not a kafka offset. At this point the current plan is to remove the ordering requirement for sql Offset, which makes the whole discussion of

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-23 Thread jwbear
Github user jwbear commented on the issue: https://github.com/apache/spark/pull/15102 Which offset does getOffset() return one from the partition or something you created? Because it looked like when you were hashing them, you were returning partition offsets. --- If your project

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-23 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 We aren't comparing ordering of offsets across partitions, and I don't think that was ever in consideration. At this point the most likely candidate for the global ordering is implicit in

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread jwbear
Github user jwbear commented on the issue: https://github.com/apache/spark/pull/15102 Just curious looking at this, if you are comparing "sequential" offsets across partitions a rebalance would definitely affect this and, unless something has changed, it probably not a good idea to

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 > I agree that if/when we add that ability to add existing partitions midstream we'd probably need to add two offsets in to the SQL offset for new partitions. It's not just existing

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15102 > "I want to be able to add a topicpartition mid stream, but I don't want to start it from the beginning." I see, I was thinking only of new topics that appear that match your pattern. I

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 @tdas I think as long as marmbrus' PR to remove comparable from the interface works for sane variations of subscription changes it's the best way to go. I'm honestly fine with someone getting

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15102 Comparable requirement removed in #15207. > I think in the absence of prior information about the position in a topicpartition, you start a new batch on topic B starting from wherever the

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/15102 @koeninger I did some independent brainstorming with @zsxwing on topic deletion, and yeah I agree with you that attempting to account for deleted topics in the offset in the KafkaSourceOffset such

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 > For streaming you already know what the global order is, because you know when you asked for A and B. I agree that we should probably remove the comparable requirement from Offset in favor of

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15102 For streaming you already know what the global order is, because you know when you asked for A and B. I agree that we should probably remove the comparable requirement from `Offset` in favor of

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 @tdas moving this conversation back to the PR that's linked from the public jira > yeah, i am trying to figure out all the options and write up something to so that we are clear on the

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65785/ Test FAILed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #65785 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65785/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #65785 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65785/consoleFull)** for PR 15102 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 This is pretty much the fundamental issue. Kafka offsets alone aren't capable of meeting the SQL Offset interface as defined. I think that means the Offset interface needs to be

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15102 > PR with failing test indicating at least one reason why it's wrong from an end-user perspective: @koeninger Thanks for writing the test. Yes, we are aware of this issue. However, it's

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-21 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 > I'd want to see some test cases though that show why the current implementation is wrong from an end-user perspective if it needs to block merging initial kafka support. PR with

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-21 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15102 I asked @koeninger to clarify the specific suggestions he is referring to above, here's my response: > [Comments here and on JIRA relating to concerns with the `Offset` implementation]

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65694/ Test PASSed. ---

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

  1   2   >