[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-29 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17308 LGTM. Merging to master and 2.2. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77499/ Test PASSed. ---

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #77499 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77499/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-29 Thread ScrapCodes
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/17308 Thanks @viirya and @zsxwing. I have tried to address you comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #77499 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77499/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-26 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/17308 LGTM and few minor comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread ScrapCodes
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/17308 @marmbrus Thank you for taking a look again. Surely, shut down hook is not ideal for closing kafka producers. In fact, for the case of kafka sink, it might be correct to skip cleanup step. I

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77410/ Test PASSed. ---

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #77410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77410/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #77410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77410/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread ScrapCodes
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/17308 Jenkins, retest this please ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread ScrapCodes
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/17308 Build is failing due to "Our attempt to download sbt locally to build/sbt-launch-0.13.13.jar failed. Please install sbt manually from http://www.scala-sbt.org/; --- If your project is set up

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77358/ Test FAILed. ---

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #77358 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77358/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #77358 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77358/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread ScrapCodes
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/17308 Jenkins, retest this please ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77357/ Test FAILed. ---

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #77357 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77357/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #77357 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77357/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #76932 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76932/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76932/ Test PASSed. ---

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #76932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76932/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-15 Thread ScrapCodes
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/17308 SPARK-20737 is created to look into cleanup mechanism in a separate JIRA. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76890/ Test PASSed. ---

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #76890 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76890/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #76890 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76890/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76872/ Test FAILed. ---

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #76872 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76872/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #76872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76872/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76869/ Test PASSed. ---

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #76869 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76869/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #76869 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76869/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76868/ Test PASSed. ---

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #76868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76868/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #76868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76868/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-08 Thread ScrapCodes
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/17308 Hi @marmbrus and @brkyvz, Thanks a lot of taking a look. @marmbrus You are right, we should have another way to canonicalize kafka params. I can only think of appending a unique id to

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-04 Thread brkyvz
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/17308 Taking a look --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-05-04 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/17308 @marmbrus @zsxwing @tdas This needs attention from someone --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-04-12 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/17308 Just to throw in my two cents, a change like this is definitely needed, as is made clear by the second sentence of the docs

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-04-03 Thread ScrapCodes
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/17308 I can further confirm, that in logs, a kafkaproducer instance is created almost every instant. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-03-30 Thread ScrapCodes
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/17308 @tdas ping ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-03-21 Thread ScrapCodes
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/17308 @tcondie and @zsxwing any comments on this patch. I would be happy, if this bug is fixed before 2.2 is released. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-03-17 Thread ScrapCodes
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/17308 Please take a look, @tcondie @zsxwing ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74658/ Test PASSed. ---

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #74658 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74658/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #74658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74658/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74646/ Test FAILed. ---

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #74646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74646/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74644/ Test FAILed. ---

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #74646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74646/testReport)** for PR 17308 at commit

[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...

2017-03-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #74644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74644/testReport)** for PR 17308 at commit