[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19134 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19134 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19134 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81674/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19134 **[Test build #81674 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81674/testReport)** for PR 19134 at commit [`d888f7b`](https://github.com/apache/spark/commit/d888f7b4b457d537c6875de31cbd77f5460c7d3b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19134 **[Test build #81674 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81674/testReport)** for PR 19134 at commit [`d888f7b`](https://github.com/apache/spark/commit/d888f7b4b457d537c6875de31cbd77f5460c7d3b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19134 OK by me, if it's all on-purpose. So, am I right that all of the Pyspark Kafka integration is effectively deprecated as of this change, because it depends on now-deprecated 0.8 support? just want to make sure I'm reading that right, and if so I'll try to mark it as such. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19134 There's already a jira about why 0.10 doesn't have python support, https://issues-test.apache.org/jira/browse/SPARK-16534 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19134 LGTM. The pyspark vs. 0.10 discussion seems like it belongs in a separate bug. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19134 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81607/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19134 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19134 **[Test build #81607 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81607/testReport)** for PR 19134 at commit [`8f1deee`](https://github.com/apache/spark/commit/8f1deeec075f42edd440982dae643dd498eaa641). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19134 **[Test build #81607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81607/testReport)** for PR 19134 at commit [`8f1deee`](https://github.com/apache/spark/commit/8f1deeec075f42edd440982dae643dd498eaa641). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19134 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81586/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19134 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19134 **[Test build #81586 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81586/testReport)** for PR 19134 at commit [`6373c02`](https://github.com/apache/spark/commit/6373c02079455689bad7ad09b23a34927e0c0335). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19134 **[Test build #81586 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81586/testReport)** for PR 19134 at commit [`6373c02`](https://github.com/apache/spark/commit/6373c02079455689bad7ad09b23a34927e0c0335). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19134 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19134 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81564/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19134 **[Test build #81564 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81564/testReport)** for PR 19134 at commit [`2a6f5de`](https://github.com/apache/spark/commit/2a6f5decf9be4a81f4c1bf905172b036722fb23c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19134 **[Test build #81564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81564/testReport)** for PR 19134 at commit [`2a6f5de`](https://github.com/apache/spark/commit/2a6f5decf9be4a81f4c1bf905172b036722fb23c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19134 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81515/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19134 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19134 **[Test build #81515 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81515/testReport)** for PR 19134 at commit [`ff76dab`](https://github.com/apache/spark/commit/ff76dab229f2ad7be9a6b75eb1046d1a01df87af). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19134 **[Test build #81515 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81515/testReport)** for PR 19134 at commit [`ff76dab`](https://github.com/apache/spark/commit/ff76dab229f2ad7be9a6b75eb1046d1a01df87af). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19134 Good feedback, all. I think I can fix the MiMa failure, and now the Python tests. Let me try to get that far. Yes, the examples use Kafka 0.8 in some places. It seems excessive to set up a new optional module of Kafka 0.8-only examples, so, perhaps it's time to remove them? Because there's also the suggestion to deprecate 0.8. And then that entails making 0.10 stable. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19134 I see @felixcheung . Since we have a solution to turn off Python Kafka unit test as mentioned by @vanzin , so it is fine to just mark as deprecated, not remove the code. Another thing is that we also have some Kafka related examples, maybe we should also handle them. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/19134 re kafka -https://github.com/apache/spark/pull/19134#issuecomment-327180847 - pretty sure that is still needed for DStream - just as any API we shouldn't just remove it in point releases... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19134 For python I think you just do what kinesis does: ``` # Must be same as the variable and condition defined in KinesisTestUtils.scala kinesis_test_environ_var = "ENABLE_KINESIS_TESTS" are_kinesis_tests_enabled = os.environ.get(kinesis_test_environ_var) == '1' ``` The comment is misleading though, since that variable is set in `modules.py` which you are already modifying. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19134 I think it makes sense to go ahead and put deprecation warnings in this PR as well --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19134 Yes, Python kafka.py itself is OK to leave without calling it, but the UT will involve Scala Kafka module to do the test. Currently I don't know how to address this issue. Ideally it would be better to put Python Kafka related codes to this python module, and only load in when specifically opt-in. But I'm not sure how to achieve this. Another option is to remove this python Kafka code. Previously we stopped supporting Python Spark Streaming work (https://github.com/apache/spark/pull/14340#issuecomment-235457517), so it might be OK to remove it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19134 Ah, this is where I need some advice. Is there an element of the Pyspark code that has to vary too? obviously Python itself isn't dependent on Scala, but the tests obviously can't test 0.8 if it's not there. Yes I think there are a few more issues to iron out before this works, so input is welcome. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19134 @srowen how do you handle python kafka.py, should it also be opt-in? As far as I understand looks like you don't address it in this PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19134 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81409/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19134 **[Test build #81409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81409/testReport)** for PR 19134 at commit [`6980c6a`](https://github.com/apache/spark/commit/6980c6a78b274693606d7d9578dc049c3d67aa19). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19134 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19134 **[Test build #81409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81409/testReport)** for PR 19134 at commit [`6980c6a`](https://github.com/apache/spark/commit/6980c6a78b274693606d7d9578dc049c3d67aa19). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
