[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/15355 I have re-enables the kafka test. This PR now only contains a change to build infrastructure. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15355 **[Test build #66487 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66487/consoleFull)** for PR 15355 at commit [`a2ba414`](https://github.com/apache/spark/commit/a2ba414ae97e859b4c50d96e8ecebb0d1bc727c8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15355 @zsxwing good eye, thanks. It's not that auto.offset.reset.earliest doesn't work, it's that there's a potential race condition that poll gets called twice slowly enough for consumer position to be modified before topicpartitions are paused. https://github.com/apache/spark/pull/15387 should address that. It's something that whoever works on the duplicated equivalent code in the structured streaming module is going to have to address, also. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15355 @koeninger you can download the unit test logs from https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.6/1756/artifact/ I saw the offsets of the first batch (pat1 0 16 16) was wrong in the unit test logs: ``` 16/10/05 20:42:46.313 streaming-start INFO JobGenerator: Started JobGenerator at 1475725367000 ms 16/10/05 20:42:46.313 streaming-start INFO JobScheduler: Started JobScheduler 16/10/05 20:42:46.313 pool-1-thread-1-ScalaTest-running-DirectKafkaStreamSuite INFO StreamingContext: StreamingContext started 16/10/05 20:42:47.015 JobGenerator INFO JobScheduler: Added jobs for time 1475725367000 ms 16/10/05 20:42:47.016 JobScheduler INFO JobScheduler: Starting job streaming job 1475725367000 ms.0 from job set of time 1475725367000 ms 16/10/05 20:42:47.017 streaming-job-executor-0 INFO DirectKafkaStreamSuite: pat2 0 3 16 16/10/05 20:42:47.017 streaming-job-executor-0 INFO DirectKafkaStreamSuite: pat1 0 16 16 ``` Seems the `earliest` config does not work. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15355 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66463/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15355 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15355 **[Test build #66463 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66463/consoleFull)** for PR 15355 at commit [`5ba78b6`](https://github.com/apache/spark/commit/5ba78b66c39c27c5e518f2cf394b64c8df88603e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15355 **[Test build #66463 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66463/consoleFull)** for PR 15355 at commit [`5ba78b6`](https://github.com/apache/spark/commit/5ba78b66c39c27c5e518f2cf394b64c8df88603e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15355 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15355 **[Test build #66461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66461/consoleFull)** for PR 15355 at commit [`0bac9ff`](https://github.com/apache/spark/commit/0bac9ffac90dbcd0e40e55f9812796fa6e9c8a28). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15355 **[Test build #66461 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66461/consoleFull)** for PR 15355 at commit [`0bac9ff`](https://github.com/apache/spark/commit/0bac9ffac90dbcd0e40e55f9812796fa6e9c8a28). * This patch **fails executing the `dev/run-tests` script**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15355 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66461/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15355 Good point, I don't see separate config for the 0.10 module in `dev/sparktestsupport/modules.py` though there is for 0.8 module. That could be a better way forward. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15355 Actually we do have the infra to not run these tests if it is just an unrelated module change. Was those not setup for Kafka? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15355 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15355 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66361/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15355 **[Test build #66361 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66361/consoleFull)** for PR 15355 at commit [`b7074d4`](https://github.com/apache/spark/commit/b7074d48159804035eaf00e1abed35e408684b42). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15355 I have generally been unable to reproduce these kinds of test failures on my local environment, and don't have access to the build server, so trying fix without repro is pretty much shooting randomly in the dark. It does seem unfortunate to me that we're effectively doing full integration tests on every PR, even if a patch has changed something (e.g. MLLib) that couldn't possibly affect the modules in /external --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING] Disable Kafka 010 pattern based...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15355 **[Test build #66361 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66361/consoleFull)** for PR 15355 at commit [`b7074d4`](https://github.com/apache/spark/commit/b7074d48159804035eaf00e1abed35e408684b42). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15355: [SPARK-17782][STREAMING]Disable Kafka 010 pattern based ...
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/15355 cc @koeninger any idea why this flaky? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
