[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5210/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1555/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89028/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20958 **[Test build #89028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89028/testReport)** for PR 20958 at commit [`919bd11`](https://github.com/apache/spark/commit/919bd112a4a556abe21aa0d2c609e2468af89316). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20958 **[Test build #89028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89028/testReport)** for PR 20958 at commit [`919bd11`](https://github.com/apache/spark/commit/919bd112a4a556abe21aa0d2c609e2468af89316). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2074/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20958 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89027/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20958 **[Test build #89027 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89027/testReport)** for PR 20958 at commit [`919bd11`](https://github.com/apache/spark/commit/919bd112a4a556abe21aa0d2c609e2468af89316). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2073/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20958 **[Test build #89027 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89027/testReport)** for PR 20958 at commit [`919bd11`](https://github.com/apache/spark/commit/919bd112a4a556abe21aa0d2c609e2468af89316). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20958 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89025/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20958 **[Test build #89025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89025/testReport)** for PR 20958 at commit [`919bd11`](https://github.com/apache/spark/commit/919bd112a4a556abe21aa0d2c609e2468af89316). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20958 @tdas , by thought about your suggestion about "failOnDataLoss" option, I made a similar proposal on socket source, would you please review again. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2072/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20958 **[Test build #89025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89025/testReport)** for PR 20958 at commit [`919bd11`](https://github.com/apache/spark/commit/919bd112a4a556abe21aa0d2c609e2468af89316). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20958 Thanks @tdas for your comments. I agree that socket source should only be used in testing. But it doesn't mean that it can throw weird exception in testing env. For example, if we're dumping socket source data to parquet in test env, parquet source requires checkpoint (even in test env), but socket source should not honor checkpoint data, this basically not works also in test env. My thinking is that cases in test env should also be worked basically, but doesn't have in-production features like checkpointing. Maybe we can improve error log, but still cases like socket -> parquet doesn't work. Let me think about how to better address this issue. Thanks again. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user tdas commented on the issue: https://github.com/apache/spark/pull/20958 We have made it clear that sockets is ONLY for testing and will not recover data from checkpoints. So I see no problem that it throws errors when attempting to recover. May we can improve the error message by making it clear that recovery is not supported. If you indeed want to forget lost data and proceed, then that should be an opt-in. We could do this by explicitly setting a source option (like failOnDataLoss = false in Kafka source). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20958 Not sure why the test is not triggered, maybe jenkins is down. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20958 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20958 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20958 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20958 Jenkins, test this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
