[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21220 Yeah. This refactoring was needed. Now it should be easier to make such changes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user jose-torres commented on the issue: https://github.com/apache/spark/pull/21220 LGTM. Obviously shouldn't block this PR, but MicroBatchExecution is structured in a way that makes it hard to review changes like this. It seems like changing the condition under which new batches are run should have been a much more local change than it ended up having to be. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21220 @brkyvz Answers to your questions. 1. We have already fixed those emptyDF optimizations. The optimization only quick in `df.isStreaming = false`, and emptyDFs generated by sources should be have isStreaming = true. That's for v1 sources. For v2 sources, the engine already takes care of that by making sure that `StreamingDataSourceV2Relation.isStreaming` is true where `StreamingDataSourceV2Relation` is the logical plan leaf inserted into the micro-batch logical plan irrespective of empty or not. 2. I will do both type of timeouts in flatMapGroupsWithState in later PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21220 @brkyvz Answers to your questions. 1. We have already fixed those emptyDF optimizations. The optimization only quick in `df.isStreaming = false`, and emptyDFs generated by sources should be have isStreaming = true. That's for v1 sources. For v2 sources, the engine already takes care of that by making sure that `StreamingDataSourceV2Relation.isStreaming` is true where `StreamingDataSourceV2Relation` is the logical plan leaf inserted into the micro-batch logical plan irrespective of empty or not. 2. I will do both type of timeouts in flatMapGroupsWithState in later PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90172/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21220 **[Test build #90172 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90172/testReport)** for PR 21220 at commit [`da3fd2f`](https://github.com/apache/spark/commit/da3fd2f8510482e3e71cc37a9da2207e3aef1ef0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2891/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21220 **[Test build #90172 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90172/testReport)** for PR 21220 at commit [`da3fd2f`](https://github.com/apache/spark/commit/da3fd2f8510482e3e71cc37a9da2207e3aef1ef0). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90078/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21220 **[Test build #90078 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90078/testReport)** for PR 21220 at commit [`7fa11c0`](https://github.com/apache/spark/commit/7fa11c0ac362ace43ce02dee6309a3a632b0c3ee). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class WatermarkTracker extends Logging ` * `trait MemorySinkBase extends BaseStreamingSink ` * `class MemorySink(val schema: StructType, outputMode: OutputMode) extends Sink` * `class MemorySinkV2 extends DataSourceV2 with StreamWriteSupport with MemorySinkBase with Logging ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90080/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21220 **[Test build #90080 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90080/testReport)** for PR 21220 at commit [`7fa11c0`](https://github.com/apache/spark/commit/7fa11c0ac362ace43ce02dee6309a3a632b0c3ee). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class WatermarkTracker extends Logging ` * `trait MemorySinkBase extends BaseStreamingSink ` * `class MemorySink(val schema: StructType, outputMode: OutputMode) extends Sink` * `class MemorySinkV2 extends DataSourceV2 with StreamWriteSupport with MemorySinkBase with Logging ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2830/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21220 **[Test build #90080 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90080/testReport)** for PR 21220 at commit [`7fa11c0`](https://github.com/apache/spark/commit/7fa11c0ac362ace43ce02dee6309a3a632b0c3ee). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21220 @brkyvz @zsxwing @jose-torres --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2829/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21220 jenkins retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2828/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21220 **[Test build #90078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90078/testReport)** for PR 21220 at commit [`7fa11c0`](https://github.com/apache/spark/commit/7fa11c0ac362ace43ce02dee6309a3a632b0c3ee). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
