[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21700 **[Test build #92550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92550/testReport)** for PR 21700 at commit [`345b33a`](https://github.com/apache/spark/commit/345b33ab5b9042eb7be86b2993dc9b6306480f5d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21700 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21700 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92549/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21700 **[Test build #92549 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92549/testReport)** for PR 21700 at commit [`0819412`](https://github.com/apache/spark/commit/081941248792612000fe4a1d92be917d771117eb). * This patch **fails RAT tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21700 **[Test build #92549 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92549/testReport)** for PR 21700 at commit [`0819412`](https://github.com/apache/spark/commit/081941248792612000fe4a1d92be917d771117eb). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21700 **[Test build #92548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92548/testReport)** for PR 21700 at commit [`cab25df`](https://github.com/apache/spark/commit/cab25dfd8599a2edfdefe83ad9b9be1f827aaad0). * This patch **fails to generate documentation**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public class BoundedSortedMap extends TreeMap ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21700 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92548/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21700 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21700 **[Test build #92548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92548/testReport)** for PR 21700 at commit [`cab25df`](https://github.com/apache/spark/commit/cab25dfd8599a2edfdefe83ad9b9be1f827aaad0). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21700 Missing new line in EOF for two new Java files. Just addressed. Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21700 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21700 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92547/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21700 **[Test build #92547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92547/testReport)** for PR 21700 at commit [`45796d8`](https://github.com/apache/spark/commit/45796d8c74d0a55bf0d3a22f1c526dc764c0e924). * This patch **fails Java style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public class BoundedSortedMap extends TreeMap ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21700 **[Test build #92547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92547/testReport)** for PR 21700 at commit [`45796d8`](https://github.com/apache/spark/commit/45796d8c74d0a55bf0d3a22f1c526dc764c0e924). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21700 cc. @tdas @zsxwing @jose-torres @jerryshao @arunmahadevan @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21700 retest this, please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21700 **[Test build #92546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92546/testReport)** for PR 21700 at commit [`22f0e22`](https://github.com/apache/spark/commit/22f0e220f661b5457584ef83b1ecddc18212fa73). * This patch **fails Java style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public class BoundedSortedMap extends TreeMap ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21700 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92546/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21700 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21700 **[Test build #92546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92546/testReport)** for PR 21700 at commit [`22f0e22`](https://github.com/apache/spark/commit/22f0e220f661b5457584ef83b1ecddc18212fa73). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21700 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21700 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21700: SPARK-24717 Split out min retain version of state for me...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21700 Pasting JIRA issue description to explain why this patch is needed: As default version of "spark.sql.streaming.minBatchesToRetain" is set to high (100), which doesn't require strictly 100x of memory, but I'm seeing 10x ~ 80x of memory consumption for various workloads. In addition, in some cases, requiring 2x of memory is even unacceptable, so we should split out configuration for memory and let users adjust to trade-off between memory usage vs cache miss (building state from files). In normal case, default value '2' would cover both cases: success and restoring failure with less than or around 2x of memory usage, and '1' would only cover success case but no longer require more than 1x of memory. In extreme case, user can set the value to '0' to completely disable the map cache to maximize executor memory (covers #21500). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
