[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19940 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85273/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #85273 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85273/testReport)** for PR 19940 at commit [`7397cf3`](https://github.com/apache/spark/commit/7397cf3957add3e0ce335f904f4a20b62060b566). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #85273 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85273/testReport)** for PR 19940 at commit [`7397cf3`](https://github.com/apache/spark/commit/7397cf3957add3e0ce335f904f4a20b62060b566). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19940 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85261/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #85261 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85261/testReport)** for PR 19940 at commit [`6143b22`](https://github.com/apache/spark/commit/6143b221ffe6387231dbdacc9a4aebb997f8e815). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #85261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85261/testReport)** for PR 19940 at commit [`6143b22`](https://github.com/apache/spark/commit/6143b221ffe6387231dbdacc9a4aebb997f8e815). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19940 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85252/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #85252 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85252/testReport)** for PR 19940 at commit [`6143b22`](https://github.com/apache/spark/commit/6143b221ffe6387231dbdacc9a4aebb997f8e815). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19940 the test errors are unrelated to this change. Any other comments @cloud-fan @kiszk @viirya ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #85252 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85252/testReport)** for PR 19940 at commit [`6143b22`](https://github.com/apache/spark/commit/6143b221ffe6387231dbdacc9a4aebb997f8e815). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19940 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85212/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #85212 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85212/testReport)** for PR 19940 at commit [`6143b22`](https://github.com/apache/spark/commit/6143b221ffe6387231dbdacc9a4aebb997f8e815). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #85212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85212/testReport)** for PR 19940 at commit [`6143b22`](https://github.com/apache/spark/commit/6143b221ffe6387231dbdacc9a4aebb997f8e815). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85202/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #85202 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85202/testReport)** for PR 19940 at commit [`c650196`](https://github.com/apache/spark/commit/c650196a4c0c4391d3112678803bd601af9aa5fb). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #85202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85202/testReport)** for PR 19940 at commit [`c650196`](https://github.com/apache/spark/commit/c650196a4c0c4391d3112678803bd601af9aa5fb). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84972/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #84972 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84972/testReport)** for PR 19940 at commit [`e2725d2`](https://github.com/apache/spark/commit/e2725d2ac8c7c66fb99332414b25f549de42b40f). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #84972 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84972/testReport)** for PR 19940 at commit [`e2725d2`](https://github.com/apache/spark/commit/e2725d2ac8c7c66fb99332414b25f549de42b40f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19940 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19940 the test failure is not related to the PR. It looks the same R failure we had some days ago, I thought it was solved: ``` checking CRAN incoming feasibility ...Error in .check_package_CRAN_incoming(pkgdir) : dims [product 22] do not match the length of object [0] ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84956/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #84956 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84956/testReport)** for PR 19940 at commit [`e2725d2`](https://github.com/apache/spark/commit/e2725d2ac8c7c66fb99332414b25f549de42b40f). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #84956 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84956/testReport)** for PR 19940 at commit [`e2725d2`](https://github.com/apache/spark/commit/e2725d2ac8c7c66fb99332414b25f549de42b40f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19940 @cloud-fan @kiszk @viirya any more comments on this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84846/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #84846 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84846/testReport)** for PR 19940 at commit [`90f517c`](https://github.com/apache/spark/commit/90f517c38364eda0e905ea0f23b413806307ec10). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #84846 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84846/testReport)** for PR 19940 at commit [`90f517c`](https://github.com/apache/spark/commit/90f517c38364eda0e905ea0f23b413806307ec10). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19940 I don't think so, it would be very risky, since `splitExpressions` might put the initialization outside the constructor. What about 1? Any thought? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84775/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #84775 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84775/testReport)** for PR 19940 at commit [`ce76015`](https://github.com/apache/spark/commit/ce76015c9ae2f5a0d04941af6bcd8e6b99168c00). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19940 For 2., I noticed there are two types of initialization. One is in `init` method by using `addPartitionInitializationStatement`. The other is in constructor that includes an initialization statement in `addSingleMutableState`. Can we declare `final` for latter cases? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19940 1. Yes, to ensure that there are no problems, variables' value should not be changed. I agree that we can find a better API name, to enforce this: what about `addImmutableStateIfNotExists`? Or do you have any suggestion? 2. Beacause as @viirya pointed out [here](https://github.com/apache/spark/pull/19940#issuecomment-350724795), they are not initialized in the declaration, but in the `init` method of the generated class, thus it cannot be made final unfortunately. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19940 Got it. It seems to be . I have two questions. 1. Is this mutable? It looks it is global, but immutable. IIUC, is there any other better API name? 2. As @viirya pointed out [here](https://github.com/apache/spark/pull/19940#issuecomment-350724536), why don't we declare `final`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #84775 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84775/testReport)** for PR 19940 at commit [`ce76015`](https://github.com/apache/spark/commit/ce76015c9ae2f5a0d04941af6bcd8e6b99168c00). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19940 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19940 @kiszk for instance it can remove one entry for every timestamp function (to_timestamp or from_utc_timestamp). Of course #19811 is the most important PR, because it solves the problem. But I think we all agree that if we can avoid to waste global variables it is better and there have been many PRs to avoid the usage of global variables. This is one of the many. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19940 I have one question. We are implementing #19811 to compact mutable states. Even when it will be merged, does this PR can reduce large number of constant pool entries? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84714/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #84714 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84714/testReport)** for PR 19940 at commit [`ce76015`](https://github.com/apache/spark/commit/ce76015c9ae2f5a0d04941af6bcd8e6b99168c00). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #84714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84714/testReport)** for PR 19940 at commit [`ce76015`](https://github.com/apache/spark/commit/ce76015c9ae2f5a0d04941af6bcd8e6b99168c00). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19940 Oh, the initialization is not right away in declaration. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19940 Shall we make them as `final` variables to guarantee this? I think this is an important requirement to prevent something wrong when wrongly using the shared global variables. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19940 @viirya this is the requirement I followed in this change which ensures that is it safe to share the variable across all the operators, since all the access are read only and there cannot be influences. Maybe this might be relaxed in the future, but if we follow this requirement, we are sure that this is safe. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19940 @mgaido91 Do you mean the shared global variables are required to be only assigned once (initialization) and never changed again? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19940 @viirya we have seen that using arrays affects performance. Thus if we can reduce their usage it is better. I don't think that debugging is harder. These variables I made shared are never assigned, but in the initialization. Do you have an other opinion? Or are you thinking for something specific? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19940 A high level question is, do we need to share mutable status, if we can compact global variables into array later? Will sharing mutable status increase the difficulty of debugging codegen in the future? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19940 @cloud-fan @kiszk @viirya might you please help reviewing this? Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19940 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84695/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #84695 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84695/testReport)** for PR 19940 at commit [`978bfd6`](https://github.com/apache/spark/commit/978bfd6a490cfef2c0f2da1190830ef6a64287af). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19940 **[Test build #84695 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84695/testReport)** for PR 19940 at commit [`978bfd6`](https://github.com/apache/spark/commit/978bfd6a490cfef2c0f2da1190830ef6a64287af). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
