[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23166 thanks @redsanket @squito, committed master, 2.4, and 2.3.2 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99401/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23166 **[Test build #99401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99401/testReport)** for PR 23166 at commit [`001dfff`](https://github.com/apache/spark/commit/001dfff1b444d3f90febe1487eac4a1411a582de). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99399/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23166 **[Test build #99399 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99399/testReport)** for PR 23166 at commit [`d9994b7`](https://github.com/apache/spark/commit/d9994b7f9b6aaaf9f87ff09b1d45f3c204f7b4d3). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99394/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23166 **[Test build #99394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99394/testReport)** for PR 23166 at commit [`605ed93`](https://github.com/apache/spark/commit/605ed934fa109f7473a19d23af49e345f19392c8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/23166 cc @HyukjinKwon @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5475/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23166 **[Test build #99401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99401/testReport)** for PR 23166 at commit [`001dfff`](https://github.com/apache/spark/commit/001dfff1b444d3f90febe1487eac4a1411a582de). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23166 **[Test build #99399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99399/testReport)** for PR 23166 at commit [`d9994b7`](https://github.com/apache/spark/commit/d9994b7f9b6aaaf9f87ff09b1d45f3c204f7b4d3). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5472/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user redsanket commented on the issue: https://github.com/apache/spark/pull/23166 @squito yeah looking at the code we could maintain a soft reference so that we dont have to keep requesting it from the disk. It is definitely an optimization --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23166 **[Test build #99394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99394/testReport)** for PR 23166 at commit [`605ed93`](https://github.com/apache/spark/commit/605ed934fa109f7473a19d23af49e345f19392c8). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5468/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user squito commented on the issue: https://github.com/apache/spark/pull/23166 > The only question I have is does this cause more memory usage on the driver because it has a reference to that broadcast value or is something else already holding on to it? yeah good point. I guess that would make you hold onto a reference to the original value, even if it was never used on the driver. the proposal here is more like what is done in TorrentBroadcast.java -- its written to the driver's block manager, and read back if you request the value. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23166 Yeah certainly seems like a good idea. The only question I have is does this cause more memory usage on the driver because it has a reference to that broadcast value or is something else already holding on to it? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user squito commented on the issue: https://github.com/apache/spark/pull/23166 maybe a dumb question -- couldn't you just add ```python self._value = value ``` [inside `__init__()` when on the driver](https://github.com/apache/spark/blob/master/python/pyspark/broadcast.py#L78)? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5462/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99387/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23166 **[Test build #99387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99387/testReport)** for PR 23166 at commit [`67a2ac8`](https://github.com/apache/spark/commit/67a2ac87fb6e2d3fd4a5f260047a37bd2858228d). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23166 **[Test build #99387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99387/testReport)** for PR 23166 at commit [`67a2ac8`](https://github.com/apache/spark/commit/67a2ac87fb6e2d3fd4a5f260047a37bd2858228d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23166 cc @squito --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23166 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23166 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
