[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22024 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22024 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user vincent-grosbois commented on the issue: https://github.com/apache/spark/pull/22024 Hello ! is there anybody interested in merging this ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/22024 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user vincent-grosbois commented on the issue: https://github.com/apache/spark/pull/22024 Hi, sorry I haven't done any benchmark on this. But this solved an Out Of Memory issue for: - before this fix with Spark 2.3 we had OOM issues in specific jobs when the Partition size was big (1.9 GB) and made of few big objects, without using the spark.maxRemoteBlockSizeFetchToMem trick - with this fix our OOM dissapeared, without having to use spark.maxRemoteBlockSizeFetchToMem (ie everything is done on memory, no disk spill) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22024 @vincent-grosbois I don't look into this change yet. Do you have reply for @dbtsai's comment https://github.com/apache/spark/pull/22024#issuecomment-412659532? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22024 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user vincent-grosbois commented on the issue: https://github.com/apache/spark/pull/22024 Hi @viirya , any news on this? do you need me to do something? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22024 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22024 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94708/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22024 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22024 **[Test build #94708 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94708/testReport)** for PR 22024 at commit [`6b29a12`](https://github.com/apache/spark/commit/6b29a122189ffe1cf75f1b84f181db2abf7ee337). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22024 **[Test build #94708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94708/testReport)** for PR 22024 at commit [`6b29a12`](https://github.com/apache/spark/commit/6b29a122189ffe1cf75f1b84f181db2abf7ee337). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/22024 test this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22024 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94696/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22024 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22024 **[Test build #94696 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94696/testReport)** for PR 22024 at commit [`6b29a12`](https://github.com/apache/spark/commit/6b29a122189ffe1cf75f1b84f181db2abf7ee337). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/22024 @vincent-grosbois do you have some benchmark or profiling on this change? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22024 **[Test build #94696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94696/testReport)** for PR 22024 at commit [`6b29a12`](https://github.com/apache/spark/commit/6b29a122189ffe1cf75f1b84f181db2abf7ee337). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/22024 test this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user vincent-grosbois commented on the issue: https://github.com/apache/spark/pull/22024 I updated my branch to take into account the TorrentBroadcast issue, can anyone have a look? In general some parts of the source code are unclear, it seems that some cases are requiring the ChunkedByteBuffer instances to only have 1 component but I don't know why and I don't know if it was intended --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user vincent-grosbois commented on the issue: https://github.com/apache/spark/pull/22024 Actually I spoke too soon! We spotted some regression in TorrentBroadcast: https://github.com/apache/spark/blob/46110a589f4e91cd7605c5a2c34c3db6b2635830/core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala#L164 it seems that the code in charge of checking the CheckSum is assuming that we only receive 1 block in the chunkedByteBuffer (cf checksum of only chunk(0)). Does anyone know why ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22024 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22024 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22024 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
