[GitHub] spark issue #21933: [SPARK-24917][CORE] make chunk size configurable

Github user vincent-grosbois commented on the issue: https://github.com/apache/spark/pull/21933 Hello, I updated the description and title --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #21933: [SPARK-24917][CORE] make chunk size configurable

Github user vincent-grosbois commented on a diff in the pull request: https://github.com/apache/spark/pull/21933#discussion_r207557109 --- Diff: core/src/main/scala/org/apache/spark/serializer/SerializerManager.scala --- @@ -70,6 +70,8 @@ private[spark] class SerializerManager

[GitHub] spark pull request #21933: [SPARK-24917] make chunk size configurable

GitHub user vincent-grosbois opened a pull request: https://github.com/apache/spark/pull/21933 [SPARK-24917] make chunk size configurable Add an option in Spark configuration to change the chunk size (which is by default 4 Mb). This would allow to bypass the issue

[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...

Github user vincent-grosbois commented on the issue: https://github.com/apache/spark/pull/22024 I updated my branch to take into account the TorrentBroadcast issue, can anyone have a look? In general some parts of the source code are unclear, it seems that some cases

[GitHub] spark pull request #22024: [SPARK-25034][CORE] Remove allocations in onBlock...

GitHub user vincent-grosbois opened a pull request: https://github.com/apache/spark/pull/22024 [SPARK-25034][CORE] Remove allocations in onBlockFetchSuccess This method is only transferring a ManagedBuffer to the caller, so there is no reason why it should allocate 2

[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...

Github user vincent-grosbois commented on the issue: https://github.com/apache/spark/pull/22024 Actually I spoke too soon! We spotted some regression in TorrentBroadcast: https://github.com/apache/spark/blob/46110a589f4e91cd7605c5a2c34c3db6b2635830/core/src/main/scala/org

[GitHub] spark issue #21933: [SPARK-24917][CORE] make chunk size configurable

Github user vincent-grosbois commented on the issue: https://github.com/apache/spark/pull/21933 Not really! This would be a useful features to backport in all spark branches that will never benefit from an upgrade to netty 4.1.28. However I think for spark 2.4 and the master branch

[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...

Github user vincent-grosbois commented on the issue: https://github.com/apache/spark/pull/22024 Hi, sorry I haven't done any benchmark on this. But this solved an Out Of Memory issue for: - before this fix with Spark 2.3 we had OOM issues in specific jobs when

[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...

Github user vincent-grosbois commented on the issue: https://github.com/apache/spark/pull/22024 Hi @viirya , any news on this? do you need me to do something? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22024: [SPARK-25034][CORE] Remove allocations in onBlock...

Github user vincent-grosbois commented on a diff in the pull request: https://github.com/apache/spark/pull/22024#discussion_r213037747 --- Diff: core/src/main/scala/org/apache/spark/network/BlockTransferService.scala --- @@ -101,15 +101,7 @@ abstract class BlockTransferService

[GitHub] spark issue #21473: [SPARK-21896][SQL] Fix StackOverflow caused by window fu...

Github user vincent-grosbois commented on the issue: https://github.com/apache/spark/pull/21473 Hello! Is there anyway this can be backported in 2.3? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSu...

Github user vincent-grosbois commented on the issue: https://github.com/apache/spark/pull/22024 Hello ! is there anybody interested in merging this ? --- - To unsubscribe, e-mail: reviews-unsubscr