Repository: spark Updated Branches: refs/heads/master 2fbbcd0d2 -> c77aa42f5
[SPARK-25757][BUILD] Upgrade netty-all from 4.1.17.Final to 4.1.30.Final ## What changes were proposed in this pull request? Upgrade netty dependency from 4.1.17 to 4.1.30. Explanation: Currently when sending a ChunkedByteBuffer with more than 16 chunks over the network will trigger a "merge" of all the blocks into one big transient array that is then sent over the network. This is problematic as the total memory for all chunks can be high (2GB) and this would then trigger an allocation of 2GB to merge everything, which will create OOM errors. And we can avoid this issue by upgrade the netty. https://github.com/netty/netty/pull/8038 ## How was this patch tested? Manual tests in some spark jobs. Closes #22765 from lipzhu/SPARK-25757. Authored-by: Zhu, Lipeng <lip...@ebay.com> Signed-off-by: Dongjoon Hyun <dongj...@apache.org> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c77aa42f Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c77aa42f Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c77aa42f Branch: refs/heads/master Commit: c77aa42f55fe2c92bb9fcf93c92af0a768edfb68 Parents: 2fbbcd0 Author: Zhu, Lipeng <lip...@ebay.com> Authored: Sat Oct 20 22:17:37 2018 -0700 Committer: Dongjoon Hyun <dongj...@apache.org> Committed: Sat Oct 20 22:17:37 2018 -0700 ---------------------------------------------------------------------- dev/deps/spark-deps-hadoop-2.7 | 2 +- dev/deps/spark-deps-hadoop-3.1 | 2 +- pom.xml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/c77aa42f/dev/deps/spark-deps-hadoop-2.7 ---------------------------------------------------------------------- diff --git a/dev/deps/spark-deps-hadoop-2.7 b/dev/deps/spark-deps-hadoop-2.7 index 3b17f88..06173f7 100644 --- a/dev/deps/spark-deps-hadoop-2.7 +++ b/dev/deps/spark-deps-hadoop-2.7 @@ -149,7 +149,7 @@ metrics-json-3.1.5.jar metrics-jvm-3.1.5.jar minlog-1.3.0.jar netty-3.9.9.Final.jar -netty-all-4.1.17.Final.jar +netty-all-4.1.30.Final.jar objenesis-2.5.1.jar okhttp-3.8.1.jar okio-1.13.0.jar http://git-wip-us.apache.org/repos/asf/spark/blob/c77aa42f/dev/deps/spark-deps-hadoop-3.1 ---------------------------------------------------------------------- diff --git a/dev/deps/spark-deps-hadoop-3.1 b/dev/deps/spark-deps-hadoop-3.1 index c818b2c..62fddf0 100644 --- a/dev/deps/spark-deps-hadoop-3.1 +++ b/dev/deps/spark-deps-hadoop-3.1 @@ -165,7 +165,7 @@ metrics-jvm-3.1.5.jar minlog-1.3.0.jar mssql-jdbc-6.2.1.jre7.jar netty-3.9.9.Final.jar -netty-all-4.1.17.Final.jar +netty-all-4.1.30.Final.jar nimbus-jose-jwt-4.41.1.jar objenesis-2.5.1.jar okhttp-2.7.5.jar http://git-wip-us.apache.org/repos/asf/spark/blob/c77aa42f/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index 58a2841..b1f0a53 100644 --- a/pom.xml +++ b/pom.xml @@ -581,7 +581,7 @@ <dependency> <groupId>io.netty</groupId> <artifactId>netty-all</artifactId> - <version>4.1.17.Final</version> + <version>4.1.30.Final</version> </dependency> <dependency> <groupId>io.netty</groupId> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
