[
https://issues.apache.org/jira/browse/SPARK-41318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17640498#comment-17640498
]
Francesco Nigro commented on SPARK-41318:
-----------------------------------------
FYI [~yangjie01] [~hyukjin.kwon]
> Validate scalability improvements due to JDK-8180450 on Netty 4.1.84.Final
> --------------------------------------------------------------------------
>
> Key: SPARK-41318
> URL: https://issues.apache.org/jira/browse/SPARK-41318
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.4.0
> Reporter: Francesco Nigro
> Priority: Major
> Labels: jdk, netty, performance, scalability
>
> [https://netty.io/news/2022/10/11/4-1-84-Final.html] has introduced a
> bug-fix/improvement to prevent a JDK scalability issue due to type check
> against interfaces (see [https://github.com/netty/netty/pull/12806] for more
> details - and
> [https://redhatperf.github.io/post/type-check-scalability-issue/] for a
> deeper and wider analysis).
> it seems that Spark network make uses of Netty channel duplex handlers on
> [TransportContext|https://github.com/apache/spark/blob/94de3ca2942bb04852510abccf06df1fa8b2dab3/common/network-common/src/main/java/org/apache/spark/network/TransportContext.java#L201]
> and it's likely to be affected by it (or better, depending which netty
> version).
> Moreover, I suggest to verify if there are other places where the issue can
> happen and eventually, report them on
> https://bugs.openjdk.org/browse/JDK-8180450 as well (if they use type checks
> in an idiomatic form).
> The blog post has linked https://github.com/RedHatPerf/type-pollution-agent
> to detect dangerous candidate code paths.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
