Re: Apache Arrow integration issue with Spark involving Netty
Update! Netty has reverted the affecting change in v4.1.96. See netty commit here[1] and arrow PR to upgrade here[2]. The upcoming release of arrow-memory-netty v13 should work with netty versions <4.1.94 and >=4.1.96. [1] https://github.com/netty/netty/commit/dc16c5818a5cd0711f17e0a966783cdc84c9db01 [2] https://github.com/apache/arrow/pull/36926 On Thu, Jul 13, 2023 at 11:47 AM Dane Pitkin wrote: > I just want to add that there is a Spark Jira issue[1] for upgrading Netty > once Arrow v13.0.0 is released this month. > > [1] https://issues.apache.org/jira/projects/SPARK/issues/SPARK-44212 > > On Thu, Jul 6, 2023 at 2:25 PM Dane Pitkin wrote: > >> Hi all, >> >> The next release of Apache Arrow v13.0.0 coming this month[1] has >> upgraded Netty to v4.1.94.Final[2] due to a moderate severity CVE[3]. We >> are seeing that Spark using Netty v4.1.93.Final is not compatible with >> Arrow v13.0.0, throwing an exception at runtime[4]. There has been some >> talk in a Spark PR about upgrading to Netty v4.1.94.Final once the new >> arrow-memory-netty is released[5]. >> >> Should the Spark POM be updated to shade arrow-memory-netty? >> >> Thanks, >> Dane >> >> [1] https://lists.apache.org/thread/f9r0dsd65ohdtcvc7fnnlfs23n3z0n7f >> [2] https://github.com/apache/arrow/pull/36211 >> [3] https://github.com/advisories/GHSA-6mjq-h674-j845 >> [4] https://github.com/apache/arrow/issues/36332 >> [5] https://github.com/apache/spark/pull/41681 >> >>
Re: Apache Arrow integration issue with Spark involving Netty
I just want to add that there is a Spark Jira issue[1] for upgrading Netty once Arrow v13.0.0 is released this month. [1] https://issues.apache.org/jira/projects/SPARK/issues/SPARK-44212 On Thu, Jul 6, 2023 at 2:25 PM Dane Pitkin wrote: > Hi all, > > The next release of Apache Arrow v13.0.0 coming this month[1] has upgraded > Netty to v4.1.94.Final[2] due to a moderate severity CVE[3]. We are seeing > that Spark using Netty v4.1.93.Final is not compatible with Arrow > v13.0.0, throwing an exception at runtime[4]. There has been some talk in a > Spark PR about upgrading to Netty v4.1.94.Final once the new > arrow-memory-netty is released[5]. > > Should the Spark POM be updated to shade arrow-memory-netty? > > Thanks, > Dane > > [1] https://lists.apache.org/thread/f9r0dsd65ohdtcvc7fnnlfs23n3z0n7f > [2] https://github.com/apache/arrow/pull/36211 > [3] https://github.com/advisories/GHSA-6mjq-h674-j845 > [4] https://github.com/apache/arrow/issues/36332 > [5] https://github.com/apache/spark/pull/41681 > >
Apache Arrow integration issue with Spark involving Netty
Hi all, The next release of Apache Arrow v13.0.0 coming this month[1] has upgraded Netty to v4.1.94.Final[2] due to a moderate severity CVE[3]. We are seeing that Spark using Netty v4.1.93.Final is not compatible with Arrow v13.0.0, throwing an exception at runtime[4]. There has been some talk in a Spark PR about upgrading to Netty v4.1.94.Final once the new arrow-memory-netty is released[5]. Should the Spark POM be updated to shade arrow-memory-netty? Thanks, Dane [1] https://lists.apache.org/thread/f9r0dsd65ohdtcvc7fnnlfs23n3z0n7f [2] https://github.com/apache/arrow/pull/36211 [3] https://github.com/advisories/GHSA-6mjq-h674-j845 [4] https://github.com/apache/arrow/issues/36332 [5] https://github.com/apache/spark/pull/41681