I think @pjfanning means you configured the pekko version with the 1.2.0 
snapshot
https://repository.apache.org/content/groups/snapshots/org/apache/pekko/pekko-actor_2.13/1.2.0-M0+55-a75bc7a7-SNAPSHOT/

with `unpooled-heap` allocator type.

and change the failing test with 7MB to see if it succeeds, then he is OK with 
releasing it in 1.1.4

But I think option 2 will not cause many problems in real production, who will 
run a Flink with 7MB? Even my phone has 16GB RAM.

On 2025/01/20 15:06:09 Alexander Fedulov wrote:
> Hi Ferenc,
> 
> Impacting data exchange performance is definitely not an option. I'm simply
> not entirely sure if Flink's network stack delegates buffer allocation to
> Netty or handles it "manually."
> That said, even if we could confirm that this part is not impacted, setting
> the global configuration is still not a good idea. We already have the HDFS
> filesystem dependency inside the framework that depends on Netty 4, plus
> potentially multiple externalized connectors.
> 
> Let me summarize the options we have, knowing that the allocator control
> localized to Pekko is currently not available:
> 
> 1) Revert the changes
>     Pros:
>       - Unblocks the release
>     Cons:
>       - Leaves us with 20 unaddressed critical CVEs
> 
> 2) Upgrade to Netty 4 and let the RPC module run with Netty's default
> settings (also bump the limits for the memory-restricted test, as done on
> master):
>     Pros:
>       - Resolves CVEs
>       - Unblocks the release
>     Cons:
>       - Higher memory usage with the risk of tipping containers already at
> their memory limit into OOM
>       - Unclear if the fractional autoallocation of Flink memory needs to
> be adjusted according to the new defaults
> 
> 3) Help the colleagues working on Pekko to confirm that the patch for
> settings fulfills our purposes [1] and wait for a new Pekko release that
> allows isolated Netty settings solely for flink-rpc:
>     Pros:
>       - Resolves the CVEs
>       - Memory usage similar to Netty 3, reducing the risk of OOM surprises
> for users
>     Cons:
>       - Could take time for the new Pekko release
> 
> [1] https://github.com/apache/pekko/pull/1709#issuecomment-2599698240
> 
> Unless we can hope for a timely Pekko release, my vote would be to go ahead
> with option 2 and clearly document the potential need for increasing
> container memory limits for the sake of improved security.
> 
> Best regards,
> Alex
> 
> On Mon, 20 Jan 2025 at 14:11, Ferenc Csaky <ferenc.cs...@pm.me.invalid>
> wrote:
> 
> > Since `flink-runtime` uses Netty4 for quite a while, I believe
> > enforcing UNPOOLED will affect shuffle performance. I did not
> > performed actual tests comparing Netty3 and Netty4 in this regard,
> > so I cannot back this with actual numbers, but I think losing
> > shuffle performance would affect more real-world use-cases and be a
> > bigger problem, than a bit more overall memory consumption for
> > RPC communication.
> >
> > To cover highly resource-limited use-cases where it might be useful
> > to spare some memory and performance is not critical, I would
> > suggest to document these options in the release notes and in the
> > product documentation as well. I already created a ticket for
> > that [1], so I plan to deliver it in the next couple days.
> >
> > WDYT?
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-37099
> >
> >
> >
> > On Monday, January 20th, 2025 at 13:10, ConradJam <jam.gz...@gmail.com>
> > wrote:
> >
> > >
> > >
> > > +1
> > >
> > > Alexis Sarda-Espinosa sarda.espin...@gmail.com 于2025年1月20日周一 18:38写道:
> > >
> > > > Hello,
> > > >
> > > > what about io.netty.maxDirectMemory [1]? Is it relevant? I haven't been
> > > > able to understand exactly how much that changes, but I find it odd
> > that,
> > > > for the default, <"practical max direct memory" would be 2 * max
> > memory as
> > > > defined by the JDK>.
> > > >
> > > > [1]
> > > >
> > > >
> > https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java#L162
> > > >
> > > > Regards,
> > > > Alexis.
> > > >
> > > > Am Mo., 20. Jan. 2025 um 04:53 Uhr schrieb He Pin he...@apache.org:
> > > >
> > > > > I think so, not sure how Flink works, but if they share the same key
> > and
> > > > > running in the same JVM process, which can be.
> > > > >
> > > > > On 2025/01/18 16:58:15 Alexander Fedulov wrote:
> > > > >
> > > > > > @He Pin,
> > > > > > Thanks for bringing this up.
> > > > > > So, if I understand correctly, the problem is that there is
> > currently
> > > > > > no
> > > > > > way to control the underlying allocator exclusively for Pekko.
> > Setting
> > > > > > `-Dio.netty.allocator.type=unpooled` would impact Netty's behavior
> > > > > > across
> > > > > > other parts of the framework.
> > > > > >
> > > > > > Does anyone know if this could potentially affect the data exchange
> > > > > > network
> > > > > > stack in `flink-runtime`, which is also based on Netty?
> > > > > >
> > > > > > Best,
> > > > > > Alex
> > > > > >
> > > > > > On Sat, 18 Jan 2025 at 04:10, He Pin he...@apache.org wrote:
> > > > > >
> > > > > > > > +1 for Netty4 with UNPOOLED memory allocator to not change the
> > > > > > > > default
> > > > > > > > memory footprint.
> > > > > > >
> > > > > > > That can only be done with another release, otherwise if will
> > reduce
> > > > > > > the
> > > > > > > performance.
> > > > > > >
> > > > > > > see https://github.com/apache/pekko/pull/1709
> > > > > > >
> > > > > > > On 2025/01/17 17:05:06 Maximilian Michels wrote:
> > > > > > >
> > > > > > > > +1 for Netty4 with UNPOOLED memory allocator to not change the
> > > > > > > > default
> > > > > > > > memory footprint.
> > > > > > > >
> > > > > > > > -Max
> > > > > > > >
> > > > > > > > On Fri, Jan 17, 2025 at 1:15 PM Samrat Deb
> > decordea...@gmail.com
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 to move to netty4.
> > > > > > > > >
> > > > > > > > > bests,
> > > > > > > > > Samrat
> > > > > > > > >
> > > > > > > > > On Fri, 17 Jan 2025 at 5:30 PM, Luke Chen show...@gmail.com
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for the summary!
> > > > > > > > > >
> > > > > > > > > > +1 to upgrade Pekko to have netty 4 in 1.19.2 and 1.20.1
> > > > > > > > > > releases.
> > > > > > > > > >
> > > > > > > > > > Thanks.
> > > > > > > > > > Luke
> > > > > > > > > >
> > > > > > > > > > On Fri, Jan 17, 2025 at 7:50 PM He Pin he...@apache.org
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 to Netty 4
> > > > > > > > > > >
> > > > > > > > > > > On 2025/01/16 15:12:40 Alexander Fedulov wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > We have one remaining blocker for the 1.19.2 and 1.20.1
> > > > > > > > > > > > releases,
> > > > > > > > > > > > namely
> > > > > > > > > > > > the issue associated with ticket FLINK-36510: "Upgrade
> > > > > > > > > > > > Pekko
> > > > > > > > > > > > from
> > > > > > > > > > > > 1.0.1
> > > > > > > > > > > > to
> > > > > > > > > > > > 1.1.2" [1]. Here is the context:
> > > > > > > > > > > >
> > > > > > > > > > > > - The flink-rpc module is currently based on Pekko
> > > > > > > > > > > > 1.0.1,
> > > > > > > > > > > > which
> > > > > > > > > > > > bundles
> > > > > > > > > > > > Netty version 3.10.6. Netty 3.10.6 is the last 3.x
> > > > > > > > > > > > release and
> > > > > > > > > > > > officially
> > > > > > > > > > > > reached EOL more than eight years ago. It contains at
> > > > > > > > > > > > least
> > > > > > > > > > > > 20 known
> > > > > > > > > > > > critical vulnerabilities [2].
> > > > > > > > > > > > - FLINK-36510 [1] upgrades flink-rpc to Pekko 1.1.2,
> > > > > > > > > > > > which
> > > > > > > > > > > > introduces
> > > > > > > > > > > > a
> > > > > > > > > > > > long-awaited migration to Netty 4.x.
> > > > > > > > > > > > - Memory allocation in Netty 4.x differs from Netty 3.x
> > > > > > > > > > > > and
> > > > > > > > > > > > has a
> > > > > > > > > > > > larger
> > > > > > > > > > > > memory footprint with default settings [3].
> > > > > > > > > > > > - Norman Mauerer, Netty's project lead, strongly
> > > > > > > > > > > > recommends
> > > > > > > > > > > > moving
> > > > > > > > > > > > away
> > > > > > > > > > > > from Netty 3 as soon as possible [4].
> > > > > > > > > > > > - According to Norman, setting
> > > > > > > > > > > > -Dio.netty.allocator.type=unpooled
> > > > > > > > > > > > should
> > > > > > > > > > > > approximate Netty 3's memory behavior at the expense of
> > > > > > > > > > > > performance
> > > > > > > > > > > > improvements that Netty 4 would otherwise provide. That
> > > > > > > > > > > > said,
> > > > > > > > > > > > Netty
> > > > > > > > > > > > 4
> > > > > > > > > > > > with
> > > > > > > > > > > > -Dio.netty.allocator.type=unpooled is not expected to
> > > > > > > > > > > > perform
> > > > > > > > > > > > worse
> > > > > > > > > > > > than
> > > > > > > > > > > > Netty 3.
> > > > > > > > > > > > - Although this change might seem too substantial for a
> > > > > > > > > > > > patch
> > > > > > > > > > > > release, I
> > > > > > > > > > > > propose proceeding with it due to the accumulated risks
> > > > > > > > > > > > of
> > > > > > > > > > > > staying
> > > > > > > > > > > > on
> > > > > > > > > > > > Netty
> > > > > > > > > > > > 3.10.6. This will need to be addressed in a 1.20 as a
> > > > > > > > > > > > patch
> > > > > > > > > > > > release
> > > > > > > > > > > > anyway,
> > > > > > > > > > > > given that 1.20 is designated as LTS, and we can expect
> > > > > > > > > > > > Netty
> > > > > > > > > > > > 3 to
> > > > > > > > > > > > accrue
> > > > > > > > > > > > even more CVEs over time.
> > > > > > > > > > > >
> > > > > > > > > > > > Here you can find more details of the ongoing
> > discussion
> > > > > > > > > > > > [5].
> > > > > > > > > > > >
> > > > > > > > > > > > Looking forward to hearing the community's thoughts on
> > > > > > > > > > > > whether we
> > > > > > > > > > > > should
> > > > > > > > > > > > proceed with the proposed changes.
> > > > > > > > > > > >
> > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-36510
> > > > > > > > > > > > [2]
> > > > > > > > > > > >
> > https://mvnrepository.com/artifact/io.netty/netty/3.10.6.Final
> > > > > > > > > > > > [3]
> > > >
> > > >
> > https://issues.apache.org/jira/browse/FLINK-36510?focusedCommentId=17911219&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17911219
> > > >
> > > > > > > > > > > > [4]
> > > > > > > > > > > >
> > https://github.com/apache/flink/pull/25866#issuecomment-2595168560
> > > > > > > > > > > > [5] https://github.com/apache/flink/pull/25866
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Alex
> >
> 

Reply via email to