Hi Kartikey,
I like the idea and I agree with general direction, thank you for
putting it together!
I have one concern about making this modification "forced", imho there
should be a room for "guaranteed important events delivery" from the
operations point of view. If Flink job is struggling/backpressured it
may make sense to emit some events at priority that would be used for
external triggers like "autoscaling" or external dynamic configuration
tuning.
Imho, interfaces should either allow to choose "sync" vs "non guaranteed
async" delivery for different events (or event reporters). With proposal
"as is" it won't be possible to "ensure" that important messages have
been delivered and can be actioned by external monitoring system. Could
we make "queue / async" behaviour opt-in?
Second question I had was around FileEventReporter implementation, at a
glance, "append to file" is a fairly cheap operation, do you have a
concern that amount of events is large enough to have significant
bottleneck on disk IO and requires memory queue?
Kind regards,
Aleksandr Iushmanov
On 2025/08/19 06:56:36 Kartikey Pant wrote:
> Hi everyone,
>
> I'd like to propose a new FLIP that builds directly on the excellent
> foundation laid by FLIP-481 (Introduce Event Reporting). For anyone
> needing context, the original proposal is available here:
>
https://cwiki.apache.org/confluence/display/FLINK/FLIP-481%3A+Introduce+Event+Reporting
>
> Now that the community has this powerful API, the logical next step is
> to ensure it's fully robust for large-scale production environments
> where users will be writing their own diverse, custom reporters.
>
> This proposal focuses on one key enhancement: introducing a resilient,
> asynchronous dispatch core. The goal is to decouple event generation
> from the reporter's execution, ensuring that a slow or experimental
> sink can never impact Flink's core stability.
>
> I've drafted a detailed design document that I hope can form the basis
> of this new FLIP:
>
https://docs.google.com/document/d/1CCu7Js0ATOAgqRMS-kWj_0v0G_jt2r9IfMB2Oty7KJo/edit?usp=sharing
>
> I'm keen to get the community's initial feedback on this direction
> before moving forward with the formal process.
>
> Thanks,
> Kartikey Pant
>
