Hi Martijn Thanks for the heads-up.
That sounds good to me. Since the proposal has been stable for a while, could you share a rough idea of when you expect to finish your final review? A tentative date would be enough, and I can hold off starting the vote until then. Best, Guowei On Thu, May 21, 2026 at 4:47 PM Martijn Visser <[email protected]> wrote: > Hi Guowei, > > I would like to have a couple of more days to take a closer final look > with a final round of comments, I haven't been able to finalize > everything yet on this subject. > > Best regards, > > Martijn > > Op wo 20 mei 2026 om 15:04 schreef Guowei Ma <[email protected]>: > > > > Hi all, > > > > Thanks everyone for the discussion and feedback on the FLIP. > > > > If there are no further comments or concerns, I plan to start the vote on > > Thursday. > > > > > > Best, > > Guowei > > > > > > On Fri, May 15, 2026 at 12:21 PM Jiangang Liu <[email protected] > > > > wrote: > > > > > I Greatly support this proposal. AI-oriented and multimodal data > processing > > > is already becoming a real production workload, and Flink is well > > > positioned to serve as the dataflow backbone for these pipelines. > > > > > > The proposal correctly focuses on engine-level capabilities rather than > > > turning Flink into an AI framework or model serving platform. In > > > particular, the shared runtime contract across multimodal types, object > > > references, inference invocation, GPU resource management, scaling, and > > > recovery is essential. Without such coordination, users would have to > > > stitch together fragmented abstractions themselves. I also appreciate > the > > > incremental and backward-compatible approach, as well as the > conservative > > > scope for correctness-sensitive runtime changes. > > > > > > This direction can help Flink extend its existing strengths in > streaming, > > > checkpointing, and production operations to a new class of AI-driven > > > workloads. > > > > > > Best > > > Jiangang Liu > > > > > > > > > Lincoln Lee <[email protected]> 于2026年5月11日周一 14:47写道： > > > > > > > Hi Guowei and all, > > > > > > > > Big +1 on the overall direction of FLIP-577. > > > > > > > > From the SQL ecosystem perspective, over the past year the community > > > > has introduced AI functions (e.g., ML_PREDICT, VECTOR_SEARCH) to help > > > > users leverage Flink in AI scenarios. Through community interactions > > > > and enterprise user feedback since then, we can strongly sense the > > > > growing expectation for Flink to go further — broader multimodal data > > > > processing capabilities, improved runtime stability under AI > > > > workloads, and more optimized execution for inference-heavy > pipelines. > > > > > > > > This FLIP addresses exactly those gaps. In particular, I'm excited > about: > > > > > > > > - Multimodal data type extensions — first-class types like > > > > Tensor, Image, Embedding, and Audio in the type system will unlock > > > > native SQL/Table expressions over multimodal data, rather than > > > > forcing users to work around opaque BYTES columns. > > > > - Object reference mechanism — a critical optimization for > > > > large-payload multimodal data. Passing multi-megabyte blobs through > > > > the standard data path is a known pain point today; a > > > > reference-based approach will significantly reduce serialization > > > > overhead and memory pressure. > > > > - Richer built-in AI functions — standardizing common multimodal > > > > operations (embedding generation, similarity search, model > > > > inference) as engine-level functions will eliminate the repeated > > > > ad-hoc UDF implementations we see across organizations today, and > > > > open the door for engine-level optimizations like shared model > > > > handles and adaptive batching. > > > > > > > > As Jark mentioned, we'd like to participate in and contribute to the > > > > relevant sub-FLIPs. > > > > > > > > > > > > Best, > > > > Lincoln Lee > > > > > > > > > > > > Guowei Ma <[email protected]> 于2026年5月10日周日 21:26写道： > > > > > > > > > Hi all, > > > > > > > > > > Thanks again for the thoughtful feedback and the valuable > perspectives > > > > > shared in this discussion. > > > > > > > > > > I have updated FLIP-577 [1] based on the discussion in this > thread. The > > > > > overall direction remains the same, but I have tried to make the > scope, > > > > > motivation, and boundaries clearer. > > > > > > > > > > The main changes are: > > > > > > > > > > 1. > > > > > > > > > > Clarified the target workload as AI-oriented data processing > > > > workloads, > > > > > instead of relying only on the broader "AI-Native" wording. > > > > > 2. > > > > > > > > > > Added explicit non-goals to make clear that this proposal does > not > > > aim > > > > > to turn Flink into an AI framework, ML platform, or model > serving > > > > > system. > > > > > 3. > > > > > > > > > > Added "Why Now" and "Why Flink" sections to better explain the > > > > > production signals, ecosystem trends, and why Flink's existing > > > runtime > > > > > strengths are relevant here. > > > > > 4. > > > > > > > > > > Reworked the umbrella rationale. The key point is not that every > > > > > sub-FLIP is AI-specific, but that these mechanisms need a shared > > > > runtime > > > > > contract across data representation, service invocation, GPU > > > > resources, > > > > > scaling, and recovery. > > > > > 5. > > > > > > > > > > Clarified that engine-level primitives should consider > SQL/Table, > > > Java > > > > > DataStream, and Python DataFrame. > > > > > 6. > > > > > > > > > > Made the initial correctness scope of runtime mechanisms — > > > > > non-disruptive scaling, UAC enhancements, and Pipeline Region > > > > > checkpoints — > > > > > more conservative, with explicit opt-in where default behavior > is > > > > > affected. > > > > > > > > > > I also tried to reflect the earlier questions raised in this > thread in > > > > the > > > > > corresponding sections of the updated document. > > > > > > > > > > Looking forward to the continued discussion. > > > > > > > > > > [1] > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=421957275 > > > > > > > > > > Best, > > > > > Guowei > > > > > > > > > > > > > > > On Sun, May 10, 2026 at 4:19 PM Xintong Song < > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > First, big thanks to Guowei for kicking off this important > > > discussion, > > > > > and > > > > > > to the community for the substantive engagement over the past > days. > > > I'd > > > > > > also like to share some of my own thoughts. > > > > > > > > > > > > My apologies for joining late — I've been tied up with other > things > > > and > > > > > > only recently started catching up. Given the length of this > thread, > > > > it's > > > > > > possible I haven't fully digested every detail. If I miss > anything, > > > > > please > > > > > > bear with me and feel free to point it out. > > > > > > > > > > > > Let me state my position upfront: a big +1 to this FLIP. Based > on the > > > > > > workload evolution I've been observing, I believe moving Flink > toward > > > > > > AI-Native is necessary — this is a project- and community-level > > > effort > > > > > that > > > > > > matters deeply for how Flink responds to the industry shift > driven by > > > > AI, > > > > > > and how it captures the new opportunities that come with it. We > may > > > > still > > > > > > need more discussion to align on specific technical details, but > at > > > the > > > > > > strategic level of where the community should head, I strongly > > > support > > > > > this > > > > > > umbrella. > > > > > > > > > > > > Below are my thoughts on a few key questions raised in the > discussion > > > > so > > > > > > far. > > > > > > > > > > > > > > > > > > 1. On the scope of the umbrella > > > > > > > > > > > > Robert, Yaroslav, and Martijn have all pointed out that some > items in > > > > the > > > > > > FLIP — Pipeline Region independent checkpointing, UAC > enhancements, > > > > > > non-disruptive scaling, etc. — are not unique to AI scenarios and > > > would > > > > > > also apply elsewhere. That observation is fair. > > > > > > > > > > > > However, I'd argue the right criterion for whether something > belongs > > > > > under > > > > > > this umbrella is not whether it is *exclusively* used for AI, but > > > > whether > > > > > > it is *essential* to the central goal here — AI-Native Flink — > i.e., > > > > > > whether AI-Native Flink would actually fall short without it. > > > > > > > > > > > > From that angle, I agree with what Guowei has already laid out: > under > > > > AI > > > > > > workload characteristics (per-record processing cost orders of > > > > magnitude > > > > > > higher, GPU resources expensive and scarce, long and long-tailed > > > > > > asynchronous inference), these otherwise-general mechanisms get > > > pushed > > > > to > > > > > > the limit of their current implementations. They may be adequate > for > > > > > > traditional BI/ETL workloads, but they become real bottlenecks > under > > > AI > > > > > > workloads. Without them, the AI-Native Flink story would be > > > incomplete. > > > > > So > > > > > > I support including them under this umbrella. > > > > > > > > > > > > > > > > > > 2. On usage and maintenance complexity > > > > > > > > > > > > I understand the concerns Yaroslav and Martijn raised about > > > complexity > > > > > > costs. That said, I don't think we should conservatively reject a > > > > > > capability because of complexity, if that capability is critical > and > > > > > brings > > > > > > substantial benefits. On the necessity of RpcOperator > specifically, > > > Zhu > > > > > > Zhu, Guowei, and others have already provided answers earlier in > this > > > > > > thread, and I'll add some context from the Flink Agents side > below. > > > > > > > > > > > > As a side observation: with the rapid progress of AI-assisted > > > > development > > > > > > over the past year or two, the way engineers work is shifting in > a > > > > > > noticeable way. On one hand, AI coding tools are helping > developers > > > > > > maintain complex software systems at lower effort. On the other > hand, > > > > > more > > > > > > and more users are starting to ask AI how to use Flink > correctly, and > > > > > even > > > > > > letting AI help them develop and operate Flink workloads. Usage > and > > > > > > maintenance complexity still matter, but their relative weight is > > > > > arguably > > > > > > going down — this is a trend worth factoring into long-term > tradeoffs > > > > at > > > > > > the community level. > > > > > > > > > > > > > > > > > > 3. On "wait for the industry to stabilize before integrating" > > > > > > > > > > > > Yaroslav raised a concern about the timing — the LLM tooling > > > landscape > > > > is > > > > > > changing fast, and it's hard to predict what will be needed > tomorrow. > > > > I'd > > > > > > offer a different perspective. > > > > > > > > > > > > It's *precisely because* the industry is evolving fast and > hasn't yet > > > > > > settled into a stable shape that Flink should join earlier — to > > > > influence > > > > > > and help define how that shape forms. If we just stand by > waiting for > > > > the > > > > > > landscape to settle, by the time it does, there may not be a > place > > > for > > > > > > Flink in it, and we'd have missed an important window. > > > > > > > > > > > > In fact, projects like Daft and Ray Data are increasingly > defining > > > the > > > > de > > > > > > facto standards in this space, as Leonard pointed out earlier in > this > > > > > > thread. If Flink doesn't engage proactively, even keeping up > later > > > will > > > > > put > > > > > > us in a reactive position. > > > > > > > > > > > > Engaging early does carry some cost of trial and error — some > > > > > capabilities > > > > > > may need to be deprecated or restructured down the line. But I'd > > > argue > > > > > this > > > > > > is a normal part of how open-source projects evolve, and it's > both > > > > > > necessary and worth it. > > > > > > > > > > > > > > > > > > 4. RpcOperator from a Flink Agents perspective > > > > > > > > > > > > The benefits of RpcOperator have already been articulated by > several > > > > > people > > > > > > earlier in the thread. I'd like to add a concrete scenario from > the > > > > Flink > > > > > > Agents side — the subagent pattern. > > > > > > > > > > > > > > > > > > 4.1 Brief intro to the subagent pattern > > > > > > > > > > > > In agent practice, after planning, the main agent often spawns > one or > > > > > more > > > > > > subagents to handle specific subtasks. Each subagent has its own > LLM > > > > > > context — its own system prompt, its own context window, its own > > > > toolset > > > > > > and permissions. The main agent only receives the final result > from > > > the > > > > > > subagent, not the intermediate steps. This pattern is now widely > > > > adopted > > > > > in > > > > > > the industry, and many agent skills explicitly say "spawn a > subagent > > > to > > > > > do > > > > > > X". > > > > > > > > > > > > Flink Agents wants to support this pattern for several reasons: > > > > > > compatibility with the existing agent skill ecosystem; context > > > > isolation, > > > > > > so that intermediate state during subagent execution doesn't > pollute > > > > the > > > > > > main agent's context; and one additional advantage that > enterprise > > > > > > production scenarios have over single-machine personal-assistant > > > > agents — > > > > > > the ability to dispatch resource-intensive subtasks to a shared > > > > resource > > > > > > pool for efficient resource reuse. > > > > > > > > > > > > > > > > > > 4.2 Why this needs RpcOperator instead of in-place execution > inside > > > the > > > > > > Flink Agents operator > > > > > > > > > > > > Why not just spawn subagents directly inside the Flink Agents > > > operator? > > > > > > Because subagent workloads are highly bursty — within the same > job, > > > one > > > > > > minute's input might need several subagents for deep processing; > the > > > > next > > > > > > minute's input might need none. Under that kind of randomness, > > > > statically > > > > > > allocating subagent resources per operator parallelism leaves no > good > > > > > > answer: under-allocate and the operator gets blown out the moment > > > > demand > > > > > > arrives; over-allocate and resources sit idle most of the time. > > > > > > > > > > > > The natural answer is to put subagent execution into a shared > > > resource > > > > > pool > > > > > > that can be used by multiple upstream operator subtasks, and > that can > > > > > scale > > > > > > elastically based on load. This maps directly onto the design and > > > core > > > > > > benefits of RpcOperator that Zhu Zhu laid out earlier — each > > > > RpcOperator > > > > > > instance forming its own Pipelined Region, which in turn enables > > > > > > independent scaling and flexible load balancing. > > > > > > > > > > > > > > > > > > 4.3 Why this shared resource pool should be inside Flink rather > than > > > > > > deployed externally > > > > > > > > > > > > Two reasons. > > > > > > > > > > > > First, the prompts, toolsets, and permission configurations a > > > subagent > > > > > > needs at execution time are essentially part of the Flink Agents > *job > > > > > > definition*. Stripping them out of the job and deploying them > > > > separately > > > > > > effectively splits the job definition in two, leaving users to > > > > maintain a > > > > > > synchronization relationship between an external deployment and > the > > > > Flink > > > > > > job. This goes against the developer experience Flink Agents > aims to > > > > > > provide. > > > > > > > > > > > > Second, subagent execution needs to be incorporated into Flink's > > > > > checkpoint > > > > > > mechanism. The notion of "stateful" here deserves a brief > > > > clarification: > > > > > > each subagent task is self-contained — the full context it needs > is > > > > > handed > > > > > > over by the main agent at dispatch time in one shot. That is, > > > subagents > > > > > > don't need to share state across records, so at the task level > they > > > > look > > > > > > stateless. However, executing a single subagent task takes a > > > > significant > > > > > > amount of time and involves multiple rounds of model calls and > tool > > > > > calls, > > > > > > where tool calls may have externally observable side effects > (sending > > > > > > emails, writing to external systems, etc.). This means we need > > > failure > > > > > > recovery for the in-flight computation state during task > execution, > > > so > > > > > that > > > > > > subagent execution preserves exactly-once semantics across > failovers. > > > > > > > > > > > > > > > > > > To wrap up, let me reiterate my support for the overall > direction of > > > > this > > > > > > FLIP. There are clearly technical details still to be worked out, > > > but I > > > > > > believe the direction described by this umbrella is both > necessary > > > and > > > > > > important for the long-term evolution of the Flink community. > Looking > > > > > > forward to continued discussion in the sub-FLIPs ahead. > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Best, > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 8, 2026 at 3:43 PM Guowei Ma <[email protected]> > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > Thanks everyone for the in-depth discussion over the past few > days. > > > > Let > > > > > > me > > > > > > > first summarize my understanding of the discussion so far. I > see > > > > strong > > > > > > > interest in making Flink better support AI-oriented data > processing > > > > > > > workloads, especially multimodal and inference-oriented > pipelines. > > > At > > > > > the > > > > > > > same time, a recurring concern is that RpcOperator and some of > the > > > > > Layer > > > > > > 3 > > > > > > > runtime improvements also look like general Flink > capabilities, so > > > > why > > > > > > > should they be discussed together under the AI-Native / > multimodal > > > > > > > processing umbrella? > > > > > > > > > > > > > > I think the key question is not whether these mechanisms can > only > > > be > > > > > used > > > > > > > for AI scenarios, but whether they jointly form the runtime > > > contract > > > > > > Flink > > > > > > > needs to support AI-oriented data processing workloads, > especially > > > > > > > multimodal inference pipelines. Capabilities such as > RpcOperator, > > > > > > > checkpointing, scaling, and resource management can certainly > serve > > > > > > broader > > > > > > > use cases. However, under this class of workloads, they require > > > more > > > > > > > consistent runtime semantics and coordinated design, and > together > > > > > > determine > > > > > > > whether the system can provide production-grade execution, > > > recovery, > > > > > and > > > > > > > elasticity. > > > > > > > > > > > > > > Compared with traditional streaming workloads, this class of > > > > workloads > > > > > > > changes the data shape, computation pattern, and resource model > > > > > > > significantly. Data is no longer only small structured > records; it > > > > may > > > > > be > > > > > > > images, video, audio, tensors, embeddings, or object > references to > > > > > > external > > > > > > > large objects. Many pipelines also have a relatively > shuffle-light > > > > > shape, > > > > > > > such as URI → preprocessing → inference → sink. The computation > > > logic > > > > > > often > > > > > > > includes model inference or service-style invocation, either as > > > > remote > > > > > > > inference service invocation or local GPU / accelerator-backed > > > > > execution. > > > > > > > Therefore, the system needs to handle not only ordinary RPC > calls, > > > > but > > > > > > also > > > > > > > service discovery, backpressure propagation, batching / > concurrency > > > > > > > control, timeout / retry, in-flight request draining, model > > > loading, > > > > > GPU > > > > > > > warmup, resource scheduling, and fault recovery. > > > > > > > > > > > > > > From this perspective, RpcOperator is not just “another way to > call > > > > an > > > > > > > external service,” nor is it merely a deployment mechanism for > GPU > > > > > > > operators. More importantly, it defines a service-style > operator > > > > > > > abstraction: when inference becomes part of the logical data > flow, > > > > > Flink > > > > > > > needs to understand and coordinate these runtime semantics, > rather > > > > than > > > > > > > hiding inference completely behind external black-box calls > inside > > > > user > > > > > > > code. > > > > > > > > > > > > > > Some of the Layer 3 runtime improvements follow the same logic. > > > While > > > > > > > mechanisms such as checkpointing or scaling are not exclusive > to AI > > > > > > > workloads, inference-oriented workloads fundamentally change > their > > > > > > > operational assumptions and cost model, making runtime > behavior far > > > > > more > > > > > > > critical than in traditional data processing systems. When > > > per-record > > > > > > > computation is expensive, GPU warmup and model loading are > costly, > > > > and > > > > > > the > > > > > > > execution environment may involve elastic / preemptible > resources, > > > > the > > > > > > cost > > > > > > > of global rollback or disruptive scaling becomes much higher > than > > > in > > > > > > > traditional row-at-a-time BI / ETL workloads. As a result, > runtime > > > > > > behavior > > > > > > > that may have been acceptable for traditional workloads can > > > directly > > > > > > affect > > > > > > > stability and resource efficiency in inference-oriented > workloads. > > > > > > > > > > > > > > At the same time, the umbrella proposal helps provide a shared > > > > context > > > > > > for > > > > > > > discussing how these capabilities relate to each other and what > > > > common > > > > > > > runtime assumptions they rely on. The more important value of > the > > > > > > umbrella > > > > > > > is to align on the workload model, design principles, > boundaries, > > > and > > > > > > > dependencies between capabilities, so that independently > evolving > > > > > pieces > > > > > > > such as RpcOperator, GPU resource declaration, batching / > > > concurrency > > > > > > > control, non-disruptive scaling, and regional checkpointing do > not > > > > end > > > > > up > > > > > > > with inconsistent runtime semantics. > > > > > > > > > > > > > > Based on this discussion, I will update the proposal to make > the > > > > > workload > > > > > > > model, RpcOperator boundary, and Layer 3 dependency > relationship > > > > > clearer. > > > > > > > > > > > > > > Best, > > > > > > > Guowei > > > > > > > Best, > > > > > > > Guowei > > > > > > > > > > > > > > > > > > > > > On Fri, May 8, 2026 at 12:37 PM zhangjiaogg < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > Hi Guowei and all, > > > > > > > > > > > > > > > > Thank you for driving this initiative. Strong +1 on the > overall > > > > > > > direction. > > > > > > > > > > > > > > > > From our perspective, the core value of this proposal lies > in two > > > > > > areas: > > > > > > > > extending Flink's intelligent processing capabilities for > > > > multimodal > > > > > > > data, > > > > > > > > and enabling native, in-pipeline local inference. As AI > > > > capabilities > > > > > > > > continue to advance, multimodal data is accounting for an > > > > > increasingly > > > > > > > > large share of overall data volume, and the ability to > perform > > > > > > > intelligent, > > > > > > > > real-time processing on this data — not just ingestion or > > > routing, > > > > > but > > > > > > > > actual inference and transformation within the stream — is > > > > becoming a > > > > > > > > critical requirement across industries. Today, most pipelines > > > treat > > > > > > > > multimodal objects as opaque blobs and push inference to > external > > > > > > > systems, > > > > > > > > which works but at the cost of complexity, latency, and > > > > consistency. > > > > > > > > > > > > > > > > This is exactly the pain one of our customers experiences in > > > their > > > > > > > > autonomous driving data pipelines. Video and image data > captured > > > by > > > > > > > onboard > > > > > > > > cameras must go through annotation, frame extraction, quality > > > > > > filtering, > > > > > > > > and both unstructured and structured data transformation > before > > > it > > > > > can > > > > > > be > > > > > > > > used for model training — a workflow that today requires > > > combining > > > > > > > multiple > > > > > > > > specialized systems: a stream engine for structured > processing, a > > > > > > > separate > > > > > > > > framework (e.g., Ray Data) for multimodal processing, and an > > > > external > > > > > > > model > > > > > > > > serving layer for inference. Each system boundary introduces > > > > > > intermediate > > > > > > > > storage, operational overhead, and data consistency > challenges > > > > across > > > > > > the > > > > > > > > full pipeline. If Flink can handle this end-to-end — with > native > > > > > > > multimodal > > > > > > > > types, local GPU inference operators, and unified > checkpointing — > > > > the > > > > > > > > entire workflow becomes a single Flink job, intermediate > storage > > > is > > > > > > > > eliminated, and fault recovery covers the pipeline as a > whole. > > > > > > > > > > > > > > > > We look forward to the sub-FLIP discussions and would be > happy to > > > > > > > > contribute. > > > > > > > > > > > > > > > > Best regards > > > > > > > > Jiao Zhang > > > > > > > > > > > > > > > > At 2026-05-07 12:58:15, "zihao chen" <[email protected]> > > > > wrote: > > > > > > > > >Hi, all, > > > > > > > > > > > > > > > > > >I’d like to share some thoughts based on our internal > experience > > > > > > > > >with AI workloads on Flink. > > > > > > > > > > > > > > > > > >At Tencent, we have production scenarios where Flink is > used in > > > > > > > > >AI-related pipelines. > > > > > > > > > > > > > > > > > >Based on these workloads, we explored elasticity and > autoscaling > > > > > > > > >for cloud-native stream processing systems and published our > > > > > > > > >experience in SIGMOD 2025: > > > > > > > > > > > > > > > > > >"Oceanus: Enable SLO-Aware Vertical Autoscaling for > > > > > > > > >Cloud-Native Streaming Services in Tencent" [1] > > > > > > > > > > > > > > > > > >As our workloads evolved, we also started to see increasing > > > > > > > > >GPU-based training and inference scenarios. > > > > > > > > > > > > > > > > > >Our current solution integrates Flink with external GPU > > > services. > > > > > > > > >While this works functionally, it also introduces several > > > > practical > > > > > > > > >issues, such as: > > > > > > > > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > fragmented lifecycle management > > > > > > > > > - > > > > > > > > > > > > > > > > > > operational complexity > > > > > > > > > - > > > > > > > > > > > > > > > > > > inconsistent scaling/recovery behavior across systems > > > > > > > > > > > > > > > > > >From this perspective, I think FLIP-577 is exploring a very > > > > > > > > >meaningful direction. > > > > > > > > > > > > > > > > > >In particular, I agree with the idea of integrating > GPU-backed > > > > > > > > >computation more naturally into Flink’s runtime model, > instead > > > of > > > > > > > > >treating it purely as an external service integration > problem. > > > > > > > > > > > > > > > > > >Besides, from the elasticity perspective, our experience is > that > > > > > > > > >GPU workloads have very different characteristics compared > with > > > > > > > > >traditional CPU workloads: > > > > > > > > > > > > > > > > > > - > > > > > > > > > > > > > > > > > > GPU resources are expensive and scarce > > > > > > > > > - > > > > > > > > > > > > > > > > > > Startup and replay costs are significantly higher > > > > > > > > > - > > > > > > > > > > > > > > > > > > Long-running tasks make scaling and recovery more > challenging > > > > > > > > > > > > > > > > > >In our experience, GPU elasticity cannot simply reuse the > > > > > > > > >assumptions behind traditional CPU elasticity. > > > > > > > > > > > > > > > > > >Because of this, elasticity becomes especially important for > > > > > > > > >production AI workloads, not only for resource efficiency, > but > > > > also > > > > > > > > >for reducing scaling and recovery overhead. > > > > > > > > > > > > > > > > > >More broadly, AI workloads increasingly require Flink to > > > > collaborate > > > > > > > > >more naturally with GPU-backed computation, and I believe > > > > > > > > >FLIP-577 is exploring an important direction toward > addressing > > > > > > > > >this gap. > > > > > > > > > > > > > > > > > >Overall, I’m looking forward to further discussions about > this > > > > FLIP. > > > > > > > > > > > > > > > > > >[1] https://dl.acm.org/doi/abs/10.1145/3722212.3724445 > > > > > > > > > > > > > > > > > > > > > > > > > > >Best, > > > > > > > > >Zihao Chen > > > > > > > > > > > > > > > > > >Yong Fang <[email protected]> 于2026年5月7日周四 11:11写道： > > > > > > > > > > > > > > > > > >> Hi devs, > > > > > > > > >> > > > > > > > > >> Thanks Guowei for initiating this proposal. I think this > is an > > > > > > > important > > > > > > > > >> step for Flink towards the era of AI data processing, > very big > > > > +1. > > > > > > > > >> > > > > > > > > >> I’d like to share some scenarios and requirements of > > > leveraging > > > > > > > PyFlink > > > > > > > > for > > > > > > > > >> AI data processing at ByteDance. Currently, we run tens of > > > > > thousands > > > > > > > of > > > > > > > > >> pyflink/flink jobs, using millions of CPU cores. > > > > > > > > >> > > > > > > > > >> 1) Multimodal Data Processing > > > > > > > > >> We want to use PyFlink to generate multimodal feature > data. > > > The > > > > > > > typical > > > > > > > > >> workflow starts by reading ID-based raw data and > performing > > > > large > > > > > > > table > > > > > > > > >> joins and ETL computations. We then fetch multimodal > assets > > > such > > > > > as > > > > > > > > images, > > > > > > > > >> videos, texts and audios from object storage by ID. These > > > > > multimodal > > > > > > > > data > > > > > > > > >> are either sent to an RPC service (backed by local models > or > > > > > remote > > > > > > > > large > > > > > > > > >> models), or processed via local GPU computing for frame > > > > > extraction, > > > > > > > > >> embedding generation and other tasks. After multimodal > > > > > computation, > > > > > > > > output > > > > > > > > >> results including embeddings, processed images and > multimodal > > > > > > metadata > > > > > > > > are > > > > > > > > >> generated and persisted into the downstream multimodal > data > > > > lake. > > > > > > > > >> > > > > > > > > >> 2) Stream-Batch Unified Data Training > > > > > > > > >> We use PyFlink to consume processed sample data from MQ or > > > data > > > > > > lakes. > > > > > > > > >> Within the data pipeline, data may be shuffled by key, > then > > > fed > > > > > into > > > > > > > > >> parameter servers or local services for CPU-based or > GPU-based > > > > > model > > > > > > > > >> training. Such workloads strongly demand optimized CPU & > GPU > > > > > hybrid > > > > > > > > >> scheduling, worker node restart capability, fast scaling, > as > > > > well > > > > > as > > > > > > > > native > > > > > > > > >> support for unified streaming and batch training. > > > > > > > > >> > > > > > > > > >> While supporting the above two categories of workloads, we > > > have > > > > > > > several > > > > > > > > >> common requirements for PyFlink and Flink Core: > > > > > > > > >> > > > > > > > > >> 1) Native Python Computing Capability > > > > > > > > >> We need more user-friendly DataFrame APIs, comprehensive > > > > built-in > > > > > > data > > > > > > > > >> types for image and audio, as well as richer multimodal > > > > computing > > > > > > > > >> operators. This enables users to develop multimodal data > > > > > processing > > > > > > > jobs > > > > > > > > >> more efficiently, and allows better optimization and > > > scheduling > > > > at > > > > > > the > > > > > > > > >> execution plan layer. > > > > > > > > >> > > > > > > > > >> 2) CPU & GPU Scheduling and Resource Management > > > > > > > > >> It is expected to tag resource requirements at > fine-grained > > > > levels > > > > > > > such > > > > > > > > as > > > > > > > > >> python user-defined functions. The scheduler should > provide > > > > > enhanced > > > > > > > > >> resource orchestration and task scheduling, enabling more > > > > flexible > > > > > > > > >> heterogeneous resource management. > > > > > > > > >> > > > > > > > > >> 3) Embedded Server Node Capability in Pipeline > > > > > > > > >> We hope to launch dedicated server nodes inside a single > > > > pipeline, > > > > > > to > > > > > > > > load > > > > > > > > >> local models or access remote models, which may be shared > > > > between > > > > > > > > different > > > > > > > > >> operators in the pipeline. This unifies data ETL and > > > multimodal > > > > > > > > computing > > > > > > > > >> within one end-to-end pipeline, greatly simplifying > operation > > > > and > > > > > > > > >> maintenance for business teams. > > > > > > > > >> > > > > > > > > >> 4) Performance and Stability Optimization > > > > > > > > >> Key enhancements include zero-copy data transfer between > Flink > > > > TM > > > > > > > > processes > > > > > > > > >> and Python processes, fast scaling of compute nodes, fast > > > > > > > checkpointing > > > > > > > > >> mechanism, and shuffle optimization for large-scale > datasets. > > > > > These > > > > > > > > >> improvements will significantly boost the performance and > > > > > stability > > > > > > of > > > > > > > > >> PyFlink multimodal workloads. > > > > > > > > >> > > > > > > > > >> I’m really excited to see that FLIP-577 has covered all > the > > > real > > > > > > > > production > > > > > > > > >> scenarios and requirements mentioned above. It's a good > chance > > > > to > > > > > > > > iterate > > > > > > > > >> and enrich the core capabilities of PyFlink and Flink Core > > > > > targeting > > > > > > > > these > > > > > > > > >> AI data processing scenarios, and build Flink into a > > > first-class > > > > > AI > > > > > > > data > > > > > > > > >> processing engine. > > > > > > > > >> > > > > > > > > >> I’m very much looking forward to the progress. > > > > > > > > >> > > > > > > > > >> Best, > > > > > > > > >> FangYong > > > > > > > > >> > > > > > > > > >> On Wed, May 6, 2026 at 8:36 PM FeatZhang < > > > [email protected]> > > > > > > > wrote: > > > > > > > > >> > > > > > > > > >> > Hi all, > > > > > > > > >> > > > > > > > > > >> > This is a great topic, and honestly long overdue. > > > > > > > > >> > > > > > > > > > >> > With the rapid growth of AI applications, we have been > > > seeing > > > > a > > > > > > > > >> > significant increase in real-world demands from users > who > > > are > > > > > > > already > > > > > > > > >> > building on Flink and other traditional data processing > or > > > BI > > > > > > > engines. > > > > > > > > >> > From a platform perspective, more and more teams are > trying > > > to > > > > > > > > >> > integrate AI capabilities directly into their existing > > > > streaming > > > > > > > > >> > pipelines, rather than treating them as separate > systems. > > > > > > > > >> > > > > > > > > > >> > This is not an isolated trend — it is becoming a common > > > > > > requirement > > > > > > > > >> > across industries. > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 1. What is changing in real systems > > > > > > > > >> > > > > > > > > > >> > We are observing a consistent shift: > > > > > > > > >> > > > > > > > > > >> > AI is moving from offline analysis or request-time > scoring > > > to > > > > > > > > >> > continuous, event-driven decision making. > > > > > > > > >> > > > > > > > > > >> > In other words: > > > > > > > > >> > > > > > > > > > >> > AI is becoming part of the data stream itself. > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 2. Representative production scenarios > > > > > > > > >> > > > > > > > > > >> > 2.1 Real-time fraud detection (per-event decision under > > > strict > > > > > > > > latency) > > > > > > > > >> > > > > > > > > > >> > Typical setup > > > > > > > > >> > > > > > > > > > >> > Continuous transaction stream (payments, logins, > transfers) > > > > > > > > >> > Each event must be evaluated within milliseconds > > > > > > > > >> > Decision depends on: > > > > > > > > >> > > > > > > > > > >> > recent user behavior > > > > > > > > >> > device / IP patterns > > > > > > > > >> > short-term aggregates > > > > > > > > >> > > > > > > > > > >> > What is happening in practice > > > > > > > > >> > > > > > > > > > >> > Models are already deeply integrated into decision flow > > > > > > > > >> > Feature freshness directly impacts detection accuracy > > > > > > > > >> > > > > > > > > > >> > Current pain > > > > > > > > >> > > > > > > > > > >> > Features computed in streaming, but inference is remote > > > > > > > > >> > Network overhead adds to critical path latency > > > > > > > > >> > Hard to ensure training/serving consistency > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 2.2 Real-time recommendation and ads (continuous > re-ranking) > > > > > > > > >> > > > > > > > > > >> > Typical setup > > > > > > > > >> > > > > > > > > > >> > User interaction stream (click, view, dwell time) > > > > > > > > >> > Continuous feature updates (session + short-term > behavior) > > > > > > > > >> > Inference triggered per interaction > > > > > > > > >> > > > > > > > > > >> > What is happening in practice > > > > > > > > >> > > > > > > > > > >> > Increasing reliance on real-time context > > > > > > > > >> > Model-based ranking becomes core logic > > > > > > > > >> > > > > > > > > > >> > Current pain > > > > > > > > >> > > > > > > > > > >> > Offline and online feature pipelines diverge > > > > > > > > >> > Training-serving skew is common > > > > > > > > >> > Inference orchestration is ad-hoc > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 2.3 Streaming RAG / knowledge systems (continuous > indexing) > > > > > > > > >> > > > > > > > > > >> > Typical setup > > > > > > > > >> > > > > > > > > > >> > Continuous ingestion of documents, logs, or knowledge > > > > > > > > >> > Pipeline: > > > > > > > > >> > > > > > > > > > >> > chunking → embedding → indexing → retrieval > > > > > > > > >> > > > > > > > > > >> > Typical use cases > > > > > > > > >> > > > > > > > > > >> > AI copilots > > > > > > > > >> > enterprise knowledge assistants > > > > > > > > >> > observability systems > > > > > > > > >> > > > > > > > > > >> > Current pain > > > > > > > > >> > > > > > > > > > >> > Built via loosely coupled services or scripts > > > > > > > > >> > No strong consistency guarantees > > > > > > > > >> > Difficult to scale and recover > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 2.4 Real-time feedback loop (continuous evaluation) > > > > > > > > >> > > > > > > > > > >> > Typical setup > > > > > > > > >> > > > > > > > > > >> > Prediction at time T > > > > > > > > >> > Label arrives at T + Δ > > > > > > > > >> > > > > > > > > > >> > Required processing > > > > > > > > >> > > > > > > > > > >> > prediction stream JOIN label stream → metrics → > optimization > > > > > > > > >> > > > > > > > > > >> > Current pain > > > > > > > > >> > > > > > > > > > >> > Alignment logic duplicated across systems > > > > > > > > >> > Late data handling is complex > > > > > > > > >> > No reusable evaluation abstraction > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 2.5 AI replacing rule-based decision logic > > > > > > > > >> > > > > > > > > > >> > Evolution > > > > > > > > >> > > > > > > > > > >> > From: > > > > > > > > >> > > > > > > > > > >> > rule engine / CEP > > > > > > > > >> > > > > > > > > > >> > To: > > > > > > > > >> > > > > > > > > > >> > model / LLM → decision > > > > > > > > >> > > > > > > > > > >> > Implication > > > > > > > > >> > > > > > > > > > >> > AI is becoming the core decision layer inside streaming > > > > systems. > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 3. Architectural shift > > > > > > > > >> > > > > > > > > > >> > Across all scenarios: > > > > > > > > >> > > > > > > > > > >> > From: > > > > > > > > >> > > > > > > > > > >> > data processing → feature system → model serving → > > > evaluation > > > > > > > > >> > > > > > > > > > >> > To: > > > > > > > > >> > > > > > > > > > >> > stream = feature + inference + decision + feedback > > > > > > > > >> > > > > > > > > > >> > This reflects a fundamental change in system boundaries. > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 4. Why externalized architectures break down > > > > > > > > >> > > > > > > > > > >> > Most current implementations rely on multiple systems: > > > > > > > > >> > > > > > > > > > >> > stream processing > > > > > > > > >> > feature store > > > > > > > > >> > model serving > > > > > > > > >> > vector database > > > > > > > > >> > > > > > > > > > >> > This introduces several fundamental issues in real-time > > > > > scenarios. > > > > > > > > >> > > > > > > > > > >> > 4.1 Latency dominated by system boundaries > > > > > > > > >> > > > > > > > > > >> > stream → network → model service → response > > > > > > > > >> > > > > > > > > > >> > network overhead is unavoidable > > > > > > > > >> > batching is not controlled by the stream runtime > > > > > > > > >> > no end-to-end backpressure > > > > > > > > >> > > > > > > > > > >> > Latency becomes a system-level artifact rather than a > > > compute > > > > > > > > property. > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 4.2 Inconsistent data between training and serving > > > > > > > > >> > > > > > > > > > >> > offline vs online features > > > > > > > > >> > different definitions or time windows > > > > > > > > >> > > > > > > > > > >> > Models operate on inconsistent data distributions. > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 4.3 State fragmentation > > > > > > > > >> > > > > > > > > > >> > user/session context must be rebuilt or fetched > > > > > > > > >> > loss of data locality > > > > > > > > >> > processing becomes call-driven > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 4.4 Feedback loop is not composable > > > > > > > > >> > > > > > > > > > >> > difficult alignment of prediction and label streams > > > > > > > > >> > no unified handling of late data > > > > > > > > >> > duplicated evaluation logic > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 4.5 Operational complexity > > > > > > > > >> > > > > > > > > > >> > multiple systems to scale > > > > > > > > >> > multiple failure domains > > > > > > > > >> > complex debugging paths > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 5. Why this aligns with Flink > > > > > > > > >> > > > > > > > > > >> > These workloads require: > > > > > > > > >> > > > > > > > > > >> > event-driven execution > > > > > > > > >> > strong state management > > > > > > > > >> > precise time semantics > > > > > > > > >> > continuous feedback > > > > > > > > >> > > > > > > > > > >> > These are exactly Flink’s core strengths. > > > > > > > > >> > > > > > > > > > >> > The key insight is: > > > > > > > > >> > > > > > > > > > >> > Inference should be modeled as a dataflow operator, not > an > > > > > > external > > > > > > > > >> > service. > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 6. Implication > > > > > > > > >> > > > > > > > > > >> > If we model: > > > > > > > > >> > > > > > > > > > >> > data → feature → inference → decision → feedback > > > > > > > > >> > > > > > > > > > >> > within Flink, we can achieve: > > > > > > > > >> > > > > > > > > > >> > unified scheduling > > > > > > > > >> > shared state > > > > > > > > >> > consistent time semantics > > > > > > > > >> > end-to-end fault tolerance > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > 7. Conclusion > > > > > > > > >> > > > > > > > > > >> > This is not simply about adding AI support to Flink. > > > > > > > > >> > > > > > > > > > >> > It is about recognizing that: > > > > > > > > >> > > > > > > > > > >> > Real-time AI systems are fundamentally streaming > systems. > > > > > > > > >> > > > > > > > > > >> > The question is whether Flink evolves to support this > > > > natively, > > > > > or > > > > > > > > >> > remains a preprocessing layer in front of external AI > > > stacks. > > > > > > > > >> > > > > > > > > > >> > ________________________________ > > > > > > > > >> > > > > > > > > > >> > Happy to follow up with a more concrete proposal (e.g., > > > > > inference > > > > > > > > >> > operator abstraction) if there is interest. > > > > > > > > >> > > > > > > > > > >> > Thanks. > > > > > > > > >> > > > > > > > > > >> > On Mon, May 4, 2026 at 10:31 PM Gen Luo < > > > [email protected]> > > > > > > > wrote: > > > > > > > > >> > > > > > > > > > > >> > > Hi all, > > > > > > > > >> > > > > > > > > > > >> > > Thank you Guowei Ma for driving this discussion, and > > > thanks > > > > > > > everyone > > > > > > > > >> for > > > > > > > > >> > > the valuable insights. Inspired by this exchange, I’d > like > > > > to > > > > > > > share > > > > > > > > a > > > > > > > > >> few > > > > > > > > >> > > thoughts. > > > > > > > > >> > > > > > > > > > > >> > > While “AI-Native” covers broad ground, I believe this > FLIP > > > > > does > > > > > > > not > > > > > > > > >> > > overextend Flink’s scope. It’s a necessary iteration > > > driven > > > > by > > > > > > > > evolving > > > > > > > > >> > > user scenarios and AI advancements, particularly > > > multimodal > > > > > > > > processing. > > > > > > > > >> > > Given the growing adoption of multimodal applications > and > > > > > > > increasing > > > > > > > > >> > > interest in low-latency inference, initiating these > > > > > enhancements > > > > > > > is > > > > > > > > a > > > > > > > > >> > > timely step to better align Flink with evolving AI > > > > workloads. > > > > > > > > >> > > > > > > > > > > >> > > From our engagements with customers and developers, we > > > > > observe a > > > > > > > > clear > > > > > > > > >> > > shift in both workloads and user expectations. Model > > > > inference > > > > > > is > > > > > > > > >> > > increasingly central to data pipelines, with > multimodal AI > > > > > tasks > > > > > > > > >> growing > > > > > > > > >> > > rapidly. Traditional real-time scenarios (e.g., > monitoring > > > > and > > > > > > > > >> analytics) > > > > > > > > >> > > now leverage models and agent frameworks like Flink > Agent > > > > for > > > > > > > > >> > intelligent, > > > > > > > > >> > > multi-turn decision-making, while large-scale offline > > > > compute > > > > > is > > > > > > > > also > > > > > > > > >> > > shifting toward LLMs and vision models. Alongside this > > > > > workload > > > > > > > > >> > evolution, > > > > > > > > >> > > developer workflows have adapted: AI practitioners > > > naturally > > > > > > > prefer > > > > > > > > >> > Python > > > > > > > > >> > > and DataFrame-style APIs. As AI-assisted coding > matures, > > > > > > aligning > > > > > > > > >> system > > > > > > > > >> > > interfaces with these familiar patterns will directly > > > > improve > > > > > > > > >> > AI-generated > > > > > > > > >> > > code quality and significantly lower adoption > barriers for > > > > the > > > > > > AI > > > > > > > > >> > community. > > > > > > > > >> > > > > > > > > > > >> > > Today, many AI evaluation tools don’t yet recommend > Flink > > > > for > > > > > AI > > > > > > > > >> > > workloads—largely due to limited visibility of Flink’s > > > > > relevant > > > > > > > > >> > > capabilities rather than fundamental incompatibility. > In > > > > > > reality, > > > > > > > > Flink > > > > > > > > >> > has > > > > > > > > >> > > unique strengths here. For example, generating > multimodal > > > > > > samples > > > > > > > is > > > > > > > > >> > often > > > > > > > > >> > > a multi-day, GPU-heavy process. Flink’s streaming > model, > > > > > > combined > > > > > > > > with > > > > > > > > >> > > checkpointing and reduced disk I/O, is well-suited for > > > such > > > > > > > > >> long-running > > > > > > > > >> > > tasks—a direction also pursued by engines like Daft > and > > > Ray > > > > > > Data. > > > > > > > > With > > > > > > > > >> > > Flink’s proven production stability, we’re > well-positioned > > > > for > > > > > > > both > > > > > > > > >> batch > > > > > > > > >> > > and future real-time multimodal streaming inference. > > > > Targeted > > > > > > > > >> > improvements > > > > > > > > >> > > can make these advantages visible, driving better user > > > > > > experiences > > > > > > > > and > > > > > > > > >> > > healthier ecosystem growth. > > > > > > > > >> > > > > > > > > > > >> > > I’d also note a lesson from FlinkML. It attempted to > cover > > > > > model > > > > > > > > >> training > > > > > > > > >> > > but struggled to align with the fast-iteration, > > > > > > > > Python/notebook-centric > > > > > > > > >> > > workflows preferred by ML researchers. Flink’s core > > > strength > > > > > > lies > > > > > > > in > > > > > > > > >> > > high-concurrency, production-grade inference > > > > orchestration—not > > > > > > > > training > > > > > > > > >> > > lifecycle management (e.g., experiment tracking, > > > > versioning). > > > > > > This > > > > > > > > >> > mismatch > > > > > > > > >> > > limited its adoption. > > > > > > > > >> > > > > > > > > > > >> > > This proposal, however, takes a different path. It > doesn’t > > > > aim > > > > > > to > > > > > > > > >> replace > > > > > > > > >> > > training frameworks. Instead, it introduces modern AI > > > > concepts > > > > > > > > >> > (multimodal > > > > > > > > >> > > data, LLMs) as first-class citizens for inference, > built > > > > atop > > > > > > > > Flink’s > > > > > > > > >> > > computation strengths. Think Ray Data’s scope (plus > simple > > > > > > > > co-located > > > > > > > > >> > > serving), not Train/Tune. Crucially, unlike the > FlinkML > > > era, > > > > > > > today’s > > > > > > > > >> > models > > > > > > > > >> > > use standardized interfaces and mature serving > frameworks, > > > > > > > allowing > > > > > > > > >> Flink > > > > > > > > >> > > to integrate external models seamlessly without heavy > > > > > > > > >> > > customization—significantly lowering project risk. > > > > > > > > >> > > > > > > > > > > >> > > This FLIP marks Flink’s another starting point for > the AI > > > > era. > > > > > > > While > > > > > > > > >> > > details need refinement, I believe this direction > aligns > > > > with > > > > > > both > > > > > > > > >> > current > > > > > > > > >> > > and future user needs and Flink’s evolution. > > > > > > > > >> > > > > > > > > > > >> > > Best, > > > > > > > > >> > > Gen > > > > > > > > >> > > > > > > > > > > >> > > On Mon, May 4, 2026 at 12:15 PM Jark Wu < > [email protected] > > > > > > > > > > wrote: > > > > > > > > >> > > > > > > > > > > >> > > > Hi Guowei, > > > > > > > > >> > > > > > > > > > > > >> > > > Thanks for driving this. +1 on the overall > direction. > > > > > Flink's > > > > > > > > >> > > > streaming processing and checkpoint mechanism give > it a > > > > > > > structural > > > > > > > > >> > > > advantage over systems like Daft and Ray. But today, > > > these > > > > > > > runtime > > > > > > > > >> > > > strengths are held back by gaps in Python API, GPU > > > > > scheduling, > > > > > > > and > > > > > > > > >> > > > native multimodal data handling. This umbrella FLIP > > > > > addresses > > > > > > > > exactly > > > > > > > > >> > > > that gap, comprehensively and systematically. I > believe > > > > > > > multimodal > > > > > > > > >> > > > data processing is the biggest opportunity for > > > traditional > > > > > > data > > > > > > > > infra > > > > > > > > >> > > > to transition into AI infra, and this is one of the > most > > > > > > > important > > > > > > > > >> > > > FLIPs for Flink in the AI era. > > > > > > > > >> > > > > > > > > > > > >> > > > As one of the Table/SQL module maintainers, we would > > > like > > > > to > > > > > > > > >> > > > contribute the built-in multimodal processing UDFs > > > (audio, > > > > > > > video, > > > > > > > > >> > > > image, text) and native multimodal data types > (Tensor, > > > > > Image, > > > > > > > > >> > > > Embedding, etc.) as first-class citizens in the type > > > > system. > > > > > > > > Looking > > > > > > > > >> > > > forward to the sub-FLIP discussions. > > > > > > > > >> > > > > > > > > > > > >> > > > Best, > > > > > > > > >> > > > Jark > > > > > > > > >> > > > > > > > > > > > >> > > > On Thu, 30 Apr 2026 at 18:42, Guowei Ma < > > > > > [email protected] > > > > > > > > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > > > > > >> > > > > Hi,Yaroslav > > > > > > > > >> > > > > > > > > > > > > >> > > > > Thanks for taking the time to write this detailed > > > > > feedback. > > > > > > > Let > > > > > > > > me > > > > > > > > >> > > > clarify > > > > > > > > >> > > > > the intent of the proposal first. > > > > > > > > >> > > > > > > > > > > > > >> > > > > I am not saying that Flink should become an AI > > > > framework, > > > > > an > > > > > > > ML > > > > > > > > >> > platform, > > > > > > > > >> > > > > or a model serving system. The way I use > "AI-Native" > > > in > > > > > this > > > > > > > > >> > proposal is > > > > > > > > >> > > > to > > > > > > > > >> > > > > say that Flink should support, as first-class > > > citizens, > > > > > the > > > > > > > core > > > > > > > > >> > objects > > > > > > > > >> > > > > and execution patterns that frequently show up in > > > > > > AI-oriented > > > > > > > > data > > > > > > > > >> > > > > processing — instead of leaving them entirely to > > > > external > > > > > > > > systems > > > > > > > > >> or > > > > > > > > >> > ad > > > > > > > > >> > > > hoc > > > > > > > > >> > > > > user-defined integrations. > > > > > > > > >> > > > > > > > > > > > > >> > > > > These objects and execution patterns include: > > > > > > > > >> > > > > > > > > > > > > >> > > > > - Multimodal and unstructured data objects > such as > > > > > > images, > > > > > > > > >> video, > > > > > > > > >> > > > audio, > > > > > > > > >> > > > > tensors, embeddings, and object references. > > > > > > > > >> > > > > - Model inference as part of the data flow, > rather > > > > than > > > > > > an > > > > > > > > >> > entirely > > > > > > > > >> > > > > external black-box service call. > > > > > > > > >> > > > > - Operators backed by heterogeneous resources > such > > > as > > > > > > GPUs. > > > > > > > > >> > > > > - Pythonic and vectorized processing styles. > > > > > > > > >> > > > > - Long-running, long-tailed asynchronous > > > computation. > > > > > > > > >> > > > > > > > > > > > > >> > > > > "AI-Native" is just a shorthand here, meaning that > > > Flink > > > > > > > should > > > > > > > > >> > natively > > > > > > > > >> > > > > understand and support the core abstractions of > this > > > > class > > > > > > of > > > > > > > > >> > workloads. > > > > > > > > >> > > > > The FLIP needs to make the target workload class > > > > clearer. > > > > > > What > > > > > > > > we > > > > > > > > >> > care > > > > > > > > >> > > > > about is not any specific model paradigm — LLM, > CV, > > > > > > > > recommendation, > > > > > > > > >> > or > > > > > > > > >> > > > > traditional ML inference — but a class of data > > > > processing > > > > > > > > workloads > > > > > > > > >> > with > > > > > > > > >> > > > > shared runtime and topology characteristics: > > > > > > > > >> > > > > > > > > > > > > >> > > > > - A single computation may take seconds or even > > > > > minutes, > > > > > > > > instead > > > > > > > > >> > of > > > > > > > > >> > > > > microseconds as in traditional row-at-a-time > > > > > processing. > > > > > > > > >> > > > > - Execution often involves heterogeneous > resources > > > > such > > > > > > as > > > > > > > > CPU + > > > > > > > > >> > GPU, > > > > > > > > >> > > > > where GPUs are expensive and scarce. > > > > > > > > >> > > > > - Data is often multimodal large objects > (images, > > > > > video, > > > > > > > > audio, > > > > > > > > >> > > > tensors, > > > > > > > > >> > > > > embeddings), rather than structured small > records. > > > > > > > > >> > > > > - Computation logic often includes model > inference > > > or > > > > > > > > >> > service-style > > > > > > > > >> > > > > invocations as part of the pipeline. > > > > > > > > >> > > > > - Many target topologies are relatively > > > shuffle-light > > > > > and > > > > > > > > don't > > > > > > > > >> > > > > necessarily involve complex keyed-state > migration, > > > > e.g. > > > > > > > URI → > > > > > > > > >> > > > preprocessing > > > > > > > > >> > > > > → inference → sink. > > > > > > > > >> > > > > > > > > > > > > >> > > > > Ten years ago, many ML workloads took the form of > > > > offline > > > > > > > > training > > > > > > > > >> > plus > > > > > > > > >> > > > > online feature serving. Flink already played a > strong > > > > role > > > > > > in > > > > > > > > >> feature > > > > > > > > >> > > > > engineering, streaming feature computation, and > > > > real-time > > > > > > data > > > > > > > > >> > > > preparation, > > > > > > > > >> > > > > so there was no strong need to reshape Flink into > an > > > > > > > "ML-Native" > > > > > > > > >> > engine. > > > > > > > > >> > > > > > > > > > > > > >> > > > > What is changing today is that model inference > itself > > > is > > > > > > > > >> increasingly > > > > > > > > >> > > > > becoming part of the data processing pipeline; > > > > multimodal > > > > > > > > objects > > > > > > > > >> > are no > > > > > > > > >> > > > > longer just opaque blobs in external storage, but > data > > > > > > objects > > > > > > > > that > > > > > > > > >> > need > > > > > > > > >> > > > to > > > > > > > > >> > > > > be referenced, passed, transformed, inferred > over, and > > > > > > landed > > > > > > > > >> inside > > > > > > > > >> > the > > > > > > > > >> > > > > engine. This is not simply one more ML use case — > it > > > is > > > > a > > > > > > > > change in > > > > > > > > >> > the > > > > > > > > >> > > > > shape of workloads Flink needs to support. > > > > > > > > >> > > > > > > > > > > > > >> > > > > On whether the user demand is real, the validation > > > > signals > > > > > > we > > > > > > > > are > > > > > > > > >> > > > currently > > > > > > > > >> > > > > seeing include: > > > > > > > > >> > > > > > > > > > > > > >> > > > > - Within Alibaba, multimodal data processing is > > > > already > > > > > > in > > > > > > > > >> > production, > > > > > > > > >> > > > > covering image, video, audio, and text > modalities. > > > > > > > > >> > > > > - In offline conversations with several > companies > > > > > > > (including > > > > > > > > >> > ByteDance > > > > > > > > >> > > > > and Tencent), we have heard substantial demand > for > > > > > Flink > > > > > > to > > > > > > > > >> > support > > > > > > > > >> > > > AI data > > > > > > > > >> > > > > processing / multimodal data processing. > > > > > > > > >> > > > > - On the ecosystem side, we are working with > NVIDIA > > > > on > > > > > a > > > > > > > > joint > > > > > > > > >> > demo > > > > > > > > >> > > > > focused on multimodal data processing, planned > for > > > > > Flink > > > > > > > > Forward > > > > > > > > >> > Asia. > > > > > > > > >> > > > > - The emergence and growth of systems such as > Daft, > > > > Ray > > > > > > > Data, > > > > > > > > >> > > > > Data-Juicer, and LAS also reflect rapidly > growing > > > > > demand > > > > > > > for > > > > > > > > >> > > > multimodal > > > > > > > > >> > > > > data processing. > > > > > > > > >> > > > > - There have also been independent discussions > in > > > > this > > > > > > > > direction > > > > > > > > >> > > > within > > > > > > > > >> > > > > the community — for example, the > "Streaming-native > > > AI > > > > > > > > Inference > > > > > > > > >> > > > Runtime > > > > > > > > >> > > > > Layer" proposal on the dev list. > > > > > > > > >> > > > > > > > > > > > > >> > > > > On "why now, instead of waiting for > standardization" > > > — I > > > > > > > > understand > > > > > > > > >> > the > > > > > > > > >> > > > > concern. LLM-related frameworks, APIs, and > > > > > application-level > > > > > > > > >> > patterns are > > > > > > > > >> > > > > indeed changing quickly. If this FLIP were trying > to > > > > bake > > > > > a > > > > > > > > >> specific > > > > > > > > >> > LLM > > > > > > > > >> > > > > API, agent framework, or prompt protocol into > Flink, > > > the > > > > > > risk > > > > > > > > would > > > > > > > > >> > be > > > > > > > > >> > > > high. > > > > > > > > >> > > > > > > > > > > > > >> > > > > But most of the capabilities in this proposal are > not > > > > > > > > LLM-specific. > > > > > > > > >> > They > > > > > > > > >> > > > > are more fundamental data processing and runtime > > > > > > capabilities: > > > > > > > > >> > Pipeline > > > > > > > > >> > > > > Region-level checkpointing, Object Reference, GPU > > > > resource > > > > > > > > >> > declaration, > > > > > > > > >> > > > > columnar data transfer, service-style operator > > > > invocation, > > > > > > > > >> > long-running > > > > > > > > >> > > > > async execution. These are useful for today's LLM > > > > > workloads, > > > > > > > and > > > > > > > > >> > equally > > > > > > > > >> > > > > useful for future AI workloads in shapes we cannot > > > fully > > > > > > > predict > > > > > > > > >> > yet. The > > > > > > > > >> > > > > fast-changing parts should live in the ecosystem > and > > > SDK > > > > > > > layer; > > > > > > > > the > > > > > > > > >> > FLIP > > > > > > > > >> > > > > should focus on more stable engine-level > capabilities. > > > > > > > > >> > > > > > > > > > > > > >> > > > > On tactical changes vs. umbrella, I partly agree > with > > > > you. > > > > > > > Each > > > > > > > > >> > sub-FLIP > > > > > > > > >> > > > > should be discussed, reviewed, and accepted or > > > rejected > > > > on > > > > > > its > > > > > > > > own > > > > > > > > >> > > > merits. > > > > > > > > >> > > > > The umbrella should not bypass the normal FLIP > > > process, > > > > > and > > > > > > > > >> > accepting the > > > > > > > > >> > > > > umbrella does not mean accepting all sub-FLIPs. > That > > > > > said, I > > > > > > > > still > > > > > > > > >> > think > > > > > > > > >> > > > > the umbrella is valuable. Its purpose is not to > bind > > > the > > > > > 11 > > > > > > > > changes > > > > > > > > >> > into > > > > > > > > >> > > > a > > > > > > > > >> > > > > single inseparable package, but to help the > community > > > > > align > > > > > > on > > > > > > > > >> > > > principles, > > > > > > > > >> > > > > clarify boundaries and dependencies, and avoid > > > > conflicting > > > > > > or > > > > > > > > >> > duplicated > > > > > > > > >> > > > > abstractions across related capabilities. > > > > > > > > >> > > > > > > > > > > > > >> > > > > For example, if RpcOperator is not considered > together > > > > > with > > > > > > > > >> > > > non-disruptive > > > > > > > > >> > > > > scaling, it is hard to give GPU operator > elasticity > > > > > coherent > > > > > > > > >> > semantics. > > > > > > > > >> > > > > Deploying inference services independently is > only the > > > > > first > > > > > > > > step; > > > > > > > > >> > the > > > > > > > > >> > > > > harder question is how Flink uniformly handles > service > > > > > > > > discovery, > > > > > > > > >> > > > in-flight > > > > > > > > >> > > > > request draining, backpressure, and failover > during > > > > > scaling. > > > > > > > > >> Without > > > > > > > > >> > an > > > > > > > > >> > > > > umbrella, these capabilities can certainly be > advanced > > > > as > > > > > > > > tactical > > > > > > > > >> > > > changes, > > > > > > > > >> > > > > but we may end up with a set of abstractions that > are > > > > > > locally > > > > > > > > >> usable > > > > > > > > >> > but > > > > > > > > >> > > > > globally inconsistent. > > > > > > > > >> > > > > > > > > > > > > >> > > > > On RpcOperator, I agree that we need to be very > > > careful > > > > in > > > > > > > > defining > > > > > > > > >> > the > > > > > > > > >> > > > > boundary between the Flink runtime and external > > > > > > orchestration > > > > > > > > >> > systems. > > > > > > > > >> > > > > Kubernetes or the Kubernetes Operator may well be > the > > > > > right > > > > > > > > choice > > > > > > > > >> > at the > > > > > > > > >> > > > > physical deployment level. But I still believe > Flink > > > > > needs a > > > > > > > > >> > first-class > > > > > > > > >> > > > > RpcOperator abstraction, because deployment is > only > > > part > > > > > of > > > > > > > the > > > > > > > > >> > problem — > > > > > > > > >> > > > > the harder part is its semantic integration with > the > > > > Flink > > > > > > > job. > > > > > > > > >> > > > > > > > > > > > > >> > > > > If model inference is part of the logical data > flow, > > > > Flink > > > > > > > > needs at > > > > > > > > >> > > > minimum > > > > > > > > >> > > > > to be aware of its service discovery, backpressure > > > > > behavior, > > > > > > > > >> failover > > > > > > > > >> > > > > behavior, in-flight request draining, and scaling > > > > > > > coordination. > > > > > > > > If > > > > > > > > >> > it is > > > > > > > > >> > > > > hidden entirely behind an external black-box > service, > > > it > > > > > is > > > > > > > hard > > > > > > > > >> for > > > > > > > > >> > > > Flink > > > > > > > > >> > > > > to provide consistent job-level semantics and > > > > operational > > > > > > > > >> experience. > > > > > > > > >> > > > > > > > > > > > > >> > > > > So the point of RpcOperator is not necessarily > that > > > > "every > > > > > > > > physical > > > > > > > > >> > > > process > > > > > > > > >> > > > > must be directly launched and managed by Flink > core," > > > > but > > > > > > that > > > > > > > > >> Flink > > > > > > > > >> > > > needs > > > > > > > > >> > > > > to define a service-style operator contract that > > > allows > > > > > such > > > > > > > > >> > operators to > > > > > > > > >> > > > > be invoked correctly by the data flow, coordinated > > > > > correctly > > > > > > > by > > > > > > > > the > > > > > > > > >> > > > > runtime, and understood and operated by users as > part > > > > of a > > > > > > > Flink > > > > > > > > >> job. > > > > > > > > >> > > > > > > > > > > > > >> > > > > On vectorized batch processing, I agree the > long-term > > > > > > > direction > > > > > > > > >> > should > > > > > > > > >> > > > not > > > > > > > > >> > > > > stop at Python. Native columnar / vectorized > execution > > > > is > > > > > an > > > > > > > > >> > end-to-end > > > > > > > > >> > > > > problem that touches connectors, formats, the type > > > > system, > > > > > > > > runtime, > > > > > > > > >> > Java, > > > > > > > > >> > > > > SQL, and Python. The current proposal starts from > the > > > > > > > > Java/Python > > > > > > > > >> > > > boundary > > > > > > > > >> > > > > because that is where the row/column conversion > > > overhead > > > > > is > > > > > > > most > > > > > > > > >> > visible. > > > > > > > > >> > > > > End-to-end columnar execution on the Java and SQL > side > > > > > > > deserves > > > > > > > > to > > > > > > > > >> be > > > > > > > > >> > > > > discussed further as a separate, larger FLIP. > > > > > > > > >> > > > > > > > > > > > > >> > > > > On multimodal types and SerDes complexity, I agree > > > this > > > > > > needs > > > > > > > > to be > > > > > > > > >> > > > handled > > > > > > > > >> > > > > carefully. Making AI-related objects first-class > does > > > > not > > > > > > > imply > > > > > > > > >> that > > > > > > > > >> > > > every > > > > > > > > >> > > > > connector must immediately and fully support > image, > > > > video, > > > > > > > > audio, > > > > > > > > >> > tensor, > > > > > > > > >> > > > > and so on. The concrete incremental path, fallback > > > > > strategy, > > > > > > > and > > > > > > > > >> the > > > > > > > > >> > > > > boundary between formats, connector API, and the > type > > > > > system > > > > > > > > will > > > > > > > > >> be > > > > > > > > >> > > > > discussed further in the corresponding sub-FLIPs. > > > > > > > > >> > > > > > > > > > > > > >> > > > > Coming back to the core of the proposal: it is not > > > about > > > > > > > turning > > > > > > > > >> > Flink > > > > > > > > >> > > > into > > > > > > > > >> > > > > an AI framework. It is about making the core > objects > > > and > > > > > > > > execution > > > > > > > > >> > > > patterns > > > > > > > > >> > > > > of AI-oriented data processing first-class > citizens in > > > > > > Flink. > > > > > > > > >> > > > > > > > > > > > > >> > > > > Best, > > > > > > > > >> > > > > Guowei > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > On Thu, Apr 30, 2026 at 5:37 AM Yaroslav > Tkachenko < > > > > > > > > >> > [email protected] > > > > > > > > >> > > > > > > > > > > > > >> > > > > wrote: > > > > > > > > >> > > > > > > > > > > > > >> > > > > > Hi Guowei, > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Thank you for writing this proposal. > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > I may be in the minority here, but I hope my > voice > > > > will > > > > > be > > > > > > > > >> heard. I > > > > > > > > >> > > > > > disagree with turning Flink into an "AI-Native" > > > > engine. > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Regarding your "Data processing is entering the > AI > > > > era, > > > > > > and > > > > > > > > Flink > > > > > > > > >> > > > needs to > > > > > > > > >> > > > > > evolve from a traditional BI compute engine > into a > > > > data > > > > > > > engine > > > > > > > > >> that > > > > > > > > >> > > > > > natively supports AI workloads" claim: > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > - How exactly do you define "AI"? I don't > believe > > > > there > > > > > > is a > > > > > > > > >> > standard > > > > > > > > >> > > > > > definition. For example, Machine Learning have > been > > > > > around > > > > > > > for > > > > > > > > >> more > > > > > > > > >> > > > than a > > > > > > > > >> > > > > > decade, but there were no proposals (or need, > in my > > > > > > opinion) > > > > > > > > to > > > > > > > > >> > turn > > > > > > > > >> > > > Flink > > > > > > > > >> > > > > > into an "ML-Native" engine. Flink, in its > current > > > > state, > > > > > > has > > > > > > > > >> > > > > > been successfully used in many systems alongside > > > > > dedicated > > > > > > > ML > > > > > > > > >> > > > technologies, > > > > > > > > >> > > > > > like feature stores. Based on the context of > your > > > > > > proposal, > > > > > > > it > > > > > > > > >> > looks > > > > > > > > >> > > > like > > > > > > > > >> > > > > > you mostly mean LLMs, so could you be specific > about > > > > the > > > > > > > > >> language? > > > > > > > > >> > > > > > - I wouldn't call Flink "a traditional BI > compute > > > > > engine". > > > > > > > > Flink > > > > > > > > >> > is a > > > > > > > > >> > > > > > general data processing technology which can be > used > > > > > for a > > > > > > > > >> variety > > > > > > > > >> > of > > > > > > > > >> > > > use > > > > > > > > >> > > > > > cases without any BI involvement. > > > > > > > > >> > > > > > - Do you have any proof that "Users' core > workloads > > > > are > > > > > > > > rapidly > > > > > > > > >> > > > evolving" > > > > > > > > >> > > > > > and that they require your proposed changes? > Case > > > > > studies, > > > > > > > > user > > > > > > > > >> > > > surveys, or > > > > > > > > >> > > > > > submitted issues about the lack of support? Big > > > > changes > > > > > > like > > > > > > > > that > > > > > > > > >> > > > require > > > > > > > > >> > > > > > extensive validation. > > > > > > > > >> > > > > > - And even if there is a real need to adopt some > > > > > > LLM-driven > > > > > > > > >> > changes, > > > > > > > > >> > > > why > > > > > > > > >> > > > > > now? The LLM-related tooling has been changing > so > > > > > rapidly, > > > > > > > and > > > > > > > > >> it's > > > > > > > > >> > > > hard to > > > > > > > > >> > > > > > predict what will be needed tomorrow. Why does > it > > > make > > > > > > sense > > > > > > > > to > > > > > > > > >> > > > introduce > > > > > > > > >> > > > > > changes now, and not wait for more > standardization > > > and > > > > > > > > >> > consolidation? > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > To summarize, I think there are a lot of great > ideas > > > > in > > > > > > the > > > > > > > > >> > proposal, > > > > > > > > >> > > > but > > > > > > > > >> > > > > > in my mind, they need to be addressed as > tactical, > > > > > focused > > > > > > > > >> > changes, not > > > > > > > > >> > > > > > under the "AI-Native" umbrella. > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > I also wanted to address a few more specific > points: > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > - RpcOperator, why does it need to be managed by > > > > Flink? > > > > > I > > > > > > > see > > > > > > > > >> > > > absolutely no > > > > > > > > >> > > > > > need to introduce the additional complexity of > > > > > > orchestrating > > > > > > > > >> > standalone > > > > > > > > >> > > > > > components into the core Flink engine. I can > > > imagine a > > > > > > > > separate > > > > > > > > >> > > > sub-project > > > > > > > > >> > > > > > for an RpcOperator, which could potentially be > > > managed > > > > > by > > > > > > > the > > > > > > > > >> > > > Kubernetes > > > > > > > > >> > > > > > Operator. > > > > > > > > >> > > > > > - You make the case for the vectorized batch > > > > processing, > > > > > > but > > > > > > > > only > > > > > > > > >> > on > > > > > > > > >> > > > the > > > > > > > > >> > > > > > Python side. Why stop there? Native columnar > > > > vectorized > > > > > > > > execution > > > > > > > > >> > will > > > > > > > > >> > > > > > require end-to-end changes, including > connectors, > > > data > > > > > > > format > > > > > > > > >> > support, > > > > > > > > >> > > > Type > > > > > > > > >> > > > > > system support, runtime changes, etc. It seems > > > logical > > > > > to > > > > > > me > > > > > > > > to > > > > > > > > >> > support > > > > > > > > >> > > > > > this execution mode for Java and SQL as well. > > > > > > > > >> > > > > > - Supporting many more data types natively > (images, > > > > > video, > > > > > > > > audio, > > > > > > > > >> > > > tensors) > > > > > > > > >> > > > > > will make connector serializers and > deserializers > > > > > (SerDes) > > > > > > > > much > > > > > > > > >> > more > > > > > > > > >> > > > > > challenging to implement. Even today, many > SerDes in > > > > > > > > officially > > > > > > > > >> > > > supported > > > > > > > > >> > > > > > connectors don't fully implement types like > arrays > > > and > > > > > > > > structs. > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Thank you. > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > On Wed, Apr 29, 2026 at 1:18 AM Guowei Ma < > > > > > > > > [email protected]> > > > > > > > > >> > > > wrote: > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > Hi Z > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > Thanks for the kind words and the thoughtful > > > > > questions. > > > > > > > Let > > > > > > > > me > > > > > > > > >> > take > > > > > > > > >> > > > them > > > > > > > > >> > > > > > > one by one. > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > 1. Throughput and latency targets > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > To be honest, I don't have concrete numbers to > > > share > > > > > > yet. > > > > > > > > What > > > > > > > > >> I > > > > > > > > >> > can > > > > > > > > >> > > > say > > > > > > > > >> > > > > > is > > > > > > > > >> > > > > > > that our internal testing has already surfaced > > > > several > > > > > > > > >> directions > > > > > > > > >> > > > where > > > > > > > > >> > > > > > > Flink can be improved, and at the same time we > > > want > > > > to > > > > > > > fully > > > > > > > > >> > leverage > > > > > > > > >> > > > > > > Flink's existing streaming shuffle > capabilities. > > > As > > > > > the > > > > > > > > >> > multimodal > > > > > > > > >> > > > > > operator > > > > > > > > >> > > > > > > library matures, we'll progressively publish > > > > benchmark > > > > > > > > results. > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > 2. Built-in operators > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > You're absolutely right. From what I've seen, > our > > > > > > internal > > > > > > > > >> users > > > > > > > > >> > > > already > > > > > > > > >> > > > > > > rely on a fairly large set of multimodal > > > operators — > > > > > > > > >> potentially > > > > > > > > >> > > > 100+. > > > > > > > > >> > > > > > The > > > > > > > > >> > > > > > > exact set the community should provide is best > > > > > discussed > > > > > > > in > > > > > > > > >> > FLIP-XXX: > > > > > > > > >> > > > > > > Built-in Multimodal Operators and AI > Functions, > > > and > > > > > > > > >> contributions > > > > > > > > >> > > > from > > > > > > > > >> > > > > > the > > > > > > > > >> > > > > > > community are very welcome there. > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > 3. Plan for the 11 sub-FLIPs > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > The sequencing follows the layering in the > > > umbrella: > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > - Layer 1 (Core Primitives) should be > discussed > > > > and > > > > > > > > aligned > > > > > > > > >> > first, > > > > > > > > >> > > > > > since > > > > > > > > >> > > > > > > the second and third layers build on it. > > > > > > > > >> > > > > > > - Layer 2 (API + compilation + single-node > > > > > execution) > > > > > > > > starts > > > > > > > > >> > with > > > > > > > > >> > > > > > > getting the API discussion right — the > Python > > > > API, > > > > > > how > > > > > > > > UDFs > > > > > > > > >> > > > declare > > > > > > > > >> > > > > > > resources, etc. — after which the > single-node > > > > > > execution > > > > > > > > work > > > > > > > > >> > can > > > > > > > > >> > > > build > > > > > > > > >> > > > > > > on > > > > > > > > >> > > > > > > top. > > > > > > > > >> > > > > > > - Layer 3 (distributed scheduling and > > > > > checkpointing) > > > > > > > can > > > > > > > > >> > largely > > > > > > > > >> > > > > > proceed > > > > > > > > >> > > > > > > independently in parallel. > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > So while each sub-FLIP is indeed a substantial > > > piece > > > > > of > > > > > > > > work, > > > > > > > > >> > most of > > > > > > > > >> > > > > > them > > > > > > > > >> > > > > > > can be advanced in parallel by different > > > > contributors > > > > > > once > > > > > > > > the > > > > > > > > >> > Layer > > > > > > > > >> > > > 1 > > > > > > > > >> > > > > > > primitives are settled. > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > 4. GPU scheduling roadmap > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > Could you expand a bit on which aspect of GPU > > > > > scheduling > > > > > > > you > > > > > > > > >> > have in > > > > > > > > >> > > > mind > > > > > > > > >> > > > > > > as the complex one? "GPU scheduling" covers a > > > fairly > > > > > > wide > > > > > > > > >> surface > > > > > > > > >> > > > area > > > > > > > > >> > > > > > > (resource declaration, operator-level > deployment, > > > > > > elastic > > > > > > > > >> > scaling, > > > > > > > > >> > > > > > > heterogeneous GPU types, fine-grained > > > partitioning, > > > > > > etc.), > > > > > > > > and > > > > > > > > >> > the > > > > > > > > >> > > > answer > > > > > > > > >> > > > > > > differs quite a bit depending on which > dimension > > > > we're > > > > > > > > >> > discussing. > > > > > > > > >> > > > Once I > > > > > > > > >> > > > > > > understand your specific concern I can give a > more > > > > > > useful > > > > > > > > >> > response. > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > Thanks again for the support — looking > forward to > > > > the > > > > > > > > continued > > > > > > > > >> > > > > > discussion. > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > Best, > > > > > > > > >> > > > > > > Guowei > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > On Tue, Apr 28, 2026 at 4:34 PM zl z < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > Hey Guowei, > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > Thanks for the proposal, and I think this is > > > very > > > > > > > > valuable. I > > > > > > > > >> > have > > > > > > > > >> > > > some > > > > > > > > >> > > > > > > > question about it: > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > 1. What are our expected throughput and > latency > > > > > > targets? > > > > > > > > Do > > > > > > > > >> we > > > > > > > > >> > > > have any > > > > > > > > >> > > > > > > > forward-looking tests for this? > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > 2. AI involves a very large number of > operators. > > > > > > Besides > > > > > > > > >> > allowing > > > > > > > > >> > > > users > > > > > > > > >> > > > > > > to > > > > > > > > >> > > > > > > > use them through UDFs, will we also provide > > > > commonly > > > > > > > used > > > > > > > > >> > built-in > > > > > > > > >> > > > > > > > operators? > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > 3. Each of the 11 sub-FLIPs is a major > project > > > > > > > involving a > > > > > > > > >> > > > significant > > > > > > > > >> > > > > > > > amount of changes. What is our plan for > this? > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > 4. GPU scheduling is extremely complex. > What is > > > > our > > > > > > > > current > > > > > > > > >> > > > roadmap for > > > > > > > > >> > > > > > > > this? > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > This is a very high-quality and exciting > > > proposal. > > > > > > > Making > > > > > > > > >> > Flink an > > > > > > > > >> > > > > > > > AI-native data processing engine will make > it > > > far > > > > > more > > > > > > > > >> > valuable in > > > > > > > > >> > > > the > > > > > > > > >> > > > > > AI > > > > > > > > >> > > > > > > > era. Look forward to seeing it land and > come to > > > > > > fruition > > > > > > > > >> soon. > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > Robert Metzger <[email protected]> > > > > 于2026年4月28日周二 > > > > > > > > 14:38写道： > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > Hey Guowei, > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > Thanks for the proposal. I just took a > brief > > > > look, > > > > > > > here > > > > > > > > are > > > > > > > > >> > some > > > > > > > > >> > > > high > > > > > > > > >> > > > > > > > level > > > > > > > > >> > > > > > > > > questions: > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > Regarding the RPC Operator: What is the > > > > difference > > > > > > to > > > > > > > > the > > > > > > > > >> > async > > > > > > > > >> > > > io > > > > > > > > >> > > > > > > > operator > > > > > > > > >> > > > > > > > > we have already? > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > "Connector API for Multimodal Data > > > Source/Sink": > > > > > Why > > > > > > > do > > > > > > > > we > > > > > > > > >> > need > > > > > > > > >> > > > to > > > > > > > > >> > > > > > > touch > > > > > > > > >> > > > > > > > > the connector API for supporting > multimodal > > > > data? > > > > > > > Isn't > > > > > > > > >> this > > > > > > > > >> > > > more of > > > > > > > > >> > > > > > a > > > > > > > > >> > > > > > > > > formats concern? > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > "Non-Disruptive Scaling for CPU > Operators": > > > How > > > > do > > > > > > you > > > > > > > > want > > > > > > > > >> > to > > > > > > > > >> > > > > > > guarantee > > > > > > > > >> > > > > > > > > exactly-once on that kind of scaling? > E.g. you > > > > > need > > > > > > to > > > > > > > > >> > somehow > > > > > > > > >> > > > make a > > > > > > > > >> > > > > > > > > handover between the old and new new > pipeline > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > Overall, I find the proposal has some > things > > > > which > > > > > > > seem > > > > > > > > >> > related > > > > > > > > >> > > > to > > > > > > > > >> > > > > > > making > > > > > > > > >> > > > > > > > > Flink more AI native, but other changes > seem > > > > > > > orthogonal > > > > > > > > to > > > > > > > > >> > that. > > > > > > > > >> > > > For > > > > > > > > >> > > > > > > > > example the checkpoint or scaling changes > are > > > > > > actually > > > > > > > > >> > unrelated > > > > > > > > >> > > > to > > > > > > > > >> > > > > > AI, > > > > > > > > >> > > > > > > > and > > > > > > > > >> > > > > > > > > just engine improvements. > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > On Tue, Apr 28, 2026 at 5:48 AM Guowei Ma > < > > > > > > > > >> > [email protected]> > > > > > > > > >> > > > > > > wrote: > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > Hi everyone, > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > I'd like to start a discussion on an > > > umbrella > > > > > > > FLIP[1] > > > > > > > > >> that > > > > > > > > >> > lays > > > > > > > > >> > > > > > out a > > > > > > > > >> > > > > > > > > > direction for evolving Flink into a data > > > > engine > > > > > > that > > > > > > > > >> > natively > > > > > > > > >> > > > > > > supports > > > > > > > > >> > > > > > > > AI > > > > > > > > >> > > > > > > > > > workloads. > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > The short version: user workloads are > > > shifting > > > > > > from > > > > > > > BI > > > > > > > > >> > > > analytics to > > > > > > > > >> > > > > > > > > > multimodal data processing centered on > model > > > > > > > > inference, > > > > > > > > >> and > > > > > > > > >> > > > this > > > > > > > > >> > > > > > > > triggers > > > > > > > > >> > > > > > > > > > cascading changes across the stack — > > > > multimodal > > > > > > data > > > > > > > > >> > flowing > > > > > > > > >> > > > > > through > > > > > > > > >> > > > > > > > > > pipelines, heterogeneous CPU/GPU > resources, > > > > > > > vectorized > > > > > > > > >> > > > execution, > > > > > > > > >> > > > > > and > > > > > > > > >> > > > > > > > > > inference tasks that run for seconds to > > > > minutes > > > > > on > > > > > > > > Spot > > > > > > > > >> > > > instances. > > > > > > > > >> > > > > > > The > > > > > > > > >> > > > > > > > > > proposal sketches an evolution along > five > > > > > > directions > > > > > > > > >> > > > (development > > > > > > > > >> > > > > > > > > paradigm, > > > > > > > > >> > > > > > > > > > data model, heterogeneous resources, > > > execution > > > > > > > engine, > > > > > > > > >> > fault > > > > > > > > >> > > > > > > > tolerance), > > > > > > > > >> > > > > > > > > > decomposed into 11 sub-FLIPs organized > into > > > > > three > > > > > > > > layers: > > > > > > > > >> > core > > > > > > > > >> > > > > > > runtime > > > > > > > > >> > > > > > > > > > primitives, AI workload expression and > > > > > execution, > > > > > > > and > > > > > > > > >> > > > > > > production-grade > > > > > > > > >> > > > > > > > > > operational guarantees. Most sub-FLIPs > have > > > no > > > > > > hard > > > > > > > > >> > > > dependencies on > > > > > > > > >> > > > > > > > each > > > > > > > > >> > > > > > > > > > other and can be advanced in parallel. > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > A note on scope, since it's an umbrella: > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > - In scope here: whether the evolution > > > > > directions > > > > > > > are > > > > > > > > >> > > > reasonable, > > > > > > > > >> > > > > > > > whether > > > > > > > > >> > > > > > > > > > each sub-FLIP's motivation and proposed > > > > approach > > > > > > are > > > > > > > > >> > > > well-founded, > > > > > > > > >> > > > > > > and > > > > > > > > >> > > > > > > > > > whether the boundaries and dependencies > > > > between > > > > > > > > sub-FLIPs > > > > > > > > >> > are > > > > > > > > >> > > > > > clear. > > > > > > > > >> > > > > > > > > > - Out of scope here: detailed designs, > API > > > > > > > specifics, > > > > > > > > and > > > > > > > > >> > > > > > > > implementation > > > > > > > > >> > > > > > > > > > plans of individual sub-FLIPs — those > will > > > go > > > > > > > through > > > > > > > > >> > their own > > > > > > > > >> > > > > > > FLIPs. > > > > > > > > >> > > > > > > > > > - Consensus criteria: agreement on the > > > overall > > > > > > > > direction > > > > > > > > >> is > > > > > > > > >> > > > > > > sufficient > > > > > > > > >> > > > > > > > > for > > > > > > > > >> > > > > > > > > > the umbrella to pass; passing it does > not > > > lock > > > > > in > > > > > > > any > > > > > > > > >> > > > sub-FLIP's > > > > > > > > >> > > > > > > > design — > > > > > > > > >> > > > > > > > > > sub-FLIPs may still be adjusted, > deferred, > > > or > > > > > > > > withdrawn > > > > > > > > >> as > > > > > > > > >> > they > > > > > > > > >> > > > > > > > progress. > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > All proposed changes are incremental — > no > > > > > existing > > > > > > > > API or > > > > > > > > >> > > > behavior > > > > > > > > >> > > > > > is > > > > > > > > >> > > > > > > > > > removed or altered. Compatibility > details > > > are > > > > > > > covered > > > > > > > > at > > > > > > > > >> > the > > > > > > > > >> > > > end of > > > > > > > > >> > > > > > > the > > > > > > > > >> > > > > > > > > > document. > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > Looking forward to your feedback on the > > > > overall > > > > > > > > direction > > > > > > > > >> > and > > > > > > > > >> > > > the > > > > > > > > >> > > > > > > > > layering. > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > [1] > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=421957275 > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > Thanks, > > > > > > > > >> > > > > > > > > > Guowei > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
