Hey All I have opened the initial PR for the StateCatalog implementation: https://github.com/apache/flink/pull/28837
I know that this is an unusually large PR, but I think it represents a somewhat complete key feature set that we are covering here instead of breaking it up into functionally unusable bits. Once we have agreed and merged this, I will follow up with opening PRs for the windowed and non-keyed state support. I have completed all the implementation but including those in the initial PR would be totally unreviewable. Cheers Gyula On Fri, Jul 10, 2026 at 10:50 PM Weiqing Yang <[email protected]> wrote: > Thanks Gyula. Agreed, decoupling name storage from the schema-evolution > enable/disable logic is the right move. > > > +1 on splitting the unconditional name storage into its own JIRA and doing > it independently. I've filed FLINK-40120 > <https://issues.apache.org/jira/browse/FLINK-40120> for it. The change is > self-contained and both the catalog and schema evolution need it, so it's > better as a shared capability than owned by either FLIP. Happy to take a > first cut. I've left the exact scope for us to confirm on the ticket (e.g., > whether the catalog needs names only from RowData state or something more > general) so we can keep this thread focused on the FLIP-599 proposal. > > > On not baking config into the snapshot: agreed. I had floated a toggle flag > in the snapshot, but you're right that the opt-in shouldn't be persisted; > it should be decided at restore. FLIP-527 doesn't persist it today: it > reads the option once at serializer construction (which runs at restore), > and that read determines whether field names get written. The compatibility > check then keys off name-presence, since it can't read the job config > directly. Making names unconditional removes that implicit signal, so we'll > need another way to carry the restore-time opt-in to the check. I'll > explore the cleanest option (an in-memory flag on the serializer, or > passing the job config into the check as you suggest) and follow up. > > > Glad to pair on the JIRA! > > > Cheers, > > Weiqing > > > On Fri, Jul 10, 2026 at 5:42 AM Gyula Fóra <[email protected]> wrote: > > > Hi Devs! > > > > Thank you all for the thoughtful comments, I have tried to incorporate > most > > of the larger design additions and changes to the FLIP document. > > I have also extended with a bit more details and included a high level > > design to represent window operator state (along the same core principles > > as we had the keyed states) > > > > Overall I feel a strong general consensus about the feature and key > design > > principles. Given the scope and somewhat experimental nature of it, I > > expect to make some significant iterations over the coming months to > > converge to a good solution but we definitely have a really solid > starting > > point. > > > > Unless there is further significant feedback, I will start the vote at > the > > beginning of next week. > > > > Cheers > > Gyula > > > > > > On Fri, Jul 10, 2026 at 6:58 AM <[email protected]> wrote: > > > > > Hi Weiqing! > > > > > > Thanks for flagging this, I wasn’t aware of the other discussion > details! > > > > > > First of all I agree that we have to decouple the storage of the names > in > > > the snapshot from the schema evolution enabled/disabled logic . > > > > > > Given the obvious need for names by multiple flips and the > > straightforward > > > change , I think we can actually separate this into a jira and > implement > > it > > > already independently. > > > > > > A small note on the schema evolution and maybe I need to review the > > design > > > and comment on the other FLIP, but it would be best to avoid baking > > config > > > logic into the snapshot itself and maybe providing access to the job > > config > > > during the state restore is a cleaner overall approach? > > > > > > Cheers > > > Gyula > > > > > > Sent from my iPhone > > > > > > > On 9 Jul 2026, at 22:23, Weiqing Yang <[email protected]> > > wrote: > > > > > > > > Hi Gyula, > > > > > > > > Thanks for putting this FLIP together. I'm very supportive of the > goals > > > of > > > > FLIP-599. A robust State Catalog is a fantastic step forward for > state > > > > observability and management. > > > > > > > > You and Shengkai landed on bumping RowDataSerializerSnapshot to > include > > > > field names to support the catalog's schema inference. I agree with > the > > > > direction, and FLIP-527 (State Schema Evolution for RowData) [1], > which > > > I'm > > > > driving, proposes the exact same version bump. I wanted to flag a > > design > > > > overlap early, while both FLIPs are still in DISCUSS, so we can align > > our > > > > approaches. > > > > > > > > When implementing FLIP-527, one option is that the presence of these > > > field > > > > names acts as the explicit opt-in signal for schema evolution. The > > > feature > > > > is off by default, the serializer is built with names only when it's > > > > enabled, and the compatibility/migration logic keys off whether those > > > names > > > > are present. > > > > > > > > The catch with that approach is a coupling worth designing around. > For > > > the > > > > catalog to recover real names on jobs that never enabled schema > > > evolution, > > > > those names would need to be written regardless of the evolution > > opt-in. > > > > But if name presence is what signals the opt-in, then writing names > by > > > > default would flip the gate on for jobs that never asked, and trigger > > > > migration logic they didn't opt into. Since we're still discussing > the > > > > high-level architecture for both FLIPs, this looks very solvable. We > > just > > > > need to decouple the "names persisted" metadata from the "evolution > > > > permitted" signal, perhaps by introducing an explicit toggle flag in > > the > > > > snapshot format rather than relying solely on name presence. That's > the > > > > direction I'd favor for FLIP-527 as well. > > > > > > > > Getting this right while both FLIPs are in DISCUSS means whichever > > lands > > > > first can establish a format that serves both the catalog's metadata > > > needs > > > > and the migration's safety requirements. Looking forward to your > > thoughts > > > > on how best to converge these designs. > > > > > > > > Best, > > > > Weiqing > > > > > > > > [1] > > > > > > > > > > https://cwiki.apache.org/confluence/spaces/FLINK/pages/353601981/FLIP-527+State+Schema+Evolution+for+RowData > > > > > > > > > > > >> On Wed, Jul 8, 2026 at 6:41 AM Jingsong Li <[email protected]> > > > wrote: > > > >> > > > >> Hi Gyula, > > > >> > > > >> Thanks for your response. > > > >> > > > >> Sounds good to me. > > > >> > > > >> Best, > > > >> Jingsong > > > >> > > > >>> On Wed, Jul 8, 2026 at 5:38 PM Gyula Fóra <[email protected]> > > > wrote: > > > >>> > > > >>> Hi Jingsong! > > > >>> > > > >>> Thanks for the feedback. > > > >>> I totally agree with you that the current design is going to be > most > > > >> useful > > > >>> for datastream / user defined operators and may surface some > > "strange" > > > or > > > >>> slightly unexpected internals to the users for SQL jobs and SQL > > > >> operators. > > > >>> > > > >>> Big +1 for the idea that we should add operator specific views > later > > to > > > >>> expose some important sql operator internals in a more human > readable > > > >>> format. And I also completely agree that this is out of scope for > the > > > >>> initial V1 version.Based on the interest I see on this FLIP I think > > > there > > > >>> will be a demand for a lot of similar extension capabilities once > we > > > get > > > >>> the initial physical/raw state view version out. > > > >>> > > > >>> Regarding the documentation requirements for transparency, this is > a > > > very > > > >>> important point and I couldn't agree more. In addition I think we > > will > > > >> have > > > >>> to clearly document and treat the contents of the state catalog as > > > fully > > > >>> experimental initially. With the great variety of internal > > > >> representations > > > >>> and use-cases surrounding Flink states, it's impossible to get this > > > right > > > >>> on the first try and our goal should be to iterate on this over a > few > > > >> flink > > > >>> minor versions. > > > >>> > > > >>> So specifically: > > > >>> 1. The physical state representation of SQL and other built in > > > operators > > > >>> is a Flink internal and not part of the public api. Therefore the > > > >> structure > > > >>> / schema of these tables are subject to change without notice > across > > > >> Flink > > > >>> versions. > > > >>> 2. The StateCatalog itself including configuration, operator/state > to > > > >>> table mapping, metadata views, etc. are all initially experimental > > and > > > >>> provide no backward compatibility guarantees and can change without > > > >> notice > > > >>> initially. > > > >>> > > > >>> I think 1. will stay like this for the foreseeable future , we do > not > > > >> want > > > >>> to make internal implementations as part of the public api. We > should > > > aim > > > >>> to stabilize 2. and make it part of the public api eventually, and > > the > > > >>> views on the internal operators could hopefully become part of 2 as > > > well. > > > >>> > > > >>> Cheers > > > >>> Gyula > > > >>> > > > >>> On Wed, Jul 8, 2026 at 2:55 AM Jingsong Li <[email protected] > > > > > >> wrote: > > > >>> > > > >>>> Hi Gyula and all, > > > >>>> > > > >>>> Thanks for driving this FLIP. I like the overall direction and > think > > > >> making > > > >>>> savepoint/checkpoint state discoverable and queryable from SQL > would > > > be > > > >>>> very > > > >>>> useful. > > > >>>> > > > >>>> I have one concern/question around how the catalog presents state > > from > > > >> SQL > > > >>>> operators. > > > >>>> > > > >>>> For many SQL operators, the internal state layout can be quite > > > >> different > > > >>>> from > > > >>>> the logical SQL model. For example, streaming joins, window > > > >> aggregations, > > > >>>> deduplication, TopN/rank, and other optimized operators may > maintain > > > >>>> multiple > > > >>>> internal states, timers, namespaces, accumulators, or > > > >> optimization-specific > > > >>>> auxiliary structures. If these are exposed directly as SQL tables, > > the > > > >>>> result > > > >>>> may be difficult for users to understand. It may also accidentally > > > >> make an > > > >>>> internal state layout look like a stable user-facing contract, > even > > > >> though > > > >>>> it > > > >>>> can change with planner/runtime optimizations or Flink versions. > > > >>>> > > > >>>> Would it make sense to explicitly distinguish two levels in the > > FLIP? > > > >>>> > > > >>>> 1. Physical/raw state views > > > >>>> > > > >>>> These are generic views derived from checkpoint/savepoint metadata > > and > > > >>>> serializer snapshots. They expose the actual operator UID, state > > name, > > > >>>> state > > > >>>> type, key/namespace/value schema, serializer information, etc. I > > think > > > >>>> this is > > > >>>> a very reasonable scope for the first version, especially for > > > >> debugging, > > > >>>> observability, migration, and advanced operational use cases. > > > >>>> > > > >>>> However, it would be good to document that these tables represent > > > >> Flink's > > > >>>> physical state layout and should not be treated as stable logical > > SQL > > > >>>> tables. > > > >>>> > > > >>>> 2. Optional logical/operator-aware views > > > >>>> > > > >>>> For some common SQL/runtime operators, we could later add > > > >> operator-specific > > > >>>> views or descriptors that explain the state in terms of operator > > > >> semantics. > > > >>>> For example, a join operator could expose left-side rows, > right-side > > > >> rows, > > > >>>> and > > > >>>> timers in a more understandable way; a window aggregate could > expose > > > >> window > > > >>>> accumulators and timer/namespace metadata. > > > >>>> > > > >>>> I do not think this needs to be part of the initial > implementation, > > > but > > > >>>> making > > > >>>> the distinction explicit would help set the right expectations and > > > >> avoid > > > >>>> over-promising what automatic state-to-table mapping can provide. > > > >>>> > > > >>>> So my suggestion is that V1 focuses on the generic physical/raw > > state > > > >> view, > > > >>>> with clear metadata and documentation, while leaving > > > >> logical/operator-aware > > > >>>> views as a possible extension. > > > >>>> > > > >>>> Best, > > > >>>> Jingsong > > > >>>> > > > >>>> On Wed, Jul 8, 2026 at 3:39 AM Roman Khachatryan < > [email protected]> > > > >> wrote: > > > >>>>> > > > >>>>>> 1. The catalog built on the regular state processor api (and > > > >> therefore > > > >>>>>> flink state restore) capabilities has limited scope to detect > > > >> exactly > > > >>>> what > > > >>>>>> happens when a state is no longer there. This will probably lead > > to > > > >>>> read > > > >>>>>> errors/not found exceptions etc, some of which happens in code > > > >> that is > > > >>>> a > > > >>>>>> bit tricky to control this way. Let's see how well this works in > > > >>>> practice > > > >>>>>> and error handling can always be improved in general. This is > not > > > >> part > > > >>>> of > > > >>>>>> the catalog design itself. > > > >>>>> > > > >>>>> Makes sense. I think this can also be an extension. > > > >>>>> To clarify the ownership/pinning follow-up idea: it could use FS > > > >>>> mechanisms > > > >>>>> rather than HA (ZK/etcd). For example, a separate file > > > >>>>> with the lowest checkpoint that Flink should keep, protected by > CAS > > > >> and > > > >>>>> limited to a specified TTL. > > > >>>>> > > > >>>>>> 2. The proposal in it's current form includes a global state > > > >> metadata > > > >>>> view > > > >>>>>> based on the existing metadata information ([1]) and based on > > > >>>> Shengkai's > > > >>>>>> feedback a per operator granular metadata table/view that would > > > >> expose > > > >>>>>> information of individual states. I don't see where file level > > > >>>> information > > > >>>>>> fits into this but if you have a good way / idea how to > represent > > > >> this > > > >>>> as > > > >>>>> a > > > >>>>>> table this can definitely be a future extension/addition > > > >>>>> > > > >>>>> I was thinking about something like: > > > >>>>> > > > >>>>> USE `00000000000000000000/chk-42`; > > > >>>>> SELECT * > > > >>>>> FROM state_handles; > > > >>>>> > > > >>>>> -- id type parent path > > > >>>>> size timestamp > > operator/state/subtask_index > > > >>>>> local_path key_range > > > >>>>> -- 0 FileStateHandle - > > > >>>> s3://.../_metadata > > > >>>>> 0.8Kb 2026-07-02T22:56:30 - > > > >>>>> - - > > > >>>>> -- 1 IncrementalRemoteKeyedStateHandle 0 > > > >>>>> - 2026-07-02T22:56:14 > > > >>>>> uid_transaction_aggregator_keyed/users#0 - 0 .. > > 127 > > > >>>>> -- 2 FileStateHandle 1 > > > >>>>> s3://.../xxxx-xxxx... 5.4Mb 2026-07-02T22:56:12 - > > > >>>>> 000034.SST - > > > >>>>> -- 3 FileStateHandle 1 > > > >>>>> s3://.../yyyy-yyyy... 872Kb 2026-07-02T22:56:12 - > > > >>>>> 000035.SST - > > > >>>>> -- 4 IncrementalRemoteKeyedStateHandle 0 > > > >>>>> - 2026-07-02T22:56:24 > > > >>>>> uid_transaction_aggregator_keyed/users#1 - 128 > .. > > > >> 255 > > > >>>>> -- 5 ByteStreamStateHandle 4 > > > >>>>> 2Kb 2026-07-02T22:56:24 - > > > >>>>> 000001.SST - > > > >>>>> > > > >>>>> The idea is to represent CompletedCheckpoint as a DAG so that it > > maps > > > >>>>> directly to the layout of the object in-memory. > > > >>>>> > > > >>>>>> 3. Did not think about this but if this becomes a requirement we > > > >> could > > > >>>> add > > > >>>>>> a flag to enable metadata only in the catalog. > > > >>>>> > > > >>>>> For our use-case (multi-tenant cloud environment), separate > access > > > >> models > > > >>>>> for > > > >>>>> data and metadata are very likely a must have because of the > > security > > > >>>>> concerns: > > > >>>>> - internally, the operators should have access to metadata, but > not > > > >> to > > > >>>> the > > > >>>>> customer data > > > >>>>> - externally, the users should have access to their data but not > to > > > >> the > > > >>>>> metadata > > > >>>>> > > > >>>>> 5, 6. Thanks! :) > > > >>>>> > > > >>>>> 7. Yes, this should be available since metadata V4. > > > >>>>> > > > >>>>> Regards, > > > >>>>> Roman > > > >>>>> > > > >>>>> > > > >>>>> On Tue, Jul 7, 2026 at 9:56 AM Gyula Fóra <[email protected]> > > > >> wrote: > > > >>>>> > > > >>>>>> Hey Shengkai! > > > >>>>>> > > > >>>>>> Thanks for the questions, you hit on some very good practical > > > >> points. > > > >>>> Let > > > >>>>>> me provide my answers below, in the meantime I have already > > > >> updated the > > > >>>>>> FLIP to include some of your suggestions :) > > > >>>>>> > > > >>>>>> 1. How would schema inference work for RowDataSerializer? > > > >>>>>> > > > >>>>>> That's a good observation, I did not notice this. Probably the > > > >> simplest > > > >>>>>> solution would be to introduce a new version in the > > > >>>>>> RowDataSerializerSnapshot and include the names for this > use-case. > > > >>>>>> This would not really impact checkpointing times/performance but > > > >> would > > > >>>>>> allow a straightforward mapping for sql states. > > > >>>>>> > > > >>>>>> If we feel that this is too much internal change, then we could > > > >> also > > > >>>> keep > > > >>>>>> it as is for now using simply f0, f1... > > > >>>>>> > > > >>>>>> 2. Is one keyed-state table per operator the right abstraction? > > > >>>>>> > > > >>>>>> This is a very good point and something that has bothered me as > > > >> well > > > >>>> from a > > > >>>>>> design perspective. There is no single good abstraction here > > > >> because > > > >>>> there > > > >>>>>> are completely different use cases. > > > >>>>>> When you just want to look into the state for a single / > multiple > > > >> keys > > > >>>> and > > > >>>>>> you mostly have simple value list states, the single table > > > >>>> representation > > > >>>>>> is superior from both query syntax and performance perspective. > It > > > >>>> avoids > > > >>>>>> JOINS and maps to the simple mental model that for a certain key > > > >> you > > > >>>> have > > > >>>>>> state x,y,z. Due to this straightforward mental model I still > > think > > > >>>> this is > > > >>>>>> the good default representation. With projection pushdown, it's > > > >> easy to > > > >>>>>> select one/several specific columns without reading / touching > any > > > >>>> other. > > > >>>>>> > > > >>>>>> The big issue is however with large collection states that > simply > > > >>>> cannot be > > > >>>>>> represented within a single row. This happens very often and is > > > >> one of > > > >>>> the > > > >>>>>> main reasons someone would even use a list state (if they > > > >> understand > > > >>>> how > > > >>>>>> they work internally, but not all users do...). Large map, > window, > > > >> list > > > >>>>>> states won't work in the simple row model and are completely > > > >>>> impractical. > > > >>>>>> > > > >>>>>> Based on this, my recommendation would be to keep all keyed > states > > > >> in a > > > >>>>>> single table as per the original proposal (one column per keyed > > > >> state) > > > >>>> but > > > >>>>>> also add an extra table per list / map state with the flattened > > > >> schema. > > > >>>>>> So if the operator has a value and list state, then there would > be > > > >> 2 > > > >>>>>> tables. One with both states as columns (as per original > design) + > > > >> 1 > > > >>>>>> flattened list state table (key, index, value) or for map states > > > >> (key, > > > >>>>>> map_key, value). > > > >>>>>> > > > >>>>>> This way we cover both use cases naturally. I am also open to > > > >> making > > > >>>> this > > > >>>>>> configurable on the catalog level. > > > >>>>>> > > > >>>>>> I have added this to the FLIP > > > >>>>>> > > > >>>>>> 3. Could you clarify the assumption of "reading state without > user > > > >>>>>> classes"? > > > >>>>>> > > > >>>>>> Turns out from an implementation perspective it's not too bad > and > > > >>>> pojo/avro > > > >>>>>> state schemas can be inferred quite naturally for most cases. > > > >> However > > > >>>> if > > > >>>>>> the user indeed provides the user jar on the classpath then the > > > >> whole > > > >>>>>> schema resolution will become even simpler because then we do > not > > > >> need > > > >>>> any > > > >>>>>> custom inference. For our own use-cases and in general I would > not > > > >>>> like to > > > >>>>>> assume that user classes will be easily available or that a > > catalog > > > >>>> will > > > >>>>>> represent mostly a single application. On the contrary the way > We > > > >>>> intend to > > > >>>>>> use this, is definitely mostly without userjars and to represent > > > >>>> multiple > > > >>>>>> applications at the same time. > > > >>>>>> > > > >>>>>> 4. Could StateCatalog expose more fine-grained metadata? > > > >>>>>> > > > >>>>>> I think this is a very good idea. I have updated the FLIP to > > > >> include an > > > >>>>>> operator level metadata table as well (one for each operator). I > > > >> would > > > >>>> love > > > >>>>>> to include everything that you suggested, I think the practical > > > >> limit > > > >>>> is > > > >>>>>> what kind of information is part of the checkpoint and what > isn't > > . > > > >>>> This > > > >>>>>> also ties to some questions Roman had about more detailed > > metadata. > > > >>>> Makes > > > >>>>>> sense > > > >>>>>> > > > >>>>>> Cheers > > > >>>>>> Gyula > > > >>>>>> > > > >>>>>> > > > >>>>>> On Tue, Jul 7, 2026 at 4:59 AM Shengkai Fang <[email protected] > > > > > >>>> wrote: > > > >>>>>> > > > >>>>>>> Hi Gyula, > > > >>>>>>> > > > >>>>>>> Thanks for the FLIP. I like the direction of making > > > >>>> savepoint/checkpoint > > > >>>>>>> state discoverable and queryable from SQL. I have a few > > > >> questions and > > > >>>>>>> concerns about the proposed abstraction. > > > >>>>>>> > > > >>>>>>> 1. How would schema inference work for RowDataSerializer? > > > >>>>>>> > > > >>>>>>> From the current RowDataSerializer snapshot, it looks like the > > > >>>> snapshot > > > >>>>>>> persists the LogicalType[] and nested serializer snapshots, so > > > >> the > > > >>>> field > > > >>>>>>> types can be restored. However, the top-level RowDataSerializer > > > >>>>>> constructed > > > >>>>>>> from a RowType seems to store only the child LogicalTypes, not > > > >> the > > > >>>>>> RowType > > > >>>>>>> field names. Would StateCatalog expose generated names such as > > > >>>> f0/f1, or > > > >>>>>> is > > > >>>>>>> there another source for recovering the original field names? > > > >>>>>>> > > > >>>>>>> 2. Is one keyed-state table per operator the right abstraction? > > > >>>>>>> > > > >>>>>>> I wonder whether one table per named keyed state would be a > > > >> better > > > >>>> base > > > >>>>>>> abstraction, with an optional operator-level wide view on top. > In > > > >>>>>>> particular, MapState can contain an unbounded or highly > variable > > > >>>> number > > > >>>>>> of > > > >>>>>>> entries per state key. Exposing it as a MAP<K,V> column may > > > >> require > > > >>>> fully > > > >>>>>>> deserializing the map for a key into heap. A normalized table > > > >> shape > > > >>>> such > > > >>>>>>> as: > > > >>>>>>> > > > >>>>>>> (state_key, map_key, map_value) > > > >>>>>>> > > > >>>>>>> seems more scalable and SQL-friendly for MapState. Similarly, > > > >>>>>>> ValueState/ListState/MapState have different natural table > > > >> shapes, so > > > >>>>>> tying > > > >>>>>>> the physical table boundary to the operator may be too coarse. > > > >>>>>>> > > > >>>>>>> 3. Could you clarify the assumption of "reading state without > > > >> user > > > >>>>>>> classes"? > > > >>>>>>> > > > >>>>>>> This is a very attractive goal, but it also seems to introduce > > > >>>>>> substantial > > > >>>>>>> complexity for POJOs, Avro SpecificRecord, subclasses, and > custom > > > >>>>>>> serializers. If StateCatalog is positioned as a > > > >>>>>> job-level/application-level > > > >>>>>>> catalog, would requiring the job jar or user artifacts be > > > >> acceptable > > > >>>> as a > > > >>>>>>> first step? That might simplify the design while still covering > > > >> many > > > >>>>>>> operational/debugging use cases. > > > >>>>>>> > > > >>>>>>> 4. Could StateCatalog expose more fine-grained metadata? > > > >>>>>>> > > > >>>>>>> For debugging state, it would be useful to expose state-level > > > >>>> metadata > > > >>>>>> such > > > >>>>>>> as state name, state type, serializer snapshot/serializer > class, > > > >> TTL > > > >>>>>>> configuration, namespace/window information where applicable, > > > >> backend > > > >>>>>> state > > > >>>>>>> type, and possibly whether a state can be read > > > >> lazily/streamingly. > > > >>>>>>> > > > >>>>>>> Best, > > > >>>>>>> Shengkai > > > >>>>>>> > > > >>>>>>> Roman Khachatryan <[email protected]> 于2026年7月7日周二 08:44写道： > > > >>>>>>> > > > >>>>>>>> Hi Gyula, > > > >>>>>>>> > > > >>>>>>>> Thanks for the proposal, this looks very useful! A few > > > >> questions > > > >>>> and > > > >>>>>>>> comments: > > > >>>>>>>> > > > >>>>>>>> 1. Following up on Han's question about checkpoint retention: > I > > > >>>>>>> understand > > > >>>>>>>> external coordination is out of scope for now, but could the > > > >>>> catalog at > > > >>>>>>>> least detect that a checkpoint was subsumed/deleted mid-query > > > >> and > > > >>>> fail > > > >>>>>>> with > > > >>>>>>>> a clear error, rather than a low-level file-not-found? And do > > > >> you > > > >>>> see > > > >>>>>>>> ownership/pinning as a possible follow-up FLIP once checkpoint > > > >>>> reading > > > >>>>>>>> picks up adoption? > > > >>>>>>>> 2. Does the proposal allow querying file-level metadata (file > > > >> size, > > > >>>>>>>> creation date, etc.)? This would be useful for debugging > > > >>>>>>> compaction-related > > > >>>>>>>> issues. > > > >>>>>>>> 3. If yes, could data and metadata queries have separate > access > > > >>>> modes? > > > >>>>>> In > > > >>>>>>>> many environments access to data is much stricter than access > > > >> to > > > >>>>>>> metadata, > > > >>>>>>>> so being able to grant metadata-only access to the catalog > > > >> would > > > >>>>>> broaden > > > >>>>>>>> where it can be deployed. > > > >>>>>>>> 4. Just to confirm: incremental checkpoints are expected to > > > >> work > > > >>>>>> through > > > >>>>>>>> the regular restore mechanisms, given sufficient retention? > > > >>>>>>>> 5. +1 on bringing non-keyed state into scope — a concrete use > > > >> case: > > > >>>>>>>> inspecting Kafka transaction state (currently stored in > > > >> non-keyed > > > >>>>>>> operator > > > >>>>>>>> state) would be very valuable for debugging EOS issues. > > > >>>>>>>> 6. Could you explain why timers are not supported? They live > in > > > >>>> keyed > > > >>>>>>> state > > > >>>>>>>> and the state processor API can read registered timers, so I'm > > > >>>>>> wondering > > > >>>>>>>> whether this is a fundamental limitation or just table-mapping > > > >>>> scope. > > > >>>>>>>> 7. Does the proposal allow querying checkpoint metadata (such > > > >> as > > > >>>>>>>> SharingFilesStrategy, isSavepoint, etc.)? This could be useful > > > >> for > > > >>>>>>>> debugging CLAIM mode issues. > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> Regards, > > > >>>>>>>> Roman > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> On Mon, Jul 6, 2026 at 1:02 PM Gyula Fóra < > > > >> [email protected]> > > > >>>>>> wrote: > > > >>>>>>>> > > > >>>>>>>>> Hi Zakelly! > > > >>>>>>>>> > > > >>>>>>>>> That's a good point and we have to ensure that it works. In > > > >>>> theory > > > >>>>>> SQL > > > >>>>>>>>> related states are relatively easy to cover and represent. > > > >> The > > > >>>>>> RowData > > > >>>>>>>>> state would be mapped directly to ROW<...> similar to other > > > >> pojo > > > >>>> key > > > >>>>>>>>> states. > > > >>>>>>>>> > > > >>>>>>>>> Cheers > > > >>>>>>>>> Gyula > > > >>>>>>>>> > > > >>>>>>>>> On Sun, Jul 5, 2026 at 4:03 PM Zakelly Lan < > > > >>>> [email protected]> > > > >>>>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>>> Hi Gyula, > > > >>>>>>>>>> > > > >>>>>>>>>> Thanks for driving this, it's a nice addition and I fully > > > >>>> support > > > >>>>>> it. > > > >>>>>>>> One > > > >>>>>>>>>> thing to make sure: > > > >>>>>>>>>> > > > >>>>>>>>>> For the state generated by some Flink SQL jobs, does the > > > >>>>>> StateCatalog > > > >>>>>>>>> infer > > > >>>>>>>>>> this internal `RowData` structure and expose it as a SQL > > > >>>> `ROW<...>` > > > >>>>>>>> type? > > > >>>>>>>>>> For example, a regular streaming join side may be stored > > > >> as a > > > >>>> state > > > >>>>>>>> such > > > >>>>>>>>> as > > > >>>>>>>>>> `left-records` / `right-records`, whose value or map > > > >> key/value > > > >>>>>>>> contains a > > > >>>>>>>>>> `RowData` for the original input row. > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> Best, > > > >>>>>>>>>> Zakelly > > > >>>>>>>>>> > > > >>>>>>>>>> On Fri, Jul 3, 2026 at 4:13 PM Dennis-Mircea Ciupitu < > > > >>>>>>>>>> [email protected]> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>>> Hi Gyula, > > > >>>>>>>>>>> > > > >>>>>>>>>>> Thanks for the detailed answers. This addresses my > > > >> questions > > > >>>> well > > > >>>>>>> and > > > >>>>>>>>> the > > > >>>>>>>>>>> direction sounds great. > > > >>>>>>>>>>> > > > >>>>>>>>>>> +1 (non-binding) from my side. > > > >>>>>>>>>>> > > > >>>>>>>>>>> Best regards, > > > >>>>>>>>>>> Dennis > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> On Thu, Jul 2, 2026 at 3:26 PM Gyula Fóra < > > > >>>> [email protected]> > > > >>>>>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>>> Hi Dennis! > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Thank you for the questions. Much recent work in the > > > >> state > > > >>>>>>>> connector > > > >>>>>>>>>> api > > > >>>>>>>>>>>> has been done basically towards this type of nice > > > >>>> cataloging > > > >>>>>> and > > > >>>>>>>>>> flexible > > > >>>>>>>>>>>> access. There are a few holes and things that have to > > > >> be > > > >>>>>> changed, > > > >>>>>>>> not > > > >>>>>>>>>>>> everything is enumerated in the FLIP but we have to > > > >> have an > > > >>>>>> open > > > >>>>>>>> mind > > > >>>>>>>>>> and > > > >>>>>>>>>>>> make all necessary changes as you said to make this > > > >> truly > > > >>>> nice > > > >>>>>>> and > > > >>>>>>>>>>>> comprehensive as much as possible. Most state processor > > > >>>> apis > > > >>>>>> are > > > >>>>>>>>> marked > > > >>>>>>>>>>>> experimental so we can be flexible within reason :) > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Now to the concrete questions: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> 1. Non-keyed state support / scope > > > >>>>>>>>>>>> I think non-keyed states should definitely be in the > > > >> scope > > > >>>> of > > > >>>>>> the > > > >>>>>>>>> FLIP > > > >>>>>>>>>> in > > > >>>>>>>>>>>> terms of design , and my intention was not to exclude > > > >> them > > > >>>> I > > > >>>>>> just > > > >>>>>>>>>> focused > > > >>>>>>>>>>>> on the keyed state as that is readily available in our > > > >>>>>> prototype > > > >>>>>>>>>>>> implementation (without much changes to the existing > > > >>>>>>> connectors). I > > > >>>>>>>>>> will > > > >>>>>>>>>>>> try to update the FLIP to include non-keyed states > > > >> more in > > > >>>>>> detail > > > >>>>>>>>> but I > > > >>>>>>>>>>>> think the case is pretty straightforward. From a table > > > >>>>>>>> representation > > > >>>>>>>>>>>> perspective, they can follow a similar pattern such as: > > > >>>>>>>>>>>> uid_opUID_statename_broadcast , > > > >> uid_opUID_statename_list > > > >>>> . A > > > >>>>>>>>>>> corresponding > > > >>>>>>>>>>>> SQL connector can easily be added to support these > > > >> based > > > >>>> on the > > > >>>>>>>>>> existing > > > >>>>>>>>>>>> datastream connector. I will make sure to add separate > > > >>>> tickets > > > >>>>>>> for > > > >>>>>>>>>> these > > > >>>>>>>>>>>> types of states once the FLIP is accepted and this > > > >> work can > > > >>>>>> very > > > >>>>>>>>> easily > > > >>>>>>>>>>> be > > > >>>>>>>>>>>> parallelized across different state types within the > > > >>>> existing > > > >>>>>>>> catalog > > > >>>>>>>>>>>> frameworks. This way keyed/non-keyed states will live > > > >>>> directly > > > >>>>>>>>> together > > > >>>>>>>>>>> in > > > >>>>>>>>>>>> a single catalog/db. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> In the future we can even go a step further and include > > > >>>>>> connector > > > >>>>>>>>>>> specific > > > >>>>>>>>>>>> state views such as kafka offsets etc with custom > > > >> connector > > > >>>>>>>> specific > > > >>>>>>>>>>>> plugins > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> 2/3. Serializer transparency and robustness > > > >>>>>>>>>>>> From a practical standpoint both generated (synthetic) > > > >>>>>>> serializers > > > >>>>>>>>> and > > > >>>>>>>>>>>> custom classes / kryo and pluggable logic could work > > > >> but > > > >>>> the > > > >>>>>>> whole > > > >>>>>>>>>>> catalog > > > >>>>>>>>>>>> concepts requires a certain behaviour to be useful. The > > > >>>> catalog > > > >>>>>>>> would > > > >>>>>>>>>>> point > > > >>>>>>>>>>>> to savepoint directories and discover all state in it > > > >>>>>>> (potentially > > > >>>>>>>>> from > > > >>>>>>>>>>>> multiple jobs). Configuration has to be done in a > > > >> generic > > > >>>> way, > > > >>>>>> I > > > >>>>>>>>> don't > > > >>>>>>>>>>> see > > > >>>>>>>>>>>> a problem with introducing configs for specifying > > > >> custom > > > >>>>>>>>>>>> serializers/factories either generically for certain > > > >>>> specific > > > >>>>>>>>> classes. > > > >>>>>>>>>> In > > > >>>>>>>>>>>> most cases however this won't be necessary as the state > > > >>>>>> snapshot > > > >>>>>>>>> itself > > > >>>>>>>>>>>> usually has a reference (classname) of the original > > > >> user > > > >>>>>> classes. > > > >>>>>>>> If > > > >>>>>>>>>> the > > > >>>>>>>>>>>> catalog process has access to those classes it will use > > > >>>> that > > > >>>>>>>>> directly, > > > >>>>>>>>>> or > > > >>>>>>>>>>>> other confugred serializers, and only if not available > > > >> fall > > > >>>>>> back > > > >>>>>>> to > > > >>>>>>>>>>>> generating serializers for POJO/TUPLE types. There is > > > >>>>>> obviously a > > > >>>>>>>>> limit > > > >>>>>>>>>>> to > > > >>>>>>>>>>>> what is possible here initially, Kryo being one > > > >> exception > > > >>>> where > > > >>>>>>> you > > > >>>>>>>>>>> either > > > >>>>>>>>>>>> have the class or not. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> I would like to however point out that we do not have > > > >> to > > > >>>>>> support > > > >>>>>>>>>>> everything > > > >>>>>>>>>>>> initially, we can start with what is currently > > > >> available, > > > >>>> use > > > >>>>>> the > > > >>>>>>>>>>> classpath > > > >>>>>>>>>>>> / generated serializers and as we develop we will find > > > >> the > > > >>>>>> limits > > > >>>>>>>> of > > > >>>>>>>>>> this > > > >>>>>>>>>>>> approach and then can extend with configuration as it > > > >> feels > > > >>>>>>> natural > > > >>>>>>>>>>> instead > > > >>>>>>>>>>>> of trying to create a super complex initial solution. > > > >> But I > > > >>>>>>>>> definitely > > > >>>>>>>>>>>> agree that we should support custom serializer already > > > >>>>>> specified > > > >>>>>>> in > > > >>>>>>>>> the > > > >>>>>>>>>>>> config that is otherwise used by flink for the jobs > > > >> (but I > > > >>>>>> think > > > >>>>>>>> this > > > >>>>>>>>>>>> should more or less work out of the box). > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> 4. The metadata view is currently reused based on the > > > >>>> existing > > > >>>>>>>> table > > > >>>>>>>>>>> valued > > > >>>>>>>>>>>> function. Let's take this as a followup under this > > > >>>> umbrella to > > > >>>>>>>>> improve > > > >>>>>>>>>> / > > > >>>>>>>>>>>> extend the metadata view. I don't think we need a > > > >> separate > > > >>>> FLIP > > > >>>>>>> but > > > >>>>>>>>> it > > > >>>>>>>>>>> also > > > >>>>>>>>>>>> feels out of scope here. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Cheers > > > >>>>>>>>>>>> Gyula > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> On Thu, Jul 2, 2026 at 1:02 PM Dennis-Mircea Ciupitu < > > > >>>>>>>>>>>> [email protected]> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> Hi all, > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Thank you for driving this. Being able to discover > > > >>>>>>>>>>> savepoints/checkpoints > > > >>>>>>>>>>>>> and query their state as SQL tables without shipping > > > >> the > > > >>>>>>> original > > > >>>>>>>>>> user > > > >>>>>>>>>>>>> classes is a genuinely valuable addition, and it's > > > >> nice > > > >>>> that > > > >>>>>> it > > > >>>>>>>>>> builds > > > >>>>>>>>>>> on > > > >>>>>>>>>>>>> the existing state-table connector and > > > >> savepoint_metadata > > > >>>>>> work > > > >>>>>>>>> rather > > > >>>>>>>>>>>> than > > > >>>>>>>>>>>>> starting from scratch. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> A few points and questions, mostly around scope and > > > >> the > > > >>>>>>>> serializer > > > >>>>>>>>>>> story: > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> 1. Non-keyed state and the DataStream path. > > > >>>>>>>>>>>>> - The FLIP scopes out BroadcastState, operator > > > >>>>>> ListState > > > >>>>>>>> and > > > >>>>>>>>>>>>> UnionState because "no readily available Table > > > >> API > > > >>>>>>>> connectors > > > >>>>>>>>>>> exist > > > >>>>>>>>>>>>> for > > > >>>>>>>>>>>>> these state types." That's a fair > > > >> characterization > > > >>>> of > > > >>>>>> the > > > >>>>>>>>> Table > > > >>>>>>>>>>>>> layer, but > > > >>>>>>>>>>>>> the state-processor DataStream API already > > > >> reads > > > >>>> all > > > >>>>>>> three > > > >>>>>>>>>> today > > > >>>>>>>>>>>>> (SavepointReader#readBroadcastState / > > > >>>> #readUnionState / > > > >>>>>>>>>>>>> #readListState). So > > > >>>>>>>>>>>>> the limitation is really in the keyed-only SQL > > > >>>> mapping > > > >>>>>>>>>>>>> (KeyedStateReader > > > >>>>>>>>>>>>> runs inside a keyed backend), not in the > > > >> snapshots > > > >>>>>>>>> themselves. > > > >>>>>>>>>>>>> - Is the keyed-only scope a deliberate > > > >>>> UX/table-mapping > > > >>>>>>>>>> decision, > > > >>>>>>>>>>>> or > > > >>>>>>>>>>>>> would a DataStream-backed reader be considered > > > >> so > > > >>>> the > > > >>>>>>>> catalog > > > >>>>>>>>>>> isn't > > > >>>>>>>>>>>>> strictly less capable than the API it extends? > > > >>>> Even if > > > >>>>>>>>>> non-keyed > > > >>>>>>>>>>>>> contents > > > >>>>>>>>>>>>> stay out of scope initially, it would be good > > > >> to > > > >>>> frame > > > >>>>>>> this > > > >>>>>>>>>>>>> explicitly as a > > > >>>>>>>>>>>>> Table-mapping constraint rather than a general > > > >> one. > > > >>>>>>>>>>>>> 2. Serializer transparency - the "no user classes" > > > >>>> premise > > > >>>>>>> vs. > > > >>>>>>>>>>> custom > > > >>>>>>>>>>>>> serializers. > > > >>>>>>>>>>>>> - The design relies on Flink's transparent > > > >>>> serializer > > > >>>>>>>> formats > > > >>>>>>>>>> to > > > >>>>>>>>>>>>> decode state without user dependencies, which > > > >> is > > > >>>> great > > > >>>>>>> for > > > >>>>>>>>>>>>> POJO/Avro/basic > > > >>>>>>>>>>>>> types. But two serialization efforts point the > > > >>>> other > > > >>>>>> way: > > > >>>>>>>>>>> FLIP-398 > > > >>>>>>>>>>>>> [1] > > > >>>>>>>>>>>>> (released) already lets users configure > > > >>>> serializers per > > > >>>>>>>> type > > > >>>>>>>>>> via > > > >>>>>>>>>>>>> pipeline.serialization-config, and FLIP-538 > > > >> [2] (in > > > >>>>>>>>> discussion) > > > >>>>>>>>>>>> adds > > > >>>>>>>>>>>>> pluggable custom generic-type serializers (e.g. > > > >>>> Apache > > > >>>>>>>> Fory) > > > >>>>>>>>>> and > > > >>>>>>>>>>>>> promotes > > > >>>>>>>>>>>>> TypeSerializer/TypeSerializerSnapshot to > > > >> @Public. > > > >>>> As > > > >>>>>>>> FLIP-538 > > > >>>>>>>>>>>>> itself notes, > > > >>>>>>>>>>>>> state written with a custom serializer becomes > > > >>>>>> dependent > > > >>>>>>> on > > > >>>>>>>>>> that > > > >>>>>>>>>>>>> serializer > > > >>>>>>>>>>>>> to decode - external tooling without it cannot > > > >> read > > > >>>>>> those > > > >>>>>>>>>> bytes. > > > >>>>>>>>>>>>> - Could we make the deserialization side > > > >> pluggable > > > >>>> and > > > >>>>>>>>>>>> config-driven, > > > >>>>>>>>>>>>> mirroring FLIP-398's serialization-config, > > > >> with a > > > >>>>>>> graceful > > > >>>>>>>>>>> fallback > > > >>>>>>>>>>>>> (e.g. > > > >>>>>>>>>>>>> expose the raw bytes / skip the column) when a > > > >>>> format > > > >>>>>>> isn't > > > >>>>>>>>>>>>> transparently > > > >>>>>>>>>>>>> decodable? There already seems to be a seam for > > > >>>> this > > > >>>>>>>>>>>>> (SavepointTypeInformationFactory), and making > > > >> it a > > > >>>>>>>>> first-class, > > > >>>>>>>>>>>>> config-selectable option would keep the catalog > > > >>>>>>>>>>> forward-compatible > > > >>>>>>>>>>>> as > > > >>>>>>>>>>>>> serialization support grows. > > > >>>>>>>>>>>>> 3. Robustness of the transparent decoding path. > > > >>>>>>>>>>>>> - Related to (2): reconstructing values by > > > >>>> mirroring > > > >>>>>> the > > > >>>>>>>>> binary > > > >>>>>>>>>>>>> layout (PojoToRowDataDeserializer) is the most > > > >>>> powerful > > > >>>>>>> but > > > >>>>>>>>>> also > > > >>>>>>>>>>>> the > > > >>>>>>>>>>>>> most > > > >>>>>>>>>>>>> fragile part of the design. How is it expected > > > >> to > > > >>>>>> behave > > > >>>>>>>>> across > > > >>>>>>>>>>>>> serializer > > > >>>>>>>>>>>>> schema evolution / state migration (a > > > >> serializer > > > >>>>>> snapshot > > > >>>>>>>>> that > > > >>>>>>>>>>>>> differs from > > > >>>>>>>>>>>>> the writer's), Kryo-fallback fields, > > > >> nested/generic > > > >>>>>>> types, > > > >>>>>>>>> and > > > >>>>>>>>>>>>> nullability? > > > >>>>>>>>>>>>> - It would help to spell out the supported > > > >> matrix > > > >>>> and > > > >>>>>> the > > > >>>>>>>>>> failure > > > >>>>>>>>>>>>> mode (hard error vs. degrade to raw bytes) up > > > >>>> front, > > > >>>>>>> since > > > >>>>>>>>> this > > > >>>>>>>>>>>>> is exactly > > > >>>>>>>>>>>>> where "read without the user classes" is most > > > >>>> likely to > > > >>>>>>>> break > > > >>>>>>>>>> in > > > >>>>>>>>>>>>> practice. > > > >>>>>>>>>>>>> 4. Observability / summary reporting. > > > >>>>>>>>>>>>> - The metadata view is a great start. Two small > > > >>>> asks: > > > >>>>>>>>>>>>> - per-subtask (or per-key-group) size > > > >>>> granularity in > > > >>>>>>>>>> addition > > > >>>>>>>>>>> to > > > >>>>>>>>>>>>> per-operator, since skew is usually what > > > >> you are > > > >>>>>>> chasing > > > >>>>>>>>> on > > > >>>>>>>>>>>>> large state; > > > >>>>>>>>>>>>> - optionally rounding out the size breakdown > > > >>>> with > > > >>>>>>>>>> managed/raw > > > >>>>>>>>>>>>> operator state and channel state sizes for a > > > >>>> full > > > >>>>>>>> picture > > > >>>>>>>>>>>> (noting > > > >>>>>>>>>>>>> the > > > >>>>>>>>>>>>> latter are in-flight / unaligned-checkpoint > > > >>>> buffers > > > >>>>>>>> rather > > > >>>>>>>>>>>>> than user state). > > > >>>>>>>>>>>>> - A prominent upfront summary of the largest > > > >>>> operators > > > >>>>>> / > > > >>>>>>>>> state > > > >>>>>>>>>> is > > > >>>>>>>>>>>>> often what users want before drilling in. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Best regards, > > > >>>>>>>>>>>>> Dennis > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> [1] > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>> > > > >> > > > > > > https://cwiki.apache.org/confluence/spaces/FLINK/pages/282102217/FLIP-398+Improve+Serialization+Configuration+And+Usage+In+Flink > > > >>>>>>>>>>>>> [2] > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>> > > > >> > > > > > > https://cwiki.apache.org/confluence/spaces/FLINK/pages/373886828/FLIP-538+Support+Custom+Generic+Type+Serializer > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On Mon, Jun 29, 2026 at 12:53 PM Gyula Fóra < > > > >>>>>> [email protected] > > > >>>>>>>> > > > >>>>>>>>>> wrote: > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Hi Flink Devs! > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> I would like to start the discussion about > > > >> FLIP-599: > > > >>>> State > > > >>>>>>>>> Catalog > > > >>>>>>>>>>> [1] > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> State and stateful processing has always been one > > > >> of > > > >>>> the > > > >>>>>> most > > > >>>>>>>>>>>> fundamental > > > >>>>>>>>>>>>>> features of Flink and a major contributor to its > > > >>>> success > > > >>>>>> and > > > >>>>>>>>> global > > > >>>>>>>>>>>>>> adoption. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Over the years several apis and methods have been > > > >>>> developed > > > >>>>>>> to > > > >>>>>>>>>>> address > > > >>>>>>>>>>>>> the > > > >>>>>>>>>>>>>> need for external access and analytics such as the > > > >>>> state > > > >>>>>>>>> processor > > > >>>>>>>>>>>>>> datastream / java apis, the since deprecated > > > >> queryable > > > >>>>>> state > > > >>>>>>>>>>>> abstractions > > > >>>>>>>>>>>>>> and more recently a number of table / SQL api > > > >>>> connectors to > > > >>>>>>>>> access > > > >>>>>>>>>>>> state > > > >>>>>>>>>>>>>> metadata and keyed states in a somewhat limited > > > >> way. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Extending the current capabilities of the > > > >>>>>> state-process-api, > > > >>>>>>>> this > > > >>>>>>>>>>> FLIP > > > >>>>>>>>>>>>> aims > > > >>>>>>>>>>>>>> to lift state processing, analytics and > > > >> observability > > > >>>> to a > > > >>>>>>> new > > > >>>>>>>>>> level > > > >>>>>>>>>>>> by > > > >>>>>>>>>>>>>> introducing the State Catalog. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> State Catalog is a Flink SQL Catalog implementation > > > >>>> that > > > >>>>>>> allows > > > >>>>>>>>>>>>> discovering > > > >>>>>>>>>>>>>> savepoints/checkpoints and mapping their state > > > >>>>>> automatically > > > >>>>>>> to > > > >>>>>>>>> SQL > > > >>>>>>>>>>>>> tables. > > > >>>>>>>>>>>>>> The tables are derived for the different operators > > > >> and > > > >>>>>> their > > > >>>>>>>>> keyed > > > >>>>>>>>>>>> states > > > >>>>>>>>>>>>>> with schema matching the state structure. Most > > > >>>> importantly > > > >>>>>> it > > > >>>>>>>>>>> supports > > > >>>>>>>>>>>>>> reading POJO / Avro and other structured and basic > > > >> type > > > >>>>>>> states > > > >>>>>>>>>>> without > > > >>>>>>>>>>>>> the > > > >>>>>>>>>>>>>> original user classes (dependencies) by relying on > > > >>>> Flink's > > > >>>>>>>>>>> transparent > > > >>>>>>>>>>>>> and > > > >>>>>>>>>>>>>> efficiently structured serializer formats. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> We have a fully functional prototype implementation > > > >>>>>> developed > > > >>>>>>>>> with > > > >>>>>>>>>>>> Gabor > > > >>>>>>>>>>>>>> Somogyi that we will be happy to share if the > > > >> community > > > >>>>>>> accepts > > > >>>>>>>>> the > > > >>>>>>>>>>>>>> proposal! > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Looking forward to your feedback and suggestions! > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Gyula > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> [1] > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>> > > > >> > > > > > > https://cwiki.apache.org/confluence/spaces/FLINK/pages/438009922/FLIP-599+State+Catalog > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>> > > > >> > > > > > >
