Hi Gabor, To be more specific, I'm thinking of introducing `RichSourceReaderContext`, and having a `RichSourceReaderFactory` creating `SourceReader` on top of the rich context. Or we still use the `SourceReaderFactory` but do type conversion from `SourceReaderContext` to `RichSourceReaderContext` in implementation of state processing source's `createReader`. Well this is not a big change, WDTY?
BTW, I'm curious if there is a solution for implementing a Source V2 based on `InputFormat`. Otherwise if the `InputFormat` is only for V1 then why is it not deprecated. Best, Zakelly On Thu, May 29, 2025 at 10:18 PM Gabor Somogyi <gabor.g.somo...@gmail.com> wrote: > Hi Zakelly, > > Before we go into any details I can say that if we can expose the less the > better so agree with you on the direction. > > On the technical level not yet understand what exactly you're suggesting. > `RichInputFormat` is a \@Public API > where we expose `RuntimeContext` now on V1. I would like to understand the > motivation what would be the difference > between `SourceReaderContext.getRuntimeContext` vs > `V2RichInputFormat.getRuntimeContext`. > Or you mean something like `RichSourceReaderContext`? > > If you can be more explicit on how you imagine the other path with some new > interface/class names, > call flow or anything we can pursue that. > > On coding level now I'm sitting on ~3-4K lines of code but there are some > quite things needs to be done. > The whole code is depending on `SourceReaderContext.getRuntimeContext` but > I think when it's stable > and there is another approach how to expose `RuntimeContext` then it's only > matter of time to make that > adjustments. Hope that we're not shooting for another 10k change :) > > BR, > G > > > On Thu, May 29, 2025 at 10:23 AM Zakelly Lan <zakelly....@gmail.com> > wrote: > > > Hi Gabor, > > > > Good point for the migration. > > > > I took a brief look. I thought the `RuntimeContext` is too powerful and > > might not be suitable to expose on \@Public API. Is it possible to > > introduce another 'rich' code path just like the difference between > > `RichInputFormat` and `InputFormat`, and keep that internal? WDYT > > > > > > Best, > > Zakelly > > > > On Thu, May 22, 2025 at 6:30 PM Gabor Somogyi <gabor.g.somo...@gmail.com > > > > wrote: > > > > > Hi All, > > > > > > I've just had a look at it how it would be possible to migrate state > > > processor API from source v1 to source v2 API. > > > > > > The whole concept in the state processor API actual implementation is > > based > > > on to get > > > RuntimeContext from RichInputFormat [1] which then allows us to access > > > state backend, > > > keyed state store, etc... > > > > > > Since RuntimeContext is not exposed on source v2 API the question is > > > obvious. > > > Does it make sense to expose it on SourceReaderContext [2] ? > > > > > > If somebody has better idea then please share. > > > > > > [1] > > > > > > > > > https://github.com/apache/flink/blob/4c77daaf3554446e67f6b75ac18d54da84208b9a/flink-libraries/flink-state-processing-api/src/main/java/org/apache/flink/state/api/input/KeyedStateInputFormat.java#L156 > > > [2] > > > > > > > > > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/SourceReaderContext.java > > > > > > BR, > > > G > > > > > >
