Hi Gabriele, use (or extend) the window operator provided by Flink is a better idea. A window operator in Flink manages two types of state:
- Window state: accumlate data for windows, and provide data to window function when a window comes to its end time. - Timer state: store the end times of windows, and provide the minimum end time to the window operator. The source codes are mainly in org.apache.flink.streaming.runtime.operators.windowing.WindowOperator. - "InternalAppendingState<K, W, IN, ACC, ACC> windowState" refers to the window state. - "InternalTimerService<W> internalTimerService" refers to the timer state. Best regards Rui Xia On Mon, Mar 4, 2024 at 7:39 PM Gabriele Mencagli < gabriele.menca...@gmail.com> wrote: > Dear Flink Community, > > I am using Flink with the DataStream API and operators implemented using > RichedFunctions. I know that Flink provides a set of window-based operators > with time-based semantics and tumbling/sliding windows. > > By reading the Flink documentation, I understand that there is the > possibility to change the memory backend utilized for storing the in-flight > state of the operators. For example, using RocksDB for this purpose to cope > with a larger-than-memory state. If I am not wrong, to transparently change > the backend (e.g., from in-memory to RocksDB) we have to use a proper API > to access the state. For example, the Keyed State API with different > abstractions such as ValueState<T>, ListState<T>, etc... as reported here > <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/> > . > > My question is related to the utilization of time-based window operators > with the RocksDB backend. Suppose for example very large temporal windows > with a huge number of keys in the stream. I am wondering if there is a > possibility to use the built-in window operators of Flink (e.g., with an > AggregateFunction or a more generic ProcessWindowFunction as here > <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/operators/windows/>) > transparently with RocksDB support as a state back-end, or if I have to > develop the window operator in a raw manner using the Keyed State API > (e.g., ListState, AggregateState) for this purpose by implementing the > underlying window logic manually in the code of RichedFunction of the > operator (e.g., a FlatMap). > Thanks for your support, > > -- > Gabriele Mencagli > >
