Hi Stephan, Great to hear about your experience with StateFun so far!
I think what you are looking for is a way to read StateFun checkpoints, which are basically an immutable consistent point-in-time snapshot of all the states across all your functions, and run some computation or simply to explore the state values. StateFun checkpoints are essentially adopted from Flink, so you can find more detail about that here [1]. Currently, StateFun does provide a means for state "bootstrapping": running a batch offline job to write and compose a StateFun checkpoint [2]. What is still missing is the "reading / analysis" side of things, to do exactly what you described: running a separate batch offline job for reading and processing an existing StateFun checkpoint. Before we dive into details on how that may look like, do you think that is what you would need? Although I don't think we would be able to support such a feature yet since we're currently focused on reworking the SDKs and request-reply protocol, in any case it would be interesting to discuss if this feature would be important for multiple users already. Cheers, Gordon [1] https://ci.apache.org/projects/flink/flink-docs-master/concepts/stateful-stream-processing.html#checkpointing [2] https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/deployment-and-operations/state-bootstrap.html On Wed, Jan 27, 2021 at 11:41 PM Stephan Pelikan <stephan.peli...@phactum.at> wrote: > Hi, > > > > We are trying to use Statefuns for our tool and it seems to be a good fit. > I already adopted it and it works quite well. However, we have millions of > different states (all the same FunctionType but different ids) and each > state consists of several @Persisted values (values and tables). We want to > build an administration tool for examining the crowd of states (count, > histogram, etc.) and each state in detail (the persisted-tables and > -values). > > > > Additionally we need some kind of dig-down functionality for finding those > individual states. For example some of those persisted values can be used > to categorize the crowd of states. > > > > My question now is how to achieve this. Is there a way to browse and > examine statefuns in a read-only fashion (their ids, their persisted > values)? How can one achieve this without duplicating status in e.g. a > relational database? > > > > Thanks, > > Stephan > > > > PS: I have another questions but I will send them in separate mails to > avoid mixing up topics. >
