Re: Streaming KV store abstraction

>Sorry for the late answer, I completely missed this email. (Thanks Robert for pointing out). No problem ;) >Now that you have everything set up, in flatMap1 (for events) you would query the state : state.value() and enrich your data >in flatMap2 you would update the state: state.update(newState)

Re: Streaming KV store abstraction

Hi! Sorry for the late answer, I completely missed this email. (Thanks Robert for pointing out). You won't be able to use that project as it was dependent on an earlier snapshot version that still had completely different state semantics. I don't think it is realistic that I will re-implment

Re: Streaming KV store abstraction

Hi Gyula, I'm currently looking after ways to enrich streams with external data. Have you got any update on the topic in general or on StreamKV? I've checked out the code but it won't build, mainly because StateCheckpointer has been removed since [FLINK-2808]. Any hint on a quick replacement,

Re: Streaming KV store abstraction

Hey All, We decided to make this a standalone library until it is stable enough and then we can decide whether we want to keep it like that or include in the project: https://github.com/gyfora/StreamKV Cheers, Gyula Gianmarco De Francisci Morales ezt írta (időpont: 2015.

Re: Streaming KV store abstraction

I think that is actually a cool way to kick of an addition to the system. Gives you a lot of flexibility and releasing and testing... It helps, though, to upload maven artifacts for it! On Tue, Sep 15, 2015 at 7:18 PM, Gyula Fóra wrote: > Hey All, > > We decided to make this

Re: Streaming KV store abstraction

Just a silly question. For the example you described, in a data flow model, you would do something like this: Have query ids added to the city pairs (qid, city1, city2), then split the query stream on the two cities and co-group it with the updates stream ((city1, qid) , (city, temp)), same for

Re: Streaming KV store abstraction

Hey Gianmarco, So the implementation looks something different: The update stream is received by a stateful KVStoreOperator which stores the K-V pairs as their partitioned state. The query for the 2 cities is assigned an ID yes, and is split to the 2 cities, and each of these are sent to the

Re: Streaming KV store abstraction

Yes, pretty clear. I guess semantically it's still a co-group, but implemented slightly differently. Thanks! -- Gianmarco On 9 September 2015 at 15:37, Gyula Fóra wrote: > Hey Gianmarco, > > So the implementation looks something different: > > The update stream is

Re: Streaming KV store abstraction

Hello, As for use cases, in my old job at Ericsson we were building a streaming system that was processing data from telephone networks, and it was using key-value stores a LOT. For example, keeping track of various state info of the users (which cell are they currently connected to, what bearers

Streaming KV store abstraction

Hey All, The last couple of days I have been playing around with the idea of building a streaming key-value store abstraction using stateful streaming operators that can be used within Flink Streaming programs seamlessly. Operations executed on this KV store would be fault tolerant as it

Re: Streaming KV store abstraction

@Gyula Can you explain a bit what this KeyValue store would do more then the partitioned key/value state? On Tue, Sep 8, 2015 at 2:49 PM, Gábor Gévay wrote: > Hello, > > As for use cases, in my old job at Ericsson we were building a > streaming system that was processing data

Re: Streaming KV store abstraction

@Stephan: Technically speaking this is really just a partitioned key-value state and a fancy operator executing special operations on this state. >From the user's perspective though this is something hard to implement. If you want to share state between two stream for instance this way (getting