It depends on definition of slow and data stored of course, my guess is that few million of keys might take a minute? Pure guess. Redis is a key value store, you give it a key and you can perform an operation on its value. Iterating over all keys is the slowest operation in redis. I think it will also block all other operations while this one is executing. I know this is a storm and not redis group, I am not sure there is a storm solution if redis is your partial data storage. It's not a relational database so it's not great at joins, aggregations, etc. just my 2c. Time series aggregations in redis are done with 1 key per interval, for example. 2016-06-01: 1:30pm event would execute a counter increment in 2016 key, 2016-06, 2016-06-01, etc down to your smallest interval. Then to pull count for a day you would get 1 key only, 2016-06-01. This approach is fast because all operations are key value based, accessing only 1 key at a time. There is no way to pull data you need at the same time before you store that key into redis? You can use redis as your queue and process it once a minute with a topology, then create a new time based queue key and keep going. You would store your data a bit differently though. Instead of many keys, you would have one key with array of values. You keep pushing into it based on a time stamp, then when it lapses you process it with storm and pop those values out one at a time. Lookup the data you need, keep an aggregate and keep going till the queue is empty.
> On May 30, 2016, at 1:17 PM, Daniela S <[email protected]> wrote: > > I have to pull the entries and to add a specific value to every entry. This > value is stored in another database and therefore I would like to make the > join. based on some conditions, in Storm. I need this value to build the sum, > as the entries do not contain any information for the sum. > > What would be very few keys? > > Thank you and regards, > Daniela > > Gesendet: Montag, 30. Mai 2016 um 20:11 Uhr > Von: "Yuri Kostine" <[email protected]> > An: [email protected] > Betreff: Re: Pull from Redis > > Do you pull entries only to sum them up? Why not keep a running total in > redis in a time stamped key by minute? Generally speaking redis is not great > for pulling all keys unless there are very few keys. > > On May 30, 2016, at 12:49 PM, Daniela S <[email protected]> wrote: > > Hi > > I have a topology that stores entries in Redis. Now I would like to pull all > entries from Redis every minute or as soon as a value has changed. How can I > do that? Can I add another bolt to my topology for this task or do I have to > use a spout or even a new topology? I would like to build a sum over all > entries every minute. Do you have any advice for that? > > Thank you in advance. > > Regards, > Daniela
