Hi Ravi Thank you for your reply.
Unfortunately I cannot set the expiry time to 2 minutes as I need the value as long as there is no new value. So I need one value for each device every minute and always the most recent value for each device. Regards, Daniela 2016-04-11 13:16 GMT+02:00 Ravi Sharma <ping2r...@gmail.com>: > Hi Daniel, > > What about creating your Redis Key in format YYYYMMDDHHmm_DeviceId or may > be just HHmm_DeviceId > > i.e. > 201604111213_A or 1213_A > > and give the key expiry time as 2 minutes. i assume you dont care about > past data. > > Now whenevr a new value come set this key Value (YYYYMMDDHHmm_DeviceId or > HHmm_DeviceId) > > and on aggregation side whenever you have to add all values, just get the > list of all of ur devices, build all keys with ur format and do a multiget > for all keys i.e 201604111213_A, 201604111213_B etc and add it. I think > redis also has a way to do the sum and save the result into other key. > > > Thanks > Ravi. > > > > On Sun, Apr 10, 2016 at 9:46 PM, Daniela Stoiber < > daniela.stoi...@gmail.com> > wrote: > > > HI Arun, > > > > thank you for your reply. > > > > But my problem is that I need to add up the values over all devices, but > I > > am only allowed to use the most recent value of each device. A value is > > valid as long as there is no new value for this device available. > > > > So if I receive a message with device A with value 1, value 1 should be > > used for the sum as long as the value of A does not change. > > When I receive a new value for A, the new value should be used for the > sum > > and the old one should be replaced. > > > > Therefore I thought to use Redis to store this information: > > Device Value > > A 1 > > B 10 > > C 4 > > > > Then I would like to pull every minute the most recent value of each > > device to build the sum. Therefore I would like to use the windowed bolt. > > But I am not sure if it is possible to pull data out of Redis within a > > windowed bolt. > > > > Thank you in advance. > > > > Regards, > > Daniela > > > > > > -----Ursprüngliche Nachricht----- > > Von: Arun Iyer [mailto:ai...@hortonworks.com] Im Auftrag von Arun > > Mahadevan > > Gesendet: Sonntag, 10. April 2016 20:55 > > An: dev@storm.apache.org > > Betreff: Re: AW: Use only latest values > > > > Hi Matthias, > > > > WindowedBolt does support event time. In trident its is not yet exposed. > > > > Hi Daniela, > > > > You could solve your use cases in different ways. One would be to have a > > WindowedBolt with a 1 min tumbling window, do your custom aggregation > (e.g. > > sum) every time the window tumbles and emit the results to another bolt > > where you update the count in Redis. Most of your state saving could also > > be automated by defining a Stateful bolt that would periodically > checkpoint > > your state (sum per device). You could also club both windowing and state > > into a StatefulWindowedBolt implementation. You can evaluate the options > > and decide based on your use cases. > > > > Take a look at the sample topologies (SlidingWindowTopology, > > SlidingTupleTsTopology, StatefulTopology, StatefulWindowingTopology) in > > storm-starter and the docs for more info. > > > > https://github.com/apache/storm/blob/master/docs/Windowing.md > > > > https://github.com/apache/storm/blob/master/docs/State-checkpointing.md > > > > > > -Arun > > > > > > > > > > On 4/10/16, 4:30 PM, "Matthias J. Sax" <mj...@apache.org> wrote: > > > > >A tumbling window (ie, non-overlapping window) is the right approach (a > > >sliding window is overlapping). > > > > > >The window goes into your aggregation bolt (windowing and aggregation > > >goes hand in hand, ie, when the window gets closed, the aggregation is > > >triggered and the window content is handed over to the aggregation > > >function). > > > > > >Be aware that Storm (currently) only supports processing time window > > >(an no event time windows). > > > > > >-Matthias > > > > > > > > >On 04/10/2016 09:56 AM, Daniela Stoiber wrote: > > >> Hi, > > >> > > >> thank you for your reply. > > >> > > >> How can I ensure that the latest values are pulled from Redis the sum > > >> is updated every minute? Do I need a sliding window with an interval > > >> of 1 minute? Where would this sliding window be located in my > topology? > > >> > > >> Thank you in advance. > > >> > > >> Regards, > > >> Daniela > > >> > > >> -----Ursprüngliche Nachricht----- > > >> Von: Matthias J. Sax [mailto:mj...@apache.org] > > >> Gesendet: Samstag, 9. April 2016 12:13 > > >> An: dev@storm.apache.org > > >> Betreff: Re: Use only latest values > > >> > > >> Sounds reasonable. > > >> > > >> > > >> On 04/09/2016 08:34 AM, Daniela Stoiber wrote: > > >>> Hi, > > >>> > > >>> > > >>> > > >>> I would like to cache values and to use only the latest "valid" > > >>> values to build a sum. > > >>> > > >>> In more detail, I receive values from devices periodically. I would > > >>> like to add up all the valid values each minute. But not every > > >>> device sends a new value every minute. And as long as there is no > > >>> new value the old one should be used for the sum. As soon as I > > >>> receive a new value from a device I would like to overwrite the old > > >>> value and to use the new one for the sum. Would that be possible > > >>> with the combination of > > >> Storm and Redis? > > >>> > > >>> > > >>> > > >>> My idea was to use the following: > > >>> > > >>> > > >>> > > >>> - Kafka Spout > > >>> > > >>> - Storm Bolt for storing the tuples in Redis and for overwriting the > > >>> values as soon as a new one is delivered > > >>> > > >>> - Storm Bolt for reading the latest tuples from Redis > > >>> > > >>> - Storm Bolt for grouping (I would like to group the devices per > > >>> region) > > >>> > > >>> - Storm Bolt for aggregation > > >>> > > >>> - Storm Bolt for storing the results again in Redis > > >>> > > >>> > > >>> > > >>> Thank you in advance. > > >>> > > >>> > > >>> > > >>> Regards, > > >>> > > >>> Daniela > > >>> > > >>> > > >> > > >> > > > > > > > > > >
