I have been struggling to figure out how to get trident to aggregate data over a certain time period then flush the data to an external data store. The reasoning behind this is to reduce the number of reads and writes sent to the database.
I've seen that Storm allows for tick tuples to be inserted into the stream, however I can't figure out how to do this with trident. I had thought that this functionality was added with Storm version 0.8.0 ? Is this the case ? One thing I had tried was to create a new stream that emitted a tuple once every X time period, then I tried to merge this stream into my actual data stream. However, doing this would result in a non transactional stream which would be no good. Also it didn't work, as the resulting stream only consisted of tuples from my clock stream. Can anybody help me figure out how to have Trident aggregate data over a certain time frame, flush it out to an external datastore, then rinse and repeat ? there are some blogs out there regarding how to use a sliding window in storm, however I just want sequential windows in Trident. Thanks -- Raphael Hsieh
