Depending on the jobs you may use spark for these regular jobs together with Ignite and Kafka.
Nevertheless the focus of storm is event-based processing. If this is not your use case you may want to go along with the architecture below. Do you have fault-tolerance requirements? You may need a fault tolerant storage for Kafka (and maybe Ignite), such as HDFS. Do you need order guarantees? Do you need once and only once delivery? Does Kafka meet your use case? > On 29 Nov 2016, at 12:26, Shawn Du <[email protected]> wrote: > > Hi experts, > > After days trying with ignite, I am impressed by its powerful capability. > > Now in our architecture, ignite works with storm and act as cache/persist > layer. I am considering replace storm with ignite at all. > > The reasons are : > #1 In essential, we are recovering a stateful object according to the > incoming events and generate some metrics periodically. > Maintaining stateful objects, personally I think it is not storm strong > point, but ignite it is. > #2 in order to generate some metrics, the storm codes become very complex and > hard to debug and maintain. I want to use ignite > to simplify the codes. > > Now my rough ideas is: > #1 Integrate kafka with ignite and rebuild the state according to the > incoming events > #2 schedule some periodically tasks to query the cache and generate metrics > and store them into cache again. > > Please comments on this solution. > > saying, there are several caches, for each cache I execute dozens of queries > every 30 seconds (such as group by different dimensions) and put them into > the cache again. > Can ignite support it? > > Thanks > Shawn >
