I am really glad you brought this question up as it does come up in conversation often and clearly the way we're talking about it so far isn't really helping.
We have been using the terms 'Simple Event Processing' (SEP) and 'Complex Event Processing' (CEP) quite a lot to explain that NiFi is designed for the SEP side and systems like Storm and Spark and others are designed for both SEP and CEP. I typically also make the case that systems which support CEP tend to make SEP cases harder and that is unfortunate because really what most people are doing is SEP. The real issue has been that 'simple' and 'complex' are *really* bad terms to convey meaning in this context because they are really loaded. There are a lot of really 'complicated', 'hard', 'challenging' things that are considered 'Simple' Event processing tasks. But, they do not rise to the type of functions that were meant when the term CEP was defined. Here is what is what I think is a pretty good CEP definition as found on the wikipedia page [1]. The heart of that definition is the that a process is CEP if there is temporal, causal, spatial correlation of events to make a higher level assertion (ie: new data) or to take an action (ie: respond to the event). But, it is true that the line between something being SEP vs it being CEP can be quite blurry. I also thing these terms have fallen a bit out of favor and when you look at documentation and articles around systems like Storm, Spark, and Flink and so on you don't really see people using them. It is much more common to call it 'stream processing'. ....ok so back to helping you do great things... You are absolutely right that you can use NiFi with great effect to do transformation. This includes things like powerful enrichment, format/schema conversion, filtering, normalization, etc.. I apologize for taking you offtrack with imprecise language but glad you brought it up so we can talk through it and help tell the story better for others. If you want to talk through your use cases a bit more I'd be happy to. Thanks Joe [1] https://en.wikipedia.org/wiki/Event-driven_architecture#Complex_event_processing On Sat, Oct 3, 2015 at 8:50 AM, Adam Williams <[email protected]> wrote: > I'm trying to grasp why it wouldn't be a good idea to use NiFi instead of > Storm etc for handling data transformation and normalization. I've read, > and also watched in this video https://www.youtube.com/watch?v=sQCgtCoZyFQ > and I keep hearing that its not best suited for that, is there a general > reason why? I've been able to do pretty complex transformations on data in > NiFi without any problems, but I want to make sure I'm using it properly. > > Any thoughts on why it shouldn't be used for that purpose would be greatly > appreciated. > > Thanks!
