I totally "bird dogged the previous thread" as Casey likes to call it. :) I am extracting this thought into a separate thread before I start throwing out even more, crazier ideas.
In general, Metron is very opinionated about data flows right now. We have > Parser topologies that feed an Enrichment topology, which then feeds an > Indexing topology. We have useful bits of functionality (think Stellar > transforms, Geo enrichment, etc) that are closely coupled with these > topologies (aka data flows). > > When a user wants to parse heterogenous data from a single topic, that's > not easy. When a user wants enriched output to land in unique topics by > sensor type, well, that's also not easy. When a user wanted to skip > enrichment of data sources, we actually re-architected the data flow to add > the Indexing topology. > > In an ideal world, a user should be responsible for defining the data > flow, not Metron. Metron should provide the "useful bits of functionality" > that a user can "plugin" wherever they like. Metron itself should not care > how the data is moving or what step in the process it is at. -- Nick Allen <[email protected]>
