What about having multiple instances of the enrichment topology with different configurations? Then you would just have the last one terminate to indexing? Or something like that.
On January 9, 2017 at 10:56:38, Casey Stella ([email protected]) wrote: I think that would be a good feature to add to have arbitrary number of phases, though it might be tricky to code (the way I envisioned it would involve a loop in storm, which is possible[1]), might have unintended consequences to guarantees (e.g. updating enrichments might not be able to be applied in realtime) and could be tricky to reason about performance-wise. As it stands, the number of phases is a consequence of the topology itself. We do not currently have an architecture which would allow an arbitrary number of phases without changing the flux file itself. What you can do, though, in a stellar enrichment is stack enrichments (e.g. depend on previous enrichments) because it's just a list of stellar statements. The consequence, of course, is that these statements get run within the same worker, which is unfortunate, but may be a stopgap workaround. *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8 On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler <[email protected]> wrote: > Maybe the naming of the phases is misleading? What if you could set up an > arbitrary number of stages, with defaults? > > > On January 8, 2017 at 16:25:01, Casey Stella ([email protected]) wrote: > > You could do the geo enrichment normally and do a stellar hbase enrichment > in the threat Intel phase. > > On Sun, Jan 8, 2017 at 16:22 Ryan Merriman <[email protected]> wrote: > > > Hbase enrichments and geo enrichments are done in parallel so I would not > > expect this to work. You could do the Hbase enrichment as a threat Intel > > enrichment and that should work because enrichments and threat Intel are > > done in series. > > > > > > > > The ideal way would be to chain together Stellar enrichments but I don't > > think there is a geo enrichment function created yet. I think that should > > be a Jira. I know someone is working on an update to how we do geo > > enrichments so I will file a follow on Jira if it's not included in the > > scope of that work. > > > > > > > > Ryan > > > > > > > > > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov <[email protected]> > > wrote: > > > > > > > > > > Is it possible to enrich enrichment? > > > > > > > > > > For example I have IP address, I enrich it with geo and get City name, > > > > > now I want to enrich City name with city crime level (assume I have > that > > > > > data). But when I do that it just does not work. I specify enrichment > > > > > like that: > > > > >> { > > > > >> "index" : "msexchange", > > > > >> "batchSize" : 5, > > > > >> "enrichment" : { > > > > >> "fieldMap" : { > > > > >> "geo" : [ "destination_ip", "source_ip" ], > > > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ], > > > > >> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ], > > > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ] > > > > >> }, > > > > >> "fieldToTypeMap" : { > > > > >> "enrichments.geo.destination_ip.country" : [ "city_crime_level" ], > > > > >> "enrichments:geo:destination_ip:country" : [ "city_crime_level" ], > > > > >> "enrichments.geo.destination_ip:country" : [ "city_crime_level" ] > > > > >> }, > > > > >> "config" : { } > > > > >> }, > > > > >> "threatIntel" : { > > > > >> "fieldMap" : { }, > > > > >> "fieldToTypeMap" : { }, > > > > >> "config" : { }, > > > > >> "triageConfig" : { > > > > >> "riskLevelRules" : { }, > > > > >> "aggregator" : "MAX", > > > > >> "aggregationConfig" : { } > > > > >> } > > > > >> }, > > > > >> "configuration" : { } > > > > >> } > > > > > I tried all the ways how enrichment field can be entered just to be > sure > > > > > I do not mistype it. > > > > > > > > > > - Dima > > > > > >
