You could do that, but at the moment, I believe the indexing topology is looking for its configs in the same place in zookeeper, so that'd get in the way of that.
On Mon, Jan 9, 2017 at 11:08 AM, Otto Fowler <[email protected]> wrote: > What about having multiple instances of the enrichment topology with > different configurations? Then you would just have the last one terminate > to indexing? Or something like that. > > > On January 9, 2017 at 10:56:38, Casey Stella ([email protected]) wrote: > > I think that would be a good feature to add to have arbitrary number of > phases, though it might be tricky to code (the way I envisioned it would > involve a loop in storm, which is possible[1]), might have unintended > consequences to guarantees (e.g. updating enrichments might not be able to > be applied in realtime) and could be tricky to reason about > performance-wise. > > As it stands, the number of phases is a consequence of the topology > itself. We do not currently have an architecture which would allow an > arbitrary number of phases without changing the flux file itself. What you > can do, though, in a stellar enrichment is stack enrichments (e.g. depend > on previous enrichments) because it's just a list of stellar statements. > The consequence, of course, is that these statements get run within the > same worker, which is unfortunate, but may be a stopgap workaround. > > *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8 > > On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler <[email protected]> > wrote: > >> Maybe the naming of the phases is misleading? What if you could set up >> an arbitrary number of stages, with defaults? >> >> >> On January 8, 2017 at 16:25:01, Casey Stella ([email protected]) wrote: >> >> You could do the geo enrichment normally and do a stellar hbase enrichment >> in the threat Intel phase. >> >> On Sun, Jan 8, 2017 at 16:22 Ryan Merriman <[email protected]> wrote: >> >> > Hbase enrichments and geo enrichments are done in parallel so I would >> not >> > expect this to work. You could do the Hbase enrichment as a threat Intel >> > enrichment and that should work because enrichments and threat Intel are >> > done in series. >> > >> > >> > >> > The ideal way would be to chain together Stellar enrichments but I don't >> > think there is a geo enrichment function created yet. I think that >> should >> > be a Jira. I know someone is working on an update to how we do geo >> > enrichments so I will file a follow on Jira if it's not included in the >> > scope of that work. >> > >> > >> > >> > Ryan >> > >> > >> > >> > > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov <[email protected]> >> > wrote: >> > >> > > >> > >> > > Is it possible to enrich enrichment? >> > >> > > >> > >> > > For example I have IP address, I enrich it with geo and get City name, >> > >> > > now I want to enrich City name with city crime level (assume I have >> that >> > >> > > data). But when I do that it just does not work. I specify enrichment >> > >> > > like that: >> > >> > >> { >> > >> > >> "index" : "msexchange", >> > >> > >> "batchSize" : 5, >> > >> > >> "enrichment" : { >> > >> > >> "fieldMap" : { >> > >> > >> "geo" : [ "destination_ip", "source_ip" ], >> > >> > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ], >> > >> > >> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ], >> > >> > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ] >> > >> > >> }, >> > >> > >> "fieldToTypeMap" : { >> > >> > >> "enrichments.geo.destination_ip.country" : [ "city_crime_level" ], >> > >> > >> "enrichments:geo:destination_ip:country" : [ "city_crime_level" ], >> > >> > >> "enrichments.geo.destination_ip:country" : [ "city_crime_level" ] >> > >> > >> }, >> > >> > >> "config" : { } >> > >> > >> }, >> > >> > >> "threatIntel" : { >> > >> > >> "fieldMap" : { }, >> > >> > >> "fieldToTypeMap" : { }, >> > >> > >> "config" : { }, >> > >> > >> "triageConfig" : { >> > >> > >> "riskLevelRules" : { }, >> > >> > >> "aggregator" : "MAX", >> > >> > >> "aggregationConfig" : { } >> > >> > >> } >> > >> > >> }, >> > >> > >> "configuration" : { } >> > >> > >> } >> > >> > > I tried all the ways how enrichment field can be entered just to be >> sure >> > >> > > I do not mistype it. >> > >> > > >> > >> > > - Dima >> > >> > >> >> >
