Rahul, I have a client that uses Mongo for most of their data storage. What I am doing for them for enrichment like this is I took all of their enrichment data and store it now in a Mongo collection. Then as data comes in, I use a LookupService to merge all of the fields from that collection into the records as they come into the pipeline. Their current process sounds similar to yours if I understand it correctly: load both data sets, then reload the one, enrich it and save the enrichment. If you look at the record processors and use a LookupService, you can do the enrichment on the fly as you get data so there is no moving data around between tables/collections.
On Mon, Nov 6, 2017 at 5:20 PM, Malhotra, Rahul <[email protected]> wrote: > Thanks Eric. This helps. > > > > > > *Rahul Malhotra* > > Group Development Manager – *Data Labs* *|* Consumer Tax Group *| *Intuit > Inc. *|* w: (858) 215-7150 | c: (760) 889-7206 > > https://bluejeans.com/7608897206 > > > > > > *From: *<[email protected]> on behalf of Eric Chaves <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Sunday, November 5, 2017 at 3:05 PM > *To: *"[email protected]" <[email protected]> > *Subject: *Re: Enrichment > > > > Hi Rahul, I'm working something similar and my first shoot was writing a > Scriptedookup service to perform database lookups. Not sure if this is the > recommended approach but it is working. > > > > In case you want to give it a try you can take a look on Matt's articles > "Scripting Cookbooks" to get a hang on NiFi scripting capabilities and then > use Andy Lopresto groovy lookup as starting point: > > > > - https://community.hortonworks.com/articles/75032/ > executescript-cookbook-part-1.html > > - https://gist.github.com/alopresto/78eb1a2c2b878f75f61481269af38a9f > > - https://gist.github.com/alopresto/beb62a15c82c6d68528474085b1a9610 > > > > Best regards, > > > Eric > > > > 2017-11-05 0:08 GMT-02:00 Malhotra, Rahul <[email protected]>: > > Hi, > > I am trying to enrich my data by looking up in a dimension table and > moving from one table to another. My warehouse is Vertica, is there a > recommended approach we can take? Or is it something we have to write as a > custom process? > > > > > > Thanks, > > > > Rahul > > >
