Have now implemented SQLLookupService which is a LookupService<Record> implementation.
This works as advertised to provide a service based method of assigning attributes to a flow file based on a Lookup Service. https://github.com/brettryan/nifi-drunken-bundle/blob/dev/nifi-drunken-services/src/main/java/com/drunkendev/nifi/services/SQLLookupService.java <https://github.com/brettryan/nifi-drunken-bundle/blob/dev/nifi-drunken-services/src/main/java/com/drunkendev/nifi/services/SQLLookupService.java> The matching LookupAttributeFromRecord https://github.com/brettryan/nifi-drunken-bundle/blob/dev/nifi-drunken-processors/src/main/java/com/drunkendev/nifi/processors/LookupAttributeFromRecord.java <https://github.com/brettryan/nifi-drunken-bundle/blob/dev/nifi-drunken-processors/src/main/java/com/drunkendev/nifi/processors/LookupAttributeFromRecord.java> If y'all think this could be worth contributing I'll invest further time in neatening these up and provide full test cases, atm these are just a PoC that's been proven. > On 9 Jan 2018, at 18:50, Brett Ryan <[email protected]> wrote: > > >> On 7 Jan 2018, at 22:23, Brett Ryan <[email protected] >> <mailto:[email protected]>> wrote: >> >> (1) Write a processor that updates an attribute on a FF from a Record based >> LookupService. Say I call it UpdateAttributeFromRecordLookup > > Ok, I've implemented this now as a PoC and have tested it with the > MongoDBLookupService which is working great. > > The processor takes a record based lookup service and any number of > attributes. The processor has a required attribute that specifies a key > prefix (default = "key."), this prefix identifies further attributes as a key > lookup, any other attributes are considered names to include as > attribute/field pairs for the resulting FF, if none of these non-key custom > attributes exist then all fields from the Record are added. > > if we take the (LookupRecord with MongoDB) as an example > > Key Prefix = ".key" > key.id_store = ${id_store} > > We would end up with the following attributes added to the resulting FF > > address = 177 Boulevard Haussmann, 75008 Paris > address_city = Paris > capacity = 464600 > id_store = 1 > manager = Jean Ricca > Given all this, do you think I'm now on the right track for a more general > purpose processor that could be used for multiple lookups? > > I could then implement the SQL Lookup Service to feed this. > > [1]: > https://community.hortonworks.com/articles/146198/data-flow-enrichment-with-nifi-part-3-lookuprecord.html > > <https://community.hortonworks.com/articles/146198/data-flow-enrichment-with-nifi-part-3-lookuprecord.html> >> (2) Write a service SQLLookupService which given a query as a lookup >> coordinate would return a Record for the first found entry. >> (3) Write a processor that updates an attribute from a record lookup, call >> it UpdateAttributeFromRecordLookup. >> >> It's a bit more long winded to achieve what I originally had with a single >> processor, but I guess UpdateAttributeFromRecordLookup would have the >> benefit of updating attributes from LookupService implementations that do >> not meet the LookupAttribute criteria. >> >> If you think this is viable, please let me know and I'll have a look at >> starting it tomorrow afternoon. >
signature.asc
Description: Message signed with OpenPGP
