Re: Enrichment plugin for adding attributes from SQL

Have now implemented SQLLookupService which is a LookupService implementation. This works as advertised to provide a service based method of assigning attributes to a flow file based on a Lookup Service.

Re: Enrichment plugin for adding attributes from SQL

> On 7 Jan 2018, at 22:23, Brett Ryan wrote: > > (1) Write a processor that updates an attribute on a FF from a Record based > LookupService. Say I call it UpdateAttributeFromRecordLookup Ok, I've implemented this now as a PoC and have tested it with the

Re: Enrichment plugin for adding attributes from SQL

Brett, I think it’s great that you brought this up and made some specific suggestions, because it’s easy for people to overlook and hard to know how to do the right thing without that kind of feedback. -joey On Jan 7, 2018, 5:51 AM -0600, Brett Ryan , wrote: > > Probably

Re: Enrichment plugin for adding attributes from SQL

Hey Brett, So as for the LookupAttribute only doing a single key lookup, that was a bit of a coin flip on whether it makes more sense to allow multiple lookups at once, each with a single key, or a single lookup with multiple AND-ed constraints. I opted for the former since that was the use

Re: Enrichment plugin for adding attributes from SQL

> On 5 Jan 2018, at 10:17, Andrew Lim wrote: > > Hi Brett, > > Thanks for your feedback on the “Add Processor” window. I’m sorry you had > trouble identifying the right processor to use for your data flow. As it turns out the right processor doesn't yet exist :)

Re: Enrichment plugin for adding attributes from SQL

MongoDBLookupService can't be used with an UpdateAttribute processor though, it returns a Record type, and has no required keys. The whole purpose of the processor that I originally wrote was to update an attribute from SQL on a FF. The implementation of LookupAttribute requires -

Re: Enrichment plugin for adding attributes from SQL

Take a look at the mongo lookup service. I think it could serve as a good example here. On Fri, Jan 5, 2018 at 10:49 PM Brett Ryan wrote: > Looking at using a lookupservice, this doesn't seem to support sending > multiple keys to the LookupService at the same time. > > What

Re: Enrichment plugin for adding attributes from SQL

Looking at using a lookupservice, this doesn't seem to support sending multiple keys to the LookupService at the same time. What I was thinking of doing was implement a LookupService that took an attribute "sql.query" which would use this to evaluate the query but then pass in a map of

Re: Enrichment plugin for adding attributes from SQL

Hi Brett, Thanks for your feedback on the “Add Processor” window. I’m sorry you had trouble identifying the right processor to use for your data flow. There are tags on the left of the “Add Processor” window that categorize many of the processors into functional groups. For example,

Re: Enrichment plugin for adding attributes from SQL

I should qualify what my security concern on the map cache was. Given a shared cache server any other flow not related to mine could read my keys. > On 5 Jan 2018, at 08:32, Andy LoPresto wrote: > > UpdateAttribute doesn’t pull from a database, it uses static or dynamic

Re: Enrichment plugin for adding attributes from SQL

Cool, thanks for the help. I’ll investigate implementing this as a LookupService as there’s no present service for connecting to SQL, I think; man, I could be wrong, I did try looking at all available ;) To help me learn all the processors I actually dragged every processor into process groups

Re: Enrichment plugin for adding attributes from SQL

Ooo, i shall take a look at this, that sounds great. Yeah, my inexperience is probably a sore point. You know what would be great, either in the add processor browser to have categories to find processors. Trying to find enrichment processors only is probably the hardest part of identifying

Re: Enrichment plugin for adding attributes from SQL

UpdateAttribute doesn’t pull from a database, it uses static or dynamic attribute values and supports NiFi Expression Language. In your original message, you didn’t mention any database interaction, so I thought you were just trying to accomplish "I wanted to add some attributes to a FlowFile

Re: Enrichment plugin for adding attributes from SQL

Thanks Andy, how would update attribute be able to get the value from sql? Consider a flow where a piece of information needs to be obtained from a DB but i do not want the contents of the current FF to be altered, using ExecuteSQL anywhere prior would not be possible due to replacing the FF

Re: Enrichment plugin for adding attributes from SQL

Andy, Brett, Taking a quick glance at the code it looks like it's enriching attributes from a database according to a query. If that's correct, there's a LookupAttribute processor that delegates lookups to a "LookupService" and adds attributes without altering content. There are a variety of

Re: Enrichment plugin for adding attributes from SQL

Hi Brett, It’s great that you found it easy to write a new processor for Apache NiFi. It is probably an indicator that we need to improve education/evangelism/documentation, however, that you did not find UpdateAttribute [1], which should do exactly what you were looking for. [1]

Enrichment plugin for adding attributes from SQL

Hi all, having used NiFi for a couple days I wanted to add some attributes to a FlowFile while not altering the contents of that FlowFile. I had suggestions to use a script processor but that just sounded like a hack which could become a nuisance to replicate. Anyway, I figured I'd write a