MongoDBLookupService can't be used with an UpdateAttribute processor though, it returns a Record type, and has no required keys. The whole purpose of the processor that I originally wrote was to update an attribute from SQL on a FF.
The implementation of LookupAttribute requires - getRequiredKeys to return a single element set - getValueType to return String.class I guess what I could do is (1) Write a processor that updates an attribute on a FF from a Record based LookupService. Say I call it UpdateAttributeFromRecordLookup (2) Write a service SQLLookupService which given a query as a lookup coordinate would return a Record for the first found entry. (3) Write a processor that updates an attribute from a record lookup, call it UpdateAttributeFromRecordLookup. It's a bit more long winded to achieve what I originally had with a single processor, but I guess UpdateAttributeFromRecordLookup would have the benefit of updating attributes from LookupService implementations that do not meet the LookupAttribute criteria. If you think this is viable, please let me know and I'll have a look at starting it tomorrow afternoon. > On 7 Jan 2018, at 03:23, Mike Thomsen <mikerthom...@gmail.com> wrote: > > Take a look at the mongo lookup service. I think it could serve as a good > example here. > On Fri, Jan 5, 2018 at 10:49 PM Brett Ryan <brett.r...@gmail.com > <mailto:brett.r...@gmail.com>> wrote: > >> Looking at using a lookupservice, this doesn't seem to support sending >> multiple keys to the LookupService at the same time. >> >> What I was thinking of doing was implement a LookupService that took an >> attribute "sql.query" which would use this to evaluate the query but then >> pass in a map of key/value pairs for attribute-name/column-name to set the >> attributes. >> >> I could implement this as I imagined it to work, however it will evaluate >> the SQL expression multiple times for the same query on the same flow. >> >> I am also wondering why LookupService#getRequiredKeys must return a >> Set<String>, yet; this set must only contain one value. >> >> >> On 5 Jan 2018, at 08:32, Andy LoPresto <alopre...@apache.org> wrote: >> >> UpdateAttribute doesn’t pull from a database, it uses static or dynamic >> attribute values and supports NiFi Expression Language. In your original >> message, you didn’t mention any database interaction, so I thought you were >> just trying to accomplish "I wanted to add some attributes to a FlowFile >> while not altering the contents of that FlowFile”, which is indeed what >> UpdateAttribute does. >> >> If you need to retrieve those values from a database, as Joey mentions, >> the LookupService is the right tool. >> >> With your prior setup, the distributed map cache is as secure as the NiFi >> configuration — if using secured NiFi, the communication between that node >> and any other is over TLS, and within the node it’s a memory access. >> >> A big part of the NiFi philosophy is the same as the Unix philosophy — >> each tool should do one job very well, and to perform complicated tasks, >> chain those tools together. This helps with provenance reporting, usage >> reporting, debugging, flow development lifecycle, maintenance, etc. A >> processor which retrieves attributes from a database and updates the >> incoming flowfile with them is certainly useful in the use case you >> describe, but is not a generic pattern. There’s no intent to discourage >> custom development, and whatever makes your flow work is fine. Just >> explaining why you likely won’t see a solution like that in the NiFi >> bundled components. >> >> >> Andy LoPresto >> alopre...@apache.org >> *alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> >> <alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com>>* >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> On Jan 4, 2018, at 4:07 PM, Brett Ryan <brett.r...@gmail.com >> <mailto:brett.r...@gmail.com>> wrote: >> >> Thanks Andy, how would update attribute be able to get the value from sql? >> >> Consider a flow where a piece of information needs to be obtained from a >> DB but i do not want the contents of the current FF to be altered, using >> ExecuteSQL anywhere prior would not be possible due to replacing the FF >> contents. >> >> What I had was two seperate flows, one that updates an oauth key in a db >> keeping it fresh, the main flow would then read the db just before an >> invokehttp. >> >> I was originally using a distributed map cache but had concerns that it >> might not be secure and was also advised the cache server has been known to >> go down. >> >> On 5 Jan 2018, at 04:01, Joey Frazee <joey.fra...@icloud.com >> <mailto:joey.fra...@icloud.com>> wrote: >> >> Andy, Brett, >> >> Taking a quick glance at the code it looks like it's enriching attributes >> from a database according to a query. >> >> If that's correct, there's a LookupAttribute processor that delegates >> lookups to a "LookupService" and adds attributes without altering content. >> There are a variety of these LookupServices included. I think what you >> implemented would make sense as a LookupService and then you could just >> configure the processor to use that. It could also be used with >> LookupRecord then too so there'd be a double payoff. >> >> -joey >> >> On Jan 4, 2018, 8:44 AM -0800, Andy LoPresto <alopre...@apache.org >> <mailto:alopre...@apache.org>>, >> wrote: >> Hi Brett, >> >> It’s great that you found it easy to write a new processor for Apache >> NiFi. It is probably an indicator that we need to improve >> education/evangelism/documentation, however, that you did not find >> UpdateAttribute [1], which should do exactly what you were looking for. >> >> [1] >> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/org.apache.nifi.processors.attributes.UpdateAttribute/index.html >> >> <https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/org.apache.nifi.processors.attributes.UpdateAttribute/index.html> >> >> Andy LoPresto >> alopre...@apache.org <mailto:alopre...@apache.org> >> alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> On Jan 4, 2018, at 7:03 AM, Brett Ryan <brett.r...@gmail.com> wrote: >> >> Hi all, having used NiFi for a couple days I wanted to add some attributes >> to a FlowFile while not altering the contents of that FlowFile. >> >> I had suggestions to use a script processor but that just sounded like a >> hack which could become a nuisance to replicate. >> >> Anyway, I figured I'd write a processor to do this, anyone interested you >> can find it here. >> >> https://github.com/brettryan/nifi-drunken-bundle >> >> Feel free to do with it what you please. >> >> I've published to maven central but it will take a day to appear in the >> search. >> >> <dependency> >> <groupId>com.drunkendev</groupId> >> <artifactId>nifi-drunken-nar</artifactId> >> <version>1.0.0</version> >> <type>nar</type> >> </dependency> >> <dependency> >> <groupId>com.drunkendev</groupId> >> <artifactId>nifi-drunken-processors</artifactId> >> <version>1.0.0</version> >> </dependency> >> <dependency> >> <groupId>com.drunkendev</groupId> >> <artifactId>nifi-drunken-bundle</artifactId> >> <version>1.0.0</version> >> <type>pom</type> >> </dependency>
signature.asc
Description: Message signed with OpenPGP
