Re: Getting lost a bit

[Brett Ryan]

I must be an odd one, I’m more comfortable resolving conflicts from the cli ;)

The simplest way is the CSV resolution strategy. Edit each file looking for <<< 
—- >>> blocks where conflicts are presented. Keep the section you want deleting 
the other side, then git add the file. Do for all conflicts then git commit.

Fetch often to try and keep your base aligned in order to keep conflicts as 
small as possible.

> On 8 Jan 2018, at 17:03, James Wing  wrote:
> 
> Mike,
> 
> I believe you will want to rebase your branch on the latest master before
> you submit your PR.  First, update your master branch if you have not
> already done so, with commands like the following:
> 
> git checkout master
> git fetch origin
> git merge origin/master
> 
> Second, rebase your feature branch so that your commits are relative to the
> latest master:
> 
> git checkout NIFI-4731
> git rebase master
> 
> Given the recent changes in the GCP bundle, you may have some conflicts to
> resolve.  A GUI tool like IntelliJ can be very handy for resolving git
> conflicts if you are not familiar with the command-line git resolution
> process (almost nobody is).
> 
> If that does not work for you, an alternative would be to create a new
> branch from the latest master, then manually copy/paste your changes to
> make a more concise commit.
> 
> Thanks,
> 
> James
> 
> On Sun, Jan 7, 2018 at 7:16 PM, Mikhail Sosonkin > wrote:
> 
>> Hi Devs,
>> 
>> I'm trying to create a PR for this branch in my fork
>> https://github.com/nologic/nifi/tree/NIFI-4731. I see that master has
>> moved
>> on to 1.5.0 but I'd like to have the processor built for 1.4 and later.
>> It's the version we're using. However, I don't see an origin/1.4.0 based on
>> the instructions from the contributor guide. How do I go about this? Please
>> give simple instructions, I'm not exactly a git master :)
>> 
>> Thanks,
>> Mike.
>> 
>> --
>> This email may contain material that is confidential for the sole use of
>> the intended recipient(s).  Any review, reliance or distribution or
>> disclosure by others without express permission is strictly prohibited.  If
>> you are not the intended recipient, please contact the sender and delete
>> all copies of this message.
>> 

Getting lost a bit

[Mikhail Sosonkin]

Hi Devs,

I'm trying to create a PR for this branch in my fork
https://github.com/nologic/nifi/tree/NIFI-4731. I see that master has moved
on to 1.5.0 but I'd like to have the processor built for 1.4 and later.
It's the version we're using. However, I don't see an origin/1.4.0 based on
the instructions from the contributor guide. How do I go about this? Please
give simple instructions, I'm not exactly a git master :)

Thanks,
Mike.

-- 
This email may contain material that is confidential for the sole use of 
the intended recipient(s).  Any review, reliance or distribution or 
disclosure by others without express permission is strictly prohibited.  If 
you are not the intended recipient, please contact the sender and delete 
all copies of this message.

Re: Enrichment plugin for adding attributes from SQL

[Joey Frazee]

Brett, I think it’s great that you brought this up and made some specific 
suggestions, because it’s easy for people to overlook and hard to know how to 
do the right thing without that kind of feedback.

-joey

On Jan 7, 2018, 5:51 AM -0600, Brett Ryan , wrote:
>
> Probably my biggest suggestion overall for NiFi is for accessibility support, 
> it's almost non-existant and makes it really hard for folk like me who are 
> quite visually impaired. NiFi is almost exclusively a mouse driven tool, 
> which really is hard for the visually impaired.

Re: Enrichment plugin for adding attributes from SQL

[Joey Frazee]

Hey Brett,

So as for the LookupAttribute only doing a single key lookup, that was a bit of 
a coin flip on whether it makes more sense to allow multiple lookups at once, 
each with a single key, or a single lookup with multiple AND-ed constraints. I 
opted for the former since that was the use case I was going after and we don’t 
really have a principled way of doing a map or set-valued property, so we do 
multiple properties and pretend it’s a map.

I think I considered enabling a flag to choose between the two alternatives, so 
that’d be the easy, dumb thing I could do and then we could relax those 
requirements (which are correct to have at the moment). I think the difficult 
part would really be more in writing a custom validator — and that wouldn’t be 
hard either.

-joey

On Jan 7, 2018, 5:23 AM -0600, Brett Ryan , wrote:
> MongoDBLookupService can't be used with an UpdateAttribute processor though, 
> it returns a Record type, and has no required keys. The whole purpose of the 
> processor that I originally wrote was to update an attribute from SQL on a FF.
>
> The implementation of LookupAttribute requires
>
> - getRequiredKeys to return a single element set
> - getValueType to return String.class
>
> I guess what I could do is
>
> (1) Write a processor that updates an attribute on a FF from a Record based 
> LookupService. Say I call it UpdateAttributeFromRecordLookup
> (2) Write a service SQLLookupService which given a query as a lookup 
> coordinate would return a Record for the first found entry.
> (3) Write a processor that updates an attribute from a record lookup, call it 
> UpdateAttributeFromRecordLookup.
>
> It's a bit more long winded to achieve what I originally had with a single 
> processor, but I guess UpdateAttributeFromRecordLookup would have the benefit 
> of updating attributes from LookupService implementations that do not meet 
> the LookupAttribute criteria.
>
> If you think this is viable, please let me know and I'll have a look at 
> starting it tomorrow afternoon.
>
>
> > On 7 Jan 2018, at 03:23, Mike Thomsen  wrote:
> >
> > Take a look at the mongo lookup service. I think it could serve as a good
> > example here.
> > On Fri, Jan 5, 2018 at 10:49 PM Brett Ryan  wrote:
> >
> > > Looking at using a lookupservice, this doesn't seem to support sending
> > > multiple keys to the LookupService at the same time.
> > >
> > > What I was thinking of doing was implement a LookupService that took an
> > > attribute "sql.query" which would use this to evaluate the query but then
> > > pass in a map of key/value pairs for attribute-name/column-name to set the
> > > attributes.
> > >
> > > I could implement this as I imagined it to work, however it will evaluate
> > > the SQL expression multiple times for the same query on the same flow.
> > >
> > > I am also wondering why LookupService#getRequiredKeys must return a
> > > Set, yet; this set must only contain one value.
> > >
> > >
> > > On 5 Jan 2018, at 08:32, Andy LoPresto  wrote:
> > >
> > > UpdateAttribute doesn’t pull from a database, it uses static or dynamic
> > > attribute values and supports NiFi Expression Language. In your original
> > > message, you didn’t mention any database interaction, so I thought you 
> > > were
> > > just trying to accomplish "I wanted to add some attributes to a FlowFile
> > > while not altering the contents of that FlowFile”, which is indeed what
> > > UpdateAttribute does.
> > >
> > > If you need to retrieve those values from a database, as Joey mentions,
> > > the LookupService is the right tool.
> > >
> > > With your prior setup, the distributed map cache is as secure as the NiFi
> > > configuration — if using secured NiFi, the communication between that node
> > > and any other is over TLS, and within the node it’s a memory access.
> > >
> > > A big part of the NiFi philosophy is the same as the Unix philosophy —
> > > each tool should do one job very well, and to perform complicated tasks,
> > > chain those tools together. This helps with provenance reporting, usage
> > > reporting, debugging, flow development lifecycle, maintenance, etc. A
> > > processor which retrieves attributes from a database and updates the
> > > incoming flowfile with them is certainly useful in the use case you
> > > describe, but is not a generic pattern. There’s no intent to discourage
> > > custom development, and whatever makes your flow work is fine. Just
> > > explaining why you likely won’t see a solution like that in the NiFi
> > > bundled components.
> > >
> > >
> > > Andy LoPresto
> > > alopre...@apache.org
> > > *alopresto.apa...@gmail.com *
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > >
> > > On Jan 4, 2018, at 4:07 PM, Brett Ryan  wrote:
> > >
> > > Thanks Andy, how would update attribute be able to get the value from sql?
> > >
> > > 

Re: Enrichment plugin for adding attributes from SQL

[Brett Ryan]

> On 5 Jan 2018, at 10:17, Andrew Lim  wrote:
> 
> Hi Brett,
> 
> Thanks for your feedback on the “Add Processor” window.  I’m sorry you had 
> trouble identifying the right processor to use for your data flow.

As it turns out the right processor doesn't yet exist :) I'll look to create it.

> There are tags on the left of the “Add Processor” window that categorize many 
> of the processors into functional groups.  For example, selecting the 
> “attributes” tag would limit the processors listed to the those related to 
> attributes.  Additionally, there is a Filter field at the top that can also 
> be used to search processors.  More information on using tags and filters can 
> be found in the documentation [1].

While this is handy there's several problems with this approach.

Tag clouds are really hard to read for visually impaired people. I am very 
visually impaired and my eyes shake so it's hard to focus on this sort of UX. 
lists are always better. Given that, an "enrichment" tag would be handy.

> Having said that, the UX of finding/adding a processor can definitely be 
> improved.  There are two Jiras related to this effort that I am aware of 
> [2][3].  If you have the opportunity to review those proposed improvements, 
> additional thoughts/suggestions are welcomed and greatly appreciated!

Probably my biggest suggestion overall for NiFi is for accessibility support, 
it's almost non-existant and makes it really hard for folk like me who are 
quite visually impaired. NiFi is almost exclusively a mouse driven tool, which 
really is hard for the visually impaired.

I would imagine a categorical type browser similar to that of Atlassian 
Confluence macro browser would be a better approach.


The description field can be very hard to read when text is long, the tooltip 
thing doesn't help much, it would be better for this box to scroll.

Source IMHO isn't really relevant for the non-programmer or someone that didn't 
install the NAR. End users should be abstracted from the implementation of 
where the processors came from and good organisational standards would better 
support this.

One bug that's present is a large amount of whitespace between the description 
and dialog buttons.

And finally CamelCase is hard to read for variable width fonts, processor 
names, it would be better for a label name to be used.

> [1] 
> https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#adding-components-to-the-canvas
> [2] https://issues.apache.org/jira/browse/NIFI-3338
> [3] https://issues.apache.org/jira/browse/NIFI-4249
> 
> -Drew
> 
> 
>> On Jan 4, 2018, at 4:33 PM, Brett Ryan  wrote:
>> 
>> Ooo, i shall take a look at this, that sounds great.
>> 
>> Yeah, my inexperience is probably a sore point. You know what would be 
>> great, either in the add processor browser to have categories to find 
>> processors. Trying to find enrichment processors only is probably the 
>> hardest part of identifying the right processor.
>> 
>> Where I’m working they’re on nifi 1.1 and iirc they had > 200 processors. 
>> It’s also possible that newer processors are available in newer versions.
>> 
>> 
>>> On 5 Jan 2018, at 04:01, Joey Frazee  wrote:
>>> 
>>> Andy, Brett,
>>> 
>>> Taking a quick glance at the code it looks like it's enriching attributes 
>>> from a database according to a query.
>>> 
>>> If that's correct, there's a LookupAttribute processor that delegates 
>>> lookups to a "LookupService" and adds attributes without altering content. 
>>> There are a variety of these LookupServices included. I think what you 
>>> implemented would make sense as a LookupService and then you could just 
>>> configure the processor to use that. It could also be used with 
>>> LookupRecord then too so there'd be a double payoff.
>>> 
>>> -joey
>>> 
 On Jan 4, 2018, 8:44 AM -0800, Andy LoPresto , wrote:
 Hi Brett,
 
 It’s great that you found it easy to write a new processor for Apache 
 NiFi. It is probably an indicator that we need to improve 
 education/evangelism/documentation, however, that you did not find 
 UpdateAttribute [1], which should do exactly what you were looking for.
 
 [1] 
 https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/org.apache.nifi.processors.attributes.UpdateAttribute/index.html
 
 Andy LoPresto
 alopre...@apache.org
 alopresto.apa...@gmail.com
 PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
 
> On Jan 4, 2018, at 7:03 AM, Brett Ryan  wrote:
> 
> Hi all, having used NiFi for a couple days I wanted to add some 
> attributes to a FlowFile while not altering the contents of that FlowFile.
> 
> I had suggestions to use a script processor but that just sounded like a 
> hack which could become a nuisance to replicate.
> 

Re: Enrichment plugin for adding attributes from SQL

[Brett Ryan]

MongoDBLookupService can't be used with an UpdateAttribute processor though, it 
returns a Record type, and has no required keys. The whole purpose of the 
processor that I originally wrote was to update an attribute from SQL on a FF.

The implementation of LookupAttribute requires

- getRequiredKeys to return a single element set
- getValueType to return String.class

I guess what I could do is

(1) Write a processor that updates an attribute on a FF from a Record based 
LookupService. Say I call it UpdateAttributeFromRecordLookup
(2) Write a service SQLLookupService which given a query as a lookup coordinate 
would return a Record for the first found entry.
(3) Write a processor that updates an attribute from a record lookup, call it 
UpdateAttributeFromRecordLookup.

It's a bit more long winded to achieve what I originally had with a single 
processor, but I guess UpdateAttributeFromRecordLookup would have the benefit 
of updating attributes from LookupService implementations that do not meet the 
LookupAttribute criteria.

If you think this is viable, please let me know and I'll have a look at 
starting it tomorrow afternoon.


> On 7 Jan 2018, at 03:23, Mike Thomsen  wrote:
> 
> Take a look at the mongo lookup service. I think it could serve as a good
> example here.
> On Fri, Jan 5, 2018 at 10:49 PM Brett Ryan  > wrote:
> 
>> Looking at using a lookupservice, this doesn't seem to support sending
>> multiple keys to the LookupService at the same time.
>> 
>> What I was thinking of doing was implement a LookupService that took an
>> attribute "sql.query" which would use this to evaluate the query but then
>> pass in a map of key/value pairs for attribute-name/column-name to set the
>> attributes.
>> 
>> I could implement this as I imagined it to work, however it will evaluate
>> the SQL expression multiple times for the same query on the same flow.
>> 
>> I am also wondering why LookupService#getRequiredKeys must return a
>> Set, yet; this set must only contain one value.
>> 
>> 
>> On 5 Jan 2018, at 08:32, Andy LoPresto  wrote:
>> 
>> UpdateAttribute doesn’t pull from a database, it uses static or dynamic
>> attribute values and supports NiFi Expression Language. In your original
>> message, you didn’t mention any database interaction, so I thought you were
>> just trying to accomplish "I wanted to add some attributes to a FlowFile
>> while not altering the contents of that FlowFile”, which is indeed what
>> UpdateAttribute does.
>> 
>> If you need to retrieve those values from a database, as Joey mentions,
>> the LookupService is the right tool.
>> 
>> With your prior setup, the distributed map cache is as secure as the NiFi
>> configuration — if using secured NiFi, the communication between that node
>> and any other is over TLS, and within the node it’s a memory access.
>> 
>> A big part of the NiFi philosophy is the same as the Unix philosophy —
>> each tool should do one job very well, and to perform complicated tasks,
>> chain those tools together. This helps with provenance reporting, usage
>> reporting, debugging, flow development lifecycle, maintenance, etc. A
>> processor which retrieves attributes from a database and updates the
>> incoming flowfile with them is certainly useful in the use case you
>> describe, but is not a generic pattern. There’s no intent to discourage
>> custom development, and whatever makes your flow work is fine. Just
>> explaining why you likely won’t see a solution like that in the NiFi
>> bundled components.
>> 
>> 
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com  
>> >*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
>> On Jan 4, 2018, at 4:07 PM, Brett Ryan > > wrote:
>> 
>> Thanks Andy, how would update attribute be able to get the value from sql?
>> 
>> Consider a flow where a piece of information needs to be obtained from a
>> DB but i do not want the contents of the current FF to be altered, using
>> ExecuteSQL anywhere prior would not be possible due to replacing the FF
>> contents.
>> 
>> What I had was two seperate flows, one that updates an oauth key in a db
>> keeping it fresh, the main flow would then read the db just before an
>> invokehttp.
>> 
>> I was originally using a distributed map cache but had concerns that it
>> might not be secure and was also advised the cache server has been known to
>> go down.
>> 
>> On 5 Jan 2018, at 04:01, Joey Frazee > > wrote:
>> 
>> Andy, Brett,
>> 
>> Taking a quick glance at the code it looks like it's enriching attributes
>> from a database according to a query.
>> 
>> If that's correct, there's a LookupAttribute processor that delegates
>> lookups to a