Re: LookupService + flowfile attributes

2018-06-08 Thread Koji Kawamura
Thanks Mike for starting the discussion. Yes, I believe that will make LookupService and Schema access strategy much easier, reusable, and useful. What I was imagined is not adding new method signature, but simply copy certain FlowFile attributes into the coordinates map. We can add that at Looku

Apache NiFi Flow Design System 0.1.0 RC1 Release Helper Guide

2018-06-08 Thread Scott Aslan
Hello Apache NiFi community, Please find the associated guidance to help those interested in validating/verifying the Apache NiFi Flow Design System release so they can vote. # Download latest KEYS file: https://dist.apache.org/repos/dist/dev/nifi/KEYS # Import keys file: gpg --import KEY

[VOTE] Release Apache NiFi Flow Design System 0.1.0

2018-06-08 Thread Scott Aslan
Hello, I am pleased to be calling this vote for the source release of Apache NiFi Flow Design System 0.1.0. The source zip, including signatures, etc. can be found at: https://dist.apache.org/repos/dist/dev/nifi/nifi-fds/nifi-fds-0.1.0/ The Git tag is nifi-fds-0.1.0-RC1 The Git commit ID i

Re: Copying small portions of code form other Apache Projects

2018-06-08 Thread Joe Witt
Yes - we should reference the code being utilized. See the top level nifi NOTICE for similar examples. This is the source code NOTICE and it should be reflected there. Thanks On Fri, Jun 8, 2018 at 1:47 PM, Peter Wicks (pwicks) wrote: > I'm working on some changes and found that the exact code

Copying small portions of code form other Apache Projects

2018-06-08 Thread Peter Wicks (pwicks)
I'm working on some changes and found that the exact code I need, along with a lot of code I can make derivatives from, are buried in the Apache Ant project. I'm not going to include Ant as a dependency, just copy some code from it. Should there be some annotation on the code/license entry/anythi

Re: [EXT] Re: Primary Only Content Migration

2018-06-08 Thread Joe Witt
Certainly there are simpler approaches to take in the mean time as this case. Specifically using ListenHTTP/PostHTTP with an intentional configuration to do all merging of split items on a specific node can do the trick. For a more general purpose and consumable solution it is worth noting: - rel

Re: [EXT] Re: Primary Only Content Migration

2018-06-08 Thread Brandon DeVries
In this particular case, the easiest thing to do may be to keep track of the hostname of the original node of the file (before split and distribution). Then, just send all of the pieces back to the originator node when done, and merge there. It would be nice to have automatic queue balancing of s

LookupService + flowfile attributes

2018-06-08 Thread Mike Thomsen
On the RestLookupService PR I think Koji mentioned the idea of expanding the lookup capability to include flowfile attributes. That sort of thing would be immensely useful on two PRs I have already open for lookup service changes for ES and Mongo. Koji, add your thoughts, but what I'm thinking woul

Re: [DISCUSS] First Release of NiFi Flow Design System

2018-06-08 Thread Scott Aslan
Hey guys, It looks like all of the outstanding issues have been resolved. I will go ahead and start the RM process and kick out an RC. -Scotty On Fri, Jun 1, 2018 at 9:54 AM, Scott Aslan wrote: > Hello, > > The github repo [1] is now referenced as "Mirror of > Apache NiFi Flow Design System"

Re: [EXT] Re: Primary Only Content Migration

2018-06-08 Thread Bryan Bende
I'm also curious to know how Peter is currently handling this since he mentioned already sending back to primary node, and we don't yet have any of the functionality that Joe, Koji, and Pierre mentioned, so the only way I can think of doing that would be to use PostHttp/InvokeHttp and point directl

Re: [EXT] Re: Primary Only Content Migration

2018-06-08 Thread Pierre Villard
Hi guys, Koji is right, I initially filed NIFI-4026 to cover this kind of use case. There is a lot of challenges and ways to address this subject. Auto-balancing in queues will be a super nice way forward this goal. I think an easy first step would be to add a checkbox in RPG configuration allow