IMO the concept of "functions" is different enough to warrant its own set of components, rather than trying to shoehorn them into a wormhole / PG discussion. Perhaps a Function component which may feel like a PG (but with UI distinction such as rounded corners?), but would be referred to by name (uniqueness to be enforced by the framework), possibly with a Function Input Port (a single instance also enforced) and Function Output Port (single instance) located inside the Function (rather than an Input Port outside linked to an Input Port inside). Invocation of the Function would be done with some other component akin to an Output Port or PG, possibly called a Function Call. Function Call could act like a processor in the sense that it would send flow files (with the return destination attached for use by the Function Output Port) to the Function's Input Port and receive them when available, then send them to downstream connection(s). Best practice may be to put all Functions in a single PG to isolate them, but because invocation is done by unique name, we wouldn't need a "wormhole", rather the framework can keep a reference to where the Function is located, and thus send the flow files from the Function Call directly to the Function. If this entails some of the same work to implement wormholes then we get two birds with one stone :) Thoughts?
Regards, Matt On Sun, May 13, 2018 at 3:31 PM, Ed B <bdes...@gmail.com> wrote: > Joe, Aldrin, > Wormholes is pretty interesting thing. I played around with that and could > make it working. Though, this approach has downsides. > I'll create an article for this, but you can take a look at it now > (attaching template for root canvas). > > So, what I've found while playing around this topic, is that removing > restrictions for remote input/output port being on root canvas only would be > nice, but not sufficient. > When we distribute flowfiles over the nodes within the same cluster - we > need to make it easy to indicate, so RPG will be using properties of the > cluster, instead of manually provided ones. I would even go further to add > distribution capabilities on relationship level. That would really reduce > amount of entities we put into our flows, and reduce complexity. > > > On Sun, May 13, 2018 at 1:20 PM Aldrin Piri <aldrinp...@gmail.com> wrote: >> >> I think what you highlighted is kind of how I had it worked out in my >> mind. Although maybe I read too much into the description of the proposal >> about the framework managing context. In terms of what we have now, I >> think I pictured this to be "Tag this data as from this source" and then >> when leaving such a group, the framework would send it back to that "tag." >> >> I will avoid showing my blissful ignorance of all the internals by saying >> how it could work but will try to draw the analogs from functionality >> currently in place. I imagined feeding the reference-able group similar >> to >> a virtual funnel of sorts where we use framework knowledge of the >> connection to it (and perhaps said connection's source) to track that >> state >> in shipping it back via some slightly smarter port that is, in effect, a >> router back to virtual ports (wormholes?) to where the data came from. >> Or, >> perhaps, in more concrete terms: >> >> We have >> * a Process Group has several input ports (source processors), >> * that all feed an UpdateAttribute which tags each flowfile as the source >> via EL, >> * carry out the functions of the referenceable group, >> * with the end of this "block" feeding a RouteOnAttribute on this tag to >> an equivalent number of output ports. >> >> On Sun, May 13, 2018 at 12:20 PM, Joe Witt <joe.w...@gmail.com> wrote: >> >> > Aldrin >> > >> > Referencable groups would have to work like a single instance of a PG in >> > terms of flow definition but caller specific instances in reality. >> > Otherwise youd have no way to avoid cross contaminating flowfiles from >> > various callers as thered be no caller specific stack (in our case >> > caller >> > specific queues and other resources). >> > >> > The point about keeping versions of instances up to date with registry >> > based versioned instances is true but can be addressed with auto >> > updating >> > instances of versioned flows which we will need to add anyway. >> > >> > In either case having PG operate like a callable function reusable >> > across >> > flows will likely need to operate as mentioned above. The former being >> > less consistent with the user experience and more work than the latter. >> > >> > Do you see some other way to make referencable groups work. >> > >> > Wormhole connections need to be implemented for sure to help keep flows >> > concise. >> > >> > Thanks >> > Joe >> > >> > On Sun, May 13, 2018, 11:42 AM Aldrin Piri <aldrinp...@gmail.com> wrote: >> > >> > > I think the Registry solves part of the issue but even that would lead >> > > to >> > > duplication of units where we are "copying and pasting" the "code." >> > > Versioning would aid in keeping all components in lock step, but will >> > > not >> > > remedy manual intervention with n-many instances of them. After one >> > > was >> > > altered, there would still be the manual process where the PGs would >> > > each >> > > need to be updated when that change was committed and changes were >> > realized >> > > after some time delta. >> > > >> > > I think the previously discussed Reference-able Process Groups [1] are >> > > likely better aligned in conjunction with the Wormhole Connections >> > > [2]. >> > > >> > > [1] https://cwiki.apache.org/confluence/display/NIFI/ >> > > Reference-able+Process+Groups >> > > [2] https://cwiki.apache.org/confluence/display/NIFI/Wormhole+Co >> > nnections >> > > >> > > >> > > >> > > On Sat, May 12, 2018 at 10:19 PM, Joe Witt <joe.w...@gmail.com> wrote: >> > > >> > > > Scott >> > > > >> > > > Youre very right there must be a better way. The flow registry with >> > > > versioned flows is the answer. You can version control the common >> > logic >> > > > and reuse it in as many instances as you need. >> > > > >> > > > This is like having a flow Class in java terms where you can >> > instantiate >> > > as >> > > > many objects of that type Class you need. >> > > > >> > > > It was definitely a long missing solution that was addressed in nifi >> > > 1.5.0 >> > > > and with the flow registry. >> > > > >> > > > Also, we should just remove the root group remote port limitation. >> > > > It >> > > was >> > > > an implementation choice long before we had multi tenant auth and >> > > > now >> > it >> > > no >> > > > longer makes sense to force root group only. Still though the above >> > > > scenario of versioned flows and the flow registry solves the main >> > > problem. >> > > > >> > > > >> > > > thanks >> > > > >> > > > On Sat, May 12, 2018, 9:22 PM Charlie Meyer < >> > > > charlie.me...@civitaslearning.com> wrote: >> > > > >> > > > > We do this often by leveraging the variable registery and the >> > > expression >> > > > > language to make components be more dynamic and reusable >> > > > > >> > > > > -Charlie >> > > > > >> > > > > On Sat, May 12, 2018, 20:01 scott <tcots8...@gmail.com> wrote: >> > > > > >> > > > > > Hi Devs, >> > > > > > >> > > > > > I've got a question about an observation I've had while working >> > with >> > > > > > NiFi. Is there a better way to re-use process groups similar to >> > > > > > how >> > > > > > programming languages reference functions, libraries, classes, >> > > > > > or >> > > > > > pointers. I know about remote process groups and templates, but >> > > neither >> > > > > > do exactly what I was thinking. RPGs are great, but I think the >> > > output >> > > > > > goes to the root canvas level, and you have to have have >> > > > > > connectors >> > > all >> > > > > > the way back up your flow hierarchy, and that's not practical. >> > > > > > Ultimately, I'm looking for an easy way to re-use process groups >> > that >> > > > > > contain common logic in many of my flows, so that I reduce the >> > amount >> > > > of >> > > > > > places I have to change. >> > > > > > >> > > > > > Hopefully that made sense. Appreciate your thoughts. >> > > > > > >> > > > > > Scott >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> >
