I like the idea. I'm thinking about how it would be implemented in the UI (and API). Specifically, I'm comparing this feature of setting the prioritizer to setting default connection properties via the configuration of a process group. For the default connection properties, updates to the process group configuration only change the default settings for any new connection created; it does not affect existing connections. This was intentional because it seemed heavy-handed to apply such settings to all existing connections - especially expiration settings which could result in data loss.
However, the recommendation here is for actively changing the prioritizer in all existing connections (and potentially nested connections in child process groups). I understand the benefit and use case, but it seems the two modification styles (new connections or existing connections) would easily become confused. Would a checkbox for "apply prioritizer to all existing connections" be appropriate? And, if so, we still need to somehow make it clear that this applies to just prioritizer settings. I do not believe we want the other connection settings to be recursively applied to existing connections. Do we want to introduce a new context menu item for process groups, "Connection Settings" or something similar? Is there a JIRA ticket for this proposed new feature? On Tue, Apr 12, 2022 at 11:48 PM Ryan Hendrickson < [email protected]> wrote: > @Bryan - Correct, everything is still per queue, just with that convenience > feature. > > Totally agree with @Salvatore too. I hadn't even thought of nested process > groups. > > On Tue, Apr 12, 2022 at 10:14 PM Salvatore <[email protected]> > wrote: > > > #2 would definitely be convenient. Maybe also include an option whether > to > > recurse down through nested process groups, or just apply to the selected > > process group. > > > > On Wed, 13 Apr 2022 at 06:05, Bryan Bende <[email protected]> wrote: > > > > > Makes sense. For # 2, it is still per queue with an "Apply All" > > > convenience right? Just trying to differentiate with prioritizing > > > across all queues. > > > > > > On Tue, Apr 12, 2022 at 3:22 PM Ryan Hendrickson > > > <[email protected]> wrote: > > > > > > > > I see two things as particularly useful... > > > > > > > > 1) Default Prioritizer for new Relationships (Bound to a process > > group, > > > > similar to how the "Default FlowFile Expiration" can be changed). > > > > 2) Applying a prioritizer to an entire Process Group as a one-time > > > action. > > > > > > > > Some background... I'm hand-converting two super-legacy v0.7.3 > canvases > > > to > > > > v1.15.3. Part of that is applying flow priorities all over the place > > in > > > > the new system. Probably not a common task, but I could see this > > feature > > > > being useful for other week-to-week work too. > > > > > > > > Ryan > > > > > > > > On Tue, Apr 12, 2022 at 1:32 PM Bryan Bende <[email protected]> > wrote: > > > > > > > > > I think there are two different concepts here... The original > > > > > discussion is just about default settings for new connections. The > > > > > idea in NIFI-6831 is about prioritizing data across multiple > queues, > > > > > either for all of nifi or within a given process group. > > > > > > > > > > > > > > > On Tue, Apr 12, 2022 at 1:13 PM Mark Bean <[email protected]> > > > wrote: > > > > > > > > > > > > We experimented with the idea of a custom "Global Prioritizer". > One > > > of > > > > > the > > > > > > problems with this approach is that it ran the risk of breaking > the > > > > > > multi-tenancy philosophy. If there were a truly global priority, > it > > > would > > > > > > affect all flows, each may have different priority rules. > However, > > if > > > > > this > > > > > > could be applied only at the process group level, it might have > > legs. > > > > > > > > > > > > You can follow the initial approach to such a mechanism in the > JIRA > > > > > ticket. > > > > > > https://issues.apache.org/jira/browse/NIFI-6831 > > > > > > > > > > > > > > > > > > On Tue, Apr 12, 2022 at 12:06 PM Ryan Hendrickson < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > I just went to the config button in my process group, hoping to > > > set all > > > > > > > relationships in there to priority first.... Lots of right > > > clicking & > > > > > > > dragging instead. > > > > > > > > > > > > > > +1 for an approach like that. > > > > > > > > > > > > > > On Tue, Apr 12, 2022 at 11:44 AM Joe Witt <[email protected]> > > > wrote: > > > > > > > > > > > > > > > Hello > > > > > > > > > > > > > > > > Certainly the spirit of this is a good idea. Would likely > need > > > to > > > > > > > approach > > > > > > > > it at a more flow/process group centric level. > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson < > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > This would be very helpful. > > > > > > > > > > > > > > > > > > Ryan > > > > > > > > > > > > > > > > > > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss < > > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > Do you see much value in being able to specify an > > > instance-wide > > > > > (or > > > > > > > > > > cluster-wide) default prioritizer for all connections > that > > > do not > > > > > > > have > > > > > > > > > one > > > > > > > > > > manually set? > > > > > > > > > > > > > > > > > > > > Along with the the following properties in > nifi.properties: > > > > > > > > > > nifi.queue.backpressure.count=10000 > > > > > > > > > > nifi.queue.backpressure.size=1 GB > > > > > > > > > > > > > > > > > > > > I'd like to see see something like > > > > > > > > > > > nifi.queue.prioritizer.default=org.apache.nifi.prioritize. > > > > > > > > > > PriorityAttributePrioritizer > > > > > > > > > > > > > > > > > > > > Thoughts? My only concern would be if connection > > prioritizers > > > > > have a > > > > > > > > > > noticeable impact on system resources. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
