Our team is considering ways to accelerate delivery for certain subsets of content we process through NiFi. We are using Apache NiFi 0.7.x as our baseline.
This link discusses a recommended approach to content prioritization using PriorityAttributePrioritizer on a connector (queue) to tailor throughput based on a priority attribute we set upstream in our flow: https://stackoverflow.com/questions/42528993/how-to-specify-priority-attributes-for-individual-flowfiles How often does the connector queue have to re-sort contents in order to enforce our priority attribute? Is it re-sorting *every *single time new flowFiles hit the queue? Won't that markedly and negatively impact performance? If our priority 1s are a huge volume of flowfiles that persists over time, won't this approach cause our priority 2s, 3s, etc etc to languish in queue? The described approach seems to embed significant business logic in the NiFi workflows. In an environment where priorities change often, would that be considered a poor approach? Might it be better to enforce priority processing at a higher architectural level - a lightweight NiFi server to accelerate delivery of priority one content and email alerts, a priority two suite of NiFi servers for standard flowfile volume, a priority three suite of servers to handle long-term bulk processing, etc etc? Thanks in advance for your help. -Jim
