Brandon DeVries created NIFI-4871:

             Summary: Data Queuing at Process Group Input Port
                 Key: NIFI-4871
             Project: Apache NiFi
          Issue Type: Bug
            Reporter: Brandon DeVries

When a flow is under heavy load, we've seen data queue on the way in to a 
process group.  There is nothing queued *inside* the process group, and no back 
pressure... it simply isn't moving files fast enough.  As a workaround, 
multiple input ports were added... but that's ugly.

The first issue may be the limit of 100 files transferred on any call to 
onTrigger()[1].  100 seems low, ideally it would be nice to fill to the back 
pressure point (or some reasonable limit if set to "0").  This may also be 
compounded by some scheduling issues we've seen, but not reliably enough to 
actually diagnose.  But the combination of not transferring enough files on 
each run and not running as often as it "should" is definitely causing issues.  
The scheduling issue may be complex, but transferring more files should be 
pretty straightforward...




This message was sent by Atlassian JIRA

Reply via email to