I sent this from the wrong email account a few days ago, but am still interested in any thoughts.
Got an interesting message that prompted me to follow up on a few questions I have. The message is the screenshot below (if it works). It says: "WARNING The rate of the dataflow is exceeding the provenance recording rate. Slowing down flow to accommodate." [image: Inline image 1] This flow has queued up a few hundred thousand files (mostly very small) and I'm not sure that's ideal. I read that there is some automatic swapping that takes place at 20k file queues. It does eventually process the files, but I would like to make sure we're taking advantage of any performance options. 1. Would it be more efficient to let the files queue up, or to try to match the process rate with timing or back pressure? 2. I've made the suggested system edits in the administrator's guide, as well as increasing xms and xmx somewhat. Any additional suggestions? 3. Somewhat related, I think that logging / provenance is eating up disk space. My install directory stays large, even after processing has finished, until I stop and start NiFi which significantly reduces the disk in use. 4. Maybe unrelated, what does the number that appears in a little white box at the top right of a processor indicate? It seems to show up on processors that have a large queue in front of them. [image: Inline image 2] Thanks, Charlie
