[ https://issues.apache.org/jira/browse/NIFI-7501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Thomsen resolved NIFI-7501. -------------------------------- Fix Version/s: 1.12.0 Resolution: Fixed > Generate Flowfile does not scale > -------------------------------- > > Key: NIFI-7501 > URL: https://issues.apache.org/jira/browse/NIFI-7501 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions > Affects Versions: 1.11.4 > Reporter: Dennis Jaheruddin > Priority: Minor > Fix For: 1.12.0 > > Attachments: generationperformance.xml > > Time Spent: 20m > Remaining Estimate: 0h > > One of the purposes of Generate Flowfile is load testing. However, > unfortunately it often appears to become the bottleneck itself. I have found > it not to scale well. > Example result from my laptop: > I want to generate messages and bring them to a single processor, lets call > it processor X. > With 1 concurrent task, and a batch size of 1, and a message size of 10MB and > uniqueness false it can generate approximately 2 GB/sec. > When allowing for more concurrent tasks, or a larger batch size, no > noticeable change is found. > However, if instead of increasing the batchsize I route the success > relationship to multiple processors that do 'nothing' (like updateattribute), > and then bring the success relations of all these to processor X, I can get > much more than 2 GB/sec. > > In conclusion: I don't appear to be hitting a hardware limit as I am able to > generate the number of messages in this inelegant way, but no matter how I > set up my generateflowfile processor, it just will not scale. Suggesting > there may be a smarter way to generate data when uniqueness is not required. > > I have attached a template to illustrate my findings. -- This message was sent by Atlassian Jira (v8.3.4#803005)