Robert, Can you please detail the record readers and writers involved and how schemas are accessed? There can be very important performance related changes in the parsers/serializers of the given formats. And we've added a lot to make schema caching really capable but you have to opt into it. It is of course possible MergeRecord itself is the culprit for performance reduction but lets get a more full picture here.
Are you able to share a template and sample data which we can use to replicate? Thanks On Fri, Apr 24, 2020 at 4:38 PM Robert R. Bruno <[email protected]> wrote: > I wanted to see if anyone else has experienced performance issues with the > newest version of nifi and MergeRecord? We have been running on nifi 1.9.2 > for awhile now, and recently upgraded to nifi 1.11.4. Once upgraded, our > identical flows were no longer able to keep up with our data mainly at > MergeRecord processors. > > We ended up downgrading back to nifi 1.9.2. Once we downgraded, all was > keeping up again. There were no errors to speak of when we were running > the flow with 1.11.4. We did see higher load on the OS, but this may have > been caused by the fact there was such a tremendous backlog built up in the > flow. > > Another side note, we saw one UpdateRecord processor producing errors when > I tested the flow with nifi 1.11.4 with a small test flow. I was able to > fix this issue by changing some parameters in my RecordWriter. So perhaps > some underlying ways records are being handled since 1.9.2 caused the > performance issue we saw? > > Any insight anyone has would be greatly appreciated, as we very much would > like to upgrade to nifi 1.11.4. One thought was switching the MergeRecord > processors to MergeContent since I've been told MergeContent seems to > perform better, but not sure if this is actually true. We are using the > pattern of chaining a few MergeRecord processors together to help with > performance. > > Thanks in advance! >
