Might be worth experimenting with KeyDB and see if that helps. It's a mutli-threaded fork of Redis that's supposedly about as fast in a single node as a same size Redis cluster when you compare cluster nodes to KeyDB thread pool size.
https://keydb.dev/ On Tue, Mar 31, 2020 at 4:49 PM Bryan Bende <[email protected]> wrote: > Hi Brian, > > I'm not sure what can really be done with the existing processor besides > what you have already done. Have you configured your overall Timer Driven > thread pool appropriately? > > Most likely there would need to be a new PutRedis processor that didn't > have to adhere to the DistributedMapCacheInterface and could use MSET or > whatever specific Redis functionality was needed. > > Another option might be a record-based variation of PutDistributedMapCache > where you could keep thousands of records together and stream them to the > cache. It would take a record-path to specify the key for each record and > serialize the record as the value (assuming your data fits into one of the > record formats like JSON, Avro, CSV). > > -Bryan > > On Tue, Mar 31, 2020 at 4:23 PM Hesselmann, Brian < > [email protected]> wrote: > >> Hi, >> >> We currently run a flow that puts about 700.000 entries/flowfiles into >> Redis every 5 minutes. I'm looking for ways to improve performance. >> >> Currently we've been upping the number of concurrent tasks and run >> duration of the PutDistributedMapCache processor to be able to process >> everything. I know Redis supports setting multiple keys at once using MSET( >> https://redis.io/commands/mset), however using Nifi this command is not >> available. >> >> Short of simply upgrading the system we run Nifi/Redis on, do you have >> any suggestions for improving performance of PutDistributedMapCache? >> >> Best, >> Brian >> >
