Hi Brian, I'm not sure what can really be done with the existing processor besides what you have already done. Have you configured your overall Timer Driven thread pool appropriately?
Most likely there would need to be a new PutRedis processor that didn't have to adhere to the DistributedMapCacheInterface and could use MSET or whatever specific Redis functionality was needed. Another option might be a record-based variation of PutDistributedMapCache where you could keep thousands of records together and stream them to the cache. It would take a record-path to specify the key for each record and serialize the record as the value (assuming your data fits into one of the record formats like JSON, Avro, CSV). -Bryan On Tue, Mar 31, 2020 at 4:23 PM Hesselmann, Brian <[email protected]> wrote: > Hi, > > We currently run a flow that puts about 700.000 entries/flowfiles into > Redis every 5 minutes. I'm looking for ways to improve performance. > > Currently we've been upping the number of concurrent tasks and run > duration of the PutDistributedMapCache processor to be able to process > everything. I know Redis supports setting multiple keys at once using MSET( > https://redis.io/commands/mset), however using Nifi this command is not > available. > > Short of simply upgrading the system we run Nifi/Redis on, do you have any > suggestions for improving performance of PutDistributedMapCache? > > Best, > Brian >
