Re: Performance of adding many keys to redis with PutDistributedMapCache

Might be worth experimenting with KeyDB and see if that helps. It's a mutli-threaded fork of Redis that's supposedly about as fast in a single node as a same size Redis cluster when you compare cluster nodes to KeyDB thread pool size. https://keydb.dev/ On Tue, Mar 31, 2020 at 4:49 PM Bryan

Re: Performance of adding many keys to redis with PutDistributedMapCache

Hi Brian, I'm not sure what can really be done with the existing processor besides what you have already done. Have you configured your overall Timer Driven thread pool appropriately? Most likely there would need to be a new PutRedis processor that didn't have to adhere to the

Performance of adding many keys to redis with PutDistributedMapCache

Hi, We currently run a flow that puts about 700.000 entries/flowfiles into Redis every 5 minutes. I'm looking for ways to improve performance. Currently we've been upping the number of concurrent tasks and run duration of the PutDistributedMapCache processor to be able to process everything. I

Re: How to use delta storage format

Let me share answers from the delta community: Answer to Q1: Structured streaming queries can do commits every minute, even every 20-30 seconds. This definitely creates small files. But that is okay, because it is expected that people will periodically compact the files. The same timing should

Re: Adding Nested Properties/JSON

With Jolt transformation, you can do something like : Input { "name": "this and that", "field": "value" } Transformation 1 [ { "operation":"modify-overwrite-beta", "spec":{ "others":{ "[0]":[ { "name":"here and there"

Re: Adding Nested Properties/JSON

Sure. Thank you. Processor #1 creates this JSON { "name":"this and that", "field":"value" } passes to Processor #2 which adds a record to a sub-field { "name":"this and that", "field":"value", "others": [ {"name":"here and there"} ] } passes to Processor #3 which also

Re: Adding Nested Properties/JSON

Can you post example of input and expected result. For adding, you can use default or modify-overwrite-beta Garanti sans virus. www.avast.com

Re: Adding Nested Properties/JSON

Hi. Thank you. In looking at the Jolt docs these are the operations: shift, sort, cardinality, modify-default-beta, modify-overwrite-beta, modify-define-beta, or remove I primarily need "add" such that I can add nested elements or add elements to an array already in the JSON. Can a single

Re: Adding Nested Properties/JSON

Hello. Jolt transformation. Etienne Le mar. 31 mars 2020 à 14:40, Darren Govoni a écrit : > Hi, >I want to use Nifi to design a flow that modifies, updates, etc a > nested JSON document (or that can finally output one at the end). > > For example: > > { >"name":"this and that", >

Adding Nested Properties/JSON

Hi, I want to use Nifi to design a flow that modifies, updates, etc a nested JSON document (or that can finally output one at the end). For example: { "name":"this and that", "field":"value", "others": [ {"name":"here and there"}, ... ] } What's the best approach