The DistributedMapCacheServer is, I believe, meant as a reference implementation of the service to be used as an example rather than in production. The kind of scenario you describe is exactly the reason to not use this in-memory (optionally locally persisted on disk) in a clustered production environment.
That said, it can be used if the use case of the Flow doesn't have problems if a node goes offline, etc. The recommended approach is to use an external service such as Redis with the RedisDistributedMapCacheClientService [1]. This can interface with your external Redis cluster/instance using the same API. Other external services can be used, see the selection of related Controller Services in the nifi docs [2] (e.g. search for "cache"). [1]: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-redis-nar/1.18.0/org.apache.nifi.redis.service.RedisDistributedMapCacheClientService/index.html [2]: https://nifi.apache.org/docs/nifi-docs On Fri, 14 Oct 2022, 17:13 Jörg Hammerbacher, <[email protected]> wrote: > Hi, > > > I have one thing where i am looking for a solution. Maybe someone can > help me out or give me hint how to do. > > > Problem: > > I often use a NiFi Clusters with "DistributedMapCacheClientService" > which uses a "DistributedMapCacheServer" for cluster wide key/value > storage. Per default the DMCS uses "in memory" and sockets for > synchronization. We use a persistence directory to make the data > persistent and to avoid that the data is gone after restarting the > entire cluster. But in the case, if the primary node changes, i think > the data will be outdated or used from a potential outdated other node. > If this other Node takes the primary node role, old data will be used > for next FecthDistrubutedMapCache. The latest updates over the old > primar node are gone. > > Is there a service using e.g. zookeeper "int the backgroud" to get a > real distributed persitent Cache - even after restarting the entire > cluster / all nodes? > > > I know, the standard cache is able to provide a hugh frequent > read/update servise if the data is in memory. But if we need just one or > max a few updates per minute ... > > Yes, using another system like a Database (as persistent singleton) can > be a solution - a not really matching solution. Why is there no standard > service in NiFi for this? Isn't it a good idea or i am the only one with > this problem in the past? > > > Thanks in advance for answers, > > Jörg (Hammerbacher) > > >
