thanks for the explanation, Bryan! it helps! Boris
On Thu, Nov 29, 2018 at 12:26 PM Bryan Bende <[email protected]> wrote: > Boris, > > Yes the "distributed" name is confusing... it is referring to the fact > that it is a cache that can be accessed across the cluster, rather > than a local cache on each node, but you are correct that that DMC > server is a single point of failure. > > It is important to separate the DMC client and server, there are > multiple implementations of the DMC client that can interact with > different caches (Redis, HBase, etc), the trade-off being you then > have to run/maintain these external systems, instead of the DMC server > which is fully managed by NiFi. > > Regarding ZK... I don't think there is a good answer other than the > fact that DMC existed when NiFi was open sourced, and NiFi didn't > start using ZK for clustering until the 1.0.0 release, so originally > ZK wasn't in the picture. I assume we could implement a DMC client > that talked to ZK, just like we have done for Redis, HBase, and > others. > > I'm not aware of any issues with the DMC server persisting to file > system or handling concurrent connections, it should be stable. > > Thanks, > > Bryan > > On Thu, Nov 29, 2018 at 11:52 AM Boris Tyukin <[email protected]> > wrote: > > > > Hi guys, > > > > I have a few questions about DistributedMapCacheServer. > > > > First question, I am confused by "Distributed" part. If I get it, the > server actually runs on a single node and if it fails, it is game over. Is > that right? Why NiFi is not using ZK for that since ZK is already used by > NiFi cluster? I see most of the use cases / examples are about using > DistributedMapCacheServer as a lookup or state store and this is exactly > what ZK was designed for and provides redundancy, scalability and 5-10k ops > per sec on 3 node ZK cluster. > > > > Second question, I did not find any tools to interact with it other than > Matt's groovy tool. > > > > Third question, how DistributedMapCacheServer that persists to file > system, handles concurrency and locking? Is it reliable and can be trusted? > > > > And lastly, is there additional overhead to support > DistributedMapCacheServer as another system or it is pretty much hands off > once a controller is set up? > > > > Thanks! > > Boris >
