On Wed, Jul 17, 2024 at 2:31 AM Mark Miller <markrmil...@gmail.com> wrote: > And if you really > wanted to keep an immutable cluster state object, that still doesn’t mean > ZkStateReader has to use one for its cluster state structure just because > that’s what it gives out with getClusterState.
Right -- absolutely. I _think_ most consumers can deal with mutability fine; that there are relatively few consumers that require a snapshot. They should explicitly ask for that so that the snapshotting cost isn't needlessly paid. > The separation of replica states from cluster structure is useful in > addressing an efficient cluster state structure and update strategy in > ZkStateReader, if not just so that you know what you actually *need* to > update. If you get a collection worth of JSON, you have to update it all or > do some silly gymnastics to reverse engineer what the update actually is. > For me, a concurrent hashmap in ZkStateReader was better than a cluster > state object. It mapped a collection to some kind of collection state, and > in the replica state, I put an atomic integer indicating the state. Then, > you can throw out the global lock and forget about any lock or object > creation when updating replica states. You speak in the past tense as if it worked this way. Maybe you mean "would have been" and hopefully you still think so? Practically speaking, I think "ClusterState" should continue to exist but simply be mutable (containing a ConcurrentHashMap), effectively shifting some of ZkStateReader's extensive responsibilities to this collaborator. No sense in updating lots of callers needlessly. > The entire communication path can be made easily 100x+ more scalable with > changes that are just as simple and straightforward. “Oh, this is crazy. > And it’s easy to do something that’s not.” And without any brain sweat, you > end up with a system that works in parallel on independent work, transmits > state sizes close to the actual state that needs transmitting, doesn’t spam > updates that are either unnecessary or already outdated, and operates at a > designed developer drum beat rather than an arbitrary army of drummers. Then I'm looking forward to seeing you take a stab at this ;-) even if it's just a draft PR / straw-man. You can count on me for giving a code review. ~ David --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
