[
https://issues.apache.org/jira/browse/SOLR-18507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18135183#comment-18135183
]
David Smiley commented on SOLR-18507:
-------------------------------------
I thoroughly enjoyed the presentation you gave on this topic at CoC in Glasgow!
> 5. (more of an architectural shift) scrap the fingerprint altogether.
+1 I love this direction! I strongly suspect a dense sequence number would
lead to other improvements in the indexing flow aside from the NRT fingerprint
matter. Do you think you could scope the work involved in this for Solr 11?
There will be backwards-compatibility concerns, of course.
> Make Index Fingerprint Faster in Leader Election Path
> -----------------------------------------------------
>
> Key: SOLR-18507
> URL: https://issues.apache.org/jira/browse/SOLR-18507
> Project: Solr
> Issue Type: Wish
> Reporter: Lucas Kot-Zaniewski
> Priority: Major
> Labels: newdev
>
> Leader election causes the new candidate to run PeerSync against all
> replicas. The reason is it must ensure all replicas under its stewardship are
> "document equivalent" before it can safely pronounce its leadership. Document
> equivalence is ascertained with the help of a checksum called the index
> fingerprint. The fingerprint is computed over all the document versions in
> the index and is O(# of docs in the index) in the worst case. Yes, there is a
> patchwork of caching and parallelization to speed up the average case.
> However, our alarming informs us about the \~worst case often enough for us
> to consider it a major pain point. The issue is especially noticeable when
> you have a lot of docs relative to your allocated compute. For instance, if
> you have low billions of documents on a node with ~16 cores, this computation
> can block leader elections for tens of seconds (at least in our environment).
> This amounts to a write-outage that makes Solr look bad.
> The slowness stems ~99% (literally) from decoding Solr's \_version\_ field.
> Reading doc-values at this scale is slow; it requires increment doc Id
> iterator and, more expensively, decompressing the long value from the DV
> long-point format billions of times. Note this is CPU time and not waiting
> for mmap to load into resident memory. We run on clusters with ample RAM but
> I imagine this problem may be even worse when you run more memory constrained
> processes.
> I have thought about this problem for a bit and can volunteer some general
> directions (obviously not exhaustive) where improvements could be made.
> 1. Keep existing in-memory cache but compute it more eagerly, i.e. on
> start-up and after a full recovery. I find these scenarios cause cache
> invalidation the most often. The downside is you make initial node-start-up a
> function of index size. You have to consider deletions also invalidate the
> cache so you'd want to subtract deletions here in order to avoid total
> re-computation after delete-heavy workloads. Which brings me to my next
> suggestion.
> 2. Persisting a gross fingerprint during segment creation and then
> subtracting deletions during fingerprint calculation. Deletions are easily
> invertible with regards to the fingerprint. You can compute a gross
> fingerprint when you create the segment and store it, perhaps along with
> commit info or in the segment attributes themselves. Then when you need the
> fingerprint you subtract the checksum of the deleted docs from this gross
> checksum. This allows you to scan the version long-values of only the deleted
> docs. I've ran a PoC of this kind of solution and find that it results in a
> ~7X improvement when the deletion ratio is ~10%. The downside is your
> fingerprint computation cost is now a function of your deletion ratio which,
> although much better than a full index scan, is still typically O(# of docs
> in the index).
> 3. Carrying the idea even further, you could incrementally update the
> fingerprint with each generation of deletes as you index. So you can store
> not only the gross-fingerprint but the net fingerprint as well. This net
> fingerprint could track the generation of your live docs. I have a PoC of
> this kind of solution as well and find it reduces the peer sync cost
> dramatically (~100X). The downsides are you need a custom codec and you pay a
> bigger tax during indexing than 2. My profiling found this tax to be rather
> minuscule (less than ~1% of indexing CPU samples) but it exists nonetheless.
> Also, it adds a bit of complexity, most prominently in the form of a custom
> codec. The PoC also has no optimizations for in-place DV updates which would
> require even _more_ complexity to optimize (did I mention dislike in-place dv
> updates?).
> 4. Disable fingerprint computation for tlog+pull replicas. Tlog+pull replicas
> don't really benefit from tracking document-equivalence via a checksum over
> the individual document versions. They converge to the same underlying
> segments so I think this is totally wasted work (unless we do something along
> the lines of the custom codec proposed in points 2 or 3. There you could see
> how a leader tlog replica could keep track of fingerprint just in case a
> benefitting NRT replica ever gets created in the future though this seems
> like a rather small corner case).
> 5. (more of an architectural shift) scrap the fingerprint altogether. The
> only reason it is needed is because Solr's "update identifier" of choice (the
> document \_version\_) is sparse. It's effectively a Lamport timestamp. But if
> it were a dense sequence number instead, you could track the high watermark
> of the cluster via a simple checkpoint. Elastic and open search do this for
> their "document replication" mode. Each replica tracks a checkpoint which is
> the point up to which it knows the full history "densely". The minimum
> checkpoint of all active replicas is a high watermark, below which you don't
> need to check _anything_.
> I welcome any other discussions or suggestions you may have regarding this
> ticket!
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
