[jira] [Commented] (SOLR-14776) Precompute the fingerprint during PeerSync
[ https://issues.apache.org/jira/browse/SOLR-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213082#comment-17213082 ] ASF subversion and git services commented on SOLR-14776: Commit 0f2e389c1346edaed76987cf73c53218110863bc in lucene-solr's branch refs/heads/branch_8x from Cao Manh Dat [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=0f2e389 ] SOLR-14776: Precompute the fingerprint during PeerSync (#1814) After heavy indexing, the call to compute fingerprint takes awhile and slows the leader election. This commit computes the fingerprint in parallel with fetching the fingerprint from the other replicas. Co-authored-by: Shalin Shekhar Mangar (cherry picked from commit 9594ab3ac01c3449fde8163d44c9c27261585c9f) > Precompute the fingerprint during PeerSync > -- > > Key: SOLR-14776 > URL: https://issues.apache.org/jira/browse/SOLR-14776 > Project: Solr > Issue Type: Improvement >Reporter: Cao Manh Dat >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: master (9.0), 8.7 > > Time Spent: 20m > Remaining Estimate: 0h > > Computing fingerprint can very costly and take time. But the current > implementation will send requests for getting fingerprint for multiple > replicas, then on the first response it will then compute its own fingerprint > for comparison. A very simple but effective improvement here is compute its > own fingerprint right after send requests to other replicas. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14776) Precompute the fingerprint during PeerSync
[ https://issues.apache.org/jira/browse/SOLR-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213060#comment-17213060 ] ASF subversion and git services commented on SOLR-14776: Commit 9594ab3ac01c3449fde8163d44c9c27261585c9f in lucene-solr's branch refs/heads/master from Cao Manh Dat [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=9594ab3 ] SOLR-14776: Precompute the fingerprint during PeerSync (#1814) After heavy indexing, the call to compute fingerprint takes awhile and slows the leader election. This commit computes the fingerprint in parallel with fetching the fingerprint from the other replicas. Co-authored-by: Shalin Shekhar Mangar > Precompute the fingerprint during PeerSync > -- > > Key: SOLR-14776 > URL: https://issues.apache.org/jira/browse/SOLR-14776 > Project: Solr > Issue Type: Improvement >Reporter: Cao Manh Dat >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: master (9.0), 8.7 > > Time Spent: 10m > Remaining Estimate: 0h > > Computing fingerprint can very costly and take time. But the current > implementation will send requests for getting fingerprint for multiple > replicas, then on the first response it will then compute its own fingerprint > for comparison. A very simple but effective improvement here is compute its > own fingerprint right after send requests to other replicas. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14776) Precompute the fingerprint during PeerSync
[ https://issues.apache.org/jira/browse/SOLR-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213059#comment-17213059 ] ASF subversion and git services commented on SOLR-14776: Commit 33d1a6edcefd4bb1511fe79db6b114b996a55e8f in lucene-solr's branch refs/heads/jira/SOLR-14776 from Shalin Shekhar Mangar [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=33d1a6e ] Merge branch 'master' into jira/SOLR-14776 > Precompute the fingerprint during PeerSync > -- > > Key: SOLR-14776 > URL: https://issues.apache.org/jira/browse/SOLR-14776 > Project: Solr > Issue Type: Improvement >Reporter: Cao Manh Dat >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: master (9.0), 8.7 > > Time Spent: 10m > Remaining Estimate: 0h > > Computing fingerprint can very costly and take time. But the current > implementation will send requests for getting fingerprint for multiple > replicas, then on the first response it will then compute its own fingerprint > for comparison. A very simple but effective improvement here is compute its > own fingerprint right after send requests to other replicas. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14776) Precompute the fingerprint during PeerSync
[ https://issues.apache.org/jira/browse/SOLR-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213051#comment-17213051 ] ASF subversion and git services commented on SOLR-14776: Commit 8c554379aca6ab6c811eddd9e4428d598c2e7336 in lucene-solr's branch refs/heads/jira/SOLR-14776 from Shalin Shekhar Mangar [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=8c55437 ] SOLR-14776: Adding entry to CHANGES.txt > Precompute the fingerprint during PeerSync > -- > > Key: SOLR-14776 > URL: https://issues.apache.org/jira/browse/SOLR-14776 > Project: Solr > Issue Type: Improvement >Reporter: Cao Manh Dat >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: master (9.0), 8.7 > > Time Spent: 10m > Remaining Estimate: 0h > > Computing fingerprint can very costly and take time. But the current > implementation will send requests for getting fingerprint for multiple > replicas, then on the first response it will then compute its own fingerprint > for comparison. A very simple but effective improvement here is compute its > own fingerprint right after send requests to other replicas. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14776) Precompute the fingerprint during PeerSync
[ https://issues.apache.org/jira/browse/SOLR-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213015#comment-17213015 ] Shalin Shekhar Mangar commented on SOLR-14776: -- I have added a comment as Mike suggested. I'll commit this once tests pass locally. > Precompute the fingerprint during PeerSync > -- > > Key: SOLR-14776 > URL: https://issues.apache.org/jira/browse/SOLR-14776 > Project: Solr > Issue Type: Improvement >Reporter: Cao Manh Dat >Assignee: Shalin Shekhar Mangar >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Computing fingerprint can very costly and take time. But the current > implementation will send requests for getting fingerprint for multiple > replicas, then on the first response it will then compute its own fingerprint > for comparison. A very simple but effective improvement here is compute its > own fingerprint right after send requests to other replicas. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14776) Precompute the fingerprint during PeerSync
[ https://issues.apache.org/jira/browse/SOLR-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213011#comment-17213011 ] ASF subversion and git services commented on SOLR-14776: Commit f0a9588e6402c81ae4c2e38e85ee8aedcc7848cb in lucene-solr's branch refs/heads/jira/SOLR-14776 from Shalin Shekhar Mangar [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=f0a9588 ] SOLR-14776: Adding comment on how/why leader candidate's fingerprint is computed in parallel with other replicas > Precompute the fingerprint during PeerSync > -- > > Key: SOLR-14776 > URL: https://issues.apache.org/jira/browse/SOLR-14776 > Project: Solr > Issue Type: Improvement >Reporter: Cao Manh Dat >Assignee: Cao Manh Dat >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Computing fingerprint can very costly and take time. But the current > implementation will send requests for getting fingerprint for multiple > replicas, then on the first response it will then compute its own fingerprint > for comparison. A very simple but effective improvement here is compute its > own fingerprint right after send requests to other replicas. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14776) Precompute the fingerprint during PeerSync
[ https://issues.apache.org/jira/browse/SOLR-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17194462#comment-17194462 ] Mike Drob commented on SOLR-14776: -- > Yeah, but we only compute fingerprint during leader election. Therefore after >a very heavy indexing, then the leader goes away the first call to fingerprint >gonna takes awhile then slowdown the leader election Oh, this makes sense and is a pretty big limitation. Can you add a comment to the code that reminds us of this? > Precompute the fingerprint during PeerSync > -- > > Key: SOLR-14776 > URL: https://issues.apache.org/jira/browse/SOLR-14776 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Cao Manh Dat >Assignee: Cao Manh Dat >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Computing fingerprint can very costly and take time. But the current > implementation will send requests for getting fingerprint for multiple > replicas, then on the first response it will then compute its own fingerprint > for comparison. A very simple but effective improvement here is compute its > own fingerprint right after send requests to other replicas. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14776) Precompute the fingerprint during PeerSync
[ https://issues.apache.org/jira/browse/SOLR-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193951#comment-17193951 ] Cao Manh Dat commented on SOLR-14776: - > Is there a chance that the core is changing between subsequent calls and so > it is not safe to use the same, maybe from incoming updates? During leader election there shouldn't be any updates coming from the old leader (we are not doing anything to prevent that), if there are the current fingerprint strategy's here is also fail > It looks like there is already caching on SolrCore.getIndexFingerprint, is >that broken or insufficient in some way? Yeah, but we only compute fingerprint during leader election. Therefore after a very heavy indexing, then the leader goes away the first call to fingerprint gonna takes awhile then slowdown the leader election > Thinking about this more, is the big win that we compute our own fingerprint >during the time that would normally be spent waiting, and we decrease latency >that way? This can be a way to do that, another solution here is on commit the segment we already compute fingerprint for that segment, but I think it is worth to do in another issue. > Precompute the fingerprint during PeerSync > -- > > Key: SOLR-14776 > URL: https://issues.apache.org/jira/browse/SOLR-14776 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Cao Manh Dat >Assignee: Cao Manh Dat >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Computing fingerprint can very costly and take time. But the current > implementation will send requests for getting fingerprint for multiple > replicas, then on the first response it will then compute its own fingerprint > for comparison. A very simple but effective improvement here is compute its > own fingerprint right after send requests to other replicas. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14776) Precompute the fingerprint during PeerSync
[ https://issues.apache.org/jira/browse/SOLR-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190410#comment-17190410 ] Mike Drob commented on SOLR-14776: -- Thinking about this more, is the big win that we compute our own fingerprint during the time that would normally be spent waiting, and we decrease latency that way? > Precompute the fingerprint during PeerSync > -- > > Key: SOLR-14776 > URL: https://issues.apache.org/jira/browse/SOLR-14776 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Cao Manh Dat >Assignee: Cao Manh Dat >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Computing fingerprint can very costly and take time. But the current > implementation will send requests for getting fingerprint for multiple > replicas, then on the first response it will then compute its own fingerprint > for comparison. A very simple but effective improvement here is compute its > own fingerprint right after send requests to other replicas. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14776) Precompute the fingerprint during PeerSync
[ https://issues.apache.org/jira/browse/SOLR-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190408#comment-17190408 ] Mike Drob commented on SOLR-14776: -- Is there a chance that the core is changing between subsequent calls and so it is not safe to use the same, maybe from incoming updates? It looks like there is already caching on SolrCore.getIndexFingerprint, is that broken or insufficient in some way? > Precompute the fingerprint during PeerSync > -- > > Key: SOLR-14776 > URL: https://issues.apache.org/jira/browse/SOLR-14776 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Cao Manh Dat >Assignee: Cao Manh Dat >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Computing fingerprint can very costly and take time. But the current > implementation will send requests for getting fingerprint for multiple > replicas, then on the first response it will then compute its own fingerprint > for comparison. A very simple but effective improvement here is compute its > own fingerprint right after send requests to other replicas. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14776) Precompute the fingerprint during PeerSync
[ https://issues.apache.org/jira/browse/SOLR-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188127#comment-17188127 ] ASF subversion and git services commented on SOLR-14776: Commit 71de2cd4a265ffce211e40ea17daefc8b456973c in lucene-solr's branch refs/heads/jira/SOLR-14776 from Cao Manh Dat [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=71de2cd ] SOLR-14776: Precompute the fingerprint during PeerSync > Precompute the fingerprint during PeerSync > -- > > Key: SOLR-14776 > URL: https://issues.apache.org/jira/browse/SOLR-14776 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Cao Manh Dat >Assignee: Cao Manh Dat >Priority: Major > > Computing fingerprint can very costly and take time. But the current > implementation will send requests for getting fingerprint for multiple > replicas, then on the first response it will then compute its own fingerprint > for comparison. A very simple but effective improvement here is compute its > own fingerprint right after send requests to other replicas. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
