[jira] [Commented] (SOLR-10524) Better ZkStateWriter batching
[ https://issues.apache.org/jira/browse/SOLR-10524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297192#comment-16297192 ] ASF GitHub Bot commented on SOLR-10524: --- GitHub user slackhappy opened a pull request: https://github.com/apache/lucene-solr/pull/294 ZkStateReader: cache LazyCollectionRef SOLR-10524 introduced zk state update batching, with a default interval of 2 seconds. That opens the door for a simple, time-based cache on the read side to address the issue described in SOLR-8327 You can merge this pull request into a Git repository by running: $ git pull https://github.com/slackhappy/lucene-solr cloud_cache_lazy_collection_ref Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/294.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #294 commit e7c6a6773f1d01d2ddb0dbce6cdbaeff52e78376 Author: John Gallagher Date: 2017-12-19T16:57:25Z ZkStateReader: cache LazyCollectionRef SOLR-10524 introduced zk state update batching, with a default interval of 2 seconds. That opens the door for a simple, time-based cache on the read side to address the issue described in SOLR-8327 > Better ZkStateWriter batching > - > > Key: SOLR-10524 > URL: https://issues.apache.org/jira/browse/SOLR-10524 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson >Assignee: Cao Manh Dat > Fix For: 6.6, 7.0 > > Attachments: SOLR-10524-NPE-fix.patch, SOLR-10524-dragonsinth.patch, > SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch > > > There are several JIRAs (I'll link in a second) about trying to be more > efficient about processing overseer messages as the overseer can become a > bottleneck, especially with very large numbers of replicas in a cluster. One > of the approaches mentioned near the end of SOLR-5872 (15-Mar) was to "read > large no:of items say 1. put them into in memory buckets and feed them > into overseer". > This JIRA is to break out that part of the discussion as it might be an easy > win whereas "eliminating the Overseer queue" would be quite an undertaking. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10524) Better ZkStateWriter batching
[ https://issues.apache.org/jira/browse/SOLR-10524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002192#comment-16002192 ] ASF subversion and git services commented on SOLR-10524: Commit 5c626dc9e0a488d43e9a2f41947fd2ec3b0b046f in lucene-solr's branch refs/heads/branch_6x from [~dragonsinth] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5c626dc ] SOLR-10524: Build fix for NPE Introduced by ZkStateWriter batching optimizations. > Better ZkStateWriter batching > - > > Key: SOLR-10524 > URL: https://issues.apache.org/jira/browse/SOLR-10524 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson > Attachments: SOLR-10524-dragonsinth.patch, SOLR-10524-NPE-fix.patch, > SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch > > > There are several JIRAs (I'll link in a second) about trying to be more > efficient about processing overseer messages as the overseer can become a > bottleneck, especially with very large numbers of replicas in a cluster. One > of the approaches mentioned near the end of SOLR-5872 (15-Mar) was to "read > large no:of items say 1. put them into in memory buckets and feed them > into overseer". > This JIRA is to break out that part of the discussion as it might be an easy > win whereas "eliminating the Overseer queue" would be quite an undertaking. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10524) Better ZkStateWriter batching
[ https://issues.apache.org/jira/browse/SOLR-10524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002191#comment-16002191 ] ASF subversion and git services commented on SOLR-10524: Commit 9cab9c0cf2777a21a81386be2262e84da2bca751 in lucene-solr's branch refs/heads/branch_6x from [~caomanhdat] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9cab9c0 ] SOLR-10524: Explore in-memory partitioning for processing Overseer queue messages > Better ZkStateWriter batching > - > > Key: SOLR-10524 > URL: https://issues.apache.org/jira/browse/SOLR-10524 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson > Attachments: SOLR-10524-dragonsinth.patch, SOLR-10524-NPE-fix.patch, > SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch > > > There are several JIRAs (I'll link in a second) about trying to be more > efficient about processing overseer messages as the overseer can become a > bottleneck, especially with very large numbers of replicas in a cluster. One > of the approaches mentioned near the end of SOLR-5872 (15-Mar) was to "read > large no:of items say 1. put them into in memory buckets and feed them > into overseer". > This JIRA is to break out that part of the discussion as it might be an easy > win whereas "eliminating the Overseer queue" would be quite an undertaking. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10524) Better ZkStateWriter batching
[ https://issues.apache.org/jira/browse/SOLR-10524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001856#comment-16001856 ] Scott Blum commented on SOLR-10524: --- Thanks [~caomanhdat], committed to master. > Better ZkStateWriter batching > - > > Key: SOLR-10524 > URL: https://issues.apache.org/jira/browse/SOLR-10524 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson > Attachments: SOLR-10524-dragonsinth.patch, SOLR-10524-NPE-fix.patch, > SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch > > > There are several JIRAs (I'll link in a second) about trying to be more > efficient about processing overseer messages as the overseer can become a > bottleneck, especially with very large numbers of replicas in a cluster. One > of the approaches mentioned near the end of SOLR-5872 (15-Mar) was to "read > large no:of items say 1. put them into in memory buckets and feed them > into overseer". > This JIRA is to break out that part of the discussion as it might be an easy > win whereas "eliminating the Overseer queue" would be quite an undertaking. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10524) Better ZkStateWriter batching
[ https://issues.apache.org/jira/browse/SOLR-10524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001855#comment-16001855 ] ASF subversion and git services commented on SOLR-10524: Commit 972e342fee7a02e71300a9739b9971e63708589b in lucene-solr's branch refs/heads/master from [~dragonsinth] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=972e342 ] SOLR-10524: Build fix for NPE Introduced by ZkStateWriter batching optimizations. > Better ZkStateWriter batching > - > > Key: SOLR-10524 > URL: https://issues.apache.org/jira/browse/SOLR-10524 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson > Attachments: SOLR-10524-dragonsinth.patch, SOLR-10524-NPE-fix.patch, > SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch > > > There are several JIRAs (I'll link in a second) about trying to be more > efficient about processing overseer messages as the overseer can become a > bottleneck, especially with very large numbers of replicas in a cluster. One > of the approaches mentioned near the end of SOLR-5872 (15-Mar) was to "read > large no:of items say 1. put them into in memory buckets and feed them > into overseer". > This JIRA is to break out that part of the discussion as it might be an easy > win whereas "eliminating the Overseer queue" would be quite an undertaking. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10524) Better ZkStateWriter batching
[ https://issues.apache.org/jira/browse/SOLR-10524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001846#comment-16001846 ] Cao Manh Dat commented on SOLR-10524: - Looks good +1 for commit > Better ZkStateWriter batching > - > > Key: SOLR-10524 > URL: https://issues.apache.org/jira/browse/SOLR-10524 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson > Attachments: SOLR-10524-dragonsinth.patch, SOLR-10524-NPE-fix.patch, > SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch > > > There are several JIRAs (I'll link in a second) about trying to be more > efficient about processing overseer messages as the overseer can become a > bottleneck, especially with very large numbers of replicas in a cluster. One > of the approaches mentioned near the end of SOLR-5872 (15-Mar) was to "read > large no:of items say 1. put them into in memory buckets and feed them > into overseer". > This JIRA is to break out that part of the discussion as it might be an easy > win whereas "eliminating the Overseer queue" would be quite an undertaking. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
