[
https://issues.apache.org/jira/browse/SOLR-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704299#comment-14704299
]
Shalin Shekhar Mangar commented on SOLR-6760:
---------------------------------------------
This gives us a great speedup as well. The rate of processing state operations
went up from 4550 requests/min to 26083 requests/min i.e. a boost of 473%!
{code}
stateFormat = 2, 10 collections, With refactoring (trunk):
=============================================================
2> 56663 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 Overseer loop finished processing:
2> 56664 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 totalTime: 46405.595921
2> 56664 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 avgRequestsPerMinute: 0.021548541967677345
2> 56664 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 5minRateRequestsPerMinute: 0.0
2> 56664 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 15minRateRequestsPerMinute: 0.0
2> 56664 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 avgTimePerRequest: 46405.595921
2> 56664 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 medianRequestTime: 46405.595921
2> 56665 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 75thPctlRequestTime: 46405.595921
2> 56665 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 95thPctlRequestTime: 46405.595921
2> 56665 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 99thPctlRequestTime: 46405.595921
2> 56665 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 999thPctlRequestTime: 46405.595921
2> 56665 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 op: am_i_leader, success: 3, failure: 0
2> 56665 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 totalTime: 7.714871
2> 56665 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 avgRequestsPerMinute: 3.9009249281419294
2> 56665 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 5minRateRequestsPerMinute: 21.00415965703074
2> 56665 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 15minRateRequestsPerMinute: 22.9566897384727
2> 56666 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 avgTimePerRequest: 2.5716236666666665
2> 56666 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 medianRequestTime: 0.390077
2> 56666 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 75thPctlRequestTime: 7.016754
2> 56666 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 95thPctlRequestTime: 7.016754
2> 56666 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 99thPctlRequestTime: 7.016754
2> 56667 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 999thPctlRequestTime: 7.016754
2> 56667 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 op: update_state, success: 20011, failure: 0
2> 56669 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 totalTime: 8677.820981
2> 56669 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 avgRequestsPerMinute: 26054.58092669684
2> 56669 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 5minRateRequestsPerMinute: 28115.736803934476
2> 56669 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 15minRateRequestsPerMinute: 28300.17714580809
2> 56669 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 avgTimePerRequest: 0.4336525401529159
2> 56669 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 medianRequestTime: 0.4546175
2> 56669 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 75thPctlRequestTime: 0.5425845
2> 56669 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 95thPctlRequestTime: 0.68700805
2> 56670 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 99thPctlRequestTime: 0.9623780800000007
2> 56670 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 999thPctlRequestTime: 3.872901139000002
2> 56670 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 op: state, success: 20001, failure: 0
2> 56671 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 totalTime: 7268.609771
2> 56671 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 avgRequestsPerMinute: 26083.69778407265
2> 56671 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 5minRateRequestsPerMinute: 28469.456291781717
2> 56671 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 15minRateRequestsPerMinute: 28689.269577883602
2> 56671 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 avgTimePerRequest: 0.36341231793410333
2> 56671 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 medianRequestTime: 0.37234449999999997
2> 56671 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 75thPctlRequestTime: 0.451628
2> 56671 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 95thPctlRequestTime: 0.5609881
2> 56671 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 99thPctlRequestTime: 0.6850270100000001
2> 56671 INFO (TEST-OverseerTest.testPerformance-seed#[4A854829B14EB462]) [
] o.a.s.SolrTestCaseJ4 999thPctlRequestTime: 3.4499434830000055
stateFormat = 2, 10 collections, Without refactoring (trunk):
=============================================================
[junit4] 2> 274160 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 Overseer loop finished processing:
[junit4] 2> 274161 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 totalTime: 264346.50677
[junit4] 2> 274161 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 avgRequestsPerMinute: 0.003782893537381759
[junit4] 2> 274161 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 5minRateRequestsPerMinute: 0.0
[junit4] 2> 274162 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 15minRateRequestsPerMinute: 0.0
[junit4] 2> 274162 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 avgTimePerRequest: 264346.50677
[junit4] 2> 274162 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 medianRequestTime: 264346.50677
[junit4] 2> 274162 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 75thPctlRequestTime: 264346.50677
[junit4] 2> 274162 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 95thPctlRequestTime: 264346.50677
[junit4] 2> 274162 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 99thPctlRequestTime: 264346.50677
[junit4] 2> 274162 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 999thPctlRequestTime: 264346.50677
[junit4] 2> 274163 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 op: am_i_leader, success: 2, failure: 0
[junit4] 2> 274163 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 totalTime: 10.991148
[junit4] 2> 274163 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 avgRequestsPerMinute: 0.4544215783246879
[junit4] 2> 274163 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 5minRateRequestsPerMinute: 10.257958366769442
[junit4] 2> 274163 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 15minRateRequestsPerMinute: 18.078447754911757
[junit4] 2> 274163 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 avgTimePerRequest: 5.495574
[junit4] 2> 274164 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 medianRequestTime: 5.4955739999999995
[junit4] 2> 274164 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 75thPctlRequestTime: 7.035856
[junit4] 2> 274164 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 95thPctlRequestTime: 7.035856
[junit4] 2> 274164 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 99thPctlRequestTime: 7.035856
[junit4] 2> 274165 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 999thPctlRequestTime: 7.035856
[junit4] 2> 274165 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 op: update_state, success: 20011, failure: 0
[junit4] 2> 274166 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 totalTime: 12397.475361
[junit4] 2> 274166 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 avgRequestsPerMinute: 4547.833695111793
[junit4] 2> 274166 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 5minRateRequestsPerMinute: 3847.5781990107857
[junit4] 2> 274166 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 15minRateRequestsPerMinute: 2998.648631930923
[junit4] 2> 274167 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 avgTimePerRequest: 0.6195330248863126
[junit4] 2> 274167 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 medianRequestTime: 0.5545475
[junit4] 2> 274167 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 75thPctlRequestTime: 0.6847515
[junit4] 2> 274167 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 95thPctlRequestTime: 0.8879762999999998
[junit4] 2> 274167 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 99thPctlRequestTime: 1.1280661800000003
[junit4] 2> 274167 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 999thPctlRequestTime: 3.7108806920000013
[junit4] 2> 274167 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 op: state, success: 20001, failure: 0
[junit4] 2> 274168 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 totalTime: 10719.864045
[junit4] 2> 274169 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 avgRequestsPerMinute: 4550.14580363099
[junit4] 2> 274169 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 5minRateRequestsPerMinute: 3861.3377008032517
[junit4] 2> 274169 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 15minRateRequestsPerMinute: 3010.807350864149
[junit4] 2> 274169 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 avgTimePerRequest: 0.5359664039298035
[junit4] 2> 274169 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 medianRequestTime: 0.483754
[junit4] 2> 274169 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 75thPctlRequestTime: 0.58192225
[junit4] 2> 274169 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 95thPctlRequestTime: 0.71668745
[junit4] 2> 274169 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 99thPctlRequestTime: 0.8258062500000001
[junit4] 2> 274169 INFO
(TEST-OverseerTest.testPerformance-seed#[1CE0793468424C19]) [ ]
o.a.s.SolrTestCaseJ4 999thPctlRequestTime: 13.507429244000047
{code}
> New optimized DistributedQueue implementation for overseer
> ----------------------------------------------------------
>
> Key: SOLR-6760
> URL: https://issues.apache.org/jira/browse/SOLR-6760
> Project: Solr
> Issue Type: Bug
> Reporter: Noble Paul
> Assignee: Shalin Shekhar Mangar
> Attachments: SOLR-6760.patch, SOLR-6760.patch, SOLR-6760.patch,
> SOLR-6760.patch, deadlock.patch
>
>
> Currently the DQ works as follows
> * read all items in the directory
> * sort them all
> * take the head and return it and discard everything else
> * rinse and repeat
> This works well when we have only a handful of items in the Queue. If the
> items in the queue is much larger (in tens of thousands) , this is
> counterproductive
> As the overseer queue is a multiple producers + single consumer queue, We can
> read them all in bulk and before processing each item , just do a
> zk.exists(itemname) and if all is well we don't need to do the fetch all +
> sort thing again
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
