[ 
https://issues.apache.org/jira/browse/OAK-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362564#comment-16362564
 ] 

Andrei Dulceanu commented on OAK-7262:
--------------------------------------

Here are the first results for patched vs. trunk (on my local machine):

{noformat}
# BasicWriteTest                   C     min     10%     50%     90%     max    
   N
Patched                            1      19      21      22      25      67    
2569
Trunk                              1      19      21      22      26      63    
2501

# ConcurrentReadTest               C     min     10%     50%     90%     max    
   N
Patched                            1      54     112     153     217     314    
 379
Trunk                              1      50     115     140     165     249    
 427

# ConcurrentWriteTest              C     min     10%     50%     90%     max    
   N
Patched                            1      81      84      89     104     222    
 645
Trunk                              1      83      85      90     110     267    
 627

# ConcurrentReadWriteTest          C     min     10%     50%     90%     max    
   N
Patched                            1      52     136     188     276     359    
 307
Trunk                              1      49     127     170     242     372    
 340

# ConcurrentWriteReadTest          C     min     10%     50%     90%     max    
   N
Patched                            1       6       8      17      76     226    
1871
Trunk                              1       6       8      16      66     250    
2057
{noformat}

[~frm], it appears that there's a slight performance drop for some tests (i.e., 
{{BasicWriteTest}}, {{ConcurrentReadTest}}). One thing to be noted is that 
usually there's a lot of variance when running the benchmarks locally. 
Moreover, as noted in previous experiments, these benchmarks might not be 
really accurate at predicting how the system performs under heavy-load with 
multiple concurrent writers ({{SegmentCompactionIT}} is the only example which 
comes to mind here).

My proposal is to move forward with this patch. We can easily revert it later, 
provided real-world usage proves it was a wrong solution.

> LockBasedScheduler#getHeadNodeState poor performance due to lock contention 
> in commitTimeHistogram implementation
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: OAK-7262
>                 URL: https://issues.apache.org/jira/browse/OAK-7262
>             Project: Jackrabbit Oak
>          Issue Type: Improvement
>          Components: segment-tar
>            Reporter: Andrei Dulceanu
>            Assignee: Andrei Dulceanu
>            Priority: Major
>             Fix For: 1.9.0, 1.10
>
>
> With OAK-4732, we slightly prioritised reads, allowing callers of 
> {{LockBasedScheduler#getHeadNodeState}} to wait for a certain amount of time 
> on the commit semaphore, in order to fetch the latest head. The amount of 
> time to wait for was computed as the median of the latest 1000 commits, but 
> the implementation was flexible enough to replace the 0.5 quantile with a 
> different value.
> The initial implementation used {{SynchronizedDescriptiveStatistics}} from 
> {{commons-math3}}. In OAK-6430, we replaced that with a {{Histogram}} backed 
> by a {{SlindingWindowReservoir}} from the dropwizard metric library 
> ({{io.dropwizard.metrics}}). 
> One of the problems observed with the current implementation is that, under 
> heavy loading, the performance of {{LockBasedScheduler#getHeadNodeState}} 
> decreases due to lock contention in {{commitTimeHistogram}} implementation. 
> Namely,  {{SlidingWindowReservoir#getSnapshot}} seems to be a bottleneck. It 
> copies an array of 1000 elements at each invocation of 
> {{LockBasedScheduler#getHeadNodeState}}, acquiring and releasing a lock for 
> each element in the array.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to