[ 
https://issues.apache.org/jira/browse/KUDU-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504989#comment-15504989
 ] 

Todd Lipcon commented on KUDU-1567:
-----------------------------------

I re-ran the benchmarks and got more consistent results:

*First the throughput:*
{code}
results/log_retention=50/sync=false/ycsb-load-workloada.json:  "measurement" : 
"Throughput(ops/sec)",
results/log_retention=50/sync=false/ycsb-load-workloada.json-  "value" : 
381219.30708053865
--
results/log_retention=50/sync=false/ycsb-run-workloada.json:  "measurement" : 
"Throughput(ops/sec)",
results/log_retention=50/sync=false/ycsb-run-workloada.json-  "value" : 
1628.2733269662172
--
results/log_retention=50/sync=false/ycsb-run-workloadc.json:  "measurement" : 
"Throughput(ops/sec)",
results/log_retention=50/sync=false/ycsb-run-workloadc.json-  "value" : 
252.7126705999234
--
results/log_retention=default/sync=false/ycsb-load-workloada.json:  
"measurement" : "Throughput(ops/sec)",
results/log_retention=default/sync=false/ycsb-load-workloada.json-  "value" : 
186764.10274416232
--
results/log_retention=default/sync=false/ycsb-run-workloada.json:  
"measurement" : "Throughput(ops/sec)",
results/log_retention=default/sync=false/ycsb-run-workloada.json-  "value" : 
856.4117934372364
--
results/log_retention=default/sync=false/ycsb-run-workloadc.json:  
"measurement" : "Throughput(ops/sec)",
results/log_retention=default/sync=false/ycsb-run-workloadc.json-  "value" : 
283.99350631361403
{code}

Slightly odd that the 'workload C' (read-only) throughput appeared to go down 
10-15%, but I'm guessing this wasn't really affected and is just noise.

*And some metrics:*

Bloom lookups per op (captured for the four tablets in YCSB) are about half:
{code}
results/log_retention=50/sync=false/metrics-after-load-workloada.txt:           
     "name": "bloom_lookups_per_op",
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-           
     "total_count": 135524910,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-           
     "min": 0,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-           
     "mean": 17.4312,
--
results/log_retention=50/sync=false/metrics-after-load-workloada.txt:           
     "name": "bloom_lookups_per_op",
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-           
     "total_count": 135524034,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-           
     "min": 0,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-           
     "mean": 17.1001,
--
results/log_retention=50/sync=false/metrics-after-load-workloada.txt:           
     "name": "bloom_lookups_per_op",
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-           
     "total_count": 135526576,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-           
     "min": 0,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-           
     "mean": 17.4048,
--
results/log_retention=50/sync=false/metrics-after-load-workloada.txt:           
     "name": "bloom_lookups_per_op",
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-           
     "total_count": 93424480,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-           
     "min": 0,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-           
     "mean": 12.9333,
--
results/log_retention=default/sync=false/metrics-after-load-workloada.txt:      
          "name": "bloom_lookups_per_op",
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-      
          "total_count": 93424480,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-      
          "min": 0,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-      
          "mean": 25.2444,
--
results/log_retention=default/sync=false/metrics-after-load-workloada.txt:      
          "name": "bloom_lookups_per_op",
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-      
          "total_count": 135526576,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-      
          "min": 0,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-      
          "mean": 37.8615,
--
results/log_retention=default/sync=false/metrics-after-load-workloada.txt:      
          "name": "bloom_lookups_per_op",
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-      
          "total_count": 135524034,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-      
          "min": 0,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-      
          "mean": 39.7972,
--
results/log_retention=default/sync=false/metrics-after-load-workloada.txt:      
          "name": "bloom_lookups_per_op",
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-      
          "total_count": 135524910,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-      
          "min": 0,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-      
          "mean": 39.3984,
{code}

and total IO counters substantially lower:
{code}
results/log_retention=50/sync=false/metrics-after-load-workloada.txt:           
     "name": "block_manager_total_bytes_written",
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-           
     "value": 168846243283
--
results/log_retention=50/sync=false/metrics-after-load-workloada.txt:           
     "name": "block_manager_total_bytes_read",
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-           
     "value": 4969048266023
--
results/log_retention=default/sync=false/metrics-after-load-workloada.txt:      
          "name": "block_manager_total_bytes_written",
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-      
          "value": 285113846979
--
results/log_retention=default/sync=false/metrics-after-load-workloada.txt:      
          "name": "block_manager_total_bytes_read",
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-      
          "value": 12224412188591
{code}

So I think the benchmarks confirm that fixing this would be good 
bang-for-the-buck perf improvement.

> Short default for log retention increases write amplification
> -------------------------------------------------------------
>
>                 Key: KUDU-1567
>                 URL: https://issues.apache.org/jira/browse/KUDU-1567
>             Project: Kudu
>          Issue Type: Improvement
>          Components: perf, tserver
>    Affects Versions: 0.10.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>
> Currently the maintenance manager prioritizes flushes over compactions if the 
> flush operations are retaining WAL segments. The goal here is to prevent the 
> amount of in-memory data from getting so large that restarts would be 
> incredibly slow. However, it has a somewhat unintuitive negative effect on 
> performance:
> - with the default of retaining just two segments, flushes become highly 
> prioritized when the MRS only has ~128MB of data, regardless of the 
> "flush_threshold_mb" configuration
> - this creates lots of overlapping rowsets in the case of random-write 
> applications
> - because flushes are prioritized over compactions, compactions rarely run
> - the frequent flushes, combined with low priority of compactions, means that 
> after a few days of constant inserts, we often end up with average "bloom 
> lookups per op" metrics of 50-100, which is quite slow even if the blooms fit 
> in cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to