[
https://issues.apache.org/jira/browse/KUDU-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504989#comment-15504989
]
Todd Lipcon commented on KUDU-1567:
-----------------------------------
I re-ran the benchmarks and got more consistent results:
*First the throughput:*
{code}
results/log_retention=50/sync=false/ycsb-load-workloada.json: "measurement" :
"Throughput(ops/sec)",
results/log_retention=50/sync=false/ycsb-load-workloada.json- "value" :
381219.30708053865
--
results/log_retention=50/sync=false/ycsb-run-workloada.json: "measurement" :
"Throughput(ops/sec)",
results/log_retention=50/sync=false/ycsb-run-workloada.json- "value" :
1628.2733269662172
--
results/log_retention=50/sync=false/ycsb-run-workloadc.json: "measurement" :
"Throughput(ops/sec)",
results/log_retention=50/sync=false/ycsb-run-workloadc.json- "value" :
252.7126705999234
--
results/log_retention=default/sync=false/ycsb-load-workloada.json:
"measurement" : "Throughput(ops/sec)",
results/log_retention=default/sync=false/ycsb-load-workloada.json- "value" :
186764.10274416232
--
results/log_retention=default/sync=false/ycsb-run-workloada.json:
"measurement" : "Throughput(ops/sec)",
results/log_retention=default/sync=false/ycsb-run-workloada.json- "value" :
856.4117934372364
--
results/log_retention=default/sync=false/ycsb-run-workloadc.json:
"measurement" : "Throughput(ops/sec)",
results/log_retention=default/sync=false/ycsb-run-workloadc.json- "value" :
283.99350631361403
{code}
Slightly odd that the 'workload C' (read-only) throughput appeared to go down
10-15%, but I'm guessing this wasn't really affected and is just noise.
*And some metrics:*
Bloom lookups per op (captured for the four tablets in YCSB) are about half:
{code}
results/log_retention=50/sync=false/metrics-after-load-workloada.txt:
"name": "bloom_lookups_per_op",
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-
"total_count": 135524910,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-
"min": 0,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-
"mean": 17.4312,
--
results/log_retention=50/sync=false/metrics-after-load-workloada.txt:
"name": "bloom_lookups_per_op",
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-
"total_count": 135524034,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-
"min": 0,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-
"mean": 17.1001,
--
results/log_retention=50/sync=false/metrics-after-load-workloada.txt:
"name": "bloom_lookups_per_op",
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-
"total_count": 135526576,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-
"min": 0,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-
"mean": 17.4048,
--
results/log_retention=50/sync=false/metrics-after-load-workloada.txt:
"name": "bloom_lookups_per_op",
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-
"total_count": 93424480,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-
"min": 0,
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-
"mean": 12.9333,
--
results/log_retention=default/sync=false/metrics-after-load-workloada.txt:
"name": "bloom_lookups_per_op",
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-
"total_count": 93424480,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-
"min": 0,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-
"mean": 25.2444,
--
results/log_retention=default/sync=false/metrics-after-load-workloada.txt:
"name": "bloom_lookups_per_op",
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-
"total_count": 135526576,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-
"min": 0,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-
"mean": 37.8615,
--
results/log_retention=default/sync=false/metrics-after-load-workloada.txt:
"name": "bloom_lookups_per_op",
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-
"total_count": 135524034,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-
"min": 0,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-
"mean": 39.7972,
--
results/log_retention=default/sync=false/metrics-after-load-workloada.txt:
"name": "bloom_lookups_per_op",
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-
"total_count": 135524910,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-
"min": 0,
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-
"mean": 39.3984,
{code}
and total IO counters substantially lower:
{code}
results/log_retention=50/sync=false/metrics-after-load-workloada.txt:
"name": "block_manager_total_bytes_written",
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-
"value": 168846243283
--
results/log_retention=50/sync=false/metrics-after-load-workloada.txt:
"name": "block_manager_total_bytes_read",
results/log_retention=50/sync=false/metrics-after-load-workloada.txt-
"value": 4969048266023
--
results/log_retention=default/sync=false/metrics-after-load-workloada.txt:
"name": "block_manager_total_bytes_written",
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-
"value": 285113846979
--
results/log_retention=default/sync=false/metrics-after-load-workloada.txt:
"name": "block_manager_total_bytes_read",
results/log_retention=default/sync=false/metrics-after-load-workloada.txt-
"value": 12224412188591
{code}
So I think the benchmarks confirm that fixing this would be good
bang-for-the-buck perf improvement.
> Short default for log retention increases write amplification
> -------------------------------------------------------------
>
> Key: KUDU-1567
> URL: https://issues.apache.org/jira/browse/KUDU-1567
> Project: Kudu
> Issue Type: Improvement
> Components: perf, tserver
> Affects Versions: 0.10.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
>
> Currently the maintenance manager prioritizes flushes over compactions if the
> flush operations are retaining WAL segments. The goal here is to prevent the
> amount of in-memory data from getting so large that restarts would be
> incredibly slow. However, it has a somewhat unintuitive negative effect on
> performance:
> - with the default of retaining just two segments, flushes become highly
> prioritized when the MRS only has ~128MB of data, regardless of the
> "flush_threshold_mb" configuration
> - this creates lots of overlapping rowsets in the case of random-write
> applications
> - because flushes are prioritized over compactions, compactions rarely run
> - the frequent flushes, combined with low priority of compactions, means that
> after a few days of constant inserts, we often end up with average "bloom
> lookups per op" metrics of 50-100, which is quite slow even if the blooms fit
> in cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
