Will Berkeley created KUDU-2318:
-----------------------------------
Summary: Kudu flushes aggressively even if it can't relieve memory
pressure.
Key: KUDU-2318
URL: https://issues.apache.org/jira/browse/KUDU-2318
Project: Kudu
Issue Type: Bug
Affects Versions: 1.6.0
Reporter: Will Berkeley
Attachments: rowsetblackbarofdeath.png
Kudu starts flushing aggressively when memory usage exceeds 60% of the hard
limit. In pathological cases, this can cause Kudu to flush an extremely large
amount of small rowsets.
For example, if -block_cache_capacity_mb is set to over 60% of the hard limit,
and the block cache fills, then Kudu will always be under memory pressure.
Flushing won't ever be able to reduce memory usage under the aggressive
flushing threshold. However, Kudu will still flush, producing lots of tiny
rowsets. This can eventually cause problems like KUDU-2317.
Attached is the rowset diagram from a tablet showing this phenomenon. To
produce this I
1. Ran the tablet servers with -memory_limit_hard_bytes=1048576000 (1GiB) and
-block_cache_capacity_mb=750.
2. Started an insert workload (single-threaded so insert would be sequential):
build/latest/bin/kudu perf loadgen -keep_auto_table -num_threads=1
-num_rows_per_thread=100000000.
3. Waited a bit for tablet servers to have a gigabyte or two of data.
4. Ran a ksck checksum set to cache blocks: build/latest/bin/kudu cluster ksck
-checksum_scan -checksum_cache_blocks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
