wxmimperio created KUDU-2995:
--------------------------------
Summary: Tuning kudu write performance
Key: KUDU-2995
URL: https://issues.apache.org/jira/browse/KUDU-2995
Project: Kudu
Issue Type: Test
Components: client
Affects Versions: 1.7.0
Environment: kudu 1.7.0-cdh5.16.2
Reporter: wxmimperio
Kudu version： kudu 1.7.0-cdh5.16.2
System memary pre node：256G
4 SSDs per machine：512G
Three Master nodes and three Tserver nodes.
{code:java}
// Master config
--fs_wal_dir=/mnt/disk1/kudu/var/wal
--fs_data_dirs=/mnt/disk1/kudu/var/data,/mnt/disk2/kudu/var/data,/mnt/disk3/kudu/var/data,/mnt/disk4/kudu/var/data
--fs_metadata_dir=/mnt/disk1/kudu/var/metadata
--log_dir=/mnt/disk1/kudu/var/logs
--master_addresses=xxxx
--maintenance_manager_num_threads=2
--block_cache_capacity_mb=6144
--memory_limit_hard_bytes=34359738368
--max_log_size=40
// Tserver config
--fs_wal_dir=/mnt/disk1/kudu/var/wal
--fs_data_dirs=/mnt/disk1/kudu/var/data,/mnt/disk2/kudu/var/data,/mnt/disk3/kudu/var/data,/mnt/disk4/kudu/var/data
--fs_metadata_dir=/mnt/disk1/kudu/var/metadata
--log_dir=/mnt/disk1/kudu/var/logs
--tserver_master_addrs=xxxx
--block_cache_capacity_mb=6144
--memory_limit_hard_bytes=34359738368
--max_log_size=40
// Table schema
// _key is UUID for each msg
// event_time is data time
// Schema has only 15 columns
// Single message does not exceed 100Bytes
HASH (_key) PARTITIONS 3,
RANGE (event_time) (
PARTITION 2019-10-31T16:00:00.000000Z <= VALUES <
2019-11-30T16:00:00.000000Z
)
{code}
I write a project to write data to kudu.
Whether manual or automatic flush mode write speed is only 6M/s.
I think SSD should be more than this speed, and the network and memory have not
reached the bottleneck.
Is this the normal level of kudu writing? How to tuning?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
