Some thoughts on how you might increase your write speed: - Don't use the same disk for both WAL and data directories. If you have enough disks, dedicate one for the WAL and the rest for data directories. - Since each disk is an SSD, experiment with a higher ratio of MM threads to data directories. We typically recommend 1:3, but that's for spinning disks. I see you've configured 2 MM threads for the masters but are still using just 1 for the tservers? Consider using 2-4. - How is your schema structured? Are you using hash partitioning? Range partitioning? Both? What's your primary key look like and does incoming data arrive in sorted order (or mostly sorted order) w.r.t. that key? Random order? https://kudu.apache.org/docs/schema_design.html is an excellent resource for understanding how schema can impact writes and reads.
On Wed, Nov 13, 2019 at 3:07 PM wei ximing <wxmimpe...@outlook.com> wrote: > > Hi! > > I have some questions about kudu performance tuning. > > Kudu version： kudu 1.7.0-cdh5.16.2 > > System memary pre node：256G > > 4 SSDs per machine：512G > > Three Master nodes and three Tserver nodes. > > // Master config > --fs_wal_dir=/mnt/disk1/kudu/var/wal > --fs_data_dirs=/mnt/disk1/kudu/var/data,/mnt/disk2/kudu/var/data,/mnt/disk3/kudu/var/data,/mnt/disk4/kudu/var/data > --fs_metadata_dir=/mnt/disk1/kudu/var/metadata > --log_dir=/mnt/disk1/kudu/var/logs > --master_addresses=xxxx > --maintenance_manager_num_threads=2 > --block_cache_capacity_mb=6144 > --memory_limit_hard_bytes=34359738368 > --max_log_size=40 > > // Tserver config > --fs_wal_dir=/mnt/disk1/kudu/var/wal > --fs_data_dirs=/mnt/disk1/kudu/var/data,/mnt/disk2/kudu/var/data,/mnt/disk3/kudu/var/data,/mnt/disk4/kudu/var/data > --fs_metadata_dir=/mnt/disk1/kudu/var/metadata > --log_dir=/mnt/disk1/kudu/var/logs > --tserver_master_addrs=xxxx > --block_cache_capacity_mb=6144 > --memory_limit_hard_bytes=34359738368 > --max_log_size=40 > > // Table schema > // _key is UUID for each msg > // event_time is data time > // Schema has only 15 columns > // Single message does not exceed 100Bytes > > HASH (_key) PARTITIONS 3, > RANGE (event_time) ( > PARTITION 2019-10-31T16:00:00.000000Z <= VALUES < > 2019-11-30T16:00:00.000000Z > ) > > I write a project to write data to kudu. > > Whether manual or automatic flush mode write speed is only 6MB/s. > > I think SSD should be more than this speed, and the network and memory have > not reached the bottleneck. > > Is this the normal level of kudu writing? How to tuning? > > > Thanks.
