on 2019/9/10 13:47, 蒋涛涛 wrote:
尝试手段：
1. 手动迁移IO比较高的任务到其他机器，但是yarn任务提交比较随机，只能偶尔为之
2. 目前没有SSD，只能用普通STATA盘，目前加了两块盘提示磁盘IO能力，但是单盘对单任务的磁盘IO瓶颈还在
还有哪些策略可以解决或者缓解么？
It seems the tricks to improve RocksDB's throughput might be helpfu.
With writes and reads accessing mostly the recent data, our goal is to
let them stay in memory as much as possible without using up all the
memory on the server. The following parameters are worth tuning:
Block cache size: When uncompressed blocks are read from SSTables, they
are cached in memory. The amount of data that can be stored before
eviction policies apply is determined by the block cache size. The
bigger the better.
Write buffer size: How big can Memtable get before it is frozen.
Generally, the bigger the better. The tradeoff is that big write buffer
takes more memory and longer to flush to disk and to recover.
Write buffer number: How many Memtables to keep before flushing to
SSTable. Generally, the bigger the better. Similarly, the tradeoff is
that too many write buffers take up more memory and longer to flush to disk.
Minimum write buffers to merge: If most recently written keys are
frequently changed, it is better to only flush the latest version to
SSTable. This parameter controls how many Memtables it will try to merge
before flushing to SSTable. It should be less than the write buffer
number. A suggested value is 2. If the number is too big, it takes
longer to merge buffers and there is less chance of duplicate keys in
that many buffers.
The list above is far from being exhaustive, but tuning them correctly
can have a big impact on performance. Please refer to RocksDB’s Tuning
Guide for more details on these parameters. Figuring out the optimal
combination of values for all of them is an art in itself.
please ref: https://klaviyo.tech/flinkperf-c7bd28acc67
regards.
