?????? ??????????????????????????????????????rocksDB

Wed, 26 Feb 2020 17:54:18 -0800 

??????????1.10????????????


????????????????????????????????/data/hadoop/tmp/nm-local-dir/usercache/flink-user/appcache/application_1582163794765_0026/flink-io-21b91014-03a5-4d58-bafc-7e5b73f11ed0??????2??????????43G??????44G??


jstack??????????rocksdb????????????????????????????????????????????????????????rocksdb??????rocksdb:low??rocksdb:high??CPU????????????????1????????????????????tps????????????io????????????80??????????????rocksdb??????????????????128??????


??????????????????????????????????????????????????????????yarn??????????????node??32????128G????????5??????????????????32??????????????????????5????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????




------------------ ???????? ------------------
??????:&nbsp;"Yun Tang"<[email protected]&gt;;
????????:&nbsp;2020??2??27??(??????) ????2:07
??????:&nbsp;"user-zh"<[email protected]&gt;;

????:&nbsp;Re: ??????????????????????????????????????rocksDB



Hi

??????????rocksDB state size??????????????????????????metrics [1] 
??????????RocksDB????????????????RocksDB??????size??
????????????????????????rocksDB 
??????????????????????IO????????????????????????????????????????????jstack??????TM????????????????????task????????????????RocksDB??get????????????

RocksDB????????????????????kv????????????????????????????????block 
cache????????????????????????????????????????????????????????????????????????????rocksDB??????state??????

??????1.10 
??????????RocksDB??????????????????????????????????????????????????????????????

[1] 
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#state-backend-rocksdb-metrics-total-sst-files-size

????
????

________________________________
From: claylin <[email protected]&gt;
Sent: Wednesday, February 26, 2020 23:27
To: user-zh <[email protected]&gt;
Subject: ??????????????????????????????????????rocksDB

Hi 
????????????????????????rocksDB??????????????????????????tps??10w??????????????????????????rocksDB????????????sar
 -dp ??????????io??????
Average:&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; DEV&amp;nbsp; 
&amp;nbsp; &amp;nbsp; &amp;nbsp;tps&amp;nbsp; rd_sec/s&amp;nbsp; 
wr_sec/s&amp;nbsp; avgrq-sz&amp;nbsp; avgqu-sz&amp;nbsp; &amp;nbsp; 
&amp;nbsp;await&amp;nbsp; &amp;nbsp; &amp;nbsp;svctm&amp;nbsp; &amp;nbsp; 
&amp;nbsp;%util
Average:&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; sda&amp;nbsp; 
&amp;nbsp; 285.36&amp;nbsp; &amp;nbsp;2152.91&amp;nbsp; 88322.99&amp;nbsp; 
&amp;nbsp; 317.06&amp;nbsp; &amp;nbsp; &amp;nbsp;21.48&amp;nbsp; &amp;nbsp; 
&amp;nbsp;75.27&amp;nbsp; &amp;nbsp; &amp;nbsp; 0.58&amp;nbsp; &amp;nbsp; 
&amp;nbsp;16.60
??????????????????????????????????????????https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#tuning-rocksdb????????????????????Manager
 
Memory??????rocksDB??flush??compact??????????????????????????????????????????????????????????????state????????????


??????????????????????????????????????????????

回复