[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135181#comment-17135181
]
Danil Lipovoy edited comment on HBASE-23887 at 6/14/20, 2:47 PM:
-----------------------------------------------------------------
[~bharathv]
I found out why scan was slow - the bottle neck was network (I sent requests
form other PC). Now I scan local and the network is not the problem.
So I run a test scenario:
1. Scan (25 threads, batch = 100)
2. After 5 minutes add multi-gets (25 threads, batch = 100)
3. After 5 minutes switch off multi-gets (only scan again)
During the test have checked the ratio between single and multi caches (added
it into log file).
The first run with *hbase.lru.cache.heavy.eviction.count.limit* = 10000
(disable the feature) and the second the limit = 0.
!scan_and_gets.png!
Take a look on the ratio (single/multi).
Log file of the first run (the feature is disabled):
evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0,
current caching DataBlock (%): 100
evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter:
0, current caching DataBlock (%): 100
evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter:
0, current caching DataBlock (%): 100
evicted (MB): 114, ratio 5.78, overhead (%): -43, heavy eviction counter: 0,
current caching DataBlock (%): 100 < start the test
evicted (MB): 5704, ratio 1.07, overhead (%): 2752, heavy eviction counter: 1,
current caching DataBlock (%): 100
evicted (MB): 5245, ratio 1.06, overhead (%): 2522, heavy eviction counter: 2,
current caching DataBlock (%): 100
evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter: 3,
current caching DataBlock (%): 100
evicted (MB): 4788, ratio 1.06, overhead (%): 2294, heavy eviction counter: 4,
current caching DataBlock (%): 100
evicted (MB): 5132, ratio 1.06, overhead (%): 2466, heavy eviction counter: 5,
current caching DataBlock (%): 100
evicted (MB): 5018, ratio 1.07, overhead (%): 2409, heavy eviction counter: 6,
current caching DataBlock (%): 100
evicted (MB): 5244, ratio 1.06, overhead (%): 2522, heavy eviction counter: 7,
current caching DataBlock (%): 100
evicted (MB): 5019, ratio 1.07, overhead (%): 2409, heavy eviction counter: 8,
current caching DataBlock (%): 100
evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter: 9,
current caching DataBlock (%): 100
evicted (MB): 4904, ratio 1.06, overhead (%): 2352, heavy eviction counter:
10, current caching DataBlock (%): 100
evicted (MB): 5017, ratio 1.06, overhead (%): 2408, heavy eviction counter:
11, current caching DataBlock (%): 100
evicted (MB): 4563, ratio 1.06, overhead (%): 2181, heavy eviction counter:
12, current caching DataBlock (%): 100
evicted (MB): 4338, ratio 1.06, overhead (%): 2069, heavy eviction counter:
13, current caching DataBlock (%): 100
evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter:
14, current caching DataBlock (%): 100
evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter:
15, current caching DataBlock (%): 100
evicted (MB): 5130, ratio 1.06, overhead (%): 2465, heavy eviction counter:
16, current caching DataBlock (%): 100
evicted (MB): 5017, ratio 1.06, overhead (%): 2408, heavy eviction counter:
17, current caching DataBlock (%): 100
evicted (MB): 4795, ratio 1.06, overhead (%): 2297, heavy eviction counter:
18, current caching DataBlock (%): 100
evicted (MB): 4905, ratio 1.07, overhead (%): 2352, heavy eviction counter:
19, current caching DataBlock (%): 100
evicted (MB): 4911, ratio 1.06, overhead (%): 2355, heavy eviction counter:
20, current caching DataBlock (%): 100
evicted (MB): 5019, ratio 1.06, overhead (%): 2409, heavy eviction counter:
21, current caching DataBlock (%): 100
evicted (MB): 5134, ratio 1.06, overhead (%): 2467, heavy eviction counter:
22, current caching DataBlock (%): 100
evicted (MB): 5016, ratio 1.06, overhead (%): 2408, heavy eviction counter:
23, current caching DataBlock (%): 100
evicted (MB): 4450, ratio 1.06, overhead (%): 2125, heavy eviction counter:
24, current caching DataBlock (%): 100
evicted (MB): 4904, ratio 1.07, overhead (%): 2352, heavy eviction counter:
25, current caching DataBlock (%): 100
evicted (MB): 4561, ratio 1.06, overhead (%): 2180, heavy eviction counter:
26, current caching DataBlock (%): 100
evicted (MB): 4334, ratio 1.06, overhead (%): 2067, heavy eviction counter:
27, current caching DataBlock (%): 100
evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter:
28, current caching DataBlock (%): 100
evicted (MB): 4792, ratio 1.06, overhead (%): 2296, heavy eviction counter:
29, current caching DataBlock (%): 100
evicted (MB): 4903, ratio 1.07, overhead (%): 2351, heavy eviction counter:
30, current caching DataBlock (%): 100
evicted (MB): 4791, ratio 1.06, overhead (%): 2295, heavy eviction counter:
31, current caching DataBlock (%): 100
evicted (MB): 4677, ratio 1.07, overhead (%): 2238, heavy eviction counter:
32, current caching DataBlock (%): 100
evicted (MB): 4903, ratio 1.06, overhead (%): 2351, heavy eviction counter:
33, current caching DataBlock (%): 100
evicted (MB): 4791, ratio 1.06, overhead (%): 2295, heavy eviction counter:
34, current caching DataBlock (%): 100
evicted (MB): 4677, ratio 1.06, overhead (%): 2238, heavy eviction counter:
35, current caching DataBlock (%): 100
evicted (MB): 4909, ratio 1.06, overhead (%): 2354, heavy eviction counter:
36, current caching DataBlock (%): 100
evicted (MB): 4678, ratio 1.06, overhead (%): 2239, heavy eviction counter:
37, current caching DataBlock (%): 100
evicted (MB): 4788, ratio 1.06, overhead (%): 2294, heavy eviction counter:
38, current caching DataBlock (%): 100
evicted (MB): 5133, ratio 1.06, overhead (%): 2466, heavy eviction counter:
39, current caching DataBlock (%): 100
evicted (MB): 5017, ratio 1.06, overhead (%): 2408, heavy eviction counter:
40, current caching DataBlock (%): 100
evicted (MB): 4903, ratio 1.06, overhead (%): 2351, heavy eviction counter:
41, current caching DataBlock (%): 100
evicted (MB): 4789, ratio 1.07, overhead (%): 2294, heavy eviction counter:
42, current caching DataBlock (%): 100
evicted (MB): 4790, ratio 1.06, overhead (%): 2295, heavy eviction counter:
43, current caching DataBlock (%): 100
evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter:
44, current caching DataBlock (%): 100
evicted (MB): 4449, ratio 1.04, overhead (%): 2124, heavy eviction counter:
45, current caching DataBlock (%): 100
evicted (MB): 5672, ratio 1.11, overhead (%): 2736, heavy eviction counter:
46, current caching DataBlock (%): 100 < added gets
evicted (MB): 5658, ratio 1.11, overhead (%): 2729, heavy eviction counter:
47, current caching DataBlock (%): 100
evicted (MB): 5891, ratio 1.11, overhead (%): 2845, heavy eviction counter:
48, current caching DataBlock (%): 100
evicted (MB): 6008, ratio 1.11, overhead (%): 2904, heavy eviction counter:
49, current caching DataBlock (%): 100
evicted (MB): 5832, ratio 1.15, overhead (%): 2816, heavy eviction counter:
50, current caching DataBlock (%): 100
evicted (MB): 6393, ratio 1.12, overhead (%): 3096, heavy eviction counter:
51, current caching DataBlock (%): 100
evicted (MB): 6218, ratio 1.11, overhead (%): 3009, heavy eviction counter:
52, current caching DataBlock (%): 100
evicted (MB): 5939, ratio 1.12, overhead (%): 2869, heavy eviction counter:
53, current caching DataBlock (%): 100
evicted (MB): 6369, ratio 1.11, overhead (%): 3084, heavy eviction counter:
54, current caching DataBlock (%): 100
evicted (MB): 6198, ratio 1.14, overhead (%): 2999, heavy eviction counter:
55, current caching DataBlock (%): 100
evicted (MB): 6170, ratio 1.13, overhead (%): 2985, heavy eviction counter:
56, current caching DataBlock (%): 100
evicted (MB): 6014, ratio 1.12, overhead (%): 2907, heavy eviction counter:
57, current caching DataBlock (%): 100
evicted (MB): 6359, ratio 1.11, overhead (%): 3079, heavy eviction counter:
58, current caching DataBlock (%): 100
evicted (MB): 6164, ratio 1.1, overhead (%): 2982, heavy eviction counter: 59,
current caching DataBlock (%): 100
evicted (MB): 6283, ratio 1.11, overhead (%): 3041, heavy eviction counter:
60, current caching DataBlock (%): 100
evicted (MB): 6037, ratio 1.12, overhead (%): 2918, heavy eviction counter:
61, current caching DataBlock (%): 100
evicted (MB): 6216, ratio 1.11, overhead (%): 3008, heavy eviction counter:
62, current caching DataBlock (%): 100
evicted (MB): 6053, ratio 1.11, overhead (%): 2926, heavy eviction counter:
63, current caching DataBlock (%): 100
evicted (MB): 6026, ratio 1.1, overhead (%): 2913, heavy eviction counter: 64,
current caching DataBlock (%): 100
evicted (MB): 5759, ratio 1.11, overhead (%): 2779, heavy eviction counter:
65, current caching DataBlock (%): 100
evicted (MB): 6313, ratio 1.1, overhead (%): 3056, heavy eviction counter: 66,
current caching DataBlock (%): 100
evicted (MB): 5979, ratio 1.11, overhead (%): 2889, heavy eviction counter:
67, current caching DataBlock (%): 100
evicted (MB): 6237, ratio 1.12, overhead (%): 3018, heavy eviction counter:
68, current caching DataBlock (%): 100
evicted (MB): 6354, ratio 1.13, overhead (%): 3077, heavy eviction counter:
69, current caching DataBlock (%): 100
evicted (MB): 6409, ratio 1.11, overhead (%): 3104, heavy eviction counter:
70, current caching DataBlock (%): 100
evicted (MB): 5788, ratio 1.11, overhead (%): 2794, heavy eviction counter:
71, current caching DataBlock (%): 100
evicted (MB): 5526, ratio 1.11, overhead (%): 2663, heavy eviction counter:
72, current caching DataBlock (%): 100
evicted (MB): 6032, ratio 1.12, overhead (%): 2916, heavy eviction counter:
73, current caching DataBlock (%): 100
evicted (MB): 6307, ratio 1.11, overhead (%): 3053, heavy eviction counter:
74, current caching DataBlock (%): 100
evicted (MB): 6277, ratio 1.1, overhead (%): 3038, heavy eviction counter: 75,
current caching DataBlock (%): 100
evicted (MB): 6091, ratio 1.11, overhead (%): 2945, heavy eviction counter:
76, current caching DataBlock (%): 100
evicted (MB): 5905, ratio 1.1, overhead (%): 2852, heavy eviction counter: 77,
current caching DataBlock (%): 100
evicted (MB): 6185, ratio 1.1, overhead (%): 2992, heavy eviction counter: 78,
current caching DataBlock (%): 100
evicted (MB): 6169, ratio 1.12, overhead (%): 2984, heavy eviction counter:
79, current caching DataBlock (%): 100
evicted (MB): 6268, ratio 1.11, overhead (%): 3034, heavy eviction counter:
80, current caching DataBlock (%): 100
evicted (MB): 6246, ratio 1.12, overhead (%): 3023, heavy eviction counter:
81, current caching DataBlock (%): 100
evicted (MB): 6332, ratio 1.13, overhead (%): 3066, heavy eviction counter:
82, current caching DataBlock (%): 100
evicted (MB): 6001, ratio 1.11, overhead (%): 2900, heavy eviction counter:
83, current caching DataBlock (%): 100
evicted (MB): 5931, ratio 1.1, overhead (%): 2865, heavy eviction counter: 84,
current caching DataBlock (%): 100
evicted (MB): 6197, ratio 1.1, overhead (%): 2998, heavy eviction counter: 85,
current caching DataBlock (%): 100
evicted (MB): 6291, ratio 1.11, overhead (%): 3045, heavy eviction counter:
86, current caching DataBlock (%): 100
evicted (MB): 6283, ratio 1.11, overhead (%): 3041, heavy eviction counter:
87, current caching DataBlock (%): 100
evicted (MB): 6040, ratio 1.07, overhead (%): 2920, heavy eviction counter:
88, current caching DataBlock (%): 100
evicted (MB): 5358, ratio 1.07, overhead (%): 2579, heavy eviction counter:
89, current caching DataBlock (%): 100 < interrupt gets
evicted (MB): 4903, ratio 1.06, overhead (%): 2351, heavy eviction counter:
90, current caching DataBlock (%): 100
evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter:
91, current caching DataBlock (%): 100
evicted (MB): 5016, ratio 1.06, overhead (%): 2408, heavy eviction counter:
92, current caching DataBlock (%): 100
evicted (MB): 4904, ratio 1.06, overhead (%): 2352, heavy eviction counter:
93, current caching DataBlock (%): 100
evicted (MB): 5019, ratio 1.07, overhead (%): 2409, heavy eviction counter:
94, current caching DataBlock (%): 100
evicted (MB): 4904, ratio 1.06, overhead (%): 2352, heavy eviction counter:
95, current caching DataBlock (%): 100
evicted (MB): 5130, ratio 1.06, overhead (%): 2465, heavy eviction counter:
96, current caching DataBlock (%): 100
evicted (MB): 4904, ratio 1.06, overhead (%): 2352, heavy eviction counter:
97, current caching DataBlock (%): 100
evicted (MB): 4903, ratio 1.07, overhead (%): 2351, heavy eviction counter:
98, current caching DataBlock (%): 100
evicted (MB): 4904, ratio 1.06, overhead (%): 2352, heavy eviction counter:
99, current caching DataBlock (%): 100
evicted (MB): 5017, ratio 1.06, overhead (%): 2408, heavy eviction counter:
100, current caching DataBlock (%): 100
evicted (MB): 4903, ratio 1.06, overhead (%): 2351, heavy eviction counter:
101, current caching DataBlock (%): 100
evicted (MB): 4788, ratio 1.07, overhead (%): 2294, heavy eviction counter:
102, current caching DataBlock (%): 100
evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter:
103, current caching DataBlock (%): 100
evicted (MB): 4903, ratio 1.06, overhead (%): 2351, heavy eviction counter:
104, current caching DataBlock (%): 100
evicted (MB): 4903, ratio 1.06, overhead (%): 2351, heavy eviction counter:
105, current caching DataBlock (%): 100
evicted (MB): 4674, ratio 1.06, overhead (%): 2237, heavy eviction counter:
106, current caching DataBlock (%): 100
evicted (MB): 4906, ratio 1.06, overhead (%): 2353, heavy eviction counter:
107, current caching DataBlock (%): 100
evicted (MB): 4904, ratio 1.06, overhead (%): 2352, heavy eviction counter:
108, current caching DataBlock (%): 100
evicted (MB): 5018, ratio 1.06, overhead (%): 2409, heavy eviction counter:
109, current caching DataBlock (%): 100
evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter:
110, current caching DataBlock (%): 100
evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter:
111, current caching DataBlock (%): 100
evicted (MB): 4902, ratio 1.07, overhead (%): 2351, heavy eviction counter:
112, current caching DataBlock (%): 100
evicted (MB): 5017, ratio 1.06, overhead (%): 2408, heavy eviction counter:
113, current caching DataBlock (%): 100
evicted (MB): 4674, ratio 1.06, overhead (%): 2237, heavy eviction counter:
114, current caching DataBlock (%): 100
evicted (MB): 4674, ratio 1.06, overhead (%): 2237, heavy eviction counter:
115, current caching DataBlock (%): 100
evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter:
116, current caching DataBlock (%): 100
evicted (MB): 4793, ratio 1.07, overhead (%): 2296, heavy eviction counter:
117, current caching DataBlock (%): 100
evicted (MB): 4790, ratio 1.06, overhead (%): 2295, heavy eviction counter:
118, current caching DataBlock (%): 100
evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter:
119, current caching DataBlock (%): 100
evicted (MB): 4562, ratio 1.06, overhead (%): 2181, heavy eviction counter:
120, current caching DataBlock (%): 100
evicted (MB): 4788, ratio 1.06, overhead (%): 2294, heavy eviction counter:
121, current caching DataBlock (%): 100
evicted (MB): 3801, ratio 1.02, overhead (%): 1800, heavy eviction counter:
122, current caching DataBlock (%): 100
evicted (MB): 0, ratio 0.0, overhead (%): -100, heavy eviction counter: 0,
current caching DataBlock (%): 100
So the ratio about 1.05 - 1.1 all the way.
continue in the next message...
was (Author: pustota):
[~bharathv]
I found out why scan was slow - the bottle neck was net (I sent requests form
other PC). Now I scan local and the net is not the problem.
So I run a test scenario:
1. Scan (25 threads, batch = 100)
2. After 5 minutes add multi-gets (25 threads, batch = 100)
3. After 5 minutes switch off multi-gets (only scan again)
During the test have checked the ratio between single and multi caches (added
it into log file).
The first run with *hbase.lru.cache.heavy.eviction.count.limit* = 10000
(disable the feature) and the second the limit = 0.
!scan_and_gets.png!
Take a look on the ratio (single/multi).
Log file of the first run (the feature is disabled):
evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0,
current caching DataBlock (%): 100
evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0,
current caching DataBlock (%): 100
evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0,
current caching DataBlock (%): 100
evicted (MB): 114, ratio 5.78, overhead (%): -43, heavy eviction counter: 0,
current caching DataBlock (%): 100 < start the test
evicted (MB): 5704, ratio 1.07, overhead (%): 2752, heavy eviction counter: 1,
current caching DataBlock (%): 100
evicted (MB): 5245, ratio 1.06, overhead (%): 2522, heavy eviction counter: 2,
current caching DataBlock (%): 100
evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter: 3,
current caching DataBlock (%): 100
evicted (MB): 4788, ratio 1.06, overhead (%): 2294, heavy eviction counter: 4,
current caching DataBlock (%): 100
evicted (MB): 5132, ratio 1.06, overhead (%): 2466, heavy eviction counter: 5,
current caching DataBlock (%): 100
evicted (MB): 5018, ratio 1.07, overhead (%): 2409, heavy eviction counter: 6,
current caching DataBlock (%): 100
evicted (MB): 5244, ratio 1.06, overhead (%): 2522, heavy eviction counter: 7,
current caching DataBlock (%): 100
evicted (MB): 5019, ratio 1.07, overhead (%): 2409, heavy eviction counter: 8,
current caching DataBlock (%): 100
evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter: 9,
current caching DataBlock (%): 100
evicted (MB): 4904, ratio 1.06, overhead (%): 2352, heavy eviction counter: 10,
current caching DataBlock (%): 100
evicted (MB): 5017, ratio 1.06, overhead (%): 2408, heavy eviction counter: 11,
current caching DataBlock (%): 100
evicted (MB): 4563, ratio 1.06, overhead (%): 2181, heavy eviction counter: 12,
current caching DataBlock (%): 100
evicted (MB): 4338, ratio 1.06, overhead (%): 2069, heavy eviction counter: 13,
current caching DataBlock (%): 100
evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter: 14,
current caching DataBlock (%): 100
evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter: 15,
current caching DataBlock (%): 100
evicted (MB): 5130, ratio 1.06, overhead (%): 2465, heavy eviction counter: 16,
current caching DataBlock (%): 100
evicted (MB): 5017, ratio 1.06, overhead (%): 2408, heavy eviction counter: 17,
current caching DataBlock (%): 100
evicted (MB): 4795, ratio 1.06, overhead (%): 2297, heavy eviction counter: 18,
current caching DataBlock (%): 100
evicted (MB): 4905, ratio 1.07, overhead (%): 2352, heavy eviction counter: 19,
current caching DataBlock (%): 100
evicted (MB): 4911, ratio 1.06, overhead (%): 2355, heavy eviction counter: 20,
current caching DataBlock (%): 100
evicted (MB): 5019, ratio 1.06, overhead (%): 2409, heavy eviction counter: 21,
current caching DataBlock (%): 100
evicted (MB): 5134, ratio 1.06, overhead (%): 2467, heavy eviction counter: 22,
current caching DataBlock (%): 100
evicted (MB): 5016, ratio 1.06, overhead (%): 2408, heavy eviction counter: 23,
current caching DataBlock (%): 100
evicted (MB): 4450, ratio 1.06, overhead (%): 2125, heavy eviction counter: 24,
current caching DataBlock (%): 100
evicted (MB): 4904, ratio 1.07, overhead (%): 2352, heavy eviction counter: 25,
current caching DataBlock (%): 100
evicted (MB): 4561, ratio 1.06, overhead (%): 2180, heavy eviction counter: 26,
current caching DataBlock (%): 100
evicted (MB): 4334, ratio 1.06, overhead (%): 2067, heavy eviction counter: 27,
current caching DataBlock (%): 100
evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter: 28,
current caching DataBlock (%): 100
evicted (MB): 4792, ratio 1.06, overhead (%): 2296, heavy eviction counter: 29,
current caching DataBlock (%): 100
evicted (MB): 4903, ratio 1.07, overhead (%): 2351, heavy eviction counter: 30,
current caching DataBlock (%): 100
evicted (MB): 4791, ratio 1.06, overhead (%): 2295, heavy eviction counter: 31,
current caching DataBlock (%): 100
evicted (MB): 4677, ratio 1.07, overhead (%): 2238, heavy eviction counter: 32,
current caching DataBlock (%): 100
evicted (MB): 4903, ratio 1.06, overhead (%): 2351, heavy eviction counter: 33,
current caching DataBlock (%): 100
evicted (MB): 4791, ratio 1.06, overhead (%): 2295, heavy eviction counter: 34,
current caching DataBlock (%): 100
evicted (MB): 4677, ratio 1.06, overhead (%): 2238, heavy eviction counter: 35,
current caching DataBlock (%): 100
evicted (MB): 4909, ratio 1.06, overhead (%): 2354, heavy eviction counter: 36,
current caching DataBlock (%): 100
evicted (MB): 4678, ratio 1.06, overhead (%): 2239, heavy eviction counter: 37,
current caching DataBlock (%): 100
evicted (MB): 4788, ratio 1.06, overhead (%): 2294, heavy eviction counter: 38,
current caching DataBlock (%): 100
evicted (MB): 5133, ratio 1.06, overhead (%): 2466, heavy eviction counter: 39,
current caching DataBlock (%): 100
evicted (MB): 5017, ratio 1.06, overhead (%): 2408, heavy eviction counter: 40,
current caching DataBlock (%): 100
evicted (MB): 4903, ratio 1.06, overhead (%): 2351, heavy eviction counter: 41,
current caching DataBlock (%): 100
evicted (MB): 4789, ratio 1.07, overhead (%): 2294, heavy eviction counter: 42,
current caching DataBlock (%): 100
evicted (MB): 4790, ratio 1.06, overhead (%): 2295, heavy eviction counter: 43,
current caching DataBlock (%): 100
evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter: 44,
current caching DataBlock (%): 100
evicted (MB): 4449, ratio 1.04, overhead (%): 2124, heavy eviction counter: 45,
current caching DataBlock (%): 100
evicted (MB): 5672, ratio 1.11, overhead (%): 2736, heavy eviction counter: 46,
current caching DataBlock (%): 100 < added gets
evicted (MB): 5658, ratio 1.11, overhead (%): 2729, heavy eviction counter: 47,
current caching DataBlock (%): 100
evicted (MB): 5891, ratio 1.11, overhead (%): 2845, heavy eviction counter: 48,
current caching DataBlock (%): 100
evicted (MB): 6008, ratio 1.11, overhead (%): 2904, heavy eviction counter: 49,
current caching DataBlock (%): 100
evicted (MB): 5832, ratio 1.15, overhead (%): 2816, heavy eviction counter: 50,
current caching DataBlock (%): 100
evicted (MB): 6393, ratio 1.12, overhead (%): 3096, heavy eviction counter: 51,
current caching DataBlock (%): 100
evicted (MB): 6218, ratio 1.11, overhead (%): 3009, heavy eviction counter: 52,
current caching DataBlock (%): 100
evicted (MB): 5939, ratio 1.12, overhead (%): 2869, heavy eviction counter: 53,
current caching DataBlock (%): 100
evicted (MB): 6369, ratio 1.11, overhead (%): 3084, heavy eviction counter: 54,
current caching DataBlock (%): 100
evicted (MB): 6198, ratio 1.14, overhead (%): 2999, heavy eviction counter: 55,
current caching DataBlock (%): 100
evicted (MB): 6170, ratio 1.13, overhead (%): 2985, heavy eviction counter: 56,
current caching DataBlock (%): 100
evicted (MB): 6014, ratio 1.12, overhead (%): 2907, heavy eviction counter: 57,
current caching DataBlock (%): 100
evicted (MB): 6359, ratio 1.11, overhead (%): 3079, heavy eviction counter: 58,
current caching DataBlock (%): 100
evicted (MB): 6164, ratio 1.1, overhead (%): 2982, heavy eviction counter: 59,
current caching DataBlock (%): 100
evicted (MB): 6283, ratio 1.11, overhead (%): 3041, heavy eviction counter: 60,
current caching DataBlock (%): 100
evicted (MB): 6037, ratio 1.12, overhead (%): 2918, heavy eviction counter: 61,
current caching DataBlock (%): 100
evicted (MB): 6216, ratio 1.11, overhead (%): 3008, heavy eviction counter: 62,
current caching DataBlock (%): 100
evicted (MB): 6053, ratio 1.11, overhead (%): 2926, heavy eviction counter: 63,
current caching DataBlock (%): 100
evicted (MB): 6026, ratio 1.1, overhead (%): 2913, heavy eviction counter: 64,
current caching DataBlock (%): 100
evicted (MB): 5759, ratio 1.11, overhead (%): 2779, heavy eviction counter: 65,
current caching DataBlock (%): 100
evicted (MB): 6313, ratio 1.1, overhead (%): 3056, heavy eviction counter: 66,
current caching DataBlock (%): 100
evicted (MB): 5979, ratio 1.11, overhead (%): 2889, heavy eviction counter: 67,
current caching DataBlock (%): 100
evicted (MB): 6237, ratio 1.12, overhead (%): 3018, heavy eviction counter: 68,
current caching DataBlock (%): 100
evicted (MB): 6354, ratio 1.13, overhead (%): 3077, heavy eviction counter: 69,
current caching DataBlock (%): 100
evicted (MB): 6409, ratio 1.11, overhead (%): 3104, heavy eviction counter: 70,
current caching DataBlock (%): 100
evicted (MB): 5788, ratio 1.11, overhead (%): 2794, heavy eviction counter: 71,
current caching DataBlock (%): 100
evicted (MB): 5526, ratio 1.11, overhead (%): 2663, heavy eviction counter: 72,
current caching DataBlock (%): 100
evicted (MB): 6032, ratio 1.12, overhead (%): 2916, heavy eviction counter: 73,
current caching DataBlock (%): 100
evicted (MB): 6307, ratio 1.11, overhead (%): 3053, heavy eviction counter: 74,
current caching DataBlock (%): 100
evicted (MB): 6277, ratio 1.1, overhead (%): 3038, heavy eviction counter: 75,
current caching DataBlock (%): 100
evicted (MB): 6091, ratio 1.11, overhead (%): 2945, heavy eviction counter: 76,
current caching DataBlock (%): 100
evicted (MB): 5905, ratio 1.1, overhead (%): 2852, heavy eviction counter: 77,
current caching DataBlock (%): 100
evicted (MB): 6185, ratio 1.1, overhead (%): 2992, heavy eviction counter: 78,
current caching DataBlock (%): 100
evicted (MB): 6169, ratio 1.12, overhead (%): 2984, heavy eviction counter: 79,
current caching DataBlock (%): 100
evicted (MB): 6268, ratio 1.11, overhead (%): 3034, heavy eviction counter: 80,
current caching DataBlock (%): 100
evicted (MB): 6246, ratio 1.12, overhead (%): 3023, heavy eviction counter: 81,
current caching DataBlock (%): 100
evicted (MB): 6332, ratio 1.13, overhead (%): 3066, heavy eviction counter: 82,
current caching DataBlock (%): 100
evicted (MB): 6001, ratio 1.11, overhead (%): 2900, heavy eviction counter: 83,
current caching DataBlock (%): 100
evicted (MB): 5931, ratio 1.1, overhead (%): 2865, heavy eviction counter: 84,
current caching DataBlock (%): 100
evicted (MB): 6197, ratio 1.1, overhead (%): 2998, heavy eviction counter: 85,
current caching DataBlock (%): 100
evicted (MB): 6291, ratio 1.11, overhead (%): 3045, heavy eviction counter: 86,
current caching DataBlock (%): 100
evicted (MB): 6283, ratio 1.11, overhead (%): 3041, heavy eviction counter: 87,
current caching DataBlock (%): 100
evicted (MB): 6040, ratio 1.07, overhead (%): 2920, heavy eviction counter: 88,
current caching DataBlock (%): 100
evicted (MB): 5358, ratio 1.07, overhead (%): 2579, heavy eviction counter: 89,
current caching DataBlock (%): 100 < interrupt gets
evicted (MB): 4903, ratio 1.06, overhead (%): 2351, heavy eviction counter: 90,
current caching DataBlock (%): 100
evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter: 91,
current caching DataBlock (%): 100
evicted (MB): 5016, ratio 1.06, overhead (%): 2408, heavy eviction counter: 92,
current caching DataBlock (%): 100
evicted (MB): 4904, ratio 1.06, overhead (%): 2352, heavy eviction counter: 93,
current caching DataBlock (%): 100
evicted (MB): 5019, ratio 1.07, overhead (%): 2409, heavy eviction counter: 94,
current caching DataBlock (%): 100
evicted (MB): 4904, ratio 1.06, overhead (%): 2352, heavy eviction counter: 95,
current caching DataBlock (%): 100
evicted (MB): 5130, ratio 1.06, overhead (%): 2465, heavy eviction counter: 96,
current caching DataBlock (%): 100
evicted (MB): 4904, ratio 1.06, overhead (%): 2352, heavy eviction counter: 97,
current caching DataBlock (%): 100
evicted (MB): 4903, ratio 1.07, overhead (%): 2351, heavy eviction counter: 98,
current caching DataBlock (%): 100
evicted (MB): 4904, ratio 1.06, overhead (%): 2352, heavy eviction counter: 99,
current caching DataBlock (%): 100
evicted (MB): 5017, ratio 1.06, overhead (%): 2408, heavy eviction counter:
100, current caching DataBlock (%): 100
evicted (MB): 4903, ratio 1.06, overhead (%): 2351, heavy eviction counter:
101, current caching DataBlock (%): 100
evicted (MB): 4788, ratio 1.07, overhead (%): 2294, heavy eviction counter:
102, current caching DataBlock (%): 100
evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter:
103, current caching DataBlock (%): 100
evicted (MB): 4903, ratio 1.06, overhead (%): 2351, heavy eviction counter:
104, current caching DataBlock (%): 100
evicted (MB): 4903, ratio 1.06, overhead (%): 2351, heavy eviction counter:
105, current caching DataBlock (%): 100
evicted (MB): 4674, ratio 1.06, overhead (%): 2237, heavy eviction counter:
106, current caching DataBlock (%): 100
evicted (MB): 4906, ratio 1.06, overhead (%): 2353, heavy eviction counter:
107, current caching DataBlock (%): 100
evicted (MB): 4904, ratio 1.06, overhead (%): 2352, heavy eviction counter:
108, current caching DataBlock (%): 100
evicted (MB): 5018, ratio 1.06, overhead (%): 2409, heavy eviction counter:
109, current caching DataBlock (%): 100
evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter:
110, current caching DataBlock (%): 100
evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter:
111, current caching DataBlock (%): 100
evicted (MB): 4902, ratio 1.07, overhead (%): 2351, heavy eviction counter:
112, current caching DataBlock (%): 100
evicted (MB): 5017, ratio 1.06, overhead (%): 2408, heavy eviction counter:
113, current caching DataBlock (%): 100
evicted (MB): 4674, ratio 1.06, overhead (%): 2237, heavy eviction counter:
114, current caching DataBlock (%): 100
evicted (MB): 4674, ratio 1.06, overhead (%): 2237, heavy eviction counter:
115, current caching DataBlock (%): 100
evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter:
116, current caching DataBlock (%): 100
evicted (MB): 4793, ratio 1.07, overhead (%): 2296, heavy eviction counter:
117, current caching DataBlock (%): 100
evicted (MB): 4790, ratio 1.06, overhead (%): 2295, heavy eviction counter:
118, current caching DataBlock (%): 100
evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter:
119, current caching DataBlock (%): 100
evicted (MB): 4562, ratio 1.06, overhead (%): 2181, heavy eviction counter:
120, current caching DataBlock (%): 100
evicted (MB): 4788, ratio 1.06, overhead (%): 2294, heavy eviction counter:
121, current caching DataBlock (%): 100
evicted (MB): 3801, ratio 1.02, overhead (%): 1800, heavy eviction counter:
122, current caching DataBlock (%): 100
evicted (MB): 0, ratio 0.0, overhead (%): -100, heavy eviction counter: 0,
current caching DataBlock (%): 100
So the ratio about 1.05 - 1.1 all the way.
continue in the next message...
> BlockCache performance improve by reduce eviction rate
> ------------------------------------------------------
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
> Reporter: Danil Lipovoy
> Assignee: Danil Lipovoy
> Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new2_100p.png, requests_new_100p.png, scan.png, scan_and_gets.png,
> wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
