keith-turner commented on PR #2707: URL: https://github.com/apache/accumulo/pull/2707#issuecomment-1130599393
I circled back and did my test and these changes resulted in a really significant improvement. I was running [this bit of code](https://github.com/keith-turner/accumulo-testing/blob/scan-server-testing/src/main/java/org/apache/accumulo/testing/continuous/ContinuousQuery.java) that I created to do random queries against CI data. I ran the code with parameters such that it would do small random queries in a single tablet using 500 threads. Below are the results which show the average times and entries returned for every 1000 scans done. With these changes and 500 concurrent queries the average time is 400ms and the max times are around 1000ms (ignoring the first few lines because JIT probably has not kicked in scan servers). Without these changes the average times are around 1600ms to 1700ms and the max times 15,000ms to 20,000ms. ``` With 2 scan server lock contention fixes, with the sha512 cache : hadoop@manager:/opt/accumulo-testing/sources/accumulo-testing-repo$ ./bin/cingest query 733333333333333e 79999999999999a5 16 500 2>&1 | grep STAT 2022-05-18T21:41:01,226 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:7263 avgTime:1717.988 minResults:0 maxResults:98 avgResults:22.314 2022-05-18T21:41:02,333 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:8380 avgTime:871.568 minResults:0 maxResults:98 avgResults:23.173 2022-05-18T21:41:03,221 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:9268 avgTime:1399.021 minResults:0 maxResults:97 avgResults:23.728 2022-05-18T21:41:04,003 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:25 maxTime:9967 avgTime:620.181 minResults:0 maxResults:94 avgResults:22.873 2022-05-18T21:41:04,736 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:33 maxTime:10187 avgTime:480.048 minResults:0 maxResults:101 avgResults:22.382 2022-05-18T21:41:05,482 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:38 maxTime:10965 avgTime:533.736 minResults:0 maxResults:106 avgResults:22.628 2022-05-18T21:41:06,250 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:30 maxTime:11534 avgTime:374.372 minResults:0 maxResults:94 avgResults:22.656 2022-05-18T21:41:07,038 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:18 maxTime:999 avgTime:402.28 minResults:0 maxResults:94 avgResults:23.957 2022-05-18T21:41:07,857 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:22 maxTime:962 avgTime:381.623 minResults:0 maxResults:95 avgResults:22.933 2022-05-18T21:41:08,688 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:15 maxTime:951 avgTime:420.283 minResults:0 maxResults:93 avgResults:21.669 2022-05-18T21:41:09,494 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:25 maxTime:1133 avgTime:417.134 minResults:0 maxResults:91 avgResults:22.704 2022-05-18T21:41:10,262 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:18 maxTime:998 avgTime:379.985 minResults:0 maxResults:103 avgResults:22.826 2022-05-18T21:41:11,063 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:15 maxTime:1174 avgTime:391.562 minResults:0 maxResults:98 avgResults:22.415 2022-05-18T21:41:11,861 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:21 maxTime:1005 avgTime:405.976 minResults:0 maxResults:91 avgResults:21.523 2022-05-18T21:41:12,669 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:34 maxTime:1061 avgTime:403.049 minResults:0 maxResults:105 avgResults:22.965 2022-05-18T21:41:13,430 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:17 maxTime:1077 avgTime:377.985 minResults:0 maxResults:98 avgResults:22.559 2022-05-18T21:41:14,223 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:21 maxTime:1003 avgTime:392.969 minResults:0 maxResults:112 avgResults:22.2 2022-05-18T21:41:14,986 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:18 maxTime:923 avgTime:383.816 minResults:0 maxResults:93 avgResults:22.59 2022-05-18T21:41:15,759 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:15 maxTime:970 avgTime:385.021 minResults:0 maxResults:97 avgResults:21.352 2022-05-18T21:41:16,562 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:11 maxTime:1098 avgTime:401.247 minResults:0 maxResults:104 avgResults:23.041 2022-05-18T21:41:17,336 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:11 maxTime:1074 avgTime:387.938 minResults:0 maxResults:105 avgResults:23.56 2022-05-18T21:41:18,131 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:19 maxTime:1522 avgTime:400.732 minResults:0 maxResults:104 avgResults:23.852 2022-05-18T21:41:18,890 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:24 maxTime:1347 avgTime:377.983 minResults:0 maxResults:90 avgResults:22.291 2022-05-18T21:41:19,669 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:16 maxTime:2075 avgTime:379.598 minResults:0 maxResults:97 avgResults:24.757 With 2 scan server lock contention fixes, Without the sha512 cache : hadoop@manager:/opt/accumulo-testing/sources/accumulo-testing-repo$ ./bin/cingest query 733333333333333e 79999999999999a5 16 500 2>&1 | grep STAT 2022-05-18T21:13:07,128 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:9294 avgTime:2934.887 minResults:0 maxResults:95 avgResults:21.559 2022-05-18T21:13:11,032 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:12401 avgTime:2558.46 minResults:0 maxResults:95 avgResults:22.671 2022-05-18T21:13:14,814 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:15932 avgTime:1787.61 minResults:0 maxResults:106 avgResults:22.142 2022-05-18T21:13:18,361 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:19075 avgTime:1757.055 minResults:0 maxResults:95 avgResults:22.1 2022-05-18T21:13:21,627 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:14242 avgTime:1787.193 minResults:0 maxResults:108 avgResults:23.424 2022-05-18T21:13:24,935 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:15593 avgTime:1668.114 minResults:0 maxResults:92 avgResults:22.891 2022-05-18T21:13:28,543 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:14493 avgTime:1664.027 minResults:0 maxResults:94 avgResults:24.173 2022-05-18T21:13:31,744 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:3 maxTime:14075 avgTime:1452.26 minResults:0 maxResults:93 avgResults:22.112 2022-05-18T21:13:35,330 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:15807 avgTime:1659.093 minResults:0 maxResults:103 avgResults:23.123 2022-05-18T21:13:38,536 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:19485 avgTime:1626.846 minResults:0 maxResults:103 avgResults:22.99 2022-05-18T21:13:42,135 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:21982 avgTime:1709.725 minResults:0 maxResults:103 avgResults:22.758 2022-05-18T21:13:45,235 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:17912 avgTime:1797.937 minResults:0 maxResults:91 avgResults:23.651 2022-05-18T21:13:48,547 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:16882 avgTime:1488.331 minResults:0 maxResults:104 avgResults:23.036 2022-05-18T21:13:51,931 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:12888 avgTime:1337.462 minResults:0 maxResults:107 avgResults:23.906 2022-05-18T21:13:55,143 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:20292 avgTime:1833.693 minResults:0 maxResults:100 avgResults:22.504 2022-05-18T21:13:58,369 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:18291 avgTime:1550.546 minResults:0 maxResults:100 avgResults:23.073 2022-05-18T21:14:01,646 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:19390 avgTime:1840.628 minResults:0 maxResults:93 avgResults:21.93 2022-05-18T21:14:05,563 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:16744 avgTime:2057.819 minResults:0 maxResults:85 avgResults:21.94 2022-05-18T21:14:08,630 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:17751 avgTime:1918.439 minResults:0 maxResults:94 avgResults:23.095 2022-05-18T21:14:12,124 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:24605 avgTime:1785.754 minResults:0 maxResults:106 avgResults:22.681 2022-05-18T21:14:15,330 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:12315 avgTime:1558.236 minResults:0 maxResults:100 avgResults:23.369 Without 2 scan server lock contention fixes, Without the sha512 cache : hadoop@manager:/opt/accumulo-testing/sources/accumulo-testing-repo$ ./bin/cingest query 733333333333333e 79999999999999a5 16 500 2>&1 | grep STAT 2022-05-18T21:26:54,209 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:9955 avgTime:3377.346 minResults:0 maxResults:104 avgResults:24.251 2022-05-18T21:26:58,112 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:12942 avgTime:2398.26 minResults:0 maxResults:101 avgResults:22.639 2022-05-18T21:27:01,732 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:16860 avgTime:1872.668 minResults:0 maxResults:93 avgResults:22.595 2022-05-18T21:27:05,113 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:12454 avgTime:1858.168 minResults:0 maxResults:89 avgResults:22.467 2022-05-18T21:27:08,592 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:13245 avgTime:1816.611 minResults:0 maxResults:109 avgResults:23.52 2022-05-18T21:27:11,941 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:10074 avgTime:1577.603 minResults:0 maxResults:98 avgResults:23.139 2022-05-18T21:27:15,537 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:10367 avgTime:1617.97 minResults:0 maxResults:99 avgResults:23.271 2022-05-18T21:27:18,940 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:16847 avgTime:1717.488 minResults:0 maxResults:103 avgResults:23.523 2022-05-18T21:27:22,121 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:3 maxTime:11905 avgTime:1754.433 minResults:0 maxResults:96 avgResults:23.501 2022-05-18T21:27:25,418 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:11685 avgTime:1628.917 minResults:0 maxResults:107 avgResults:22.256 2022-05-18T21:27:28,822 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:10381 avgTime:1533.018 minResults:0 maxResults:98 avgResults:21.994 2022-05-18T21:27:31,979 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:13414 avgTime:1715.61 minResults:0 maxResults:105 avgResults:22.88 2022-05-18T21:27:35,447 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:12916 avgTime:1939.198 minResults:0 maxResults:105 avgResults:22.622 2022-05-18T21:27:38,921 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:15182 avgTime:1690.404 minResults:0 maxResults:113 avgResults:23.795 2022-05-18T21:27:42,426 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:13339 avgTime:1672.008 minResults:0 maxResults:96 avgResults:21.484 2022-05-18T21:27:46,246 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:10578 avgTime:1716.462 minResults:0 maxResults:96 avgResults:22.72 2022-05-18T21:27:49,048 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:13267 avgTime:1575.544 minResults:0 maxResults:92 avgResults:23.481 2022-05-18T21:27:52,529 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:10375 avgTime:1833.692 minResults:0 maxResults:99 avgResults:23.189 2022-05-18T21:27:55,927 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:11099 avgTime:1505.09 minResults:0 maxResults:101 avgResults:22.699 2022-05-18T21:27:59,122 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:12778 avgTime:1692.452 minResults:0 maxResults:93 avgResults:22.965 2022-05-18T21:28:02,413 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:19010 avgTime:1612.813 minResults:0 maxResults:98 avgResults:22.725 2022-05-18T21:28:05,514 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:12986 avgTime:1662.435 minResults:0 maxResults:89 avgResults:22.258 2022-05-18T21:28:08,849 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:13979 avgTime:1559.41 minResults:0 maxResults:95 avgResults:22.965 2022-05-18T21:28:12,458 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:4 maxTime:13302 avgTime:1701.877 minResults:0 maxResults:93 avgResults:22.371 2022-05-18T21:28:15,937 [testing.continuous.ContinuousScanner] INFO : STATS count:1000 minTime:5 maxTime:11030 avgTime:1794.829 minResults:0 maxResults:97 avgResults:21.826 ``` Also another interesting data point is that when I profile without these changes all of the hottest methods are sha512 related in addition to the lock contention problem with getting the sha512 provider, do not see any Accumulo scan code in the hot methods. With these changes the hottest methods in profiling are all Accumulo scan code and there is not much lock contention (just a little bit around the secure random used to generate session ids). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
