keith-turner commented on PR #4568: URL: https://github.com/apache/accumulo/pull/4568#issuecomment-2117942635
Without the changes from this PR and #4563 seeing SplitMillionIT take a total of 11 min 8 sec (upped the timeout). Below are some highlights from the logs. ``` 2024-05-17T11:16:28,184 32 [functional.SplitMillionIT] INFO : Time to compact all tablets : 177606ms 2024-05-17T11:16:28,297 32 [functional.SplitMillionIT] INFO : Time to verify 200 rows was 113ms 2024-05-17T11:17:04,720 32 [functional.SplitMillionIT] INFO : Time to clone table : 36422ms 2024-05-17T11:17:05,433 32 [clientImpl.ClientTabletCacheImpl] DEBUG: Requesting hosting for 100 ondemand tablets for table id 2. 2024-05-17T11:17:27,405 32 [functional.SplitMillionIT] INFO : Time to verify 200 rows was 22685ms 2024-05-17T11:20:35,323 32 [functional.SplitMillionIT] INFO : Time to merge all tablets : 187917ms 2024-05-17T11:20:36,123 745 [clientImpl.ClientTabletCacheImpl] DEBUG: Requesting hosting for 1 ondemand tablets for table id 2. 2024-05-17T11:20:41,610 32 [functional.SplitMillionIT] INFO : Time to verify 200 rows was 6287ms 2024-05-17T11:22:13,490 32 [functional.SplitMillionIT] INFO : Time to delete table : 91880ms ``` With the changes from this PR and #4563 seeing SplitMillionIT take a total of 9 min 44 sec. Its squeaking by under the 10 min threshold. Still seems like it used to be faster when running it locally. Below are some highlights. See noticeable improvements in compact and delete table, but not merge. I think merge is what I need to look into next. Also going to open an issue about improving the time to host ondemand tablets based on some thing I have seen. Below and above where it request hosting of 100 tablets and takes 10s of seconds that should be faster. Part of the reason that was slow was because of how long it takes to scan all of the metadata, which is what lead to opening this PR. However more could be done to improve on demand hosting. ``` 2024-05-17T11:41:05,680 34 [functional.SplitMillionIT] INFO : Time to compact all tablets : 148425ms 2024-05-17T11:41:05,817 34 [functional.SplitMillionIT] INFO : Time to verify 200 rows was 137ms 2024-05-17T11:41:32,089 34 [functional.SplitMillionIT] INFO : Time to clone table : 26271ms 2024-05-17T11:41:32,813 34 [clientImpl.ClientTabletCacheImpl] DEBUG: Requesting hosting for 100 ondemand tablets for table id 2. 2024-05-17T11:41:45,478 34 [functional.SplitMillionIT] INFO : Time to verify 200 rows was 13389ms 2024-05-17T11:44:49,188 34 [functional.SplitMillionIT] INFO : Time to merge all tablets : 183710ms 2024-05-17T11:44:49,445 747 [clientImpl.ClientTabletCacheImpl] DEBUG: Requesting hosting for 1 ondemand tablets for table id 2. 2024-05-17T11:44:50,221 34 [functional.SplitMillionIT] INFO : Time to verify 200 rows was 1033ms 2024-05-17T11:46:08,001 34 [functional.SplitMillionIT] INFO : Time to delete table : 77780ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
