Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
github-actions[bot] commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3604740309 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
hello-stephen commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3603591354 # BE Regression && UT Coverage Report Increment line coverage `92.00% (69/75)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/58567_ba10ea9e23c84e6b3a1cf767a5ac5b518a10d961_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/58567_ba10ea9e23c84e6b3a1cf767a5ac5b518a10d961_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 72.28% (24812/34328) | | Line Coverage | 58.90% (259991/441385) | | Region Coverage | 53.73% (215792/401607) | | Branch Coverage | 55.30% (92352/166997) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
doris-robot commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3603203190 # BE UT Coverage Report Increment line coverage `88.00% (66/75)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/ba10ea9e23c84e6b3a1cf767a5ac5b518a10d961_ba10ea9e23c84e6b3a1cf767a5ac5b518a10d961/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/ba10ea9e23c84e6b3a1cf767a5ac5b518a10d961_ba10ea9e23c84e6b3a1cf767a5ac5b518a10d961/report/index.html) | Category | Coverage | |---|| | Function Coverage | 53.51% (18736/35012) | | Line Coverage | 39.11% (172784/441780) | | Region Coverage | 33.73% (133823/396797) | | Branch Coverage | 34.66% (57576/166100) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
doris-robot commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3603078886 ClickBench: Total hot run time: 27.24 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit ba10ea9e23c84e6b3a1cf767a5ac5b518a10d961, data reload: false query1 0.050.040.05 query2 0.100.050.05 query3 0.260.090.08 query4 1.620.110.10 query5 0.260.250.26 query6 1.160.640.64 query7 0.030.030.03 query8 0.060.040.05 query9 0.580.520.50 query10 0.550.550.56 query11 0.160.100.12 query12 0.150.120.12 query13 0.620.600.62 query14 1.000.991.00 query15 0.820.790.82 query16 0.400.420.40 query17 1.091.051.00 query18 0.240.220.21 query19 1.951.801.82 query20 0.010.010.01 query21 15.46 0.300.14 query22 4.760.050.04 query23 16.00 0.290.11 query24 1.780.270.28 query25 0.100.050.05 query26 0.140.140.13 query27 0.080.060.05 query28 3.051.231.02 query29 12.63 3.963.18 query30 0.270.140.12 query31 2.810.630.39 query32 3.240.560.47 query33 3.073.093.11 query34 16.91 5.144.52 query35 4.544.504.56 query36 0.650.510.49 query37 0.110.070.06 query38 0.070.050.04 query39 0.050.030.04 query40 0.180.150.15 query41 0.090.030.03 query42 0.040.030.03 query43 0.040.040.03 Total cold run time: 97.18 s Total hot run time: 27.24 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
doris-robot commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3603047652 TPC-DS: Total hot run time: 182329 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit ba10ea9e23c84e6b3a1cf767a5ac5b518a10d961, data reload: false query1 1035416 422 416 query2 6581118211551155 query3 6743228 227 227 query4 25165 23428 23034 23034 query5 5885667 496 496 query6 343 245 223 223 query7 4659517 307 307 query8 347 253 235 235 query9 8721259725822582 query10 556 362 312 312 query11 15397 15089 14927 14927 query12 181 120 119 119 query13 1683559 450 450 query14 10931 603459755975 query15 257 205 183 183 query16 7709698 490 490 query17 1597806 645 645 query18 2058443 348 348 query19 269 216 190 190 query20 139 125 122 122 query21 236 140 116 116 query22 3832397738313831 query23 32926 32212 31925 31925 query24 8161244523992399 query25 633 549 502 502 query26 1227293 175 175 query27 2663523 353 353 query28 4332215821712158 query29 803 647 519 519 query30 312 244 219 219 query31 807 705 647 647 query32 93 76 78 76 query33 625 402 341 341 query34 846 877 592 592 query35 806 836 753 753 query36 908 925 850 850 query37 128 117 91 91 query38 3868390338713871 query39 1475140014101400 query40 226 134 123 123 query41 65 68 61 61 query42 131 118 111 111 query43 447 443 419 419 query44 1328757 745 745 query45 200 197 181 181 query46 910 1029667 667 query47 1699173116621662 query48 404 434 332 332 query49 785 536 401 401 query50 700 706 411 411 query51 3827392438883888 query52 110 121 112 112 query53 249 262 196 196 query54 306 298 287 287 query55 99 100 95 95 query56 348 336 329 329 query57 1155117010931093 query58 286 281 282 281 query59 2342253223242324 query60 381 365 330 330 query61 159 159 158 158 query62 809 708 668 668 query63 234 196 199 196 query64 45011212912 912 query65 4043397039583958 query66 1037442 342 342 query67 15449 15068 14888 14888 query68 8420993 631 631 query69 560 355 315 315 query70 11481040985 985 query71 530 343 332 332 query72 5836496448534853 query73 672 600 346 346 query74 8711872985748574 query75 3849307825302530 query76 37091149757 757 query77 815 428 318 318 query78 9616959388678867 query79 2725811 586 586 query80 695 600 501 501 query81 513 273 239 239 query82 492 142 125 125 query83 294 273 257 257 query84 267 127 115 115 query85 901 495 453 453 query86 341 292 287 287 query87 4131411640214021 query88 4191228622702270 query89 398 336 313 313 query90 1961226 220 220 query91 177 168 154 154 query92 91 71 77 71 query93 24491018656 656 query94 720 459 336 336 query95 493 414 409 409 query96 541 565 284 284 query97 2615266426292629 query98 246 225 215 215 query99 1381140012561256 Total cold run time: 276655 ms Total hot run time: 182329 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
doris-robot commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3602994362 TPC-H: Total hot run time: 34346 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit ba10ea9e23c84e6b3a1cf767a5ac5b518a10d961, data reload: false -- Round 1 -- q1 17641 508049584958 q2 2014353 214 214 q3 10223 1321725 725 q4 10213 896 328 328 q5 7557239721682168 q6 196 173 139 139 q7 962 758 640 640 q8 9359141411001100 q9 7045537253545354 q10 6798220117941794 q11 530 313 290 290 q12 342 367 240 240 q13 17776 370730203020 q14 236 231 221 221 q15 578 519 510 510 q16 908 855 832 832 q17 701 811 519 519 q18 7545720669626962 q19 1242988 611 611 q20 368 352 234 234 q21 4065319825362536 q22 10241043951 951 Total cold run time: 107323 ms Total hot run time: 34346 ms - Round 2, with runtime_filter_mode=off - q1 4960500149834983 q2 322 380 310 310 q3 2135265923432343 q4 1352176713461346 q5 4251418641924186 q6 211 169 126 126 q7 1895182016791679 q8 2533248623972397 q9 6982698469916984 q10 2909312126602660 q11 588 514 484 484 q12 670 711 564 564 q13 3310380330953095 q14 290 288 281 281 q15 558 508 519 508 q16 891 902 842 842 q17 1110130913101309 q18 7319709369886988 q19 854 812 826 812 q20 1938197818231823 q21 4738430841924192 q22 1086105610291029 Total cold run time: 50902 ms Total hot run time: 48941 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
Yukang-Lian commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3602908091 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
doris-robot commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3602903184 ClickBench: Total hot run time: 27.35 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit ba10ea9e23c84e6b3a1cf767a5ac5b518a10d961, data reload: false query1 0.050.050.05 query2 0.100.050.04 query3 0.260.100.09 query4 1.610.120.12 query5 0.280.270.27 query6 1.180.660.64 query7 0.040.030.03 query8 0.060.040.05 query9 0.560.500.51 query10 0.550.550.56 query11 0.160.110.12 query12 0.150.120.11 query13 0.620.620.61 query14 1.000.980.99 query15 0.810.800.79 query16 0.420.410.40 query17 1.001.051.02 query18 0.230.220.21 query19 1.891.891.79 query20 0.020.010.01 query21 15.46 0.300.14 query22 4.810.060.05 query23 16.14 0.280.11 query24 0.941.030.17 query25 0.090.050.08 query26 0.140.140.14 query27 0.040.050.05 query28 4.841.231.03 query29 12.63 3.983.30 query30 0.280.150.12 query31 2.820.640.39 query32 3.230.590.48 query33 3.083.103.08 query34 16.90 5.254.53 query35 4.614.524.56 query36 0.670.500.48 query37 0.110.070.06 query38 0.060.050.04 query39 0.040.030.03 query40 0.180.140.13 query41 0.080.030.04 query42 0.040.030.03 query43 0.050.040.03 Total cold run time: 98.23 s Total hot run time: 27.35 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
doris-robot commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3602880983 TPC-DS: Total hot run time: 182697 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit ba10ea9e23c84e6b3a1cf767a5ac5b518a10d961, data reload: false query1 1043418 401 401 query2 6594117211791172 query3 6743227 232 227 query4 25683 23482 22997 22997 query5 5092646 492 492 query6 334 250 231 231 query7 4655532 310 310 query8 316 244 247 244 query9 8751265126462646 query10 575 369 327 327 query11 15350 14933 14872 14872 query12 187 124 116 116 query13 1701589 446 446 query14 9434592559495925 query15 210 204 187 187 query16 7672711 545 545 query17 1228787 651 651 query18 2061446 357 357 query19 221 214 183 183 query20 133 128 124 124 query21 227 137 119 119 query22 3872404339203920 query23 33037 32062 32157 32062 query24 8533244324322432 query25 665 569 504 504 query26 1252293 175 175 query27 2701519 347 347 query28 4333216821562156 query29 824 648 515 515 query30 312 245 216 216 query31 832 700 644 644 query32 86 78 81 78 query33 653 408 348 348 query34 849 884 534 534 query35 843 828 760 760 query36 910 958 851 851 query37 127 116 91 91 query38 3826394938253825 query39 1471140214081402 query40 229 138 130 130 query41 67 61 61 61 query42 128 118 114 114 query43 453 454 408 408 query44 1331764 736 736 query45 201 193 183 183 query46 917 1021653 653 query47 1674170816431643 query48 415 427 336 336 query49 770 513 452 452 query50 701 710 431 431 query51 3992385638143814 query52 111 116 108 108 query53 242 262 192 192 query54 314 305 297 297 query55 98 93 90 90 query56 353 348 306 306 query57 1151118010801080 query58 288 283 273 273 query59 2265246123552355 query60 359 352 336 336 query61 195 153 157 153 query62 786 732 654 654 query63 236 190 200 190 query64 44761215909 909 query65 4018397839573957 query66 1122444 339 339 query67 15099 15113 14895 14895 query68 8457997 635 635 query69 513 341 318 318 query70 11261037997 997 query71 518 366 336 336 query72 5799489249054892 query73 722 637 351 351 query74 8724879288308792 query75 3722303825202520 query76 38491148757 757 query77 824 438 313 313 query78 9458962288818881 query79 2022858 578 578 query80 647 607 511 511 query81 495 274 240 240 query82 464 141 112 112 query83 263 272 250 250 query84 258 110 95 95 query85 902 483 442 442 query86 338 289 283 283 query87 4017402340514023 query88 3944230522872287 query89 388 329 299 299 query90 1977230 222 222 query91 168 171 141 141 query92 80 71 71 71 query93 15531055674 674 query94 723 450 309 309 query95 515 423 410 410 query96 569 565 293 293 query97 2682269925812581 query98 250 224 210 210 query99 1407136813471347 Total cold run time: 272701 ms Total hot run time: 182697 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
doris-robot commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3602810493 TPC-H: Total hot run time: 34658 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit ba10ea9e23c84e6b3a1cf767a5ac5b518a10d961, data reload: false -- Round 1 -- q1 17604 514649404940 q2 2015349 211 211 q3 10223 1306748 748 q4 10218 845 314 314 q5 7576244121622162 q6 188 173 143 143 q7 952 803 647 647 q8 9349142811381138 q9 6996533352995299 q10 6853219217941794 q11 527 311 293 293 q12 340 371 231 231 q13 18319 375432163216 q14 242 267 221 221 q15 588 517 515 515 q16 919 894 832 832 q17 712 811 579 579 q18 7929709569876987 q19 1184979 611 611 q20 366 359 234 234 q21 4041342226012601 q22 10341016942 942 Total cold run time: 108175 ms Total hot run time: 34658 ms - Round 2, with runtime_filter_mode=off - q1 5007497449484948 q2 338 413 318 318 q3 2150270022752275 q4 1334175013391339 q5 4248449646114496 q6 232 182 143 143 q7 2084208918871887 q8 2684259625812581 q9 7582746575937465 q10 3085325628472847 q11 605 525 521 521 q12 681 760 616 616 q13 3509406032243224 q14 290 320 291 291 q15 545 502 519 502 q16 931 928 887 887 q17 1225141414031403 q18 7955770076347634 q19 957 916 893 893 q20 2027197618221822 q21 4654430342514251 q22 10871030970 970 Total cold run time: 53210 ms Total hot run time: 51313 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
Yukang-Lian commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3602612944 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
hello-stephen commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3601866729 # BE Regression && UT Coverage Report Increment line coverage `70.67% (53/75)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/58567_8150ec43d78b06cd44e92d845936c1c9288fc45b_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/58567_8150ec43d78b06cd44e92d845936c1c9288fc45b_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 72.28% (24811/34328) | | Line Coverage | 58.91% (260032/441385) | | Region Coverage | 53.69% (215607/401607) | | Branch Coverage | 55.32% (92377/166997) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
doris-robot commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3601455698 # BE UT Coverage Report Increment line coverage `50.67% (38/75)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/8150ec43d78b06cd44e92d845936c1c9288fc45b_8150ec43d78b06cd44e92d845936c1c9288fc45b/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/8150ec43d78b06cd44e92d845936c1c9288fc45b_8150ec43d78b06cd44e92d845936c1c9288fc45b/report/index.html) | Category | Coverage | |---|| | Function Coverage | 53.51% (18735/35012) | | Line Coverage | 39.10% (172724/441780) | | Region Coverage | 33.70% (133730/396797) | | Branch Coverage | 34.66% (57569/166100) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
doris-robot commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3601398440 ClickBench: Total hot run time: 27.3 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 8150ec43d78b06cd44e92d845936c1c9288fc45b, data reload: false query1 0.060.060.05 query2 0.100.060.05 query3 0.250.090.09 query4 1.610.120.11 query5 0.280.260.25 query6 1.180.640.64 query7 0.040.030.02 query8 0.060.040.04 query9 0.590.500.51 query10 0.560.550.56 query11 0.150.120.11 query12 0.160.110.12 query13 0.630.620.61 query14 1.000.980.98 query15 0.820.810.81 query16 0.420.450.41 query17 0.991.031.03 query18 0.230.220.22 query19 1.891.781.88 query20 0.010.010.01 query21 15.45 0.290.15 query22 4.940.050.05 query23 16.22 0.280.10 query24 1.440.530.25 query25 0.100.060.06 query26 0.140.130.12 query27 0.060.050.05 query28 3.831.211.02 query29 12.58 4.123.23 query30 0.270.140.12 query31 2.810.650.40 query32 3.230.560.47 query33 3.043.093.04 query34 16.90 5.214.53 query35 4.594.504.57 query36 0.660.500.49 query37 0.110.060.06 query38 0.070.050.03 query39 0.040.030.03 query40 0.160.140.14 query41 0.080.040.03 query42 0.050.030.03 query43 0.050.030.03 Total cold run time: 97.85 s Total hot run time: 27.3 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
doris-robot commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3601380103 TPC-DS: Total hot run time: 182055 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 8150ec43d78b06cd44e92d845936c1c9288fc45b, data reload: false query1 1057404 394 394 query2 6601119611611161 query3 6744228 221 221 query4 25521 23310 22918 22918 query5 5062672 512 512 query6 339 244 236 236 query7 4657551 312 312 query8 312 267 248 248 query9 8767265426242624 query10 548 359 313 313 query11 15545 14907 14686 14686 query12 187 128 117 117 query13 1696595 472 472 query14 9550602160416021 query15 219 206 187 187 query16 7705716 534 534 query17 1216800 689 689 query18 2041425 332 332 query19 214 203 176 176 query20 134 128 120 120 query21 217 134 120 120 query22 3893393838603860 query23 32855 31752 32030 31752 query24 8425243224152415 query25 606 520 473 473 query26 1250272 173 173 query27 2756500 354 354 query28 4343215221322132 query29 812 633 495 495 query30 313 245 222 222 query31 871 727 620 620 query32 91 78 70 70 query33 594 449 334 334 query34 848 883 556 556 query35 811 851 748 748 query36 905 925 849 849 query37 123 121 91 91 query38 3862397738323832 query39 1464142714041404 query40 227 136 127 127 query41 66 63 63 63 query42 129 121 114 114 query43 428 452 418 418 query44 1373751 749 749 query45 204 192 186 186 query46 905 1038650 650 query47 1659171016431643 query48 396 433 330 330 query49 765 516 405 405 query50 713 715 420 420 query51 3979388541103885 query52 111 116 106 106 query53 259 270 195 195 query54 329 298 282 282 query55 99 97 90 90 query56 345 350 345 345 query57 1140116110801080 query58 301 287 285 285 query59 2351247023012301 query60 360 356 329 329 query61 185 158 153 153 query62 823 710 696 696 query63 236 204 200 200 query64 44221188886 886 query65 4036393939973939 query66 1125428 328 328 query67 15128 14976 14793 14793 query68 46041034635 635 query69 524 357 311 311 query70 11201052992 992 query71 448 357 320 320 query72 5789501951135019 query73 670 585 351 351 query74 8862883286988698 query75 3077306225202520 query76 33141145754 754 query77 516 425 345 345 query78 9414973288568856 query79 2045894 629 629 query80 1689623 582 582 query81 553 270 246 246 query82 404 142 114 114 query83 366 264 256 256 query84 262 120 93 93 query85 936 501 450 450 query86 388 294 278 278 query87 4128409039483948 query88 3068234822872287 query89 386 336 301 301 query90 1764230 227 227 query91 180 171 138 138 query92 73 72 70 70 query93 12271019662 662 query94 738 438 327 327 query95 510 414 406 406 query96 573 563 288 288 query97 2604269825902590 query98 243 221 212 212 query99 1308140413791379 Total cold run time: 267031 ms Total hot run time: 182055 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
github-actions[bot] commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3601376254 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
github-actions[bot] commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3601376461 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
doris-robot commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3601337869 TPC-H: Total hot run time: 34707 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 8150ec43d78b06cd44e92d845936c1c9288fc45b, data reload: false -- Round 1 -- q1 17600 507049534953 q2 2032320 208 208 q3 10245 1308754 754 q4 10220 873 325 325 q5 7521247422532253 q6 194 170 135 135 q7 987 796 663 663 q8 9345146311461146 q9 7062537353415341 q10 6877217918001800 q11 535 329 296 296 q12 353 379 225 225 q13 17762 376630653065 q14 236 244 217 217 q15 590 522 513 513 q16 894 873 813 813 q17 680 822 582 582 q18 7430717370257025 q19 1101950 611 611 q20 364 349 229 229 q21 4033341826042604 q22 1035974 949 949 Total cold run time: 107096 ms Total hot run time: 34707 ms - Round 2, with runtime_filter_mode=off - q1 5022498849824982 q2 334 396 323 323 q3 2131262923402340 q4 1337187813381338 q5 4246456646504566 q6 229 174 129 129 q7 2050199018871887 q8 2668252325872523 q9 7564750675297506 q10 3162325228222822 q11 613 530 500 500 q12 695 769 661 661 q13 3638398333923392 q14 287 295 277 277 q15 575 532 506 506 q16 925 943 927 927 q17 1271145914381438 q18 7832781975987598 q19 918 864 908 864 q20 1919196518301830 q21 4689431442254225 q22 11151020988 988 Total cold run time: 53220 ms Total hot run time: 51622 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
Yukang-Lian commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3601133197 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
gavinchou commented on code in PR #58567:
URL: https://github.com/apache/doris/pull/58567#discussion_r2579863870
##
be/src/olap/compaction.cpp:
##
@@ -1522,6 +1522,146 @@ int64_t CloudCompactionMixin::initiator() const {
return HashUtil::hash64(_uuid.data(), _uuid.size(), 0) &
std::numeric_limits::max();
}
+size_t
CloudCompactionMixin::truncate_rowsets_by_txn_size(std::vector&
rowsets,
+ int64_t&
estimated_size_bytes,
+ bool
enable_oplog_overhead,
+ int64_t&
rowset_meta_size_out,
+ int64_t&
delete_bitmap_size_out,
+ int64_t&
oplog_overhead_out) {
+if (rowsets.empty()) {
+estimated_size_bytes = 0;
+rowset_meta_size_out = 0;
+delete_bitmap_size_out = 0;
+oplog_overhead_out = 0;
+return 0;
+}
+
+const int64_t max_size = config::compaction_txn_max_size_bytes;
+const double oplog_ratio = config::compaction_oplog_overhead_ratio;
+const bool is_mow = _tablet->keys_type() == KeysType::UNIQUE_KEYS &&
+_tablet->enable_unique_key_merge_on_write();
+
+int64_t cumulative_meta_size = 0;
+int64_t cumulative_delete_bitmap_size = 0;
+int64_t cumulative_oplog_overhead = 0;
+int64_t cumulative_total_size = 0;
+size_t keep_count = 0;
+
+for (size_t i = 0; i < rowsets.size(); ++i) {
+const auto& rs = rowsets[i];
+
+// 1. Estimate rowset meta PB size
+int64_t rowset_meta_size =
rs->rowset_meta()->get_rowset_pb().ByteSizeLong();
Review Comment:
overlarge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
gavinchou commented on code in PR #58567: URL: https://github.com/apache/doris/pull/58567#discussion_r2579847275 ## be/src/cloud/config.h: ## @@ -96,6 +96,9 @@ DECLARE_mInt32(check_auto_compaction_interval_seconds); DECLARE_mInt32(max_base_compaction_task_num_per_disk); DECLARE_mBool(prioritize_query_perf_in_compaction); DECLARE_mInt32(compaction_max_rowset_count); +DECLARE_mInt64(compaction_txn_max_size_bytes); +DECLARE_mDouble(compaction_oplog_overhead_ratio); Review Comment: remove redundant configs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
hello-stephen commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3600457775 # BE Regression && UT Coverage Report Increment line coverage `68.55% (85/124)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/58567_638dd8dc1278b57f2f6319eec0a1d33bac82a7db_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/58567_638dd8dc1278b57f2f6319eec0a1d33bac82a7db_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 72.26% (24805/34329) | | Line Coverage | 58.90% (260001/441431) | | Region Coverage | 53.84% (216230/401618) | | Branch Coverage | 55.32% (92391/166999) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
hello-stephen commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3600243074 # BE UT Coverage Report Increment line coverage `50.00% (62/124)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/638dd8dc1278b57f2f6319eec0a1d33bac82a7db_638dd8dc1278b57f2f6319eec0a1d33bac82a7db/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/638dd8dc1278b57f2f6319eec0a1d33bac82a7db_638dd8dc1278b57f2f6319eec0a1d33bac82a7db/report/index.html) | Category | Coverage | |---|| | Function Coverage | 53.51% (18737/35013) | | Line Coverage | 39.11% (172798/441826) | | Region Coverage | 33.69% (133695/396808) | | Branch Coverage | 34.67% (57580/166102) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
Yukang-Lian commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3599961475 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
Yukang-Lian commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3597386129 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
gavinchou commented on code in PR #58567:
URL: https://github.com/apache/doris/pull/58567#discussion_r2576197203
##
cloud/src/meta-service/meta_service_job.cpp:
##
@@ -1290,8 +1289,44 @@ void process_compaction_job(MetaServiceCode& code,
std::string& msg, std::string
.tag("instance_id", instance_id)
.tag("operation_log_key", hex(operation_log_key))
.tag("tablet_id", tablet_id)
-.tag("recycle_rowsets_count", num_recycled_rowsets);
+.tag("recycle_rowsets_count", num_recycled_rowsets)
+.tag("operation_log_size", operation_log.ByteSizeLong());
}
+
+if (config::enable_compaction_txn_size_check) {
Review Comment:
can we check size of rowset meta when picking input rowsets on BE side?
is fails
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Compaction) Add compaction metadata size guard to prevent FDB transaction limit exceeded [doris]
hello-stephen commented on PR #58567: URL: https://github.com/apache/doris/pull/58567#issuecomment-3595098995 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
