Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
dataroaring merged PR #32980: URL: https://github.com/apache/doris/pull/32980 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2098039788 run cloud_p1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2097746227 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
github-actions[bot] commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2097364918 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
github-actions[bot] commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2097364847 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2097246766 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2096368618 TPC-DS: Total hot run time: 186442 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit f41f417864445dbb578e3030cf6dce6702c8a2e3, data reload: false query1 921 373 362 362 query2 6443238723042304 query3 6641213 210 210 query4 22758 21161 21198 21161 query5 4129414 418 414 query6 260 177 182 177 query7 4580282 279 279 query8 233 201 187 187 query9 8474240023782378 query10 429 250 250 250 query11 14662 14222 14126 14126 query12 133 94 91 91 query13 1647366 372 366 query14 9676763975447544 query15 215 172 178 172 query16 7749274 272 272 query17 1595570 584 570 query18 1948269 270 269 query19 206 142 151 142 query20 91 87 87 87 query21 198 128 127 127 query22 5021484248214821 query23 33707 33384 33360 33360 query24 5799284528842845 query25 486 370 367 367 query26 698 151 148 148 query27 1858309 322 309 query28 3785204620502046 query29 833 597 609 597 query30 246 150 152 150 query31 929 751 729 729 query32 58 51 52 51 query33 472 246 241 241 query34 881 465 466 465 query35 780 688 670 670 query36 1009921 892 892 query37 101 64 84 64 query38 3136303630043004 query39 1603151215091509 query40 204 125 124 124 query41 41 38 38 38 query42 105 97 98 97 query43 588 558 553 553 query44 1080732 755 732 query45 262 260 247 247 query46 1072720 719 719 query47 1962187118941871 query48 368 295 298 295 query49 765 410 395 395 query50 776 385 391 385 query51 6906680567636763 query52 108 92 88 88 query53 349 278 287 278 query54 279 259 241 241 query55 79 71 70 70 query56 243 221 221 221 query57 1206113311371133 query58 209 191 192 191 query59 3311324931933193 query60 276 232 233 232 query61 90 91 88 88 query62 566 455 463 455 query63 302 282 273 273 query64 7545721372477213 query65 3134305830253025 query66 787 340 337 337 query67 15386 15009 14979 14979 query68 5471521 522 521 query69 541 304 309 304 query70 1131113910941094 query71 436 274 269 269 query72 8025262223452345 query73 712 321 317 317 query74 6486621361236123 query75 3612268926072607 query76 37081040875 875 query77 572 263 264 263 query78 11039 10364 10227 10227 query79 4898523 523 523 query80 1833446 469 446 query81 511 217 230 217 query82 163691 96 91 query83 304 166 169 166 query84 264 86 92 86 query85 1165272 263 263 query86 467 289 326 289 query87 3306308330893083 query88 4891235223412341 query89 503 392 385 385 query90 2093181 186 181 query91 129 99 98 98 query92 57 47 47 47 query93 5843518 504 504 query94 1170186 185 185 query95 396 306 305 305 query96 606 266 259 259 query97 3143296829622962 query98 242 215 222 215 query99 1218874 859 859 Total cold run time: 278797 ms Total hot run time: 186442 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2096326801 TPC-H: Total hot run time: 41325 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit f41f417864445dbb578e3030cf6dce6702c8a2e3, data reload: false -- Round 1 -- q1 17605 467642914291 q2 2008191 192 191 q3 10491 117612061176 q4 10201 705 765 705 q5 7483272027392720 q6 220 132 132 132 q7 1038617 613 613 q8 9235222721252125 q9 9827684067756775 q10 9325386739033867 q11 463 256 246 246 q12 496 231 235 231 q13 18205 318632163186 q14 263 218 225 218 q15 509 462 477 462 q16 522 420 395 395 q17 997 690 658 658 q18 8236777177877771 q19 2415158115481548 q20 657 318 323 318 q21 5341428134223422 q22 355 275 282 275 Total cold run time: 115892 ms Total hot run time: 41325 ms - Round 2, with runtime_filter_mode=off - q1 4553443644494436 q2 389 265 270 265 q3 3202293129772931 q4 1847158015711571 q5 5512551355185513 q6 217 120 121 120 q7 2369199920091999 q8 3247345033603360 q9 8768880388368803 q10 4025389539893895 q11 594 541 526 526 q12 850 639 641 639 q13 17137 315231623152 q14 318 265 249 249 q15 524 479 486 479 q16 463 416 411 411 q17 1759147814711471 q18 7812757474077407 q19 4458157915321532 q20 1992177917661766 q21 5136495948714871 q22 575 517 503 503 Total cold run time: 75747 ms Total hot run time: 55899 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2096226167 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1590486660
##
fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionEntry.java:
##
@@ -261,22 +278,62 @@ public long abortTransaction()
}
}
+private void beforeFinishTransaction() {
+if (isTransactionBegan) {
+List tableIds =
transactionState.getTableIdList().stream().distinct().collect(Collectors.toList());
+transactionState.setTableIdList(tableIds);
+subTransactionStates.sort((s1, s2) -> {
+if (s1.getSubTransactionType() == SubTransactionType.INSERT
Review Comment:
1. We provide that, each statement cannot read the modifications made by
other statements within the same transaction.
2. For delete command, there are 2 implementations, one is delete
condition, one is insert.
3. If the delete condition is committed after the insert, the delete will
work for the insert, for example:
```
begin;
insert into t select * from t1 where id = 1;
delete from t where id = 1;
commit;
```
4. So we make the delete condition commit before the insert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
morningman commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1589872449
##
fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionEntry.java:
##
@@ -261,22 +278,62 @@ public long abortTransaction()
}
}
+private void beforeFinishTransaction() {
+if (isTransactionBegan) {
+List tableIds =
transactionState.getTableIdList().stream().distinct().collect(Collectors.toList());
+transactionState.setTableIdList(tableIds);
+subTransactionStates.sort((s1, s2) -> {
+if (s1.getSubTransactionType() == SubTransactionType.INSERT
Review Comment:
Why using this order?
##
fe/fe-core/src/main/java/org/apache/doris/load/EtlStatus.java:
##
@@ -34,10 +35,13 @@
public class EtlStatus implements Writable {
public static final String DEFAULT_TRACKING_URL = FeConstants.null_string;
-
+@SerializedName(value = "state")
private TEtlState state;
+@SerializedName(value = "trackingUrl")
Review Comment:
use short name to reduce the edit log size, eg: `tu
Same comment for all other SerializedName
##
fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java:
##
@@ -243,6 +245,22 @@ public void commitTransaction(long dbId, List
tableList, long transaction
dbTransactionMgr.commitTransaction(tableList, transactionId,
tabletCommitInfos, txnCommitAttachment, false);
}
+/**
+ * @note callers should get all tables' write locks before call this api
+ */
+public void commitTransaction(long dbId, List tableList, long
transactionId,
Review Comment:
the timeout is unused
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2084696393 TPC-DS: Total hot run time: 187105 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 6720dd0dc0d7b4ec0608c3a7d31cba960b17c16f, data reload: false query1 928 370 341 341 query2 7036236922642264 query3 6653219 218 218 query4 23131 21156 21195 21156 query5 4134431 436 431 query6 260 176 176 176 query7 4592294 292 292 query8 243 194 196 194 query9 8707246024482448 query10 438 252 262 252 query11 14859 14183 14091 14091 query12 140 89 90 89 query13 1663372 389 372 query14 8711858480248024 query15 259 170 171 170 query16 8235268 270 268 query17 1914584 573 573 query18 2126287 285 285 query19 344 155 158 155 query20 95 88 91 88 query21 198 138 132 132 query22 4999483648284828 query23 33915 33295 33158 33158 query24 10837 289329212893 query25 651 387 379 379 query26 1404156 156 156 query27 2891323 349 323 query28 7632209320652065 query29 899 628 623 623 query30 297 156 155 155 query31 951 733 732 732 query32 90 54 60 54 query33 757 255 256 255 query34 1028488 481 481 query35 818 682 680 680 query36 1087935 926 926 query37 138 70 71 70 query38 3187301029772977 query39 1591154015431540 query40 233 131 132 131 query41 44 44 40 40 query42 104 100 102 100 query43 582 520 557 520 query44 1221731 749 731 query45 265 264 263 263 query46 1091747 733 733 query47 1951185418541854 query48 451 303 301 301 query49 1023398 399 398 query50 779 410 403 403 query51 6793679267736773 query52 101 98 91 91 query53 358 296 276 276 query54 292 245 240 240 query55 80 74 72 72 query56 245 216 226 216 query57 1219115411241124 query58 214 200 200 200 query59 3529325031503150 query60 267 240 243 240 query61 92 90 90 90 query62 662 446 442 442 query63 311 284 279 279 query64 8735721672037203 query65 3137308330553055 query66 1405339 350 339 query67 15822 14919 14936 14919 query68 9373561 610 561 query69 609 319 311 311 query70 1315114311161116 query71 521 274 271 271 query72 8414255723642364 query73 1601332 338 332 query74 6567614360046004 query75 4503264826122612 query76 5828110710501050 query77 642 269 269 269 query78 11022 10195 10197 10195 query79 12835 515 524 515 query80 2356432 434 432 query81 507 227 218 218 query82 452 94 93 93 query83 216 166 204 166 query84 270 85 84 84 query85 1107271 256 256 query86 343 305 314 305 query87 3319309330623062 query88 5259243024182418 query89 532 398 381 381 query90 2068186 188 186 query91 126 97 96 96 query92 59 47 46 46 query93 7432523 519 519 query94 1369187 183 183 query95 393 302 298 298 query96 607 270 266 266 query97 3119292329242923 query98 238 221 215 215 query99 1245885 861 861 Total cold run time: 313091 ms Total hot run time: 187105 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2084673900 TPC-H: Total hot run time: 40318 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 6720dd0dc0d7b4ec0608c3a7d31cba960b17c16f, data reload: false -- Round 1 -- q1 17870 447142994299 q2 2040201 202 201 q3 10590 119412481194 q4 10506 779 852 779 q5 7646274827012701 q6 224 136 138 136 q7 1051609 624 609 q8 9334215821012101 q9 9174679267326732 q10 9080379638053796 q11 460 251 241 241 q12 464 225 221 221 q13 17785 296329862963 q14 260 218 227 218 q15 505 468 472 468 q16 516 390 370 370 q17 994 681 681 681 q18 8022732573277325 q19 1849156015081508 q20 663 315 308 308 q21 5110319340393193 q22 349 274 280 274 Total cold run time: 114492 ms Total hot run time: 40318 ms - Round 2, with runtime_filter_mode=off - q1 4341419341874187 q2 371 279 267 267 q3 3039274927722749 q4 1874166916061606 q5 5280533353025302 q6 209 125 124 124 q7 2247186818551855 q8 3200332933443329 q9 8492845584738455 q10 3906372537413725 q11 586 476 478 476 q12 740 616 606 606 q13 17157 297330082973 q14 287 294 274 274 q15 516 481 482 481 q16 469 420 413 413 q17 1774147714391439 q18 7635735374227353 q19 1663154015321532 q20 1952179117591759 q21 5208474949154749 q22 596 516 507 507 Total cold run time: 71542 ms Total hot run time: 54161 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2084598177 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2084398990 TPC-DS: Total hot run time: 187497 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 85b9878701dd5e3c9c9b028a83c08b592bdcd41d, data reload: false query1 921 371 363 363 query2 6314243625042436 query3 6646207 208 207 query4 24368 21957 21860 21860 query5 3807430 426 426 query6 256 178 177 177 query7 4582298 291 291 query8 231 208 178 178 query9 8751247625082476 query10 416 254 255 254 query11 15269 14951 14819 14819 query12 123 104 85 85 query13 1652386 366 366 query14 8535777668086808 query15 245 176 167 167 query16 8141264 267 264 query17 1690575 557 557 query18 2124268 265 265 query19 264 157 143 143 query20 89 84 82 82 query21 195 127 125 125 query22 5059486148294829 query23 33824 33444 33437 33437 query24 10675 286029392860 query25 610 375 358 358 query26 1338148 150 148 query27 3048355 355 355 query28 7264207120642064 query29 857 610 586 586 query30 259 147 149 147 query31 950 710 730 710 query32 86 50 50 50 query33 732 247 250 247 query34 1043474 474 474 query35 808 665 673 665 query36 1050866 900 866 query37 130 66 62 62 query38 3146302730123012 query39 1600151715271517 query40 198 123 126 123 query41 40 40 38 38 query42 107 96 91 91 query43 574 556 527 527 query44 1164719 738 719 query45 263 238 255 238 query46 1086718 708 708 query47 1974189318521852 query48 387 290 289 289 query49 904 451 381 381 query50 765 387 378 378 query51 6769666367266663 query52 102 90 94 90 query53 345 275 275 275 query54 290 228 234 228 query55 77 72 69 69 query56 240 225 213 213 query57 1235114211531142 query58 221 195 195 195 query59 3356317331353135 query60 264 235 228 228 query61 90 87 89 87 query62 654 440 437 437 query63 308 276 277 276 query64 8647724972247224 query65 3113303830493038 query66 1368327 321 321 query67 15535 15366 15138 15138 query68 5237530 545 530 query69 479 304 314 304 query70 1169115211141114 query71 419 274 284 274 query72 7244257524072407 query73 709 326 330 326 query74 6501620261496149 query75 3432267626492649 query76 31821029936 936 query77 398 264 285 264 query78 10849 10214 10212 10212 query79 4647526 515 515 query80 2008429 425 425 query81 552 226 232 226 query82 851 114 98 98 query83 282 173 164 164 query84 265 89 87 87 query85 2149265 261 261 query86 510 309 294 294 query87 3317306031163060 query88 4740242024252420 query89 498 366 372 366 query90 2043184 186 184 query91 127 96 98 96 query92 65 48 46 46 query93 4941515 496 496 query94 1208186 182 182 query95 389 302 310 302 query96 604 280 268 268 query97 3155297529492949 query98 237 213 217 213 query99 1268841 887 841 Total cold run time: 290649 ms Total hot run time: 187497 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2084330754 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2081474550 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2081430730 TPC-DS: Total hot run time: 186860 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 4120bb12f027007362a97ed819609b5c9c7d09d4, data reload: false query1 917 370 339 339 query2 6481234522972297 query3 6656207 211 207 query4 23800 21174 21158 21158 query5 4156415 419 415 query6 257 174 172 172 query7 4595298 290 290 query8 249 203 193 193 query9 8470236023502350 query10 429 242 256 242 query11 14843 14159 14144 14144 query12 133 87 85 85 query13 1642352 358 352 query14 9773845484728454 query15 275 182 176 176 query16 8243269 271 269 query17 1899579 572 572 query18 2116277 283 277 query19 334 156 155 155 query20 98 92 85 85 query21 203 126 130 126 query22 4985484648334833 query23 34148 33378 33213 33213 query24 12054 299829822982 query25 685 395 384 384 query26 1750154 152 152 query27 2918317 317 317 query28 7319204720282028 query29 1021628 600 600 query30 289 148 151 148 query31 955 741 698 698 query32 96 55 52 52 query33 746 249 253 249 query34 1022489 478 478 query35 829 700 696 696 query36 1104927 927 927 query37 269 66 68 66 query38 3136300030143000 query39 1579156115131513 query40 279 128 127 127 query41 42 40 41 40 query42 107 96 96 96 query43 586 530 525 525 query44 1255723 743 723 query45 283 257 245 245 query46 1079729 705 705 query47 1903181818401818 query48 386 298 296 296 query49 1145396 402 396 query50 768 399 417 399 query51 6710668366256625 query52 101 92 91 91 query53 350 277 283 277 query54 304 238 235 235 query55 79 74 72 72 query56 238 229 222 222 query57 1211113511441135 query58 228 196 196 196 query59 3421325130343034 query60 271 245 234 234 query61 94 88 89 88 query62 640 457 445 445 query63 305 279 282 279 query64 9553710371257103 query65 3108304230453042 query66 1393337 332 332 query67 15600 14835 14807 14807 query68 6141549 557 549 query69 525 318 315 315 query70 1222115210691069 query71 460 279 272 272 query72 7812264424252425 query73 738 329 333 329 query74 6555612559985998 query75 3798266826462646 query76 4233103410461034 query77 620 272 264 264 query78 11075 10182 10113 10113 query79 7825520 512 512 query80 1465445 436 436 query81 523 230 232 230 query82 917 94 95 94 query83 203 162 175 162 query84 268 89 85 85 query85 1446274 282 274 query86 464 330 307 307 query87 3253307930983079 query88 5028242224282422 query89 509 388 366 366 query90 2034187 186 186 query91 130 97 98 97 query92 66 47 48 47 query93 6246526 508 508 query94 1124185 182 182 query95 389 300 304 300 query96 616 262 265 262 query97 3115295029592950 query98 243 215 216 215 query99 1267851 877 851 Total cold run time: 302223 ms Total hot run time: 186860 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2081427738 TPC-H: Total hot run time: 40448 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 4120bb12f027007362a97ed819609b5c9c7d09d4, data reload: false -- Round 1 -- q1 17616 451242834283 q2 2017192 187 187 q3 10692 134212391239 q4 10505 830 785 785 q5 7710272326882688 q6 217 129 132 129 q7 1047631 603 603 q8 9459211520852085 q9 92446732 q10 9006385238133813 q11 452 255 252 252 q12 396 221 230 221 q13 18819 296429732964 q14 279 231 234 231 q15 514 471 470 470 q16 496 383 377 377 q17 973 693 708 693 q18 8029737474747374 q19 1632154615341534 q20 648 312 317 312 q21 5122327440333274 q22 348 268 271 268 Total cold run time: 115221 ms Total hot run time: 40448 ms - Round 2, with runtime_filter_mode=off - q1 4280422742384227 q2 365 279 272 272 q3 2964276827552755 q4 1817157315791573 q5 5330531152895289 q6 206 120 122 120 q7 2243183218271827 q8 3201333033413330 q9 8544853585548535 q10 3948370136943694 q11 603 495 479 479 q12 786 576 574 574 q13 17288 295129782951 q14 305 283 267 267 q15 516 473 482 473 q16 470 421 417 417 q17 1766151014491449 q18 7459752173077307 q19 1634156915301530 q20 1940178317601760 q21 4997482547884788 q22 579 489 499 489 Total cold run time: 71241 ms Total hot run time: 54106 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2081422968 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2079313582 run cloud_p1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2079313406 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2079312319 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2079136010 TPC-DS: Total hot run time: 185285 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit f40fd3f3aad028c1532478904745fbb614fb5fcb, data reload: false query1 909 367 335 335 query2 6489229724252297 query3 6663209 211 209 query4 23437 21417 21369 21369 query5 4128409 416 409 query6 262 185 186 185 query7 4593291 282 282 query8 244 191 194 191 query9 8404231922992299 query10 424 240 249 240 query11 14686 14126 14047 14047 query12 131 87 83 83 query13 1635354 361 354 query14 10234 754769416941 query15 211 173 172 172 query16 7810256 254 254 query17 1874560 552 552 query18 1953287 277 277 query19 212 151 151 151 query20 96 84 86 84 query21 201 132 122 122 query22 5003478548144785 query23 33625 33023 33165 33023 query24 11947 298129902981 query25 680 366 371 366 query26 1845151 144 144 query27 3046309 303 303 query28 7666200219981998 query29 1131605 591 591 query30 310 147 147 147 query31 972 741 732 732 query32 86 52 52 52 query33 735 245 244 244 query34 1074490 463 463 query35 805 688 676 676 query36 1051928 852 852 query37 272 66 68 66 query38 3195297330252973 query39 1596153415391534 query40 306 124 122 122 query41 42 38 38 38 query42 101 93 99 93 query43 562 504 532 504 query44 1256720 741 720 query45 299 280 266 266 query46 1071710 722 710 query47 1918184618221822 query48 366 291 303 291 query49 1191415 397 397 query50 762 374 377 374 query51 6907673867846738 query52 100 93 90 90 query53 347 279 277 277 query54 316 230 232 230 query55 84 70 73 70 query56 239 220 214 214 query57 1231112511071107 query58 255 200 218 200 query59 3350299230472992 query60 263 234 258 234 query61 93 92 89 89 query62 642 433 455 433 query63 301 274 271 271 query64 9495709270897089 query65 3114303830533038 query66 1396375 340 340 query67 15320 15232 14985 14985 query68 5260534 523 523 query69 476 298 312 298 query70 1149112811331128 query71 410 270 270 270 query72 7266259524582458 query73 696 316 324 316 query74 6622624962176217 query75 3438264026862640 query76 3416985 1007985 query77 391 272 263 263 query78 11001 10310 10344 10310 query79 3765520 517 517 query80 1827431 429 429 query81 532 220 222 220 query82 811 97 90 90 query83 270 168 165 165 query84 264 85 83 83 query85 2152266 261 261 query86 525 299 292 292 query87 3249305330993053 query88 4510232223192319 query89 501 383 372 372 query90 2069182 182 182 query91 130 107 105 105 query92 63 49 49 49 query93 4932506 491 491 query94 1255187 183 183 query95 388 300 303 300 query96 608 259 262 259 query97 3150291929472919 query98 240 223 222 222 query99 1230895 849 849 Total cold run time: 293558 ms Total hot run time: 185285 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2079118399 TPC-H: Total hot run time: 40160 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit f40fd3f3aad028c1532478904745fbb614fb5fcb, data reload: false -- Round 1 -- q1 17607 438943134313 q2 2011189 186 186 q3 10477 121512151215 q4 10147 784 845 784 q5 7488266826832668 q6 221 137 130 130 q7 996 626 586 586 q8 9213210720512051 q9 9535662266466622 q10 8846375536983698 q11 494 254 231 231 q12 444 219 216 216 q13 17774 295829902958 q14 273 236 231 231 q15 518 477 477 477 q16 505 384 378 378 q17 966 673 716 673 q18 8083750075407500 q19 1657150315171503 q20 651 306 303 303 q21 4971316041093160 q22 348 278 277 277 Total cold run time: 113225 ms Total hot run time: 40160 ms - Round 2, with runtime_filter_mode=off - q1 4347431142334233 q2 381 270 267 267 q3 3047282427782778 q4 1870154215541542 q5 5315533053285328 q6 211 124 124 124 q7 2240190118871887 q8 3216339733483348 q9 8622855285198519 q10 3898368036983680 q11 588 487 493 487 q12 769 570 608 570 q13 16039 297829322932 q14 308 292 276 276 q15 514 464 464 464 q16 485 408 419 408 q17 1762147914741474 q18 7603752175337521 q19 1643157315091509 q20 1966176417651764 q21 4928473046824682 q22 570 487 486 486 Total cold run time: 70322 ms Total hot run time: 54279 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2079066336 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2079050394 TPC-H: Total hot run time: 40112 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 26d2812be5e617d4d28388815ac35a7a99c93064, data reload: false -- Round 1 -- q1 17617 458642134213 q2 2013183 195 183 q3 10495 122211861186 q4 10547 828 786 786 q5 7739277626832683 q6 218 133 132 132 q7 1049618 606 606 q8 9520214620942094 q9 9028674466316631 q10 8702374837173717 q11 463 239 230 230 q12 453 215 209 209 q13 17769 294029262926 q14 273 230 230 230 q15 517 471 479 471 q16 492 379 368 368 q17 950 654 765 654 q18 8060741575067415 q19 4184151814911491 q20 648 305 307 305 q21 4934330639303306 q22 339 276 281 276 Total cold run time: 116010 ms Total hot run time: 40112 ms - Round 2, with runtime_filter_mode=off - q1 4377423142014201 q2 382 264 278 264 q3 3055277427642764 q4 1852157016091570 q5 5286532053775320 q6 207 124 123 123 q7 2251188118961881 q8 3206336733703367 q9 8605857085828570 q10 3883366336623662 q11 614 472 497 472 q12 773 620 594 594 q13 17372 291829372918 q14 308 287 276 276 q15 515 477 473 473 q16 471 411 421 411 q17 1762147614851476 q18 7727757974767476 q19 1632149815191498 q20 1936174317491743 q21 7945485648964856 q22 586 496 480 480 Total cold run time: 74745 ms Total hot run time: 54395 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2079020689 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2078999174 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
morningman commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1580576119
##
fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionState.java:
##
@@ -298,6 +299,16 @@ public SchemaInfo(OlapTable olapTable) {
// table id -> schema info
private Map txnSchemas = new HashMap<>();
+@Getter
+@Setter
+private List subTransactionStates;
+@Getter
+@SerializedName(value = "subTxnIdToTableCommitInfo")
Review Comment:
Use short name, such as:
```suggestion
@SerializedName(value = "sti")
```
##
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadJobFinalOperation.java:
##
@@ -38,13 +39,20 @@
* It is used to edit the job final state.
*/
public class LoadJobFinalOperation extends TxnCommitAttachment implements
Writable {
+@SerializedName(value = "id")
private long id;
+@SerializedName(value = "loadingStatus")
private EtlStatus loadingStatus = new EtlStatus();
+@SerializedName(value = "progress")
private int progress;
+@SerializedName(value = "loadStartTimestamp")
Review Comment:
use short name. like `lst`.
Same comment for all other fields
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
dataroaring commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1579500655
##
regression-test/suites/insert_p0/txn_insert.groovy:
##
@@ -328,19 +480,44 @@ suite("txn_insert") {
where txn_insert_dt4.id = txn_insert_dt2.id;
"""
sql """
-delete from txn_insert_dt2 where id = 1 or id = 5;
+delete from txn_insert_dt2 where id = 1;
"""
sql """
-delete from txn_insert_dt5 partition(p_2102) where id = 1
or id = 5;
+delete from txn_insert_dt2 where id = 5;
+"""
Review Comment:
what if
begin
insert into t1;
delete from t1;
insert into t2 select * from t1;
commit;
result of first insert is visible to delete but not visible to the second
insert.
##
regression-test/suites/insert_p0/txn_insert.groovy:
##
@@ -328,19 +480,44 @@ suite("txn_insert") {
where txn_insert_dt4.id = txn_insert_dt2.id;
"""
sql """
-delete from txn_insert_dt2 where id = 1 or id = 5;
+delete from txn_insert_dt2 where id = 1;
"""
sql """
-delete from txn_insert_dt5 partition(p_2102) where id = 1
or id = 5;
+delete from txn_insert_dt2 where id = 5;
+"""
Review Comment:
> > Thread1 t2: insert into table1 Thread2 t1: insert into table2 select
table1, t3: insert into table2 select table1.
> > Is result of t2 visible to insert at t3?
>
> yes
For clarity,
Thread1 t2: begin insert into table1; commit
thread2 t1:begin insert into table 2 select table2; t3: insert into table2
select table1; commit
If yes, what's our isolation level? ReadCommitted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2076875183 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2076544832 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2076488459 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1577699808
##
fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionState.java:
##
@@ -695,41 +730,21 @@ public void prolongPublishTimeout() {
@Override
public void write(DataOutput out) throws IOException {
-out.writeLong(transactionId);
-Text.writeString(out, label);
-out.writeLong(dbId);
-out.writeInt(idToTableCommitInfos.size());
-for (TableCommitInfo info : idToTableCommitInfos.values()) {
-info.write(out);
-}
-out.writeInt(txnCoordinator.sourceType.value());
-Text.writeString(out, txnCoordinator.ip);
-out.writeInt(transactionStatus.value());
-out.writeInt(sourceType.value());
-out.writeLong(prepareTime);
-out.writeLong(preCommitTime);
-out.writeLong(commitTime);
-out.writeLong(finishTime);
-Text.writeString(out, reason);
-out.writeInt(errorReplicas.size());
-for (long errorReplciaId : errorReplicas) {
-out.writeLong(errorReplciaId);
-}
-
-if (txnCommitAttachment == null) {
-out.writeBoolean(false);
+Text.writeString(out, GsonUtils.GSON.toJson(this));
+}
+
+public static TransactionState read(DataInput in) throws IOException {
+if (Env.getCurrentEnvJournalVersion() < FeMetaVersion.VERSION_132) {
+TransactionState transactionState = new TransactionState();
+transactionState.readFields(in);
+return transactionState;
} else {
-out.writeBoolean(true);
-txnCommitAttachment.write(out);
-}
-out.writeLong(callbackId);
-out.writeLong(timeoutMs);
-out.writeInt(tableIdList.size());
-for (Long aLong : tableIdList) {
-out.writeLong(aLong);
+String json = Text.readString(in);
Review Comment:
In this case, the fe meta version is 131 and use the original `readFields`
method to read txn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
yujun777 commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1577621060
##
fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionState.java:
##
@@ -695,41 +730,21 @@ public void prolongPublishTimeout() {
@Override
public void write(DataOutput out) throws IOException {
-out.writeLong(transactionId);
-Text.writeString(out, label);
-out.writeLong(dbId);
-out.writeInt(idToTableCommitInfos.size());
-for (TableCommitInfo info : idToTableCommitInfos.values()) {
-info.write(out);
-}
-out.writeInt(txnCoordinator.sourceType.value());
-Text.writeString(out, txnCoordinator.ip);
-out.writeInt(transactionStatus.value());
-out.writeInt(sourceType.value());
-out.writeLong(prepareTime);
-out.writeLong(preCommitTime);
-out.writeLong(commitTime);
-out.writeLong(finishTime);
-Text.writeString(out, reason);
-out.writeInt(errorReplicas.size());
-for (long errorReplciaId : errorReplicas) {
-out.writeLong(errorReplciaId);
-}
-
-if (txnCommitAttachment == null) {
-out.writeBoolean(false);
+Text.writeString(out, GsonUtils.GSON.toJson(this));
+}
+
+public static TransactionState read(DataInput in) throws IOException {
+if (Env.getCurrentEnvJournalVersion() < FeMetaVersion.VERSION_132) {
+TransactionState transactionState = new TransactionState();
+transactionState.readFields(in);
+return transactionState;
} else {
-out.writeBoolean(true);
-txnCommitAttachment.write(out);
-}
-out.writeLong(callbackId);
-out.writeLong(timeoutMs);
-out.writeInt(tableIdList.size());
-for (Long aLong : tableIdList) {
-out.writeLong(aLong);
+String json = Text.readString(in);
Review Comment:
suppose fe has some history txns, upgrade fe. at the first time fe upgrade
and restart, can it read the history txns with json format ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2073920318 ClickBench: Total hot run time: 31.57 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit f0fea970c97d9a61fb244ac146a4c4ae0d53dd03, data reload: false query1 0.030.030.04 query2 0.090.040.03 query3 0.230.050.05 query4 1.690.080.08 query5 0.510.540.51 query6 1.230.880.82 query7 0.020.010.01 query8 0.040.040.04 query9 0.510.440.46 query10 0.500.510.50 query11 0.140.100.10 query12 0.140.110.11 query13 0.630.640.64 query14 0.880.991.06 query15 0.840.840.85 query16 0.380.380.37 query17 1.041.031.03 query18 0.250.200.26 query19 1.911.821.78 query20 0.010.020.01 query21 15.42 0.650.65 query22 4.266.151.81 query23 18.34 1.321.26 query24 2.020.240.26 query25 0.150.100.10 query26 0.270.170.17 query27 0.080.080.09 query28 13.26 1.031.01 query29 12.63 3.473.48 query30 0.280.080.06 query31 3.180.400.39 query32 3.600.490.50 query33 2.863.042.77 query34 17.18 4.694.62 query35 4.634.704.73 query36 0.660.470.48 query37 0.210.190.17 query38 0.200.190.19 query39 0.060.050.04 query40 0.180.150.15 query41 0.120.070.05 query42 0.070.070.06 query43 0.050.040.05 Total cold run time: 110.78 s Total hot run time: 31.57 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2073876742 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2071341468 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2071279767 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2069252931 ClickBench: Total hot run time: 30.76 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 731a923c8d3271792984d56f3d8f5b9ecb790188, data reload: false query1 0.040.030.03 query2 0.080.040.04 query3 0.230.050.05 query4 1.680.070.07 query5 0.470.470.48 query6 1.490.730.72 query7 0.020.010.01 query8 0.060.040.04 query9 0.560.490.49 query10 0.540.560.54 query11 0.160.120.12 query12 0.150.120.12 query13 0.600.590.58 query14 0.740.790.78 query15 0.810.810.79 query16 0.360.370.37 query17 0.981.021.04 query18 0.210.260.23 query19 1.891.741.77 query20 0.010.010.02 query21 15.40 0.650.66 query22 4.776.362.29 query23 18.33 1.301.25 query24 1.890.250.21 query25 0.150.080.08 query26 0.270.170.17 query27 0.090.090.08 query28 13.28 1.000.98 query29 12.61 3.293.26 query30 0.260.070.07 query31 2.840.370.38 query32 3.310.460.47 query33 2.842.772.83 query34 16.94 4.424.49 query35 4.464.454.44 query36 0.650.450.46 query37 0.210.150.15 query38 0.160.150.14 query39 0.050.040.04 query40 0.180.140.14 query41 0.090.050.05 query42 0.050.050.04 query43 0.040.030.04 Total cold run time: 109.95 s Total hot run time: 30.76 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2069241870 TPC-DS: Total hot run time: 185151 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 731a923c8d3271792984d56f3d8f5b9ecb790188, data reload: false query1 3016378 365 365 query2 7829243624352435 query3 8296218 219 218 query4 24698 21260 21420 21260 query5 4141402 412 402 query6 275 171 177 171 query7 4579285 281 281 query8 242 198 199 198 query9 8532226022592259 query10 555 275 249 249 query11 14663 14238 14216 14216 query12 137 92 89 89 query13 1651374 367 367 query14 9565752278057522 query15 287 200 192 192 query16 8214273 262 262 query17 2127576 553 553 query18 2103276 268 268 query19 330 145 152 145 query20 89 91 84 84 query21 195 128 123 123 query22 5012483848464838 query23 33881 33528 34164 33528 query24 8621299430122994 query25 600 386 394 386 query26 693 161 150 150 query27 2263375 394 375 query28 5570204219891989 query29 857 627 650 627 query30 225 185 178 178 query31 976 777 776 776 query32 88 52 51 51 query33 610 242 251 242 query34 962 496 503 496 query35 789 728 731 728 query36 1069950 938 938 query37 112 69 66 66 query38 3509340933443344 query39 1080101410371014 query40 170 126 126 126 query41 46 46 45 45 query42 107 99 94 94 query43 584 564 569 564 query44 1106771 741 741 query45 291 270 283 270 query46 749 757 749 query47 2032196219511951 query48 370 301 309 301 query49 806 405 394 394 query50 796 408 404 404 query51 6850663665966596 query52 100 90 89 89 query53 348 281 279 279 query54 266 224 231 224 query55 80 70 69 69 query56 237 229 232 229 query57 1183110311121103 query58 214 194 195 194 query59 3376325531493149 query60 248 239 254 239 query61 106 107 107 107 query62 588 441 448 441 query63 305 280 282 280 query64 4786390139863901 query65 3050303630243024 query66 754 328 336 328 query67 15423 14856 14932 14856 query68 5197538 538 538 query69 506 302 303 302 query70 1215112112111121 query71 1401126512691265 query72 6308266524252425 query73 704 323 321 321 query74 6756633864486338 query75 3367265726062606 query76 3340922 919 919 query77 576 262 265 262 query78 10832 10343 10272 10272 query79 4266524 534 524 query80 1820415 418 415 query81 536 250 243 243 query82 153096 97 96 query83 309 177 172 172 query84 259 88 79 79 query85 1413262 258 258 query86 483 294 279 279 query87 3459323932473239 query88 4873240024202400 query89 468 366 368 366 query90 1971178 175 175 query91 121 98 96 96 query92 58 46 45 45 query93 5468493 496 493 query94 1128185 176 176 query95 381 298 305 298 query96 601 271 267 267 query97 3173296629622962 query98 229 214 213 213 query99 1182864 860 860 Total cold run time: 287704 ms Total hot run time: 185151 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2069218758 TPC-H: Total hot run time: 38404 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 731a923c8d3271792984d56f3d8f5b9ecb790188, data reload: false -- Round 1 -- q1 17617 430042254225 q2 2014197 185 185 q3 10447 114011331133 q4 10192 757 722 722 q5 7508270126152615 q6 217 133 133 133 q7 1006605 598 598 q8 9229204120362036 q9 7336656165306530 q10 8610349535183495 q11 463 227 227 227 q12 520 222 219 219 q13 17771 289429262894 q14 269 230 234 230 q15 535 482 483 482 q16 523 371 387 371 q17 958 756 680 680 q18 7288674466426642 q19 5846155515121512 q20 652 296 297 296 q21 3632288328862883 q22 358 296 305 296 Total cold run time: 112991 ms Total hot run time: 38404 ms - Round 2, with runtime_filter_mode=off - q1 4314423641874187 q2 359 273 260 260 q3 2947271027892710 q4 1886157815971578 q5 5382536652995299 q6 210 121 124 121 q7 2245186618961866 q8 3174335633443344 q9 8679865486658654 q10 4158394740463947 q11 619 520 503 503 q12 824 614 646 614 q13 17315 314931943149 q14 334 281 273 273 q15 533 501 496 496 q16 490 445 470 445 q17 1850156815121512 q18 8189807279237923 q19 1715157215611561 q20 2054189318701870 q21 8987505950825059 q22 609 497 509 497 Total cold run time: 76873 ms Total hot run time: 55868 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2069151796 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2068906132 ClickBench: Total hot run time: 29.67 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 32c3dca60d2e90c86a6f54dfc1fa4bb68517cd2d, data reload: false query1 0.040.040.03 query2 0.080.040.04 query3 0.230.050.06 query4 1.670.070.07 query5 0.500.500.49 query6 1.460.730.72 query7 0.020.010.02 query8 0.040.040.04 query9 0.550.480.49 query10 0.540.560.54 query11 0.160.120.11 query12 0.140.110.11 query13 0.600.600.58 query14 0.750.750.77 query15 0.820.800.80 query16 0.340.360.34 query17 0.980.940.95 query18 0.210.250.22 query19 1.731.661.70 query20 0.010.010.01 query21 15.44 0.660.65 query22 4.397.741.49 query23 18.27 1.381.27 query24 2.060.200.21 query25 0.140.080.08 query26 0.260.160.16 query27 0.070.070.07 query28 13.34 0.990.99 query29 12.58 3.223.26 query30 0.260.060.06 query31 2.850.360.38 query32 3.300.460.46 query33 2.792.882.78 query34 17.35 4.404.43 query35 4.464.504.46 query36 0.640.460.46 query37 0.180.160.15 query38 0.150.140.14 query39 0.050.040.04 query40 0.200.150.13 query41 0.090.050.04 query42 0.050.050.04 query43 0.040.030.04 Total cold run time: 109.83 s Total hot run time: 29.67 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2068894649 TPC-DS: Total hot run time: 182913 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 32c3dca60d2e90c86a6f54dfc1fa4bb68517cd2d, data reload: false query1 911 372 362 362 query2 6517257522952295 query3 6672205 204 204 query4 22852 21547 21427 21427 query5 4202410 398 398 query6 265 175 186 175 query7 4577293 284 284 query8 234 184 181 181 query9 8496234423382338 query10 582 263 263 263 query11 14738 14432 14157 14157 query12 136 95 87 87 query13 1632356 354 354 query14 10265 700877807008 query15 258 185 184 184 query16 8171265 253 253 query17 1924568 547 547 query18 2103269 259 259 query19 201 143 151 143 query20 92 85 82 82 query21 202 132 123 123 query22 5003483248334832 query23 33803 33280 33008 33008 query24 11537 292728182818 query25 646 363 393 363 query26 1781149 148 148 query27 3003304 316 304 query28 7474198319991983 query29 1024597 587 587 query30 310 174 168 168 query31 979 727 737 727 query32 98 52 52 52 query33 745 245 235 235 query34 1086482 478 478 query35 826 706 677 677 query36 1058878 912 878 query37 197 71 67 67 query38 3375320631873187 query39 1567154515421542 query40 280 120 126 120 query41 46 44 43 43 query42 103 93 96 93 query43 546 543 540 540 query44 1219732 732 732 query45 277 269 235 235 query46 1072707 757 707 query47 1922184618461846 query48 361 296 288 288 query49 1175393 379 379 query50 757 377 367 367 query51 6801672267096709 query52 98 91 89 89 query53 342 288 281 281 query54 325 235 252 235 query55 79 73 69 69 query56 253 231 229 229 query57 1199111211391112 query58 234 196 202 196 query59 3236311031803110 query60 261 265 230 230 query61 88 88 102 88 query62 642 448 430 430 query63 300 276 275 275 query64 6299384729142914 query65 3373305730563056 query66 1381346 323 323 query67 15379 15167 15201 15167 query68 5150516 528 516 query69 479 304 297 297 query70 1251113811041104 query71 1385126012691260 query72 6410262824322432 query73 707 314 312 312 query74 6784636463586358 query75 3499266626192619 query76 3272994 973 973 query77 404 261 256 256 query78 11014 10228 10183 10183 query79 2814529 509 509 query80 1943430 444 430 query81 572 238 242 238 query82 767 99 97 97 query83 271 165 165 165 query84 267 82 85 82 query85 2096265 255 255 query86 523 311 289 289 query87 3521331032583258 query88 4260230823012301 query89 463 375 381 375 query90 2015179 183 179 query91 127 95 96 95 query92 69 45 47 45 query93 3931520 496 496 query94 1239178 175 175 query95 387 306 293 293 query96 588 257 262 257 query97 3166294229422942 query98 238 219 222 219 query99 1205873 862 862 Total cold run time: 288407 ms Total hot run time: 182913 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2068871176 TPC-H: Total hot run time: 39205 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 32c3dca60d2e90c86a6f54dfc1fa4bb68517cd2d, data reload: false -- Round 1 -- q1 17613 433342304230 q2 2024193 185 185 q3 10533 117111971171 q4 10337 779 842 779 q5 7593272526712671 q6 227 139 139 139 q7 1039632 607 607 q8 9327213220802080 q9 8094677867036703 q10 8886372936613661 q11 465 235 253 235 q12 624 234 238 234 q13 18574 298429842984 q14 274 223 235 223 q15 524 495 489 489 q16 520 386 376 376 q17 970 714 738 714 q18 7381680667636763 q19 1593154115331533 q20 646 321 302 302 q21 3560282330472823 q22 371 316 303 303 Total cold run time: 75 ms Total hot run time: 39205 ms - Round 2, with runtime_filter_mode=off - q1 4193419341904190 q2 373 281 272 272 q3 3021278527032703 q4 1869158916051589 q5 5346535153255325 q6 212 125 124 124 q7 2262191118641864 q8 3192337633753375 q9 8606855086108550 q10 3926373537433735 q11 597 487 490 487 q12 768 596 635 596 q13 15917 294029612940 q14 304 260 297 260 q15 508 471 469 469 q16 487 416 409 409 q17 1790148514571457 q18 7499749874067406 q19 1627156015191519 q20 1936175117291729 q21 5034479348344793 q22 520 444 438 438 Total cold run time: 69987 ms Total hot run time: 54230 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2068821993 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2065705935 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1571711166
##
regression-test/suites/insert_p0/txn_insert.groovy:
##
@@ -328,19 +480,44 @@ suite("txn_insert") {
where txn_insert_dt4.id = txn_insert_dt2.id;
"""
sql """
-delete from txn_insert_dt2 where id = 1 or id = 5;
+delete from txn_insert_dt2 where id = 1;
"""
sql """
-delete from txn_insert_dt5 partition(p_2102) where id = 1
or id = 5;
+delete from txn_insert_dt2 where id = 5;
+"""
Review Comment:
> If we do insert and delete on the same table, then there is a problem
that, if the delete is a predicate and implemented via push handler, then the
delete take effect on the insert result. while if the delete is in `delete
from` format then the delete does not take effect on the insert result.
>
> It is ambiguous for users.
For `delete from t1` and `insert t1` case, the delete does not work the next
insert, see case 14.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2065621068 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2063649274 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1570519758
##
regression-test/suites/insert_p0/txn_insert.groovy:
##
@@ -328,19 +480,44 @@ suite("txn_insert") {
where txn_insert_dt4.id = txn_insert_dt2.id;
"""
sql """
-delete from txn_insert_dt2 where id = 1 or id = 5;
+delete from txn_insert_dt2 where id = 1;
"""
sql """
-delete from txn_insert_dt5 partition(p_2102) where id = 1
or id = 5;
+delete from txn_insert_dt2 where id = 5;
+"""
Review Comment:
> Thread1 t2: insert into table1 Thread2 t1: insert into table2 select
table1, t3: insert into table2 select table1.
>
> Is result of t2 visible to insert at t3?
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2063583099 > What if we change session var during a transaction, like timeout. When begin txn, calculate a timeout timestamp. So the changed timeout does not work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2063564160 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2063562373 > Add a parallel case, in which each instance writes to several same tables continuously. done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2063561008 > A case ensure that group commit is disabled with begin commit. done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
dataroaring commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1566621595
##
fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java:
##
@@ -286,6 +303,33 @@ public boolean commitAndPublishTransaction(DatabaseIf db,
List tableList,
return dbTransactionMgr.waitForTransactionFinished(db, transactionId,
publishTimeoutMillis);
}
+public boolean commitAndPublishTransaction(DatabaseIf db, long
transactionId,
+List subTransactionStates, long
timeoutMillis) throws UserException {
+StopWatch stopWatch = new StopWatch();
+stopWatch.start();
+List tableList =
subTransactionStates.stream().map(SubTransactionState::getTable).distinct()
Review Comment:
Are table in tableList ordered? otherwise, deadlock may happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
dataroaring commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1566618862
##
regression-test/suites/insert_p0/txn_insert.groovy:
##
@@ -214,7 +228,145 @@ suite("txn_insert") {
order_qt_select24 """select * from ${table}_2"""
}
-// 7. update stmt
+// 7. insert into select to same table
+if (use_nereids_planner) {
+sql """ begin; """
+sql """ insert into ${table}_0 select * from ${table}_1; """
+sql """ insert into ${table}_0 select * from ${table}_2; """
+sql """ insert into ${table}_0 select * from ${table}_1; """
+sql """ insert into ${table}_0 select * from ${table}_2; """
+sql """ commit; """
+sql "sync"
+order_qt_select25 """select * from ${table}_0"""
+
+sql """ insert into ${table}_0 select * from ${table}_1; """
+sql "sync"
+order_qt_select26 """select * from ${table}_0"""
+
+sql """ insert into ${table}_0 values(1001, 2.2, "abc", [], []) """
+sql "sync"
+order_qt_select27 """select * from ${table}_0"""
+
+// select from observer fe
+def observer_fe_url = get_observer_fe_url()
+if (observer_fe_url != null) {
+logger.info("observer url: $observer_fe_url")
+connect(user = context.config.jdbcUser, password =
context.config.jdbcPassword, url = observer_fe_url) {
+result = sql """ select count() from
regression_test_insert_p0.${table}_0 """
+logger.info("select from observer result: $result")
+assertEquals(79, result[0][0])
+}
+}
+}
+
+// 8. insert into tables in different database
+if (use_nereids_planner) {
+def db2 = "regression_test_insert_p0_1"
+sql """ create database if not exists $db2 """
+
+try {
+sql """ create table ${db2}.${table} like ${table} """
+sql """ begin; """
+sql """ insert into ${table} select * from ${table}_0; """
+test {
+sql """ insert into $db2.${table} select * from
${table}_0; """
+exception """Transaction insert must be in the same
database, expect db_id"""
+}
+} finally {
+sql """rollback"""
Review Comment:
Please add a case that does not finally rollback. We also should add a case
for timeout without explicit rollback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
dataroaring commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1566617348
##
fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionEntry.java:
##
@@ -163,39 +162,44 @@ public void setpLoadId(Types.PUniqueId pLoadId) {
this.pLoadId = pLoadId;
}
-// Used for insert into select
-public void beginTransaction(DatabaseIf database, TableIf table)
-throws DdlException, BeginTransactionException,
MetaNotFoundException, AnalysisException,
-QuotaExceedException {
+// Used for insert into select, return the sub_txn_id for this insert
+public long beginTransaction(TableIf table) throws UserException {
if (isTxnBegin()) {
Review Comment:
Does isTxnBegin indicate there is already a insert into values? If so, maybe
we should change name of isTxnBegin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
dataroaring commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1566610654
##
fe/fe-core/src/main/java/org/apache/doris/catalog/TabletInvertedIndex.java:
##
@@ -362,6 +353,25 @@ && isLocal(tabletMeta.getStorageMedium())) {
tabletRecoveryMap.size(), (end - start));
}
+// the transactionId may be sub transaction id or transaction id
+private TPartitionVersionInfo
generatePartitionVersionInfoWhenReport(TransactionState transactionState,
+long transactionId, TabletMeta tabletMeta, long partitionId) {
+TableCommitInfo tableCommitInfo;
+if (transactionState.getSubTransactionStates() != null) {
Review Comment:
should be == null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
dataroaring commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1566605801
##
regression-test/suites/insert_p0/txn_insert.groovy:
##
@@ -328,19 +480,44 @@ suite("txn_insert") {
where txn_insert_dt4.id = txn_insert_dt2.id;
"""
sql """
-delete from txn_insert_dt2 where id = 1 or id = 5;
+delete from txn_insert_dt2 where id = 1;
"""
sql """
-delete from txn_insert_dt5 partition(p_2102) where id = 1
or id = 5;
+delete from txn_insert_dt2 where id = 5;
+"""
Review Comment:
If we do insert and delete on the same table, then there is a problem that,
if the delete is a predicate and implemented via push handler, then the delete
take effect on the insert result. while if the delete is in `delete from`
format then the delete does not take effect on the insert result.
It is ambiguous for users.
##
regression-test/suites/insert_p0/txn_insert.groovy:
##
@@ -328,19 +480,44 @@ suite("txn_insert") {
where txn_insert_dt4.id = txn_insert_dt2.id;
"""
sql """
-delete from txn_insert_dt2 where id = 1 or id = 5;
+delete from txn_insert_dt2 where id = 1;
"""
sql """
-delete from txn_insert_dt5 partition(p_2102) where id = 1
or id = 5;
+delete from txn_insert_dt2 where id = 5;
+"""
Review Comment:
BTW, delete sign also works as delete via push handler.
##
regression-test/suites/insert_p0/txn_insert.groovy:
##
@@ -328,19 +480,44 @@ suite("txn_insert") {
where txn_insert_dt4.id = txn_insert_dt2.id;
"""
sql """
-delete from txn_insert_dt2 where id = 1 or id = 5;
+delete from txn_insert_dt2 where id = 1;
"""
sql """
-delete from txn_insert_dt5 partition(p_2102) where id = 1
or id = 5;
+delete from txn_insert_dt2 where id = 5;
+"""
Review Comment:
Thread1 t2: insert into table1
Thread2 t1: insert into table2 select table1, t3: insert into table2 select
table1.
Is result of t2 visible to insert at t3?
##
regression-test/suites/insert_p0/txn_insert.groovy:
##
@@ -328,19 +480,44 @@ suite("txn_insert") {
where txn_insert_dt4.id = txn_insert_dt2.id;
"""
sql """
-delete from txn_insert_dt2 where id = 1 or id = 5;
+delete from txn_insert_dt2 where id = 1;
"""
sql """
-delete from txn_insert_dt5 partition(p_2102) where id = 1
or id = 5;
+delete from txn_insert_dt2 where id = 5;
+"""
Review Comment:
If a dba changes the replication num of a table between two insert to a
table, then what happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2056372370 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit d8905e1aeb1903b233eea10e1325793e3fabc2c1 with default session variables Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s Stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s Insert into select: 13.4 seconds inserted 1000 Rows, about 746K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2056365933 ClickBench: Total hot run time: 30.59 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit d8905e1aeb1903b233eea10e1325793e3fabc2c1, data reload: false query1 0.050.030.03 query2 0.080.040.04 query3 0.230.050.05 query4 1.670.070.07 query5 0.490.510.51 query6 1.460.720.72 query7 0.020.010.01 query8 0.040.040.04 query9 0.540.490.50 query10 0.550.570.56 query11 0.140.110.11 query12 0.140.110.13 query13 0.620.590.59 query14 0.780.750.78 query15 0.830.800.82 query16 0.360.360.36 query17 1.031.040.99 query18 0.220.230.25 query19 1.791.781.78 query20 0.020.010.00 query21 15.41 0.660.65 query22 3.787.652.09 query23 18.32 1.451.21 query24 1.960.200.20 query25 0.130.070.08 query26 0.260.160.16 query27 0.080.080.08 query28 13.42 1.000.98 query29 12.58 3.333.25 query30 0.260.070.06 query31 2.860.380.37 query32 3.270.480.46 query33 2.812.892.84 query34 17.18 4.474.43 query35 4.524.464.52 query36 0.650.460.46 query37 0.180.150.16 query38 0.150.140.15 query39 0.050.030.04 query40 0.170.150.14 query41 0.090.050.04 query42 0.050.050.04 query43 0.040.040.04 Total cold run time: 109.28 s Total hot run time: 30.59 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2056351425 TPC-DS: Total hot run time: 183439 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit d8905e1aeb1903b233eea10e1325793e3fabc2c1, data reload: false query1 891 1129347 347 query2 7491246523042304 query3 6655204 206 204 query4 37169 21514 21352 21352 query5 4154392 387 387 query6 232 190 187 187 query7 4056295 290 290 query8 215 167 169 167 query9 5726228322972283 query10 358 231 241 231 query11 14537 14304 14262 14262 query12 136 87 84 84 query13 983 356 345 345 query14 8559681467766776 query15 214 180 179 179 query16 7254268 259 259 query17 1677575 547 547 query18 1597290 272 272 query19 198 151 150 150 query20 96 85 88 85 query21 198 124 130 124 query22 5062481047814781 query23 33560 33025 33325 33025 query24 11838 302630213021 query25 581 416 406 406 query26 1435159 159 159 query27 3197362 379 362 query28 7656206120522052 query29 899 658 633 633 query30 298 179 174 174 query31 935 751 750 750 query32 63 55 54 54 query33 571 261 248 248 query34 939 491 500 491 query35 837 709 725 709 query36 1060920 938 920 query37 138 70 76 70 query38 3434331732923292 query39 1631157915961579 query40 182 134 125 125 query41 51 48 48 48 query42 106 100 103 100 query43 565 529 523 523 query44 1337744 746 744 query45 288 284 270 270 query46 1087743 734 734 query47 2024197719591959 query48 374 318 332 318 query49 832 380 378 378 query50 789 404 401 401 query51 6984685468216821 query52 94 89 88 88 query53 346 281 277 277 query54 257 230 228 228 query55 157 73 70 70 query56 231 218 228 218 query57 1184110811181108 query58 213 200 193 193 query59 3250303430423034 query60 242 229 232 229 query61 93 90 92 90 query62 591 443 452 443 query63 297 282 278 278 query64 4224396137603760 query65 3065303730273027 query66 719 321 319 319 query67 15763 15279 15017 15017 query68 10647 567 560 560 query69 617 345 303 303 query70 1415118311361136 query71 526 274 266 266 query72 6525264524322432 query73 1897324 320 320 query74 7045641964556419 query75 4005262826532628 query76 6704118311221122 query77 632 245 250 245 query78 10995 10212 10201 10201 query79 11899 510 517 510 query80 2305416 420 416 query81 569 241 233 233 query82 303 93 92 92 query83 221 167 167 167 query84 265 80 79 79 query85 1143273 261 261 query86 347 282 264 264 query87 3511331232543254 query88 5863235122792279 query89 507 366 376 366 query90 2386175 182 175 query91 122 96 94 94 query92 59 47 48 47 query93 6762509 510 509 query94 1570185 183 183 query95 395 285 285 285 query96 597 267 263 263 query97 3112290328942894 query98 231 224 226 224 query99 1094851 911 851 Total cold run time: 316144 ms Total hot run time: 183439 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2056320937 TPC-H: Total hot run time: 38312 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit d8905e1aeb1903b233eea10e1325793e3fabc2c1, data reload: false -- Round 1 -- q1 17636 433442494249 q2 2017194 180 180 q3 10464 112011841120 q4 10187 801 794 794 q5 7499268926202620 q6 213 133 135 133 q7 1006607 589 589 q8 9222204520222022 q9 7947665265416541 q10 8532354134903490 q11 454 237 231 231 q12 433 219 214 214 q13 19018 291429422914 q14 267 231 240 231 q15 514 476 467 467 q16 507 403 377 377 q17 962 662 665 662 q18 7318685866086608 q19 6471155615101510 q20 683 310 299 299 q21 3545276227932762 q22 356 299 310 299 Total cold run time: 115251 ms Total hot run time: 38312 ms - Round 2, with runtime_filter_mode=off - q1 4337425341784178 q2 367 265 273 265 q3 2957278227902782 q4 1868160415671567 q5 5352531952675267 q6 211 125 125 125 q7 2223185918381838 q8 3187332933173317 q9 8544846986708469 q10 4075386240233862 q11 600 506 504 504 q12 827 629 609 609 q13 16125 317231093109 q14 309 281 290 281 q15 498 464 483 464 q16 493 447 438 438 q17 1819151515611515 q18 8123787477717771 q19 1704173516291629 q20 2038182518641825 q21 5067493048534853 q22 547 469 460 460 Total cold run time: 71271 ms Total hot run time: 55128 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2056181223 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2051550674 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1561923173
##
fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java:
##
@@ -2290,4 +2480,177 @@ protected void
updateMultiTableRunningTransactionTableIds(long transactionId, Li
}
idToRunningTransactionState.get(transactionId).setTableIdList(tableIds);
}
+
+private PublishResult finishCheckQuorumReplicas(TransactionState
transactionState, Set errorReplicaIds) {
+long now = System.currentTimeMillis();
+long firstPublishVersionTime =
transactionState.getFirstPublishVersionTime();
+boolean allowPublishOneSucc = false;
+if (Config.publish_wait_time_second > 0 && firstPublishVersionTime > 0
+&& now >= firstPublishVersionTime +
Config.publish_wait_time_second * 1000L) {
+allowPublishOneSucc = true;
+}
+List logs = Lists.newArrayList();
+
+Map> publishTasks =
transactionState.getPublishVersionTasks();
+PublishResult publishResult = PublishResult.QUORUM_SUCC;
+for (SubTransactionState subTransactionState :
transactionState.getSubTransactionStates()) {
+long subTxnId = subTransactionState.getSubTransactionId();
+TableCommitInfo tableCommitInfo =
transactionState.getSubTxnIdToTableCommitInfo().get(subTxnId);
+if (tableCommitInfo == null) {
+continue;
+}
+OlapTable table = (OlapTable) subTransactionState.getTable();
+long tableId = table.getId();
+for (Entry entry :
tableCommitInfo.getIdToPartitionCommitInfo().entrySet()) {
+long partitionId = entry.getValue().getPartitionId();
+Partition partition = table.getPartition(partitionId);
+if (partition == null) {
+continue;
+}
+int loadRequiredReplicaNum =
table.getLoadRequiredReplicaNum(partitionId);
+
+// TODO should use sub transaction load indexes
+List allIndices;
+if (transactionState.getLoadedTblIndexes().isEmpty()) {
+allIndices =
partition.getMaterializedIndices(MaterializedIndex.IndexExtState.ALL);
+} else {
+allIndices = Lists.newArrayList();
+for (long indexId :
transactionState.getLoadedTblIndexes().get(tableId)) {
+MaterializedIndex index = partition.getIndex(indexId);
+if (index != null) {
+allIndices.add(index);
+}
+}
+}
+
+boolean alterReplicaLoadedTxn =
isAlterReplicaLoadedTxn(transactionState.getTransactionId(), table);
+
+// check success replica number for each tablet.
+// a success replica means:
+// 1. Not in errorReplicaIds: succeed in both commit and
publish phase
+// 2. last failed version < 0: is a health replica before
+// 3. version catch up: not with a stale version
+// Here we only check number, the replica version will be
updated in updateCatalogAfterVisible()
+for (MaterializedIndex index : allIndices) {
+for (Tablet tablet :
partition.getIndex(index.getId()).getTablets()) {
+List tabletSuccReplicas =
Lists.newArrayList();
+List tabletWriteFailedReplicas =
Lists.newArrayList();
+List tabletVersionFailedReplicas =
Lists.newArrayList();
+// TODO always use the visible version because the
replica version is not changed
+long newVersion = partition.getVisibleVersion() + 1;
+for (Replica replica : tablet.getReplicas()) {
+for (PublishVersionTask publishVersionTask :
publishTasks.get(replica.getBackendId())) {
+boolean needCheck =
publishVersionTask.getTransactionId()
+==
subTransactionState.getSubTransactionId()
+&&
publishVersionTask.getPartitionVersionInfos().stream()
+.anyMatch(s -> s.getPartitionId() ==
partitionId);
+if (needCheck) {
+
checkReplicaContinuousVersionSucc(tablet.getId(), replica,
alterReplicaLoadedTxn,
+newVersion, publishVersionTask,
+errorReplicaIds,
tabletSuccReplicas, tabletWriteFailedReplicas,
+tabletVersionFailedReplicas);
+LOG.debug("after
checkReplicaContinuousVersion for
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2049484422 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2047369839 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2044680325 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2042114445 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 9b75e5b32beabd65db623865223d004260299b85 with default session variables Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s Insert into select: 14.5 seconds inserted 1000 Rows, about 689K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2042108698 ClickBench: Total hot run time: 29.87 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 9b75e5b32beabd65db623865223d004260299b85, data reload: false query1 0.040.040.03 query2 0.090.040.04 query3 0.230.060.06 query4 1.660.070.07 query5 0.480.480.48 query6 1.130.650.66 query7 0.020.010.01 query8 0.040.040.04 query9 0.540.480.50 query10 0.540.550.54 query11 0.170.120.11 query12 0.150.120.11 query13 0.600.580.59 query14 0.790.770.76 query15 0.820.800.80 query16 0.370.350.36 query17 0.971.010.92 query18 0.220.210.26 query19 1.791.661.68 query20 0.010.020.01 query21 15.43 0.660.64 query22 5.336.131.92 query23 18.28 1.361.11 query24 1.730.230.19 query25 0.130.080.08 query26 0.260.160.16 query27 0.090.080.07 query28 13.57 0.990.99 query29 12.61 3.273.31 query30 0.300.070.06 query31 3.020.380.38 query32 3.290.470.46 query33 2.772.812.79 query34 17.34 4.334.45 query35 4.534.494.48 query36 0.630.460.45 query37 0.190.150.15 query38 0.140.140.14 query39 0.050.040.03 query40 0.170.150.14 query41 0.090.050.04 query42 0.060.050.05 query43 0.040.040.03 Total cold run time: 110.71 s Total hot run time: 29.87 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2042098357 TPC-DS: Total hot run time: 181807 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 9b75e5b32beabd65db623865223d004260299b85, data reload: false query1 883 112611171117 query2 6473190019231900 query3 6655204 213 204 query4 37442 21656 21259 21259 query5 4171412 400 400 query6 229 175 179 175 query7 4047288 287 287 query8 230 176 177 176 query9 5808229622702270 query10 504 245 254 245 query11 14702 14462 14287 14287 query12 138 92 85 85 query13 972 349 355 349 query14 8994673268616732 query15 206 177 179 177 query16 6928254 248 248 query17 1745572 539 539 query18 1607286 270 270 query19 191 147 143 143 query20 93 87 84 84 query21 203 133 123 123 query22 4912479947924792 query23 33570 33116 32895 32895 query24 6997296929392939 query25 453 352 368 352 query26 731 155 149 149 query27 2532314 318 314 query28 4416190718781878 query29 830 594 586 586 query30 247 167 161 161 query31 876 696 730 696 query32 59 55 58 55 query33 501 244 245 244 query34 856 459 465 459 query35 801 670 680 670 query36 1034883 911 883 query37 106 70 70 70 query38 3500346934573457 query39 1606153715451537 query40 209 132 139 132 query41 50 46 47 46 query42 107 103 98 98 query43 486 458 443 443 query44 1189712 709 709 query45 268 262 262 262 query46 1050714 724 714 query47 1913187518221822 query48 353 302 289 289 query49 702 364 374 364 query50 761 377 377 377 query51 6916670965066506 query52 108 99 97 97 query53 353 282 282 282 query54 273 241 244 241 query55 83 77 82 77 query56 251 228 225 225 query57 1184110411131104 query58 230 206 206 206 query59 2805264929852649 query60 266 254 265 254 query61 113 111 109 109 query62 545 427 460 427 query63 312 283 288 283 query64 4343405940194019 query65 3076305530403040 query66 849 335 333 333 query67 15625 14931 15164 14931 query68 7398532 548 532 query69 613 315 315 315 query70 1247114311641143 query71 499 294 276 276 query72 6682271325612561 query73 830 315 319 315 query74 7266636963596359 query75 3382227122712271 query76 4160110511281105 query77 702 247 250 247 query78 10968 10175 10231 10175 query79 12283 552 539 539 query80 2251431 424 424 query81 499 237 228 228 query82 155596 96 96 query83 311 166 165 165 query84 264 81 84 81 query85 896 262 258 258 query86 477 304 305 304 query87 3612349735143497 query88 6790235322772277 query89 549 376 371 371 query90 2228179 181 179 query91 121 94 97 94 query92 61 49 47 47 query93 7059505 500 500 query94 1054179 180 179 query95 396 308 307 307 query96 611 260 265 260 query97 2732243524812435 query98 225 237 211 211 query99 1174810 818 810 Total cold run time: 299563 ms Total hot run time: 181807 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2042075692 TPC-H: Total hot run time: 38335 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 9b75e5b32beabd65db623865223d004260299b85, data reload: false -- Round 1 -- q1 4864422742114211 q2 404 193 187 187 q3 5441114911451145 q4 8159742 696 696 q5 2656278328872783 q6 226 140 132 132 q7 1013615 553 553 q8 1950204520292029 q9 6697658065286528 q10 3660359335223522 q11 354 232 241 232 q12 386 214 206 206 q13 17602 291329532913 q14 276 229 236 229 q15 523 496 487 487 q16 496 407 380 380 q17 938 665 597 597 q18 7264678966866686 q19 1531151614941494 q20 517 307 306 306 q21 3421271727262717 q22 359 305 302 302 Total cold run time: 68737 ms Total hot run time: 38335 ms - Round 2, with runtime_filter_mode=off - q1 4221419441844184 q2 354 267 254 254 q3 2987277027742770 q4 1882155415981554 q5 5288527452915274 q6 209 122 123 122 q7 2275190618791879 q8 3190331233263312 q9 8582855984968496 q10 3892368337233683 q11 581 474 477 474 q12 736 603 597 597 q13 15643 297429602960 q14 298 275 256 256 q15 510 471 468 468 q16 472 418 418 418 q17 1776147114641464 q18 7550758973537353 q19 1637150614791479 q20 1910176317511751 q21 4978484248424842 q22 546 461 457 457 Total cold run time: 69517 ms Total hot run time: 54047 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2042043077 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2041996141 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2041839874 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 8ed7e2d1f979eda02f060b4ccb98d95d39336544 with default session variables Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s Stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 13.5 seconds inserted 1000 Rows, about 740K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2041837530 ClickBench: Total hot run time: 31.14 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 8ed7e2d1f979eda02f060b4ccb98d95d39336544, data reload: false query1 0.040.040.03 query2 0.080.030.04 query3 0.220.050.05 query4 1.680.080.08 query5 0.500.490.50 query6 1.110.660.66 query7 0.020.010.01 query8 0.050.040.04 query9 0.560.510.50 query10 0.550.550.54 query11 0.160.110.12 query12 0.150.120.12 query13 0.610.590.59 query14 0.770.780.81 query15 0.850.830.83 query16 0.380.380.37 query17 1.010.940.94 query18 0.200.260.23 query19 1.851.711.87 query20 0.020.010.01 query21 15.42 0.650.65 query22 4.526.332.66 query23 18.27 1.231.15 query24 1.660.310.21 query25 0.120.080.08 query26 0.260.150.15 query27 0.080.080.07 query28 13.32 1.021.00 query29 12.62 3.323.33 query30 0.250.060.06 query31 2.850.400.38 query32 3.300.470.47 query33 2.932.862.85 query34 17.07 4.504.44 query35 4.554.484.55 query36 0.650.470.48 query37 0.200.160.14 query38 0.150.150.14 query39 0.060.040.03 query40 0.190.140.14 query41 0.090.050.04 query42 0.060.050.05 query43 0.040.040.04 Total cold run time: 109.47 s Total hot run time: 31.14 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2041833272 TPC-DS: Total hot run time: 181749 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 8ed7e2d1f979eda02f060b4ccb98d95d39336544, data reload: false query1 888 11091109 query2 6464188318681868 query3 6654202 212 202 query4 37546 21732 21454 21454 query5 4137395 391 391 query6 223 176 175 175 query7 4044286 283 283 query8 218 165 167 165 query9 5751228322732273 query10 510 236 242 236 query11 14572 14211 14151 14151 query12 138 91 88 88 query13 979 354 353 353 query14 10275 675468096754 query15 207 184 177 177 query16 6749261 263 261 query17 1673595 557 557 query18 1397291 271 271 query19 194 153 148 148 query20 91 88 82 82 query21 199 131 132 131 query22 4985489948494849 query23 33569 33017 32992 32992 query24 12254 299929302930 query25 541 358 377 358 query26 1849150 158 150 query27 3085301 318 301 query28 7419191218781878 query29 855 590 584 584 query30 306 162 163 162 query31 887 677 711 677 query32 59 52 51 51 query33 584 247 245 245 query34 888 468 476 468 query35 816 685 700 685 query36 1038924 898 898 query37 230 69 71 69 query38 3551338834173388 query39 1579153315531533 query40 276 131 138 131 query41 50 47 46 46 query42 106 98 98 98 query43 503 436 456 436 query44 1445717 720 717 query45 280 264 275 264 query46 1083716 733 716 query47 1903187118331833 query48 375 290 289 289 query49 1150372 364 364 query50 761 383 393 383 query51 6824657666966576 query52 106 93 96 93 query53 356 280 282 280 query54 265 234 230 230 query55 78 73 74 73 query56 241 222 227 222 query57 1216113111511131 query58 230 204 208 204 query59 2845271025592559 query60 259 241 252 241 query61 107 111 109 109 query62 624 434 422 422 query63 313 284 286 284 query64 4996414539643964 query65 3153306330293029 query66 1345340 328 328 query67 15431 14963 15219 14963 query68 8461537 532 532 query69 607 311 313 311 query70 1254115011271127 query71 548 272 267 267 query72 6621274124382438 query73 833 317 316 316 query74 7109638363476347 query75 3686228323542283 query76 5007110011001100 query77 647 238 238 238 query78 10869 10260 10187 10187 query79 9961535 510 510 query80 2055413 418 413 query81 502 230 234 230 query82 131393 98 93 query83 216 170 167 167 query84 263 79 82 79 query85 256 258 256 query86 500 291 280 280 query87 3729355635373537 query88 6119228722922287 query89 533 366 376 366 query90 1981182 181 181 query91 120 96 95 95 query92 61 46 49 46 query93 7212507 497 497 query94 1214176 177 176 query95 397 304 310 304 query96 600 265 261 261 query97 2678247424982474 query98 241 219 210 210 query99 1167863 809 809 Total cold run time: 311340 ms Total hot run time: 181749 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2041824131 TPC-H: Total hot run time: 38735 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 8ed7e2d1f979eda02f060b4ccb98d95d39336544, data reload: false -- Round 1 -- q1 18399 449543904390 q2 2636189 189 189 q3 11164 121612131213 q4 10375 771 726 726 q5 7500274426442644 q6 217 135 135 135 q7 1003596 586 586 q8 9221208520652065 q9 7915660265486548 q10 8512359135283528 q11 463 232 225 225 q12 384 220 208 208 q13 18713 298029642964 q14 268 242 243 242 q15 511 479 472 472 q16 523 402 374 374 q17 972 718 698 698 q18 7437670067416700 q19 1588150614971497 q20 690 300 312 300 q21 3316273627502736 q22 360 295 305 295 Total cold run time: 112167 ms Total hot run time: 38735 ms - Round 2, with runtime_filter_mode=off - q1 4253418442264184 q2 371 263 256 256 q3 3005275627192719 q4 1920163616091609 q5 5289529853065298 q6 212 124 125 124 q7 2219186518651865 q8 3188335133153315 q9 8665863285948594 q10 3937370636473647 q11 579 480 481 480 q12 726 573 563 563 q13 17537 296629332933 q14 279 270 269 269 q15 502 461 471 461 q16 480 424 421 421 q17 1782149114711471 q18 7482745673207320 q19 1625154315811543 q20 1975173917401739 q21 4958469747234697 q22 529 450 457 450 Total cold run time: 71513 ms Total hot run time: 53958 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2041810893 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2041452598 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 729cc3012a86c45146c309916d4ad1538470cc2d with default session variables Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 16.7 seconds inserted 1000 Rows, about 598K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2041451828 ClickBench: Total hot run time: 30.1 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 729cc3012a86c45146c309916d4ad1538470cc2d, data reload: false query1 0.040.030.03 query2 0.070.040.04 query3 0.240.040.04 query4 1.670.060.06 query5 0.490.480.49 query6 1.140.660.65 query7 0.020.020.01 query8 0.040.040.04 query9 0.550.490.50 query10 0.570.570.56 query11 0.160.120.11 query12 0.140.120.12 query13 0.600.600.60 query14 0.780.800.78 query15 0.860.850.85 query16 0.340.350.36 query17 0.960.970.96 query18 0.250.240.25 query19 1.841.721.75 query20 0.010.000.01 query21 15.42 0.650.64 query22 3.816.661.83 query23 18.04 1.371.26 query24 1.510.200.19 query25 0.160.070.08 query26 0.270.180.17 query27 0.080.080.08 query28 13.85 0.970.94 query29 12.59 3.263.27 query30 0.260.060.05 query31 2.880.400.39 query32 3.240.480.48 query33 2.862.912.88 query34 15.51 4.344.33 query35 4.384.344.36 query36 0.680.480.47 query37 0.180.160.15 query38 0.160.140.14 query39 0.040.040.04 query40 0.180.140.15 query41 0.100.050.05 query42 0.050.040.05 query43 0.040.040.03 Total cold run time: 107.06 s Total hot run time: 30.1 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2041450280 TPC-DS: Total hot run time: 181653 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 729cc3012a86c45146c309916d4ad1538470cc2d, data reload: false query1 905 111211101110 query2 6305204119501950 query3 8266226 221 221 query4 26608 21685 21300 21300 query5 4182401 399 399 query6 285 191 191 191 query7 4603312 291 291 query8 239 187 171 171 query9 849822022202 query10 464 245 251 245 query11 14882 14453 14406 14406 query12 139 92 86 86 query13 1633370 384 370 query14 9352699669396939 query15 219 180 178 178 query16 7119268 270 268 query17 1711563 551 551 query18 1817268 271 268 query19 203 154 148 148 query20 98 86 85 85 query21 195 123 124 123 query22 5005481147784778 query23 33020 32629 32556 32556 query24 9984318431563156 query25 594 383 390 383 query26 723 158 169 158 query27 3019364 392 364 query28 5741192318671867 query29 953 620 617 617 query30 340 177 175 175 query31 1004780 796 780 query32 101 55 58 55 query33 573 262 257 257 query34 905 508 521 508 query35 828 720 696 696 query36 1018904 879 879 query37 106 75 77 75 query38 3737361736103610 query39 1624160415931593 query40 176 130 128 128 query41 53 46 46 46 query42 109 102 108 102 query43 467 441 429 429 query44 1107763 732 732 query45 288 264 251 251 query46 1090821 776 776 query47 1977186318891863 query48 397 314 316 314 query49 833 371 369 369 query50 830 419 414 414 query51 6841673867686738 query52 108 93 95 93 query53 357 291 299 291 query54 266 233 232 232 query55 87 77 76 76 query56 245 240 248 240 query57 1261116711881167 query58 232 219 227 219 query59 2714242923792379 query60 249 258 243 243 query61 111 115 87 87 query62 663 456 450 450 query63 304 277 279 277 query64 4248315031253125 query65 3045302529802980 query66 735 322 308 308 query67 15703 14883 14741 14741 query68 8945578 583 578 query69 552 295 304 295 query70 1315112010901090 query71 487 261 272 261 query72 6269256824412441 query73 970 319 325 319 query74 6693629163576291 query75 3663230022992299 query76 5546120812261208 query77 650 244 255 244 query78 10838 10150 10147 10147 query79 9946521 524 521 query80 1872419 419 419 query81 496 237 232 232 query82 528 98 90 90 query83 210 165 163 163 query84 273 86 87 86 query85 1055274 270 270 query86 371 293 286 286 query87 3654349035173490 query88 3724234623532346 query89 565 374 366 366 query90 1943180 177 177 query91 132 105 101 101 query92 59 48 48 48 query93 6651521 516 516 query94 1118177 181 177 query95 418 311 308 308 query96 607 270 268 268 query97 2621249924802480 query98 231 216 225 216 query99 1297859 841 841 Total cold run time: 295193 ms Total hot run time: 181653 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2041446593 TPC-H: Total hot run time: 38520 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 729cc3012a86c45146c309916d4ad1538470cc2d, data reload: false -- Round 1 -- q1 17615 413540674067 q2 2023191 185 185 q3 10564 117913261179 q4 10242 804 974 804 q5 7486293729152915 q6 220 137 135 135 q7 1118625 615 615 q8 9409198320161983 q9 6621617961536153 q10 8441349034963490 q11 417 248 244 244 q12 380 211 213 211 q13 17783 290529132905 q14 265 236 236 236 q15 524 485 488 485 q16 517 375 373 373 q17 941 886 889 886 q18 7201653964086408 q19 1588152215201520 q20 539 311 310 310 q21 3484313131183118 q22 361 298 314 298 Total cold run time: 107739 ms Total hot run time: 38520 ms - Round 2, with runtime_filter_mode=off - q1 4069404040594040 q2 325 220 221 220 q3 2968295329612953 q4 1853183218251825 q5 5231522752275227 q6 210 127 127 127 q7 2265181717641764 q8 3226330032843284 q9 8457850185088501 q10 3767394840093948 q11 562 498 454 454 q12 782 576 609 576 q13 17162 312731373127 q14 307 283 279 279 q15 523 473 495 473 q16 494 454 449 449 q17 1786175817571757 q18 8311773678217736 q19 1711169016931690 q20 2040182518761825 q21 5216504850095009 q22 494 444 433 433 Total cold run time: 71759 ms Total hot run time: 55697 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2041436808 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2029375842 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
dataroaring commented on code in PR #32980:
URL: https://github.com/apache/doris/pull/32980#discussion_r1546035141
##
fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java:
##
@@ -2290,4 +2480,177 @@ protected void
updateMultiTableRunningTransactionTableIds(long transactionId, Li
}
idToRunningTransactionState.get(transactionId).setTableIdList(tableIds);
}
+
+private PublishResult finishCheckQuorumReplicas(TransactionState
transactionState, Set errorReplicaIds) {
+long now = System.currentTimeMillis();
+long firstPublishVersionTime =
transactionState.getFirstPublishVersionTime();
+boolean allowPublishOneSucc = false;
+if (Config.publish_wait_time_second > 0 && firstPublishVersionTime > 0
+&& now >= firstPublishVersionTime +
Config.publish_wait_time_second * 1000L) {
+allowPublishOneSucc = true;
+}
+List logs = Lists.newArrayList();
+
+Map> publishTasks =
transactionState.getPublishVersionTasks();
+PublishResult publishResult = PublishResult.QUORUM_SUCC;
+for (SubTransactionState subTransactionState :
transactionState.getSubTransactionStates()) {
+long subTxnId = subTransactionState.getSubTransactionId();
+TableCommitInfo tableCommitInfo =
transactionState.getSubTxnIdToTableCommitInfo().get(subTxnId);
+if (tableCommitInfo == null) {
+continue;
+}
+OlapTable table = (OlapTable) subTransactionState.getTable();
+long tableId = table.getId();
+for (Entry entry :
tableCommitInfo.getIdToPartitionCommitInfo().entrySet()) {
+long partitionId = entry.getValue().getPartitionId();
+Partition partition = table.getPartition(partitionId);
+if (partition == null) {
+continue;
+}
+int loadRequiredReplicaNum =
table.getLoadRequiredReplicaNum(partitionId);
+
+// TODO should use sub transaction load indexes
+List allIndices;
+if (transactionState.getLoadedTblIndexes().isEmpty()) {
+allIndices =
partition.getMaterializedIndices(MaterializedIndex.IndexExtState.ALL);
+} else {
+allIndices = Lists.newArrayList();
+for (long indexId :
transactionState.getLoadedTblIndexes().get(tableId)) {
+MaterializedIndex index = partition.getIndex(indexId);
+if (index != null) {
+allIndices.add(index);
+}
+}
+}
+
+boolean alterReplicaLoadedTxn =
isAlterReplicaLoadedTxn(transactionState.getTransactionId(), table);
+
+// check success replica number for each tablet.
+// a success replica means:
+// 1. Not in errorReplicaIds: succeed in both commit and
publish phase
+// 2. last failed version < 0: is a health replica before
+// 3. version catch up: not with a stale version
+// Here we only check number, the replica version will be
updated in updateCatalogAfterVisible()
+for (MaterializedIndex index : allIndices) {
+for (Tablet tablet :
partition.getIndex(index.getId()).getTablets()) {
+List tabletSuccReplicas =
Lists.newArrayList();
+List tabletWriteFailedReplicas =
Lists.newArrayList();
+List tabletVersionFailedReplicas =
Lists.newArrayList();
+// TODO always use the visible version because the
replica version is not changed
+long newVersion = partition.getVisibleVersion() + 1;
+for (Replica replica : tablet.getReplicas()) {
+for (PublishVersionTask publishVersionTask :
publishTasks.get(replica.getBackendId())) {
+boolean needCheck =
publishVersionTask.getTransactionId()
+==
subTransactionState.getSubTransactionId()
+&&
publishVersionTask.getPartitionVersionInfos().stream()
+.anyMatch(s -> s.getPartitionId() ==
partitionId);
+if (needCheck) {
+
checkReplicaContinuousVersionSucc(tablet.getId(), replica,
alterReplicaLoadedTxn,
+newVersion, publishVersionTask,
+errorReplicaIds,
tabletSuccReplicas, tabletWriteFailedReplicas,
+tabletVersionFailedReplicas);
+LOG.debug("after
checkReplicaContinuousVersion
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2026658730 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit c9064b19b33ba6d3d773f219b495c028707f82bf with default session variables Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 13.7 seconds inserted 1000 Rows, about 729K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2026655743 ClickBench: Total hot run time: 28.9 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit c9064b19b33ba6d3d773f219b495c028707f82bf, data reload: false query1 0.040.040.04 query2 0.060.020.02 query3 0.220.060.06 query4 1.640.100.09 query5 0.510.480.50 query6 1.150.630.65 query7 0.020.020.01 query8 0.040.030.03 query9 0.560.480.49 query10 0.540.560.56 query11 0.130.090.09 query12 0.120.100.09 query13 0.580.580.57 query14 0.770.770.74 query15 0.830.800.79 query16 0.390.370.38 query17 1.000.990.95 query18 0.230.240.22 query19 1.781.661.74 query20 0.010.000.01 query21 15.56 0.560.54 query22 2.792.891.27 query23 17.24 0.720.75 query24 3.501.081.08 query25 0.140.220.10 query26 0.560.130.12 query27 0.030.030.07 query28 11.07 0.840.87 query29 12.51 3.253.14 query30 0.280.090.08 query31 2.830.350.36 query32 3.340.440.44 query33 2.782.842.76 query34 16.09 4.354.44 query35 4.494.414.42 query36 0.640.480.48 query37 0.080.060.06 query38 0.040.040.03 query39 0.030.020.02 query40 0.170.140.14 query41 0.070.020.02 query42 0.020.020.02 query43 0.030.020.02 Total cold run time: 104.91 s Total hot run time: 28.9 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2026651620 TPC-DS: Total hot run time: 182424 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit c9064b19b33ba6d3d773f219b495c028707f82bf, data reload: false query1 920 366 355 355 query2 6369205219891989 query3 6703225 209 209 query4 31822 21232 21263 21232 query5 4332417 413 413 query6 281 187 182 182 query7 4640288 293 288 query8 234 174 176 174 query9 9199226622482248 query10 445 257 268 257 query11 17266 14199 14298 14199 query12 142 86 89 86 query13 1622427 420 420 query14 10151 786272217221 query15 240 186 181 181 query16 8197269 273 269 query17 1923584 552 552 query18 2113316 293 293 query19 273 159 168 159 query20 102 95 88 88 query21 204 131 126 126 query22 5023485647564756 query23 33410 32996 33191 32996 query24 12208 296429642964 query25 660 413 411 411 query26 1646177 167 167 query27 3198372 372 372 query28 7889192119221921 query29 1005642 647 642 query30 320 157 156 156 query31 1029776 782 776 query32 99 61 59 59 query33 778 272 264 264 query34 1334503 518 503 query35 937 733 747 733 query36 1067920 924 920 query37 266 69 66 66 query38 3774370436133613 query39 1655163015921592 query40 257 116 119 116 query41 53 49 49 49 query42 111 98 104 98 query43 490 472 454 454 query44 1187759 760 759 query45 297 256 267 256 query46 1107719 709 709 query47 2042193519201920 query48 475 373 376 373 query49 1031369 376 369 query50 788 396 394 394 query51 6912684268356835 query52 105 95 95 95 query53 344 283 288 283 query54 307 240 265 240 query55 86 84 86 84 query56 254 234 241 234 query57 1249125812131213 query58 222 227 233 227 query59 2893264024432443 query60 255 245 238 238 query61 96 95 93 93 query62 607 437 435 435 query63 303 272 278 272 query64 6253411939343934 query65 3076302430143014 query66 1321348 351 348 query67 15539 14917 14747 14747 query68 9270550 540 540 query69 672 377 385 377 query70 1451111911611119 query71 525 267 273 267 query72 6795252123582358 query73 1654327 321 321 query74 6912640464006400 query75 3904220721952195 query76 5464843 901 843 query77 603 260 264 260 query78 10958 10151 10181 10151 query79 11586 538 540 538 query80 1777376 378 376 query81 497 221 223 221 query82 238 82 88 82 query83 201 148 155 148 query84 290 80 80 80 query85 1045312 316 312 query86 351 301 287 287 query87 3836356235463546 query88 5476237723642364 query89 487 379 370 370 query90 2064178 176 176 query91 173 136 137 136 query92 63 47 47 47 query93 6230497 504 497 query94 1628190 181 181 query95 408 317 316 316 query96 605 267 271 267 query97 2668249625132496 query98 233 214 215 214 query99 1035855 859 855 Total cold run time: 320235 ms Total hot run time: 182424 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2026641704 TPC-H: Total hot run time: 37824 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit c9064b19b33ba6d3d773f219b495c028707f82bf, data reload: false -- Round 1 -- q1 17630 422541304130 q2 2126163 155 155 q3 10583 117911691169 q4 10224 744 798 744 q5 7474298129362936 q6 207 124 124 124 q7 1034589 575 575 q8 9336202419921992 q9 7224633663416336 q10 8441346135773461 q11 422 232 219 219 q12 370 198 192 192 q13 17801 284628452845 q14 233 210 200 200 q15 496 470 461 461 q16 466 370 368 368 q17 956 567 647 567 q18 7311654565276527 q19 2708143915151439 q20 556 254 257 254 q21 3602292928402840 q22 347 290 307 290 Total cold run time: 109547 ms Total hot run time: 37824 ms - Round 2, with runtime_filter_mode=off - q1 4102406740824067 q2 328 228 243 228 q3 2966285228192819 q4 1891160615991599 q5 5223524652435243 q6 194 117 115 115 q7 2216180417981798 q8 3177328632733273 q9 8434838483708370 q10 3834389139493891 q11 569 462 459 459 q12 754 552 603 552 q13 16954 309530903090 q14 304 283 260 260 q15 538 497 492 492 q16 516 473 463 463 q17 1796153215281528 q18 7928777674757475 q19 3488153915841539 q20 2010181918721819 q21 4969494847844784 q22 562 443 449 443 Total cold run time: 72753 ms Total hot run time: 54307 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2026569870 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2025215125 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit f2b70a5247bbccb71d9a68f6ac8e657ad5b9c27b with default session variables Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 13.8 seconds inserted 1000 Rows, about 724K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2025195418 TPC-DS: Total hot run time: 181883 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit f2b70a5247bbccb71d9a68f6ac8e657ad5b9c27b, data reload: false query1 926 363 369 363 query2 6379201719971997 query3 6706205 201 201 query4 31760 21338 21296 21296 query5 4325406 390 390 query6 274 197 181 181 query7 4625296 297 296 query8 236 195 170 170 query9 9371230622712271 query10 438 256 269 256 query11 14796 14287 14237 14237 query12 138 94 86 86 query13 1619425 428 425 query14 10344 767179867671 query15 252 181 182 181 query16 8173240 248 240 query17 1973558 525 525 query18 2084277 277 277 query19 321 152 144 144 query20 94 84 81 81 query21 206 134 131 131 query22 5018484547034703 query23 33594 32903 33268 32903 query24 10847 293729902937 query25 580 400 409 400 query26 708 175 165 165 query27 2292361 362 361 query28 6194190019741900 query29 929 682 647 647 query30 306 153 157 153 query31 1017808 757 757 query32 95 60 58 58 query33 669 254 260 254 query34 1030520 508 508 query35 902 750 704 704 query36 1036926 910 910 query37 105 67 68 67 query38 3828368636283628 query39 1121105510571055 query40 179 112 113 112 query41 51 48 45 45 query42 109 102 98 98 query43 494 467 475 467 query44 1159771 756 756 query45 289 260 271 260 query46 1117703 724 703 query47 1988193919481939 query48 447 375 369 369 query49 876 357 340 340 query50 789 405 401 401 query51 6862684168156815 query52 126 98 90 90 query53 353 309 284 284 query54 308 254 247 247 query55 88 82 100 82 query56 248 232 233 232 query57 1251119911821182 query58 227 212 212 212 query59 3022268126222622 query60 248 232 237 232 query61 95 89 90 89 query62 606 468 439 439 query63 303 288 280 280 query64 4915398938843884 query65 3076306630513051 query66 785 363 358 358 query67 16509 14881 14859 14859 query68 8583533 544 533 query69 642 396 376 376 query70 1227121211941194 query71 518 262 276 262 query72 6321254723482348 query73 938 307 316 307 query74 8009633863596338 query75 3731217322072173 query76 5250858 887 858 query77 613 258 252 252 query78 10877 10234 10034 10034 query79 12893 520 511 511 query80 2000377 377 377 query81 534 217 212 212 query82 760 85 79 79 query83 202 141 137 137 query84 286 80 75 75 query85 1259329 306 306 query86 395 310 288 288 query87 3745350934763476 query88 5372232823182318 query89 508 367 366 366 query90 1977172 178 172 query91 175 132 144 132 query92 57 45 46 45 query93 7103482 475 475 query94 1174179 172 172 query95 404 315 303 303 query96 591 272 276 272 query97 2619248824862486 query98 252 217 210 210 query99 1150858 819 819 Total cold run time: 311996 ms Total hot run time: 181883 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
doris-robot commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2025170329 TPC-H: Total hot run time: 37510 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit f2b70a5247bbccb71d9a68f6ac8e657ad5b9c27b, data reload: false -- Round 1 -- q1 17608 431941044104 q2 2105157 166 157 q3 10563 115111671151 q4 10229 834 817 817 q5 7451289228922892 q6 205 126 124 124 q7 1015586 576 576 q8 9333196019481948 q9 6952636162826282 q10 8435346036113460 q11 445 227 217 217 q12 403 197 197 197 q13 17790 283428462834 q14 244 199 204 199 q15 517 464 460 460 q16 483 366 363 363 q17 955 619 529 529 q18 7093648263786378 q19 4370142714581427 q20 547 264 265 264 q21 3566287128392839 q22 336 294 292 292 Total cold run time: 110645 ms Total hot run time: 37510 ms - Round 2, with runtime_filter_mode=off - q1 4125412140704070 q2 336 227 233 227 q3 2942279428252794 q4 1827156615251525 q5 5193522752465227 q6 196 120 121 120 q7 2228180418301804 q8 3159331332933293 q9 8438835884088358 q10 3776390939623909 q11 552 461 460 460 q12 759 585 584 584 q13 17076 303930803039 q14 313 260 280 260 q15 513 473 475 473 q16 510 472 477 472 q17 1810156015331533 q18 7762759575147514 q19 1660155116021551 q20 2048179817941794 q21 5074478649034786 q22 540 485 457 457 Total cold run time: 70837 ms Total hot run time: 54250 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2025099184 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](txn insert) txn insert support write to one table many times [doris]
mymeiyi commented on PR #32980: URL: https://github.com/apache/doris/pull/32980#issuecomment-2024956361 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
