Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
doris-robot commented on PR #44361: URL: https://github.com/apache/doris/pull/44361#issuecomment-2488737644 TPC-DS: Total hot run time: 191241 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit b14cd87a133700eaeadc0a17dc78c69c1e1eb82e, data reload: false query1 981 386 374 374 query2 6524224121562156 query3 6727215 216 215 query4 34278 23510 23787 23510 query5 4353457 434 434 query6 277 175 173 173 query7 4623297 294 294 query8 277 221 219 219 query9 9718267826622662 query10 482 258 249 249 query11 18236 15233 15176 15176 query12 158 103 103 103 query13 1657429 430 429 query14 10231 716174907161 query15 299 177 174 174 query16 8202455 460 455 query17 1845588 584 584 query18 2139301 303 301 query19 364 158 154 154 query20 118 114 115 114 query21 212 103 102 102 query22 4516442541044104 query23 35144 34102 34064 34064 query24 11621 246625442466 query25 672 400 384 384 query26 1794149 151 149 query27 2869281 270 270 query28 8026245924662459 query29 1020408 401 401 query30 302 148 150 148 query31 1008796 844 796 query32 99 54 59 54 query33 772 286 289 286 query34 1011501 522 501 query35 857 714 728 714 query36 1099968 960 960 query37 274 73 84 73 query38 4425418142864181 query39 1617145614431443 query40 284 96 102 96 query41 46 43 44 43 query42 120 96 97 96 query43 562 501 508 501 query44 1268801 813 801 query45 183 165 162 162 query46 1160684 711 684 query47 1939180118501801 query48 405 313 304 304 query49 1292391 379 379 query50 815 381 391 381 query51 7343712870187018 query52 98 91 88 88 query53 254 177 179 177 query54 1223400 413 400 query55 78 75 85 75 query56 251 247 232 232 query57 1325116611691166 query58 235 244 210 210 query59 3425323830573057 query60 275 242 246 242 query61 140 104 109 104 query62 882 665 677 665 query63 211 195 186 186 query64 5145644 672 644 query65 3329320432423204 query66 1411318 320 318 query67 16321 15830 15709 15709 query68 5006545 576 545 query69 420 249 259 249 query70 1216107911461079 query71 328 242 250 242 query72 6343403040104010 query73 769 362 357 357 query74 10539 899091508990 query75 3483269726772677 query76 2986115711531153 query77 511 272 277 272 query78 10410 946494599459 query79 2247615 606 606 query80 1118423 443 423 query81 547 229 234 229 query82 908 116 117 116 query83 251 148 144 144 query84 237 73 72 72 query85 1382293 288 288 query86 433 280 302 280 query87 4739464644814481 query88 3779222421792179 query89 400 291 295 291 query90 2164188 191 188 query91 138 104 102 102 query92 62 50 53 50 query93 1602543 555 543 query94 1124284 296 284 query95 361 249 247 247 query96 621 280 283 280 query97 2891267526532653 query98 220 211 215 211 query99 1547130613181306 Total cold run time: 307629 ms Total hot run time: 191241 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
doris-robot commented on PR #44361: URL: https://github.com/apache/doris/pull/44361#issuecomment-2488751408 ClickBench: Total hot run time: 32.92 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit b14cd87a133700eaeadc0a17dc78c69c1e1eb82e, data reload: false query1 0.030.030.03 query2 0.070.030.03 query3 0.230.080.07 query4 1.620.100.11 query5 0.440.400.40 query6 1.140.660.66 query7 0.020.020.01 query8 0.040.030.03 query9 0.560.520.51 query10 0.560.560.56 query11 0.140.090.10 query12 0.140.120.11 query13 0.620.610.61 query14 2.812.732.73 query15 0.900.820.83 query16 0.390.370.39 query17 1.001.021.00 query18 0.240.210.22 query19 1.951.892.00 query20 0.020.020.01 query21 15.38 0.610.60 query22 2.762.281.80 query23 16.98 1.000.80 query24 3.241.212.13 query25 0.280.110.20 query26 0.500.150.14 query27 0.040.040.04 query28 9.621.121.08 query29 12.52 3.213.17 query30 0.260.060.06 query31 2.870.390.38 query32 3.270.460.46 query33 3.073.073.16 query34 17.04 4.454.48 query35 4.544.554.53 query36 0.640.500.48 query37 0.090.060.06 query38 0.050.040.03 query39 0.030.020.02 query40 0.150.140.12 query41 0.080.030.02 query42 0.040.020.02 query43 0.030.030.03 Total cold run time: 106.4 s Total hot run time: 32.92 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
doris-robot commented on PR #44361: URL: https://github.com/apache/doris/pull/44361#issuecomment-2488705456 TPC-H: Total hot run time: 39934 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b14cd87a133700eaeadc0a17dc78c69c1e1eb82e, data reload: false -- Round 1 -- q1 17607 744872987298 q2 2035184 177 177 q3 10530 113311411133 q4 10208 709 650 650 q5 7586275627442744 q6 244 147 149 147 q7 984 629 612 612 q8 9245178719441787 q9 6484642063876387 q10 6957229823052298 q11 460 260 264 260 q12 411 217 215 215 q13 17751 306730853067 q14 251 208 209 208 q15 577 532 533 532 q16 663 593 597 593 q17 974 644 585 585 q18 7535672267186718 q19 1346952 970 952 q20 454 189 179 179 q21 4554318630763076 q22 387 325 316 316 Total cold run time: 107243 ms Total hot run time: 39934 ms - Round 2, with runtime_filter_mode=off - q1 7324731672577257 q2 314 229 226 226 q3 2899273928162739 q4 1968168416881684 q5 5412540454785404 q6 215 140 141 140 q7 2129178517281728 q8 3266341834203418 q9 8480850185498501 q10 3511345634573456 q11 593 507 520 507 q12 808 576 581 576 q13 10398 306530513051 q14 289 261 289 261 q15 566 515 510 510 q16 691 628 650 628 q17 1805161516061606 q18 7853758573887388 q19 1691159715621562 q20 2108183718331833 q21 5471540050845084 q22 663 590 567 567 Total cold run time: 68454 ms Total hot run time: 58126 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
qidaye commented on code in PR #44361: URL: https://github.com/apache/doris/pull/44361#discussion_r1850380608 ## fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java: ## @@ -440,7 +441,9 @@ private boolean processDropColumn(DropColumnClause alterClause, OlapTable olapTa newBfCols.add(bfCol); } } -olapTable.setBloomFilterInfo(newBfCols, olapTable.getBfFpp()); Review Comment: Record the new bf columns to `propertyMap`, the actual bf columns drop operation is happening in the following schema change job which created by directly schema change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
airborne12 commented on code in PR #44361: URL: https://github.com/apache/doris/pull/44361#discussion_r1850375254 ## fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java: ## @@ -440,7 +441,9 @@ private boolean processDropColumn(DropColumnClause alterClause, OlapTable olapTa newBfCols.add(bfCol); } } -olapTable.setBloomFilterInfo(newBfCols, olapTable.getBfFpp()); Review Comment: why remove? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
qidaye commented on PR #44361: URL: https://github.com/apache/doris/pull/44361#issuecomment-2488658190 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
doris-robot commented on PR #44361: URL: https://github.com/apache/doris/pull/44361#issuecomment-2488656276 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
airborne12 commented on code in PR #41369:
URL: https://github.com/apache/doris/pull/41369#discussion_r1796616983
##
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java:
##
@@ -430,6 +430,18 @@ private boolean processDropColumn(DropColumnClause
alterClause, OlapTable olapTa
throw new DdlException("Column does not exists: " +
dropColName);
}
+// drop bloom filter column
+Set bfCols = olapTable.getCopiedBfColumns();
+if (bfCols != null) {
+Set newBfCols = new HashSet<>();
+for (String bfCol : bfCols) {
+if (!bfCol.equalsIgnoreCase(dropColName)) {
+newBfCols.add(bfCol);
+}
+}
+olapTable.setBloomFilterInfo(newBfCols, olapTable.getBfFpp());
+}
Review Comment:
may be this can be better
```
Set bfCols = olapTable.getCopiedBfColumns();
if (bfCols != null) {
Set newBfCols = bfCols.stream()
.filter(bfCol ->
!bfCol.equalsIgnoreCase(dropColName))
.collect(Collectors.toSet());
olapTable.setBloomFilterInfo(newBfCols, olapTable.getBfFpp());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
qidaye merged PR #41369: URL: https://github.com/apache/doris/pull/41369 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
doris-robot commented on PR #41369: URL: https://github.com/apache/doris/pull/41369#issuecomment-2376986593 TPC-DS: Total hot run time: 192543 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 96977a45d4985d99039021badb7501e6b37daaf5, data reload: false query1 858 391 420 391 query2 6304224920652065 query3 8688193 213 193 query4 33679 23597 23494 23494 query5 3411464 460 460 query6 275 178 172 172 query7 4193306 330 306 query8 298 238 208 208 query9 9252265226772652 query10 458 269 271 269 query11 17975 15209 15143 15143 query12 157 95 97 95 query13 1519426 424 424 query14 9591756476627564 query15 256 172 180 172 query16 7916497 490 490 query17 1636609 588 588 query18 2002327 339 327 query19 364 162 160 160 query20 124 112 111 111 query21 217 118 109 109 query22 4522452044534453 query23 35568 34317 33912 33912 query24 11033 279628682796 query25 629 413 411 411 query26 1104168 172 168 query27 2239297 309 297 query28 7751242024272420 query29 806 453 433 433 query30 258 153 152 152 query31 1032809 810 809 query32 107 54 58 54 query33 771 301 299 299 query34 907 519 505 505 query35 856 734 729 729 query36 1105981 957 957 query37 160 93 89 89 query38 3980403338763876 query39 1513143214251425 query40 218 98 99 98 query41 51 48 53 48 query42 115 98 96 96 query43 530 490 490 490 query44 1251805 806 805 query45 201 166 168 166 query46 1159736 726 726 query47 1948184218511842 query48 477 391 378 378 query49 890 424 430 424 query50 821 416 414 414 query51 7026702769446944 query52 103 88 97 88 query53 261 184 189 184 query54 1209505 495 495 query55 84 79 80 79 query56 273 271 267 267 query57 1228110811161108 query58 244 251 250 250 query59 3182315728742874 query60 314 264 265 264 query61 102 104 102 102 query62 817 656 670 656 query63 221 192 194 192 query64 4007657 618 618 query65 3276327132213221 query66 1058307 301 301 query67 15748 15487 15416 15416 query68 4704584 557 557 query69 572 301 306 301 query70 1149107510981075 query71 403 272 272 272 query72 7648399040873990 query73 768 360 351 351 query74 10223 923591119111 query75 3993270926942694 query76 3523955 926 926 query77 695 302 289 289 query78 10497 973495829582 query79 4783601 603 601 query80 1819477 473 473 query81 587 242 252 242 query82 712 142 135 135 query83 322 133 136 133 query84 293 73 84 73 query85 1889303 286 286 query86 445 305 305 305 query87 4482439243714371 query88 4830240923762376 query89 415 293 295 293 query90 1998200 185 185 query91 185 141 149 141 query92 69 47 47 47 query93 3450572 558 558 query94 870 306 307 306 query95 364 260 276 260 query96 636 284 289 284 query97 3250315730983098 query98 227 196 194 194 query99 1535131513101310 Total cold run time: 307664 ms Total hot run time: 192543 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
doris-robot commented on PR #41369: URL: https://github.com/apache/doris/pull/41369#issuecomment-237666 ClickBench: Total hot run time: 32.72 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 96977a45d4985d99039021badb7501e6b37daaf5, data reload: false query1 0.050.050.04 query2 0.070.030.03 query3 0.230.070.07 query4 1.640.100.10 query5 0.490.500.49 query6 1.140.740.73 query7 0.020.010.01 query8 0.040.030.04 query9 0.550.500.49 query10 0.540.590.53 query11 0.130.110.11 query12 0.140.110.11 query13 0.600.610.60 query14 2.692.722.83 query15 0.910.820.82 query16 0.380.390.38 query17 1.011.021.06 query18 0.190.190.19 query19 1.931.841.98 query20 0.020.010.01 query21 15.49 0.590.59 query22 2.702.302.59 query23 17.30 0.840.83 query24 2.391.460.47 query25 0.210.110.14 query26 0.280.140.14 query27 0.040.050.04 query28 11.31 1.091.06 query29 12.60 3.243.24 query30 0.240.060.06 query31 2.890.380.38 query32 3.280.460.46 query33 3.013.003.00 query34 16.93 4.454.45 query35 4.514.504.52 query36 0.670.480.47 query37 0.080.060.06 query38 0.050.030.04 query39 0.040.020.02 query40 0.160.120.12 query41 0.080.020.03 query42 0.030.020.02 query43 0.040.030.03 Total cold run time: 107.1 s Total hot run time: 32.72 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
doris-robot commented on PR #41369: URL: https://github.com/apache/doris/pull/41369#issuecomment-2376953878 TPC-H: Total hot run time: 40697 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 96977a45d4985d99039021badb7501e6b37daaf5, data reload: false -- Round 1 -- q1 17599 734672327232 q2 2034288 282 282 q3 12210 106511731065 q4 10569 737 787 737 q5 7734283028012801 q6 240 149 148 148 q7 980 613 608 608 q8 9578193019341930 q9 8170635564046355 q10 6993225323242253 q11 442 248 244 244 q12 416 215 221 215 q13 17792 298030132980 q14 234 215 216 215 q15 568 537 513 513 q16 654 593 573 573 q17 966 584 466 466 q18 7054670667316706 q19 1391104810481048 q20 484 194 192 192 q21 3941331031413141 q22 10881025993 993 Total cold run time: 37 ms Total hot run time: 40697 ms - Round 2, with runtime_filter_mode=off - q1 7242717271897172 q2 320 223 229 223 q3 3076288529522885 q4 2100184518001800 q5 5722578357035703 q6 231 151 150 150 q7 2221185918491849 q8 3374346234753462 q9 8903881288498812 q10 3601353635203520 q11 588 487 493 487 q12 878 625 631 625 q13 8048318031963180 q14 305 272 281 272 q15 588 514 534 514 q16 699 648 634 634 q17 1816162715651565 q18 8316786974817481 q19 1700152813561356 q20 2117187418881874 q21 5772529353175293 q22 1137105610281028 Total cold run time: 68754 ms Total hot run time: 59885 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
github-actions[bot] commented on PR #41369: URL: https://github.com/apache/doris/pull/41369#issuecomment-2376862916 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
github-actions[bot] commented on PR #41369: URL: https://github.com/apache/doris/pull/41369#issuecomment-2376862844 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
qidaye commented on PR #41369: URL: https://github.com/apache/doris/pull/41369#issuecomment-2376842596 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](bloom filter)Fix drop column with bloom filter [doris]
doris-robot commented on PR #41369: URL: https://github.com/apache/doris/pull/41369#issuecomment-2376835186 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
