Re: [PR] [chore](info) Record `data_footprint` and `total_data_size` in SegmentFooterPB and ColumnMeaPB [doris]
doris-robot commented on PR #51001: URL: https://github.com/apache/doris/pull/51001#issuecomment-2890819631 ClickBench: Total hot run time: 29.99 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit c8f569a243eb8819215910c5a5b9adb7d23fa7c7, data reload: false query1 0.040.040.03 query2 0.150.100.11 query3 0.350.200.21 query4 1.600.210.21 query5 0.480.460.46 query6 1.170.650.66 query7 0.020.010.02 query8 0.060.060.05 query9 0.620.520.54 query10 0.590.590.58 query11 0.260.130.12 query12 0.260.130.14 query13 0.640.620.63 query14 0.800.810.84 query15 0.980.900.87 query16 0.380.370.38 query17 1.041.071.06 query18 0.190.190.19 query19 1.991.911.84 query20 0.020.010.01 query21 15.50 0.970.68 query22 0.941.030.84 query23 14.69 1.620.77 query24 5.410.570.28 query25 0.170.100.08 query26 0.550.230.19 query27 0.090.080.09 query28 11.01 1.220.59 query29 12.54 4.063.42 query30 0.290.090.07 query31 2.840.630.44 query32 3.230.600.51 query33 3.143.133.19 query34 16.34 5.134.46 query35 4.444.504.48 query36 0.630.510.49 query37 0.200.170.16 query38 0.170.160.15 query39 0.050.050.04 query40 0.200.160.15 query41 0.100.060.05 query42 0.060.060.05 query43 0.060.050.05 Total cold run time: 104.29 s Total hot run time: 29.99 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [chore](info) Record `data_footprint` and `total_data_size` in SegmentFooterPB and ColumnMeaPB [doris]
doris-robot commented on PR #51001: URL: https://github.com/apache/doris/pull/51001#issuecomment-2890802915 TPC-DS: Total hot run time: 193869 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit c8f569a243eb8819215910c5a5b9adb7d23fa7c7, data reload: false query1 1384111010461046 query2 6201180317851785 query3 11057 453745044504 query4 55373 25596 23161 23161 query5 5054581 485 485 query6 344 244 195 195 query7 4920519 305 305 query8 308 272 231 231 query9 5329267326732673 query10 438 341 280 280 query11 15101 14973 14899 14899 query12 162 114 103 103 query13 1016557 404 404 query14 10291 643566276435 query15 206 194 177 177 query16 7114672 496 496 query17 1099719 618 618 query18 1596416 321 321 query19 204 202 181 181 query20 136 123 116 116 query21 205 135 113 113 query22 4487451943354335 query23 34295 33774 33634 33634 query24 6467249925222499 query25 535 487 421 421 query26 680 284 156 156 query27 2232522 366 366 query28 3277216921852169 query29 607 578 474 474 query30 287 220 196 196 query31 843 884 788 788 query32 74 69 72 69 query33 450 386 323 323 query34 820 929 560 560 query35 828 837 737 737 query36 963 1014901 901 query37 119 105 82 82 query38 4234432542624262 query39 1533145715011457 query40 217 121 149 121 query41 59 66 53 53 query42 132 110 113 110 query43 505 520 481 481 query44 1435885 869 869 query45 182 175 170 170 query46 891 1062674 674 query47 1793187017831783 query48 419 463 358 358 query49 677 541 448 448 query50 685 697 412 412 query51 4287426842624262 query52 114 118 104 104 query53 241 268 195 195 query54 604 603 535 535 query55 90 87 92 87 query56 334 317 325 317 query57 1201119911291129 query58 280 283 266 266 query59 2682276526812681 query60 386 350 344 344 query61 149 164 122 122 query62 711 736 654 654 query63 233 190 187 187 query64 14861020714 714 query65 4323425242854252 query66 705 401 311 311 query67 16130 15762 15230 15230 query68 6535910 547 547 query69 550 324 286 286 query70 1173111211161112 query71 463 344 305 305 query72 5915492450674924 query73 1341739 367 367 query74 9309904289118911 query75 3247320727332733 query76 38031198780 780 query77 530 376 292 292 query78 10206 10156 93739373 query79 2950835 599 599 query80 818 527 461 461 query81 491 261 221 221 query82 544 127 99 99 query83 330 258 245 245 query84 306 110 90 90 query85 775 366 316 316 query86 384 297 287 287 query87 4538444944214421 query88 3451238923692369 query89 406 332 285 285 query90 1700214 206 206 query91 138 141 123 123 query92 76 59 61 59 query93 2542969 587 587 query94 728 402 276 276 query95 372 299 280 280 query96 494 579 290 290 query97 2761277726942694 query98 239 208 202 202 query99 1430140813031303 Total cold run time: 299765 ms Total hot run time: 193869 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [chore](info) Record `data_footprint` and `total_data_size` in SegmentFooterPB and ColumnMeaPB [doris]
doris-robot commented on PR #51001: URL: https://github.com/apache/doris/pull/51001#issuecomment-2890767339 TPC-H: Total hot run time: 34139 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit c8f569a243eb8819215910c5a5b9adb7d23fa7c7, data reload: false -- Round 1 -- q1 26493 507549564956 q2 2083280 193 193 q3 10556 1276708 708 q4 10231 1031545 545 q5 7771235123862351 q6 189 166 133 133 q7 908 755 632 632 q8 9334137212551255 q9 6832506450915064 q10 6882233419321932 q11 503 298 279 279 q12 352 356 217 217 q13 17764 369430973097 q14 234 226 214 214 q15 533 486 488 486 q16 440 442 372 372 q17 608 861 365 365 q18 7515726972187218 q19 1472958 551 551 q20 334 340 218 218 q21 3960317323802380 q22 1089983 973 973 Total cold run time: 116083 ms Total hot run time: 34139 ms - Round 2, with runtime_filter_mode=off - q1 5155504550975045 q2 243 340 233 233 q3 2210264823122312 q4 1333179014031403 q5 4442441744374417 q6 209 174 128 128 q7 2038197718061806 q8 2637274325342534 q9 7186716270507050 q10 3071318727562756 q11 595 519 493 493 q12 667 777 601 601 q13 3499392833033303 q14 295 319 302 302 q15 527 492 480 480 q16 446 491 454 454 q17 1190154514481448 q18 7782767874907490 q19 836 949 1107949 q20 2015206718501850 q21 4752444744474447 q22 1117106710541054 Total cold run time: 52245 ms Total hot run time: 50555 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [chore](info) Record `data_footprint` and `total_data_size` in SegmentFooterPB and ColumnMeaPB [doris]
doris-robot commented on PR #51001: URL: https://github.com/apache/doris/pull/51001#issuecomment-2890489283 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/c8f569a243eb8819215910c5a5b9adb7d23fa7c7_c8f569a243eb8819215910c5a5b9adb7d23fa7c7_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/c8f569a243eb8819215910c5a5b9adb7d23fa7c7_c8f569a243eb8819215910c5a5b9adb7d23fa7c7_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 83.31% (1113/1336) | | Line Coverage | 66.20% (18660/28187) | | Region Coverage | 65.79% (9260/14076) | | Branch Coverage | 55.58% (4982/8964) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [chore](info) Record `data_footprint` and `total_data_size` in SegmentFooterPB and ColumnMeaPB [doris]
wyxxxcat commented on PR #51001: URL: https://github.com/apache/doris/pull/51001#issuecomment-2890419176 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [chore](info) Record `data_footprint` and `total_data_size` in SegmentFooterPB and ColumnMeaPB [doris]
wyxxxcat commented on PR #51001: URL: https://github.com/apache/doris/pull/51001#issuecomment-2890414539 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [chore](info) Record `data_footprint` and `total_data_size` in SegmentFooterPB and ColumnMeaPB [doris]
zhannngchen commented on code in PR #51001:
URL: https://github.com/apache/doris/pull/51001#discussion_r2094794788
##
be/src/olap/rowset/segment_v2/segment_writer.cpp:
##
@@ -811,7 +813,16 @@ Status SegmentWriter::append_block(const
vectorized::Block* block, size_t row_po
}
RETURN_IF_ERROR(_column_writers[id]->append(converted_result.second->get_nullmap(),
converted_result.second->get_data(), num_rows));
+for (size_t row_id = 0; row_id < block->rows(); ++row_id) {
+const auto& stringRef =
block->get_by_position(cid).column->get_data_at(row_id);
Review Comment:
`get_data_at()` this function is used in ColumnString, ColumnFixedString,
ColumnVector, not support in ColumnArray|ColumnMap...,and should be pair with
insert_data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [chore](info) Record `data_footprint` and `total_data_size` in SegmentFooterPB and ColumnMeaPB [doris]
zhannngchen commented on code in PR #51001:
URL: https://github.com/apache/doris/pull/51001#discussion_r2094795164
##
be/src/olap/rowset/segment_v2/segment_writer.cpp:
##
@@ -811,7 +813,16 @@ Status SegmentWriter::append_block(const
vectorized::Block* block, size_t row_po
}
RETURN_IF_ERROR(_column_writers[id]->append(converted_result.second->get_nullmap(),
converted_result.second->get_data(), num_rows));
+for (size_t row_id = 0; row_id < block->rows(); ++row_id) {
+const auto& stringRef =
block->get_by_position(cid).column->get_data_at(row_id);
+total_data_size += stringRef.size;
+column_data_size += stringRef.size;
+}
+_footer.mutable_columns(id)->set_total_data_size(column_data_size);
Review Comment:
it's reset each time when `append_block()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[PR] [chore](info) Record `data_footprint` and `total_data_size` in SegmentFooterPB and ColumnMeaPB [doris]
wyxxxcat opened a new pull request, #51001: URL: https://github.com/apache/doris/pull/51001 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [chore](info) Record `data_footprint` and `total_data_size` in SegmentFooterPB and ColumnMeaPB [doris]
Thearas commented on PR #51001: URL: https://github.com/apache/doris/pull/51001#issuecomment-2889426836 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
