Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
gavinchou commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5190111879 close due to inactivity -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
gavinchou closed pull request #61518: [feature](filecache) persist table/partition context for cache meta for external table URL: https://github.com/apache/doris/pull/61518 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on code in PR #61518:
URL: https://github.com/apache/doris/pull/61518#discussion_r3654088390
##
gensrc/proto/file_cache.proto:
##
@@ -59,5 +59,7 @@ message BlockMetaPb {
optional FileCacheType type = 1;
optional uint64 size = 2;
optional uint64 ttl = 3;
+optional uint64 context_id = 4;
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5069531683 # BE Regression && UT Coverage Report Increment line coverage `88.59% (233/263)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_8c6d2db138e49afed313998a70e8cab8ddaaf2e0_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_8c6d2db138e49afed313998a70e8cab8ddaaf2e0_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 75.19% (31026/41262) | | Line Coverage | 59.71% (345390/578452) | | Region Coverage | 56.27% (289382/514293) | | Branch Coverage | 57.71% (129513/224409) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5068660813 ClickBench: Total hot run time: 18.46 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 8c6d2db138e49afed313998a70e8cab8ddaaf2e0, data reload: false query1 0.000.000.01 query2 0.090.040.04 query3 0.250.120.12 query4 1.600.130.13 query5 0.200.180.18 query6 1.190.910.93 query7 0.040.000.00 query8 0.060.030.04 query9 0.340.250.26 query10 0.470.450.47 query11 0.170.130.13 query12 0.170.140.14 query13 0.380.370.38 query14 0.590.580.59 query15 0.480.470.45 query16 0.260.260.26 query17 0.940.930.86 query18 0.240.230.22 query19 1.521.441.40 query20 0.020.010.01 query21 15.42 0.240.14 query22 4.760.050.06 query23 16.14 0.310.13 query24 2.990.410.30 query25 0.120.050.04 query26 0.720.210.16 query27 0.040.040.03 query28 3.530.860.48 query29 12.51 4.263.44 query30 0.300.170.14 query31 2.760.500.22 query32 3.320.410.30 query33 1.581.631.62 query34 15.38 2.812.15 query35 2.182.102.11 query36 0.480.320.33 query37 0.080.050.05 query38 0.060.040.03 query39 0.040.030.03 query40 0.140.110.11 query41 0.080.030.03 query42 0.040.030.03 query43 0.040.030.04 Total cold run time: 91.72 s Total hot run time: 18.46 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5068624916 TPC-DS: Total hot run time: 117902 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 8c6d2db138e49afed313998a70e8cab8ddaaf2e0, data reload: false query5 4346580 437 437 query6 451 254 208 208 query7 4910595 348 348 query8 335 186 167 167 query9 8800394639523946 query10 489 326 304 304 query11 5527136712061206 query12 158 97 106 97 query13 1251596 465 465 query14 6073288126842684 query14_12403239024022390 query15 197 158 143 143 query16 1118581 502 502 query17 1127622 505 505 query18 2526454 351 351 query19 213 194 167 167 query20 108 94 92 92 query21 229 151 127 127 query22 7568754075257525 query23 9684935990349034 query23_19076912191279121 query24 74201409981 981 query24_11026102310641023 query25 545 414 373 373 query26 1305397 211 211 query27 2515572 375 375 query28 4421217121542154 query29 1038609 688 609 query30 322 239 200 200 query31 953 696 614 614 query32 115 67 65 65 query33 569 303 239 239 query34 11741140666 666 query35 518 541 447 447 query36 10881078993 993 query37 156 104 90 90 query38 12871028978 978 query39 649 637 611 611 query39_1639 568 594 568 query40 261 169 140 140 query41 64 61 59 59 query42 95 93 97 93 query43 289 290 249 249 query44 1341777 778 777 query45 136 133 130 130 query46 919 1067710 710 query47 1395133613661336 query48 392 392 333 333 query49 585 328 250 250 query50 988 403 340 340 query51 11058 11635 10658 10658 query52 87 91 76 76 query53 258 271 204 204 query54 291 237 215 215 query55 76 72 66 66 query56 297 289 284 284 query57 889 853 835 835 query58 296 254 268 254 query59 1360135512531253 query60 314 261 246 246 query61 151 144 142 142 query62 407 297 263 263 query63 229 201 194 194 query64 2770882 744 744 query65 2116206220402040 query66 1839403 322 322 query67 16508 15677 16114 15677 query68 33951396967 967 query69 401 314 258 258 query70 1121103310541033 query71 367 355 312 312 query72 2909247122902290 query73 772 731 450 450 query74 1739153214581458 query75 1605149813031303 query76 2366985 719 719 query77 332 336 273 273 query78 6044577954405440 query79 15401086795 795 query80 1302553 467 467 query81 492 259 227 227 query82 611 148 111 111 query83 368 324 294 294 query84 283 155 134 134 query85 977 586 540 540 query86 441 318 309 309 query87 1165109410501050 query88 3823295229492949 query89 401 316 293 293 query90 1902194 193 193 query91 192 186 162 162 query92 65 59 56 56 query93 1645137210081008 query94 729 350 338 338 query95 758 573 455 455 query96 1001747 364 364 query97 1365134812831283 query98 183 178 182 178 query99 601 558 500 500 Total cold run time: 208117 ms Total hot run time: 117902 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5068557797 TPC-H: Total hot run time: 23734 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 8c6d2db138e49afed313998a70e8cab8ddaaf2e0, data reload: false -- Round 1 -- q1 17676 356335853563 q2 1990294 188 188 q3 10619 1317753 753 q4 4751316 231 231 q5 8559752 510 510 q6 338 151 114 114 q7 812 745 525 525 q8 10581 169016981690 q9 4734315331423142 q10 66951201976 976 q11 538 347 313 313 q12 709 554 435 435 q13 17939 241419501950 q14 201 197 174 174 q15 q16 545 514 466 466 q17 874 931 876 876 q18 4910376736463646 q19 1361125810821082 q20 793 661 539 539 q21 5665250622702270 q22 434 355 291 291 Total cold run time: 100724 ms Total hot run time: 23734 ms - Round 2, with runtime_filter_mode=off - q1 3973379637763776 q2 254 265 191 191 q3 2671277726022602 q4 1373135810711071 q5 2694257425292529 q6 228 160 106 106 q7 1308117111161116 q8 2048177617341734 q9 3742372036923692 q10 2466239221372137 q11 434 349 328 328 q12 589 564 410 410 q13 2640251020322032 q14 236 242 226 226 q15 q16 536 516 456 456 q17 1310127913751279 q18 5133480746104610 q19 1182118211711171 q20 11361100970 970 q21 4448378838373788 q22 510 454 422 422 Total cold run time: 38911 ms Total hot run time: 34646 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5067897069 # FE UT Coverage Report Increment line coverage ` 62.39% (141/226)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_8c6d2db138e49afed313998a70e8cab8ddaaf2e0/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_8c6d2db138e49afed313998a70e8cab8ddaaf2e0/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
github-actions[bot] commented on code in PR #61518: URL: https://github.com/apache/doris/pull/61518#discussion_r3643856895 ## be/src/io/cache/block_file_cache.cpp: ## @@ -1113,6 +1117,8 @@ FileBlockCell* BlockFileCache::add_cell(const UInt128Wrapper& hash, const CacheC key.meta.type = context.cache_type; key.meta.expiration_time = context.expiration_time; key.meta.tablet_id = context.tablet_id; +key.meta.table_name = context.table_name; Review Comment: [P2] Propagate context through normal write-through fills The ordinary write-through path can create this cell before any scanner supplies these fields. `FileCacheAllocatorBuilder::allocate_cache_holder()` gives the S3/HDFS writers a context with cache type/TTL/coldness/tablet ID only, while `PackedFileManager` independently constructs the same anonymous context; all of them can finalize and persist the block. A later named scan only reuses the existing cell, so `FILE_CACHE_INFO` keeps empty table/partition labels until eviction. Please propagate stable table/partition identity from rowset/tablet metadata into both writer paths (or safely late-bind an anonymous cell), and cover a write-through-before-query case. ## be/src/io/cache/block_file_cache.cpp: ## @@ -1113,6 +1117,8 @@ FileBlockCell* BlockFileCache::add_cell(const UInt128Wrapper& hash, const CacheC key.meta.type = context.cache_type; key.meta.expiration_time = context.expiration_time; key.meta.tablet_id = context.tablet_id; +key.meta.table_name = context.table_name; Review Comment: [P2] Define attribution for cache blocks shared by external tables External cache identity is derived only from `path + mtime`, so two catalog aliases or overlapping external-table locations that reference the same object deliberately share this hash and offsets. Persisting one table/partition pair here makes `FILE_CACHE_INFO` report whichever logical table reads the file first; the other live owner cannot be represented, and reversing cold-read order changes the result. Please define shared attribution explicitly (association records, logical identity in the key, or a shared/ambiguous marker) and test two table aliases over the same file in both read orders. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5067183817 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5067183037 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5060932845 ClickBench: Total hot run time: 25.57 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 5d7fba72569f0abf6a9ee83e2737d98edae3, data reload: false query1 0.000.000.01 query2 0.140.080.08 query3 0.360.240.24 query4 1.610.240.24 query5 0.330.310.32 query6 1.160.670.67 query7 0.040.010.00 query8 0.090.070.07 query9 0.510.370.37 query10 0.580.590.59 query11 0.310.180.17 query12 0.310.200.18 query13 0.530.540.53 query14 0.930.930.93 query15 0.670.590.60 query16 0.380.390.38 query17 1.031.021.11 query18 0.320.300.31 query19 1.901.761.84 query20 0.020.010.02 query21 15.41 0.370.32 query22 4.820.130.14 query23 15.87 0.510.30 query24 2.490.630.43 query25 0.160.110.09 query26 0.740.270.22 query27 0.100.100.10 query28 3.430.930.52 query29 12.46 4.313.34 query30 0.390.270.26 query31 2.770.630.32 query32 3.240.610.48 query33 2.903.042.96 query34 15.65 3.983.34 query35 3.303.273.24 query36 0.650.520.51 query37 0.120.100.09 query38 0.090.070.07 query39 0.070.060.06 query40 0.200.180.15 query41 0.130.070.08 query42 0.080.060.06 query43 0.080.070.07 Total cold run time: 96.37 s Total hot run time: 25.57 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5060882738 TPC-DS: Total hot run time: 177611 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 5d7fba72569f0abf6a9ee83e2737d98edae3, data reload: false query5 4327634 513 513 query6 454 234 216 216 query7 4854579 354 354 query8 335 186 168 168 query9 8782407840574057 query10 481 371 325 325 query11 5919230621762176 query12 161 103 101 101 query13 1342636 421 421 query14 6266519448954895 query14_14268425642394239 query15 211 201 189 189 query16 1052493 446 446 query17 1126711 576 576 query18 2562481 354 354 query19 229 198 158 158 query20 115 112 111 111 query21 238 160 137 137 query22 13608 13535 13278 13278 query23 17277 16475 16125 16125 query23_116213 16267 16215 16215 query24 7511179212811281 query24_11283130212851285 query25 565 468 410 410 query26 1347372 217 217 query27 2580634 405 405 query28 4435204420382038 query29 1086653 494 494 query30 343 265 232 232 query31 11611086968 968 query32 102 65 68 65 query33 517 296 243 243 query34 11461137639 639 query35 767 770 662 662 query36 1200118610951095 query37 152 110 88 88 query38 1877170816541654 query39 874 871 855 855 query39_1850 843 826 826 query40 245 161 144 144 query41 66 62 67 62 query42 93 91 91 91 query43 321 323 277 277 query44 1433778 773 773 query45 196 186 172 172 query46 10491222759 759 query47 2129209820052005 query48 394 425 288 288 query49 585 426 302 302 query50 1076453 343 343 query51 10568 10695 10782 10695 query52 86 93 75 75 query53 257 273 200 200 query54 301 224 231 224 query55 74 72 67 67 query56 294 298 299 298 query57 1306129711871187 query58 285 276 263 263 query59 1623165514871487 query60 302 272 250 250 query61 157 150 154 150 query62 553 503 431 431 query63 246 205 197 197 query64 27861031829 829 query65 4695464346744643 query66 1794505 372 372 query67 29228 29307 29037 29037 query68 3227161310431043 query69 414 290 308 290 query70 1091968 956 956 query71 390 359 315 315 query72 3045267423502350 query73 812 803 422 422 query74 5046491747184718 query75 2552252721552155 query76 23101174763 763 query77 359 386 289 289 query78 11881 11837 11324 11324 query79 13871173763 763 query80 1286559 473 473 query81 560 339 291 291 query82 617 153 118 118 query83 372 320 306 306 query84 284 157 132 132 query85 987 630 511 511 query86 437 296 286 286 query87 1827182317601760 query88 3704278527942785 query89 453 372 335 335 query90 2000200 186 186 query91 204 193 159 159 query92 62 62 58 58 query93 16851557986 986 query94 704 364 309 309 query95 803 511 484 484 query96 1047840 344 344 query97 2636261525122512 query98 213 230 205 205 query99 10861119969 969 Total cold run time: 263628 ms Total hot run time: 177611 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5060776543 TPC-H: Total hot run time: 29756 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 5d7fba72569f0abf6a9ee83e2737d98edae3, data reload: false -- Round 1 -- q1 17765 426042644260 q2 1999327 199 199 q3 10839 1411846 846 q4 4732469 350 350 q5 8197847 576 576 q6 309 185 148 148 q7 806 833 607 607 q8 10526 156416041564 q9 5747437743044304 q10 6879174814661466 q11 527 368 328 328 q12 732 591 458 458 q13 18118 335027322732 q14 258 262 241 241 q15 q16 800 785 710 710 q17 1099938 984 938 q18 6866576455565556 q19 1571131310901090 q20 770 660 593 593 q21 5667269824942494 q22 438 359 296 296 Total cold run time: 104645 ms Total hot run time: 29756 ms - Round 2, with runtime_filter_mode=off - q1 4602446044324432 q2 283 325 217 217 q3 4619500044394439 q4 2080219513671367 q5 4413432547684325 q6 279 213 153 153 q7 2030182815821582 q8 2638230322782278 q9 7917774978217749 q10 4657461244284428 q11 612 467 451 451 q12 747 796 561 561 q13 3289367729362936 q14 310 301 274 274 q15 q16 750 742 647 647 q17 1420138213821382 q18 8170748870497049 q19 1103107610951076 q20 2206222019431943 q21 5336471244954495 q22 534 457 430 430 Total cold run time: 57995 ms Total hot run time: 52214 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
gavinchou commented on code in PR #61518:
URL: https://github.com/apache/doris/pull/61518#discussion_r3638085183
##
gensrc/proto/file_cache.proto:
##
@@ -59,5 +59,7 @@ message BlockMetaPb {
optional FileCacheType type = 1;
optional uint64 size = 2;
optional uint64 ttl = 3;
+optional uint64 context_id = 4;
Review Comment:
how to generate context_id?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
github-actions[bot] commented on code in PR #61518:
URL: https://github.com/apache/doris/pull/61518#discussion_r3637906466
##
be/src/cloud/cloud_internal_service.cpp:
##
@@ -1230,7 +1236,9 @@ void
CloudInternalServiceImpl::warm_up_rowset(google::protobuf::RpcController* c
.ctx = {.is_index_data = false,
.expiration_time = expiration_time,
.is_dryrun =
config::enable_reader_dryrun_when_download_file_cache,
-.is_warmup = true},
+.is_warmup = true,
+.table_name = "",
Review Comment:
[P2] Attribute blocks created by local warmup
This and the sibling sync/periodic warmup and index-preload paths perform
real cache fills with both new fields empty even though the producer already
knows the tablet ID. `FILE_CACHE_INFO` emits only the persisted name strings,
and `add_cell()` records context only when the cell is created, so a later
named OLAP scan merely hits the anonymous cell and cannot repair it. This is
distinct from the peer/hot-redistribution threads: these local paths start
without context. Please propagate stable identity from the known tablet, or
safely late-bind the first named hit, and cover warm-before-query rows in
`FILE_CACHE_INFO`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5057822008 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5057820977 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
github-actions[bot] commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5032453425 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
github-actions[bot] commented on code in PR #61518:
URL: https://github.com/apache/doris/pull/61518#discussion_r3620702685
##
fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java:
##
@@ -563,19 +576,16 @@ private TFileRangeDesc createFileRangeDesc(FileSplit
fileSplit, List col
// fileSize only be used when format is orc or parquet and TFileType
is broker
// When TFileType is other type, it is not necessary
rangeDesc.setFileSize(fileSplit.getFileLength());
-if (!columnsFromPathKeys.isEmpty()) {
-rangeDesc.setColumnsFromPath(columnsFromPath);
-rangeDesc.setColumnsFromPathKeys(columnsFromPathKeys);
-rangeDesc.setColumnsFromPathIsNull(columnsFromPathIsNull);
-}
-
rangeDesc.setFileType(fileSplit.getLocationType());
rangeDesc.setPath(fileSplit.getPath().toStorageLocation().toString());
if (fileSplit.getLocationType() == TFileType.FILE_HDFS) {
URI fileUri = fileSplit.getPath().getPath().toUri();
rangeDesc.setFsName(fileUri.getScheme() + "://" +
fileUri.getAuthority());
}
rangeDesc.setModificationTime(fileSplit.getModificationTime());
+List partitionKeyValues =
+FileScanNode.buildPartitionKeyValues(columnsFromPathKeys,
columnsFromPath, columnsFromPathIsNull);
+FileScanNode.fillPathPartitionContext(rangeDesc, desc.getTable(),
partitionKeyValues);
Review Comment:
[P2] Refresh cache context after connector setup
This context is built before `setScanParams()`. Plugin-driven Paimon adds
its partition fields only in that later hook, so partitioned files reach both
scanners with a table-only context; the first cold fill persists it and later
hits cannot repair it. Reconstructing generically from the legacy path fields
is unsafe because mixed-spec Iceberg intentionally uses those fields only for
partial row materialization. Please populate or refresh the structured context
after connector-specific setup (or in the Paimon hook), keep intentional
omission distinguishable from legacy absence, and cover Paimon plus mixed-spec
Iceberg.
##
be/src/io/cache/cache_block_meta_store.cpp:
##
@@ -110,16 +203,25 @@ Status CacheBlockMetaStore::init() {
}
_db.reset(db_ptr);
-// Store the file_cache_meta column family handle
-// handles[0] is default column family, handles[1] is file_cache_meta
+// handles[0] is default column family, handles[1] is file_cache_meta.
+// Store context dictionary records in the default family to keep
downgrade-open compatibility.
if (handles.size() >= 2) {
+_context_dict_cf_handle.reset(handles[0]);
_file_cache_meta_cf_handle.reset(handles[1]);
-// Close default column family handle as we won't use it
-_db->DestroyColumnFamilyHandle(handles[0]);
+if (has_legacy_context_cf && handles.size() >= 3) {
+std::unique_ptr
legacy_context_cf_handle(handles[2]);
+
RETURN_IF_ERROR(_migrate_legacy_context_dict(legacy_context_cf_handle.get()));
+const auto drop_status =
_db->DropColumnFamily(legacy_context_cf_handle.get());
+if (!drop_status.ok()) {
+LOG(WARNING) << "Failed to drop legacy context dictionary
column family: "
+ << drop_status.ToString();
+}
+}
} else {
-return Status::InternalError("Failed to get file_cache_meta column
family handle");
+return Status::InternalError("Failed to get default/file_cache_meta
column family handles");
}
+RETURN_IF_ERROR(_load_next_context_id());
Review Comment:
[P2] Do not expose a partially initialized meta store
At this point the RocksDB and column-family handles are already installed,
but migration or `_load_next_context_id()` can still return before the write
worker starts. The constructor only logs that failure and
`FSFileCacheStorage::init()` continues, so later fills can allocate from the
still-default ID 1 and overwrite an association referenced by existing block
metadata; every `put()` also queues work with no consumer. Please propagate the
initialization failure (or tear down and retry before exposing the store), and
add a fault-injection reopen test that proves IDs are not reused and the worker
is running.
##
be/src/io/cache/block_file_cache_downloader.cpp:
##
@@ -280,6 +280,8 @@ void FileCacheBlockDownloader::download_file_cache_block(
.expiration_time = meta.expiration_time(),
.is_dryrun =
config::enable_reader_dryrun_when_download_file_cache,
.is_warmup = true,
+.table_name = "",
Review Comment:
[P2] Preserve context during hot-block redistribution
The source block can already have a valid table/partition
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5031274990 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5031154937 # BE Regression && UT Coverage Report Increment line coverage `77.06% (420/545)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_6bd1126e65e25e43f488367bc2d521f44d815978_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_6bd1126e65e25e43f488367bc2d521f44d815978_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 75.08% (30570/40718) | | Line Coverage | 59.23% (337321/569503) | | Region Coverage | 55.85% (282457/505729) | | Branch Coverage | 57.21% (125549/219453) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5030516751 ClickBench: Total hot run time: 25.95 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 6bd1126e65e25e43f488367bc2d521f44d815978, data reload: false query1 0.010.010.03 query2 0.140.090.08 query3 0.360.240.24 query4 1.600.250.23 query5 0.330.320.33 query6 1.160.680.68 query7 0.040.010.01 query8 0.100.070.07 query9 0.520.390.39 query10 0.620.620.64 query11 0.300.190.17 query12 0.320.180.19 query13 0.520.540.54 query14 0.940.930.94 query15 0.670.610.61 query16 0.400.390.40 query17 1.021.031.05 query18 0.310.310.30 query19 1.961.831.82 query20 0.020.020.01 query21 15.45 0.380.33 query22 4.870.140.13 query23 15.79 0.510.31 query24 2.480.610.44 query25 0.160.120.10 query26 0.740.270.21 query27 0.110.100.10 query28 3.360.890.55 query29 12.48 4.223.33 query30 0.370.250.27 query31 2.770.600.33 query32 3.240.610.48 query33 3.083.063.01 query34 15.71 4.053.38 query35 3.293.273.28 query36 0.650.540.51 query37 0.120.100.09 query38 0.080.070.06 query39 0.070.060.06 query40 0.190.170.17 query41 0.130.080.09 query42 0.090.060.06 query43 0.080.070.07 Total cold run time: 96.65 s Total hot run time: 25.95 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5030485549 TPC-DS: Total hot run time: 178035 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 6bd1126e65e25e43f488367bc2d521f44d815978, data reload: false query5 4354636 495 495 query6 450 234 216 216 query7 4882588 336 336 query8 327 186 175 175 query9 8786410741144107 query10 490 361 351 351 query11 5939235221292129 query12 161 105 102 102 query13 1276584 443 443 query14 6705517448974897 query14_14260426042414241 query15 223 209 183 183 query16 1019487 449 449 query17 1151717 579 579 query18 2723481 359 359 query19 205 191 153 153 query20 111 111 109 109 query21 233 158 139 139 query22 13642 13565 13397 13397 query23 17273 16460 16127 16127 query23_116252 16237 16285 16237 query24 7469177413141314 query24_11295130613071306 query25 569 462 392 392 query26 1387339 221 221 query27 2472587 396 396 query28 4357203420112011 query29 1134634 525 525 query30 351 259 229 229 query31 11211097989 989 query32 113 65 63 63 query33 543 330 272 272 query34 12011114658 658 query35 765 783 668 668 query36 1188119310691069 query37 152 110 135 110 query38 1877168516711671 query39 855 861 844 844 query39_1830 841 830 830 query40 249 161 139 139 query41 66 61 64 61 query42 94 91 93 91 query43 336 336 289 289 query44 1425789 774 774 query45 198 187 174 174 query46 10791196706 706 query47 2056206619931993 query48 410 429 297 297 query49 591 417 301 301 query50 1137429 340 340 query51 10580 10643 10540 10540 query52 85 87 75 75 query53 268 274 203 203 query54 282 238 219 219 query55 77 72 69 69 query56 292 298 299 298 query57 1314131212051205 query58 297 264 251 251 query59 1576168014611461 query60 321 278 264 264 query61 169 147 151 147 query62 547 493 434 434 query63 247 206 199 199 query64 2738999 844 844 query65 4730463546304630 query66 1777510 383 383 query67 29275 29214 29056 29056 query68 3174151010221022 query69 414 297 269 269 query70 10551011968 968 query71 411 373 316 316 query72 2973262823452345 query73 859 834 481 481 query74 5084491647124712 query75 2538249421222122 query76 23311170803 803 query77 365 394 299 299 query78 11755 11879 11262 11262 query79 13781145765 765 query80 1310557 481 481 query81 523 334 291 291 query82 553 164 123 123 query83 369 333 304 304 query84 283 209 131 131 query85 968 614 513 513 query86 433 324 277 277 query87 1821181517731773 query88 3717281128082808 query89 445 388 329 329 query90 1934205 203 203 query91 203 195 167 167 query92 64 64 58 58 query93 16111519974 974 query94 728 360 308 308 query95 778 607 476 476 query96 1065818 352 352 query97 2632262124792479 query98 231 214 207 207 query99 10811126980 980 Total cold run time: 263546 ms Total hot run time: 178035 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5030425904 TPC-H: Total hot run time: 29600 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 6bd1126e65e25e43f488367bc2d521f44d815978, data reload: false -- Round 1 -- q1 17710 418240724072 q2 2022321 199 199 q3 10231 1377843 843 q4 4680483 342 342 q5 7520884 582 582 q6 186 171 136 136 q7 774 831 610 610 q8 9338157715591559 q9 6019438143224322 q10 6811172714551455 q11 511 356 327 327 q12 759 583 460 460 q13 18142 331227302730 q14 268 262 252 252 q15 q16 797 776 715 715 q17 1054997 1100997 q18 7026583655235523 q19 1569135711261126 q20 793 706 607 607 q21 5896259924452445 q22 433 354 298 298 Total cold run time: 102539 ms Total hot run time: 29600 ms - Round 2, with runtime_filter_mode=off - q1 4436435043694350 q2 292 343 211 211 q3 4637499943954395 q4 2068213513751375 q5 4428425042674250 q6 230 174 129 129 q7 1761209715481548 q8 2575224321392139 q9 7917788577307730 q10 4685468942264226 q11 558 420 391 391 q12 736 862 646 646 q13 3361365930633063 q14 316 328 289 289 q15 q16 721 731 640 640 q17 1354134313331333 q18 8093736469236923 q19 1134106310741063 q20 2224224319541954 q21 5242457644664466 q22 527 466 409 409 Total cold run time: 57295 ms Total hot run time: 51530 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5030349263 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/6bd1126e65e25e43f488367bc2d521f44d815978_6bd1126e65e25e43f488367bc2d521f44d815978_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/6bd1126e65e25e43f488367bc2d521f44d815978_6bd1126e65e25e43f488367bc2d521f44d815978_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 77.62% (1908/2458) | | Line Coverage | 64.51% (34168/52968) | | Region Coverage | 64.94% (17582/27076) | | Branch Coverage | 54.07% (9421/17424) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
github-actions[bot] commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5030204900 Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 25th, 2026 6:03 AM. Workflow run: https://github.com/apache/doris/actions/runs/29801320698 Please inspect the workflow logs and rerun the review after the underlying issue is resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5030155214 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5030154524 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5029644219 run beut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5024404837 ClickBench: Total hot run time: 25.23 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 214884ec13606628cd7e544f4ea2ff76c589ef66, data reload: false query1 0.000.000.00 query2 0.100.050.05 query3 0.260.140.13 query4 1.610.130.14 query5 0.240.260.23 query6 1.231.031.08 query7 0.040.010.01 query8 0.060.040.03 query9 0.400.320.32 query10 0.570.600.57 query11 0.190.140.14 query12 0.200.150.14 query13 0.480.480.48 query14 1.001.011.02 query15 0.610.590.61 query16 0.350.340.33 query17 1.141.111.15 query18 0.240.210.20 query19 2.001.982.00 query20 0.020.010.01 query21 15.43 0.200.13 query22 4.970.050.05 query23 16.12 0.300.11 query24 3.010.410.36 query25 0.100.040.05 query26 0.750.200.16 query27 0.040.040.03 query28 3.490.910.54 query29 12.49 4.133.29 query30 0.270.150.16 query31 2.770.600.32 query32 3.230.600.49 query33 3.233.213.23 query34 15.75 4.213.54 query35 3.513.513.55 query36 0.550.440.44 query37 0.090.070.07 query38 0.050.050.03 query39 0.040.030.03 query40 0.180.170.15 query41 0.080.030.04 query42 0.030.030.03 query43 0.040.030.03 Total cold run time: 96.96 s Total hot run time: 25.23 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5024356180 TPC-DS: Total hot run time: 177427 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 214884ec13606628cd7e544f4ea2ff76c589ef66, data reload: false query5 4316634 472 472 query6 463 220 214 214 query7 4839597 339 339 query8 348 191 168 168 query9 8774405340564053 query10 480 358 305 305 query11 5948233321152115 query12 154 101 100 100 query13 1232619 420 420 query14 6240519048634863 query14_14260423642504236 query15 207 202 175 175 query16 1018510 446 446 query17 923 694 580 580 query18 2439480 362 362 query19 215 196 162 162 query20 108 108 117 108 query21 238 160 137 137 query22 13545 13623 13317 13317 query23 17428 16410 16127 16127 query23_116222 16108 16249 16108 query24 7682178512881288 query24_11316129612881288 query25 577 467 389 389 query26 1335353 216 216 query27 2704632 382 382 query28 4509203819921992 query29 1082646 494 494 query30 352 262 225 225 query31 11161100991 991 query32 112 66 70 66 query33 539 337 259 259 query34 11841174640 640 query35 774 795 690 690 query36 1193118710011001 query37 156 120 102 102 query38 1879171216881688 query39 879 887 857 857 query39_1837 848 831 831 query40 252 174 148 148 query41 74 70 69 69 query42 98 94 92 92 query43 320 329 280 280 query44 1427795 775 775 query45 199 181 179 179 query46 10821197725 725 query47 2112209419951995 query48 426 408 304 304 query49 594 424 316 316 query50 1120446 339 339 query51 10941 10499 10647 10499 query52 88 88 78 78 query53 265 289 202 202 query54 303 245 237 237 query55 77 76 71 71 query56 324 319 299 299 query57 1307126911611161 query58 287 283 307 283 query59 1591161814281428 query60 302 276 253 253 query61 153 151 143 143 query62 548 491 432 432 query63 253 201 197 197 query64 28431041868 868 query65 4728461946704619 query66 1830513 389 389 query67 29269 29292 29053 29053 query68 3192166510611061 query69 395 303 274 274 query70 10531005951 951 query71 370 355 332 332 query72 3118263522402240 query73 807 800 454 454 query74 5042491747064706 query75 2520249721462146 query76 23421151801 801 query77 350 376 287 287 query78 11906 11814 11398 11398 query79 14091219786 786 query80 1323560 475 475 query81 543 344 287 287 query82 646 155 118 118 query83 382 325 300 300 query84 325 161 137 137 query85 995 613 536 536 query86 438 310 273 273 query87 1822182417641764 query88 3728281327992799 query89 437 382 336 336 query90 1930202 200 200 query91 197 195 171 171 query92 61 60 57 57 query93 1637148110711071 query94 727 343 306 306 query95 806 524 583 524 query96 1025778 340 340 query97 2641265325182518 query98 218 214 201 201 query99 10961108976 976 Total cold run time: 264155 ms Total hot run time: 177427 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5024247050 TPC-H: Total hot run time: 29657 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 214884ec13606628cd7e544f4ea2ff76c589ef66, data reload: false -- Round 1 -- q1 17695 414841074107 q2 2015342 200 200 q3 10342 1422809 809 q4 4752477 341 341 q5 8209897 575 575 q6 343 176 141 141 q7 872 845 634 634 q8 10651 169116321632 q9 5836439043944390 q10 6761174514511451 q11 519 357 323 323 q12 750 572 455 455 q13 18086 341828242824 q14 270 271 247 247 q15 q16 797 777 718 718 q17 10481035924 924 q18 6928583155865586 q19 1186124010441044 q20 835 714 604 604 q21 5425262923512351 q22 428 356 301 301 Total cold run time: 103748 ms Total hot run time: 29657 ms - Round 2, with runtime_filter_mode=off - q1 4506436144194361 q2 281 311 211 211 q3 4636501944334433 q4 2071219313541354 q5 4439429342784278 q6 236 177 133 133 q7 2478190016881688 q8 2539224021992199 q9 8212757078087570 q10 4696464341834183 q11 568 422 416 416 q12 744 775 546 546 q13 3334363528842884 q14 283 320 282 282 q15 q16 692 729 646 646 q17 1376133914751339 q18 8105750873397339 q19 1184106710871067 q20 2217222319071907 q21 5296461445074507 q22 520 453 406 406 Total cold run time: 58413 ms Total hot run time: 51749 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5023912972 # FE UT Coverage Report Increment line coverage ` 60.77% (127/209)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_214884ec13606628cd7e544f4ea2ff76c589ef66/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_214884ec13606628cd7e544f4ea2ff76c589ef66/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
github-actions[bot] commented on code in PR #61518:
URL: https://github.com/apache/doris/pull/61518#discussion_r3615223399
##
be/src/storage/id_manager.h:
##
@@ -82,15 +82,17 @@ struct ExternalFileMappingInfo {
* 1. If the file belongs to a partition, columns_from_path_keys and
columns_from_path in TFileRangeDesc are needed when materializing the partition
column
* 2. path, file_type, modification_time,compress_type used to read
the file
* 3. TFileFormatType can distinguish whether it is
iceberg/hive/hudi/paimon
- */
+*/
TFileRangeDesc scan_range_desc;
bool enable_file_meta_cache;
+std::string table_name;
ExternalFileMappingInfo(int plan_node_id, const TFileRangeDesc& scan_range,
-bool file_meta_cache)
+bool file_meta_cache, std::string table_name_)
Review Comment:
[P1] Update the remaining three-argument callers
This makes `table_name` mandatory, but `parquet_read_lines.cpp:223` and
`orc_read_lines.cpp:201` still construct `ExternalFileMappingInfo` with three
arguments, and `id_manager_test.cpp:49` still constructs the external
`FileMapping` overload with three. All three sources are included by the
recursive `doris_be_test` glob, so that target cannot compile. Please update
those callers (and assert the restored context) or retain a deliberate
compatible overload/default.
##
be/src/exec/scan/file_scanner_v2.cpp:
##
@@ -817,9 +819,41 @@ Status
FileScannerV2::_to_file_format(TFileFormatType::type format_type,
Status FileScannerV2::_init_io_ctx() {
_io_ctx = create_file_scan_io_context(_state);
+if (_local_state) {
+_io_ctx->table_name =
_local_state->cast().table_name();
Review Comment:
[P2] Pass this context to V2 Iceberg delete readers
This names the V2 data-file IOContext, but both position- and
equality-delete initialization create a fresh `IcebergDeleteFileIOContext` from
`RuntimeState` only. That helper never copies these names before its
Parquet/ORC reader opens the delete file, so cold delete-file blocks remain
anonymous while the associated data file is attributed. Seed the delete-file
context from the current table-reader IOContext (at least preserving the
unambiguous table name) and cover both delete types.
##
be/src/storage/id_manager.h:
##
@@ -82,15 +82,17 @@ struct ExternalFileMappingInfo {
* 1. If the file belongs to a partition, columns_from_path_keys and
columns_from_path in TFileRangeDesc are needed when materializing the partition
column
* 2. path, file_type, modification_time,compress_type used to read
the file
* 3. TFileFormatType can distinguish whether it is
iceberg/hive/hudi/paimon
- */
+*/
TFileRangeDesc scan_range_desc;
bool enable_file_meta_cache;
+std::string table_name;
Review Comment:
[P2] Preserve context for internal row-ID rereads too
The latest fix carries the table only in `ExternalFileMappingInfo`;
`InternalFileMappingInfo` still records just tablet/rowset/segment.
`SegmentIterator` emits that mapping for global row IDs, and
`RowIdStorageReader::read_doris_format_row()` later loads the segment and
creates fresh row-store/column-store IOContexts without names. A TopN
lazy-materialization miss on a second-phase-only page is therefore persisted
with `context_id = 0`. Please carry the scan context through the internal
mapping and restore it in each second-phase IOContext.
##
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java:
##
@@ -776,6 +776,40 @@ public static Map
getPartitionInfoMap(PartitionData partitionDat
return partitionInfoMap;
}
+public static Map
getIdentityPartitionInfoMapForCache(PartitionData partitionData,
+PartitionSpec partitionSpec, String timeZone) {
+Map partitionInfoMap = Maps.newLinkedHashMap();
+List fields =
partitionData.getPartitionType().asNestedType().fields();
+List partitionFields = partitionSpec.fields();
+Preconditions.checkArgument(fields.size() == partitionFields.size(),
+"PartitionData fields size does not match PartitionSpec fields
size");
+
+for (int i = 0; i < fields.size(); i++) {
+NestedField field = fields.get(i);
+PartitionField partitionField = partitionFields.get(i);
+if (!partitionField.transform().isIdentity()) {
Review Comment:
[P1] Do not drop identity values in mixed specs
For a spec such as `identity(dt), day(ts)`, `getIdentityPartitionColumns()`
still marks `dt` as a path-only/non-file slot, but this return discards the
`dt` value when the loop reaches `day(ts)`. `createIcebergSplit()` then keeps
only an empty accounting sentinel, and `setIcebergParams()` unsets the generic
path values without restoring `dt`, leaving the scan with no source for that
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5022926517 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5022925653 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5002555235 ClickBench: Total hot run time: 25.73 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 7420e233c896156426d48c937a26f9106b18c80d, data reload: false query1 0.010.010.00 query2 0.140.080.09 query3 0.370.240.23 query4 1.610.260.25 query5 0.340.320.31 query6 1.160.680.67 query7 0.040.010.00 query8 0.090.080.07 query9 0.520.380.39 query10 0.600.590.58 query11 0.300.180.19 query12 0.330.180.18 query13 0.530.530.54 query14 0.940.940.93 query15 0.690.610.60 query16 0.390.400.39 query17 1.030.970.98 query18 0.310.280.29 query19 1.921.831.76 query20 0.030.020.01 query21 15.47 0.400.32 query22 4.750.140.14 query23 15.80 0.510.30 query24 2.520.600.44 query25 0.160.110.11 query26 0.760.270.21 query27 0.100.100.09 query28 3.430.930.55 query29 12.46 4.323.33 query30 0.380.280.26 query31 2.770.630.32 query32 3.230.600.49 query33 3.032.973.18 query34 15.68 4.053.40 query35 3.303.293.32 query36 0.650.530.52 query37 0.130.100.09 query38 0.080.060.06 query39 0.080.070.06 query40 0.200.190.17 query41 0.130.090.09 query42 0.090.060.06 query43 0.070.070.06 Total cold run time: 96.62 s Total hot run time: 25.73 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5002502591 TPC-DS: Total hot run time: 178132 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 7420e233c896156426d48c937a26f9106b18c80d, data reload: false query5 4305635 521 521 query6 456 219 224 219 query7 4871612 358 358 query8 348 198 188 188 query9 8787412941194119 query10 488 368 317 317 query11 5871234621632163 query12 152 108 103 103 query13 1262612 453 453 query14 6268519649514951 query14_14277425842494249 query15 223 205 184 184 query16 1007466 470 466 query17 1129703 590 590 query18 2518484 359 359 query19 216 204 157 157 query20 112 110 111 110 query21 229 166 140 140 query22 13763 13637 13382 13382 query23 17455 16524 16123 16123 query23_116308 16325 16292 16292 query24 7545178012911291 query24_11291132212821282 query25 581 485 359 359 query26 1334356 212 212 query27 2580619 382 382 query28 4431201420192014 query29 1074610 486 486 query30 346 270 228 228 query31 11221095983 983 query32 111 62 63 62 query33 546 316 250 250 query34 11871136669 669 query35 758 807 669 669 query36 1205119910471047 query37 155 103 92 92 query38 1881170516491649 query39 869 866 848 848 query39_1852 843 857 843 query40 244 159 138 138 query41 66 63 64 63 query42 95 92 91 91 query43 333 341 290 290 query44 1485789 783 783 query45 205 183 175 175 query46 10981211705 705 query47 2136208120222022 query48 394 384 299 299 query49 582 416 301 301 query50 1066433 339 339 query51 10704 10698 10593 10593 query52 88 87 73 73 query53 260 295 198 198 query54 285 239 225 225 query55 76 75 68 68 query56 287 308 287 287 query57 1300126111641164 query58 277 276 266 266 query59 1603167514691469 query60 302 278 251 251 query61 149 149 142 142 query62 544 493 438 438 query63 250 207 209 207 query64 29351021835 835 query65 4743458146464581 query66 1821504 372 372 query67 29449 29372 29230 29230 query68 31351598948 948 query69 406 309 268 268 query70 1077971 930 930 query71 377 341 303 303 query72 3180271723152315 query73 843 744 393 393 query74 5116495847754775 query75 2567250821532153 query76 23391216780 780 query77 398 389 285 285 query78 11988 11869 11390 11390 query79 14041231744 744 query80 1011565 482 482 query81 505 334 281 281 query82 571 154 119 119 query83 391 322 304 304 query84 280 158 126 126 query85 989 620 550 550 query86 412 285 283 283 query87 1830182717541754 query88 3835282527912791 query89 441 378 337 337 query90 1774201 194 194 query91 200 195 166 166 query92 62 61 56 56 query93 16591594946 946 query94 621 368 312 312 query95 800 478 561 478 query96 1103820 368 368 query97 2598263524882488 query98 216 229 204 204 query99 11211101995 995 Total cold run time: 264328 ms Total hot run time: 178132 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5002388970 TPC-H: Total hot run time: 29625 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 7420e233c896156426d48c937a26f9106b18c80d, data reload: false -- Round 1 -- q1 17871 413941514139 q2 2040324 206 206 q3 10862 1428822 822 q4 4758476 334 334 q5 8357843 569 569 q6 330 171 137 137 q7 833 831 587 587 q8 10546 161615281528 q9 5672437143404340 q10 6801172414681468 q11 499 354 318 318 q12 751 575 460 460 q13 18088 329027242724 q14 265 261 238 238 q15 q16 785 775 703 703 q17 10241009972 972 q18 6912575657165716 q19 1277126210431043 q20 806 705 634 634 q21 5626259323942394 q22 439 357 293 293 Total cold run time: 104542 ms Total hot run time: 29625 ms - Round 2, with runtime_filter_mode=off - q1 4454432743224322 q2 293 318 225 225 q3 4588501243814381 q4 2056214913621362 q5 4389429842214221 q6 228 180 127 127 q7 2292189016121612 q8 2467214421162116 q9 7784777177547754 q10 4688467142544254 q11 772 411 391 391 q12 765 756 537 537 q13 3254362229342934 q14 310 310 282 282 q15 q16 705 737 632 632 q17 1363132913331329 q18 7957764768986898 q19 1071108910731073 q20 2252221719411941 q21 5220454443594359 q22 522 463 440 440 Total cold run time: 57430 ms Total hot run time: 51190 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-5000808950 # BE UT Coverage Report Increment line coverage `61.81% (259/419)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/7420e233c896156426d48c937a26f9106b18c80d_7420e233c896156426d48c937a26f9106b18c80d/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/7420e233c896156426d48c937a26f9106b18c80d_7420e233c896156426d48c937a26f9106b18c80d/report/index.html) | Category | Coverage | |---|| | Function Coverage | 57.43% (23942/41689) | | Line Coverage | 41.15% (235442/572106) | | Region Coverage | 36.98% (185979/502970) | | Branch Coverage | 38.15% (83474/218796) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
github-actions[bot] commented on code in PR #61518:
URL: https://github.com/apache/doris/pull/61518#discussion_r3600619383
##
be/src/exec/scan/parallel_scanner_builder.cpp:
##
@@ -296,6 +296,12 @@ Status ParallelScannerBuilder::_load() {
std::shared_ptr ParallelScannerBuilder::_build_scanner(
BaseTabletSPtr tablet, int64_t version, const
std::vector& key_ranges,
TabletReadSource&& read_source, io::FileCacheStatistics&&
initial_file_cache_stats) {
+std::string table_name;
+std::string partition_name;
+if (auto it = _tablet_contexts.find(tablet->tablet_id()); it !=
_tablet_contexts.end()) {
Review Comment:
This lookup happens only in `_build_scanner()`, after `build_scanners()` has
already run `_load()`. `_load()` calls `get_segment_num_rows()` with
`create_preload_io_context()`, which leaves both names empty; when row counts
must be read from segment footers, that path can create and persist file-cache
blocks with `context_id = 0`. Later scanner reads only reuse/touch the existing
cells and do not merge this context into them, so those footer blocks remain
anonymous. Please apply the tablet context to the preload `IOContext` before
`get_segment_num_rows()` and cover that cache-populating path.
##
be/src/io/cache/fs_file_cache_storage.cpp:
##
@@ -321,8 +322,16 @@ Status FSFileCacheStorage::finalize(const FileCacheKey&
key, const size_t size)
}
BlockMetaKey mkey(key.meta.tablet_id, UInt128Wrapper(key.hash),
key.offset);
-BlockMeta meta(key.meta.type, size, key.meta.expiration_time);
-_meta_store->put(mkey, meta);
+uint64_t context_id = 0;
+if (_meta_store) {
+// Table/partition context is only used for observability. Any
dictionary lookup/write
+// failure must degrade to an empty context id instead of affecting
cache writes.
+context_id =
+_meta_store->get_or_create_context_id(key.meta.table_name,
key.meta.partition_name);
Review Comment:
`finalize()` runs once per file-cache block, and
`get_or_create_context_id()` always issues a synchronous RocksDB `Get` even for
an existing pair. Because the default block size is 1 MiB and no context ID is
resolved before block splitting, a 1 TiB cold fill repeats roughly one million
reads for what is commonly one table/partition, while holding each block's
finalize path. Please intern/cache the forward mapping in memory or resolve the
context once upstream and propagate its ID to sibling blocks.
##
be/src/exec/scan/file_scanner.h:
##
@@ -310,6 +313,9 @@ class FileScanner : public Scanner {
Status _init_io_ctx() {
_io_ctx = create_file_scan_io_context(_state);
+if (_local_state) {
Review Comment:
This adds the table context only to FileScanner V1, but supported non-load
scans use FileScannerV2 by default (`enable_file_scanner_v2 = true`). V2's
`_init_io_ctx()` only calls `create_file_scan_io_context()`, and
`_prepare_next_split()` never copies `TFileRangeDesc.partition_name`, so
ordinary Parquet/ORC queries still reach the cache with both names empty and
persist `context_id = 0`. This is distinct from the existing V1 range-timing
thread. Please mirror both the table initialization and per-split partition
refresh in V2, with a two-partition V2 test.
##
be/src/io/cache/cache_block_meta_store.cpp:
##
@@ -97,6 +178,9 @@ Status CacheBlockMetaStore::init() {
column_families.emplace_back(rocksdb::kDefaultColumnFamilyName,
rocksdb::ColumnFamilyOptions());
// File cache meta column family
column_families.emplace_back(FILE_CACHE_META_COLUMN_FAMILY,
rocksdb::ColumnFamilyOptions());
+// File cache context dictionary column family
+column_families.emplace_back(FILE_CACHE_CONTEXT_DICT_COLUMN_FAMILY,
Review Comment:
Adding a persistent column family makes this local DB incompatible with
rolling back to the previous BE: RocksDB's descriptor-based `Open` requires
every existing family to be listed, while the old binary lists only `default`
and `file_cache_meta`. After the new BE creates `file_cache_context_dict`, the
old BE cannot reopen the metadata DB; initialization merely logs the failure
and continues with no working metadata store. Please keep the dictionary in an
existing family with namespaced keys (or otherwise preserve downgrade-open
compatibility) and add an upgrade-then-downgrade reopen test.
##
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java:
##
@@ -776,6 +777,41 @@ public static Map
getPartitionInfoMap(PartitionData partitionDat
return partitionInfoMap;
}
+public static Map
getIdentityPartitionInfoMapForCache(PartitionData partitionData,
+PartitionSpec partitionSpec, String timeZone) {
+Map partitionInfoMap = Maps.newLinkedHashMap();
+List fields =
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4999174778 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4999174291 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4934370013 # FE Regression Coverage Report Increment line coverage ` 32.57% (71/218)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_118f72fdd77676e70e9ccd1ad6b77ee81753415b_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_118f72fdd77676e70e9ccd1ad6b77ee81753415b_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4931806983 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/2cd1b2331e4031f84d7332d0cb4a94e7c7034e27_2cd1b2331e4031f84d7332d0cb4a94e7c7034e27_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/2cd1b2331e4031f84d7332d0cb4a94e7c7034e27_2cd1b2331e4031f84d7332d0cb4a94e7c7034e27_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 77.39% (1896/2450) | | Line Coverage | 64.45% (34071/52862) | | Region Coverage | 64.88% (17539/27032) | | Branch Coverage | 54.03% (9396/17390) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4931589628 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4931589624 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
github-actions[bot] commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4766594896 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4766389504 ClickBench: Total hot run time: 25.86 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 21f8facd4573775e4d880ea98dd07254de7dd2a4, data reload: false query1 0.010.010.01 query2 0.140.080.09 query3 0.380.260.24 query4 1.610.250.24 query5 0.340.310.31 query6 1.150.680.67 query7 0.040.010.00 query8 0.100.070.07 query9 0.500.390.39 query10 0.580.580.57 query11 0.310.190.19 query12 0.320.190.19 query13 0.520.530.53 query14 0.940.920.92 query15 0.670.600.61 query16 0.390.390.39 query17 1.021.001.00 query18 0.320.300.30 query19 1.971.871.78 query20 0.020.020.02 query21 15.40 0.390.31 query22 4.750.130.14 query23 15.83 0.490.31 query24 2.430.620.44 query25 0.150.110.10 query26 0.760.270.22 query27 0.100.100.11 query28 3.460.970.53 query29 12.49 4.403.52 query30 0.380.260.26 query31 2.770.620.32 query32 3.240.600.48 query33 2.952.943.04 query34 15.71 4.093.34 query35 3.283.253.26 query36 0.640.520.51 query37 0.130.090.10 query38 0.080.070.07 query39 0.070.060.07 query40 0.200.180.17 query41 0.140.080.08 query42 0.090.060.06 query43 0.080.070.07 Total cold run time: 96.46 s Total hot run time: 25.86 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4766352183 TPC-DS: Total hot run time: 173120 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 21f8facd4573775e4d880ea98dd07254de7dd2a4, data reload: false query5 4324636 490 490 query6 440 195 184 184 query7 4823550 310 310 query8 365 219 196 196 query9 8766404040374037 query10 451 318 261 261 query11 5965230521462146 query12 159 109 102 102 query13 1313593 455 455 query14 6390540550215021 query14_14400436544224365 query15 210 205 182 182 query16 991 503 417 417 query17 1146723 607 607 query18 2546485 358 358 query19 205 194 150 150 query20 117 111 110 110 query21 217 143 117 117 query22 13712 13574 13338 13338 query23 17344 16415 16238 16238 query23_116253 16368 16207 16207 query24 7563176513341334 query24_11299133613381336 query25 580 515 376 376 query26 1299320 168 168 query27 2669559 337 337 query28 4429205420592054 query29 1051612 470 470 query30 298 238 196 196 query31 11131071953 953 query32 109 63 58 58 query33 533 335 249 249 query34 11881187659 659 query35 752 804 697 697 query36 1388135012221222 query37 147 101 91 91 query38 1914169116581658 query39 919 926 882 882 query39_1880 881 874 874 query40 223 122 101 101 query41 74 63 61 61 query42 89 87 92 87 query43 320 325 292 292 query44 1420771 795 771 query45 189 185 180 180 query46 10871232734 734 query47 2369233622982298 query48 402 411 291 291 query49 626 497 357 357 query50 968 349 261 261 query51 4330435742044204 query52 83 87 70 70 query53 255 262 191 191 query54 264 210 211 210 query55 73 75 71 71 query56 253 230 212 212 query57 1445139813241324 query58 247 217 216 216 query59 1548165614161416 query60 276 243 225 225 query61 157 153 148 148 query62 703 648 598 598 query63 230 191 200 191 query64 2538799 626 626 query65 4891477147934771 query66 1788471 378 378 query67 29815 29727 29613 29613 query68 32731583953 953 query69 421 328 274 274 query70 1105989 978 978 query71 295 237 213 213 query72 2871265823602360 query73 876 785 440 440 query74 5133496648074807 query75 2642260922492249 query76 23351228774 774 query77 341 383 291 291 query78 12483 12405 11881 11881 query79 13941170792 792 query80 809 492 387 387 query81 488 282 239 239 query82 579 161 124 124 query83 348 277 247 247 query84 258 149 119 119 query85 922 512 420 420 query86 416 296 284 284 query87 1837184117721772 query88 3686279527632763 query89 435 379 337 337 query90 1775181 177 177 query91 174 161 136 136 query92 63 62 58 58 query93 15131561910 910 query94 618 361 302 302 query95 669 468 355 355 query96 1064801 365 365 query97 2706270825552555 query98 218 209 207 207 query99 1207113910501050 Total cold run time: 258208 ms Total hot run time: 173120 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4766269373 TPC-H: Total hot run time: 29329 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 21f8facd4573775e4d880ea98dd07254de7dd2a4, data reload: false -- Round 1 -- q1 17582 400140394001 q2 2013312 200 200 q3 10828 1427863 863 q4 4733471 337 337 q5 8177867 576 576 q6 319 183 142 142 q7 848 837 639 639 q8 10620 166416401640 q9 6039455445234523 q10 6831178615301530 q11 451 272 243 243 q12 639 424 290 290 q13 18119 333527412741 q14 276 257 239 239 q15 q16 793 774 708 708 q17 11411013991 991 q18 7015581555335533 q19 1837128911221122 q20 477 418 276 276 q21 6016266124402440 q22 453 362 295 295 Total cold run time: 105207 ms Total hot run time: 29329 ms - Round 2, with runtime_filter_mode=off - q1 4372426042764260 q2 328 352 228 228 q3 4556495144314431 q4 2077218113931393 q5 4450432843174317 q6 233 193 205 193 q7 2038188415871587 q8 2585210921552109 q9 8040780779137807 q10 4811475745004500 q11 581 415 384 384 q12 766 749 555 555 q13 3316360429432943 q14 313 305 276 276 q15 q16 715 740 634 634 q17 1340134413331333 q18 8035730570177017 q19 1150107711061077 q20 222519431943 q21 5731455244624462 q22 525 475 388 388 Total cold run time: 58187 ms Total hot run time: 51837 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4765274162 # FE Regression Coverage Report Increment line coverage ` 34.30% (71/207)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_21f8facd4573775e4d880ea98dd07254de7dd2a4_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_21f8facd4573775e4d880ea98dd07254de7dd2a4_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4765221146 # BE Regression && UT Coverage Report Increment line coverage `86.30% (378/438)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_21f8facd4573775e4d880ea98dd07254de7dd2a4_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_21f8facd4573775e4d880ea98dd07254de7dd2a4_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 74.03% (28387/38343) | | Line Coverage | 57.99% (309719/534054) | | Region Coverage | 54.83% (259439/473129) | | Branch Coverage | 56.12% (112571/200593) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4764820595 # FE UT Coverage Report Increment line coverage ` 56.04% (116/207)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_21f8facd4573775e4d880ea98dd07254de7dd2a4/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_21f8facd4573775e4d880ea98dd07254de7dd2a4/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4764678390 # BE UT Coverage Report Increment line coverage `61.14% (269/440)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/21f8facd4573775e4d880ea98dd07254de7dd2a4_21f8facd4573775e4d880ea98dd07254de7dd2a4/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/21f8facd4573775e4d880ea98dd07254de7dd2a4_21f8facd4573775e4d880ea98dd07254de7dd2a4/report/index.html) | Category | Coverage | |---|| | Function Coverage | 54.44% (21378/39269) | | Line Coverage | 38.06% (204462/537207) | | Region Coverage | 34.06% (160418/470988) | | Branch Coverage | 35.07% (70267/200340) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4764419506 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/21f8facd4573775e4d880ea98dd07254de7dd2a4_21f8facd4573775e4d880ea98dd07254de7dd2a4_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/21f8facd4573775e4d880ea98dd07254de7dd2a4_21f8facd4573775e4d880ea98dd07254de7dd2a4_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 77.32% (1889/2443) | | Line Coverage | 64.41% (33960/52726) | | Region Coverage | 64.77% (17453/26948) | | Branch Coverage | 53.93% (9338/17316) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4764256125 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4764256116 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4758445035 ClickBench: Total hot run time: 25.07 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 6658e9ff34d5556b6342b32543032c99cba86904, data reload: false query1 0.000.000.01 query2 0.100.050.05 query3 0.250.140.13 query4 1.610.130.13 query5 0.230.220.22 query6 1.261.091.06 query7 0.040.010.00 query8 0.060.040.04 query9 0.370.300.31 query10 0.550.570.54 query11 0.200.140.14 query12 0.190.140.17 query13 0.460.480.47 query14 1.021.000.99 query15 0.620.580.58 query16 0.320.330.32 query17 1.131.091.09 query18 0.220.210.21 query19 2.041.901.96 query20 0.020.010.01 query21 15.43 0.210.13 query22 4.910.060.05 query23 16.14 0.330.12 query24 2.980.430.27 query25 0.120.050.04 query26 0.860.210.15 query27 0.030.040.04 query28 3.440.960.54 query29 12.50 4.293.47 query30 0.270.140.14 query31 2.770.600.31 query32 3.220.600.48 query33 3.243.303.20 query34 15.58 4.243.53 query35 3.493.523.49 query36 0.550.450.44 query37 0.090.070.06 query38 0.050.030.04 query39 0.040.040.03 query40 0.180.160.15 query41 0.090.030.02 query42 0.030.040.03 query43 0.040.040.03 Total cold run time: 96.74 s Total hot run time: 25.07 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4758423513 TPC-DS: Total hot run time: 175652 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 6658e9ff34d5556b6342b32543032c99cba86904, data reload: false query5 4327624 485 485 query6 437 214 174 174 query7 4805577 306 306 query8 363 217 187 187 query9 8802417441414141 query10 449 319 261 261 query11 5911234921842184 query12 159 102 100 100 query13 1273628 436 436 query14 6411538050815081 query14_14439445844154415 query15 210 196 178 178 query16 1019446 469 446 query17 1141717 580 580 query18 2457489 352 352 query19 208 187 151 151 query20 113 109 106 106 query21 217 142 118 118 query22 13598 13594 13432 13432 query23 17402 16578 16217 16217 query23_116377 16326 16185 16185 query24 7305180113221322 query24_11329133713181318 query25 586 474 404 404 query26 1298337 179 179 query27 2654545 336 336 query28 4361206820422042 query29 1110671 514 514 query30 312 238 196 196 query31 11161082967 967 query32 114 64 67 64 query33 571 317 257 257 query34 11751204686 686 query35 756 794 662 662 query36 1361142712251225 query37 153 106 101 101 query38 3232315330493049 query39 945 914 885 885 query39_1886 865 899 865 query40 225 118 100 100 query41 64 65 60 60 query42 96 98 99 98 query43 335 326 290 290 query44 1501795 799 795 query45 195 188 175 175 query46 11041215740 740 query47 2401239622242224 query48 408 430 286 286 query49 608 448 345 345 query50 985 368 264 264 query51 4340428542744274 query52 92 89 79 79 query53 255 274 198 198 query54 278 219 205 205 query55 78 77 69 69 query56 234 225 218 218 query57 1431144213361336 query58 248 214 216 214 query59 1643162014561456 query60 282 235 239 235 query61 155 150 148 148 query62 704 651 581 581 query63 227 189 194 189 query64 2501748 629 629 query65 4855476247754762 query66 1778465 345 345 query67 29713 29791 29589 29589 query68 30691607882 882 query69 407 299 279 279 query70 1072994 942 942 query71 293 235 220 220 query72 2965268022782278 query73 855 797 452 452 query74 5150497747904790 query75 2642258922322232 query76 22111213791 791 query77 365 382 297 297 query78 12409 12545 11998 11998 query79 14621162740 740 query80 1269487 395 395 query81 526 280 243 243 query82 625 155 121 121 query83 311 286 251 251 query84 254 149 118 118 query85 878 515 419 419 query86 434 309 263 263 query87 3402335931953195 query88 3766279727882788 query89 431 386 337 337 query90 1922194 192 192 query91 175 168 132 132 query92 75 59 59 59 query93 17571442892 892 query94 709 352 298 298 query95 673 476 341 341 query96 1100826 357 357 query97 2716269725672567 query98 216 209 199 199 query99 1178113210481048 Total cold run time: 261834 ms Total hot run time: 175652 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4758378429 TPC-H: Total hot run time: 29448 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 6658e9ff34d5556b6342b32543032c99cba86904, data reload: false -- Round 1 -- q1 16858 410243384102 q2 1989321 187 187 q3 10063 1438844 844 q4 4675468 341 341 q5 7674846 580 580 q6 183 173 137 137 q7 763 852 610 610 q8 9435159016881590 q9 5864452145104510 q10 6810180515191519 q11 436 275 241 241 q12 626 429 297 297 q13 18111 331327932793 q14 260 264 238 238 q15 q16 773 770 704 704 q17 995 993 1080993 q18 7117573854895489 q19 1186121610791079 q20 512 416 249 249 q21 6180283326342634 q22 451 381 311 311 Total cold run time: 100961 ms Total hot run time: 29448 ms - Round 2, with runtime_filter_mode=off - q1 5321496648234823 q2 350 404 251 251 q3 4978522447124712 q4 2133217013851385 q5 4985468446354635 q6 238 175 133 133 q7 1961176315121512 q8 2441217221892172 q9 7919749974117411 q10 4823469242704270 q11 533 390 349 349 q12 720 734 530 530 q13 3047345228132813 q14 276 271 245 245 q15 q16 672 697 618 618 q17 1308127412751274 q18 7282675866936693 q19 1121108910921089 q20 2229222619401940 q21 5254454044494449 q22 515 441 418 418 Total cold run time: 58106 ms Total hot run time: 51722 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4757577406 # BE Regression && UT Coverage Report Increment line coverage `86.99% (381/438)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_6658e9ff34d5556b6342b32543032c99cba86904_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_6658e9ff34d5556b6342b32543032c99cba86904_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 74.06% (28398/38343) | | Line Coverage | 58.04% (309945/534050) | | Region Coverage | 54.91% (259802/473136) | | Branch Coverage | 56.19% (112723/200593) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4757319219 # BE UT Coverage Report Increment line coverage `61.14% (269/440)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/6658e9ff34d5556b6342b32543032c99cba86904_6658e9ff34d5556b6342b32543032c99cba86904/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/6658e9ff34d5556b6342b32543032c99cba86904_6658e9ff34d5556b6342b32543032c99cba86904/report/index.html) | Category | Coverage | |---|| | Function Coverage | 54.45% (21382/39269) | | Line Coverage | 38.09% (204613/537203) | | Region Coverage | 34.09% (160551/470995) | | Branch Coverage | 35.08% (70279/200340) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
github-actions[bot] commented on code in PR #61518:
URL: https://github.com/apache/doris/pull/61518#discussion_r3445755674
##
fe/be-java-extensions/lakesoul-scanner/src/main/java/org/apache/doris/lakesoul/arrow/LakeSoulArrowJniScanner.java:
##
@@ -145,6 +145,12 @@ protected void initTableInfo(Map params) {
private Integer fillMetaAddressVector(int batchSize, ColumnType
columnType, long metaAddress, Integer offset,
ValueVector valueVector) {
+OffHeap.putLong(null, metaAddress + (offset++) * 8, 0);
Review Comment:
This method builds the metadata address returned by
`LakeSoulJniScanner.getNextBatchMeta()`, so it must use the same BE-facing
layout that `JniDataBridge::fill_column()` consumes. The new leading zero is
read by BE as `null_map_ptr` because BE calls `address.next_meta_as_ptr()`
before any type-specific field, so normal LakeSoul columns now fail with
`Unsupported type ... in java side` before BE reaches the real validity buffer.
This is distinct from the shared `VectorColumn.updateMeta()` path: LakeSoul
hand-writes `metaAddress` and `VectorTable.getMetaAddress()` returns it
unchanged for `createReadableTable()`. Please keep LakeSoul's returned metadata
in the null-map-first BE layout, or translate from the Java-readable layout
before returning it to BE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4757084800 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4757084433 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4753487732 ClickBench: Total hot run time: 25.3 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 731702eb788f42689f94b330393ac0b5f5561c59, data reload: false query1 0.010.010.01 query2 0.100.050.05 query3 0.260.130.13 query4 1.610.140.13 query5 0.240.230.23 query6 1.221.081.08 query7 0.040.010.01 query8 0.060.040.05 query9 0.390.310.31 query10 0.550.560.55 query11 0.180.140.15 query12 0.180.150.14 query13 0.480.470.47 query14 1.041.001.01 query15 0.610.580.60 query16 0.330.330.33 query17 1.111.101.11 query18 0.230.230.23 query19 2.101.921.93 query20 0.020.010.01 query21 15.45 0.230.15 query22 4.890.050.05 query23 16.14 0.310.12 query24 2.990.430.31 query25 0.100.050.06 query26 0.730.200.16 query27 0.050.040.04 query28 3.520.940.53 query29 12.51 4.273.46 query30 0.270.160.17 query31 2.790.620.31 query32 3.220.600.48 query33 3.273.193.24 query34 15.54 4.263.52 query35 3.513.503.57 query36 0.550.440.43 query37 0.080.070.07 query38 0.050.040.04 query39 0.040.030.03 query40 0.180.170.15 query41 0.080.030.04 query42 0.040.030.03 query43 0.050.040.03 Total cold run time: 96.81 s Total hot run time: 25.3 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4753462618 TPC-DS: Total hot run time: 176160 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 731702eb788f42689f94b330393ac0b5f5561c59, data reload: false query5 4333608 484 484 query6 437 191 166 166 query7 4864546 311 311 query8 370 226 207 207 query9 8772410440774077 query10 448 313 258 258 query11 5921235421412141 query12 154 107 110 107 query13 1300601 455 455 query14 6421539350785078 query14_14383437043704370 query15 206 197 173 173 query16 1029453 447 447 query17 1106685 547 547 query18 2476479 333 333 query19 197 179 136 136 query20 112 105 114 105 query21 224 138 115 115 query22 13691 13561 13427 13427 query23 17398 16477 16056 16056 query23_116283 16231 16548 16231 query24 7496178113271327 query24_11309131513201315 query25 557 445 363 363 query26 1325326 172 172 query27 2643603 357 357 query28 4494204820592048 query29 1099651 511 511 query30 314 239 205 205 query31 11451076960 960 query32 120 64 61 61 query33 538 331 254 254 query34 11701102679 679 query35 761 781 679 679 query36 1387135712981298 query37 157 101 92 92 query38 3207315230563056 query39 935 925 897 897 query39_1924 868 864 864 query40 225 128 107 107 query41 70 69 65 65 query42 99 98 99 98 query43 331 325 286 286 query44 1448771 808 771 query45 197 183 180 180 query46 10811250768 768 query47 2335232921462146 query48 403 435 311 311 query49 642 476 367 367 query50 1054361 260 260 query51 4329434342944294 query52 96 93 80 80 query53 257 270 207 207 query54 281 229 209 209 query55 80 79 74 74 query56 247 235 223 223 query57 1436143813471347 query58 262 220 205 205 query59 1558162514201420 query60 295 258 247 247 query61 190 176 206 176 query62 700 659 588 588 query63 230 189 208 189 query64 2535776 600 600 query65 4900478147864781 query66 1797468 342 342 query67 29849 29751 29593 29593 query68 3120165810111011 query69 413 301 265 265 query70 1068969 976 969 query71 292 247 211 211 query72 2933264923932393 query73 876 804 424 424 query74 5165495647844784 query75 2598261122342234 query76 23111182809 809 query77 370 382 286 286 query78 12546 12672 11923 11923 query79 13941127788 788 query80 1265467 393 393 query81 525 280 237 237 query82 662 156 121 121 query83 325 270 244 244 query84 285 141 113 113 query85 899 501 416 416 query86 422 301 274 274 query87 3383332032123212 query88 3754281827982798 query89 421 386 334 334 query90 1917202 178 178 query91 171 163 135 135 query92 61 62 53 53 query93 16431391878 878 query94 713 343 323 323 query95 690 399 455 399 query96 1127830 352 352 query97 2663268925532553 query98 216 206 213 206 query99 1175118310161016 Total cold run time: 262430 ms Total hot run time: 176160 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4753402595 TPC-H: Total hot run time: 29636 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 731702eb788f42689f94b330393ac0b5f5561c59, data reload: false -- Round 1 -- q1 18133 413140574057 q2 2017312 190 190 q3 10796 1456827 827 q4 4746474 341 341 q5 8386857 582 582 q6 343 170 137 137 q7 853 848 629 629 q8 10596 164516481645 q9 6016457845254525 q10 6849178115491549 q11 455 271 243 243 q12 660 420 300 300 q13 18146 338027752775 q14 269 257 246 246 q15 q16 788 781 713 713 q17 982 964 1019964 q18 7051577157085708 q19 1177122911041104 q20 493 400 261 261 q21 5512263225442544 q22 435 358 296 296 Total cold run time: 104703 ms Total hot run time: 29636 ms - Round 2, with runtime_filter_mode=off - q1 4492430943494309 q2 331 354 232 232 q3 4573495543994399 q4 2051214513651365 q5 4461431043074307 q6 234 241 232 232 q7 2332182416621662 q8 2560225322072207 q9 8345796280967962 q10 4840472743434343 q11 587 426 387 387 q12 759 744 560 560 q13 3261358329532953 q14 310 309 279 279 q15 q16 716 730 644 644 q17 1406150813431343 q18 8107734972517251 q19 1117109010951090 q20 2225221419371937 q21 5300456744664466 q22 526 461 429 429 Total cold run time: 58533 ms Total hot run time: 52357 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
gavinchou commented on code in PR #61518: URL: https://github.com/apache/doris/pull/61518#discussion_r3443203858 ## fe/fe-authentication/fe-authentication-handler/src/test/resources/META-INF/services/org.apache.doris.authentication.spi.AuthenticationPluginFactory: ## @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more Review Comment: what is this? pls make the PR clean first, donot involve irrelevant contents. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4752054483 # FE Regression Coverage Report Increment line coverage ` 34.30% (71/207)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_731702eb788f42689f94b330393ac0b5f5561c59_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_731702eb788f42689f94b330393ac0b5f5561c59_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4751986334 # BE Regression && UT Coverage Report Increment line coverage `86.99% (381/438)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_731702eb788f42689f94b330393ac0b5f5561c59_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_731702eb788f42689f94b330393ac0b5f5561c59_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 74.06% (28397/38343) | | Line Coverage | 58.06% (310044/534050) | | Region Coverage | 54.91% (259779/473136) | | Branch Coverage | 56.15% (112626/200593) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4751395692 # BE UT Coverage Report Increment line coverage `61.14% (269/440)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/731702eb788f42689f94b330393ac0b5f5561c59_731702eb788f42689f94b330393ac0b5f5561c59/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/731702eb788f42689f94b330393ac0b5f5561c59_731702eb788f42689f94b330393ac0b5f5561c59/report/index.html) | Category | Coverage | |---|| | Function Coverage | 54.46% (21386/39269) | | Line Coverage | 38.10% (204701/537203) | | Region Coverage | 34.09% (160580/470995) | | Branch Coverage | 35.09% (70305/200340) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4750078110 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4750078112 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4702048286 # FE Regression Coverage Report Increment line coverage ` 34.78% (72/207)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_81a9d2f8d8e31ed1618834e4b09acbe9b580ef1c_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_81a9d2f8d8e31ed1618834e4b09acbe9b580ef1c_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4701742460 TPC-DS: Total hot run time: 168698 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 81a9d2f8d8e31ed1618834e4b09acbe9b580ef1c, data reload: false query5 4304618 480 480 query6 435 196 172 172 query7 4857552 303 303 query8 362 210 197 197 query9 8772410041154100 query10 480 294 255 255 query11 5924235821672167 query12 158 100 94 94 query13 1293627 409 409 query14 6342535450405040 query14_14345436943744369 query15 201 196 172 172 query16 1024447 416 416 query17 1109721 566 566 query18 2544474 356 356 query19 203 192 148 148 query20 112 108 109 108 query21 222 137 122 122 query22 13678 13599 13358 13358 query23 17429 16515 16142 16142 query23_116365 16365 16304 16304 query24 7460181113161316 query24_11328132013291320 query25 576 470 394 394 query26 1285347 176 176 query27 2631574 351 351 query28 4416205920382038 query29 1093659 491 491 query30 308 239 202 202 query31 11211080941 941 query32 112 67 61 61 query33 536 329 269 269 query34 12021137625 625 query35 757 780 683 683 query36 1381138812491249 query37 159 103 89 89 query38 3234317630373037 query39 957 916 887 887 query39_1883 888 897 888 query40 218 130 105 105 query41 69 66 67 66 query42 99 97 94 94 query43 323 337 285 285 query44 query45 205 191 185 185 query46 11051187758 758 query47 2400239422232223 query48 415 427 301 301 query49 626 479 368 368 query50 1018368 276 276 query51 4331425542674255 query52 92 89 77 77 query53 251 262 191 191 query54 282 245 239 239 query55 78 81 69 69 query56 237 236 220 220 query57 1426139513181318 query58 246 215 206 206 query59 1555167514491449 query60 280 233 217 217 query61 151 147 143 143 query62 711 657 583 583 query63 238 201 184 184 query64 2512734 596 596 query65 query66 1786472 340 340 query67 29416 29758 28964 28964 query68 query69 425 309 259 259 query70 954 941 913 913 query71 297 226 219 219 query72 2879260823312331 query73 847 816 456 456 query74 5089502147824782 query75 2693256822162216 query76 23121146800 800 query77 338 374 290 290 query78 12376 12394 11950 11950 query79 14541103775 775 query80 987 455 385 385 query81 494 278 238 238 query82 579 155 118 118 query83 353 271 245 245 query84 query85 881 502 404 404 query86 413 300 293 293 query87 3403339432113211 query88 3626274627292729 query89 429 380 339 339 query90 1786186 171 171 query91 173 162 127 127 query92 64 63 59 59 query93 14591416855 855 query94 650 361 322 322 query95 668 369 432 369 query96 1073841 358 358 query97 2711270525622562 query98 215 204 199 199 query99 1143116310341034 Total cold run time: 250629 ms Total hot run time: 168698 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4701714121 TPC-H: Total hot run time: 28922 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 81a9d2f8d8e31ed1618834e4b09acbe9b580ef1c, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17804 406940264026 q2 q3 10688 1335786 786 q4 4689472 342 342 q5 7505882 584 584 q6 180 168 132 132 q7 771 839 626 626 q8 9416148415921484 q9 5828456645064506 q10 6737181015361536 q11 439 264 255 255 q12 628 428 287 287 q13 18179 341027622762 q14 272 258 241 241 q15 q16 820 767 710 710 q17 937 889 955 889 q18 7211564954655465 q19 1874126710951095 q20 512 397 260 260 q21 6367278126052605 q22 481 370 331 331 Total cold run time: 101338 ms Total hot run time: 28922 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4930507047794779 q2 q3 4923522146094609 q4 2163219913651365 q5 4822481446574657 q6 253 184 128 128 q7 1843179915661566 q8 2344194719231923 q9 7463739474337394 q10 4721470142184218 q11 533 382 348 348 q12 729 739 529 529 q13 3029341528302830 q14 275 288 244 244 q15 q16 675 698 610 610 q17 1284125512551255 q18 7288673466816681 q19 1087110410931093 q20 2214222719521952 q21 5283457943774377 q22 516 450 409 409 Total cold run time: 56375 ms Total hot run time: 50967 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4701676078 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4701608770 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4701572707 run external -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4701419063 # FE Regression Coverage Report Increment line coverage ` 34.78% (72/207)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_102090a44327de68379df072d7b6c49b4ecae2f6_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_102090a44327de68379df072d7b6c49b4ecae2f6_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4701192528 # FE UT Coverage Report Increment line coverage ` 55.56% (115/207)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_102090a44327de68379df072d7b6c49b4ecae2f6/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_102090a44327de68379df072d7b6c49b4ecae2f6/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4701142987 TPC-DS: Total hot run time: 169619 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 102090a44327de68379df072d7b6c49b4ecae2f6, data reload: false query5 4338645 478 478 query6 447 202 185 185 query7 4847561 287 287 query8 363 221 210 210 query9 8785404840414041 query10 451 326 267 267 query11 5821236122172217 query12 163 102 97 97 query13 1284599 421 421 query14 6658539150875087 query14_14400441944124412 query15 207 198 180 180 query16 989 452 433 433 query17 1126724 580 580 query18 2716486 351 351 query19 209 194 145 145 query20 115 117 106 106 query21 214 140 122 122 query22 13573 13621 13336 13336 query23 17317 16553 16117 16117 query23_116299 16376 16250 16250 query24 7486177012951295 query24_11327131712941294 query25 572 478 394 394 query26 1301344 178 178 query27 2562553 345 345 query28 4416204920102010 query29 1086642 491 491 query30 322 240 202 202 query31 11351076971 971 query32 104 61 59 59 query33 533 342 305 305 query34 11921138661 661 query35 758 804 682 682 query36 1406141112271227 query37 154 105 101 101 query38 3209314430263026 query39 935 909 900 900 query39_1865 861 871 861 query40 218 124 99 99 query41 63 62 59 59 query42 91 96 91 91 query43 319 320 279 279 query44 query45 191 186 180 180 query46 10851185753 753 query47 2359239522752275 query48 358 396 305 305 query49 629 475 347 347 query50 1068359 270 270 query51 4434425942884259 query52 88 92 77 77 query53 244 263 194 194 query54 281 216 204 204 query55 79 76 69 69 query56 234 224 220 220 query57 1439140613071307 query58 236 209 214 209 query59 1598166114371437 query60 286 258 232 232 query61 155 151 151 151 query62 713 647 589 589 query63 239 188 186 186 query64 2459767 656 656 query65 query66 1736454 343 343 query67 29691 29649 29586 29586 query68 query69 427 310 253 253 query70 959 965 914 914 query71 309 223 217 217 query72 3017272224132413 query73 844 785 451 451 query74 5131495947514751 query75 2648258122302230 query76 22961163811 811 query77 358 372 281 281 query78 12455 12368 11997 11997 query79 14351052751 751 query80 1119481 388 388 query81 517 282 238 238 query82 571 155 122 122 query83 365 277 252 252 query84 query85 936 553 428 428 query86 404 296 276 276 query87 3396349032003200 query88 3597275727312731 query89 431 382 332 332 query90 1810181 179 179 query91 174 159 136 136 query92 64 66 55 55 query93 14791408886 886 query94 624 359 317 317 query95 664 385 439 385 query96 1062815 329 329 query97 2695272025372537 query98 213 205 205 205 query99 1169118310261026 Total cold run time: 251544 ms Total hot run time: 169619 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4701117830 TPC-H: Total hot run time: 28777 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 102090a44327de68379df072d7b6c49b4ecae2f6, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17685 405140264026 q2 q3 10652 1405798 798 q4 4685475 347 347 q5 7518874 587 587 q6 183 175 139 139 q7 764 829 618 618 q8 9347149215711492 q9 6061451045234510 q10 6760183815161516 q11 445 273 247 247 q12 625 426 298 298 q13 18200 342927542754 q14 273 263 246 246 q15 q16 821 762 712 712 q17 15791150692 692 q18 6802573654365436 q19 2112134011021102 q20 528 402 257 257 q21 6232276926902690 q22 458 388 310 310 Total cold run time: 101730 ms Total hot run time: 28777 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 5079482147524752 q2 q3 5076522346464646 q4 211013981398 q5 4857476248034762 q6 234 186 144 144 q7 1864172215471547 q8 2463204619281928 q9 7437740573537353 q10 4743470742504250 q11 527 392 357 357 q12 739 742 536 536 q13 2967340627852785 q14 276 289 256 256 q15 q16 687 694 626 626 q17 1281125112511251 q18 7513683168596831 q19 1163109711221097 q20 2220223619821982 q21 5315460744864486 q22 530 468 415 415 Total cold run time: 57081 ms Total hot run time: 51402 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
github-actions[bot] commented on code in PR #61518:
URL: https://github.com/apache/doris/pull/61518#discussion_r3409185254
##
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java:
##
@@ -884,9 +886,11 @@ private Split createIcebergSplit(FileScanTask
fileScanTask) {
split.setPartitionDataJson(IcebergUtils.getPartitionDataJson(
partitionData, partitionSpec,
sessionVariable.getTimeZone()));
Map partitionInfoMap =
partitionMapInfos.computeIfAbsent(
-partitionData, k ->
IcebergUtils.getIdentityPartitionInfoMap(
-partitionData, partitionSpec, icebergTable,
sessionVariable.getTimeZone()));
-if (!partitionInfoMap.isEmpty()) {
+partitionData, k ->
IcebergUtils.getIdentityPartitionInfoMapForCache(
Review Comment:
This helper is now used to populate `partitionMapInfos`, but it returns
`null` for any non-identity Iceberg spec. `Map.computeIfAbsent` does not record
a mapping when the function returns `null`, so transform-partitioned tables,
for example `PARTITIONED BY day(ts)`, no longer add their `PartitionData` to
`partitionMapInfos`. Later `selectedPartitionNum = partitionMapInfos.size()`
and `numApproximateSplits()` use that same map, so those scans report zero
selected partitions and batch mode estimates only one split regardless of the
number of scanned partitions.
This is separate from whether the cache context should be omitted for
non-identity specs. Please keep partition accounting independent from the
cache-context map, or store a sentinel/empty map for non-cacheable partition
specs while only skipping `setIcebergPartitionValues`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4701025079 # FE UT Coverage Report Increment line coverage ` 54.85% (113/206)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_56ebc6f12b986b3ff57e5d1bd5f1fad64ac94752/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_56ebc6f12b986b3ff57e5d1bd5f1fad64ac94752/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4701017520 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4701017528 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4700972739 TPC-DS: Total hot run time: 169718 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 56ebc6f12b986b3ff57e5d1bd5f1fad64ac94752, data reload: false query5 4332626 502 502 query6 448 196 184 184 query7 4911594 301 301 query8 362 215 205 205 query9 8792412140854085 query10 449 321 269 269 query11 5922235121622162 query12 156 104 100 100 query13 1258605 407 407 query14 6455540950795079 query14_14400440945114409 query15 209 200 174 174 query16 1039463 407 407 query17 1153744 593 593 query18 2778489 350 350 query19 206 205 144 144 query20 115 116 112 112 query21 238 140 115 115 query22 13686 13457 13365 13365 query23 17209 16465 16200 16200 query23_116272 16405 16170 16170 query24 7530178613031303 query24_11313131212911291 query25 591 468 397 397 query26 1319294 177 177 query27 2611551 342 342 query28 4369206720562056 query29 1115632 531 531 query30 319 234 199 199 query31 11101093956 956 query32 113 65 62 62 query33 524 328 302 302 query34 11871185652 652 query35 745 776 671 671 query36 1419139512211221 query37 152 105 87 87 query38 3235316730363036 query39 951 916 905 905 query39_1866 869 888 869 query40 218 124 98 98 query41 63 61 57 57 query42 93 93 93 93 query43 314 323 285 285 query44 query45 190 187 178 178 query46 10881233758 758 query47 2354235322332233 query48 408 386 298 298 query49 617 464 349 349 query50 979 358 259 259 query51 4347430942124212 query52 86 89 77 77 query53 238 287 187 187 query54 270 221 192 192 query55 79 80 70 70 query56 219 252 226 226 query57 1437141313091309 query58 246 212 202 202 query59 1590165515001500 query60 305 246 227 227 query61 150 152 147 147 query62 694 646 563 563 query63 232 193 187 187 query64 2493810 608 608 query65 query66 1748474 338 338 query67 29592 29832 29738 29738 query68 query69 429 297 254 254 query70 998 972 975 972 query71 300 231 206 206 query72 2967263823442344 query73 879 742 432 432 query74 5108493247794779 query75 26512576 query76 22841217828 828 query77 343 362 298 298 query78 12433 12485 12043 12043 query79 14711068754 754 query80 1285465 380 380 query81 488 276 242 242 query82 593 156 127 127 query83 341 274 250 250 query84 query85 1038508 418 418 query86 433 299 273 273 query87 3443337332163216 query88 3660276427452745 query89 436 385 331 331 query90 1914185 187 185 query91 170 163 136 136 query92 72 61 53 53 query93 14841438848 848 query94 741 364 322 322 query95 669 384 433 384 query96 1084768 330 330 query97 2697268925462546 query98 212 216 200 200 query99 1157119610591059 Total cold run time: 252044 ms Total hot run time: 169718 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4700950631 TPC-H: Total hot run time: 29099 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 56ebc6f12b986b3ff57e5d1bd5f1fad64ac94752, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17365 402040194019 q2 q3 10601 1438810 810 q4 4664472 344 344 q5 7616876 591 591 q6 183 176 144 144 q7 781 845 623 623 q8 9403159915921592 q9 5823448545004485 q10 6771180615541554 q11 442 271 249 249 q12 632 416 294 294 q13 18162 341127422742 q14 266 259 235 235 q15 q16 813 781 717 717 q17 1027965 992 965 q18 6755583955535553 q19 1693129710341034 q20 514 415 268 268 q21 6279274925532553 q22 464 373 327 327 Total cold run time: 100254 ms Total hot run time: 29099 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 5166478447874784 q2 q3 5091527946354635 q4 2112219714291429 q5 4886481846054605 q6 234 187 138 138 q7 1886175815201520 q8 2531209620432043 q9 7454756374277427 q10 4735465942094209 q11 536 390 358 358 q12 736 738 524 524 q13 3007343028012801 q14 290 272 256 256 q15 q16 674 713 621 621 q17 1283126812611261 q18 7350696668816881 q19 1125108611261086 q20 2233223319681968 q21 5304461644494449 q22 524 458 423 423 Total cold run time: 57157 ms Total hot run time: 51418 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4700932700 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/56ebc6f12b986b3ff57e5d1bd5f1fad64ac94752_56ebc6f12b986b3ff57e5d1bd5f1fad64ac94752_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/56ebc6f12b986b3ff57e5d1bd5f1fad64ac94752_56ebc6f12b986b3ff57e5d1bd5f1fad64ac94752_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 77.32% (1889/2443) | | Line Coverage | 64.40% (33954/52725) | | Region Coverage | 64.80% (17463/26948) | | Branch Coverage | 53.94% (9340/17316) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4700859627 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
freemandealer commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4700807060 run beut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4699833399 # FE Regression Coverage Report Increment line coverage ` 39.66% (71/179)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_3873121587e80450f7e658d4012c82cb16aaaba4_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_3873121587e80450f7e658d4012c82cb16aaaba4_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4699721879 # FE UT Coverage Report Increment line coverage ` 54.19% (97/179)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61518_3873121587e80450f7e658d4012c82cb16aaaba4/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61518_3873121587e80450f7e658d4012c82cb16aaaba4/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4699299873 TPC-DS: Total hot run time: 169365 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 3873121587e80450f7e658d4012c82cb16aaaba4, data reload: false query5 4326635 493 493 query6 430 193 175 175 query7 4800581 309 309 query8 360 218 211 211 query9 8784415041024102 query10 453 302 267 267 query11 5923235521722172 query12 154 102 98 98 query13 1260622 414 414 query14 6403544451065106 query14_14454439444354394 query15 205 196 174 174 query16 1010446 390 390 query17 1107693 556 556 query18 2698475 337 337 query19 197 181 140 140 query20 113 107 104 104 query21 212 144 117 117 query22 13688 13550 13275 13275 query23 17375 16430 16184 16184 query23_116226 16368 16270 16270 query24 7496178513151315 query24_11308131013211310 query25 549 440 391 391 query26 1303317 166 166 query27 2648567 336 336 query28 4437205420522052 query29 1075654 502 502 query30 313 231 196 196 query31 11261076948 948 query32 109 68 56 56 query33 547 320 243 243 query34 11881158652 652 query35 737 775 691 691 query36 1396138912101210 query37 157 105 94 94 query38 3226314630633063 query39 921 927 896 896 query39_1861 863 897 863 query40 216 123 104 104 query41 64 63 63 63 query42 95 94 94 94 query43 335 336 285 285 query44 query45 195 192 186 186 query46 11161184739 739 query47 2413234022492249 query48 373 436 294 294 query49 622 457 346 346 query50 1020360 255 255 query51 4388433543134313 query52 89 90 77 77 query53 255 269 188 188 query54 270 231 203 203 query55 77 76 74 74 query56 238 222 207 207 query57 1436141012851285 query58 238 214 221 214 query59 1593167114501450 query60 277 250 222 222 query61 155 149 153 149 query62 711 661 589 589 query63 233 187 189 187 query64 2534784 607 607 query65 query66 1773482 360 360 query67 29060 29667 29476 29476 query68 query69 473 328 275 275 query70 1007991 1029991 query71 313 218 223 218 query72 3084285225452545 query73 872 787 430 430 query74 5183497747494749 query75 2685256622782278 query76 23341185790 790 query77 367 399 300 300 query78 12321 12388 11820 11820 query79 14991092774 774 query80 1193474 379 379 query81 513 276 245 245 query82 572 161 121 121 query83 363 278 247 247 query84 query85 916 506 417 417 query86 410 302 278 278 query87 3386339232093209 query88 3710275327512751 query89 426 379 342 342 query90 1808186 191 186 query91 172 158 130 130 query92 62 58 58 58 query93 14741455934 934 query94 659 356 292 292 query95 687 370 348 348 query96 1064793 368 368 query97 2695269925382538 query98 207 202 203 202 query99 1136118010181018 Total cold run time: 251330 ms Total hot run time: 169365 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
hello-stephen commented on PR #61518: URL: https://github.com/apache/doris/pull/61518#issuecomment-4699273199 TPC-H: Total hot run time: 28399 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 3873121587e80450f7e658d4012c82cb16aaaba4, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17669 395939773959 q2 q3 11009 1363826 826 q4 4814495 339 339 q5 8735863 589 589 q6 357 169 132 132 q7 923 835 625 625 q8 10905 137714061377 q9 7185453044544454 q10 6875182915041504 q11 443 271 246 246 q12 653 427 292 292 q13 18170 332427762776 q14 266 253 248 248 q15 q16 834 788 701 701 q17 1020971 902 902 q18 6896563954345434 q19 1154123910371037 q20 509 400 258 258 q21 5510261023942394 q22 425 359 306 306 Total cold run time: 104352 ms Total hot run time: 28399 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4315425942284228 q2 q3 4505493143034303 q4 2083218813511351 q5 4463427542814275 q6 227 175 128 128 q7 1925190116551655 q8 2518214820502050 q9 7981782678117811 q10 4787475743344334 q11 595 411 530 411 q12 738 750 537 537 q13 3271360129832983 q14 302 311 285 285 q15 q16 703 734 676 676 q17 1357132713191319 q18 7826720470507050 q19 1116113111081108 q20 2205221819541954 q21 5242453944354435 q22 512 482 430 430 Total cold run time: 56671 ms Total hot run time: 51323 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature](filecache) persist table/partition context for cache meta for external table [doris]
github-actions[bot] commented on code in PR #61518:
URL: https://github.com/apache/doris/pull/61518#discussion_r3408185640
##
be/src/io/cache/fs_file_cache_storage.cpp:
##
@@ -879,6 +891,7 @@ void
FSFileCacheStorage::load_cache_info_into_memory_from_db(BlockFileCache* mgr
LOG(WARNING) << "Failed to create iterator for meta store";
return;
}
+std::unordered_map>
context_cache;
while (iterator->valid()) {
Review Comment:
This restores the persisted context for the async DB iterator load, but
`load_blocks_directly_unlocked()` still builds `context_original` from the same
`BlockMeta` without copying `context_id` or resolving table/partition. During
async open, a query can hit a persisted block before this iterator has loaded
it; that on-demand path adds the in-memory cell with `context_id = 0`. If that
block later changes cache type, `change_key_meta_type()` rewrites RocksDB with
the zero context id and permanently drops the table/partition context for this
cache block. Please apply the same `context_id` / `get_context()` restoration
in `load_blocks_directly_unlocked()` before `add_cell()`.
##
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java:
##
@@ -884,9 +886,11 @@ private Split createIcebergSplit(FileScanTask
fileScanTask) {
split.setPartitionDataJson(IcebergUtils.getPartitionDataJson(
partitionData, partitionSpec,
sessionVariable.getTimeZone()));
Map partitionInfoMap =
partitionMapInfos.computeIfAbsent(
-partitionData, k ->
IcebergUtils.getIdentityPartitionInfoMap(
Review Comment:
This helper returns keys from the Iceberg partition field names, while
`getOrderedPathPartitionKeys()` still returns source column names via
`getIdentityPartitionColumns()`. For an identity partition whose partition
field name differs from the source column name, `partitionValues` contains the
partition field key but `fillPartitionContextFromMap()` looks for the source
column key, skips it, and leaves both `columns_from_path` and the cache
partition context empty. The previous `getIdentityPartitionInfoMap()` used
`partitionField.sourceId()` and `table.schema().findColumnName(...)`, so it
produced source-column keys. Please preserve source-column keys while still
rejecting non-identity specs, for example by adding a helper that returns null
on non-identity fields but maps identity values through the source column name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
