Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
freemandealer merged PR #61083: URL: https://github.com/apache/doris/pull/61083 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
github-actions[bot] commented on code in PR #61083:
URL: https://github.com/apache/doris/pull/61083#discussion_r3316096792
##
be/test/io/cache/block_file_cache_test_lru_dump.cpp:
##
@@ -597,4 +597,111 @@ TEST_F(BlockFileCacheTest,
cached_remote_file_reader_direct_read_order_check) {
FileCacheFactory::instance()->_capacity = 0;
}
+TEST_F(BlockFileCacheTest, get_or_set_hit_order_check) {
+std::string cache_base_path = caches_dir /
"cache_get_or_set_hit_order_check" / "";
+
+const auto old_async_touch =
config::enable_file_cache_async_touch_on_get_or_set;
+const auto old_update_interval_ms =
config::file_cache_background_block_lru_update_interval_ms;
+const auto old_update_qps_limit =
config::file_cache_background_block_lru_update_qps_limit;
+Defer defer {[old_async_touch, old_update_interval_ms,
old_update_qps_limit] {
+config::enable_file_cache_async_touch_on_get_or_set = old_async_touch;
+config::file_cache_background_block_lru_update_interval_ms =
old_update_interval_ms;
+config::file_cache_background_block_lru_update_qps_limit =
old_update_qps_limit;
+}};
+
+config::enable_file_cache_async_touch_on_get_or_set = true;
+config::file_cache_background_block_lru_update_interval_ms = 200;
+config::file_cache_background_block_lru_update_qps_limit = 10;
+
+if (fs::exists(cache_base_path)) {
+fs::remove_all(cache_base_path);
+}
+fs::create_directories(cache_base_path);
+
+TUniqueId query_id;
+query_id.hi = 1;
+query_id.lo = 1;
+
+io::FileCacheSettings settings;
+settings.query_queue_size = 6291456;
+settings.query_queue_elements = 6;
+settings.index_queue_size = 1048576;
+settings.index_queue_elements = 1;
+settings.disposable_queue_size = 1048576;
+settings.disposable_queue_elements = 1;
+settings.capacity = 8388608;
+settings.max_file_block_size = 1048576;
+
+io::BlockFileCache cache(cache_base_path, settings);
+ASSERT_TRUE(cache.initialize());
+for (int i = 0; i < 100; i++) {
+if (cache.get_async_open_success()) {
+break;
+}
+std::this_thread::sleep_for(std::chrono::milliseconds(10));
+}
+ASSERT_TRUE(cache.get_async_open_success());
+
+io::CacheContext context;
+ReadStatistics stats;
+context.stats = &stats;
+context.cache_type = io::FileCacheType::NORMAL;
+context.query_id = query_id;
+
+auto key = io::BlockFileCache::hash("get_or_set_hit_order_check_key");
+constexpr size_t kBlockSize = 1024 * 1024;
+std::vector offsets {0, kBlockSize, 2 * kBlockSize};
+
+for (size_t offset : offsets) {
+auto holder = cache.get_or_set(key, offset, kBlockSize, context);
+auto blocks = fromHolder(holder);
+ASSERT_EQ(blocks.size(), 1);
+ASSERT_TRUE(blocks[0]->get_or_set_downloader() ==
io::FileBlock::get_caller_id());
+download_into_memory(blocks[0]);
+ASSERT_EQ(blocks[0]->state(), io::FileBlock::State::DOWNLOADED);
+}
+
+std::vector initial_offsets;
+for (auto it = cache._normal_queue.begin(); it !=
cache._normal_queue.end(); ++it) {
+initial_offsets.push_back(it->offset);
+}
+ASSERT_EQ(initial_offsets.size(), 3);
+ASSERT_EQ(initial_offsets[0], 0);
+ASSERT_EQ(initial_offsets[1], kBlockSize);
+ASSERT_EQ(initial_offsets[2], 2 * kBlockSize);
+
+auto holder = cache.get_or_set(key, 0, kBlockSize, context);
+auto blocks = fromHolder(holder);
+ASSERT_EQ(blocks.size(), 1);
+ASSERT_EQ(blocks[0]->state(), io::FileBlock::State::DOWNLOADED);
+
+std::vector before_updated_offsets;
+for (auto it = cache._normal_queue.begin(); it !=
cache._normal_queue.end(); ++it) {
+before_updated_offsets.push_back(it->offset);
+}
+ASSERT_EQ(before_updated_offsets.size(), 3);
+ASSERT_EQ(before_updated_offsets[0], 0);
+ASSERT_EQ(before_updated_offsets[1], kBlockSize);
+ASSERT_EQ(before_updated_offsets[2], 2 * kBlockSize);
+
+std::this_thread::sleep_for(std::chrono::milliseconds(
Review Comment:
This assertion still depends on the background LRU thread waking and
draining within a fixed wall-clock sleep. On a busy CI host the thread can be
delayed past `2 * interval_ms`, leaving the queue unchanged and making
`updated_offsets.back()` fail even though the implementation is correct. This
is distinct from the existing lock-wait timing thread: here the flake is the
fixed sleep used to observe the async LRU worker. Please make this
deterministic, for example by polling until
`need_update_lru_blocks_size_unsafe() == 0`/the expected order with a timeout,
or by invoking/draining the LRU update path synchronously in the test.
##
be/test/io/cache/cached_remote_file_reader_test.cpp:
##
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distri
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
freemandealer commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4561728299 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
github-actions[bot] commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4560743847 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
hello-stephen commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4555467531 # BE Regression && UT Coverage Report Increment line coverage `100.00% (23/23)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61083_0b40d997a1f5f6d7b75aa6a75fd467ba7ea1ac43_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61083_0b40d997a1f5f6d7b75aa6a75fd467ba7ea1ac43_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 73.71% (28030/38027) | | Line Coverage | 57.68% (304485/527872) | | Region Coverage | 54.94% (255348/464771) | | Branch Coverage | 56.35% (110083/195358) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
hello-stephen commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4555200588 # BE Regression && UT Coverage Report Increment line coverage `100.00% (23/23)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61083_0b40d997a1f5f6d7b75aa6a75fd467ba7ea1ac43_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61083_0b40d997a1f5f6d7b75aa6a75fd467ba7ea1ac43_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 73.71% (28028/38027) | | Line Coverage | 57.68% (304459/527872) | | Region Coverage | 54.94% (255333/464771) | | Branch Coverage | 56.34% (110074/195358) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
freemandealer commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4554589524 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
hello-stephen commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4554082024 TPC-DS: Total hot run time: 172076 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 0b40d997a1f5f6d7b75aa6a75fd467ba7ea1ac43, data reload: false query5 4326668 522 522 query6 343 228 208 208 query7 4245590 334 334 query8 330 240 239 239 query9 8831418541384138 query10 444 348 302 302 query11 5743257722322232 query12 185 140 130 130 query13 1324606 426 426 query14 6198556952385238 query14_14540453344964496 query15 221 208 188 188 query16 1035473 470 470 query17 1148758 623 623 query18 2733493 372 372 query19 230 209 171 171 query20 139 136 135 135 query21 218 137 122 122 query22 13573 13592 13343 13343 query23 17351 16500 16241 16241 query23_116500 16376 16310 16310 query24 7528182013121312 query24_11326133113281328 query25 612 521 457 457 query26 1299324 179 179 query27 2666579 359 359 query28 4456204720612047 query29 1030652 526 526 query30 314 244 201 201 query31 11151117962 962 query32 91 82 76 76 query33 532 355 304 304 query34 11721137668 668 query35 780 795 704 704 query36 1443137812301230 query37 146 99 86 86 query38 3219320630803080 query39 918 907 912 907 query39_1919 887 884 884 query40 223 144 126 126 query41 66 62 63 62 query42 114 109 107 107 query43 331 333 296 296 query44 query45 224 206 204 204 query46 11151250728 728 query47 2384241822532253 query48 424 449 299 299 query49 628 502 393 393 query50 977 354 256 256 query51 4383425342734253 query52 107 106 94 94 query53 247 284 212 212 query54 317 276 258 258 query55 96 92 84 84 query56 305 307 315 307 query57 1459143013401340 query58 301 278 267 267 query59 1572169314221422 query60 329 327 318 318 query61 152 156 148 148 query62 706 653 580 580 query63 253 203 211 203 query64 2379803 646 646 query65 query66 1665521 367 367 query67 29797 29721 29655 29655 query68 query69 464 352 307 307 query70 995 999 987 987 query71 300 270 268 268 query72 3042274624382438 query73 819 803 438 438 query74 5113500447984798 query75 2692261822942294 query76 22891158763 763 query77 409 415 351 351 query78 12337 12465 11917 11917 query79 15061063754 754 query80 679 570 458 458 query81 455 284 237 237 query82 1415166 126 126 query83 373 282 250 250 query84 266 141 112 112 query85 879 568 454 454 query86 399 317 341 317 query87 3451342932523252 query88 3637273627292729 query89 444 402 353 353 query90 1903185 177 177 query91 179 181 151 151 query92 84 81 78 78 query93 15001432849 849 query94 556 373 307 307 query95 707 483 349 349 query96 1034775 327 327 query97 2719275826572657 query98 240 226 238 226 query99 1163115810181018 Total cold run time: 254811 ms Total hot run time: 172076 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
hello-stephen commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4554005173 TPC-H: Total hot run time: 32010 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 0b40d997a1f5f6d7b75aa6a75fd467ba7ea1ac43, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17655 401539803980 q2 q3 10751 1434819 819 q4 4683474 342 342 q5 7600226621242124 q6 235 171 140 140 q7 975 785 638 638 q8 9403171216961696 q9 5459504049774977 q10 6464220319071907 q11 448 275 242 242 q12 692 429 294 294 q13 18223 344428062806 q14 264 263 241 241 q15 q16 818 776 714 714 q17 916 979 943 943 q18 6855587657115711 q19 1190136812901290 q20 574 458 281 281 q21 6055289025452545 q22 546 363 320 320 Total cold run time: 99806 ms Total hot run time: 32010 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4868488748574857 q2 q3 5091522646914691 q4 2142232113941394 q5 5097477446814681 q6 236 179 129 129 q7 1926193316021602 q8 2417213721182118 q9 7901739773917391 q10 4764465741964196 q11 534 395 354 354 q12 732 747 526 526 q13 2997336128172817 q14 269 277 267 267 q15 q16 677 697 618 618 q17 1287127412571257 q18 7154699069336933 q19 1150109411221094 q20 2231222319411941 q21 5280457843874387 q22 532 449 421 421 Total cold run time: 57285 ms Total hot run time: 51674 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
freemandealer commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4552947537 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
github-actions[bot] commented on code in PR #61083:
URL: https://github.com/apache/doris/pull/61083#discussion_r3309135691
##
be/test/io/cache/cached_remote_file_reader_lock_wait_test.cpp:
##
@@ -0,0 +1,428 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "common/config.h"
+#include "io/cache/block_file_cache.h"
+#include "io/cache/block_file_cache_factory.h"
+#include "io/cache/cached_remote_file_reader.h"
+#include "io/cache/file_cache_common.h"
+#include "io/fs/file_reader.h"
+#include "runtime/exec_env.h"
+#include "runtime/thread_context.h"
+#include "util/defer_op.h"
+#include "util/slice.h"
+
+namespace doris::io {
+
+namespace {
+
+class MockPatternFileReader final : public FileReader {
+public:
+MockPatternFileReader(std::string path, size_t file_size, uint32_t
pattern_id)
+: _path(std::move(path)), _size(file_size),
_pattern_id(pattern_id) {}
+
+Status close() override {
+_closed = true;
+return Status::OK();
+}
+
+const Path& path() const override { return _path; }
+
+size_t size() const override { return _size; }
+
+bool closed() const override { return _closed; }
+
+int64_t mtime() const override { return 0; }
+
+protected:
+Status read_at_impl(size_t offset, Slice result, size_t* bytes_read,
+const IOContext* /*io_ctx*/) override {
+if (offset >= _size) {
+*bytes_read = 0;
+return Status::OK();
+}
+const size_t readable = std::min(result.size, _size - offset);
+memset(result.data, static_cast('a' + (_pattern_id % 26)),
readable);
+*bytes_read = readable;
+return Status::OK();
+}
+
+private:
+Path _path;
+size_t _size;
+uint32_t _pattern_id;
+bool _closed {false};
+};
+
+struct LockWaitSummary {
+int64_t total_ns {0};
+int64_t max_ns {0};
+int64_t p50_ns {0};
+int64_t p95_ns {0};
+int64_t p99_ns {0};
+double avg_ns {0.0};
+size_t non_zero_samples {0};
+};
+
+int64_t get_percentile_value(const std::vector& sorted_values, double
percentile) {
+if (sorted_values.empty()) {
+return 0;
+}
+const double rank = percentile * static_cast(sorted_values.size()
- 1);
+return sorted_values[static_cast(rank)];
+}
+
+LockWaitSummary summarize_lock_wait(std::vector* values) {
+LockWaitSummary summary;
+if (values == nullptr || values->empty()) {
+return summary;
+}
+std::sort(values->begin(), values->end());
+summary.total_ns = std::accumulate(values->begin(), values->end(), int64_t
{0});
+summary.max_ns = values->back();
+summary.p50_ns = get_percentile_value(*values, 0.50);
+summary.p95_ns = get_percentile_value(*values, 0.95);
+summary.p99_ns = get_percentile_value(*values, 0.99);
+summary.avg_ns = static_cast(summary.total_ns) /
static_cast(values->size());
+summary.non_zero_samples =
+std::count_if(values->begin(), values->end(), [](int64_t v) {
return v > 0; });
+return summary;
+}
+
+struct LockWaitWorkloadConfig {
+size_t file_count {2048};
+size_t file_size {64 * 1024};
+size_t warmup_read_bytes {4 * 1024};
+size_t stress_read_bytes {4 * 1024};
+size_t ops_per_thread {2000};
+size_t thread_count {16};
+uint32_t seed_base {20260228};
+std::string file_prefix {"default"};
+};
+
+struct LockWaitWorkloadResult {
+LockWaitSummary summary;
+size_t populated_keys {0};
+size_t warmup_failed_reads {0};
+size_t failed_reads {0};
+size_t sample_count {0};
+};
+
+size_t calc_thread_count() {
+const size_t hw_threads =
+std::thread::hardware_concurrency() == 0 ? 16 :
std::thread::hardware_concurrency();
+return std::min(48, std::max(16, hw_threads));
+}
+
+} // namespace
+
+class CachedRemoteFileReaderLockWaitTest : public testing::Test {
+public:
+static void SetUpTestSuite() {
+auto* exec_env = ExecEnv::GetInstance();
+if (exec_env->file_cache_factory() == nullptr) {
+
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
freemandealer commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4552305554 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
hello-stephen commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4448058225 # BE Regression && UT Coverage Report Increment line coverage `100.00% (23/23)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61083_883b1f23ac2bcc554ebed4b9b6c9fbac657fc8c4_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61083_883b1f23ac2bcc554ebed4b9b6c9fbac657fc8c4_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 73.69% (27784/37706) | | Line Coverage | 57.60% (301176/522920) | | Region Coverage | 54.80% (251370/458677) | | Branch Coverage | 56.33% (108738/193034) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
hello-stephen commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4447920377 # BE UT Coverage Report Increment line coverage `100.00% (23/23)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/883b1f23ac2bcc554ebed4b9b6c9fbac657fc8c4_883b1f23ac2bcc554ebed4b9b6c9fbac657fc8c4/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/883b1f23ac2bcc554ebed4b9b6c9fbac657fc8c4_883b1f23ac2bcc554ebed4b9b6c9fbac657fc8c4/report/index.html) | Category | Coverage | |---|| | Function Coverage | 53.58% (20631/38506) | | Line Coverage | 37.21% (195089/524303) | | Region Coverage | 33.65% (152837/454263) | | Branch Coverage | 34.62% (66578/192306) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
hello-stephen commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4447472535 TPC-DS: Total hot run time: 172044 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 883b1f23ac2bcc554ebed4b9b6c9fbac657fc8c4, data reload: false query5 4343656 524 524 query6 339 216 202 202 query7 4210565 317 317 query8 346 257 214 214 query9 8872417241414141 query10 459 332 327 327 query11 5710240022262226 query12 188 130 128 128 query13 1274645 439 439 query14 6600539650585058 query14_14404431142834283 query15 210 204 180 180 query16 1046471 401 401 query17 1126767 623 623 query18 2724481 346 346 query19 228 205 167 167 query20 144 134 134 134 query21 215 139 113 113 query22 13764 14044 14380 14044 query23 17509 16628 16155 16155 query23_116306 16359 16344 16344 query24 7915182813491349 query24_11342132613771326 query25 561 481 430 430 query26 1310316 162 162 query27 2711562 333 333 query28 4403197320121973 query29 1004627 495 495 query30 295 236 194 194 query31 11121074931 931 query32 82 73 72 72 query33 566 346 298 298 query34 11911127629 629 query35 748 791 679 679 query36 1364134112291229 query37 152 102 87 87 query38 3210315730613061 query39 950 935 897 897 query39_1876 870 896 870 query40 240 157 147 147 query41 71 66 69 66 query42 116 110 108 108 query43 329 323 287 287 query44 query45 219 207 197 197 query46 10761192712 712 query47 2347237921912191 query48 446 406 297 297 query49 632 531 420 420 query50 718 277 218 218 query51 4325425542104210 query52 104 104 96 96 query53 262 276 205 205 query54 312 271 247 247 query55 91 91 84 84 query56 311 296 307 296 query57 1432141013221322 query58 288 271 266 266 query59 1568161513981398 query60 357 341 317 317 query61 164 149 166 149 query62 661 619 562 562 query63 247 204 215 204 query64 2311860 702 702 query65 query66 1700504 403 403 query67 30105 29990 29866 29866 query68 query69 474 375 302 302 query70 1035997 962 962 query71 320 288 273 273 query72 2904269424162416 query73 861 729 428 428 query74 5060489247584758 query75 2763262923152315 query76 23061142776 776 query77 417 423 346 346 query78 12900 12905 12323 12323 query79 1416964 720 720 query80 650 579 500 500 query81 455 281 245 245 query82 525 157 123 123 query83 353 278 251 251 query84 258 140 118 118 query85 860 515 449 449 query86 390 339 320 320 query87 3389334632233223 query88 3611271927022702 query89 417 376 338 338 query90 1922189 179 179 query91 182 171 136 136 query92 83 77 70 70 query93 947 944 559 559 query94 537 342 293 293 query95 656 383 359 359 query96 1019771 338 338 query97 2715269326122612 query98 239 228 232 228 query99 11091141996 996 Total cold run time: 253868 ms Total hot run time: 172044 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
hello-stephen commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4447420988 TPC-H: Total hot run time: 29378 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 883b1f23ac2bcc554ebed4b9b6c9fbac657fc8c4, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17744 384837873787 q2 q3 10711 884 620 620 q4 4697459 364 364 q5 7708133311331133 q6 221 166 136 136 q7 935 947 741 741 q8 9995138613191319 q9 6801557653265326 q10 6351208518101810 q11 484 271 251 251 q12 693 419 288 288 q13 18165 334227102710 q14 293 285 262 262 q15 q16 899 873 795 795 q17 10351032758 758 q18 6532563856065606 q19 1347130710491049 q20 514 395 261 261 q21 4738225718601860 q22 423 354 302 302 Total cold run time: 100286 ms Total hot run time: 29378 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4172411740994099 q2 q3 4739479941784178 q4 2084217214141414 q5 4970495352364953 q6 192 166 129 129 q7 2203192617121712 q8 3523313832203138 q9 8586848885128488 q10 4477456542414241 q11 642 431 399 399 q12 697 746 506 506 q13 3257358729272927 q14 307 326 281 281 q15 q16 757 815 689 689 q17 1357131814191318 q18 7971715370857085 q19 1170117911711171 q20 2272226619701970 q21 6204541849394939 q22 606 523 417 417 Total cold run time: 60186 ms Total hot run time: 54054 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
hello-stephen commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4446856112 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
github-actions[bot] commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4304449702 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
freemandealer commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4301554845 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
hello-stephen commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4143381017 # BE Regression && UT Coverage Report Increment line coverage `100% (0/0)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61083_6b7db78d6bdcb9b0046caa5f752c067a7465ebe3_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61083_6b7db78d6bdcb9b0046caa5f752c067a7465ebe3_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 57.34% (21166/36911) | | Line Coverage | 40.46% (206888/511373) | | Region Coverage | 37.42% (167412/447377) | | Branch Coverage | 38.14% (71728/188063) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
doris-robot commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4141677027 # BE UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/6b7db78d6bdcb9b0046caa5f752c067a7465ebe3_6b7db78d6bdcb9b0046caa5f752c067a7465ebe3/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/6b7db78d6bdcb9b0046caa5f752c067a7465ebe3_6b7db78d6bdcb9b0046caa5f752c067a7465ebe3/report/index.html) | Category | Coverage | |---|| | Function Coverage | 52.90% (19938/37689) | | Line Coverage | 36.43% (186840/512921) | | Region Coverage | 32.69% (144894/443255) | | Branch Coverage | 33.89% (63539/187497) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
doris-robot commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4141559270 TPC-DS: Total hot run time: 168987 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 6b7db78d6bdcb9b0046caa5f752c067a7465ebe3, data reload: false query5 4339649 508 508 query6 323 221 211 211 query7 4220483 264 264 query8 333 236 228 228 query9 8686271626872687 query10 519 396 359 359 query11 7006508748854885 query12 192 132 134 132 query13 1286463 353 353 query14 5762369934553455 query14_12932286628462846 query15 201 195 175 175 query16 993 489 460 460 query17 1116752 629 629 query18 2451453 361 361 query19 224 219 192 192 query20 135 125 128 125 query21 217 140 113 113 query22 13294 13985 14652 13985 query23 17199 16427 16330 16330 query23_116095 15744 15618 15618 query24 7181162012301230 query24_11227127612411241 query25 537 457 393 393 query26 1231262 149 149 query27 2768476 295 295 query28 4494183318521833 query29 825 568 479 479 query30 298 226 188 188 query31 1007943 865 865 query32 84 71 67 67 query33 506 328 280 280 query34 895 873 516 516 query35 633 676 593 593 query36 10771138983 983 query37 138 103 83 83 query38 3001292528562856 query39 869 834 824 824 query39_1797 795 786 786 query40 229 147 135 135 query41 61 60 58 58 query42 260 255 263 255 query43 240 255 225 225 query44 query45 192 183 184 183 query46 872 971 597 597 query47 2123214320802080 query48 309 313 227 227 query49 622 460 380 380 query50 692 275 205 205 query51 4098418539743974 query52 260 270 259 259 query53 310 349 296 296 query54 312 265 262 262 query55 102 92 85 85 query56 324 323 331 323 query57 1926165918011659 query58 280 292 266 266 query59 2819294927352735 query60 337 343 327 327 query61 156 154 153 153 query62 619 590 535 535 query63 304 290 276 276 query64 5014128810081008 query65 query66 1456443 364 364 query67 24283 24363 24284 24284 query68 query69 407 305 298 298 query70 884 981 918 918 query71 346 305 294 294 query72 2808268424212421 query73 529 545 309 309 query74 9658957694469446 query75 2861274224772477 query76 22731030653 653 query77 352 371 312 312 query78 11073 11120 10475 10475 query79 1124773 570 570 query80 1333623 525 525 query81 547 255 222 222 query82 1000153 121 121 query83 330 261 239 239 query84 249 119 93 93 query85 918 488 443 443 query86 411 316 294 294 query87 3157311930113011 query88 3540263926602639 query89 434 362 348 348 query90 2029168 176 168 query91 174 166 139 139 query92 82 74 69 69 query93 956 836 494 494 query94 639 337 308 308 query95 599 355 324 324 query96 645 526 227 227 query97 2431251024042404 query98 235 219 231 219 query99 1016980 916 916 Total cold run time: 250153 ms Total hot run time: 168987 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected].
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
doris-robot commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4141503763 TPC-H: Total hot run time: 26787 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 6b7db78d6bdcb9b0046caa5f752c067a7465ebe3, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17590 458643584358 q2 q3 10652 798 536 536 q4 4684358 252 252 q5 7575121910281028 q6 184 177 152 152 q7 809 856 681 681 q8 9320153813781378 q9 4889472047024702 q10 6315194516521652 q11 509 265 246 246 q12 703 592 467 467 q13 18035 272719251925 q14 233 236 220 220 q15 q16 756 744 670 670 q17 737 843 471 471 q18 6227547953115311 q19 1113981 628 628 q20 546 504 383 383 q21 4306184014401440 q22 586 392 287 287 Total cold run time: 95769 ms Total hot run time: 26787 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4749464645964596 q2 q3 3900435738323832 q4 888 1181782 782 q5 4066437444334374 q6 188 179 144 144 q7 1884165615321532 q8 2582271325942594 q9 7833731775507317 q10 3877397336273627 q11 498 445 411 411 q12 485 584 460 460 q13 2515294021132113 q14 290 346 288 288 q15 q16 726 782 737 737 q17 1210135313451345 q18 6978675766296629 q19 909 887 936 887 q20 2069213219721972 q21 3934349035373490 q22 475 420 374 374 Total cold run time: 50056 ms Total hot run time: 47504 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
github-actions[bot] commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4140563692 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
github-actions[bot] commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4140563576 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
freemandealer commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4140526804 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
freemandealer commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4035717894 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
hello-stephen commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4006057573 # BE Regression && UT Coverage Report Increment line coverage `100.00% (23/23)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61083_cbe632c41f6e353337e80523342e3349b42e24e4_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61083_cbe632c41f6e353337e80523342e3349b42e24e4_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 73.16% (26779/36603) | | Line Coverage | 56.52% (285692/505448) | | Region Coverage | 54.15% (239466/442248) | | Branch Coverage | 55.59% (102965/185234) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
doris-robot commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4005602373 # BE UT Coverage Report Increment line coverage `100.00% (23/23)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/cbe632c41f6e353337e80523342e3349b42e24e4_cbe632c41f6e353337e80523342e3349b42e24e4/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/cbe632c41f6e353337e80523342e3349b42e24e4_cbe632c41f6e353337e80523342e3349b42e24e4/report/index.html) | Category | Coverage | |---|| | Function Coverage | 52.56% (19648/37381) | | Line Coverage | 36.18% (183435/507001) | | Region Coverage | 32.51% (142408/438111) | | Branch Coverage | 33.44% (61754/184670) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
doris-robot commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4004847212 TPC-DS: Total hot run time: 152871 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit cbe632c41f6e353337e80523342e3349b42e24e4, data reload: false query5 4336642 521 521 query6 339 224 204 204 query7 4228480 264 264 query8 349 263 241 241 query9 8750282227802780 query10 533 400 374 374 query11 7283581156225622 query12 196 126 123 123 query13 1261460 361 361 query14 5690379635843584 query14_12846282928402829 query15 209 202 176 176 query16 973 482 475 475 query17 1118722 623 623 query18 2447457 357 357 query19 219 223 183 183 query20 136 133 126 126 query21 227 146 140 140 query22 4872500247414741 query23 15976 15533 15347 15347 query23_116196 15852 15945 15852 query24 7452168613491349 query24_11279126613141266 query25 593 522 490 490 query26 1419282 165 165 query27 3106540 302 302 query28 5004188618701870 query29 817 555 478 478 query30 304 244 208 208 query31 1344129612281228 query32 80 71 74 71 query33 503 338 267 267 query34 948 922 557 557 query35 627 672 593 593 query36 10941095951 951 query37 133 94 79 79 query38 2935290429322904 query39 934 851 838 838 query39_1824 829 826 826 query40 225 154 139 139 query41 66 69 67 67 query42 303 300 293 293 query43 242 241 224 224 query44 query45 199 197 180 180 query46 901 994 601 601 query47 2137216020462046 query48 315 319 230 230 query49 628 458 390 390 query50 683 287 214 214 query51 4096411640264026 query52 287 302 279 279 query53 292 332 282 282 query54 296 281 267 267 query55 92 88 82 82 query56 306 326 308 308 query57 1346134512641264 query58 283 278 281 278 query59 1342146212771277 query60 336 344 319 319 query61 158 145 152 145 query62 635 598 540 540 query63 314 279 272 272 query64 49941260991 991 query65 query66 1452458 351 351 query67 16162 16382 16320 16320 query68 query69 390 319 281 281 query70 933 967 929 929 query71 342 303 301 301 query72 2845268224622462 query73 544 563 318 318 query74 9990993697499749 query75 2836274924592459 query76 22791036655 655 query77 359 375 297 297 query78 11246 11436 10732 10732 query79 1164760 629 629 query80 1462646 534 534 query81 560 274 247 247 query82 1003148 116 116 query83 354 273 270 270 query84 245 117 108 108 query85 1216473 432 432 query86 413 284 298 284 query87 3126309730103010 query88 3509267726722672 query89 415 371 346 346 query90 1881173 173 173 query91 176 160 143 143 query92 78 74 77 74 query93 977 860 519 519 query94 675 333 302 302 query95 588 351 390 351 query96 634 509 229 229 query97 2473250324232423 query98 239 223 222 222 query99 10071002867 867 Total cold run time: 233914 ms Total hot run time: 152871 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected].
Re: [PR] [fix](filecache) add async lru update machanism and fix partial hit in cache reader [doris]
doris-robot commented on PR #61083: URL: https://github.com/apache/doris/pull/61083#issuecomment-4004790840 TPC-H: Total hot run time: 27941 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit cbe632c41f6e353337e80523342e3349b42e24e4, data reload: false -- Round 1 -- q1 17635 442242834283 q2 q3 10665 829 530 530 q4 4678390 267 267 q5 7568119210351035 q6 175 179 148 148 q7 784 852 671 671 q8 9293150213781378 q9 4853474347754743 q10 6316190816651665 q11 471 261 241 241 q12 738 564 472 472 q13 18054 294221662166 q14 229 236 223 223 q15 944 807 807 807 q16 786 723 681 681 q17 713 852 427 427 q18 5961544153665366 q19 11271000612 612 q20 513 495 405 405 q21 4448218115441544 q22 438 361 277 277 Total cold run time: 96389 ms Total hot run time: 27941 ms - Round 2, with runtime_filter_mode=off - q1 4657453845754538 q2 q3 3877433938943894 q4 873 1226774 774 q5 4055439343744374 q6 192 180 147 147 q7 1822164315421542 q8 2465287325782578 q9 7794738973207320 q10 3727399035663566 q11 519 434 445 434 q12 528 622 450 450 q13 2812320023232323 q14 287 295 279 279 q15 843 795 816 795 q16 711 778 756 756 q17 1159145013581358 q18 7183675667056705 q19 891 870 889 870 q20 2104229419951995 q21 3967350633073307 q22 436 420 401 401 Total cold run time: 50902 ms Total hot run time: 48406 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
