wenzhenghu commented on code in PR #57410:
URL: https://github.com/apache/doris/pull/57410#discussion_r2516407140
##########
be/src/common/config.cpp:
##########
@@ -1169,7 +1169,9 @@
DEFINE_mInt64(file_cache_background_lru_dump_update_cnt_threshold, "1000");
DEFINE_mInt64(file_cache_background_lru_dump_tail_record_num, "5000000");
DEFINE_mInt64(file_cache_background_lru_log_replay_interval_ms, "1000");
DEFINE_mBool(enable_evaluate_shadow_queue_diff, "false");
-
+DEFINE_mBool(enable_file_cache_normal_queue_2qlru, "true");
Review Comment:
default value better to be false as it is not widely used.
----
ok, LGTM
is there any test results to show how much this mechanism improves the hit
rate?
----
The test results indicate that after enabling the 2Q-LRU mechanism, both the
cache hit rate and query performance have improved, with a more significant
performance improvement observed under low cache allocation ratios. The testing
utilized the TPCH 100 (about 25GB data should cache) benchmark. The result data
is as follows:
| Cache Configuration | Single LRU Hit Rate (%) | Single LRU
Performance (Relative) | Dual LRU Hit Rate (%) | Dual LRU Performance
(Relative) |
|:----------------------|--------------------------:|------------------------------------:|------------------------:|----------------------------------:|
| 20G | 92.74 |
1 | 96.11 | 1.04 |
| 15G | 73.32 |
1 | 82.33 | 1.12 |
| 10G | 40.23 |
1 | 62.06 | 1.54 |
is there any upgrade/downgrade compatibility issues if a new queue
cold_normal and new file name _cold_normal introduced?
----
Thank you for the reminder. We acknowledge that the compatibility
implementation for scenarios involving switching between single LRU and dual
LRU during dump and load operations is not yet fully complete. We plan to
address this as follows:[in progress]
- When dumping with a single LRU queue and loading with a dual LRU queue:
In this scenario, all the loaded data enters the NORMAL queue. Because the
available cache space for itself is reduced, the NORMAL queue will (according
to the existing mechanism of the filecache) borrow space from the COLD_NORMAL
lqueue.
- When dumping with a dual LRU queue and loading with a single LRU queue:
It is necessary to add the data from the COLD_NORMAL dump file to the NORMAL
queue according to the LRU order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
