morningman commented on code in PR #69581:
URL: https://github.com/apache/doris/pull/69581#discussion_r4432225816
##########
be/src/common/config.cpp:
##########
@@ -1233,6 +1233,26 @@ DEFINE_Validator(variant_max_json_key_length,
DEFINE_Validator(variant_storage_parse_mode,
[](const int config) -> bool { return config >= 0 && config
<= 2; });
+// Lance uses one BE-wide session so metadata/index caches and the optional
Foyer data-file cache
+// can be shared by all Lance dataset readers.
+DEFINE_Int64(lance_index_cache_size_bytes, "10737418240"); // 10 GiB
+DEFINE_Int64(lance_metadata_cache_size_bytes, "1073741824"); // 1GB
+DEFINE_Bool(enable_lance_data_cache, "true");
+DEFINE_String(lance_data_cache_path, "${DORIS_HOME}/lance_data_cache");
+DEFINE_Int64(lance_data_cache_disk_capacity_bytes, "107374182400"); // 100GB
+DEFINE_Int64(lance_data_cache_read_block_size_bytes, "1048576"); // 1MB
Review Comment:
Fixed in `89afea9808c`. `LanceSessionManager::_initialize()` now rejects a
negative `lance_index_cache_size_bytes` or `lance_metadata_cache_size_bytes`,
and, while the data cache is enabled, a negative
`lance_data_cache_disk_capacity_bytes` or
`lance_data_cache_read_block_size_bytes`, before any `static_cast<uint64_t>`.
The InvalidArgument names the settings and tells the operator to fix be.conf
(or set `enable_lance_data_cache = false`) and restart. Before, a negative
index or metadata size was accepted by lance-c as a cache of 2^64 - 1 bytes,
that is no limit, with no error at all; a negative disk capacity that is a
multiple of 4096 was accepted the same way as an almost unlimited disk tier;
and other negative data-cache sizes failed with lance-c's complaint about the
wrapped value (`disk_capacity_bytes=18446744073709551615 must be a multiple of
4096 ...`).
The data-cache sizes stay ignored while the data cache is disabled, as
`SessionAndDataCacheConfigurationsAreIndependent` requires, and the remaining
constraints (a non-zero multiple of 4096, room for the memory-tier block and
two disk blocks) stay with lance-c, which reports them on the same path. On the
`call_once` part: keeping the first initialization status is intentional. These
settings are restart-only (`DEFINE_Int64`, read once when the manager is
created), the status says to restart after fixing them, and only Lance scans
fail while the rest of the BE keeps serving; a start-up validator would instead
stop the whole BE for a Lance-only mistake, also when the data cache is
disabled and its sizes are unused.
Covered by
`LanceSessionManagerTest.RejectsNegativeCacheSizesBeforeCreatingTheSession`:
each of the four negative sizes fails with INVALID_ARGUMENT and opens no
dataset. With both checks disabled only this test fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
