BlakeOrth commented on code in PR #18855:
URL: https://github.com/apache/datafusion/pull/18855#discussion_r2548179325
##########
datafusion/execution/src/cache/cache_unit.rs:
##########
@@ -111,64 +113,224 @@ impl CacheAccessor<Path, Arc<Statistics>> for
DefaultFileStatisticsCache {
///
/// Collected files metadata for listing files.
///
-/// Cache is not invalided until user calls [`Self::remove`] or
[`Self::clear`].
+/// # Internal details
+///
+/// The `capacity` parameter controls the maximum number of entries in the
cache, using a Least
+/// Recently Used eviction algorithm. When adding a new entry, if the total
number of entries in
+/// the cache exceeds `capacity`, the least recently used entries are evicted
until the total
+/// entries are lower than the `capacity`.
///
/// [`ListFilesCache`]: crate::cache::cache_manager::ListFilesCache
#[derive(Default)]
pub struct DefaultListFilesCache {
- statistics: DashMap<Path, Arc<Vec<ObjectMeta>>>,
+ state: Mutex<DefaultListFilesCacheState>,
+}
+
+impl DefaultListFilesCache {
+ pub fn new(capacity: usize, ttl: Duration) -> Self {
+ Self {
+ state: Mutex::new(DefaultListFilesCacheState::new(capacity, ttl)),
+ }
+ }
+
+ pub fn cache_limit(&self) -> usize {
+ self.state.lock().unwrap().capacity
+ }
+
+ pub fn cache_ttl(&self) -> Duration {
+ self.state.lock().unwrap().ttl
+ }
+}
+
+pub(super) const DEFAULT_LIST_FILES_CACHE_LIMIT: usize = 128 * 1024; // ~130k
objects
+pub(super) const DEFAULT_LIST_FILES_CACHE_TTL: Duration = Duration::new(600,
0); // 10min
Review Comment:
The actual default values here probably need to be discussed. These seemed
relatively sane to me, but any input here on what values these should have to
best accommodate a variety of workflows would be useful feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
