JingsongLi commented on code in PR #1267:
URL: https://github.com/apache/paimon-rust/pull/1267#discussion_r4478989345
##########
crates/paimon/src/io/cache/mod.rs:
##########
@@ -188,10 +193,28 @@ impl LocalCache {
BlockKey::with_namespace(&self.namespace, path, self.block_size,
block_index)
}
- pub(super) fn is_cacheable(&self, path: &str) -> bool {
+ /// Extension exclusions must not disable invalidation of existing shared
blocks.
+ pub(super) fn is_whitelisted(&self, path: &str) -> bool {
!FileType::is_mutable(path) &&
self.whitelist.contains(&FileType::classify(path))
}
+ pub(super) fn is_cacheable(&self, path: &str) -> bool {
+ if !self.is_whitelisted(path) {
+ return false;
+ }
+ if self.excluded_extensions.is_empty() {
+ return true;
+ }
+ let name = path.rsplit('/').next().unwrap_or(path);
Review Comment:
[P2] Match excluded extensions on the storage path actually opened
This checks the raw caller path. The built-in filesystem backend normalizes
trailing whitespace in OpenDAL, so `payload.blob ` opens the same object as
`payload.blob` but is classified with an extension of `blob `. With
`local-cache.whitelist=data` and `local-cache.exclude-extensions=blob`, the
alias still fills and hits the block cache.
I reproduced it through the public FileSystemCatalog/FileIO path on the
local merge with main. Both spellings read the same initial bytes. After
changing the source directly to distinguish cache hits, the canonical path read
the replacement bytes and the alias returned the cached original. A path
supplied by a FileIO caller can therefore cache the BLOB body the catalog
excluded.
Please apply extension matching after the same path normalization used for
the storage read, or reject aliases that resolve to a different file name.
Cover the alias and canonical path in the exclusion integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]