LuciferYang opened a new issue, #9641: URL: https://github.com/apache/paimon/issues/9641
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `475be566f` (2.1-SNAPSHOT). ### Compute Engine Any engine with `local-cache.enabled = true` and no `local-cache.dir`, which is the memory cache mode. ### Minimal reproduce step Read a consumer file, let a consumer reset overwrite it, and read it again through the same `CachingFileIO`. The second read returns the first version. `newInputStream` keys the memory cache by path alone, while the disk branch keys by path, length and modification time: ```java if (c instanceof LocalDiskCacheManager) { FileStatus status = delegate.getFileStatus(path); return new CachingSeekableInputStream( delegate, path, c, diskCacheKey(path, status), status.getLen()); } return new CachingSeekableInputStream(delegate, path, c, cacheNamespace + ":" + path, -1); ``` `consumer-*` and `service-*` classify as `META`, which is in the default whitelist (`meta,global-index`), and `FileType.isMutable` only excludes `EARLIEST` and `LATEST`: ```java public static boolean isMutable(Path filePath) { String name = filePath.getName(); return "EARLIEST".equals(name) || "LATEST".equals(name); } ``` Those two kinds of file are exactly the ones written in place, `ConsumerManager.resetConsumer` and `ServiceManager.resetService` both go through `overwriteFileUtf8`. So after a reset, `Consumer.fromPath` keeps reading the pre-reset `nextSnapshot` out of the cache. Nothing invalidates it: the cached file size is pinned per path as well. ### What doesn't meet your expectations? A cache keyed only by path cannot notice an in-place overwrite, and the whitelist deliberately contains the metadata files, some of which are overwritten in place. The disk mode already keys by length and modification time; the memory mode should not be weaker. Blacklisting the two prefixes in `isMutable` would fix the consumer case more cheaply, but in-place overwrite is not limited to them: `TagManager.createOrReplaceTag` overwrites `tag-*`, the Iceberg metadata files go through `overwriteFileUtf8`, and so does `_SUCCESS`. A prefix list has to be maintained forever; a version in the key does not care about prefixes. ### Anything else? Two limits worth stating. Versioning by length and modification time costs one `getFileStatus` per open in memory mode, where the size used to be resolved lazily on first read and then cached per path; on an object store that is one HEAD per open. And a delegate whose modification time has second granularity can still collide when a rewrite lands in the same second with the same length, which is true of the disk mode today as well. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
