PyRSA commented on code in PR #11037:
URL: https://github.com/apache/paimon/pull/11037#discussion_r4510738798
##########
paimon-filesystems/paimon-s3-impl/src/main/java/org/apache/paimon/s3/S3FileIO.java:
##########
@@ -115,34 +121,162 @@ private Options mirrorCertainHadoopConfig(Options
hadoopConfig) {
return hadoopConfig;
}
+ /**
+ * Overrides parent to refresh {@link #fsMap} when a cached {@link
RecoverableS3AFileSystem} has
+ * been invalidated. Avoids introducing a generic health hook on {@link
HadoopCompliantFileIO}.
+ */
+ @Override
+ protected FileSystem getFileSystem(org.apache.hadoop.fs.Path path) throws
IOException {
+ if (fsMap == null) {
+ synchronized (this) {
+ if (fsMap == null) {
+ fsMap = new ConcurrentHashMap<>();
+ }
+ }
+ }
+
+ final Map<Pair<String, String>, FileSystem> map = fsMap;
+ URI uri = path.toUri();
+ Pair<String, String> key = Pair.of(uri.getScheme(),
uri.getAuthority());
+
+ // Create outside fsMap's ConcurrentHashMap compute lock.
+ while (true) {
+ FileSystem current = map.get(key);
+ if (isUsableFileSystem(current)) {
+ return current;
+ }
+
+ FileSystem created = createFileSystem(path);
+ if (!isUsableFileSystem(created)) {
+ continue;
+ }
+
+ if (current == null) {
+ FileSystem existing = map.putIfAbsent(key, created);
+ if (existing == null) {
+ if (isUsableFileSystem(created)) {
+ return created;
+ }
+ map.remove(key, created);
+ continue;
+ }
+ if (isUsableFileSystem(existing)) {
+ return existing;
+ }
+ current = existing;
+ }
+
+ if (isUsableFileSystem(current)) {
+ return current;
+ }
+ if (!map.replace(key, current, created)) {
Review Comment:
Thanks for raising this. I investigated the filesystem lifecycle and also
looked into the previous discussions in #8962 and #9472, where similar
filesystem cleanup and ownership issues were discussed.
In our case, the S3A filesystem is shared through a static cache across
multiple `S3FileIO` instances. Replacing the entry in one instance's `fsMap`
does not guarantee that the old filesystem is no longer referenced by other
instances or in-flight streams. Closing it immediately could therefore
interrupt active operations.
As discussed in those PRs, safely closing shared filesystem instances
requires explicit lifecycle coordination, such as reference counting or leases.
The current FileIO design does not provide an end-to-end ownership or lease
contract.
For this reason, I think keeping the current eviction behavior is safer than
closing a filesystem whose lifecycle may still be owned by other users. The
existing cleanup of unpublished instances remains unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]