PyRSA commented on code in PR #11037:
URL: https://github.com/apache/paimon/pull/11037#discussion_r4510738798


##########
paimon-filesystems/paimon-s3-impl/src/main/java/org/apache/paimon/s3/S3FileIO.java:
##########
@@ -115,34 +121,162 @@ private Options mirrorCertainHadoopConfig(Options 
hadoopConfig) {
         return hadoopConfig;
     }
 
+    /**
+     * Overrides parent to refresh {@link #fsMap} when a cached {@link 
RecoverableS3AFileSystem} has
+     * been invalidated. Avoids introducing a generic health hook on {@link 
HadoopCompliantFileIO}.
+     */
+    @Override
+    protected FileSystem getFileSystem(org.apache.hadoop.fs.Path path) throws 
IOException {
+        if (fsMap == null) {
+            synchronized (this) {
+                if (fsMap == null) {
+                    fsMap = new ConcurrentHashMap<>();
+                }
+            }
+        }
+
+        final Map<Pair<String, String>, FileSystem> map = fsMap;
+        URI uri = path.toUri();
+        Pair<String, String> key = Pair.of(uri.getScheme(), 
uri.getAuthority());
+
+        // Create outside fsMap's ConcurrentHashMap compute lock.
+        while (true) {
+            FileSystem current = map.get(key);
+            if (isUsableFileSystem(current)) {
+                return current;
+            }
+
+            FileSystem created = createFileSystem(path);
+            if (!isUsableFileSystem(created)) {
+                continue;
+            }
+
+            if (current == null) {
+                FileSystem existing = map.putIfAbsent(key, created);
+                if (existing == null) {
+                    if (isUsableFileSystem(created)) {
+                        return created;
+                    }
+                    map.remove(key, created);
+                    continue;
+                }
+                if (isUsableFileSystem(existing)) {
+                    return existing;
+                }
+                current = existing;
+            }
+
+            if (isUsableFileSystem(current)) {
+                return current;
+            }
+            if (!map.replace(key, current, created)) {

Review Comment:
   Thanks for raising this. I investigated the filesystem lifecycle and also 
looked into the previous discussions in #8962 and #9472, where similar 
filesystem cleanup and ownership issues were discussed.
   
   In our case, the S3A filesystem is shared through a static cache across 
multiple `S3FileIO` instances. Replacing the entry in one instance's `fsMap` 
does not guarantee that the old filesystem is no longer referenced by other 
instances or in-flight streams. Closing it immediately could therefore 
interrupt active operations.
   
   As discussed in those PRs, safely closing shared filesystem instances 
requires explicit lifecycle coordination, such as reference counting or leases. 
The current FileIO design does not provide an end-to-end ownership or lease 
contract.
   
   For this reason, I think keeping the current eviction behavior is safer than 
closing a filesystem whose lifecycle may still be owned by other users. The 
existing cleanup of unpublished instances remains unchanged.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to