wombatu-kun commented on code in PR #8962:
URL: https://github.com/apache/paimon/pull/8962#discussion_r3705157640
##########
paimon-common/src/main/java/org/apache/paimon/fs/hadoop/HadoopFileIO.java:
##########
@@ -226,6 +259,64 @@ protected FileSystem
createFileSystem(org.apache.hadoop.fs.Path path) throws IOE
return fileSystem;
}
+ /**
+ * Whether the {@link FileSystem} instances created for the given scheme
belong to this {@link
+ * FileIO} exclusively, and may therefore be closed by it.
+ *
+ * <p>This mirrors the branch Hadoop itself takes in {@code
FileSystem#get(URI, Configuration)}:
+ * with {@code fs.<scheme>.impl.disable.cache} set, Hadoop hands out a
fresh instance that
+ * nobody else can reach, so releasing it is our responsibility. Otherwise
the instance lives in
+ * Hadoop's global cache and is shared with every other user in this JVM,
including other {@link
+ * FileIO}s and the compute engine itself; {@code FileSystem#closeAll}
releases those on
+ * shutdown and closing one here would break unrelated readers.
+ *
+ * <p>The scheme is the one taken from the path, not from {@code
FileSystem#getUri()}, and it is
+ * matched as written rather than lower cased, because that is what Hadoop
looks up. Any
+ * deviation could report a cached, shared instance as owned.
+ */
+ @VisibleForTesting
+ boolean isOwnedScheme(@Nullable String scheme) {
+ if (hadoopConf == null) {
+ return false;
+ }
+ Configuration conf = hadoopConf.get();
+ if (scheme == null) {
+ // a path without a scheme is served by the default file system
+ try {
+ scheme = FileSystem.getDefaultUri(conf).getScheme();
+ } catch (IllegalArgumentException e) {
+ // a missing or malformed fs.defaultFS, so there is no scheme
to claim ownership of
+ return false;
+ }
+ }
+ return conf.getBoolean(String.format("fs.%s.impl.disable.cache",
scheme), false);
+ }
+
+ @Override
+ public void close() throws IOException {
+ List<FileSystem> owned = new ArrayList<>();
+ synchronized (this) {
+ closed = true;
+ Map<Pair<String, String>, FileSystem> map = fsMap;
+ if (map == null) {
+ return;
+ }
+ for (Map.Entry<Pair<String, String>, FileSystem> entry :
map.entrySet()) {
+ if (isOwnedScheme(entry.getKey().getLeft())) {
Review Comment:
Done bd9a8d990. The map holds `{fileSystem, ownedAtCreation}` and both the
loser cleanup and `close()` read that bit. The same pass stops recording an
externally injected file system as owned, releases the raw instance when the
Kerberos wrapper fails, and moves the loser's close outside the monitor so a
slow object store teardown cannot stall the other callers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
