Akash3121 commented on code in PR #11037:
URL: https://github.com/apache/paimon/pull/11037#discussion_r4508932363
##########
paimon-filesystems/paimon-s3-impl/src/main/java/org/apache/paimon/s3/S3FileIO.java:
##########
@@ -115,34 +121,162 @@ private Options mirrorCertainHadoopConfig(Options
hadoopConfig) {
return hadoopConfig;
}
+ /**
+ * Overrides parent to refresh {@link #fsMap} when a cached {@link
RecoverableS3AFileSystem} has
+ * been invalidated. Avoids introducing a generic health hook on {@link
HadoopCompliantFileIO}.
+ */
+ @Override
+ protected FileSystem getFileSystem(org.apache.hadoop.fs.Path path) throws
IOException {
+ if (fsMap == null) {
+ synchronized (this) {
+ if (fsMap == null) {
+ fsMap = new ConcurrentHashMap<>();
+ }
+ }
+ }
+
+ final Map<Pair<String, String>, FileSystem> map = fsMap;
+ URI uri = path.toUri();
+ Pair<String, String> key = Pair.of(uri.getScheme(),
uri.getAuthority());
+
+ // Create outside fsMap's ConcurrentHashMap compute lock.
+ while (true) {
+ FileSystem current = map.get(key);
+ if (isUsableFileSystem(current)) {
+ return current;
+ }
+
+ FileSystem created = createFileSystem(path);
+ if (!isUsableFileSystem(created)) {
+ continue;
+ }
+
+ if (current == null) {
+ FileSystem existing = map.putIfAbsent(key, created);
+ if (existing == null) {
+ if (isUsableFileSystem(created)) {
+ return created;
+ }
+ map.remove(key, created);
+ continue;
+ }
+ if (isUsableFileSystem(existing)) {
+ return existing;
+ }
+ current = existing;
+ }
+
+ if (isUsableFileSystem(current)) {
+ return current;
+ }
+ if (!map.replace(key, current, created)) {
Review Comment:
Published unusable filesystem is replaced but never closed.
After this replacement succeeds, `current` is the unusable published
`S3AFileSystem`, but it is never closed. Once the remaining `S3FileIO` maps
refresh, that instance can become unreachable without running
`S3AFileSystem.close()`, which normally shuts down its store/client, executor
pools, audit/signer state, credentials, and other close-managed resources.
Repeated pool failures can therefore accumulate resources. Could we safely
close the displaced unusable instance after replacement - outside the SDK
callback - and add a close-tracking regression test?
Refs:
-`RecoverableS3AFileSystem.java:40-43` explicitly documents the behavior
-`S3FileIO.java:252-261` has the equivalent omission when replacement
initialization fails: `expectedOld` is removed, but only `unpublished` is closed
-`S3FileIO.java:227`
##########
paimon-filesystems/paimon-s3-impl/src/main/java/org/apache/paimon/s3/RecoverableS3AFileSystem.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.s3;
+
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.fs.s3a.S3ClientFactory;
+import org.apache.hadoop.fs.s3a.impl.ClientManager;
+import org.apache.hadoop.fs.statistics.DurationTrackerFactory;
+import software.amazon.awssdk.core.interceptor.Context;
+import software.amazon.awssdk.core.interceptor.ExecutionAttributes;
+import software.amazon.awssdk.core.interceptor.ExecutionInterceptor;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * {@link S3AFileSystem} that evicts itself from {@link S3FileIO}'s cache when
the sync Apache HTTP
+ * connection pool has been shut down.
+ *
+ * <p>Only handles {@code IllegalStateException("Connection pool shut down")}
from the Apache HTTP
+ * client. Netty async failures are out of scope. Previously published
instances are not closed, and
+ * failed requests are not retried automatically; callers must retry and
obtain a recovered
+ * filesystem through {@link S3FileIO}.
+ */
+class RecoverableS3AFileSystem extends S3AFileSystem {
Review Comment:
Existing scheme test rejects the new subclass:
Introducing this subclass changes the runtime class name, but
`paimon-filesystems/paimon-s3/src/test/java/org/apache/paimon/s3/S3ASchemeTest.java:145`
`assertEquals("org.apache.hadoop.fs.s3a.S3AFileSystem",
fs.getClass().getName());` still requires the exact name
`org.apache.hadoop.fs.s3a.S3AFileSystem`. Both parameterized scheme cases
consequently fail in the required JDK 8 and JDK 11 CI jobs. Please update that
test to verify S3A ancestry/behavior without requiring the old concrete class
name; note that this plugin test may use a separate classloader, so a simple
cross-classloader `instanceof` may not be reliable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]