yuqi1129 commented on code in PR #7782:
URL: https://github.com/apache/gravitino/pull/7782#discussion_r2242400440
##########
catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:
##########
@@ -129,8 +130,71 @@ public class FilesetCatalogOperations extends
ManagedSchemaOperations
private boolean disableFSOps;
+ @VisibleForTesting ScheduledThreadPoolExecutor scheduler;
+ @VisibleForTesting Cache<FileSystemCacheKey, FileSystem> fileSystemCache;
+
FilesetCatalogOperations(EntityStore store) {
this.store = store;
+ scheduler =
+ new ScheduledThreadPoolExecutor(
+ 1,
+ new ThreadFactoryBuilder()
+ .setDaemon(true)
+ .setNameFormat("file-system-cache-for-fileset" + "-%d")
+ .build());
+
+ this.fileSystemCache =
+ Caffeine.newBuilder()
+ .expireAfterAccess(1000 * 60 * 60 /* 1 hour */,
TimeUnit.MILLISECONDS)
+ .removalListener(
+ (ignored, value, cause) -> {
+ try {
+ ((FileSystem) value).close();
+ } catch (IOException e) {
+ LOG.warn("Failed to close FileSystem instance in cache",
e);
+ }
+ })
+ .scheduler(Scheduler.forScheduledExecutorService(scheduler))
+ .build();
+ }
+
+ static class FileSystemCacheKey {
+ private final NameIdentifier ident;
+ private final Map<String, String> conf;
+ private final String currentUser;
+
+ FileSystemCacheKey(NameIdentifier ident, Map<String, String> conf) {
Review Comment:
Do you mean that the files under a fileset are different types or different
clusters?
```
fileset1:
location1: hdfs://ip1/path
location2: hdfs://ip2/path
location3: hdfs://ip3/path
```
fileset2
```
location1: file://ip1/path
location2: hdfs://ip2/path
location3: s3://bucket/path
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
