davidradl commented on code in PR #27617:
URL: https://github.com/apache/flink/pull/27617#discussion_r2811841349
##########
flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java:
##########
@@ -338,14 +355,48 @@ public static void initialize(Configuration config,
@Nullable PluginManager plug
final List<FileSystemFactory> fileSystemFactories =
loadFileSystemFactories(factorySuppliers);
+ // Track registered priorities for factory selection
+ final HashMap<String, Integer> registeredPriorities = new
HashMap<>();
+
// configure all file system factories
for (FileSystemFactory factory : fileSystemFactories) {
factory.configure(config);
- String scheme = factory.getScheme();
+ final String scheme = factory.getScheme();
FileSystemFactory fsf =
ConnectionLimitingFactory.decorateIfLimited(factory,
scheme, config);
- FS_FACTORIES.put(scheme, fsf);
+
+ final String className = resolveFactoryClassName(factory);
+ final int registeredPriority =
+ registeredPriorities.getOrDefault(scheme,
Integer.MIN_VALUE);
+ final int newPriority =
+
config.getOptional(CoreOptions.fileSystemFactoryPriority(scheme, className))
+ .orElse(factory.getPriority());
+
+ LOG.info(
+ "{} filesystem factory {} for scheme '{}' "
+ + "with priority {} (highest registered
priority: {})",
+ newPriority >= registeredPriority ? "Registering" :
"Skipping",
+ className,
+ scheme,
+ newPriority,
+ registeredPriority);
+ if (newPriority >= registeredPriority) {
Review Comment:
There is no write up for this. The Flip and Jira do not mention priority.
The example in PR says "For new experimental fs factories, suggested default is
-1.". So I assume that this will not be picked up unless -1 is specified in
config
Please could you write up more about the motivation for this change
- how to associate a file system with a priority
- are we likely to have more that 2 matching schemes in you use case.
- why it is be useful to have multiple priorities to justify the if
`newPriority >= registeredPriority `. Rather than just targeting the required
priority
- is the concept of priority different from a version? Would it not be
better to just target a scheme version.
- without the config; how do we determine which file system is the highest
priority - I assume this could be different for different users; this implies
that one file system scheme is inherently higher priority than another.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
