AHeise commented on a change in pull request #16559:
URL: https://github.com/apache/flink/pull/16559#discussion_r679892788
##########
File path:
flink-filesystems/flink-azure-fs-hadoop/src/main/java/org/apache/flink/fs/azurefs/AbstractAzureFSFactory.java
##########
@@ -75,18 +76,27 @@ public void configure(Configuration config) {
@Override
public FileSystem create(URI fsUri) throws IOException {
checkNotNull(fsUri, "passed file system URI object should not be
null");
- LOG.info("Trying to load and instantiate Azure File System");
+ LOG.info("Trying to load and instantiate Azure File System for {}",
fsUri);
return new HadoopFileSystem(createInitializedAzureFS(fsUri,
flinkConfig));
}
- // uri is of the form:
wasb(s)://[email protected]/testDir
+ // uri is of the form:
wasb(s)://[email protected]/testDir (or)
+ // abfs(s):////[email protected]/testDir
private org.apache.hadoop.fs.FileSystem createInitializedAzureFS(
URI fsUri, Configuration flinkConfig) throws IOException {
org.apache.hadoop.conf.Configuration hadoopConfig =
configLoader.getOrLoadHadoopConfig();
-
- org.apache.hadoop.fs.FileSystem azureFS = new NativeAzureFileSystem();
- azureFS.initialize(fsUri, hadoopConfig);
-
- return azureFS;
+ String scheme = fsUri.getScheme();
+
+ if (scheme.startsWith("wasb")) {
Review comment:
How about adding a
```abstract org.apache.hadoop.fs.FileSystem createAzureFS();```
method to this factory and avoid this if-else?
Then the abstract factory doesn't need to know the scheme of the
implementations.
##########
File path:
flink-filesystems/flink-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE
##########
@@ -33,7 +48,7 @@ See bundled license files for details.
- org.codehaus.woodstox:stax2-api:3.1.4
(https://github.com/FasterXML/stax2-api/tree/stax2-api-3.1.4)
-This project bundles org.apache.hadoop:*:3.1.0 from which it inherits the
following notices:
+This project bundles org.apache.hadoop:*:3.3.1 from which it inherits the
following notices:
The Apache Hadoop project contains subcomponents with separate copyright
Review comment:
did you check if the notices changed?
##########
File path:
flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE
##########
@@ -34,14 +34,25 @@ This project bundles the following dependencies under the
Apache Software Licens
- joda-time:joda-time:2.5
- org.apache.commons:commons-configuration2:2.1.1
- org.apache.commons:commons-lang3:3.3.2
-- org.apache.hadoop:hadoop-annotations:3.1.0
-- org.apache.hadoop:hadoop-aws:3.1.0
-- org.apache.hadoop:hadoop-auth:3.1.0
-- org.apache.hadoop:hadoop-common:3.1.0
+- org.apache.commons:commons-text:1.4
+- org.apache.hadoop:hadoop-annotations:3.3.1
+- org.apache.hadoop:hadoop-aws:3.3.1
+- org.apache.hadoop:hadoop-auth:3.3.1
+- org.apache.hadoop:hadoop-common:3.3.1
+- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
+- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1
- org.apache.htrace:htrace-core4:4.1.0-incubating
-- org.apache.httpcomponents:httpcore:4.4.14
- org.apache.httpcomponents:httpclient:4.5.13
+- org.apache.httpcomponents:httpcore:4.4.14
+- org.apache.kerby:kerby-asn1:1.0.1
+- org.apache.kerby:kerb-core:1.0.1
+- org.apache.kerby:kerby-pkix:1.0.1
+- org.apache.kerby:kerby-util:1.0.1
+- org.codehaus.woodstox:stax2-api:4.2.1
+- org.xerial.snappy:snappy-java:1.1.8.3
- org.weakref:jmxutils:1.19
+- org.wildfly.openssl:wildfly-openssl:1.0.7.Final
+- dnsjava:dnsjava:2.1.7
Review comment:
Did you verify that this is indeed ASL?
##########
File path:
flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/HadoopS3AccessHelper.java
##########
@@ -52,8 +55,22 @@
private final InternalWriteOperationHelper s3accessHelper;
public HadoopS3AccessHelper(S3AFileSystem s3a, Configuration conf) {
Review comment:
The changes in this file are non-obvious to me: is it necessary because
of the update or is this an additional feature? If it's the latter, please
factor out into another commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
