zjffdu commented on a change in pull request #3411: ZEPPELIN-4272: Zeppelin
fails to use s3a configured for zeppelin.note…
URL: https://github.com/apache/zeppelin/pull/3411#discussion_r306240365
##########
File path:
zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/FileSystemStorage.java
##########
@@ -60,16 +63,42 @@
public FileSystemStorage(ZeppelinConfiguration zConf, String path) throws
IOException {
this.zConf = zConf;
this.hadoopConf = new Configuration();
+ URI zepConfigURI;
+ URI defaultFSURI;
+
+ zepConfigURI = getURI(path, "Failed to get URI path");
// disable checksum for local file system. because interpreter.json may be
updated by
// non-hadoop filesystem api
// disable caching for file:// scheme to avoid getting LocalFS which does
CRC checks
+
this.hadoopConf.setBoolean("fs.file.impl.disable.cache", true);
- this.hadoopConf.set("fs.file.impl", RawLocalFileSystem.class.getName());
+ String defaultFS = this.hadoopConf.get(FS_DEFAULTFS);
+ defaultFSURI = getURI(defaultFS, "Failed to get defaultFS URI");
+
+ if (isS3AFileSystem(defaultFSURI, zepConfigURI)) {
+ this.hadoopConf.set("fs.s3a.impl", S3AFileSystem.class.getName());
Review comment:
I am suggesting to remove it. I just want to confirm why hadoop didn't
detect the right FileSystem implementation. As my understanding, hadoop should
be able to detect the right FileSystem implementation based on the filesystem
schema (e.g. file://, hdfs:// and etc). So could you help investigate why it
doesn't work for s3. Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
