Hi Jay,
That code is not using fs.uri. Can you check whether the proper scheme is
specified for this.jobConfDirPath?
In the following case it is added to the configured path.
else if (this.cfg.hasPath(ConfigurationKeys.JOB_CONFIG_FILE_DIR_KEY)) {
File localJobConfigDir = new
File(this.cfg.getString(ConfigurationKeys.JOB_CONFIG_FILE_DIR_KEY));
this.jobConfDir = "file://" + localJobConfigDir.getAbsolutePath();
}
If you are using ConfigurationKeys.JOB_CONFIG_FILE_GENERAL_PATH_KEY then you
would need to add the scheme.
Hung.
________________________________
From: Jay Sen <[email protected]>
Sent: Friday, April 12, 2019 5:39:02 PM
To: [email protected]
Subject: fs.uri config not working with hadoop installed locally
Hi Dev Team,
I am seeing that even though fs.uri="file:///" it gets the hdfs FileSystem
probably because i have HADOOP_HOME is set and hadoop config is getting
picked up. is that right or I m missing something here ?
for ex: this line this.jobConfDirFileSystem =
this.jobConfDirPath.getFileSystem(new Configuration()); should get local
file system when fs.uri is set to "file:///" right?
basically we are using hadooo configuration class to get the utilmate
configs and Configuration.get(name="fs.defaultFS", defaultValue="file:///")
will always get fs.defaultFS from core-site.xml since that is in the
classpath.
can somebody please verify and tell if there is a way to enforce FileSystem
to be local ( basically usefs.uri from gobblin config) ?
Thanks
Jay
