Hi Hung,
Thank You for pointing this out. It works after adding the file:/// scheme.
there are couple of different config controlling the job conf dir location
jobconf.dir
jobconf.fullyQualifiedPath
job.conf.path ( with gobblin.cluster or gobblin.yarn )
Do you think we should collapse this into 1 config?
Thanks
Jay
Hi Jay,
>
> That code is not using fs.uri. Can you check whether the proper scheme is
> specified for this.jobConfDirPath?
>
> In the following case it is added to the configured path.
>
> else if (this.cfg.hasPath(ConfigurationKeys.JOB_CONFIG_FILE_DIR_KEY)) {
> File localJobConfigDir = new
> File(this.cfg.getString(ConfigurationKeys.JOB_CONFIG_FILE_DIR_KEY));
> this.jobConfDir = "file://" + localJobConfigDir.getAbsolutePath(); }
>
> If you are using ConfigurationKeys.JOB_CONFIG_FILE_GENERAL_PATH_KEY then
> you
> would need to add the scheme.
>
> Hung.
>
>
On Fri, Apr 12, 2019 at 5:39 PM Jay Sen <[email protected]> wrote:
> Hi Dev Team,
>
> I am seeing that even though fs.uri="file:///" it gets the hdfs
> FileSystem probably because i have HADOOP_HOME is set and hadoop config is
> getting picked up. is that right or I m missing something here ?
> for ex: this line this.jobConfDirFileSystem =
> this.jobConfDirPath.getFileSystem(new Configuration()); should get local
> file system when fs.uri is set to "file:///" right?
>
> basically we are using hadooo configuration class to get the utilmate
> configs and Configuration.get(name="fs.defaultFS", defaultValue="file:///")
> will always get fs.defaultFS from core-site.xml since that is in the
> classpath.
>
> can somebody please verify and tell if there is a way to enforce
> FileSystem to be local ( basically usefs.uri from gobblin config) ?
>
> Thanks
> Jay
>
