Hi Jay,
We try to maintain backwards compatibility as much as possible, so config keys are usually not removed. You can see another case of this in DefaultLimiterFactory.convertDeprecatedConfigs(). Hung. ________________________________ From: Jay Sen <[email protected]> Sent: Wednesday, April 17, 2019 7:52:27 PM To: [email protected] Subject: Re: fs.uri config not working with hadoop installed locally Hi Hung, Thank You for pointing this out. It works after adding the file:/// scheme. there are couple of different config controlling the job conf dir location jobconf.dir jobconf.fullyQualifiedPath job.conf.path ( with gobblin.cluster or gobblin.yarn ) Do you think we should collapse this into 1 config? Thanks Jay Hi Jay, > > That code is not using fs.uri. Can you check whether the proper scheme is > specified for this.jobConfDirPath? > > In the following case it is added to the configured path. > > else if (this.cfg.hasPath(ConfigurationKeys.JOB_CONFIG_FILE_DIR_KEY)) { > File localJobConfigDir = new > File(this.cfg.getString(ConfigurationKeys.JOB_CONFIG_FILE_DIR_KEY)); > this.jobConfDir = "file://" + localJobConfigDir.getAbsolutePath(); } > > If you are using ConfigurationKeys.JOB_CONFIG_FILE_GENERAL_PATH_KEY then > you > would need to add the scheme. > > Hung. > > On Fri, Apr 12, 2019 at 5:39 PM Jay Sen <[email protected]> wrote: > Hi Dev Team, > > I am seeing that even though fs.uri="file:///" it gets the hdfs > FileSystem probably because i have HADOOP_HOME is set and hadoop config is > getting picked up. is that right or I m missing something here ? > for ex: this line this.jobConfDirFileSystem = > this.jobConfDirPath.getFileSystem(new Configuration()); should get local > file system when fs.uri is set to "file:///" right? > > basically we are using hadooo configuration class to get the utilmate > configs and Configuration.get(name="fs.defaultFS", defaultValue="file:///") > will always get fs.defaultFS from core-site.xml since that is in the > classpath. > > can somebody please verify and tell if there is a way to enforce > FileSystem to be local ( basically usefs.uri from gobblin config) ? > > Thanks > Jay >
