Kasper Sørensen created METAMODEL-219:
-----------------------------------------
Summary: HdfsResource's FileSystem should be createable based on
custom core-site.xml and hdfs-site.xml files
Key: METAMODEL-219
URL: https://issues.apache.org/jira/browse/METAMODEL-219
Project: Apache MetaModel
Issue Type: Improvement
Affects Versions: 4.4.1
Reporter: Kasper Sørensen
Today we always assume a very vanilla HDFS/Hadoop configuration which is based
on defaults and overriding the one property {code}fs.defaultFS{code}. This
behaviour is very limiting for the use of HdfsResource since many other custom
properties may be needed.
Instead we should offer a way where users of the class can specify a directory
with the Hadoop configuration files. This could eventually also allow for a
more flexibly style of representing the HDFS path, namely without the hostname
and port (which would simply be dynamically loaded from the configuration
file), like this:
{code}
hdfs:///path/to/file
{code}
How the configuration directory is provided should probably also be flexible. I
would suggest that we offer three ways:
(1) a new constructor option to provide the directory, and
(2) a system property that will be checked, and
(3) an environmental variable that will be checked (I suggest HADOOP_CONF_DIR
and YARN_CONF_DIR to replicate common environmental variable names for this).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
