[
https://issues.apache.org/jira/browse/HDFS-15289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15289:
---------------------------------------
Description:
ViewFS provides flexibility to mount different filesystem types with mount
points configuration table. This approach is solving the scalability problems,
but users need to reconfigure the filesystem to ViewFS and to its scheme. This
will be problematic in the case of paths persisted in meta stores, ex: Hive. In
systems like Hive, it will store uris in meta store. So, changing the file
system scheme will create a burden to upgrade/recreate meta stores. In our
experience many users are not ready to change that.
Router based federation is another implementation to provide coordinated mount
points for HDFS federation clusters. Even though this provides flexibility to
handle mount points easily, this will not allow other(non-HDFS) file systems to
mount. So, this does not solve the purpose when users want to mount
external(non-HDFS) filesystems.
So, the problem here is: Even though many users want to adapt to the scalable
fs options available, technical challenges of changing schemes (ex: in meta
stores) in deployments are obstructing them.
So, we propose to allow hdfs scheme in ViewFS like client side mount system and
provision user to create mount links without changing URI paths.
I will upload detailed design doc shortly.
was:
ViewFS provides flexibility to mount different filesystem types with mount
points configuration table. Additionally viewFS provides flexibility to
configure any fs (not only HDFS) scheme in mount table mapping. This approach
is solving the scalability problems, but users need to reconfigure the
filesystem to ViewFS and to its scheme. This will be problematic in the case
of paths persisted in meta stores, ex: Hive. In systems like Hive, it will
store uris in meta store. So, changing the file system scheme will create a
burden to upgrade/recreate meta stores. In our experience many users are not
ready to change that.
Router based federation is another implementation to provide coordinated mount
points for HDFS federation clusters. Even though this provides flexibility to
handle mount points easily, this will not allow other(non-HDFS) file systems to
mount. So, this does not solve the purpose when users want to mount
external(non-HDFS) filesystems.
So, the problem here is: Even though many users want to adapt to the scalable
fs options available, technical challenges of changing schemes (ex: in meta
stores) in deployments are obstructing them.
So, we propose to allow hdfs scheme in ViewFS like client side mount system and
provision user to create mount links without changing URI paths.
I will upload detailed design doc shortly.
> Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table
> ---------------------------------------------------------------------
>
> Key: HDFS-15289
> URL: https://issues.apache.org/jira/browse/HDFS-15289
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: fs
> Affects Versions: 3.2.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Priority: Major
> Attachments: ViewFSOverloadScheme - V1.0.pdf, ViewFSOverloadScheme.png
>
>
> ViewFS provides flexibility to mount different filesystem types with mount
> points configuration table. This approach is solving the scalability
> problems, but users need to reconfigure the filesystem to ViewFS and to its
> scheme. This will be problematic in the case of paths persisted in meta
> stores, ex: Hive. In systems like Hive, it will store uris in meta store. So,
> changing the file system scheme will create a burden to upgrade/recreate meta
> stores. In our experience many users are not ready to change that.
> Router based federation is another implementation to provide coordinated
> mount points for HDFS federation clusters. Even though this provides
> flexibility to handle mount points easily, this will not allow
> other(non-HDFS) file systems to mount. So, this does not solve the purpose
> when users want to mount external(non-HDFS) filesystems.
> So, the problem here is: Even though many users want to adapt to the scalable
> fs options available, technical challenges of changing schemes (ex: in meta
> stores) in deployments are obstructing them.
> So, we propose to allow hdfs scheme in ViewFS like client side mount system
> and provision user to create mount links without changing URI paths.
> I will upload detailed design doc shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
