[
https://issues.apache.org/jira/browse/HDFS-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190917#comment-17190917
]
Fengnan Li commented on HDFS-15554:
-----------------------------------
Thanks for the pointer [~ayushtkn]!
I think both tickets are useful. Besides the shared consistency goal,
HDFS-13506 is saving one command. However there are many cases where the
downstream dirs should not be created. In our use case we are dealing with
daily ingestion data where for some days there is no data and creating an empty
dir will cause confusion for others. On the other hand, if we just create a dir
per mount point we would end up with a lot of unused dirs and inodes, which
will have a negative impact on NN performance since we are talking about a huge
number of tables over a couple years of daily data.
This change will have admins either create NN dir first if needed or not create
the dangling(wrong) mount points. This may be incompatible with the current
workflow but I don't think the current workflow is fully correct since we
already went through some outages due to this.
I am working on the tests as discussed with [~elgoiri]. It is non trivial I
have to say.
> RBF: force router check file existence in destinations before adding/updating
> mount points
> ------------------------------------------------------------------------------------------
>
> Key: HDFS-15554
> URL: https://issues.apache.org/jira/browse/HDFS-15554
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Fengnan Li
> Assignee: Fengnan Li
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Adding/Updating mount points right now is only a router action without
> validation in the downstream namenodes for the destination files/directories.
> In practice we have set up the dangling mount points and when clients call
> listStatus they would get the file returned, but then if they try to access
> the file FileNotFoundException would be thrown out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
