[
https://issues.apache.org/jira/browse/HDFS-15750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17288313#comment-17288313
]
zhengchenyu commented on HDFS-15750:
------------------------------------
[~ayushsaxena]
I think we need a stricter check! Let's consider this situation.
If we have an hdfs cluster which router proxy ns1, and ns2.
If ns1 is heavy, then we wanna copy data "hdfs://ns1/user/userA" from ns1 to
ns2 "hdfs://ns2/user/userA" to lower the pressure. And we wanna users are
unawre of data migration.
Here I have below solution:
A mountable hdfs://ns-fed/user/userA mounts hdfs://ns1/user/userA and
hdfs://ns2/user/userA. I set a prior nameservice, which use firstly.
At first prior ns is ns1 , hdfs client only use hdfs://ns1/user/userA through
router firstly. Then we copy all data to hdfs://ns2/user/userA.
When all data is copied, we switch the proxy, the prior is ns2, then hdfs
client use hdfs://ns2/user/userA through router。
But in our cluster, many hive table partition are rerunned. So need to delete
hdfs://ns-fed/user/userA/tableA/pt=XXX/XXX. Because copy operation may happen
before rewrite hive table, so some hdfs file are not deleted in router view.
The real reason is we regard success if only on rename operation succeed.
> RBF: Make sure the multi destination are consistent after write operation
> --------------------------------------------------------------------------
>
> Key: HDFS-15750
> URL: https://issues.apache.org/jira/browse/HDFS-15750
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: zhengchenyu
> Assignee: zhengchenyu
> Priority: Major
>
> Nowdays, RBF can't make sure the multi destination are consistent.
> Case 1: RBF can't remove multi destination's file.
> If /user/userA is mountable which mounts two nameservice: ns1, ns2. But if
> both hdfs://ns1/user/userA/a.log and hdfs://ns2/user/userA/a.log exists. I
> want to remove hdfs://ns-fed/user/userA/a.log (Note: a.log is file) to trash,
> then only one nameservice take effect. I think it means inconsistence.
> Through HDFS-14343 already solve the problem in some level, but not
> completed.
> Case 2: RBF regard the operation success, through only one of multi
> destination operations success.
> In other way, if we wanna delete hdfs://ns-fed/user/userA/dirA (Note: dirA is
> directroy.) If hdfs://ns1/user/userA/dirA's permission is not same with
> hdfs://ns2/user/userA/dirA's permission, or one of namenode is down.If one
> destination's result be success, the rbf regard the operation success
> (invokeConcurrent and invokeAll's logical). We maybe can't rename all
> location. I think it also means inconsistence.
> I think we need stricter check. If one operation (which shoud success) failed
> , we should throw exception.
> Note: In fact,If we only use hdfs://ns-fed, I think there is no problem.But
> when migration data from one ns1 to ns2 (a mountable mouts ns1 and ns2) , and
> rewrite some hive table's old partition (which mount multi destination), this
> problem would occure!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
