[
https://issues.apache.org/jira/browse/HDFS-16880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17653484#comment-17653484
]
ASF GitHub Bot commented on HDFS-16880:
---------------------------------------
hfutatzhanghb commented on PR #5262:
URL: https://github.com/apache/hadoop/pull/5262#issuecomment-1368616110
> Thanx @hfutatzhanghb for the fix. Had a quick look I was able to reproduce
the issue. But I don't think the fix is correct. It is logging the path wrt the
router not wrt the namespace. say, /mount -> ns0. /dir is a mount entry you
create /mount/fiel1 on the namenode of ns0, with your fix it would be logging
/mount/file1, which in actual should be /dir/file1
>
> You need to extend a UT as well for the change.
>
> Quick Suggestion for the fix, create a new invokeSingle method with the
current signature, which calls your new invokeSingle with some default value,
in order to avoid changes at places you don't have src
Thanx @ayushtkn for your careful review. I was totally agree with your
opinions。Yes，the src is not the dst path, but we can use the src to lookup real
dst path in mount table. Here, we can not execute
`this.subclusterResolver.getMountPoints(path);`, because the statement is
time-consuming. It violates the purpose that we use previous block. I am
looking forward to your reply and do the next steps. thanks again~
> modify invokeSingleXXX interface in order to pass actual file src to namenode
> for debug info.
> ---------------------------------------------------------------------------------------------
>
> Key: HDFS-16880
> URL: https://issues.apache.org/jira/browse/HDFS-16880
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
> Affects Versions: 3.3.4
> Reporter: ZhangHB
> Priority: Major
> Labels: pull-request-available
>
> We found lots of INFO level log like below:
> {quote}2022-12-30 15:31:04,169 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> completeFile: / is closed by
> DFSClient_attempt_1671783180362_213003_m_000077_0_1102875551_1
> 2022-12-30 15:31:04,186 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> completeFile: / is closed by DFSClient_NONMAPREDUCE_1198313144_27480
> {quote}
> It lost the real path of completeFile. Actually this is caused by :
>
> *org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient#invokeSingle(java.lang.String,
> org.apache.hadoop.hdfs.server.federation.router.RemoteMethod)*
> In this method, it instantiates a RemoteLocationContext object:
> *RemoteLocationContext loc = new RemoteLocation(nsId, "/", "/");*
> and then execute: *Object[] params = method.getParams(loc);*
> The problem is right here, becasuse we always use new RemoteParam(), so,
> context.getDest() always return "/"; That's why we saw lots of incorrect logs.
>
> After diving into invokeSingleXXX source code, I found the following RPCs
> classified as need actual src and not need actual src.
>
> *need src path RPC:*
> addBlock、abandonBlock、getAdditionalDatanode、complete
> *not need src path RPC:*
> updateBlockForPipeline、reportBadBlocks、getBlocks、updatePipeline、invokeAtAvailableNs（invoked
> by:
> getServerDefaults、getBlockKeys、getTransactionID、getMostRecentCheckpointTxId、versionRequest、getStoragePolicies）
>
> After changes, the src can be pass to NN correctly.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
