tomscut commented on code in PR #6926:
URL: https://github.com/apache/hadoop/pull/6926#discussion_r1886463808
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java:
##########
@@ -1152,4 +1153,14 @@ void setDeleteDuplicateReplicasForTests(
this.deleteDuplicateReplicas = deleteDuplicateReplicasForTests;
}
+ public File hardLinkOneBlock(File src, File srcMeta, Block dstBlock) throws
IOException {
+ File dstMeta = new File(tmpDir,
+ DatanodeUtil.getMetaName(dstBlock.getBlockName(),
dstBlock.getGenerationStamp()));
+ HardLink.createHardLink(srcMeta, dstMeta);
+
+ File dstBlockFile = new File(tmpDir, dstBlock.getBlockName());
+ HardLink.createHardLink(src, dstBlockFile);
Review Comment:
Hi @LiuGuH @haiyang1987 , do you use this feature on a large scale? Does
creating a lot of hard links have a maintenance impact on the system? Have you
considered implementing it based on rename?
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:
##########
@@ -1956,6 +1957,26 @@ protected IOStreamPair connectToDN(DatanodeInfo dn, int
timeout,
socketFactory, getConf().isConnectToDnViaHostname(), this, blockToken);
}
+ protected void copyBlockCrossNamespace(ExtendedBlock sourceBlk,
+ Token<BlockTokenIdentifier> sourceBlockToken, DatanodeInfo
sourceDatanode,
+ ExtendedBlock targetBlk, Token<BlockTokenIdentifier> targetBlockToken,
+ DatanodeInfo targetDatanode) throws IOException {
+ IOStreamPair pair =
+ DFSUtilClient.connectToDN(sourceDatanode,
getConf().getSocketTimeout(), conf, saslClient,
+ socketFactory, getConf().isConnectToDnViaHostname(), this,
sourceBlockToken);
Review Comment:
Plz reuse `connectToDN` and close connection resources. You can refer to
`inferChecksumTypeByReading`.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:
##########
@@ -4388,4 +4409,94 @@ boolean isSlownode() {
public BlockPoolManager getBlockPoolManager() {
return blockPoolManager;
}
+
+ public void copyBlockCrossNamespace(ExtendedBlock sourceBlk, ExtendedBlock
targetBlk,
+ DatanodeInfo targetDn) throws IOException {
+ if (!data.isValidBlock(sourceBlk)) {
+ // block does not exist or is under-construction
+ String errStr =
+ "copyBlock:(" + this.getInfoPort() + ") Can't send invalid block " +
sourceBlk + " "
+ + data.getReplicaString(sourceBlk.getBlockPoolId(),
sourceBlk.getBlockId());
+ LOG.info(errStr);
+ throw new IOException(errStr);
+ }
+ long onDiskLength = data.getLength(sourceBlk);
+ if (sourceBlk.getNumBytes() > onDiskLength) {
+ // Shorter on-disk len indicates corruption so report NN the corrupt
block
+ String msg = "copyBlock: Can't replicate block " + sourceBlk + " because
on-disk length "
+ + onDiskLength + " is shorter than provided length " +
sourceBlk.getNumBytes();
+ LOG.info(msg);
+ throw new IOException(msg);
+ }
+ LOG.info(getDatanodeInfo() + " copyBlock: Starting thread to transfer: " +
"block:"
+ + sourceBlk + " from " + this.getDatanodeUuid() + " to " +
targetDn.getDatanodeUuid()
+ + "(" + targetDn + ")");
Review Comment:
> Hi, @LiuGuH , can use log parameterized messages ("{} xxxx", a) instead of
string concatenation (a + xxx).
@LiuGuH Plz fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
