[jira] [Commented] (HDFS-15413) DFSStripedInputStream throws exception when datanodes close idle connections
[
https://issues.apache.org/jira/browse/HDFS-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741985#comment-17741985
]
Max Xie commented on HDFS-15413:
-
I add a available patch to solve it https://github.com/apache/hadoop/pull/5829.
After set `dfs.client.read.striped.datanode.max.attempts = 2`, the above test
passes.
> DFSStripedInputStream throws exception when datanodes close idle connections
>
>
> Key: HDFS-15413
> URL: https://issues.apache.org/jira/browse/HDFS-15413
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, erasure-coding, hdfs-client
>Affects Versions: 3.1.3
> Environment: - Hadoop 3.1.3
> - erasure coding with ISA-L and RS-3-2-1024k scheme
> - running in kubernetes
> - dfs.client.socket-timeout = 1
> - dfs.datanode.socket.write.timeout = 1
>Reporter: Andrey Elenskiy
>Priority: Critical
> Labels: pull-request-available
> Attachments: out.log
>
>
> We've run into an issue with compactions failing in HBase when erasure coding
> is enabled on a table directory. After digging further I was able to narrow
> it down to a seek + read logic and able to reproduce the issue with hdfs
> client only:
> {code:java}
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.FSDataInputStream;
> public class ReaderRaw {
> public static void main(final String[] args) throws Exception {
> Path p = new Path(args[0]);
> int bufLen = Integer.parseInt(args[1]);
> int sleepDuration = Integer.parseInt(args[2]);
> int countBeforeSleep = Integer.parseInt(args[3]);
> int countAfterSleep = Integer.parseInt(args[4]);
> Configuration conf = new Configuration();
> FSDataInputStream istream = FileSystem.get(conf).open(p);
> byte[] buf = new byte[bufLen];
> int readTotal = 0;
> int count = 0;
> try {
> while (true) {
> istream.seek(readTotal);
> int bytesRemaining = bufLen;
> int bufOffset = 0;
> while (bytesRemaining > 0) {
> int nread = istream.read(buf, 0, bufLen);
> if (nread < 0) {
> throw new Exception("nread is less than zero");
> }
> readTotal += nread;
> bufOffset += nread;
> bytesRemaining -= nread;
> }
> count++;
> if (count == countBeforeSleep) {
> System.out.println("sleeping for " + sleepDuration + "
> milliseconds");
> Thread.sleep(sleepDuration);
> System.out.println("resuming");
> }
> if (count == countBeforeSleep + countAfterSleep) {
> System.out.println("done");
> break;
> }
> }
> } catch (Exception e) {
> System.out.println("exception on read " + count + " read total "
> + readTotal);
> throw e;
> }
> }
> }
> {code}
> The issue appears to be due to the fact that datanodes close the connection
> of EC client if it doesn't fetch next packet for longer than
> dfs.client.socket-timeout. The EC client doesn't retry and instead assumes
> that those datanodes went away resulting in "missing blocks" exception.
> I was able to consistently reproduce with the following arguments:
> {noformat}
> bufLen = 100 (just below 1MB which is the size of the stripe)
> sleepDuration = (dfs.client.socket-timeout + 1) * 1000 (in our case 11000)
> countBeforeSleep = 1
> countAfterSleep = 7
> {noformat}
> I've attached the entire log output of running the snippet above against
> erasure coded file with RS-3-2-1024k policy. And here are the logs from
> datanodes of disconnecting the client:
> datanode 1:
> {noformat}
> 2020-06-15 19:06:20,697 INFO datanode.DataNode: Likely the client has stopped
> reading, disconnecting it (datanode-v11-0-hadoop.hadoop:9866:DataXceiver
> error processing READ_BLOCK operation src: /10.128.23.40:53748 dst:
> /10.128.14.46:9866); java.net.SocketTimeoutException: 1 millis timeout
> while waiting for channel to be ready for write. ch :
> java.nio.channels.SocketChannel[connected local=/10.128.14.46:9866
> remote=/10.128.23.40:53748]
> {noformat}
> datanode 2:
> {noformat}
> 2020-06-15 19:06:20,341 INFO datanode.DataNode: Likely the client has stopped
> reading, disconnecting it (datanode-v11-1-hadoop.hadoop:9866:DataXceiver
> error processing READ_BLOCK operation src: /10.128.23.40:48772 dst:
> /10.128.9.42:9866); java.net.SocketTimeoutException: 1 millis timeout
> while waiting for channel to be ready for write. ch :
>
[jira] [Assigned] (HDFS-16954) RBF: The operation of renaming a multi-subcluster directory to a single-cluster directory should throw ioexception
[ https://issues.apache.org/jira/browse/HDFS-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie reassigned HDFS-16954: --- Assignee: Max Xie > RBF: The operation of renaming a multi-subcluster directory to a > single-cluster directory should throw ioexception > -- > > Key: HDFS-16954 > URL: https://issues.apache.org/jira/browse/HDFS-16954 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > > The operation of renaming a multi-subcluster directory to a single-cluster > directory may cause inconsistent behavior of the file system. This operation > should throw exception to be reasonable. > Examples are as follows: > 1. add hash_all mount point `hdfs dfsrouteradmin -add /tmp/foo > subcluster1,subcluster2 /tmp/foo -order HASH_ALL` > 2. add mount point `hdfs dfsrouteradmin -add /user/foo subcluster1 > /user/foo` > 3. mkdir dir for all subcluster. ` hdfs dfs -mkdir /tmp/foo/123 ` > 4. check dir and all subclusters will have dir `/tmp/foo/123` > `hdfs dfs -ls /tmp/foo/` : will show dir `/tmp/foo/123`; > `hdfs dfs -ls hdfs://subcluster1/tmp/foo/` : will show dir > `hdfs://subcluster1/tmp/foo/123`; > `hdfs dfs -ls hdfs://subcluster2/tmp/foo/` : will show dir > `hdfs://subcluster2/tmp/foo/123`; > 5. rename `/tmp/foo/123` to `/user/foo/123`. The op will succeed. `hdfs dfs > -mv /tmp/foo/123 /user/foo/123 ` > 6. check dir again, rbf cluster still show dir `/tmp/foo/123` > `hdfs dfs -ls /tmp/foo/` : will show dir `/tmp/foo/123`; > `hdfs dfs -ls hdfs://subcluster1/tmp/foo/` : will no dirs; > `hdfs dfs -ls hdfs://subcluster2/tmp/foo/` : will show dir > `hdfs://subcluster2/tmp/foo/123`; > The step 5 should throw exception. > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16954) RBF: The operation of renaming a multi-subcluster directory to a single-cluster directory should throw ioexception
Max Xie created HDFS-16954: --- Summary: RBF: The operation of renaming a multi-subcluster directory to a single-cluster directory should throw ioexception Key: HDFS-16954 URL: https://issues.apache.org/jira/browse/HDFS-16954 Project: Hadoop HDFS Issue Type: Bug Components: rbf Affects Versions: 3.4.0 Reporter: Max Xie The operation of renaming a multi-subcluster directory to a single-cluster directory may cause inconsistent behavior of the file system. This operation should throw exception to be reasonable. Examples are as follows: 1. add hash_all mount point `hdfs dfsrouteradmin -add /tmp/foo subcluster1,subcluster2 /tmp/foo -order HASH_ALL` 2. add mount point `hdfs dfsrouteradmin -add /user/foo subcluster1 /user/foo` 3. mkdir dir for all subcluster. ` hdfs dfs -mkdir /tmp/foo/123 ` 4. check dir and all subclusters will have dir `/tmp/foo/123` `hdfs dfs -ls /tmp/foo/` : will show dir `/tmp/foo/123`; `hdfs dfs -ls hdfs://subcluster1/tmp/foo/` : will show dir `hdfs://subcluster1/tmp/foo/123`; `hdfs dfs -ls hdfs://subcluster2/tmp/foo/` : will show dir `hdfs://subcluster2/tmp/foo/123`; 5. rename `/tmp/foo/123` to `/user/foo/123`. The op will succeed. `hdfs dfs -mv /tmp/foo/123 /user/foo/123 ` 6. check dir again, rbf cluster still show dir `/tmp/foo/123` `hdfs dfs -ls /tmp/foo/` : will show dir `/tmp/foo/123`; `hdfs dfs -ls hdfs://subcluster1/tmp/foo/` : will no dirs; `hdfs dfs -ls hdfs://subcluster2/tmp/foo/` : will show dir `hdfs://subcluster2/tmp/foo/123`; The step 5 should throw exception. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16946) RBF: top real owners metrics can't been parsed json string
Max Xie created HDFS-16946: --- Summary: RBF: top real owners metrics can't been parsed json string Key: HDFS-16946 URL: https://issues.apache.org/jira/browse/HDFS-16946 Project: Hadoop HDFS Issue Type: Bug Components: rbf Affects Versions: 3.4.0 Reporter: Max Xie Attachments: image-2023-03-09-22-24-39-833.png After HDFS-15447, Add top real owners metrics for delegation tokens. But the metrics can't been parsed json string. RBFMetrics$getTopTokenRealOwners method just return `org.apache.hadoop.metrics2.util.Metrics2Util$NameValuePair@1` !image-2023-03-09-22-24-39-833.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16945) RBF: add RouterSecurityAuditLogger for router security manager
[
https://issues.apache.org/jira/browse/HDFS-16945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Xie reassigned HDFS-16945:
---
Assignee: Max Xie
> RBF: add RouterSecurityAuditLogger for router security manager
> --
>
> Key: HDFS-16945
> URL: https://issues.apache.org/jira/browse/HDFS-16945
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: rbf
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Minor
>
> we should add audit log for router security manager for token APIs. For
> examples,
> ```
>
> {{2023-03-02 20:53:02,712 INFO
> org.apache.hadoop.hdfs.server.federation.router.security.RouterSecurityManager.audit:
> allowed=true ugi=hadoop ip=localhost/127.0.0.1 cmd=getDelegationToken
> toeknId=HDFS_DELEGATION_TOKEN token 18359 for hadoop with renewer
> hadoopproto=webhdfs}}
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16945) RBF: add RouterSecurityAuditLogger for router security manager
Max Xie created HDFS-16945:
---
Summary: RBF: add RouterSecurityAuditLogger for router security
manager
Key: HDFS-16945
URL: https://issues.apache.org/jira/browse/HDFS-16945
Project: Hadoop HDFS
Issue Type: New Feature
Components: rbf
Affects Versions: 3.4.0
Reporter: Max Xie
we should add audit log for router security manager for token APIs. For
examples,
```
{{2023-03-02 20:53:02,712 INFO
org.apache.hadoop.hdfs.server.federation.router.security.RouterSecurityManager.audit:
allowed=true ugi=hadoop ip=localhost/127.0.0.1 cmd=getDelegationToken
toeknId=HDFS_DELEGATION_TOKEN token 18359 for hadoop with renewer hadoop
proto=webhdfs}}
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16861) RBF. Truncate API always fails when dirs use AllResolver oder on Router
[ https://issues.apache.org/jira/browse/HDFS-16861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie reassigned HDFS-16861: --- Assignee: Max Xie > RBF. Truncate API always fails when dirs use AllResolver oder on Router > - > > Key: HDFS-16861 > URL: https://issues.apache.org/jira/browse/HDFS-16861 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Major > Labels: pull-request-available > Attachments: image-2022-12-05-17-35-19-841.png > > > # prepare a directory in a HASH_ALL/SPACE/RANDOM mount point. > # put a test file with 1024 bytes to this directory > # truncate the file with 100 new length and this op will fail and throw a > exception that the file does not exist. > > After dig it, we should ignore the result of Truncate API in > RouterClientProtocol because > Truncate can return true/false, so don't expect a result. > After fix it , the code is > !image-2022-12-05-17-35-19-841.png! > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16861) RBF. Truncate API always fails when dirs use AllResolver oder on Router
Max Xie created HDFS-16861: --- Summary: RBF. Truncate API always fails when dirs use AllResolver oder on Router Key: HDFS-16861 URL: https://issues.apache.org/jira/browse/HDFS-16861 Project: Hadoop HDFS Issue Type: Bug Components: rbf Affects Versions: 3.4.0 Reporter: Max Xie Attachments: image-2022-12-05-17-35-19-841.png # prepare a directory in a HASH_ALL/SPACE/RANDOM mount point. # put a test file with 1024 bytes to this directory # truncate the file with 100 new length and this op will fail and throw a exception that the file does not exist. After dig it, we should ignore the result of Truncate API in RouterClientProtocol because Truncate can return true/false, so don't expect a result. After fix it , the code is !image-2022-12-05-17-35-19-841.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16677) Add OP_SWAP_BLOCK_LIST as an operation code in FSEditLogOpCodes.
[ https://issues.apache.org/jira/browse/HDFS-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie resolved HDFS-16677. - Resolution: Duplicate > Add OP_SWAP_BLOCK_LIST as an operation code in FSEditLogOpCodes. > > > Key: HDFS-16677 > URL: https://issues.apache.org/jira/browse/HDFS-16677 > Project: Hadoop HDFS > Issue Type: New Feature > Components: erasure-coding, hdfs >Reporter: Max Xie >Assignee: Max Xie >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Copy from HDFS-15006 > In HDFS-14989, we add a new Namenode operation "swapBlockList" to replace the > set of blocks in an INode File with a new set of blocks. This JIRA will track > the effort to add an FS Edit log op to persist this operation. > We also need to increase the NamenodeLayoutVersion for this change. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16677) Add OP_SWAP_BLOCK_LIST as an operation code in FSEditLogOpCodes.
Max Xie created HDFS-16677: --- Summary: Add OP_SWAP_BLOCK_LIST as an operation code in FSEditLogOpCodes. Key: HDFS-16677 URL: https://issues.apache.org/jira/browse/HDFS-16677 Project: Hadoop HDFS Issue Type: New Feature Components: erasure-coding, hdfs Reporter: Max Xie Assignee: Max Xie Copy from HDFS-15006 In HDFS-14989, we add a new Namenode operation "swapBlockList" to replace the set of blocks in an INode File with a new set of blocks. This JIRA will track the effort to add an FS Edit log op to persist this operation. We also need to increase the NamenodeLayoutVersion for this change. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16655) OIV: print out erasure coding policy name in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie reassigned HDFS-16655: --- Assignee: Max Xie > OIV: print out erasure coding policy name in oiv Delimited output > - > > Key: HDFS-16655 > URL: https://issues.apache.org/jira/browse/HDFS-16655 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > > By adding erasure coding policy name to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified ec > policy and to apply internal regulation based on this information. In > particular, it wiil be convenient for the platform to calculate the real > storage size of the ec file. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16655) OIV: print out erasure coding policy name in oiv Delimited output
Max Xie created HDFS-16655: --- Summary: OIV: print out erasure coding policy name in oiv Delimited output Key: HDFS-16655 URL: https://issues.apache.org/jira/browse/HDFS-16655 Project: Hadoop HDFS Issue Type: Improvement Components: tools Affects Versions: 3.4.0 Reporter: Max Xie By adding erasure coding policy name to oiv output, it will help with oiv post-analysis to have a overview of all folders/files with specified ec policy and to apply internal regulation based on this information. In particular, it wiil be convenient for the platform to calculate the real storage size of the ec file. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13671) Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet
[
https://issues.apache.org/jira/browse/HDFS-13671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511731#comment-17511731
]
Max Xie commented on HDFS-13671:
-
[~tomscut] Thank you for your share. Our cluster ' FBR period is 6 hours
(default value). There are 1 billion blocks and 100+ dn on one of our
clusters , which is branch-3.3.0 with this patch.
> Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet
> --
>
> Key: HDFS-13671
> URL: https://issues.apache.org/jira/browse/HDFS-13671
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.0.3
>Reporter: Yiqun Lin
>Assignee: Haibin Huang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.3, 3.3.2
>
> Attachments: HDFS-13671-001.patch, image-2021-06-10-19-28-18-373.png,
> image-2021-06-10-19-28-58-359.png, image-2021-06-18-15-46-46-052.png,
> image-2021-06-18-15-47-04-037.png
>
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> NameNode hung when deleting large files/blocks. The stack info:
> {code}
> "IPC Server handler 4 on 8020" #87 daemon prio=5 os_prio=0
> tid=0x7fb505b27800 nid=0x94c3 runnable [0x7fa861361000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet.compare(FoldedTreeSet.java:474)
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet.removeAndGet(FoldedTreeSet.java:849)
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet.remove(FoldedTreeSet.java:911)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.removeBlock(DatanodeStorageInfo.java:252)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.removeBlock(BlocksMap.java:194)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.removeBlock(BlocksMap.java:108)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.removeBlockFromMap(BlockManager.java:3813)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.removeBlock(BlockManager.java:3617)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeBlocks(FSNamesystem.java:4270)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:4244)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:4180)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:4164)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:871)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.delete(AuthorizationProviderProxyClientProtocol.java:311)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:625)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> {code}
> In the current deletion logic in NameNode, there are mainly two steps:
> * Collect INodes and all blocks to be deleted, then delete INodes.
> * Remove blocks chunk by chunk in a loop.
> Actually the first step should be a more expensive operation and will takes
> more time. However, now we always see NN hangs during the remove block
> operation.
> Looking into this, we introduced a new structure {{FoldedTreeSet}} to have a
> better performance in dealing FBR/IBRs. But compared with early
> implementation in remove-block logic, {{FoldedTreeSet}} seems more slower
> since It will take additional time to balance tree node. When there are large
> block to be removed/deleted, it looks bad.
> For the get type operations in {{DatanodeStorageInfo}}, we only provide the
> {{getBlockIterator}} to return blocks iterator and no other get operation
> with specified block. Still we need to use {{FoldedTreeSet}} in
> {{DatanodeStorageInfo}}? As we know {{FoldedTreeSet}} is benefit for Get not
> Update. Maybe we can revert this to the early implementation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13671) Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet
[
https://issues.apache.org/jira/browse/HDFS-13671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511573#comment-17511573
]
Max Xie commented on HDFS-13671:
-
[~tomscut] Hi , Would you share the parameters of G1 GC for Namenode? After
our cluster use branch 3.3.0 with this patch, without EC data now, GC
performance become poor. Thank you.
> Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet
> --
>
> Key: HDFS-13671
> URL: https://issues.apache.org/jira/browse/HDFS-13671
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.0.3
>Reporter: Yiqun Lin
>Assignee: Haibin Huang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.3, 3.3.2
>
> Attachments: HDFS-13671-001.patch, image-2021-06-10-19-28-18-373.png,
> image-2021-06-10-19-28-58-359.png, image-2021-06-18-15-46-46-052.png,
> image-2021-06-18-15-47-04-037.png
>
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> NameNode hung when deleting large files/blocks. The stack info:
> {code}
> "IPC Server handler 4 on 8020" #87 daemon prio=5 os_prio=0
> tid=0x7fb505b27800 nid=0x94c3 runnable [0x7fa861361000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet.compare(FoldedTreeSet.java:474)
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet.removeAndGet(FoldedTreeSet.java:849)
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet.remove(FoldedTreeSet.java:911)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.removeBlock(DatanodeStorageInfo.java:252)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.removeBlock(BlocksMap.java:194)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.removeBlock(BlocksMap.java:108)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.removeBlockFromMap(BlockManager.java:3813)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.removeBlock(BlockManager.java:3617)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeBlocks(FSNamesystem.java:4270)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:4244)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:4180)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:4164)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:871)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.delete(AuthorizationProviderProxyClientProtocol.java:311)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:625)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> {code}
> In the current deletion logic in NameNode, there are mainly two steps:
> * Collect INodes and all blocks to be deleted, then delete INodes.
> * Remove blocks chunk by chunk in a loop.
> Actually the first step should be a more expensive operation and will takes
> more time. However, now we always see NN hangs during the remove block
> operation.
> Looking into this, we introduced a new structure {{FoldedTreeSet}} to have a
> better performance in dealing FBR/IBRs. But compared with early
> implementation in remove-block logic, {{FoldedTreeSet}} seems more slower
> since It will take additional time to balance tree node. When there are large
> block to be removed/deleted, it looks bad.
> For the get type operations in {{DatanodeStorageInfo}}, we only provide the
> {{getBlockIterator}} to return blocks iterator and no other get operation
> with specified block. Still we need to use {{FoldedTreeSet}} in
> {{DatanodeStorageInfo}}? As we know {{FoldedTreeSet}} is benefit for Get not
> Update. Maybe we can revert this to the early implementation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16504) add `dfs.namenode.get-blocks.check.operation` to enable or disable checkOperation when NNs process getBlocks
[ https://issues.apache.org/jira/browse/HDFS-16504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie reassigned HDFS-16504: --- Assignee: Max Xie > add `dfs.namenode.get-blocks.check.operation` to enable or disable > checkOperation when NNs process getBlocks > > > Key: HDFS-16504 > URL: https://issues.apache.org/jira/browse/HDFS-16504 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer mover, namanode >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HDFS-13183 add a nice feature that Standby NameNode can process getBlocks > request to reduce Active load. Namenode must set `dfs.ha.allow.stale.reads > = true` to enable this feature. However, if we set `dfs.ha.allow.stale.reads > = true`, Standby Namenode will be able to process all read requests, which > may lead to yarn jobs fail because Standby Namenode is stale . > Maybe we should add a config `dfs.namenode.get-blocks.check.operation=false` > for namenode to disable check operation when namenode process getBlocks > request. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16504) add `dfs.namenode.get-blocks.check.operation` to enable or disable checkOperation when NNs process getBlocks
Max Xie created HDFS-16504: --- Summary: add `dfs.namenode.get-blocks.check.operation` to enable or disable checkOperation when NNs process getBlocks Key: HDFS-16504 URL: https://issues.apache.org/jira/browse/HDFS-16504 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover, namanode Affects Versions: 3.4.0 Reporter: Max Xie HDFS-13183 add a nice feature that Standby NameNode can process getBlocks request to reduce Active load. Namenode must set `dfs.ha.allow.stale.reads = true` to enable this feature. However, if we set `dfs.ha.allow.stale.reads = true`, Standby Namenode will be able to process all read requests, which may lead to yarn jobs fail because Standby Namenode is stale . Maybe we should add a config `dfs.namenode.get-blocks.check.operation=false` for namenode to disable check operation when namenode process getBlocks request. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16459) RBF: register RBFMetrics in MetricsSystem for promethuessink
[
https://issues.apache.org/jira/browse/HDFS-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Xie updated HDFS-16459:
Description:
Router' RBFMetrics was not register in MetricsSystem. We can't find these
metrics from PrometheusSink. Maybe we should fix it.
After fix it , some RBFMetrics will export like this
{code:java}
# HELP rbf_metrics_current_tokens_count Number of router's current tokens
# TYPE rbf_metrics_current_tokens_count gauge
rbf_metrics_current_tokens_count{processname="Router",context="dfs",hostname=""}
2 {code}
was:
Router' RBFMetrics was not register in MetricsSystem. We can't find these
metrics from PrometheusSink. Maybe we should fix it.
After fix it , some RBFMetrics will export like this
```
# HELP rbf_metrics_current_tokens_count Number of router's current tokens #
TYPE rbf_metrics_current_tokens_count gauge
rbf_metrics_current_tokens_count\{processname="Router",context="dfs",hostname="xxx"}
123
```
> RBF: register RBFMetrics in MetricsSystem for promethuessink
>
>
> Key: HDFS-16459
> URL: https://issues.apache.org/jira/browse/HDFS-16459
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Minor
>
> Router' RBFMetrics was not register in MetricsSystem. We can't find these
> metrics from PrometheusSink. Maybe we should fix it.
>
> After fix it , some RBFMetrics will export like this
> {code:java}
> # HELP rbf_metrics_current_tokens_count Number of router's current tokens
> # TYPE rbf_metrics_current_tokens_count gauge
> rbf_metrics_current_tokens_count{processname="Router",context="dfs",hostname=""}
> 2 {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16459) RBF: register RBFMetrics in MetricsSystem for promethuessink
Max Xie created HDFS-16459:
---
Summary: RBF: register RBFMetrics in MetricsSystem for
promethuessink
Key: HDFS-16459
URL: https://issues.apache.org/jira/browse/HDFS-16459
Project: Hadoop HDFS
Issue Type: Improvement
Components: rbf
Affects Versions: 3.4.0
Reporter: Max Xie
Assignee: Max Xie
Router' RBFMetrics was not register in MetricsSystem. We can't find these
metrics from PrometheusSink. Maybe we should fix it.
After fix it , some RBFMetrics will export like this
```
# HELP rbf_metrics_current_tokens_count Number of router's current tokens #
TYPE rbf_metrics_current_tokens_count gauge
rbf_metrics_current_tokens_count\{processname="Router",context="dfs",hostname="xxx"}
123
```
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16455) RBF: Router should explicitly specify the value of `jute.maxbuffer` in hadoop configuration files like core-site.xml
[
https://issues.apache.org/jira/browse/HDFS-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490912#comment-17490912
]
Max Xie commented on HDFS-16455:
-
https://github.com/apache/hadoop/pull/3983
> RBF: Router should explicitly specify the value of `jute.maxbuffer` in hadoop
> configuration files like core-site.xml
>
>
> Key: HDFS-16455
> URL: https://issues.apache.org/jira/browse/HDFS-16455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Affects Versions: 3.3.0, 3.4.0
>Reporter: Max Xie
>Priority: Minor
>
> Based on the current design for delegation token in secure Router, the total
> number of tokens store and update in zookeeper using
> ZKDelegationTokenManager.
> But the default value of system property `jute.maxbuffer` is just 4MB, if
> Router store too many tokens in zk, it will throw IOException `{{{}Packet
> lenxx is out of range{}}}` and all Router will crash.
>
> In our cluster, Routers crashed because of it. The crash logs are below
> {code:java}
> 2022-02-09 02:15:51,607 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Token renewal for identifier: (token for xxx: HDFS_DELEGATION_TOKEN
> owner=xxx/scheduler, renewer=hadoop, realUser=, issueDate=1644344146305,
> maxDate=1644948946305, sequenceNumbe
> r=27136070, masterKeyId=1107); total currentTokens 279548 2022-02-09
> 02:16:07,632 WARN org.apache.zookeeper.ClientCnxn: Session 0x1000172775a0012
> for server zkurl:2181, unexpected e
> rror, closing socket connection and attempting reconnect
> java.io.IOException: Packet len4194553 is out of range!
> at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:113)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)
> 2022-02-09 02:16:07,733 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 1254 on default port 9001, call Call#144 Retry#0
> org.apache.hadoop.hdfs.protocol.ClientProtocol.getDelegationToken from
> ip:46534
> java.lang.RuntimeException: Could not increment shared counter !!
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum(ZKDelegationTokenSecretManager.java:582)
> {code}
> When we restart a Router, it crashed again
> {code:java}
> 2022-02-09 03:14:17,308 INFO
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager:
> Starting to load key cache.
> 2022-02-09 03:14:17,310 INFO
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager:
> Loaded key cache.
> 2022-02-09 03:14:32,930 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x205584be35b0001 for server zkurl:2181, unexpected
> error, closing socket connection and attempting reconnect
> java.io.IOException: Packet len4194478 is out of range!
> at
> org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:113)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)
> 2022-02-09 03:14:33,030 ERROR
> org.apache.hadoop.hdfs.server.federation.router.security.token.ZKDelegationTokenSecretManagerImpl:
> Error starting threads for z
> kDelegationTokens
> java.io.IOException: Could not start PathChildrenCache for tokens {code}
> Finnally, we config `-Djute.maxbuffer=1000` in hadoop-env,sh to fix this
> issue.
> After dig it, we found the number of the znode
> `/ZKDTSMRoot/ZKDTSMTokensRoot`'s children node was more than 25, which's
> data size was over 4MB.
>
> Maybe we should explicitly specify the value of `jute.maxbuffer` in hadoop
> configuration files like core-site.xml, hdfs-rbf-site.xml to configure a
> larger value
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16455) RBF: Router should explicitly specify the value of `jute.maxbuffer` in hadoop configuration files like core-site.xml
[
https://issues.apache.org/jira/browse/HDFS-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Xie updated HDFS-16455:
Description:
Based on the current design for delegation token in secure Router, the total
number of tokens store and update in zookeeper using ZKDelegationTokenManager.
But the default value of system property `jute.maxbuffer` is just 4MB, if
Router store too many tokens in zk, it will throw IOException `{{{}Packet
lenxx is out of range{}}}` and all Router will crash.
In our cluster, Routers crashed because of it. The crash logs are below
{code:java}
2022-02-09 02:15:51,607 INFO
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Token renewal for identifier: (token for xxx: HDFS_DELEGATION_TOKEN
owner=xxx/scheduler, renewer=hadoop, realUser=, issueDate=1644344146305,
maxDate=1644948946305, sequenceNumbe
r=27136070, masterKeyId=1107); total currentTokens 279548 2022-02-09
02:16:07,632 WARN org.apache.zookeeper.ClientCnxn: Session 0x1000172775a0012
for server zkurl:2181, unexpected e
rror, closing socket connection and attempting reconnect
java.io.IOException: Packet len4194553 is out of range!
at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:113)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)
2022-02-09 02:16:07,733 WARN org.apache.hadoop.ipc.Server: IPC Server handler
1254 on default port 9001, call Call#144 Retry#0
org.apache.hadoop.hdfs.protocol.ClientProtocol.getDelegationToken from ip:46534
java.lang.RuntimeException: Could not increment shared counter !!
at
org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum(ZKDelegationTokenSecretManager.java:582)
{code}
When we restart a Router, it crashed again
{code:java}
2022-02-09 03:14:17,308 INFO
org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager:
Starting to load key cache.
2022-02-09 03:14:17,310 INFO
org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager:
Loaded key cache.
2022-02-09 03:14:32,930 WARN org.apache.zookeeper.ClientCnxn: Session
0x205584be35b0001 for server zkurl:2181, unexpected
error, closing socket connection and attempting reconnect
java.io.IOException: Packet len4194478 is out of range!
at
org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:113)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)
2022-02-09 03:14:33,030 ERROR
org.apache.hadoop.hdfs.server.federation.router.security.token.ZKDelegationTokenSecretManagerImpl:
Error starting threads for z
kDelegationTokens
java.io.IOException: Could not start PathChildrenCache for tokens {code}
Finnally, we config `-Djute.maxbuffer=1000` in hadoop-env,sh to fix this
issue.
After dig it, we found the number of the znode
`/ZKDTSMRoot/ZKDTSMTokensRoot`'s children node was more than 25, which's
data size was over 4MB.
Maybe we should explicitly specify the value of `jute.maxbuffer` in hadoop
configuration files like core-site.xml, hdfs-rbf-site.xml to configure a larger
value
was:
Based on the current design for delegation token in secure Router, the total
number of tokens store and update in zookeeper using ZKDelegationTokenManager.
But the default value of system property `jute.maxbuffer` is just 4MB, if
Router store too many tokens in zk, it will throw IOException `{{{}Packet
lenxx is out of range{}}}` and all Router will crash.
In our cluster, Routers crashed because of it. The crash logs are below
2022-02-09 02:15:51,607 INFO
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Token renewal for identifier: (token for xxx: HDFS_DELEGATION_TOKEN
owner=xxx/scheduler, renewer=hadoop, realUser=, issueDate=1644344146305,
maxDate=1644948946305, sequenceNumbe
r=27136070, masterKeyId=1107); total currentTokens 279548 2022-02-09
02:16:07,632 WARN org.apache.zookeeper.ClientCnxn: Session 0x1000172775a0012
for server zkurl:2181, unexpected e
rror, closing socket connection and attempting reconnect
java.io.IOException: Packet len4194553 is out of range!
at
org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:113)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)
2022-02-09 02:16:07,733
[jira] [Created] (HDFS-16455) RBF: Router should explicitly specify the value of `jute.maxbuffer` in hadoop configuration files like core-site.xml
Max Xie created HDFS-16455:
---
Summary: RBF: Router should explicitly specify the value of
`jute.maxbuffer` in hadoop configuration files like core-site.xml
Key: HDFS-16455
URL: https://issues.apache.org/jira/browse/HDFS-16455
Project: Hadoop HDFS
Issue Type: Bug
Components: rbf
Affects Versions: 3.3.0, 3.4.0
Reporter: Max Xie
Based on the current design for delegation token in secure Router, the total
number of tokens store and update in zookeeper using ZKDelegationTokenManager.
But the default value of system property `jute.maxbuffer` is just 4MB, if
Router store too many tokens in zk, it will throw IOException `{{{}Packet
lenxx is out of range{}}}` and all Router will crash.
In our cluster, Routers crashed because of it. The crash logs are below
2022-02-09 02:15:51,607 INFO
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Token renewal for identifier: (token for xxx: HDFS_DELEGATION_TOKEN
owner=xxx/scheduler, renewer=hadoop, realUser=, issueDate=1644344146305,
maxDate=1644948946305, sequenceNumbe
r=27136070, masterKeyId=1107); total currentTokens 279548 2022-02-09
02:16:07,632 WARN org.apache.zookeeper.ClientCnxn: Session 0x1000172775a0012
for server zkurl:2181, unexpected e
rror, closing socket connection and attempting reconnect
java.io.IOException: Packet len4194553 is out of range!
at
org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:113)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)
2022-02-09 02:16:07,733 WARN org.apache.hadoop.ipc.Server: IPC Server handler
1254 on default port 9001, call Call#144 Retry#0
org.apache.hadoop.hdfs.protocol.ClientProtocol.getDelegationToken from ip:46534
java.lang.RuntimeException: Could not increment shared counter !!
at
org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum(ZKDelegationTokenSecretManager.java:582)
{{}}
When we restart a Router, it crashed again
{code:java}
2022-02-09 03:14:17,308 INFO
org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager:
Starting to load key cache.
2022-02-09 03:14:17,310 INFO
org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager:
Loaded key cache.
2022-02-09 03:14:32,930 WARN org.apache.zookeeper.ClientCnxn: Session
0x205584be35b0001 for server zkurl:2181, unexpected
error, closing socket connection and attempting reconnect
java.io.IOException: Packet len4194478 is out of range!
at
org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:113)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)
2022-02-09 03:14:33,030 ERROR
org.apache.hadoop.hdfs.server.federation.router.security.token.ZKDelegationTokenSecretManagerImpl:
Error starting threads for z
kDelegationTokens
java.io.IOException: Could not start PathChildrenCache for tokens {code}
Finnally, we config `-Djute.maxbuffer=1000` in hadoop-env,sh to fix this
issue.
After dig it, we found the number of the znode
`/ZKDTSMRoot/ZKDTSMTokensRoot`'s children node was more than 25, which's
data size was over 4MB.
Maybe we should explicitly specify the value of `jute.maxbuffer` in hadoop
configuration files like core-site.xml, hdfs-rbf-site.xml to configure a larger
value
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16451) RBF: add search box for Router's tab-mounttable web page.
Max Xie created HDFS-16451: --- Summary: RBF: add search box for Router's tab-mounttable web page. Key: HDFS-16451 URL: https://issues.apache.org/jira/browse/HDFS-16451 Project: Hadoop HDFS Issue Type: New Feature Components: rbf Affects Versions: 3.4.0 Reporter: Max Xie Assignee: Max Xie Attachments: image-2022-02-09-18-17-53-498.png, image-2022-02-09-18-18-29-262.png In our cluster, we have mount many paths in HDFS Router and it may lead to take some time to load the mount-table page of Router when we open it in the browser. In order to use the mount-table page more conveniently, maybe we should add a search box style, just like the screenshot below !image-2022-02-09-18-17-53-498.png! !image-2022-02-09-18-18-29-262.png! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16447) RBF: Registry HDFS Router's rpcserver & rpcclient metrics for PrometheusSink.
[ https://issues.apache.org/jira/browse/HDFS-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17488740#comment-17488740 ] Max Xie commented on HDFS-16447: - https://github.com/apache/hadoop/pull/3965 > RBF: Registry HDFS Router's rpcserver & rpcclient metrics for PrometheusSink. > - > > Key: HDFS-16447 > URL: https://issues.apache.org/jira/browse/HDFS-16447 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > > When we enable PrometheusSink for HDFS Router, Router' prometheus sink miss > some metrics, for example `RpcClientNumActiveConnections` and so on. > > We need registry some Router's rpcserver & rpcclient metrics for > PrometheusSink. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16447) RBF: Registry HDFS Router's rpcserver & rpcclient metrics for PrometheusSink.
[ https://issues.apache.org/jira/browse/HDFS-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie reassigned HDFS-16447: --- Assignee: Max Xie > RBF: Registry HDFS Router's rpcserver & rpcclient metrics for PrometheusSink. > - > > Key: HDFS-16447 > URL: https://issues.apache.org/jira/browse/HDFS-16447 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > > When we enable PrometheusSink for HDFS Router, Router' prometheus sink miss > some metrics, for example `RpcClientNumActiveConnections` and so on. > > We need registry some Router's rpcserver & rpcclient metrics for > PrometheusSink. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16447) RBF: Registry HDFS Router's rpcserver & rpcclient metrics for PrometheusSink.
Max Xie created HDFS-16447: --- Summary: RBF: Registry HDFS Router's rpcserver & rpcclient metrics for PrometheusSink. Key: HDFS-16447 URL: https://issues.apache.org/jira/browse/HDFS-16447 Project: Hadoop HDFS Issue Type: Improvement Components: rbf Affects Versions: 3.4.0 Reporter: Max Xie When we enable PrometheusSink for HDFS Router, Router' prometheus sink miss some metrics, for example `RpcClientNumActiveConnections` and so on. We need registry some Router's rpcserver & rpcclient metrics for PrometheusSink. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16242) JournalMetrics should add JournalId MetricTag to distinguish different nameservice journal metrics.
Max Xie created HDFS-16242:
---
Summary: JournalMetrics should add JournalId MetricTag to
distinguish different nameservice journal metrics.
Key: HDFS-16242
URL: https://issues.apache.org/jira/browse/HDFS-16242
Project: Hadoop HDFS
Issue Type: Improvement
Components: journal-node
Affects Versions: 3.4.0
Reporter: Max Xie
JournalMetrics should add JournalId MetricTag to distinguish different
nameservice journal metrics when JournalNode use ganglia sink or prometheus
sink.
*JMX*
Before this patch, journal' jmx :
{code:java}
// jmx json
{
"name" : "Hadoop:service=JournalNode,name=Journal-nntest1",
"Syncs60sNumOps" : 0,
...
},
{
"name" : "Hadoop:service=JournalNode,name=Journal-nntest2",
"Syncs60sNumOps" : 0,
...
}
{code}
After this patch, journal' jmx :
{code:java}
// jmx json
{
"name" : "Hadoop:service=JournalNode,name=Journal-nntest1",
"tag.JournalId" : "nntest1", // add this tag
"Syncs60sNumOps" : 0,
...
},
{
"name" : "Hadoop:service=JournalNode,name=Journal-nntest2",
"tag.JournalId" : "nntest2",
"Syncs60sNumOps" : 0,
...
}
{code}
*PrometheusSink*
Before this patch, journal' prometheus export :
{code:java}
journal_node_syncs60s_num_ops{context="dfs",hostname="host"} 2
{code}
After this patch, journal' prometheus export :
{code:java}
journal_node_syncs60s_num_ops{context="dfs",journalid="nntest2",hostname="host"}
2
journal_node_syncs60s_num_ops{context="dfs",journalid="nntest1",hostname="host"}
75
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414003#comment-17414003 ] Max Xie commented on HDFS-14353: - branch-3.2.2 doesn't include this Jira +1. cc [~ayushtkn] > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: Baolong Mao >Assignee: Baolong Mao >Priority: Major > Fix For: 3.2.2, 3.3.1, 3.4.0 > > Attachments: HDFS-14353.001.patch, HDFS-14353.002.patch, > HDFS-14353.003.patch, HDFS-14353.004.patch, HDFS-14353.005.patch, > HDFS-14353.006.patch, HDFS-14353.007.patch, HDFS-14353.008.patch, > HDFS-14353.009.patch, HDFS-14353.010.patch, screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403006#comment-17403006
]
Max Xie edited comment on HDFS-16182 at 8/24/21, 4:06 AM:
---
In my cluster, we use BlockPlacementPolicyDefault to choose dn and the number
of SSD DN is much less than DISK DN. It may cause to some block that should
be placed to SSD DNs fallback to place DISK DNs when SSD DNs are too busy or no
enough place. Consider the following scenario.
# Create empty file /foo_file
# Set its storagepolicy to All_SSD
# Put data to /foo_file
# /foo_file gets 3 DISK dns for pipeline because SSD dns are too busy at
the beginning.
# When it transfers data in pipeline, one of 3 DISK dns shut down.
# The client need to get one new dn for existing pipeline in
DataStreamer$addDatanode2ExistingPipeline..
# If SSD dns are available at the moment, namenode will choose the 3 SSD dns
and return it to the client. However, the client just need one new dn,
namenode returns 3 new SSD dn and the client fail in
DataStreamer$findNewDatanode.
was (Author: max2049):
In my cluster, we use BlockPlacementPolicyDefault to choose dn and the number
of SSD DN is much less than DISK DN. It may cause to some block that should
be placed to SSD DNs fallback to place DISK DNs when SSD DNs are too busy or no
enough place. Consider the following scenario.
# Create empty file /foo_file
# Set its storagepolicy to All_SSD
# Put data to /foo_file
# /foo_file gets 3 DISK dns for pipeline because SSD dns are too busy at
the beginning.
# When it transfers data in pipeline, one of 3 DISK dns shut down.
# The client need to get one new dn for existing pipeline.
# If SSD dns are available at the moment, namenode will choose the 3 SSD dns
and return it to the client. However, the client just need one new dn,
namenode returns 3 new SSD dn and the client fail in DataStreamer.
> numOfReplicas is given the wrong value in
> BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
> Heterogeneous Storage
> ---
>
> Key: HDFS-16182
> URL: https://issues.apache.org/jira/browse/HDFS-16182
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-16182.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
> better performance. Sometimes hdfs client transfer data in pipline, it will
> throw IOException and exit. Exception logs are below:
> ```
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
>
> DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
>
> DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
>
> DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
>
> original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> ```
> After search it, I found when existing pipline need replace new dn to
> transfer data, the client will get one additional dn from namenode and check
> that the number of dn is the original number + 1.
> ```
> ## DataStreamer$findNewDatanode
> if (nodes.length != original.length + 1) {
> throw new IOException(
> "Failed to replace a bad datanode on the existing pipeline "
> + "due to no more good datanodes being available to try. "
> + "(Nodes: current=" + Arrays.asList(nodes)
> + ", original=" + Arrays.asList(original) + "). "
> + "The current failed datanode replacement policy is "
> + dfsClient.dtpReplaceDatanodeOnFailure
> + ", and a client may configure this via '"
> + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
> + "' in its configuration.");
> }
> ```
> The root cause is that Namenode$getAdditionalDatanode returns multi datanodes
> , not one in DataStreamer.addDatanode2ExistingPipeline.
>
> Maybe we can fix it in
[jira] [Comment Edited] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403006#comment-17403006
]
Max Xie edited comment on HDFS-16182 at 8/24/21, 4:00 AM:
---
In my cluster, we use BlockPlacementPolicyDefault to choose dn and the number
of SSD DN is much less than DISK DN. It may cause to some block that should
be placed to SSD DNs fallback to place DISK DNs when SSD DNs are too busy or no
enough place. Consider the following scenario.
# Create empty file /foo_file
# Set its storagepolicy to All_SSD
# Put data to /foo_file
# /foo_file gets 3 DISK dns for pipeline because SSD dns are too busy at
the beginning.
# When it transfers data in pipeline, one of 3 DISK dns shut down.
# The client need to get one new dn for existing pipeline.
# If SSD dns are available at the moment, namenode will choose the 3 SSD dns
and return it to the client. However, the client just need one new dn,
namenode returns 3 new SSD dn and the client fail in DataStreamer.
was (Author: max2049):
In my cluster, we use BlockPlacementPolicyDefault to choose dn and the number
of SSD DN is much less than DISK DN. It may cause to some block that should
be placed to SSD DNs fallback to place DISK DNs when SSD DNs are too busy or no
enough place. The steps are as follow.
# Create empty file /foo_file
# Set its storagepolicy to All_SSD
# Put data to /foo_file
# /foo_file gets 3 DISK dns for pipeline because SSD dns are too busy at
the beginning.
# When it transfers data in pipeline, one of 3 DISK dns shut down.
# The client need to get one new dn for existing pipeline.
# If SSD dns are available at the moment, namenode will choose the 3 SSD dns
and return it to the client. However, the client just need one new dn,
namenode returns 3 new SSD dn and the client fail.
> numOfReplicas is given the wrong value in
> BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
> Heterogeneous Storage
> ---
>
> Key: HDFS-16182
> URL: https://issues.apache.org/jira/browse/HDFS-16182
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-16182.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
> better performance. Sometimes hdfs client transfer data in pipline, it will
> throw IOException and exit. Exception logs are below:
> ```
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
>
> DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
>
> DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
>
> DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
>
> original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> ```
> After search it, I found when existing pipline need replace new dn to
> transfer data, the client will get one additional dn from namenode and check
> that the number of dn is the original number + 1.
> ```
> ## DataStreamer$findNewDatanode
> if (nodes.length != original.length + 1) {
> throw new IOException(
> "Failed to replace a bad datanode on the existing pipeline "
> + "due to no more good datanodes being available to try. "
> + "(Nodes: current=" + Arrays.asList(nodes)
> + ", original=" + Arrays.asList(original) + "). "
> + "The current failed datanode replacement policy is "
> + dfsClient.dtpReplaceDatanodeOnFailure
> + ", and a client may configure this via '"
> + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
> + "' in its configuration.");
> }
> ```
> The root cause is that Namenode$getAdditionalDatanode returns multi datanodes
> , not one in DataStreamer.addDatanode2ExistingPipeline.
>
> Maybe we can fix it in BlockPlacementPolicyDefault$chooseTarget. I think
> numOfReplicas should not be assigned by
[jira] [Comment Edited] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403006#comment-17403006
]
Max Xie edited comment on HDFS-16182 at 8/24/21, 2:57 AM:
---
In my cluster, we use BlockPlacementPolicyDefault to choose dn and the number
of SSD DN is much less than DISK DN. It may cause to some block that should
be placed to SSD DNs fallback to place DISK DNs when SSD DNs are too busy or no
enough place. The steps are as follow.
# Create empty file /foo_file
# Set its storagepolicy to All_SSD
# Put data to /foo_file
# /foo_file gets 3 DISK dns for pipeline because SSD dns are too busy at
the beginning.
# When it transfers data in pipeline, one of 3 DISK dns shut down.
# The client need to get one new dn for existing pipeline.
# If SSD dns are available at the moment, namenode will choose the 3 SSD dns
and return it to the client. However, the client just need one new dn,
namenode returns 3 new SSD dn and the client fail.
was (Author: max2049):
In my cluster, we use BlockPlacementPolicyDefault to choose dn and the number
of SSD DN is much less than DISK DN. It may cause to some block that should
be placed to SSD DNs fallback to place DISK DNs when SSD DNs are too busy or no
enough place. The steps are as follow.
# Create empty file /foo_file
# Set its storagepolicy to All_SSD
# Put data to /foo_file
# /foo_file gets 3 DISK dns for pipeline because SSD dns are too busy at
the beginning.
# When it transfers data in pipeline, one of 3 DISK dns shut down.
# The client need to get one new dn for existing pipeline.
# If SSD dns are available at the moment, namenode will choose the 3 SSD dns
and return it to the client. However, the client just need one new dn,
namenode returns 3 new SSD dn and the client fail.
> numOfReplicas is given the wrong value in
> BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
> Heterogeneous Storage
> ---
>
> Key: HDFS-16182
> URL: https://issues.apache.org/jira/browse/HDFS-16182
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-16182.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
> better performance. Sometimes hdfs client transfer data in pipline, it will
> throw IOException and exit. Exception logs are below:
> ```
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
>
> DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
>
> DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
>
> DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
>
> original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> ```
> After search it, I found when existing pipline need replace new dn to
> transfer data, the client will get one additional dn from namenode and check
> that the number of dn is the original number + 1.
> ```
> ## DataStreamer$findNewDatanode
> if (nodes.length != original.length + 1) {
> throw new IOException(
> "Failed to replace a bad datanode on the existing pipeline "
> + "due to no more good datanodes being available to try. "
> + "(Nodes: current=" + Arrays.asList(nodes)
> + ", original=" + Arrays.asList(original) + "). "
> + "The current failed datanode replacement policy is "
> + dfsClient.dtpReplaceDatanodeOnFailure
> + ", and a client may configure this via '"
> + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
> + "' in its configuration.");
> }
> ```
> The root cause is that Namenode$getAdditionalDatanode returns multi datanodes
> , not one in DataStreamer.addDatanode2ExistingPipeline.
>
> Maybe we can fix it in BlockPlacementPolicyDefault$chooseTarget. I think
> numOfReplicas should not be assigned by requiredStorageTypes.
>
>
>
[jira] [Comment Edited] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403006#comment-17403006
]
Max Xie edited comment on HDFS-16182 at 8/24/21, 2:55 AM:
---
In my cluster, we use BlockPlacementPolicyDefault to choose dn and the number
of SSD DN is much less than DISK DN. It may cause to some block that should
be placed to SSD DNs fallback to place DISK DNs when SSD DNs are too busy or no
enough place. The steps are as follow.
# Create empty file /foo_file
# Set its storagepolicy to All_SSD
# Put data to /foo_file
# /foo_file gets 3 DISK dns for pipeline because SSD dns are too busy at
the beginning.
# When it transfers data in pipeline, one of 3 DISK dns shut down.
# The client need to get one new dn for existing pipeline.
# If SSD dns are available at the moment, namenode will choose the 3 SSD dns
and return it to the client. However, the client just need one new dn,
namenode returns 3 new SSD dn and the client fail.
was (Author: max2049):
In my cluster, we use BlockPlacementPolicyDefault to choose dn and the number
of SSD DN is much less than DISK DN. It may cause to some block that should
be placed to SSD DNs fallback to place DISK DNs when SSD DNs are too busy or no
enough place.
# create empty file /foo_file
# set its storagepolicy to All_SSD
# put data to /foo_file
# /foo_file gets 3 DISK dns for pipeline because SSD dns are too busy at
the beginning.
# when it transfers data in pipeline, one of 3 DISK dns down and
[shut|http://dict.youdao.com/search?q=shut=chrome.extension] [ʃʌt]
[详细|http://dict.youdao.com/search?q=shut=chrome.extension]X
基本翻译
vt. 关闭；停业；幽禁
vi. 关上；停止营业
n. 关闭
adj. 关闭的；围绕的
n. (Shut)人名；(俄)舒特；(中)室(广东话·威妥玛)
网络释义
[Shut:|http://dict.youdao.com/search?q=Shut=chrome.extension=eng]
此路不通
[Eyes Wide
Shut:|http://dict.youdao.com/search?q=Eyes%20Wide%20Shut=chrome.extension=eng]
大开眼戒
[shut
out:|http://dict.youdao.com/search?q=shut%20out=chrome.extension=eng]
排除
> numOfReplicas is given the wrong value in
> BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
> Heterogeneous Storage
> ---
>
> Key: HDFS-16182
> URL: https://issues.apache.org/jira/browse/HDFS-16182
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-16182.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
> better performance. Sometimes hdfs client transfer data in pipline, it will
> throw IOException and exit. Exception logs are below:
> ```
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
>
> DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
>
> DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
>
> DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
>
> original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> ```
> After search it, I found when existing pipline need replace new dn to
> transfer data, the client will get one additional dn from namenode and check
> that the number of dn is the original number + 1.
> ```
> ## DataStreamer$findNewDatanode
> if (nodes.length != original.length + 1) {
> throw new IOException(
> "Failed to replace a bad datanode on the existing pipeline "
> + "due to no more good datanodes being available to try. "
> + "(Nodes: current=" + Arrays.asList(nodes)
> + ", original=" + Arrays.asList(original) + "). "
> + "The current failed datanode replacement policy is "
> + dfsClient.dtpReplaceDatanodeOnFailure
> + ", and a client may configure this via '"
> + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
> + "' in its configuration.");
> }
> ```
> The root cause is that Namenode$getAdditionalDatanode returns multi datanodes
> , not one in
[jira] [Comment Edited] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403006#comment-17403006
]
Max Xie edited comment on HDFS-16182 at 8/24/21, 2:45 AM:
---
In my cluster, we use BlockPlacementPolicyDefault to choose dn and the number
of SSD DN is much less than DISK DN. It may cause to some block that should
be placed to SSD DNs fallback to place DISK DNs when SSD DNs are too busy or no
enough place.
# create empty file /foo_file
# set its storagepolicy to All_SSD
# put data to /foo_file
# /foo_file gets 3 DISK dns for pipeline because SSD dns are too busy at
the beginning.
# when it transfers data in pipeline, one of 3 DISK dns down and
[shut|http://dict.youdao.com/search?q=shut=chrome.extension] [ʃʌt]
[详细|http://dict.youdao.com/search?q=shut=chrome.extension]X
基本翻译
vt. 关闭；停业；幽禁
vi. 关上；停止营业
n. 关闭
adj. 关闭的；围绕的
n. (Shut)人名；(俄)舒特；(中)室(广东话·威妥玛)
网络释义
[Shut:|http://dict.youdao.com/search?q=Shut=chrome.extension=eng]
此路不通
[Eyes Wide
Shut:|http://dict.youdao.com/search?q=Eyes%20Wide%20Shut=chrome.extension=eng]
大开眼戒
[shut
out:|http://dict.youdao.com/search?q=shut%20out=chrome.extension=eng]
排除
was (Author: max2049):
In my cluster, we use BlockPlacementPolicyDefault to choose dn and the number
of SSD DN is much less than DISK DN. It may cause to some block that should
be placed to SSD DNs fallback to place DISK DNs.
> numOfReplicas is given the wrong value in
> BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
> Heterogeneous Storage
> ---
>
> Key: HDFS-16182
> URL: https://issues.apache.org/jira/browse/HDFS-16182
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-16182.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
> better performance. Sometimes hdfs client transfer data in pipline, it will
> throw IOException and exit. Exception logs are below:
> ```
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
>
> DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
>
> DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
>
> DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
>
> original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> ```
> After search it, I found when existing pipline need replace new dn to
> transfer data, the client will get one additional dn from namenode and check
> that the number of dn is the original number + 1.
> ```
> ## DataStreamer$findNewDatanode
> if (nodes.length != original.length + 1) {
> throw new IOException(
> "Failed to replace a bad datanode on the existing pipeline "
> + "due to no more good datanodes being available to try. "
> + "(Nodes: current=" + Arrays.asList(nodes)
> + ", original=" + Arrays.asList(original) + "). "
> + "The current failed datanode replacement policy is "
> + dfsClient.dtpReplaceDatanodeOnFailure
> + ", and a client may configure this via '"
> + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
> + "' in its configuration.");
> }
> ```
> The root cause is that Namenode$getAdditionalDatanode returns multi datanodes
> , not one in DataStreamer.addDatanode2ExistingPipeline.
>
> Maybe we can fix it in BlockPlacementPolicyDefault$chooseTarget. I think
> numOfReplicas should not be assigned by requiredStorageTypes.
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403460#comment-17403460
]
Max Xie commented on HDFS-16182:
-
hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl,
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
Unit test failures seem unrelated. And I test it in IDEA locally again, these
unit tests pass.
> numOfReplicas is given the wrong value in
> BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
> Heterogeneous Storage
> ---
>
> Key: HDFS-16182
> URL: https://issues.apache.org/jira/browse/HDFS-16182
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-16182.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
> better performance. Sometimes hdfs client transfer data in pipline, it will
> throw IOException and exit. Exception logs are below:
> ```
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
>
> DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
>
> DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
>
> DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
>
> original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> ```
> After search it, I found when existing pipline need replace new dn to
> transfer data, the client will get one additional dn from namenode and check
> that the number of dn is the original number + 1.
> ```
> ## DataStreamer$findNewDatanode
> if (nodes.length != original.length + 1) {
> throw new IOException(
> "Failed to replace a bad datanode on the existing pipeline "
> + "due to no more good datanodes being available to try. "
> + "(Nodes: current=" + Arrays.asList(nodes)
> + ", original=" + Arrays.asList(original) + "). "
> + "The current failed datanode replacement policy is "
> + dfsClient.dtpReplaceDatanodeOnFailure
> + ", and a client may configure this via '"
> + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
> + "' in its configuration.");
> }
> ```
> The root cause is that Namenode$getAdditionalDatanode returns multi datanodes
> , not one in DataStreamer.addDatanode2ExistingPipeline.
>
> Maybe we can fix it in BlockPlacementPolicyDefault$chooseTarget. I think
> numOfReplicas should not be assigned by requiredStorageTypes.
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403009#comment-17403009
]
Max Xie commented on HDFS-16182:
-
[~weichiu] [~sodonnell] Any thought about it. Thanks for the reviews.
> numOfReplicas is given the wrong value in
> BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
> Heterogeneous Storage
> ---
>
> Key: HDFS-16182
> URL: https://issues.apache.org/jira/browse/HDFS-16182
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-16182.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
> better performance. Sometimes hdfs client transfer data in pipline, it will
> throw IOException and exit. Exception logs are below:
> ```
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
>
> DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
>
> DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
>
> DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
>
> original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> ```
> After search it, I found when existing pipline need replace new dn to
> transfer data, the client will get one additional dn from namenode and check
> that the number of dn is the original number + 1.
> ```
> ## DataStreamer$findNewDatanode
> if (nodes.length != original.length + 1) {
> throw new IOException(
> "Failed to replace a bad datanode on the existing pipeline "
> + "due to no more good datanodes being available to try. "
> + "(Nodes: current=" + Arrays.asList(nodes)
> + ", original=" + Arrays.asList(original) + "). "
> + "The current failed datanode replacement policy is "
> + dfsClient.dtpReplaceDatanodeOnFailure
> + ", and a client may configure this via '"
> + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
> + "' in its configuration.");
> }
> ```
> The root cause is that Namenode$getAdditionalDatanode returns multi datanodes
> , not one in DataStreamer.addDatanode2ExistingPipeline.
>
> Maybe we can fix it in BlockPlacementPolicyDefault$chooseTarget. I think
> numOfReplicas should not be assigned by requiredStorageTypes.
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Xie updated HDFS-16182:
Attachment: HDFS-16182.patch
Status: Patch Available (was: In Progress)
> numOfReplicas is given the wrong value in
> BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
> Heterogeneous Storage
> ---
>
> Key: HDFS-16182
> URL: https://issues.apache.org/jira/browse/HDFS-16182
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-16182.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
> better performance. Sometimes hdfs client transfer data in pipline, it will
> throw IOException and exit. Exception logs are below:
> ```
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
>
> DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
>
> DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
>
> DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
>
> original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> ```
> After search it, I found when existing pipline need replace new dn to
> transfer data, the client will get one additional dn from namenode and check
> that the number of dn is the original number + 1.
> ```
> ## DataStreamer$findNewDatanode
> if (nodes.length != original.length + 1) {
> throw new IOException(
> "Failed to replace a bad datanode on the existing pipeline "
> + "due to no more good datanodes being available to try. "
> + "(Nodes: current=" + Arrays.asList(nodes)
> + ", original=" + Arrays.asList(original) + "). "
> + "The current failed datanode replacement policy is "
> + dfsClient.dtpReplaceDatanodeOnFailure
> + ", and a client may configure this via '"
> + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
> + "' in its configuration.");
> }
> ```
> The root cause is that Namenode$getAdditionalDatanode returns multi datanodes
> , not one in DataStreamer.addDatanode2ExistingPipeline.
>
> Maybe we can fix it in BlockPlacementPolicyDefault$chooseTarget. I think
> numOfReplicas should not be assigned by requiredStorageTypes.
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Xie updated HDFS-16182:
Attachment: (was: HDFS-16182.patch)
> numOfReplicas is given the wrong value in
> BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
> Heterogeneous Storage
> ---
>
> Key: HDFS-16182
> URL: https://issues.apache.org/jira/browse/HDFS-16182
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
> better performance. Sometimes hdfs client transfer data in pipline, it will
> throw IOException and exit. Exception logs are below:
> ```
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
>
> DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
>
> DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
>
> DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
>
> original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> ```
> After search it, I found when existing pipline need replace new dn to
> transfer data, the client will get one additional dn from namenode and check
> that the number of dn is the original number + 1.
> ```
> ## DataStreamer$findNewDatanode
> if (nodes.length != original.length + 1) {
> throw new IOException(
> "Failed to replace a bad datanode on the existing pipeline "
> + "due to no more good datanodes being available to try. "
> + "(Nodes: current=" + Arrays.asList(nodes)
> + ", original=" + Arrays.asList(original) + "). "
> + "The current failed datanode replacement policy is "
> + dfsClient.dtpReplaceDatanodeOnFailure
> + ", and a client may configure this via '"
> + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
> + "' in its configuration.");
> }
> ```
> The root cause is that Namenode$getAdditionalDatanode returns multi datanodes
> , not one in DataStreamer.addDatanode2ExistingPipeline.
>
> Maybe we can fix it in BlockPlacementPolicyDefault$chooseTarget. I think
> numOfReplicas should not be assigned by requiredStorageTypes.
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403006#comment-17403006
]
Max Xie commented on HDFS-16182:
-
In my cluster, we use BlockPlacementPolicyDefault to choose dn and the number
of SSD DN is much less than DISK DN. It may cause to some block that should
be placed to SSD DNs fallback to place DISK DNs.
> numOfReplicas is given the wrong value in
> BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
> Heterogeneous Storage
> ---
>
> Key: HDFS-16182
> URL: https://issues.apache.org/jira/browse/HDFS-16182
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-16182.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
> better performance. Sometimes hdfs client transfer data in pipline, it will
> throw IOException and exit. Exception logs are below:
> ```
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
>
> DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
>
> DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
>
> DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
>
> original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> ```
> After search it, I found when existing pipline need replace new dn to
> transfer data, the client will get one additional dn from namenode and check
> that the number of dn is the original number + 1.
> ```
> ## DataStreamer$findNewDatanode
> if (nodes.length != original.length + 1) {
> throw new IOException(
> "Failed to replace a bad datanode on the existing pipeline "
> + "due to no more good datanodes being available to try. "
> + "(Nodes: current=" + Arrays.asList(nodes)
> + ", original=" + Arrays.asList(original) + "). "
> + "The current failed datanode replacement policy is "
> + dfsClient.dtpReplaceDatanodeOnFailure
> + ", and a client may configure this via '"
> + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
> + "' in its configuration.");
> }
> ```
> The root cause is that Namenode$getAdditionalDatanode returns multi datanodes
> , not one in DataStreamer.addDatanode2ExistingPipeline.
>
> Maybe we can fix it in BlockPlacementPolicyDefault$chooseTarget. I think
> numOfReplicas should not be assigned by requiredStorageTypes.
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Xie updated HDFS-16182:
Attachment: HDFS-16182.patch
> numOfReplicas is given the wrong value in
> BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
> Heterogeneous Storage
> ---
>
> Key: HDFS-16182
> URL: https://issues.apache.org/jira/browse/HDFS-16182
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-16182.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
> better performance. Sometimes hdfs client transfer data in pipline, it will
> throw IOException and exit. Exception logs are below:
> ```
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
>
> DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
>
> DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
>
> DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
>
> original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> ```
> After search it, I found when existing pipline need replace new dn to
> transfer data, the client will get one additional dn from namenode and check
> that the number of dn is the original number + 1.
> ```
> ## DataStreamer$findNewDatanode
> if (nodes.length != original.length + 1) {
> throw new IOException(
> "Failed to replace a bad datanode on the existing pipeline "
> + "due to no more good datanodes being available to try. "
> + "(Nodes: current=" + Arrays.asList(nodes)
> + ", original=" + Arrays.asList(original) + "). "
> + "The current failed datanode replacement policy is "
> + dfsClient.dtpReplaceDatanodeOnFailure
> + ", and a client may configure this via '"
> + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
> + "' in its configuration.");
> }
> ```
> The root cause is that Namenode$getAdditionalDatanode returns multi datanodes
> , not one in DataStreamer.addDatanode2ExistingPipeline.
>
> Maybe we can fix it in BlockPlacementPolicyDefault$chooseTarget. I think
> numOfReplicas should not be assigned by requiredStorageTypes.
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Xie reassigned HDFS-16182:
---
Assignee: Max Xie
> numOfReplicas is given the wrong value in
> BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
> Heterogeneous Storage
> ---
>
> Key: HDFS-16182
> URL: https://issues.apache.org/jira/browse/HDFS-16182
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
> better performance. Sometimes hdfs client transfer data in pipline, it will
> throw IOException and exit. Exception logs are below:
> ```
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
>
> DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
>
> DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
>
> DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
>
> original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> ```
> After search it, I found when existing pipline need replace new dn to
> transfer data, the client will get one additional dn from namenode and check
> that the number of dn is the original number + 1.
> ```
> ## DataStreamer$findNewDatanode
> if (nodes.length != original.length + 1) {
> throw new IOException(
> "Failed to replace a bad datanode on the existing pipeline "
> + "due to no more good datanodes being available to try. "
> + "(Nodes: current=" + Arrays.asList(nodes)
> + ", original=" + Arrays.asList(original) + "). "
> + "The current failed datanode replacement policy is "
> + dfsClient.dtpReplaceDatanodeOnFailure
> + ", and a client may configure this via '"
> + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
> + "' in its configuration.");
> }
> ```
> The root cause is that Namenode$getAdditionalDatanode returns multi datanodes
> , not one in DataStreamer.addDatanode2ExistingPipeline.
>
> Maybe we can fix it in BlockPlacementPolicyDefault$chooseTarget. I think
> numOfReplicas should not be assigned by requiredStorageTypes.
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-16182 started by Max Xie.
---
> numOfReplicas is given the wrong value in
> BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
> Heterogeneous Storage
> ---
>
> Key: HDFS-16182
> URL: https://issues.apache.org/jira/browse/HDFS-16182
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
> better performance. Sometimes hdfs client transfer data in pipline, it will
> throw IOException and exit. Exception logs are below:
> ```
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
>
> DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
>
> DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
>
> DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
>
> original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> ```
> After search it, I found when existing pipline need replace new dn to
> transfer data, the client will get one additional dn from namenode and check
> that the number of dn is the original number + 1.
> ```
> ## DataStreamer$findNewDatanode
> if (nodes.length != original.length + 1) {
> throw new IOException(
> "Failed to replace a bad datanode on the existing pipeline "
> + "due to no more good datanodes being available to try. "
> + "(Nodes: current=" + Arrays.asList(nodes)
> + ", original=" + Arrays.asList(original) + "). "
> + "The current failed datanode replacement policy is "
> + dfsClient.dtpReplaceDatanodeOnFailure
> + ", and a client may configure this via '"
> + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
> + "' in its configuration.");
> }
> ```
> The root cause is that Namenode$getAdditionalDatanode returns multi datanodes
> , not one in DataStreamer.addDatanode2ExistingPipeline.
>
> Maybe we can fix it in BlockPlacementPolicyDefault$chooseTarget. I think
> numOfReplicas should not be assigned by requiredStorageTypes.
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
Max Xie created HDFS-16182:
---
Summary: numOfReplicas is given the wrong value in
BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
Heterogeneous Storage
Key: HDFS-16182
URL: https://issues.apache.org/jira/browse/HDFS-16182
Project: Hadoop HDFS
Issue Type: Bug
Components: namanode
Affects Versions: 3.4.0
Reporter: Max Xie
In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
better performance. Sometimes hdfs client transfer data in pipline, it will
throw IOException and exit. Exception logs are below:
```
java.io.IOException: Failed to replace a bad datanode on the existing pipeline
due to no more good datanodes being available to try. (Nodes:
current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
The current failed datanode replacement policy is DEFAULT, and a client may
configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'
in its configuration.
```
After search it, I found when existing pipline need replace new dn to
transfer data, the client will get one additional dn from namenode and check
that the number of dn is the original number + 1.
```
## DataStreamer$findNewDatanode
if (nodes.length != original.length + 1) {
throw new IOException(
"Failed to replace a bad datanode on the existing pipeline "
+ "due to no more good datanodes being available to try. "
+ "(Nodes: current=" + Arrays.asList(nodes)
+ ", original=" + Arrays.asList(original) + "). "
+ "The current failed datanode replacement policy is "
+ dfsClient.dtpReplaceDatanodeOnFailure
+ ", and a client may configure this via '"
+ BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
+ "' in its configuration.");
}
```
The root cause is that Namenode$getAdditionalDatanode returns multi datanodes ,
not one in DataStreamer.addDatanode2ExistingPipeline.
Maybe we can fix it in BlockPlacementPolicyDefault$chooseTarget. I think
numOfReplicas should not be assigned by requiredStorageTypes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16180) FsVolumeImpl.nextBlock should consider that the block meta file has been deleted.
[ https://issues.apache.org/jira/browse/HDFS-16180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie reassigned HDFS-16180: --- Assignee: Max Xie > FsVolumeImpl.nextBlock should consider that the block meta file has been > deleted. > - > > Key: HDFS-16180 > URL: https://issues.apache.org/jira/browse/HDFS-16180 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.3.0, 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In my cluster, we found that when VolumeScanner run, sometime dn will throw > some error log below > ``` > > 2021-08-19 08:00:11,549 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: > Deleted BP-1020175758-nnip-1597745872895 blk_1142977964_69237147 URI > file:/disk1/dfs/data/current/BP-1020175758- > nnip-1597745872895/current/finalized/subdir0/subdir21/blk_1142977964 > 2021-08-19 08:00:48,368 ERROR > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl: > nextBlock(DS-060c8e4c-1ef6-49f5-91ef-91957356891a, BP-1020175758- > nnip-1597745872895): I/O error > java.io.IOException: Meta file not found, > blockFile=/disk1/dfs/data/current/BP-1020175758- > nnip-1597745872895/current/finalized/subdir0/subdir21/blk_1142977964 > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetUtil.findMetaFile(FsDatasetUtil.java:101) > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl$BlockIteratorImpl.nextBlock(FsVolumeImpl.java:809) > at > org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:528) > at > org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:628) > 2021-08-19 08:00:48,368 WARN > org.apache.hadoop.hdfs.server.datanode.VolumeScanner: > VolumeScanner(/disk1/dfs/data, DS-060c8e4c-1ef6-49f5-91ef-91957356891a): > nextBlock error on > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl$BlockIteratorImpl@7febc6b4 > ``` > When VolumeScanner scan block blk_1142977964, it has been deleted by > datanode, scanner can not find the meta file of blk_1142977964, so it throw > these error log. > > Maybe we should handle FileNotFoundException during nextblock to reduce error > log and nextblock retry times. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-16180) FsVolumeImpl.nextBlock should consider that the block meta file has been deleted.
[ https://issues.apache.org/jira/browse/HDFS-16180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-16180 started by Max Xie. --- > FsVolumeImpl.nextBlock should consider that the block meta file has been > deleted. > - > > Key: HDFS-16180 > URL: https://issues.apache.org/jira/browse/HDFS-16180 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.3.0, 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In my cluster, we found that when VolumeScanner run, sometime dn will throw > some error log below > ``` > > 2021-08-19 08:00:11,549 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: > Deleted BP-1020175758-nnip-1597745872895 blk_1142977964_69237147 URI > file:/disk1/dfs/data/current/BP-1020175758- > nnip-1597745872895/current/finalized/subdir0/subdir21/blk_1142977964 > 2021-08-19 08:00:48,368 ERROR > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl: > nextBlock(DS-060c8e4c-1ef6-49f5-91ef-91957356891a, BP-1020175758- > nnip-1597745872895): I/O error > java.io.IOException: Meta file not found, > blockFile=/disk1/dfs/data/current/BP-1020175758- > nnip-1597745872895/current/finalized/subdir0/subdir21/blk_1142977964 > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetUtil.findMetaFile(FsDatasetUtil.java:101) > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl$BlockIteratorImpl.nextBlock(FsVolumeImpl.java:809) > at > org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:528) > at > org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:628) > 2021-08-19 08:00:48,368 WARN > org.apache.hadoop.hdfs.server.datanode.VolumeScanner: > VolumeScanner(/disk1/dfs/data, DS-060c8e4c-1ef6-49f5-91ef-91957356891a): > nextBlock error on > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl$BlockIteratorImpl@7febc6b4 > ``` > When VolumeScanner scan block blk_1142977964, it has been deleted by > datanode, scanner can not find the meta file of blk_1142977964, so it throw > these error log. > > Maybe we should handle FileNotFoundException during nextblock to reduce error > log and nextblock retry times. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16180) FsVolumeImpl.nextBlock should consider that the block meta file has been deleted.
Max Xie created HDFS-16180: --- Summary: FsVolumeImpl.nextBlock should consider that the block meta file has been deleted. Key: HDFS-16180 URL: https://issues.apache.org/jira/browse/HDFS-16180 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.3.0, 3.4.0 Reporter: Max Xie In my cluster, we found that when VolumeScanner run, sometime dn will throw some error log below ``` 2021-08-19 08:00:11,549 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-1020175758-nnip-1597745872895 blk_1142977964_69237147 URI file:/disk1/dfs/data/current/BP-1020175758- nnip-1597745872895/current/finalized/subdir0/subdir21/blk_1142977964 2021-08-19 08:00:48,368 ERROR org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl: nextBlock(DS-060c8e4c-1ef6-49f5-91ef-91957356891a, BP-1020175758- nnip-1597745872895): I/O error java.io.IOException: Meta file not found, blockFile=/disk1/dfs/data/current/BP-1020175758- nnip-1597745872895/current/finalized/subdir0/subdir21/blk_1142977964 at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetUtil.findMetaFile(FsDatasetUtil.java:101) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl$BlockIteratorImpl.nextBlock(FsVolumeImpl.java:809) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:528) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:628) 2021-08-19 08:00:48,368 WARN org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/disk1/dfs/data, DS-060c8e4c-1ef6-49f5-91ef-91957356891a): nextBlock error on org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl$BlockIteratorImpl@7febc6b4 ``` When VolumeScanner scan block blk_1142977964, it has been deleted by datanode, scanner can not find the meta file of blk_1142977964, so it throw these error log. Maybe we should handle FileNotFoundException during nextblock to reduce error log and nextblock retry times. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15886) Add a way to get protected dirs from a special configuration file
[ https://issues.apache.org/jira/browse/HDFS-15886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie updated HDFS-15886: Attachment: (was: HDFS-15886.patch) > Add a way to get protected dirs from a special configuration file > - > > Key: HDFS-15886 > URL: https://issues.apache.org/jira/browse/HDFS-15886 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-15886.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > We used protected dirs to ensure that important data directories cannot be > deleted by mistake. But protected dirs can only be configured in > hdfs-site.xml. > For ease of management, we add a way to get the list of protected dirs from > a special configuration file. > How to use. > 1. set the config in hdfs-site.xml > ``` > > fs.protected.directories > > /hdfs/path/1,/hdfs/path/2,[file:///path/to/protected.dirs.config] > > ``` > 2. add some protected dirs to the config file > ([file:///path/to/protected.dirs.config]) > ``` > /hdfs/path/4 > /hdfs/path/5 > ``` > 3. use command to refresh fs.protected.directories instead of > FSDirectory.setProtectedDirectories(..) > ``` > hdfs dfsadmin -refreshProtectedDirectories > ``` > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15886) Add a way to get protected dirs from a special configuration file
[ https://issues.apache.org/jira/browse/HDFS-15886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie updated HDFS-15886: Attachment: HDFS-15886.patch Status: Patch Available (was: In Progress) > Add a way to get protected dirs from a special configuration file > - > > Key: HDFS-15886 > URL: https://issues.apache.org/jira/browse/HDFS-15886 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-15886.patch, HDFS-15886.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > We used protected dirs to ensure that important data directories cannot be > deleted by mistake. But protected dirs can only be configured in > hdfs-site.xml. > For ease of management, we add a way to get the list of protected dirs from > a special configuration file. > How to use. > 1. set the config in hdfs-site.xml > ``` > > fs.protected.directories > > /hdfs/path/1,/hdfs/path/2,[file:///path/to/protected.dirs.config] > > ``` > 2. add some protected dirs to the config file > ([file:///path/to/protected.dirs.config]) > ``` > /hdfs/path/4 > /hdfs/path/5 > ``` > 3. use command to refresh fs.protected.directories instead of > FSDirectory.setProtectedDirectories(..) > ``` > hdfs dfsadmin -refreshProtectedDirectories > ``` > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15175) Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog
[
https://issues.apache.org/jira/browse/HDFS-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388487#comment-17388487
]
Max Xie edited comment on HDFS-15175 at 7/28/21, 6:21 AM:
---
[~sodonnell] Agree with you . One solution is to deep copy the op.
But considering namenode performance, we just deep copy CloseOp block and merge
the patch to our hdfs cluster(250+ DNs, 270 million block ) . It run well so
far.
was (Author: max2049):
[~sodonnell] Agree with you . One solution is to deep copy the op.
> Multiple CloseOp shared block instance causes the standby namenode to crash
> when rolling editlog
>
>
> Key: HDFS-15175
> URL: https://issues.apache.org/jira/browse/HDFS-15175
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Yicong Cai
>Assignee: Wan Chang
>Priority: Critical
> Labels: NameNode
> Attachments: HDFS-15175-trunk.1.patch
>
>
>
> {panel:title=Crash exception}
> 2020-02-16 09:24:46,426 [507844305] - ERROR [Edit log
> tailer:FSEditLogLoader@245] - Encountered exception on operation CloseOp
> [length=0, inodeId=0, path=..., replication=3, mtime=1581816138774,
> atime=1581814760398, blockSize=536870912, blocks=[blk_5568434562_4495417845],
> permissions=da_music:hdfs:rw-r-, aclEntries=null, clientName=,
> clientMachine=, overwrite=false, storagePolicyId=0, opCode=OP_CLOSE,
> txid=32625024993]
> java.io.IOException: File is not under construction: ..
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:442)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:146)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:891)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:872)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:262)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:395)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:348)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:365)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:479)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:361)
> {panel}
>
> {panel:title=Editlog}
>
> OP_REASSIGN_LEASE
>
> 32625021150
> DFSClient_NONMAPREDUCE_-969060727_197760
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
>
>
> ..
>
> OP_CLOSE
>
> 32625023743
> 0
> 0
> ..
> 3
> 1581816135883
> 1581814760398
> 536870912
>
>
> false
>
> 5568434562
> 185818644
> 4495417845
>
>
> da_music
> hdfs
> 416
>
>
>
> ..
>
> OP_TRUNCATE
>
> 32625024049
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
> ..
> 185818644
> 1581816136336
>
> 5568434562
> 185818648
> 4495417845
>
>
>
> ..
>
> OP_CLOSE
>
> 32625024993
> 0
> 0
> ..
> 3
> 1581816138774
> 1581814760398
> 536870912
>
>
> false
>
> 5568434562
> 185818644
> 4495417845
>
>
> da_music
> hdfs
> 416
>
>
>
> {panel}
>
>
> The block size should be 185818648 in the first CloseOp. When truncate is
> used, the block size becomes 185818644. The CloseOp/TruncateOp/CloseOp is
> synchronized to the JournalNode in the same batch. The block used by CloseOp
> twice is the same instance, which causes the first CloseOp has wrong block
> size. When SNN rolling Editlog, TruncateOp does not make the file to the
> UnderConstruction state. Then, when the second CloseOp is executed, the file
> is not in the UnderConstruction state, and SNN crashes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15175) Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog
[
https://issues.apache.org/jira/browse/HDFS-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388487#comment-17388487
]
Max Xie edited comment on HDFS-15175 at 7/28/21, 6:14 AM:
---
[~sodonnell] Agree with you . One solution is to deep copy the op.
was (Author: max2049):
[~sodonnell] Agree with you . One solution is to deep copy the op.
> Multiple CloseOp shared block instance causes the standby namenode to crash
> when rolling editlog
>
>
> Key: HDFS-15175
> URL: https://issues.apache.org/jira/browse/HDFS-15175
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Yicong Cai
>Assignee: Wan Chang
>Priority: Critical
> Labels: NameNode
> Attachments: HDFS-15175-trunk.1.patch
>
>
>
> {panel:title=Crash exception}
> 2020-02-16 09:24:46,426 [507844305] - ERROR [Edit log
> tailer:FSEditLogLoader@245] - Encountered exception on operation CloseOp
> [length=0, inodeId=0, path=..., replication=3, mtime=1581816138774,
> atime=1581814760398, blockSize=536870912, blocks=[blk_5568434562_4495417845],
> permissions=da_music:hdfs:rw-r-, aclEntries=null, clientName=,
> clientMachine=, overwrite=false, storagePolicyId=0, opCode=OP_CLOSE,
> txid=32625024993]
> java.io.IOException: File is not under construction: ..
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:442)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:146)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:891)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:872)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:262)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:395)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:348)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:365)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:479)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:361)
> {panel}
>
> {panel:title=Editlog}
>
> OP_REASSIGN_LEASE
>
> 32625021150
> DFSClient_NONMAPREDUCE_-969060727_197760
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
>
>
> ..
>
> OP_CLOSE
>
> 32625023743
> 0
> 0
> ..
> 3
> 1581816135883
> 1581814760398
> 536870912
>
>
> false
>
> 5568434562
> 185818644
> 4495417845
>
>
> da_music
> hdfs
> 416
>
>
>
> ..
>
> OP_TRUNCATE
>
> 32625024049
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
> ..
> 185818644
> 1581816136336
>
> 5568434562
> 185818648
> 4495417845
>
>
>
> ..
>
> OP_CLOSE
>
> 32625024993
> 0
> 0
> ..
> 3
> 1581816138774
> 1581814760398
> 536870912
>
>
> false
>
> 5568434562
> 185818644
> 4495417845
>
>
> da_music
> hdfs
> 416
>
>
>
> {panel}
>
>
> The block size should be 185818648 in the first CloseOp. When truncate is
> used, the block size becomes 185818644. The CloseOp/TruncateOp/CloseOp is
> synchronized to the JournalNode in the same batch. The block used by CloseOp
> twice is the same instance, which causes the first CloseOp has wrong block
> size. When SNN rolling Editlog, TruncateOp does not make the file to the
> UnderConstruction state. Then, when the second CloseOp is executed, the file
> is not in the UnderConstruction state, and SNN crashes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15175) Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog
[
https://issues.apache.org/jira/browse/HDFS-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388487#comment-17388487
]
Max Xie commented on HDFS-15175:
-
[~sodonnell] Agree with you . One solution is to deep copy the op.
> Multiple CloseOp shared block instance causes the standby namenode to crash
> when rolling editlog
>
>
> Key: HDFS-15175
> URL: https://issues.apache.org/jira/browse/HDFS-15175
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Yicong Cai
>Assignee: Wan Chang
>Priority: Critical
> Labels: NameNode
> Attachments: HDFS-15175-trunk.1.patch
>
>
>
> {panel:title=Crash exception}
> 2020-02-16 09:24:46,426 [507844305] - ERROR [Edit log
> tailer:FSEditLogLoader@245] - Encountered exception on operation CloseOp
> [length=0, inodeId=0, path=..., replication=3, mtime=1581816138774,
> atime=1581814760398, blockSize=536870912, blocks=[blk_5568434562_4495417845],
> permissions=da_music:hdfs:rw-r-, aclEntries=null, clientName=,
> clientMachine=, overwrite=false, storagePolicyId=0, opCode=OP_CLOSE,
> txid=32625024993]
> java.io.IOException: File is not under construction: ..
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:442)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:146)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:891)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:872)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:262)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:395)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:348)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:365)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:479)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:361)
> {panel}
>
> {panel:title=Editlog}
>
> OP_REASSIGN_LEASE
>
> 32625021150
> DFSClient_NONMAPREDUCE_-969060727_197760
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
>
>
> ..
>
> OP_CLOSE
>
> 32625023743
> 0
> 0
> ..
> 3
> 1581816135883
> 1581814760398
> 536870912
>
>
> false
>
> 5568434562
> 185818644
> 4495417845
>
>
> da_music
> hdfs
> 416
>
>
>
> ..
>
> OP_TRUNCATE
>
> 32625024049
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
> ..
> 185818644
> 1581816136336
>
> 5568434562
> 185818648
> 4495417845
>
>
>
> ..
>
> OP_CLOSE
>
> 32625024993
> 0
> 0
> ..
> 3
> 1581816138774
> 1581814760398
> 536870912
>
>
> false
>
> 5568434562
> 185818644
> 4495417845
>
>
> da_music
> hdfs
> 416
>
>
>
> {panel}
>
>
> The block size should be 185818648 in the first CloseOp. When truncate is
> used, the block size becomes 185818644. The CloseOp/TruncateOp/CloseOp is
> synchronized to the JournalNode in the same batch. The block used by CloseOp
> twice is the same instance, which causes the first CloseOp has wrong block
> size. When SNN rolling Editlog, TruncateOp does not make the file to the
> UnderConstruction state. Then, when the second CloseOp is executed, the file
> is not in the UnderConstruction state, and SNN crashes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15886) Add a way to get protected dirs from a special configuration file
[ https://issues.apache.org/jira/browse/HDFS-15886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie updated HDFS-15886: Attachment: HDFS-15886.patch > Add a way to get protected dirs from a special configuration file > - > > Key: HDFS-15886 > URL: https://issues.apache.org/jira/browse/HDFS-15886 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-15886.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > We used protected dirs to ensure that important data directories cannot be > deleted by mistake. But protected dirs can only be configured in > hdfs-site.xml. > For ease of management, we add a way to get the list of protected dirs from > a special configuration file. > How to use. > 1. set the config in hdfs-site.xml > ``` > > fs.protected.directories > > /hdfs/path/1,/hdfs/path/2,[file:///path/to/protected.dirs.config] > > ``` > 2. add some protected dirs to the config file > ([file:///path/to/protected.dirs.config]) > ``` > /hdfs/path/4 > /hdfs/path/5 > ``` > 3. use command to refresh fs.protected.directories instead of > FSDirectory.setProtectedDirectories(..) > ``` > hdfs dfsadmin -refreshProtectedDirectories > ``` > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15175) Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog
[
https://issues.apache.org/jira/browse/HDFS-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17322045#comment-17322045
]
Max Xie commented on HDFS-15175:
-
hi [~tomscut] thanks for your reply.
I think it is not convenient to find a way to reproduce the issue by relevant
APIs (create/close/truncate). (flagoff)
> Multiple CloseOp shared block instance causes the standby namenode to crash
> when rolling editlog
>
>
> Key: HDFS-15175
> URL: https://issues.apache.org/jira/browse/HDFS-15175
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Yicong Cai
>Assignee: Wan Chang
>Priority: Critical
> Labels: NameNode
> Attachments: HDFS-15175-trunk.1.patch
>
>
>
> {panel:title=Crash exception}
> 2020-02-16 09:24:46,426 [507844305] - ERROR [Edit log
> tailer:FSEditLogLoader@245] - Encountered exception on operation CloseOp
> [length=0, inodeId=0, path=..., replication=3, mtime=1581816138774,
> atime=1581814760398, blockSize=536870912, blocks=[blk_5568434562_4495417845],
> permissions=da_music:hdfs:rw-r-, aclEntries=null, clientName=,
> clientMachine=, overwrite=false, storagePolicyId=0, opCode=OP_CLOSE,
> txid=32625024993]
> java.io.IOException: File is not under construction: ..
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:442)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:146)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:891)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:872)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:262)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:395)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:348)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:365)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:479)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:361)
> {panel}
>
> {panel:title=Editlog}
>
> OP_REASSIGN_LEASE
>
> 32625021150
> DFSClient_NONMAPREDUCE_-969060727_197760
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
>
>
> ..
>
> OP_CLOSE
>
> 32625023743
> 0
> 0
> ..
> 3
> 1581816135883
> 1581814760398
> 536870912
>
>
> false
>
> 5568434562
> 185818644
> 4495417845
>
>
> da_music
> hdfs
> 416
>
>
>
> ..
>
> OP_TRUNCATE
>
> 32625024049
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
> ..
> 185818644
> 1581816136336
>
> 5568434562
> 185818648
> 4495417845
>
>
>
> ..
>
> OP_CLOSE
>
> 32625024993
> 0
> 0
> ..
> 3
> 1581816138774
> 1581814760398
> 536870912
>
>
> false
>
> 5568434562
> 185818644
> 4495417845
>
>
> da_music
> hdfs
> 416
>
>
>
> {panel}
>
>
> The block size should be 185818648 in the first CloseOp. When truncate is
> used, the block size becomes 185818644. The CloseOp/TruncateOp/CloseOp is
> synchronized to the JournalNode in the same batch. The block used by CloseOp
> twice is the same instance, which causes the first CloseOp has wrong block
> size. When SNN rolling Editlog, TruncateOp does not make the file to the
> UnderConstruction state. Then, when the second CloseOp is executed, the file
> is not in the UnderConstruction state, and SNN crashes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15175) Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog
[
https://issues.apache.org/jira/browse/HDFS-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17319014#comment-17319014
]
Max Xie commented on HDFS-15175:
-
cc [~tomscut] [~sodonnell]
Hi, how do you think the test above and about this issue?
> Multiple CloseOp shared block instance causes the standby namenode to crash
> when rolling editlog
>
>
> Key: HDFS-15175
> URL: https://issues.apache.org/jira/browse/HDFS-15175
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Yicong Cai
>Assignee: Wan Chang
>Priority: Critical
> Labels: NameNode
> Attachments: HDFS-15175-trunk.1.patch
>
>
>
> {panel:title=Crash exception}
> 2020-02-16 09:24:46,426 [507844305] - ERROR [Edit log
> tailer:FSEditLogLoader@245] - Encountered exception on operation CloseOp
> [length=0, inodeId=0, path=..., replication=3, mtime=1581816138774,
> atime=1581814760398, blockSize=536870912, blocks=[blk_5568434562_4495417845],
> permissions=da_music:hdfs:rw-r-, aclEntries=null, clientName=,
> clientMachine=, overwrite=false, storagePolicyId=0, opCode=OP_CLOSE,
> txid=32625024993]
> java.io.IOException: File is not under construction: ..
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:442)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:146)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:891)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:872)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:262)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:395)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:348)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:365)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:479)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:361)
> {panel}
>
> {panel:title=Editlog}
>
> OP_REASSIGN_LEASE
>
> 32625021150
> DFSClient_NONMAPREDUCE_-969060727_197760
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
>
>
> ..
>
> OP_CLOSE
>
> 32625023743
> 0
> 0
> ..
> 3
> 1581816135883
> 1581814760398
> 536870912
>
>
> false
>
> 5568434562
> 185818644
> 4495417845
>
>
> da_music
> hdfs
> 416
>
>
>
> ..
>
> OP_TRUNCATE
>
> 32625024049
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
> ..
> 185818644
> 1581816136336
>
> 5568434562
> 185818648
> 4495417845
>
>
>
> ..
>
> OP_CLOSE
>
> 32625024993
> 0
> 0
> ..
> 3
> 1581816138774
> 1581814760398
> 536870912
>
>
> false
>
> 5568434562
> 185818644
> 4495417845
>
>
> da_music
> hdfs
> 416
>
>
>
> {panel}
>
>
> The block size should be 185818648 in the first CloseOp. When truncate is
> used, the block size becomes 185818644. The CloseOp/TruncateOp/CloseOp is
> synchronized to the JournalNode in the same batch. The block used by CloseOp
> twice is the same instance, which causes the first CloseOp has wrong block
> size. When SNN rolling Editlog, TruncateOp does not make the file to the
> UnderConstruction state. Then, when the second CloseOp is executed, the file
> is not in the UnderConstruction state, and SNN crashes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15175) Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog
[
https://issues.apache.org/jira/browse/HDFS-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317704#comment-17317704
]
Max Xie commented on HDFS-15175:
-
I think this case is related to async edit logging, too. I try to add a test
about this case.
```
@Test
public void testEditLogAsync() throws Exception {
// start a cluster
Configuration conf = getConf();
conf.setBoolean(DFSConfigKeys.DFS_NAMENODE_EDITS_ASYNC_LOGGING, true);
MiniDFSCluster cluster = null;
FileSystem fileSys = null;
try {
cluster = new
MiniDFSCluster.Builder(conf).numDataNodes(NUM_DATA_NODES).build();
cluster.waitActive();
fileSys = cluster.getFileSystem();
final FSNamesystem namesystem = cluster.getNamesystem();
FSImage fsimage = namesystem.getFSImage();
final FSEditLogAsync editLog = (FSEditLogAsync)fsimage.getEditLog();
// prepare a file with one block
int blocksPerFile = 1;
short replication =1;
long blockSize = 2048;
long blockGenStamp = 1;
BlockInfo[] blocks = new BlockInfo[blocksPerFile];
for (int iB = 0; iB < blocksPerFile; ++iB) {
blocks[iB] =
new BlockInfoContiguous(new Block(0, blockSize, blockGenStamp),
replication);
}
INodeId inodeId = new INodeId();
final INodeFile inode = new INodeFile(inodeId.nextValue(), null,
new PermissionStatus("joeDoe", "people",
new FsPermission((short)0777))
, 0L, 0L, blocks, replication, blockSize);
editLog.logCloseFile("/testfile", inode);
//Simulate truncateOp, it will change block's numBytes to newBlockSize
int newBlockSize = 1024;
inode.getBlocks()[0].setNumBytes(newBlockSize);
// Quickly get CloseOp from the FSEditLogAsync.editPendingQ
// If not quickly, it may be consumed and can't reproduce the issue.
long closeOpBlockNumByte = ((CloseOp) editLog.getEditPendingQElementOp())
.getBlocks()[0].getNumBytes();
// closeOpBlockNumByte should be equal to blockSize, but it has been set to
newBlockSize
assertEquals(closeOpBlockNumByte, blockSize);
editLog.close();
} finally {
if(fileSys != null) fileSys.close();
if(cluster != null) cluster.shutdown();
}
}
// add getEditPendingQElementOp for testing in FSEditLogAsync
@VisibleForTesting
public FSEditLogOp getEditPendingQElementOp(){
return editPendingQ.element().op;
}
```
> Multiple CloseOp shared block instance causes the standby namenode to crash
> when rolling editlog
>
>
> Key: HDFS-15175
> URL: https://issues.apache.org/jira/browse/HDFS-15175
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Yicong Cai
>Assignee: Wan Chang
>Priority: Critical
> Labels: NameNode
> Attachments: HDFS-15175-trunk.1.patch
>
>
>
> {panel:title=Crash exception}
> 2020-02-16 09:24:46,426 [507844305] - ERROR [Edit log
> tailer:FSEditLogLoader@245] - Encountered exception on operation CloseOp
> [length=0, inodeId=0, path=..., replication=3, mtime=1581816138774,
> atime=1581814760398, blockSize=536870912, blocks=[blk_5568434562_4495417845],
> permissions=da_music:hdfs:rw-r-, aclEntries=null, clientName=,
> clientMachine=, overwrite=false, storagePolicyId=0, opCode=OP_CLOSE,
> txid=32625024993]
> java.io.IOException: File is not under construction: ..
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:442)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:146)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:891)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:872)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:262)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:395)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:348)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:365)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:479)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:361)
> {panel}
>
> {panel:title=Editlog}
>
> OP_REASSIGN_LEASE
>
> 32625021150
> DFSClient_NONMAPREDUCE_-969060727_197760
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
>
>
> ..
>
>
[jira] [Comment Edited] (HDFS-15175) Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog
[
https://issues.apache.org/jira/browse/HDFS-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316997#comment-17316997
]
Max Xie edited comment on HDFS-15175 at 4/8/21, 9:01 AM:
--
We encountered this bug on hdfs 3.2.1.
Is there any progress now？
ping [~hexiaoqiao] [~wanchang] .
was (Author: max2049):
ping [~hexiaoqiao] [~wanchang] .
> Multiple CloseOp shared block instance causes the standby namenode to crash
> when rolling editlog
>
>
> Key: HDFS-15175
> URL: https://issues.apache.org/jira/browse/HDFS-15175
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Yicong Cai
>Assignee: Wan Chang
>Priority: Critical
> Labels: NameNode
> Attachments: HDFS-15175-trunk.1.patch
>
>
>
> {panel:title=Crash exception}
> 2020-02-16 09:24:46,426 [507844305] - ERROR [Edit log
> tailer:FSEditLogLoader@245] - Encountered exception on operation CloseOp
> [length=0, inodeId=0, path=..., replication=3, mtime=1581816138774,
> atime=1581814760398, blockSize=536870912, blocks=[blk_5568434562_4495417845],
> permissions=da_music:hdfs:rw-r-, aclEntries=null, clientName=,
> clientMachine=, overwrite=false, storagePolicyId=0, opCode=OP_CLOSE,
> txid=32625024993]
> java.io.IOException: File is not under construction: ..
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:442)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:146)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:891)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:872)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:262)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:395)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:348)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:365)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:479)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:361)
> {panel}
>
> {panel:title=Editlog}
>
> OP_REASSIGN_LEASE
>
> 32625021150
> DFSClient_NONMAPREDUCE_-969060727_197760
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
>
>
> ..
>
> OP_CLOSE
>
> 32625023743
> 0
> 0
> ..
> 3
> 1581816135883
> 1581814760398
> 536870912
>
>
> false
>
> 5568434562
> 185818644
> 4495417845
>
>
> da_music
> hdfs
> 416
>
>
>
> ..
>
> OP_TRUNCATE
>
> 32625024049
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
> ..
> 185818644
> 1581816136336
>
> 5568434562
> 185818648
> 4495417845
>
>
>
> ..
>
> OP_CLOSE
>
> 32625024993
> 0
> 0
> ..
> 3
> 1581816138774
> 1581814760398
> 536870912
>
>
> false
>
> 5568434562
> 185818644
> 4495417845
>
>
> da_music
> hdfs
> 416
>
>
>
> {panel}
>
>
> The block size should be 185818648 in the first CloseOp. When truncate is
> used, the block size becomes 185818644. The CloseOp/TruncateOp/CloseOp is
> synchronized to the JournalNode in the same batch. The block used by CloseOp
> twice is the same instance, which causes the first CloseOp has wrong block
> size. When SNN rolling Editlog, TruncateOp does not make the file to the
> UnderConstruction state. Then, when the second CloseOp is executed, the file
> is not in the UnderConstruction state, and SNN crashes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15175) Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog
[
https://issues.apache.org/jira/browse/HDFS-15175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316997#comment-17316997
]
Max Xie commented on HDFS-15175:
-
ping [~hexiaoqiao] [~wanchang] .
> Multiple CloseOp shared block instance causes the standby namenode to crash
> when rolling editlog
>
>
> Key: HDFS-15175
> URL: https://issues.apache.org/jira/browse/HDFS-15175
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Yicong Cai
>Assignee: Wan Chang
>Priority: Critical
> Labels: NameNode
> Attachments: HDFS-15175-trunk.1.patch
>
>
>
> {panel:title=Crash exception}
> 2020-02-16 09:24:46,426 [507844305] - ERROR [Edit log
> tailer:FSEditLogLoader@245] - Encountered exception on operation CloseOp
> [length=0, inodeId=0, path=..., replication=3, mtime=1581816138774,
> atime=1581814760398, blockSize=536870912, blocks=[blk_5568434562_4495417845],
> permissions=da_music:hdfs:rw-r-, aclEntries=null, clientName=,
> clientMachine=, overwrite=false, storagePolicyId=0, opCode=OP_CLOSE,
> txid=32625024993]
> java.io.IOException: File is not under construction: ..
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:442)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:146)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:891)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:872)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:262)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:395)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:348)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:365)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:479)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:361)
> {panel}
>
> {panel:title=Editlog}
>
> OP_REASSIGN_LEASE
>
> 32625021150
> DFSClient_NONMAPREDUCE_-969060727_197760
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
>
>
> ..
>
> OP_CLOSE
>
> 32625023743
> 0
> 0
> ..
> 3
> 1581816135883
> 1581814760398
> 536870912
>
>
> false
>
> 5568434562
> 185818644
> 4495417845
>
>
> da_music
> hdfs
> 416
>
>
>
> ..
>
> OP_TRUNCATE
>
> 32625024049
> ..
> DFSClient_NONMAPREDUCE_1000868229_201260
> ..
> 185818644
> 1581816136336
>
> 5568434562
> 185818648
> 4495417845
>
>
>
> ..
>
> OP_CLOSE
>
> 32625024993
> 0
> 0
> ..
> 3
> 1581816138774
> 1581814760398
> 536870912
>
>
> false
>
> 5568434562
> 185818644
> 4495417845
>
>
> da_music
> hdfs
> 416
>
>
>
> {panel}
>
>
> The block size should be 185818648 in the first CloseOp. When truncate is
> used, the block size becomes 185818644. The CloseOp/TruncateOp/CloseOp is
> synchronized to the JournalNode in the same batch. The block used by CloseOp
> twice is the same instance, which causes the first CloseOp has wrong block
> size. When SNN rolling Editlog, TruncateOp does not make the file to the
> UnderConstruction state. Then, when the second CloseOp is executed, the file
> is not in the UnderConstruction state, and SNN crashes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15886) Add a way to get protected dirs from a special configuration file
[ https://issues.apache.org/jira/browse/HDFS-15886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie updated HDFS-15886: Description: We used protected dirs to ensure that important data directories cannot be deleted by mistake. But protected dirs can only be configured in hdfs-site.xml. For ease of management, we add a way to get the list of protected dirs from a special configuration file. How to use. 1. set the config in hdfs-site.xml ``` fs.protected.directories /hdfs/path/1,/hdfs/path/2,[file:///path/to/protected.dirs.config] ``` 2. add some protected dirs to the config file ([file:///path/to/protected.dirs.config]) ``` /hdfs/path/4 /hdfs/path/5 ``` 3. use command to refresh fs.protected.directories instead of FSDirectory.setProtectedDirectories(..) ``` hdfs dfsadmin -refreshProtectedDirectories ``` was: We used protected dirs to ensure that important data directories cannot be deleted by mistake. But protected dirs can only be configured in hdfs-site.xml. For ease of management, we add a way to get the list of protected dirs from a special configuration file. How to use. 1. set the config in hdfs-site.xml ``` fs.protected.directories /hdfs/path/1,/hdfs/path/2,[file:///path/to/protected.dirs.config] ``` 2. add some protected dirs to the config file ([file:///path/to/protected.dirs.config]) ``` /hdfs/path/4 /hdfs/path/5 ``` 3. use command to refresh fs.protected.directories instead of ``` hdfs dfsadmin -refreshProtectedDirectories ``` > Add a way to get protected dirs from a special configuration file > - > > Key: HDFS-15886 > URL: https://issues.apache.org/jira/browse/HDFS-15886 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > We used protected dirs to ensure that important data directories cannot be > deleted by mistake. But protected dirs can only be configured in > hdfs-site.xml. > For ease of management, we add a way to get the list of protected dirs from > a special configuration file. > How to use. > 1. set the config in hdfs-site.xml > ``` > > fs.protected.directories > > /hdfs/path/1,/hdfs/path/2,[file:///path/to/protected.dirs.config] > > ``` > 2. add some protected dirs to the config file > ([file:///path/to/protected.dirs.config]) > ``` > /hdfs/path/4 > /hdfs/path/5 > ``` > 3. use command to refresh fs.protected.directories instead of > FSDirectory.setProtectedDirectories(..) > ``` > hdfs dfsadmin -refreshProtectedDirectories > ``` > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15886) Add a way to get protected dirs from a special configuration file
[ https://issues.apache.org/jira/browse/HDFS-15886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300260#comment-17300260 ] Max Xie commented on HDFS-15886: - Thank you for tomscut and Stephen review. It's a great idea to use a mixture of paths and files to get protected dirs and I will try to code it. As for reload the config contents, maybe we can add a new protocol like refreshProtectedDirectories (similar to refreshNodes command) instead of reconfigging fs.protected.directories by calling Namenode.reconfProtectedDirectories. And I have updated the patch. Please take a look. > Add a way to get protected dirs from a special configuration file > - > > Key: HDFS-15886 > URL: https://issues.apache.org/jira/browse/HDFS-15886 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > We used protected dirs to ensure that important data directories cannot be > deleted by mistake. But protected dirs can only be configured in > hdfs-site.xml. > For ease of management, we add a way to get the list of protected dirs from > a special configuration file. > How to use. > 1. set the config in hdfs-site.xml > ``` > > fs.protected.directories > > /hdfs/path/1,/hdfs/path/2,[file:///path/to/protected.dirs.config] > > ``` > 2. add some protected dirs to the config file > ([file:///path/to/protected.dirs.config]) > ``` > /hdfs/path/4 > /hdfs/path/5 > ``` > 3. use command to refresh fs.protected.directories instead of > FSDirectory.setProtectedDirectories(..) > ``` > hdfs dfsadmin -refreshProtectedDirectories > ``` > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15886) Add a way to get protected dirs from a special configuration file
[ https://issues.apache.org/jira/browse/HDFS-15886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie updated HDFS-15886: Description: We used protected dirs to ensure that important data directories cannot be deleted by mistake. But protected dirs can only be configured in hdfs-site.xml. For ease of management, we add a way to get the list of protected dirs from a special configuration file. How to use. 1. set the config in hdfs-site.xml ``` fs.protected.directories /hdfs/path/1,/hdfs/path/2,[file:///path/to/protected.dirs.config] ``` 2. add some protected dirs to the config file ([file:///path/to/protected.dirs.config]) ``` /hdfs/path/4 /hdfs/path/5 ``` 3. use command to refresh fs.protected.directories instead of ``` hdfs dfsadmin -refreshProtectedDirectories ``` was: We used protected dirs to ensure that important data directories cannot be deleted by mistake. But protected dirs can only be configured in hdfs-site.xml. For ease of management, we add a way to get the list of protected dirs from a special configuration file. How to use. 1. set the config in hdfs-site.xml ``` dfs.protected.directories.config.file.enable true fs.protected.directories [file:///path/to/protected.dirs.config] ``` 2. add some protected dirs to the config file ([file:///path/to/protected.dirs.config]) ``` /1 /2/3 ``` 3. done > Add a way to get protected dirs from a special configuration file > - > > Key: HDFS-15886 > URL: https://issues.apache.org/jira/browse/HDFS-15886 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > We used protected dirs to ensure that important data directories cannot be > deleted by mistake. But protected dirs can only be configured in > hdfs-site.xml. > For ease of management, we add a way to get the list of protected dirs from > a special configuration file. > How to use. > 1. set the config in hdfs-site.xml > ``` > > fs.protected.directories > > /hdfs/path/1,/hdfs/path/2,[file:///path/to/protected.dirs.config] > > ``` > 2. add some protected dirs to the config file > ([file:///path/to/protected.dirs.config]) > ``` > /hdfs/path/4 > /hdfs/path/5 > ``` > 3. use command to refresh fs.protected.directories instead of > ``` > hdfs dfsadmin -refreshProtectedDirectories > ``` > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-15886) Add a way to get protected dirs from a special configuration file
[ https://issues.apache.org/jira/browse/HDFS-15886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-15886 started by Max Xie. --- > Add a way to get protected dirs from a special configuration file > - > > Key: HDFS-15886 > URL: https://issues.apache.org/jira/browse/HDFS-15886 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > We used protected dirs to ensure that important data directories cannot be > deleted by mistake. But protected dirs can only be configured in > hdfs-site.xml. > For ease of management, we add a way to get the list of protected dirs from > a special configuration file. > How to use. > 1. set the config in hdfs-site.xml > ``` > > dfs.protected.directories.config.file.enable > true > > > fs.protected.directories > [file:///path/to/protected.dirs.config] > > ``` > 2. add some protected dirs to the config file > ([file:///path/to/protected.dirs.config]) > ``` > /1 > /2/3 > ``` > 3. done -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15886) Add a way to get protected dirs from a special configuration file
Max Xie created HDFS-15886: --- Summary: Add a way to get protected dirs from a special configuration file Key: HDFS-15886 URL: https://issues.apache.org/jira/browse/HDFS-15886 Project: Hadoop HDFS Issue Type: New Feature Components: namenode Affects Versions: 3.4.0 Reporter: Max Xie Assignee: Max Xie We used protected dirs to ensure that important data directories cannot be deleted by mistake. But protected dirs can only be configured in hdfs-site.xml. For ease of management, we add a way to get the list of protected dirs from a special configuration file. How to use. 1. set the config in hdfs-site.xml ``` dfs.protected.directories.config.file.enable true fs.protected.directories file:///path/to/protected.dirs.config ``` 2. add some protected dirs to the config file (file:///path/to/protected.dirs.config) ``` # protect directories /1 /2/3 ``` 3. done -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15886) Add a way to get protected dirs from a special configuration file
[ https://issues.apache.org/jira/browse/HDFS-15886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie updated HDFS-15886: Description: We used protected dirs to ensure that important data directories cannot be deleted by mistake. But protected dirs can only be configured in hdfs-site.xml. For ease of management, we add a way to get the list of protected dirs from a special configuration file. How to use. 1. set the config in hdfs-site.xml ``` dfs.protected.directories.config.file.enable true fs.protected.directories [file:///path/to/protected.dirs.config] ``` 2. add some protected dirs to the config file ([file:///path/to/protected.dirs.config]) ``` /1 /2/3 ``` 3. done was: We used protected dirs to ensure that important data directories cannot be deleted by mistake. But protected dirs can only be configured in hdfs-site.xml. For ease of management, we add a way to get the list of protected dirs from a special configuration file. How to use. 1. set the config in hdfs-site.xml ``` dfs.protected.directories.config.file.enable true fs.protected.directories file:///path/to/protected.dirs.config ``` 2. add some protected dirs to the config file (file:///path/to/protected.dirs.config) ``` # protect directories /1 /2/3 ``` 3. done > Add a way to get protected dirs from a special configuration file > - > > Key: HDFS-15886 > URL: https://issues.apache.org/jira/browse/HDFS-15886 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > > We used protected dirs to ensure that important data directories cannot be > deleted by mistake. But protected dirs can only be configured in > hdfs-site.xml. > For ease of management, we add a way to get the list of protected dirs from > a special configuration file. > How to use. > 1. set the config in hdfs-site.xml > ``` > > dfs.protected.directories.config.file.enable > true > > > fs.protected.directories > [file:///path/to/protected.dirs.config] > > ``` > 2. add some protected dirs to the config file > ([file:///path/to/protected.dirs.config]) > ``` > /1 > /2/3 > ``` > 3. done -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15720) namenode audit async logger should add some log4j config
Max Xie created HDFS-15720: --- Summary: namenode audit async logger should add some log4j config Key: HDFS-15720 URL: https://issues.apache.org/jira/browse/HDFS-15720 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Affects Versions: 3.3.0 Environment: hadoop 3.3.0 Reporter: Max Xie Hadoop project use log4j 1.2.x, we can't config some properties of logger in log4j.properties file , For example, AsyncAppender BufferSize and Blocking see https://logging.apache.org/log4j/1.2/apidocs/index.html. Namenode should add some audit async logger log4j config In order to facilitate the adjustment of log4j usage and audit log output performance adjustment. The new configuration is as follows dfs.namenode.audit.log.async.blocking false dfs.namenode.audit.log.async.buffer.size 128 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13183) Standby NameNode process getBlocks request to reduce Active load
[
https://issues.apache.org/jira/browse/HDFS-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005233#comment-17005233
]
Max Xie commented on HDFS-13183:
-
(y)
> Standby NameNode process getBlocks request to reduce Active load
>
>
> Key: HDFS-13183
> URL: https://issues.apache.org/jira/browse/HDFS-13183
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer mover, namenode
>Affects Versions: 2.7.5, 3.1.0, 2.9.1, 2.8.4, 3.0.2
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-13183-trunk.001.patch, HDFS-13183-trunk.002.patch,
> HDFS-13183-trunk.003.patch
>
>
> The performance of Active NameNode could be impact when {{Balancer}} requests
> #getBlocks, since query blocks of overly full DNs performance is extremely
> inefficient currently. The main reason is {{NameNodeRpcServer#getBlocks}}
> hold read lock for long time. In extreme case, all handlers of Active
> NameNode RPC server are occupied by one reader
> {{NameNodeRpcServer#getBlocks}} and other write operation calls, thus Active
> NameNode enter a state of false death for number of seconds even for minutes.
> The similar performance concerns of Balancer have reported by HDFS-9412,
> HDFS-7967, etc.
> If Standby NameNode can shoulder #getBlocks heavy burden, it could speed up
> the progress of balancing and reduce performance impact to Active NameNode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
