[jira] [Commented] (HDFS-16726) There is a memory-related problem about HDFS namenode
[ https://issues.apache.org/jira/browse/HDFS-16726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17797090#comment-17797090 ] Yanlei Yu commented on HDFS-16726: -- [~zinx] hi,You mean upgrade jdk to 12 and remove G1RSetRegionEntries param. After that, you can solve the problem? > There is a memory-related problem about HDFS namenode > - > > Key: HDFS-16726 > URL: https://issues.apache.org/jira/browse/HDFS-16726 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs, namenode >Affects Versions: 2.7.2 > Environment: -Xms280G -Xmx280G -XX:MaxDirectMemorySize=10G > -XX:MetaspaceSize=128M -server \ > -XX:+UseG1GC -XX:+UseStringDeduplication > -XX:MaxGCPauseMillis=250 -XX:+UnlockExperimentalVMOptions > -XX:+PrintGCApplicationStoppedTime -XX:+PrintSafepointStatistics > -XX:PrintSafepointStatisticsCount=1 \ > -XX:G1OldCSetRegionThresholdPercent=1 > -XX:G1MixedGCCountTarget=9 -XX:+SafepointTimeout > -XX:SafepointTimeoutDelay=4000 \ > -XX:ParallelGCThreads=24 -XX:ConcGCThreads=6 > -XX:G1RSetRegionEntries=4096 -XX:+AggressiveOpts -XX:+DisableExplicitGC \ > -XX:G1HeapWastePercent=9 > -XX:G1MixedGCLiveThresholdPercent=85 -XX:InitiatingHeapOccupancyPercent=75 \ > -XX:+ParallelRefProcEnabled -XX:-ResizePLAB > -XX:+PrintAdaptiveSizePolicy \ > -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps \ > -Xloggc:$HADOOP_LOG_DIR/namenode.gc.log \ > -XX:+HeapDumpOnOutOfMemoryError > -XX:ErrorFile=$HADOOP_LOG_DIR/hs_err_pid%p.log > -XX:HeapDumpPath=$HADOOP_LOG_DIR \ > -Dcom.sun.management.jmxremote \ > -Dcom.sun.management.jmxremote.port=9009 \ > -Dcom.sun.management.jmxremote.ssl=false \ > -Dcom.sun.management.jmxremote.authenticate=false \ > $HADOOP_NAMENODE_OPTS >Reporter: Yanlei Yu >Priority: Critical > > In the cluster, the memory usage of Namenode exceeds the XMX setting (XMX > =280GB). The actual memory usage of Namenode is 479GB > Output via pamp: > Address Perm Offset Device Inode Size Rss Pss > Referenced Anonymous Swap Locked Mapping > 2b42f000 rw-p 00:00 0 294174720 293756960 293756960 > 293756960 293756960 0 0 > 01e21000 rw-p 00:00 0 195245456 195240848 195240848 > 195240848 195240848 0 0 [heap] > 2b897c00 rw-p 00:00 0 9246724 9246724 9246724 > 9246724 9246724 0 0 > 2b8bb0905000 rw-p 00:00 0 1781124 1754572 1754572 > 1754572 1754572 0 0 > 2b893600 rw-p 00:00 0 1146880 1002084 1002084 > 1002084 1002084 0 0 > 2b42db652000 rwxp 00:00 0 57792 55252 55252 > 55252 55252 0 0 > 2b42ec12a000 rw-p 00:00 0 25696 24700 24700 > 24700 24700 0 0 > 2b42ef25b000 rw-p 00:00 0 9988 8972 8972 > 8972 8972 0 0 > 2b8c1d467000 rw-p 00:00 0 9216 8204 8204 > 8204 8204 0 0 > 2b8d6f8db000 rw-p 00:00 0 7160 6228 6228 > 6228 6228 0 0 > The first line should configure the memory footprint for XMX, and [heap] is > unusually large, so a memory leak is suspected! > > * [heap] is associated with malloc > After configuring JCMD in the test environment, we found that the malloc part > of Internal in JCMD increased significantly when the client was writing to a > gz file (XMX =40g in the test environment, and the Internal area was 900MB > before the client wrote) : > Total: reserved=47276MB, committed=47070MB > - Java Heap (reserved=40960MB, committed=40960MB) > (mmap: reserved=40960MB, committed=40960MB) > > - Class (reserved=53MB, committed=52MB) > (classes #7423) > (malloc=1MB #17053) > (mmap: reserved=52MB, committed=52MB) > > - Thread (reserved=2145MB, committed=2145MB) > (thread #2129) > (stack: reserved=2136MB, committed=2136MB) > (malloc=7MB #10673) > (arena=2MB #4256) > > - Code (reserved=251MB, committed=45MB) > (malloc=7MB #10661) > (mmap: reserved=244MB, committed=38MB) > > -
[jira] [Commented] (HDFS-17093) In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block reporting
[
https://issues.apache.org/jira/browse/HDFS-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744861#comment-17744861
]
Yanlei Yu commented on HDFS-17093:
--
{quote}Please push update to your same original branch (for this case which is
Tre2878:HDFS-17093), DO NOT pull repeat request for same issue.
{quote}
Sorry, I now made changes and commits in the original branch
> In the case of all datanodes sending FBR when the namenode restarts (large
> clusters), there is an issue with incomplete block reporting
> ---
>
> Key: HDFS-17093
> URL: https://issues.apache.org/jira/browse/HDFS-17093
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: Yanlei Yu
>Priority: Minor
> Labels: pull-request-available
>
> In our cluster of 800+ nodes, after restarting the namenode, we found that
> some datanodes did not report enough blocks, causing the namenode to stay in
> secure mode for a long time after restarting because of incomplete block
> reporting
> I found in the logs of the datanode with incomplete block reporting that the
> first FBR attempt failed, possibly due to namenode stress, and then a second
> FBR attempt was made as follows:
> {code:java}
>
> 2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
> report(s), of which we sent 1. The reports had 1099057 total blocks and used
> 1 RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
> processing. Got back no commands.
> 2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Successfully sent block report 0x62382416f3f055, containing 12 storage
> report(s), of which we sent 12. The reports had 1099048 total blocks and used
> 12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
> processing. Got back no commands. {code}
> There's nothing wrong with that. Retry the send if it fails But on the
> namenode side of the logic:
> {code:java}
> if (namesystem.isInStartupSafeMode()
> && !StorageType.PROVIDED.equals(storageInfo.getStorageType())
> && storageInfo.getBlockReportCount() > 0) {
> blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
> + "discarded non-initial block report from {}"
> + " because namenode still in startup phase",
> strBlockReportId, fullBrLeaseId, nodeID);
> blockReportLeaseManager.removeLease(node);
> return !node.hasStaleStorages();
> } {code}
> When a disk was identified as the report is not the first time, namely
> storageInfo. GetBlockReportCount > 0, Will remove the ticket from the
> datanode, lead to a second report failed because no lease
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17093) In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block reporting
[
https://issues.apache.org/jira/browse/HDFS-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yanlei Yu updated HDFS-17093:
-
Attachment: (was: HDFS-17093.patch)
> In the case of all datanodes sending FBR when the namenode restarts (large
> clusters), there is an issue with incomplete block reporting
> ---
>
> Key: HDFS-17093
> URL: https://issues.apache.org/jira/browse/HDFS-17093
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: Yanlei Yu
>Priority: Minor
> Labels: pull-request-available
>
> In our cluster of 800+ nodes, after restarting the namenode, we found that
> some datanodes did not report enough blocks, causing the namenode to stay in
> secure mode for a long time after restarting because of incomplete block
> reporting
> I found in the logs of the datanode with incomplete block reporting that the
> first FBR attempt failed, possibly due to namenode stress, and then a second
> FBR attempt was made as follows:
> {code:java}
>
> 2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
> report(s), of which we sent 1. The reports had 1099057 total blocks and used
> 1 RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
> processing. Got back no commands.
> 2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Successfully sent block report 0x62382416f3f055, containing 12 storage
> report(s), of which we sent 12. The reports had 1099048 total blocks and used
> 12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
> processing. Got back no commands. {code}
> There's nothing wrong with that. Retry the send if it fails But on the
> namenode side of the logic:
> {code:java}
> if (namesystem.isInStartupSafeMode()
> && !StorageType.PROVIDED.equals(storageInfo.getStorageType())
> && storageInfo.getBlockReportCount() > 0) {
> blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
> + "discarded non-initial block report from {}"
> + " because namenode still in startup phase",
> strBlockReportId, fullBrLeaseId, nodeID);
> blockReportLeaseManager.removeLease(node);
> return !node.hasStaleStorages();
> } {code}
> When a disk was identified as the report is not the first time, namely
> storageInfo. GetBlockReportCount > 0, Will remove the ticket from the
> datanode, lead to a second report failed because no lease
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17093) In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block reporting
[
https://issues.apache.org/jira/browse/HDFS-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744602#comment-17744602
]
Yanlei Yu commented on HDFS-17093:
--
[~hexiaoqiao] I have modified and resubmitted PB[：GitHub Pull Request
#5856|https://github.com/apache/hadoop/pull/5856]
> In the case of all datanodes sending FBR when the namenode restarts (large
> clusters), there is an issue with incomplete block reporting
> ---
>
> Key: HDFS-17093
> URL: https://issues.apache.org/jira/browse/HDFS-17093
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: Yanlei Yu
>Priority: Minor
> Labels: pull-request-available
> Attachments: HDFS-17093.patch
>
>
> In our cluster of 800+ nodes, after restarting the namenode, we found that
> some datanodes did not report enough blocks, causing the namenode to stay in
> secure mode for a long time after restarting because of incomplete block
> reporting
> I found in the logs of the datanode with incomplete block reporting that the
> first FBR attempt failed, possibly due to namenode stress, and then a second
> FBR attempt was made as follows:
> {code:java}
>
> 2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
> report(s), of which we sent 1. The reports had 1099057 total blocks and used
> 1 RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
> processing. Got back no commands.
> 2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Successfully sent block report 0x62382416f3f055, containing 12 storage
> report(s), of which we sent 12. The reports had 1099048 total blocks and used
> 12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
> processing. Got back no commands. {code}
> There's nothing wrong with that. Retry the send if it fails But on the
> namenode side of the logic:
> {code:java}
> if (namesystem.isInStartupSafeMode()
> && !StorageType.PROVIDED.equals(storageInfo.getStorageType())
> && storageInfo.getBlockReportCount() > 0) {
> blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
> + "discarded non-initial block report from {}"
> + " because namenode still in startup phase",
> strBlockReportId, fullBrLeaseId, nodeID);
> blockReportLeaseManager.removeLease(node);
> return !node.hasStaleStorages();
> } {code}
> When a disk was identified as the report is not the first time, namely
> storageInfo. GetBlockReportCount > 0, Will remove the ticket from the
> datanode, lead to a second report failed because no lease
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-17093) In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block reporting
[
https://issues.apache.org/jira/browse/HDFS-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744524#comment-17744524
]
Yanlei Yu edited comment on HDFS-17093 at 7/19/23 10:24 AM:
[~hexiaoqiao] ，
{quote}would you mind to submit PR via Github if need?
{quote}
PR：[GitHub Pull Request #5855|https://github.com/apache/hadoop/pull/5855]
was (Author: JIRAUSER294151):
[~hexiaoqiao] ，
{quote}would you mind to submit PR via Github if need?
{quote}
PR：[[GitHub Pull Request
#5855|https://github.com/apache/hadoop/pull/5855]|[http://example.com|https://github.com/apache/hadoop/pull/5855]]
> In the case of all datanodes sending FBR when the namenode restarts (large
> clusters), there is an issue with incomplete block reporting
> ---
>
> Key: HDFS-17093
> URL: https://issues.apache.org/jira/browse/HDFS-17093
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: Yanlei Yu
>Priority: Minor
> Attachments: HDFS-17093.patch
>
>
> In our cluster of 800+ nodes, after restarting the namenode, we found that
> some datanodes did not report enough blocks, causing the namenode to stay in
> secure mode for a long time after restarting because of incomplete block
> reporting
> I found in the logs of the datanode with incomplete block reporting that the
> first FBR attempt failed, possibly due to namenode stress, and then a second
> FBR attempt was made as follows:
> {code:java}
>
> 2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
> report(s), of which we sent 1. The reports had 1099057 total blocks and used
> 1 RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
> processing. Got back no commands.
> 2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Successfully sent block report 0x62382416f3f055, containing 12 storage
> report(s), of which we sent 12. The reports had 1099048 total blocks and used
> 12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
> processing. Got back no commands. {code}
> There's nothing wrong with that. Retry the send if it fails But on the
> namenode side of the logic:
> {code:java}
> if (namesystem.isInStartupSafeMode()
> && !StorageType.PROVIDED.equals(storageInfo.getStorageType())
> && storageInfo.getBlockReportCount() > 0) {
> blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
> + "discarded non-initial block report from {}"
> + " because namenode still in startup phase",
> strBlockReportId, fullBrLeaseId, nodeID);
> blockReportLeaseManager.removeLease(node);
> return !node.hasStaleStorages();
> } {code}
> When a disk was identified as the report is not the first time, namely
> storageInfo. GetBlockReportCount > 0, Will remove the ticket from the
> datanode, lead to a second report failed because no lease
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17093) In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block reporting
[
https://issues.apache.org/jira/browse/HDFS-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744524#comment-17744524
]
Yanlei Yu commented on HDFS-17093:
--
[~hexiaoqiao] ，
{quote}would you mind to submit PR via Github if need?
{quote}
PR：[[GitHub Pull Request
#5855|https://github.com/apache/hadoop/pull/5855]|[http://example.com|https://github.com/apache/hadoop/pull/5855]]
> In the case of all datanodes sending FBR when the namenode restarts (large
> clusters), there is an issue with incomplete block reporting
> ---
>
> Key: HDFS-17093
> URL: https://issues.apache.org/jira/browse/HDFS-17093
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: Yanlei Yu
>Priority: Minor
> Attachments: HDFS-17093.patch
>
>
> In our cluster of 800+ nodes, after restarting the namenode, we found that
> some datanodes did not report enough blocks, causing the namenode to stay in
> secure mode for a long time after restarting because of incomplete block
> reporting
> I found in the logs of the datanode with incomplete block reporting that the
> first FBR attempt failed, possibly due to namenode stress, and then a second
> FBR attempt was made as follows:
> {code:java}
>
> 2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
> report(s), of which we sent 1. The reports had 1099057 total blocks and used
> 1 RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
> processing. Got back no commands.
> 2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Successfully sent block report 0x62382416f3f055, containing 12 storage
> report(s), of which we sent 12. The reports had 1099048 total blocks and used
> 12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
> processing. Got back no commands. {code}
> There's nothing wrong with that. Retry the send if it fails But on the
> namenode side of the logic:
> {code:java}
> if (namesystem.isInStartupSafeMode()
> && !StorageType.PROVIDED.equals(storageInfo.getStorageType())
> && storageInfo.getBlockReportCount() > 0) {
> blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
> + "discarded non-initial block report from {}"
> + " because namenode still in startup phase",
> strBlockReportId, fullBrLeaseId, nodeID);
> blockReportLeaseManager.removeLease(node);
> return !node.hasStaleStorages();
> } {code}
> When a disk was identified as the report is not the first time, namely
> storageInfo. GetBlockReportCount > 0, Will remove the ticket from the
> datanode, lead to a second report failed because no lease
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-17093) In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block reporting
[
https://issues.apache.org/jira/browse/HDFS-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1778#comment-1778
]
Yanlei Yu edited comment on HDFS-17093 at 7/19/23 6:08 AM:
---
Just to add to that,Our cluster configuration is
dfs.namenode.max.full.block.report.leases=1500(we have 800+ nodes)，When the
namenode restarts, all 800+ nodes will send FBRS,This happens when the namenode
is under a lot of pressure，Of course will not rule out
dfs.namenode.max.full.block.report.leases Set to a smaller value，not sure it
will happen
was (Author: JIRAUSER294151):
Just to add to that,Our cluster configuration is
dfs.namenode.max.full.block.report.leases=1500(we have 800+ nodes)，When the
namenode restarts, all 800+ nodes will send FBRS,This happens when the namenode
is under a lot of pressure，Of course will not rule out
dfs.namenode.max.full.block.report.leases Set to a smaller value，will not happen
> In the case of all datanodes sending FBR when the namenode restarts (large
> clusters), there is an issue with incomplete block reporting
> ---
>
> Key: HDFS-17093
> URL: https://issues.apache.org/jira/browse/HDFS-17093
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: Yanlei Yu
>Priority: Minor
> Attachments: HDFS-17093.patch
>
>
> In our cluster of 800+ nodes, after restarting the namenode, we found that
> some datanodes did not report enough blocks, causing the namenode to stay in
> secure mode for a long time after restarting because of incomplete block
> reporting
> I found in the logs of the datanode with incomplete block reporting that the
> first FBR attempt failed, possibly due to namenode stress, and then a second
> FBR attempt was made as follows:
> {code:java}
>
> 2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
> report(s), of which we sent 1. The reports had 1099057 total blocks and used
> 1 RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
> processing. Got back no commands.
> 2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Successfully sent block report 0x62382416f3f055, containing 12 storage
> report(s), of which we sent 12. The reports had 1099048 total blocks and used
> 12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
> processing. Got back no commands. {code}
> There's nothing wrong with that. Retry the send if it fails But on the
> namenode side of the logic:
> {code:java}
> if (namesystem.isInStartupSafeMode()
> && !StorageType.PROVIDED.equals(storageInfo.getStorageType())
> && storageInfo.getBlockReportCount() > 0) {
> blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
> + "discarded non-initial block report from {}"
> + " because namenode still in startup phase",
> strBlockReportId, fullBrLeaseId, nodeID);
> blockReportLeaseManager.removeLease(node);
> return !node.hasStaleStorages();
> } {code}
> When a disk was identified as the report is not the first time, namely
> storageInfo. GetBlockReportCount > 0, Will remove the ticket from the
> datanode, lead to a second report failed because no lease
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17093) In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block reporting
[
https://issues.apache.org/jira/browse/HDFS-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1778#comment-1778
]
Yanlei Yu commented on HDFS-17093:
--
Just to add to that,Our cluster configuration is
dfs.namenode.max.full.block.report.leases=1500(we have 800+ nodes)，When the
namenode restarts, all 800+ nodes will send FBRS,This happens when the namenode
is under a lot of pressure，Of course will not rule out
dfs.namenode.max.full.block.report.leases Set to a smaller value，will not happen
> In the case of all datanodes sending FBR when the namenode restarts (large
> clusters), there is an issue with incomplete block reporting
> ---
>
> Key: HDFS-17093
> URL: https://issues.apache.org/jira/browse/HDFS-17093
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: Yanlei Yu
>Priority: Minor
> Attachments: HDFS-17093.patch
>
>
> In our cluster of 800+ nodes, after restarting the namenode, we found that
> some datanodes did not report enough blocks, causing the namenode to stay in
> secure mode for a long time after restarting because of incomplete block
> reporting
> I found in the logs of the datanode with incomplete block reporting that the
> first FBR attempt failed, possibly due to namenode stress, and then a second
> FBR attempt was made as follows:
> {code:java}
>
> 2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
> report(s), of which we sent 1. The reports had 1099057 total blocks and used
> 1 RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
> processing. Got back no commands.
> 2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Successfully sent block report 0x62382416f3f055, containing 12 storage
> report(s), of which we sent 12. The reports had 1099048 total blocks and used
> 12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
> processing. Got back no commands. {code}
> There's nothing wrong with that. Retry the send if it fails But on the
> namenode side of the logic:
> {code:java}
> if (namesystem.isInStartupSafeMode()
> && !StorageType.PROVIDED.equals(storageInfo.getStorageType())
> && storageInfo.getBlockReportCount() > 0) {
> blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
> + "discarded non-initial block report from {}"
> + " because namenode still in startup phase",
> strBlockReportId, fullBrLeaseId, nodeID);
> blockReportLeaseManager.removeLease(node);
> return !node.hasStaleStorages();
> } {code}
> When a disk was identified as the report is not the first time, namely
> storageInfo. GetBlockReportCount > 0, Will remove the ticket from the
> datanode, lead to a second report failed because no lease
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17093) In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block reporting
[
https://issues.apache.org/jira/browse/HDFS-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1775#comment-1775
]
Yanlei Yu commented on HDFS-17093:
--
[~xinglin] ，I think you modify some more reasonable, datanode separate disk
operation should be processed in the final set to perform
{quote}blockReportLeaseManager. RemoveLease (node);
return ! node.hasStaleStorages();
{quote}
This is all at the datanode level
> In the case of all datanodes sending FBR when the namenode restarts (large
> clusters), there is an issue with incomplete block reporting
> ---
>
> Key: HDFS-17093
> URL: https://issues.apache.org/jira/browse/HDFS-17093
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: Yanlei Yu
>Priority: Minor
> Attachments: HDFS-17093.patch
>
>
> In our cluster of 800+ nodes, after restarting the namenode, we found that
> some datanodes did not report enough blocks, causing the namenode to stay in
> secure mode for a long time after restarting because of incomplete block
> reporting
> I found in the logs of the datanode with incomplete block reporting that the
> first FBR attempt failed, possibly due to namenode stress, and then a second
> FBR attempt was made as follows:
> {code:java}
>
> 2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
> report(s), of which we sent 1. The reports had 1099057 total blocks and used
> 1 RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
> processing. Got back no commands.
> 2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Successfully sent block report 0x62382416f3f055, containing 12 storage
> report(s), of which we sent 12. The reports had 1099048 total blocks and used
> 12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
> processing. Got back no commands. {code}
> There's nothing wrong with that. Retry the send if it fails But on the
> namenode side of the logic:
> {code:java}
> if (namesystem.isInStartupSafeMode()
> && !StorageType.PROVIDED.equals(storageInfo.getStorageType())
> && storageInfo.getBlockReportCount() > 0) {
> blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
> + "discarded non-initial block report from {}"
> + " because namenode still in startup phase",
> strBlockReportId, fullBrLeaseId, nodeID);
> blockReportLeaseManager.removeLease(node);
> return !node.hasStaleStorages();
> } {code}
> When a disk was identified as the report is not the first time, namely
> storageInfo. GetBlockReportCount > 0, Will remove the ticket from the
> datanode, lead to a second report failed because no lease
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17093) In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block reporting
[
https://issues.apache.org/jira/browse/HDFS-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1774#comment-1774
]
Yanlei Yu commented on HDFS-17093:
--
{quote} Would you try to dig the log information at NameNode side when the
second FBR from DataNode?
{quote}
[~hexiaoqiao] The namenode log looks like this:
{code:java}
DEBUG blockmanagement.BlockReportLeaseManager: Created a new BR lease
0x954310c48050d79f for DN b8a8f403-4e3e-4a8a-ac51-bb512c83186b. numPending = 1
TRACE blockmanagement.BlockReportLeaseManager: BR lease 0x954310c48050d79f is
valid for DN b8a8f403-4e3e-4a8a-ac51-bb512c83186b.
INFO BlockStateChange: BLOCK* processReport 0x13dffe1dc2f6199 with lease ID
0x954310c48050d79f: discarded non-initial block report from
DatanodeRegistration(*.*.*.*:50010,
datanodeUuid=b8a8f403-4e3e-4a8a-ac51-bb512c83186b, infoPort=50075,
infoSecurePort=0, ipcPort=50020,
storageInfo=lv=-56;cid=CID-c9a83fd3-b70f-498a-be5e-f2d5a34b5aee;nsid=902715697;c=0)
because namenode still in startup phase
TRACE blockmanagement.BlockReportLeaseManager: Removed BR lease
0x954310c48050d79f for DN b8a8f403-4e3e-4a8a-ac51-bb512c83186b. numPending = 0
WARN blockmanagement.BlockReportLeaseManager: BR lease 0x954310c48050d79f is
not valid for DN b8a8f403-4e3e-4a8a-ac51-bb512c83186b, because the DN is not in
the pending set.
WARN blockmanagement.BlockReportLeaseManager: BR lease 0x954310c48050d79f is
not valid for DN b8a8f403-4e3e-4a8a-ac51-bb512c83186b, because the DN is not in
the pending set.
WARN blockmanagement.BlockReportLeaseManager: BR lease 0x954310c48050d79f is
not valid for DN b8a8f403-4e3e-4a8a-ac51-bb512c83186b, because the DN is not in
the pending set.
WARN blockmanagement.BlockReportLeaseManager: BR lease 0x954310c48050d79f is
not valid for DN b8a8f403-4e3e-4a8a-ac51-bb512c83186b, because the DN is not in
the pending set.
WARN blockmanagement.BlockReportLeaseManager: BR lease 0x954310c48050d79f is
not valid for DN b8a8f403-4e3e-4a8a-ac51-bb512c83186b, because the DN is not in
the pending set.
WARN blockmanagement.BlockReportLeaseManager: BR lease 0x954310c48050d79f is
not valid for DN b8a8f403-4e3e-4a8a-ac51-bb512c83186b, because the DN is not in
the pending set.
WARN blockmanagement.BlockReportLeaseManager: BR lease 0x954310c48050d79f is
not valid for DN b8a8f403-4e3e-4a8a-ac51-bb512c83186b, because the DN is not in
the pending set.
WARN blockmanagement.BlockReportLeaseManager: BR lease 0x954310c48050d79f is
not valid for DN b8a8f403-4e3e-4a8a-ac51-bb512c83186b, because the DN is not in
the pending set.
WARN blockmanagement.BlockReportLeaseManager: BR lease 0x954310c48050d79f is
not valid for DN b8a8f403-4e3e-4a8a-ac51-bb512c83186b, because the DN is not in
the pending set.
WARN blockmanagement.BlockReportLeaseManager: BR lease 0x954310c48050d79f is
not valid for DN b8a8f403-4e3e-4a8a-ac51-bb512c83186b, because the DN is not in
the pending set.
WARN blockmanagement.BlockReportLeaseManager: BR lease 0x954310c48050d79f is
not valid for DN b8a8f403-4e3e-4a8a-ac51-bb512c83186b, because the DN is not in
the pending set. {code}
{quote}would you mind to submit PR via Github if need?
{quote}
I tried the github commit, but it seems that I do not have the permission. This
is the first time for me to commit a patch, and I am not sure how to operate it
> In the case of all datanodes sending FBR when the namenode restarts (large
> clusters), there is an issue with incomplete block reporting
> ---
>
> Key: HDFS-17093
> URL: https://issues.apache.org/jira/browse/HDFS-17093
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: Yanlei Yu
>Priority: Minor
> Attachments: HDFS-17093.patch
>
>
> In our cluster of 800+ nodes, after restarting the namenode, we found that
> some datanodes did not report enough blocks, causing the namenode to stay in
> secure mode for a long time after restarting because of incomplete block
> reporting
> I found in the logs of the datanode with incomplete block reporting that the
> first FBR attempt failed, possibly due to namenode stress, and then a second
> FBR attempt was made as follows:
> {code:java}
>
> 2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
> report(s), of which we sent 1. The reports had 1099057 total blocks and used
> 1 RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
> processing. Got back no commands.
> 2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Successfully sent block report 0x62382416f3f055, containing 12 storage
> report(s),
[jira] [Updated] (HDFS-17093) In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block reporting
[
https://issues.apache.org/jira/browse/HDFS-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yanlei Yu updated HDFS-17093:
-
Issue Type: Bug (was: Improvement)
> In the case of all datanodes sending FBR when the namenode restarts (large
> clusters), there is an issue with incomplete block reporting
> ---
>
> Key: HDFS-17093
> URL: https://issues.apache.org/jira/browse/HDFS-17093
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: Yanlei Yu
>Priority: Minor
> Attachments: HDFS-17093.patch
>
>
> In our cluster of 800+ nodes, after restarting the namenode, we found that
> some datanodes did not report enough blocks, causing the namenode to stay in
> secure mode for a long time after restarting because of incomplete block
> reporting
> I found in the logs of the datanode with incomplete block reporting that the
> first FBR attempt failed, possibly due to namenode stress, and then a second
> FBR attempt was made as follows:
> {code:java}
>
> 2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
> report(s), of which we sent 1. The reports had 1099057 total blocks and used
> 1 RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
> processing. Got back no commands.
> 2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Successfully sent block report 0x62382416f3f055, containing 12 storage
> report(s), of which we sent 12. The reports had 1099048 total blocks and used
> 12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
> processing. Got back no commands. {code}
> There's nothing wrong with that. Retry the send if it fails But on the
> namenode side of the logic:
> {code:java}
> if (namesystem.isInStartupSafeMode()
> && !StorageType.PROVIDED.equals(storageInfo.getStorageType())
> && storageInfo.getBlockReportCount() > 0) {
> blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
> + "discarded non-initial block report from {}"
> + " because namenode still in startup phase",
> strBlockReportId, fullBrLeaseId, nodeID);
> blockReportLeaseManager.removeLease(node);
> return !node.hasStaleStorages();
> } {code}
> When a disk was identified as the report is not the first time, namely
> storageInfo. GetBlockReportCount > 0, Will remove the ticket from the
> datanode, lead to a second report failed because no lease
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15901) Solve the problem of DN repeated block reports occupying too many RPCs during Safemode
[ https://issues.apache.org/jira/browse/HDFS-15901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744402#comment-17744402 ] Yanlei Yu commented on HDFS-15901: -- I submitted a patch to [HDFS-18093|https://issues.apache.org/jira/browse/HDFS-17093] that resolves this issue, which you can take a look at > Solve the problem of DN repeated block reports occupying too many RPCs during > Safemode > -- > > Key: HDFS-15901 > URL: https://issues.apache.org/jira/browse/HDFS-15901 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > When the cluster exceeds thousands of nodes, we want to restart the NameNode > service, and all DataNodes send a full Block action to the NameNode. During > SafeMode, some DataNodes may send blocks to NameNode multiple times, which > will take up too much RPC. In fact, this is unnecessary. > In this case, some block report leases will fail or time out, and in extreme > cases, the NameNode will always stay in Safe Mode. > 2021-03-14 08:16:25,873 [78438700] - INFO [Block report > processor:BlockManager@2158] - BLOCK* processReport 0xe: discarded > non-initial block report from DatanodeRegistration(:port, > datanodeUuid=, infoPort=, infoSecurePort=, > ipcPort=, storageInfo=lv=;nsid=;c=0) because namenode > still in startup phase > 2021-03-14 08:16:31,521 [78444348] - INFO [Block report > processor:BlockManager@2158] - BLOCK* processReport 0xe: discarded > non-initial block report from DatanodeRegistration(, > datanodeUuid=, infoPort=, infoSecurePort=, > ipcPort=, storageInfo=lv=;nsid=;c=0) because namenode > still in startup phase > 2021-03-13 18:35:38,200 [29191027] - WARN [Block report > processor:BlockReportLeaseManager@311] - BR lease 0x is not valid for > DN , because the DN is not in the pending set. > 2021-03-13 18:36:08,143 [29220970] - WARN [Block report > processor:BlockReportLeaseManager@311] - BR lease 0x is not valid for > DN , because the DN is not in the pending set. > 2021-03-13 18:36:08,143 [29220970] - WARN [Block report > processor:BlockReportLeaseManager@317] - BR lease 0x is not valid for > DN , because the lease has expired. > 2021-03-13 18:36:08,145 [29220972] - WARN [Block report > processor:BlockReportLeaseManager@317] - BR lease 0x is not valid for > DN , because the lease has expired. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17093) In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block reporting
[
https://issues.apache.org/jira/browse/HDFS-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744401#comment-17744401
]
Yanlei Yu commented on HDFS-17093:
--
The problem is similar to HDFS-15901
> In the case of all datanodes sending FBR when the namenode restarts (large
> clusters), there is an issue with incomplete block reporting
> ---
>
> Key: HDFS-17093
> URL: https://issues.apache.org/jira/browse/HDFS-17093
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: Yanlei Yu
>Priority: Minor
> Attachments: HDFS-17093.patch
>
>
> In our cluster of 800+ nodes, after restarting the namenode, we found that
> some datanodes did not report enough blocks, causing the namenode to stay in
> secure mode for a long time after restarting because of incomplete block
> reporting
> I found in the logs of the datanode with incomplete block reporting that the
> first FBR attempt failed, possibly due to namenode stress, and then a second
> FBR attempt was made as follows:
> {code:java}
>
> 2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
> report(s), of which we sent 1. The reports had 1099057 total blocks and used
> 1 RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
> processing. Got back no commands.
> 2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Successfully sent block report 0x62382416f3f055, containing 12 storage
> report(s), of which we sent 12. The reports had 1099048 total blocks and used
> 12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
> processing. Got back no commands. {code}
> There's nothing wrong with that. Retry the send if it fails But on the
> namenode side of the logic:
> {code:java}
> if (namesystem.isInStartupSafeMode()
> && !StorageType.PROVIDED.equals(storageInfo.getStorageType())
> && storageInfo.getBlockReportCount() > 0) {
> blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
> + "discarded non-initial block report from {}"
> + " because namenode still in startup phase",
> strBlockReportId, fullBrLeaseId, nodeID);
> blockReportLeaseManager.removeLease(node);
> return !node.hasStaleStorages();
> } {code}
> When a disk was identified as the report is not the first time, namely
> storageInfo. GetBlockReportCount > 0, Will remove the ticket from the
> datanode, lead to a second report failed because no lease
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17093) In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block reporting
[
https://issues.apache.org/jira/browse/HDFS-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yanlei Yu updated HDFS-17093:
-
Attachment: HDFS-17093.patch
> In the case of all datanodes sending FBR when the namenode restarts (large
> clusters), there is an issue with incomplete block reporting
> ---
>
> Key: HDFS-17093
> URL: https://issues.apache.org/jira/browse/HDFS-17093
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: Yanlei Yu
>Priority: Minor
> Attachments: HDFS-17093.patch
>
>
> In our cluster of 800+ nodes, after restarting the namenode, we found that
> some datanodes did not report enough blocks, causing the namenode to stay in
> secure mode for a long time after restarting because of incomplete block
> reporting
> I found in the logs of the datanode with incomplete block reporting that the
> first FBR attempt failed, possibly due to namenode stress, and then a second
> FBR attempt was made as follows:
> {code:java}
>
> 2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
> report(s), of which we sent 1. The reports had 1099057 total blocks and used
> 1 RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
> processing. Got back no commands.
> 2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Successfully sent block report 0x62382416f3f055, containing 12 storage
> report(s), of which we sent 12. The reports had 1099048 total blocks and used
> 12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
> processing. Got back no commands. {code}
> There's nothing wrong with that. Retry the send if it fails But on the
> namenode side of the logic:
> {code:java}
> if (namesystem.isInStartupSafeMode()
> && !StorageType.PROVIDED.equals(storageInfo.getStorageType())
> && storageInfo.getBlockReportCount() > 0) {
> blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
> + "discarded non-initial block report from {}"
> + " because namenode still in startup phase",
> strBlockReportId, fullBrLeaseId, nodeID);
> blockReportLeaseManager.removeLease(node);
> return !node.hasStaleStorages();
> } {code}
> When a disk was identified as the report is not the first time, namely
> storageInfo. GetBlockReportCount > 0, Will remove the ticket from the
> datanode, lead to a second report failed because no lease
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17093) In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block reporting
Yanlei Yu created HDFS-17093:
Summary: In the case of all datanodes sending FBR when the
namenode restarts (large clusters), there is an issue with incomplete block
reporting
Key: HDFS-17093
URL: https://issues.apache.org/jira/browse/HDFS-17093
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 3.3.4
Reporter: Yanlei Yu
In our cluster of 800+ nodes, after restarting the namenode, we found that some
datanodes did not report enough blocks, causing the namenode to stay in secure
mode for a long time after restarting because of incomplete block reporting
I found in the logs of the datanode with incomplete block reporting that the
first FBR attempt failed, possibly due to namenode stress, and then a second
FBR attempt was made as follows:
{code:java}
2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
report(s), of which we sent 1. The reports had 1099057 total blocks and used 1
RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
processing. Got back no commands.
2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Successfully sent block report 0x62382416f3f055, containing 12 storage
report(s), of which we sent 12. The reports had 1099048 total blocks and used
12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
processing. Got back no commands. {code}
There's nothing wrong with that. Retry the send if it fails But on the namenode
side of the logic:
{code:java}
if (namesystem.isInStartupSafeMode()
&& !StorageType.PROVIDED.equals(storageInfo.getStorageType())
&& storageInfo.getBlockReportCount() > 0) {
blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
+ "discarded non-initial block report from {}"
+ " because namenode still in startup phase",
strBlockReportId, fullBrLeaseId, nodeID);
blockReportLeaseManager.removeLease(node);
return !node.hasStaleStorages();
} {code}
When a disk was identified as the report is not the first time, namely
storageInfo. GetBlockReportCount > 0, Will remove the ticket from the datanode,
lead to a second report failed because no lease
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15901) Solve the problem of DN repeated block reports occupying too many RPCs during Safemode
[
https://issues.apache.org/jira/browse/HDFS-15901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743628#comment-17743628
]
Yanlei Yu edited comment on HDFS-15901 at 7/17/23 6:00 AM:
---
In our cluster of 800+ nodes, after restarting the namenode, we found that some
datanodes did not report enough blocks, causing the namenode to stay in secure
mode for a long time after restarting because of incomplete block reporting
I found in the logs of the datanode with incomplete block reporting that the
first FBR attempt failed, possibly due to namenode stress, and then a second
FBR attempt was made as follows:
{code:java}
2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
report(s), of which we sent 1. The reports had 1099057 total blocks and used 1
RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
processing. Got back no commands.
2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Successfully sent block report 0x62382416f3f055, containing 12 storage
report(s), of which we sent 12. The reports had 1099048 total blocks and used
12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
processing. Got back no commands.
... {code}
There's nothing wrong with that. Retry the send if it fails But on the namenode
side of the logic:
{code:java}
if (namesystem.isInStartupSafeMode()
&& !StorageType.PROVIDED.equals(storageInfo.getStorageType())
&& storageInfo.getBlockReportCount() > 0) {
blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
+ "discarded non-initial block report from {}"
+ " because namenode still in startup phase",
strBlockReportId, fullBrLeaseId, nodeID);
blockReportLeaseManager.removeLease(node);
return !node.hasStaleStorages();
} {code}
When a disk was identified as the report is not the first time, namely
storageInfo. GetBlockReportCount > 0, Will remove the ticket from the datanode,
lead to a second report failed because no lease
was (Author: JIRAUSER294151):
In our cluster of 800+ nodes, after restarting the namenode, we found that some
datanodes did not report enough blocks, causing the namenode to stay in secure
mode for a long time after restarting because of incomplete block reporting
I found in the logs of the datanode with incomplete block reporting that the
first FBR attempt failed, possibly due to namenode stress, and then a second
FBR attempt was made as follows:
{code:java}
//代码占位符
2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
report(s), of which we sent 1. The reports had 1099057 total blocks and used 1
RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
processing. Got back no commands.
2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Successfully sent block report 0x62382416f3f055, containing 12 storage
report(s), of which we sent 12. The reports had 1099048 total blocks and used
12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
processing. Got back no commands.
... {code}
There's nothing wrong with that. Retry the send if it fails But on the namenode
side of the logic:
{code:java}
//代码占位符
if (namesystem.isInStartupSafeMode()
&& !StorageType.PROVIDED.equals(storageInfo.getStorageType())
&& storageInfo.getBlockReportCount() > 0) {
blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
+ "discarded non-initial block report from {}"
+ " because namenode still in startup phase",
strBlockReportId, fullBrLeaseId, nodeID);
blockReportLeaseManager.removeLease(node);
return !node.hasStaleStorages();
} {code}
When a disk was identified as the report is not the first time, namely
storageInfo. GetBlockReportCount > 0, Will remove the ticket from the datanode,
lead to a second report failed because no lease
> Solve the problem of DN repeated block reports occupying too many RPCs during
> Safemode
> --
>
> Key: HDFS-15901
> URL: https://issues.apache.org/jira/browse/HDFS-15901
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> When the cluster exceeds thousands of nodes, we want to restart the NameNode
> service, and all DataNodes send a full Block action to the NameNode. During
> SafeMode, some
[jira] [Commented] (HDFS-15901) Solve the problem of DN repeated block reports occupying too many RPCs during Safemode
[
https://issues.apache.org/jira/browse/HDFS-15901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743628#comment-17743628
]
Yanlei Yu commented on HDFS-15901:
--
In our cluster of 800+ nodes, after restarting the namenode, we found that some
datanodes did not report enough blocks, causing the namenode to stay in secure
mode for a long time after restarting because of incomplete block reporting
I found in the logs of the datanode with incomplete block reporting that the
first FBR attempt failed, possibly due to namenode stress, and then a second
FBR attempt was made as follows:
{code:java}
//代码占位符
2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
report(s), of which we sent 1. The reports had 1099057 total blocks and used 1
RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
processing. Got back no commands.
2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Successfully sent block report 0x62382416f3f055, containing 12 storage
report(s), of which we sent 12. The reports had 1099048 total blocks and used
12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
processing. Got back no commands.
... {code}
There's nothing wrong with that. Retry the send if it fails But on the namenode
side of the logic:
{code:java}
//代码占位符
if (namesystem.isInStartupSafeMode()
&& !StorageType.PROVIDED.equals(storageInfo.getStorageType())
&& storageInfo.getBlockReportCount() > 0) {
blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
+ "discarded non-initial block report from {}"
+ " because namenode still in startup phase",
strBlockReportId, fullBrLeaseId, nodeID);
blockReportLeaseManager.removeLease(node);
return !node.hasStaleStorages();
} {code}
When a disk was identified as the report is not the first time, namely
storageInfo. GetBlockReportCount > 0, Will remove the ticket from the datanode,
lead to a second report failed because no lease
> Solve the problem of DN repeated block reports occupying too many RPCs during
> Safemode
> --
>
> Key: HDFS-15901
> URL: https://issues.apache.org/jira/browse/HDFS-15901
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> When the cluster exceeds thousands of nodes, we want to restart the NameNode
> service, and all DataNodes send a full Block action to the NameNode. During
> SafeMode, some DataNodes may send blocks to NameNode multiple times, which
> will take up too much RPC. In fact, this is unnecessary.
> In this case, some block report leases will fail or time out, and in extreme
> cases, the NameNode will always stay in Safe Mode.
> 2021-03-14 08:16:25,873 [78438700] - INFO [Block report
> processor:BlockManager@2158] - BLOCK* processReport 0xe: discarded
> non-initial block report from DatanodeRegistration(:port,
> datanodeUuid=, infoPort=, infoSecurePort=,
> ipcPort=, storageInfo=lv=;nsid=;c=0) because namenode
> still in startup phase
> 2021-03-14 08:16:31,521 [78444348] - INFO [Block report
> processor:BlockManager@2158] - BLOCK* processReport 0xe: discarded
> non-initial block report from DatanodeRegistration(,
> datanodeUuid=, infoPort=, infoSecurePort=,
> ipcPort=, storageInfo=lv=;nsid=;c=0) because namenode
> still in startup phase
> 2021-03-13 18:35:38,200 [29191027] - WARN [Block report
> processor:BlockReportLeaseManager@311] - BR lease 0x is not valid for
> DN , because the DN is not in the pending set.
> 2021-03-13 18:36:08,143 [29220970] - WARN [Block report
> processor:BlockReportLeaseManager@311] - BR lease 0x is not valid for
> DN , because the DN is not in the pending set.
> 2021-03-13 18:36:08,143 [29220970] - WARN [Block report
> processor:BlockReportLeaseManager@317] - BR lease 0x is not valid for
> DN , because the lease has expired.
> 2021-03-13 18:36:08,145 [29220972] - WARN [Block report
> processor:BlockReportLeaseManager@317] - BR lease 0x is not valid for
> DN , because the lease has expired.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] (HDFS-15901) Solve the problem of DN repeated block reports occupying too many RPCs during Safemode
[ https://issues.apache.org/jira/browse/HDFS-15901 ] Yanlei Yu deleted comment on HDFS-15901: -- was (Author: JIRAUSER294151): This seems to be a bug, we also encountered a similar error, after restarting the namenode, we found that the datanode FBR in the namenode log, some disk block report could not be reported successfully because of invalid ticket, because it was considered as a second report. After processReport method called processFirstBlockReport storageInfo. ReceivedBlockReport (); blockReportCount++; , processFirstBlockReport processing is sent to the queue is not actual processing quick report, then it is likely that they will appear error, lead to the first piece of report by mistake for the second time, then will enter storageInfo. GetBlockReportCount () > 0, Then blockReportLeaseManager. RemoveLease (node); , causing block reports to be rejected for subsequent renewals on the datanode > Solve the problem of DN repeated block reports occupying too many RPCs during > Safemode > -- > > Key: HDFS-15901 > URL: https://issues.apache.org/jira/browse/HDFS-15901 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > When the cluster exceeds thousands of nodes, we want to restart the NameNode > service, and all DataNodes send a full Block action to the NameNode. During > SafeMode, some DataNodes may send blocks to NameNode multiple times, which > will take up too much RPC. In fact, this is unnecessary. > In this case, some block report leases will fail or time out, and in extreme > cases, the NameNode will always stay in Safe Mode. > 2021-03-14 08:16:25,873 [78438700] - INFO [Block report > processor:BlockManager@2158] - BLOCK* processReport 0xe: discarded > non-initial block report from DatanodeRegistration(:port, > datanodeUuid=, infoPort=, infoSecurePort=, > ipcPort=, storageInfo=lv=;nsid=;c=0) because namenode > still in startup phase > 2021-03-14 08:16:31,521 [78444348] - INFO [Block report > processor:BlockManager@2158] - BLOCK* processReport 0xe: discarded > non-initial block report from DatanodeRegistration(, > datanodeUuid=, infoPort=, infoSecurePort=, > ipcPort=, storageInfo=lv=;nsid=;c=0) because namenode > still in startup phase > 2021-03-13 18:35:38,200 [29191027] - WARN [Block report > processor:BlockReportLeaseManager@311] - BR lease 0x is not valid for > DN , because the DN is not in the pending set. > 2021-03-13 18:36:08,143 [29220970] - WARN [Block report > processor:BlockReportLeaseManager@311] - BR lease 0x is not valid for > DN , because the DN is not in the pending set. > 2021-03-13 18:36:08,143 [29220970] - WARN [Block report > processor:BlockReportLeaseManager@317] - BR lease 0x is not valid for > DN , because the lease has expired. > 2021-03-13 18:36:08,145 [29220972] - WARN [Block report > processor:BlockReportLeaseManager@317] - BR lease 0x is not valid for > DN , because the lease has expired. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15901) Solve the problem of DN repeated block reports occupying too many RPCs during Safemode
[ https://issues.apache.org/jira/browse/HDFS-15901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743595#comment-17743595 ] Yanlei Yu commented on HDFS-15901: -- This seems to be a bug, we also encountered a similar error, after restarting the namenode, we found that the datanode FBR in the namenode log, some disk block report could not be reported successfully because of invalid ticket, because it was considered as a second report. After processReport method called processFirstBlockReport storageInfo. ReceivedBlockReport (); blockReportCount++; , processFirstBlockReport processing is sent to the queue is not actual processing quick report, then it is likely that they will appear error, lead to the first piece of report by mistake for the second time, then will enter storageInfo. GetBlockReportCount () > 0, Then blockReportLeaseManager. RemoveLease (node); , causing block reports to be rejected for subsequent renewals on the datanode > Solve the problem of DN repeated block reports occupying too many RPCs during > Safemode > -- > > Key: HDFS-15901 > URL: https://issues.apache.org/jira/browse/HDFS-15901 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > When the cluster exceeds thousands of nodes, we want to restart the NameNode > service, and all DataNodes send a full Block action to the NameNode. During > SafeMode, some DataNodes may send blocks to NameNode multiple times, which > will take up too much RPC. In fact, this is unnecessary. > In this case, some block report leases will fail or time out, and in extreme > cases, the NameNode will always stay in Safe Mode. > 2021-03-14 08:16:25,873 [78438700] - INFO [Block report > processor:BlockManager@2158] - BLOCK* processReport 0xe: discarded > non-initial block report from DatanodeRegistration(:port, > datanodeUuid=, infoPort=, infoSecurePort=, > ipcPort=, storageInfo=lv=;nsid=;c=0) because namenode > still in startup phase > 2021-03-14 08:16:31,521 [78444348] - INFO [Block report > processor:BlockManager@2158] - BLOCK* processReport 0xe: discarded > non-initial block report from DatanodeRegistration(, > datanodeUuid=, infoPort=, infoSecurePort=, > ipcPort=, storageInfo=lv=;nsid=;c=0) because namenode > still in startup phase > 2021-03-13 18:35:38,200 [29191027] - WARN [Block report > processor:BlockReportLeaseManager@311] - BR lease 0x is not valid for > DN , because the DN is not in the pending set. > 2021-03-13 18:36:08,143 [29220970] - WARN [Block report > processor:BlockReportLeaseManager@311] - BR lease 0x is not valid for > DN , because the DN is not in the pending set. > 2021-03-13 18:36:08,143 [29220970] - WARN [Block report > processor:BlockReportLeaseManager@317] - BR lease 0x is not valid for > DN , because the lease has expired. > 2021-03-13 18:36:08,145 [29220972] - WARN [Block report > processor:BlockReportLeaseManager@317] - BR lease 0x is not valid for > DN , because the lease has expired. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16726) There is a memory-related problem about HDFS namenode
[ https://issues.apache.org/jira/browse/HDFS-16726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17688317#comment-17688317 ] Yanlei Yu commented on HDFS-16726: -- hello [~zinx] ,We can't do anything about this problem. We are using version 2.7.2, what version are you using? We are going to upgrade our cluster to version 3 to see if it improves. > There is a memory-related problem about HDFS namenode > - > > Key: HDFS-16726 > URL: https://issues.apache.org/jira/browse/HDFS-16726 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs, namenode >Affects Versions: 2.7.2 > Environment: -Xms280G -Xmx280G -XX:MaxDirectMemorySize=10G > -XX:MetaspaceSize=128M -server \ > -XX:+UseG1GC -XX:+UseStringDeduplication > -XX:MaxGCPauseMillis=250 -XX:+UnlockExperimentalVMOptions > -XX:+PrintGCApplicationStoppedTime -XX:+PrintSafepointStatistics > -XX:PrintSafepointStatisticsCount=1 \ > -XX:G1OldCSetRegionThresholdPercent=1 > -XX:G1MixedGCCountTarget=9 -XX:+SafepointTimeout > -XX:SafepointTimeoutDelay=4000 \ > -XX:ParallelGCThreads=24 -XX:ConcGCThreads=6 > -XX:G1RSetRegionEntries=4096 -XX:+AggressiveOpts -XX:+DisableExplicitGC \ > -XX:G1HeapWastePercent=9 > -XX:G1MixedGCLiveThresholdPercent=85 -XX:InitiatingHeapOccupancyPercent=75 \ > -XX:+ParallelRefProcEnabled -XX:-ResizePLAB > -XX:+PrintAdaptiveSizePolicy \ > -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps \ > -Xloggc:$HADOOP_LOG_DIR/namenode.gc.log \ > -XX:+HeapDumpOnOutOfMemoryError > -XX:ErrorFile=$HADOOP_LOG_DIR/hs_err_pid%p.log > -XX:HeapDumpPath=$HADOOP_LOG_DIR \ > -Dcom.sun.management.jmxremote \ > -Dcom.sun.management.jmxremote.port=9009 \ > -Dcom.sun.management.jmxremote.ssl=false \ > -Dcom.sun.management.jmxremote.authenticate=false \ > $HADOOP_NAMENODE_OPTS >Reporter: Yanlei Yu >Priority: Critical > > In the cluster, the memory usage of Namenode exceeds the XMX setting (XMX > =280GB). The actual memory usage of Namenode is 479GB > Output via pamp: > Address Perm Offset Device Inode Size Rss Pss > Referenced Anonymous Swap Locked Mapping > 2b42f000 rw-p 00:00 0 294174720 293756960 293756960 > 293756960 293756960 0 0 > 01e21000 rw-p 00:00 0 195245456 195240848 195240848 > 195240848 195240848 0 0 [heap] > 2b897c00 rw-p 00:00 0 9246724 9246724 9246724 > 9246724 9246724 0 0 > 2b8bb0905000 rw-p 00:00 0 1781124 1754572 1754572 > 1754572 1754572 0 0 > 2b893600 rw-p 00:00 0 1146880 1002084 1002084 > 1002084 1002084 0 0 > 2b42db652000 rwxp 00:00 0 57792 55252 55252 > 55252 55252 0 0 > 2b42ec12a000 rw-p 00:00 0 25696 24700 24700 > 24700 24700 0 0 > 2b42ef25b000 rw-p 00:00 0 9988 8972 8972 > 8972 8972 0 0 > 2b8c1d467000 rw-p 00:00 0 9216 8204 8204 > 8204 8204 0 0 > 2b8d6f8db000 rw-p 00:00 0 7160 6228 6228 > 6228 6228 0 0 > The first line should configure the memory footprint for XMX, and [heap] is > unusually large, so a memory leak is suspected! > > * [heap] is associated with malloc > After configuring JCMD in the test environment, we found that the malloc part > of Internal in JCMD increased significantly when the client was writing to a > gz file (XMX =40g in the test environment, and the Internal area was 900MB > before the client wrote) : > Total: reserved=47276MB, committed=47070MB > - Java Heap (reserved=40960MB, committed=40960MB) > (mmap: reserved=40960MB, committed=40960MB) > > - Class (reserved=53MB, committed=52MB) > (classes #7423) > (malloc=1MB #17053) > (mmap: reserved=52MB, committed=52MB) > > - Thread (reserved=2145MB, committed=2145MB) > (thread #2129) > (stack: reserved=2136MB, committed=2136MB) > (malloc=7MB #10673) > (arena=2MB #4256) > > - Code (reserved=251MB, committed=45MB) > (malloc=7MB #10661) >
[jira] [Commented] (HDFS-16863) Optimize frequency of regular block reports
[ https://issues.apache.org/jira/browse/HDFS-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655868#comment-17655868 ] Yanlei Yu commented on HDFS-16863: -- hi,[~jianghuazhu], Thank you for your reply. As for the first point you mentioned, even periodic FBR cannot provide timely notification. If this situation is abnormal, it should be regarded as the pre-judgment condition of FBR As for your second point, I don't think it has anything to do with FBR. datanode if it doesn't copy the data copy successfully, it has a mechanism to handle the unsuccessful copy. That's not what FBR does > Optimize frequency of regular block reports > --- > > Key: HDFS-16863 > URL: https://issues.apache.org/jira/browse/HDFS-16863 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yanlei Yu >Priority: Major > Attachments: HDFS-16863.patch > > > like HDFS-15162 > Avoid sending block report at regular interval, if there is no failover, > DiskError or any exception encountered in connecting to the Namenode. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7923) The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages
[ https://issues.apache.org/jira/browse/HDFS-7923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17647866#comment-17647866 ] Yanlei Yu commented on HDFS-7923: - hi，[~cmccabe] ，in my 1000+ cluster test, I found that during namenode restart, if datanode does not connect to namenode because namenode is busy, it will print IOException in offerService, However, forceFullBr is already false in the next loop, which causes the datanode to fail to send FBR during the namenode restart. As a result, the namenode restarts slowly. The block reporting phase takes about 20 hours. I can set forceFullBr to true after the exception is received so that the datanode can send the FBR in a timely manner. How do you feel? > The DataNodes should rate-limit their full block reports by asking the NN on > heartbeat messages > --- > > Key: HDFS-7923 > URL: https://issues.apache.org/jira/browse/HDFS-7923 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.8.0 >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Major > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-7923.000.patch, HDFS-7923.001.patch, > HDFS-7923.002.patch, HDFS-7923.003.patch, HDFS-7923.004.patch, > HDFS-7923.006.patch, HDFS-7923.007.patch > > > The DataNodes should rate-limit their full block reports. They can do this > by first sending a heartbeat message to the NN with an optional boolean set > which requests permission to send a full block report. If the NN responds > with another optional boolean set, the DN will send an FBR... if not, it will > wait until later. This can be done compatibly with optional fields. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15162) Optimize frequency of regular block reports
[ https://issues.apache.org/jira/browse/HDFS-15162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17644205#comment-17644205 ] Yanlei Yu commented on HDFS-15162: -- hi，[~ayushtkn]，I submitted a patch to HDFS-16863, and FBR was executed only when failover and DiskError occurred. When I lost contact with namenode, I felt that sending FBR was meaningless > Optimize frequency of regular block reports > --- > > Key: HDFS-15162 > URL: https://issues.apache.org/jira/browse/HDFS-15162 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Critical > > Avoid sending block report at regular interval, if there is no failover, > DiskError or any exception encountered in connecting to the Namenode. > This JIRA intents to limit the regular block reports to be sent only in case > of the above scenarios and during re-registration of datanode, to eliminate > the overhead of processing BlockReports at Namenode in case of huge clusters. > *Eg.* If a block report was sent at hours and the next was scheduled at > 0600 hours if there is no above mentioned scenario, it will skip sending the > BR, and schedule it to next 1200 hrs. if something of such sort happens > between 06:- 12: it would send the BR normally. > *NOTE*: This would be optional and can be turned off by default. Would add a > configuration to enable this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16863) Optimize frequency of regular block reports
[ https://issues.apache.org/jira/browse/HDFS-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17644204#comment-17644204 ] Yanlei Yu commented on HDFS-16863: -- hi，[~ayushtkn]，I submitted a patch, and FBR was executed only when failover and DiskError occurred. When I lost contact with namenode, I felt that sending FBR was meaningless > Optimize frequency of regular block reports > --- > > Key: HDFS-16863 > URL: https://issues.apache.org/jira/browse/HDFS-16863 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yanlei Yu >Priority: Major > Attachments: HDFS-16863.patch > > > like HDFS-15162 > Avoid sending block report at regular interval, if there is no failover, > DiskError or any exception encountered in connecting to the Namenode. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16863) Optimize frequency of regular block reports
[ https://issues.apache.org/jira/browse/HDFS-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanlei Yu updated HDFS-16863: - Attachment: HDFS-16863.patch > Optimize frequency of regular block reports > --- > > Key: HDFS-16863 > URL: https://issues.apache.org/jira/browse/HDFS-16863 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yanlei Yu >Priority: Major > Attachments: HDFS-16863.patch > > > like HDFS-15162 > Avoid sending block report at regular interval, if there is no failover, > DiskError or any exception encountered in connecting to the Namenode. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16863) Optimize frequency of regular block reports
Yanlei Yu created HDFS-16863: Summary: Optimize frequency of regular block reports Key: HDFS-16863 URL: https://issues.apache.org/jira/browse/HDFS-16863 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Reporter: Yanlei Yu Attachments: HDFS-16863.patch like HDFS-15162 Avoid sending block report at regular interval, if there is no failover, DiskError or any exception encountered in connecting to the Namenode. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17643702#comment-17643702 ] Yanlei Yu commented on HDFS-16432: -- hi [~qinyuren] , Similar to this [HDFS-14657|https://issues.apache.org/jira/browse/HDFS-14657], both control the lock time. You directly control the time, while HDFS-14657 controls the lock time by setting the number of blocks > Namenode block report add yield to avoid holding write lock too long > > > Key: HDFS-16432 > URL: https://issues.apache.org/jira/browse/HDFS-16432 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Labels: pull-request-available > Attachments: 20220209-120613.html, image-2022-01-20-15-19-28-384.png > > Time Spent: 2h 20m > Remaining Estimate: 0h > > !image-2022-01-20-15-19-28-384.png|width=683,height=132! > In our cluster, namenode block report will held write lock for a long time if > the storage block number more than 10. So we want to add a yield > mechanism in block reporting process to avoid holding write lock too long. > # Ensure that the processing of the same block is in the same write lock. > # Because StorageInfo.addBlock will moves the block to the head of > blockList, so we can collect blocks that have not been reported by delimiter > block. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14657) Refine NameSystem lock usage during processing FBR
[ https://issues.apache.org/jira/browse/HDFS-14657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17642233#comment-17642233 ] Yanlei Yu commented on HDFS-14657: -- hi [~zhangchen] ,Whether the new patch can be uploaded? > Refine NameSystem lock usage during processing FBR > -- > > Key: HDFS-14657 > URL: https://issues.apache.org/jira/browse/HDFS-14657 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-14657-001.patch, HDFS-14657.002.patch > > > The disk with 12TB capacity is very normal today, which means the FBR size is > much larger than before, Namenode holds the NameSystemLock during processing > block report for each storage, which might take quite a long time. > On our production environment, processing large FBR usually cause a longer > RPC queue time, which impacts client latency, so we did some simple work on > refining the lock usage, which improved the p99 latency significantly. > In our solution, BlockManager release the NameSystem write lock and request > it again for every 5000 blocks(by default) during processing FBR, with the > fair lock, all the RPC request can be processed before BlockManager > re-acquire the write lock. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10453) ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.
[
https://issues.apache.org/jira/browse/HDFS-10453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624837#comment-17624837
]
Yanlei Yu commented on HDFS-10453:
--
Hi [~hexiaoqiao].
After reading your previous discussion, I would like to confirm whether
HDFS-10453-branch-2.7.009 patch has solved this problem
> ReplicationMonitor thread could stuck for long time due to the race between
> replication and delete of same file in a large cluster.
> ---
>
> Key: HDFS-10453
> URL: https://issues.apache.org/jira/browse/HDFS-10453
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.1, 2.5.2, 2.7.1, 2.6.4
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 2.9.1, 2.8.4, 2.7.6, 3.0.3
>
> Attachments: HDFS-10453-branch-2.001.patch,
> HDFS-10453-branch-2.003.patch, HDFS-10453-branch-2.7.004.patch,
> HDFS-10453-branch-2.7.005.patch, HDFS-10453-branch-2.7.006.patch,
> HDFS-10453-branch-2.7.007.patch, HDFS-10453-branch-2.7.008.patch,
> HDFS-10453-branch-2.7.009.patch, HDFS-10453-branch-2.8.001.patch,
> HDFS-10453-branch-2.8.002.patch, HDFS-10453-branch-2.9.001.patch,
> HDFS-10453-branch-2.9.002.patch, HDFS-10453-branch-3.0.001.patch,
> HDFS-10453-branch-3.0.002.patch, HDFS-10453-trunk.001.patch,
> HDFS-10453-trunk.002.patch, HDFS-10453.001.patch
>
>
> ReplicationMonitor thread could stuck for long time and loss data with little
> probability. Consider the typical scenario：
> (1) create and close a file with the default replicas(3);
> (2) increase replication (to 10) of the file.
> (3) delete the file while ReplicationMonitor is scheduling blocks belong to
> that file for replications.
> if ReplicationMonitor stuck reappeared, NameNode will print log as:
> {code:xml}
> 2016-04-19 10:20:48,083 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 7 to reach 10
> (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
> newBlock=false) For more information, please enable DEBUG log level on
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
> ..
> 2016-04-19 10:21:17,184 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 7 to reach 10
> (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
> newBlock=false) For more information, please enable DEBUG log level on
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
> 2016-04-19 10:21:17,184 WARN
> org.apache.hadoop.hdfs.protocol.BlockStoragePolicy: Failed to place enough
> replicas: expected size is 7 but only 0 storage types can be selected
> (replication=10, selected=[], unavailable=[DISK, ARCHIVE], removed=[DISK,
> DISK, DISK, DISK, DISK, DISK, DISK], policy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]})
> 2016-04-19 10:21:17,184 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 7 to reach 10
> (unavailableStorages=[DISK, ARCHIVE], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
> newBlock=false) All required storage types are unavailable:
> unavailableStorages=[DISK, ARCHIVE], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
> {code}
> This is because 2 threads (#NameNodeRpcServer and #ReplicationMonitor)
> process same block at the same moment.
> (1) ReplicationMonitor#computeReplicationWorkForBlocks get blocks to
> replicate and leave the global lock.
> (2) FSNamesystem#delete invoked to delete blocks then clear the reference in
> blocksmap, needReplications, etc. the block's NumBytes will set
> NO_ACK(Long.MAX_VALUE) which is used to indicate that the block deletion does
> not need explicit ACK from the node.
> (3) ReplicationMonitor#computeReplicationWorkForBlocks continue to
> chooseTargets for the same blocks and no node will be selected after traverse
> whole cluster because no node choice satisfy the goodness criteria
> (remaining spaces achieve required size Long.MAX_VALUE).
> During of stage#3 ReplicationMonitor stuck for long time, especial in a large
> cluster. invalidateBlocks & neededReplications continues to grow and no
> consumes. it will loss data at the worst.
> This can mostly be avoided by skip
[jira] (HDFS-16765) "hdfs namenode -rollingUpgrade started" has long execution time
[ https://issues.apache.org/jira/browse/HDFS-16765 ] Yanlei Yu deleted comment on HDFS-16765: -- was (Author: JIRAUSER294151): !hadoop-daemon.sh.png! > "hdfs namenode -rollingUpgrade started" has long execution time > --- > > Key: HDFS-16765 > URL: https://issues.apache.org/jira/browse/HDFS-16765 > Project: Hadoop HDFS > Issue Type: Bug > Components: rolling upgrades >Affects Versions: 3.3.4 >Reporter: Haobetter >Priority: Major > > I upgraded version 3.2.1 to version 3.3.4 and executed "HDFS namode > -rollingUpgrade Started", which did not stop after 10 hours. I don't think > that's too long.。 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16765) "hdfs namenode -rollingUpgrade started" has long execution time
[ https://issues.apache.org/jira/browse/HDFS-16765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17616364#comment-17616364 ] Yanlei Yu commented on HDFS-16765: -- !hadoop-daemon.sh.png! > "hdfs namenode -rollingUpgrade started" has long execution time > --- > > Key: HDFS-16765 > URL: https://issues.apache.org/jira/browse/HDFS-16765 > Project: Hadoop HDFS > Issue Type: Bug > Components: rolling upgrades >Affects Versions: 3.3.4 >Reporter: Haobetter >Priority: Major > > I upgraded version 3.2.1 to version 3.3.4 and executed "HDFS namode > -rollingUpgrade Started", which did not stop after 10 hours. I don't think > that's too long.。 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16765) "hdfs namenode -rollingUpgrade started" has long execution time
[
https://issues.apache.org/jira/browse/HDFS-16765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17616359#comment-17616359
]
Yanlei Yu commented on HDFS-16765:
--
This is normal
Normal start and stop command is used in the namenode or datanode: sbin/hadoop
- daemon. Sh start | stop the namenode | datanode
In hadoop-daemon.sh, the final execution logic is: nohup nice -n
$HADOOP_NICENESS $hdfsScript --config $HADOOP_CONF_DIR $command "$@" > "$log"
2>&1 < /dev/null &
Look at rolling upgrade start namnode command: bin/hdfs namenode -
rollingUpgrade started, actually also is to use bin/hdfs, just no nohup to the
background
Therefore, it is normal that the bin/hdfs namode -rollingUpgrade started
command does not exit when the rolling upgrade starts. The bin/hdfs command is
used in the end, just like the normal start command.
You can have it run in the background, like this:{{{}nohup bin/hdfs namenode
{}}}{{-rollingUpgrade}} {{started > logs/nn1RollingUpgrade.log 2>&1 & }}
> "hdfs namenode -rollingUpgrade started" has long execution time
> ---
>
> Key: HDFS-16765
> URL: https://issues.apache.org/jira/browse/HDFS-16765
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rolling upgrades
>Affects Versions: 3.3.4
>Reporter: Haobetter
>Priority: Major
>
> I upgraded version 3.2.1 to version 3.3.4 and executed "HDFS namode
> -rollingUpgrade Started", which did not stop after 10 hours. I don't think
> that's too long.。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
