[
https://issues.apache.org/jira/browse/HDFS-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17386932#comment-17386932
]
Renukaprasad C commented on HDFS-16138:
---------------------------------------
Thanks [~hexiaoqiao] for the review.
This is FGL branch (Trunk & 3.1.1) and issue (AssertionError) is specific to
FGL code only. We found the cause and addressed the issue. But, any kind of
exceptions in trunk lead to the same stack and ignore the acutal issue. Which
is difficult to debug especially in production envs. So, its better to log the
complete trace which cause the issue.
> BlockReportProcessingThread exit doesnt print the acutal stack
> --------------------------------------------------------------
>
> Key: HDFS-16138
> URL: https://issues.apache.org/jira/browse/HDFS-16138
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Renukaprasad C
> Assignee: Renukaprasad C
> Priority: Major
>
> BlockReportProcessingThread thread may gets exited with multiple reasons, but
> the current logging prints only the exception message with different stack
> which is difficult to debug the issue.
>
> Existing logging:
> 2021-07-20 10:20:23,104 [Block report processor] INFO util.ExitUtil
> (ExitUtil.java:terminate(210)) - Exiting with status 1: Block report
> processor encountered fatal exception: java.lang.AssertionError
> 2021-07-20 10:20:23,104 [Block report processor] ERROR util.ExitUtil
> (ExitUtil.java:terminate(213)) - Terminate called
> 1: Block report processor encountered fatal exception:
> java.lang.AssertionError
> at
> org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:304)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:5315)
> Exception in thread "Block report processor" 1: Block report processor
> encountered fatal exception: java.lang.AssertionError
> at
> org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:304)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:5315)
>
> Actual issue found at:
> 2021-07-20 10:20:23,101 [Block report processor] ERROR
> blockmanagement.BlockManager (BlockManager.java:run(5314)) -
> java.lang.AssertionError
> java.lang.AssertionError
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addStoredBlock(BlockManager.java:3480)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processAndHandleReportedBlock(BlockManager.java:4280)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addBlock(BlockManager.java:4202)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processIncrementalBlockReport(BlockManager.java:4338)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processIncrementalBlockReport(BlockManager.java:4305)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processIncrementalBlockReport(FSNamesystem.java:4853)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer$2.run(NameNodeRpcServer.java:1657)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:5334)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:5312)
>
> This issue found while working on FGL branch. But, same issue can happen in
> Trunk also in any error scenario.
>
> [~hemanthboyina] [~hexiaoqiao]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
