[
https://issues.apache.org/jira/browse/HDFS-17097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ConfX updated HDFS-17097:
-------------------------
Description:
h2. What happened:
When setting {{dfs.namenode.blockreport.queue.size}} to a large number,
{{BlockManager}} throws an out-of-memory exception due to inappropriate
checking and handling.
*There is no checking or error-handling logic for this parameter at all.*
h2. StackTrace:
{noformat}
java.lang.OutOfMemoryError: Java heap space
at
java.base/java.util.concurrent.ArrayBlockingQueue.<init>(ArrayBlockingQueue.java:270)
at
java.base/java.util.concurrent.ArrayBlockingQueue.<init>(ArrayBlockingQueue.java:254)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.<init>(BlockManager.java:5243)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.<init>(BlockManager.java:613)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:869)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:248)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:194)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:181){noformat}
h2. Reproduce:
(1) Set {{dfs.namenode.blockreport.queue.size}} to a large value, e.g.,
1807890864
(2) Run a simple test that exercises this parameter, e.g.
{{org.apache.hadoop.hdfs.server.namenode.TestNameNodeHttpServerXFrame#testSecondaryNameNodeXFrame}}
For an easy reproduction, run the reproduce.sh in the attachment.
We are happy to provide a patch if this issue is confirmed.
was:
h2. What happened:
When setting {{dfs.namenode.blockreport.queue.size}} to a large number,
{{BlockManager}} throws an out-of-memory exception due to inappropriate
checking and handling.
*There is no checking or error-handling logic for this parameter at all.*
h2. StackTrace:
{noformat}
java.lang.OutOfMemoryError: Java heap space
at
java.base/java.util.concurrent.ArrayBlockingQueue.<init>(ArrayBlockingQueue.java:270)
at
java.base/java.util.concurrent.ArrayBlockingQueue.<init>(ArrayBlockingQueue.java:254)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.<init>(BlockManager.java:5243)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.<init>(BlockManager.java:613)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:869)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:248)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:194)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:181){noformat}
h2. Reproduce:
(1) Set {{dfs.namenode.blockreport.queue.size}} to a large value, e.g.,
1807890864
(2) Run a simple test that exercises this parameter, e.g.
{{org.apache.hadoop.hdfs.server.namenode.TestNameNodeHttpServerXFrame#testSecondaryNameNodeXFrame}}
For an easy reproduction, run the reproduce.sh in the attachment.
> Out of Memory when set dfs.namenode.blockreport.queue.size to a large number
> ----------------------------------------------------------------------------
>
> Key: HDFS-17097
> URL: https://issues.apache.org/jira/browse/HDFS-17097
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: ConfX
> Priority: Critical
> Attachments: reproduce.sh
>
>
> h2. What happened:
> When setting {{dfs.namenode.blockreport.queue.size}} to a large number,
> {{BlockManager}} throws an out-of-memory exception due to inappropriate
> checking and handling.
> *There is no checking or error-handling logic for this parameter at all.*
> h2. StackTrace:
>
> {noformat}
> java.lang.OutOfMemoryError: Java heap space
> at
> java.base/java.util.concurrent.ArrayBlockingQueue.<init>(ArrayBlockingQueue.java:270)
> at
> java.base/java.util.concurrent.ArrayBlockingQueue.<init>(ArrayBlockingQueue.java:254)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.<init>(BlockManager.java:5243)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.<init>(BlockManager.java:613)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:869)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:248)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:194)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:181){noformat}
> h2. Reproduce:
> (1) Set {{dfs.namenode.blockreport.queue.size}} to a large value, e.g.,
> 1807890864
> (2) Run a simple test that exercises this parameter, e.g.
> {{org.apache.hadoop.hdfs.server.namenode.TestNameNodeHttpServerXFrame#testSecondaryNameNodeXFrame}}
> For an easy reproduction, run the reproduce.sh in the attachment.
> We are happy to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
