[
https://issues.apache.org/jira/browse/HADOOP-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573523#action_12573523
]
Christian Kunz commented on HADOOP-2907:
----------------------------------------
Every map application streams through about 40-60 DFS files and writes directly
to 2 DFS files (besides sending data to map-reduce framework). None of the maps
accesses the same data.
Every reduce application writes to about 70 DFS files.
Keep in mind, that the datanodes ran out of memory already during the map phase.
This access pattern did not change for a while now. But we started to see
datanodes running out of memory with about nightly build #810.
I am surprised about the high number of connections. Are some of them stale?
> dead datanodes because of OutOfMemoryError
> ------------------------------------------
>
> Key: HADOOP-2907
> URL: https://issues.apache.org/jira/browse/HADOOP-2907
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Christian Kunz
>
> We see more dead datanodes than in previous releases. The common exception is
> found in the out file:
> Exception in thread "[EMAIL PROTECTED]" java.lang.OutOfMemoryError: Java heap
> space
> Exception in thread "DataNode: [dfs.data.dir-value]"
> java.lang.OutOfMemoryError: Java heap space
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
