[
https://issues.apache.org/jira/browse/HDFS-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718938#comment-13718938
]
Suresh Srinivas edited comment on HDFS-4979 at 7/24/13 10:51 PM:
-----------------------------------------------------------------
ClientCacheEntry/ClientCacheEntryWithPayload cost:
* Object cost - 16 bytes
* state - 1 byte
* clientId - 16 bytes
* callId - 4 bytes
* next elemet - 8 bytes
* Payload cost - assume on an average 24 bytes
* PriorityQueue expiration time - 8 bytes
* PriorityQueue entry reference - 8 bytes
*= 85 bytes*
Assuming 150 bytes for files and blocks, a file costs ~300 bytes of namenode
heap space.
> The total heap space for with F number of file is 300 * F. Lets generously
> assume that this uses up the entire namenode java heap space.
>> Considering number of retry cache entries for a period of 10 minutes is 1%
>> of number of files count, number of retry cache entries is ~0.01 * F.
>> The heap space for retry cache including cached response is 0.01 * F *
>> (85+8) bytes = ~F bytes. This is 0.33% of total java heap space.
>> The heap space that retry cache alone should be sized on: (8 bytes per entry
>> * 0.33 * total heap space)/85 = ~0.028% of total heap space
I plan on using 0.03% of total heap space for retry cache map. This would be
0.345% of java heap space for retry cache + cached responses.
Lets look at some examples of how this number works out:
||Heap space||Num of cache entries||Total memory for retry cache||Ops/second
(sustained for 10 minutes)||
|1M|64|6.25K|1 ops for every 9 seconds|
|256M|16384|1.6M|27|
|1G|65536|6.4M|109|
|64G|4194304|409.6M|6992|
Note in the above calculation, number of cache entries are the nearest power of
2 number.
The number of entries from the table will the max limit for retry cache.
Does this calculation look correct? Does this amount of heap sound reasonable?
was (Author: sureshms):
ClientCacheEntry/ClientCacheEntryWithPayload cost:
* Object cost - 16 bytes
* state - 1 byte
* clientId - 16 bytes
* callId - 4 bytes
* next elemet - 8 bytes
* Payload cost - assume on an average 24 bytes
* PriorityQueue expiration time - 8 bytes
* PriorityQueue entry reference - 8 bytes
*= 96 bytes*
Assuming 150 bytes for files and blocks, a file costs ~300 bytes of namenode
heap space.
> The total heap space for with F number of file is 300 * F. Lets generously
> assume that this uses up the entire namenode java heap space.
>> Considering number of retry cache entries for a period of 10 minutes is 1%
>> of number of files count, number of retry cache entries is 0.01 * F.
>> The heap space for retry cache including cached response is 0.01 * F * 100
>> bytes = F bytes. This is 0.33% of total java heap space.
>> The heap space that retry cache alone should be sized on: (8 bytes per entry
>> * 0.33 * total heap space)/100 = ~0.026% of total heap space
I plan on using 0.03% of total heap space for retry cache map. This would be
0.375% of java heap space for retry cache + cached responses.
Lets look at some examples of how this number works out:
||Heap space||Num of cache entries||Total memory for retry cache||Ops/second
(sustained for 10 minutes)||
|1M|64|6.25K|1 ops for every 9 seconds|
|256M|16384|1.6M|27|
|1G|65536|6.4M|109|
|64G|4194304|409.6M|6992|
Does this calculation look correct? Does this amount of heap sound reasonable?
> Implement retry cache on the namenode
> -------------------------------------
>
> Key: HDFS-4979
> URL: https://issues.apache.org/jira/browse/HDFS-4979
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: HDFS-4979.10.patch, HDFS-4979.1.patch,
> HDFS-4979.2.patch, HDFS-4979.3.patch, HDFS-4979.4.patch, HDFS-4979.5.patch,
> HDFS-4979.6.patch, HDFS-4979.7.patch, HDFS-4979.8.patch, HDFS-4979.9.patch,
> HDFS-4979.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
