[
https://issues.apache.org/jira/browse/HADOOP-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624100#action_12624100
]
Hairong Kuang commented on HADOOP-2330:
---------------------------------------
I launched a benchmark application that starts 300 threads, each of which keeps
on creating a zero-length file (create/close) under a random directory and then
deleting it. I ran the program for ten minutes and the following is the data
that I collected:
Trunk with the preallocation patch:
Average deletion execution time: 42.18339892175518ms
Average create execution time: 44.52994046503582ms
Average write_close execution time: 43.39569781647428ms
Average operations per second: 6937.158333333334ops/s
Trunk (without the preallocation patch):
Average deletion execution time: 56.26059929403046ms
Average create execution time: 60.724891276790764ms
Average write_close execution time: 58.710387362570394ms
Average operations per second: 5127.675ops/s
This represented a 35% throughput improvement and a latency improvement of
33.3% for deletion, 36.4% for create, and 35.3% for close.
> Preallocate transaction log to improve namenode transaction logging
> performance
> -------------------------------------------------------------------------------
>
> Key: HADOOP-2330
> URL: https://issues.apache.org/jira/browse/HADOOP-2330
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: preallocateTransactionLog.patch,
> preallocateTransactionLog.patch, preallocateTransactionLog2.patch,
> preallocateTransactionLog3.patch
>
>
> In the current implementation, the transaction log is opened in "append" mode
> and every new transaction is written to the end of the log. This means that
> new blocks get allocated to the edits file frequently.
> It is worth measuring the performance improvement when big chunks of the
> transaction log are allocated up front. Adding new transactions do not cause
> frequent block allocations for the edits log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
