[
https://issues.apache.org/jira/browse/HDFS-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218386#comment-13218386
]
Todd Lipcon commented on HDFS-3023:
-----------------------------------
bq. By option #2 vs option #3 above I meant component #2 vs component #3 in
HDFS-3010, ie is most of the overhead is due to component #3 (which this patch
does not address).
Ah, I understand now.
Unfortunately the fsync cost is a "fixed penalty" of sorts. But I think the
fsync will cost less when the data is smaller, especially when logging to NFS.
Granted there is a fixed cost for seek, but there isn't much we can do to
optimize that, short of flash or BBU write cache -- so everywhere else we can
shave off some overhead we should take the opportunity.
The good thing about the sync cost is that it might harm the performance of
individual writers, but since syncs are batched, total system throughput
shouldn't go down any more than the individual writer throughput. Will continue
to investigate with further benchmarking.
> Optimize entries in edits log for persistBlocks calls
> -----------------------------------------------------
>
> Key: HDFS-3023
> URL: https://issues.apache.org/jira/browse/HDFS-3023
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node, performance
> Affects Versions: HA branch (HDFS-1623), 0.23.2
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-3023-HDFS-1623.txt
>
>
> One of the performance issues noticed in the HA branch is due to the much
> larger edit logs, now that we are writing OP_ADD transactions to the edit log
> on every block allocation. We can condense these calls down in two ways:
> 1) use variable-length integers for the block list length, size, and genstamp
> (most of these end up fitting in far less than 8 bytes)
> 2) use delta-coding for the genstamp and block size for any blocks after the
> first block (most blocks will be the same size and only slightly higher
> genstamps)
> 3) introduce a new OP_UPDATE_BLOCKS transaction that doesn't re-serialize
> metadata information like lease owner, permissions, etc
> 4) allow OP_UPDATE_BLOCKS to only re-serialize the blocks that have changed
> for a given transaction
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
