[
https://issues.apache.org/jira/browse/HDFS-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220009#comment-13220009
]
Hudson commented on HDFS-3023:
------------------------------
Integrated in Hadoop-Hdfs-HAbranch-build #93 (See
[https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/93/])
HDFS-3023. Optimize entries in edits log for persistBlocks call.
Contributed by Todd Lipcon. (Revision 1295356)
Result = UNSTABLE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295356
Files :
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-1623.txt
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/EditsElement.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/EditsLoaderCurrent.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
> Optimize entries in edits log for persistBlocks calls
> -----------------------------------------------------
>
> Key: HDFS-3023
> URL: https://issues.apache.org/jira/browse/HDFS-3023
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node, performance
> Affects Versions: HA branch (HDFS-1623), 0.23.2
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: HA branch (HDFS-1623)
>
> Attachments: hdfs-3023-HDFS-1623.txt, hdfs-3023.txt
>
>
> One of the performance issues noticed in the HA branch is due to the much
> larger edit logs, now that we are writing OP_ADD transactions to the edit log
> on every block allocation. We can condense these calls down in two ways:
> 1) use variable-length integers for the block list length, size, and genstamp
> (most of these end up fitting in far less than 8 bytes)
> 2) use delta-coding for the genstamp and block size for any blocks after the
> first block (most blocks will be the same size and only slightly higher
> genstamps)
> 3) introduce a new OP_UPDATE_BLOCKS transaction that doesn't re-serialize
> metadata information like lease owner, permissions, etc
> 4) allow OP_UPDATE_BLOCKS to only re-serialize the blocks that have changed
> for a given transaction
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
