[
https://issues.apache.org/jira/browse/HDFS-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981404#action_12981404
]
Konstantin Shvachko commented on HDFS-1583:
-------------------------------------------
Liyin, This is a nice optimization, and thanks for measuring the performance.
I think this is critical for 0.22 release. Writing edits to BN should not be
slower than writing to disk.
> byte array with 8000 elements will grow up to 56008 elements after
> serialization
This makes RPC very inefficient.
Java arrays can not hold different type instances (see ArrayStoreException for
refs), so serializing type name for each element does not make sense. Type
should be stored only once for the entire array.
Does anybody remember discussions or jiras open for it?
We need to decide if it should be fixed in RPC or locally for BackupNode only.
Seems that RPC level fix would optimize communication in general, but will be
massively backward incompatible. It could be a good time to do that now before
the major release.
> Improve backup-node sync performance by wrapping RPC parameters
> ---------------------------------------------------------------
>
> Key: HDFS-1583
> URL: https://issues.apache.org/jira/browse/HDFS-1583
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Liyin Liang
> Fix For: 0.23.0
>
> Attachments: HDFS-1583-1.patch, HDFS-1583-2.patch
>
>
> The journal edit records are sent by the active name-node to the backup-node
> with RPC:
> {code:}
> public void journal(NamenodeRegistration registration,
> int jAction,
> int length,
> byte[] records) throws IOException;
> {code}
> During the name-node throughput benchmark, the size of byte array _records_
> is around *8000*. Then the serialization and deserialization is
> time-consuming. I wrote a simple application to test RPC with byte array
> parameter. When the size got to 8000, each RPC call need about 6 ms. While
> name-node sync 8k byte to local disk only need 0.3~0.4ms.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
