[
https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988664#comment-12988664
]
Matt Foley commented on HADOOP-6949:
------------------------------------
Doug, thanks for bringing up this important use case. After looking over the
code, it appears there is a single line in WritableRpcEngine where all RPC code
calls ObjectWritable.writeObject(). If this one call is modified to allow the
more compact array format, we get the benefit for all RPC protocols based on
WritableRpcEngine, without having to change any of the RPC APIs, and with no
danger to the other callers of ObjectWritable.writeObject(). Any other callers
of ObjectWritable.writeObject() may choose to use the new call, but for safety
sake we default to the old format.
Please take a look at this new patch. Besides the changes Todd recommended, I
broke ObjectWritable.writeObject() into a stub with the old API signature, and
a slightly modified implementation with an additional boolean argument
controlling whether compact format is used for arrays of primitives. If false,
the behavior is as before; if true the behavior is the new compact format.
Callers of the old stub always get the old ("false") behavior. And finally I
changed the one line in WritableRpcEngine to call the new API with the boolean
argument "true".
I also re-confirmed that this change gives the improvement in Block Report
processing RPC overhead time.
> Reduces RPC packet size for primitive arrays, especially long[], which is
> used at block reporting
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6949
> URL: https://issues.apache.org/jira/browse/HADOOP-6949
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Reporter: Navis
> Assignee: Matt Foley
> Fix For: 0.23.0
>
> Attachments: ObjectWritable.diff, arrayprim.patch
>
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array
> types as general object array ; array type string + array length + (element
> type string + value)*n
> It would not be needed to specify each element types for primitive arrays.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
