[
https://issues.apache.org/jira/browse/HTRACE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HTRACE-308:
----------------------------------------
Attachment: HTRACE-308.004.patch
Allocating buffers of size MAX_HRPC_BODY_LENGTH when starting the
minihtracedcluster was slowing down the unit tests. This latest patch makes it
so we only allocate big buffers if we need them. Buffers are still cached so
there will be no need to keep allocating once htraced is up and running.
> Deserialize WriteSpans requests incrementally rather than all at once to
> optimize GC
> ------------------------------------------------------------------------------------
>
> Key: HTRACE-308
> URL: https://issues.apache.org/jira/browse/HTRACE-308
> Project: HTrace
> Issue Type: Improvement
> Components: htraced
> Affects Versions: 4.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HTRACE-308.001.patch, HTRACE-308.002.patch,
> HTRACE-308.003.patch, HTRACE-308.004.patch
>
>
> We should deserialize WriteSpans requests incrementally rather than all at
> once. Currently, we can deserialize 63 MB of spans all at once, which
> immediately creates somewhere between 60k and 600k spans, depending on span
> size. This is hard on the garbage collector because it's a lot of
> allocations all at once, and because it allocates a very large array to hold
> it all.
> It would be better to deserialize spans one at a time and feed them into the
> datastore via the BatchIngestor. This will ensure that we don't have to
> allocate giant arrays of spans all at once. If the datastore lags behind the
> rate of span ingestion, this will avoid us needing to allocate a bunch of
> memory "up front" which can lead to further slowdowns due to GC.
> Also, we should reuse buffers for the RPC handlers, and use buffering while
> deserializing to avoid making lots of small reads from the socket.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
