[
https://issues.apache.org/jira/browse/HTRACE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034631#comment-15034631
]
Hudson commented on HTRACE-308:
-------------------------------
SUCCESS: Integrated in HTrace-Master #222 (See
[https://builds.apache.org/job/HTrace-Master/222/])
HTRACE-308. Deserialize WriteSpans requests incrementally rather than (cmccabe:
rev 35053cfc557c24e94b7336fc24801d8bdfafc973)
* htrace-htraced/go/src/org/apache/htrace/htracedTool/cmd.go
* htrace-htraced/go/Godeps/Godeps.json
* htrace-c/src/receiver/htraced.c
* htrace-htraced/go/src/org/apache/htrace/htraced/client_test.go
* htrace-htraced/src/main/java/org/apache/htrace/impl/RestBufferManager.java
* htrace-htraced/go/src/org/apache/htrace/client/client.go
* htrace-htraced/go/src/org/apache/htrace/htraced/hrpc.go
* htrace-htraced/go/src/org/apache/htrace/client/hclient.go
* htrace-htraced/go/src/org/apache/htrace/common/rpc.go
* htrace-htraced/go/src/org/apache/htrace/htraced/rest.go
* htrace-htraced/src/main/java/org/apache/htrace/impl/PackedBuffer.java
* htrace-htraced/src/main/java/org/apache/htrace/impl/Conf.java
* htrace-htraced/go/src/org/apache/htrace/htraced/metrics_test.go
* htrace-htraced/go/src/org/apache/htrace/htraced/datastore_test.go
> Deserialize WriteSpans requests incrementally rather than all at once to
> optimize GC
> ------------------------------------------------------------------------------------
>
> Key: HTRACE-308
> URL: https://issues.apache.org/jira/browse/HTRACE-308
> Project: HTrace
> Issue Type: Improvement
> Components: htraced
> Affects Versions: 4.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Fix For: 4.1
>
> Attachments: HTRACE-308.001.patch, HTRACE-308.002.patch,
> HTRACE-308.003.patch, HTRACE-308.004.patch
>
>
> We should deserialize WriteSpans requests incrementally rather than all at
> once. Currently, we can deserialize 63 MB of spans all at once, which
> immediately creates somewhere between 60k and 600k spans, depending on span
> size. This is hard on the garbage collector because it's a lot of
> allocations all at once, and because it allocates a very large array to hold
> it all.
> It would be better to deserialize spans one at a time and feed them into the
> datastore via the BatchIngestor. This will ensure that we don't have to
> allocate giant arrays of spans all at once. If the datastore lags behind the
> rate of span ingestion, this will avoid us needing to allocate a bunch of
> memory "up front" which can lead to further slowdowns due to GC.
> Also, we should reuse buffers for the RPC handlers, and use buffering while
> deserializing to avoid making lots of small reads from the socket.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
