[
https://issues.apache.org/jira/browse/HTRACE-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527096#comment-14527096
]
Colin Patrick McCabe commented on HTRACE-164:
---------------------------------------------
The issue is that most Linux distros don't ship {{libprotobuf.a}}, only
{{libprotobuf.so}} (shared library version). Sure you could require people to
install it, or have a "third-party" build directory like Impala does, but this
is extra complexity and things to get wrong for someone trying to build the
software or contribute to it. Linux distros also look unfavorably on bundling
so if we ever wanted to get into a yum or apt-get repo, this would be a
negative. And GRPC.io pulls in a lot more than just libproto, as you know. So
we'd have to make our build process a lot more complex and possibly start
downloading or bundling 3rd party software as part of it.
Believe me, I am the master of doing static builds, and in general horrible
things with native builds. But like karate, a real master tries never to use
these skills.
msgpack maps extremely well to JSON, because it has the same set of datatypes.
This was one of the design goals of msgpack-- to be isomorphic with JSON. The
Golang code for encoding things in msgpack is literally just {{enc :=
codec.NewEncoder(w, mh); err := enc.Encode(msg);}}. Decoding is similar. It
looks at the structures and encodes / decodes from there rather than requiring
separate structures definitions and boilerplate. So we automatically keep
everything in sync between the msgpack and JSON format for spans in htraced.
If you look at the patch, msgpack allows doing something very clever here. We
can write the trace spans to a buffer, one after another, in msgpack format,
just as they come in. Then, later on, we can write a header, also in msgpack
format, to another buffer. Then we can use writev to write a complete message
that looks like this:
{code}
part 1: [map DefaultPid -> "myDefaultPid", Spans ->]
part 2: spans buffer
{code}
This avoids doing a copy or a costly "big bang" serialization. But at the same
time, messages are sent fully structured rather than ad-hoc. This is possible
because an array of spans in msgpack is just
{{\[array-tag\]\[array-count\]\[repeated array element\]}}.
I want to emphasize again that this code is in htrace-htraced, not in common.
We are not adding any new stuff to htrace-common. We can always swap this out
later if we find something awesome to replace it with.
> htrace hrpc: use msgpack for serialization
> ------------------------------------------
>
> Key: HTRACE-164
> URL: https://issues.apache.org/jira/browse/HTRACE-164
> Project: HTrace
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HTRACE-164.002.patch
>
>
> htrace HRPC should use msgpack for serialization. Messages serialized using
> msgpack use less space on the wire and use less CPU time to encode. The CMP
> library allows us to include msgpack support easily in the htrace C client.
> There is also good Java and Golang support available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
