[
https://issues.apache.org/jira/browse/MESOS-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938944#comment-14938944
]
Benjamin Mahler edited comment on MESOS-3562 at 9/30/15 9:58 PM:
-----------------------------------------------------------------
This is just chunked encoding per the Transfer-Encoding header, your library
should de-chunk for you:
The first chunk has size 0x6c == 108 bytes of data.
The first chunk in this case contains a single record of 104 bytes from our
recordio format: "104\n...". This chunk has length 104 + strlen("104\n") == 108.
We send you data in recordio format. That data may or may not be transfer
encode with chunked encoding. Ideally we set the Content-Type to express that
it is recordio wrapped, rather than just saying json, curious to see how other
streaming APIs set content type.
was (Author: bmahler):
This is just chunked encoding per the Transfer-Encoding header, your library
should de-chunk for you:
The first chunk has size 0x6c == 108 bytes of data.
The first chunk in this case contains a single record from our recordio format:
"104\n..." which is 104 + strlen("104\n") == 108.
We send you data in recordio format. That data may or may not be transfer
encode with chunked encoding. Ideally we set the Content-Type to express that
it is recordio wrapped, rather than just saying json, curious to see how other
streaming APIs set content type.
> Anomalous bytes in stream from HTTPI Api
> ----------------------------------------
>
> Key: MESOS-3562
> URL: https://issues.apache.org/jira/browse/MESOS-3562
> Project: Mesos
> Issue Type: Bug
> Components: HTTP API
> Affects Versions: 0.24.0
> Environment: Linux 3.16.7-24-desktop #1 SMP PREEMPT Mon Aug 3
> 14:37:06 UTC 2015 (ec183cc) x86_64 x86_64 GNU/Linux
> Mesos 0.24.0
> gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch revision 212064]
> Reporter: Ben Whitehead
> Priority: Blocker
> Labels: http, wireprotocol
> Attachments: app.log, tcpdump.log
>
>
> When connecting to the new HTTP Api and attempting to {{SUBSCRIBE}} there are
> some anomalous bytes contained in the chunked stream that appear to be
> causing problems when I attempting to integrate.
> Attached are two log files. app.log represents my application trying to
> connect to mesos using RxNetty. Netty has been configured to log all data it
> sends/receives over the wire this can be seen in the byte blocks in the log.
> The client is constructing a protobuf in java for the subscribe call
> {code:java}
> final Call subscribeCall = Call.newBuilder()
> .setType(Call.Type.SUBSCRIBE)
> .setSubscribe(
> Call.Subscribe.newBuilder()
> .setFrameworkInfo(
> Protos.FrameworkInfo.newBuilder()
> .setUser("bill")
> .setName("testing_this_shit_out")
> .build()
> )
> )
> .build();
> {code}
>
> lient sends the protobuf to mesos with the following request headers:
> {code}
> POST /api/v1/scheduler HTTP/1.1
> Content-Type: application/x-protobuf
> Accept: application/json
> Content-Length: 35
> Host: localhost:5050
> User-Agent: RxNetty Client
> {code}
> The body is then serialized via protobuf and sent.
> The response from the mesos master has the following headers:
> {code}
> HTTP/1.1 200 OK
> Transfer-Encoding: chunked
> Date: Wed, 30 Sep 2015 21:07:16 GMT
> Content-Type: application/json
> {code}
> followed by
> {code}
> \r\n\r\n6c\r\n104\n{"subscribed":{"framework_id":{"value":"20150930-103028-16777343-5050-11742-0028"}},"type":"SUBSCRIBED"}
> {code}
> The {{\r\n\r\n}} is expected for standard http bodies, how ever {{6c\r\n}}
> doesn't appear to be attached to anything. {{104}} is the correct length of
> the Subscribe events JSON.
> What is this extra number and why is it there?
> This is not the first time confusion has come up related to the wire format
> for the event stream from the new http api see
> [this|http://mail-archives.apache.org/mod_mbox/mesos-user/201508.mbox/%3c94d2c9e8-2fe8-4c11-b0d3-859dac654...@me.com%3E]
> message from the mailing list.
> In the [Design
> Doc|https://docs.google.com/document/d/1pnIY_HckimKNvpqhKRhbc9eSItWNFT-priXh_urR-T0/edit#]
> there is a statement that said
> {quote}
> All subsequent events that are relevant to this framework generated by Mesos
> are streamed on this connection. Master encodes each Event in RecordIO
> format, i.e., string representation of length of the event followed by JSON
> or binary Protobuf (possibly compressed) encoded event.
> {quote}
> There is no specification I've been able to find online that actually
> explains this format. The only reference I can find to it is some sample go
> code.
> The attached tcpdump.log contains a tcp dump between the mesos master and my
> client collected using the following command {{tcpdump -xx -n -i lo "dst port
> 5050" or "src port 5050" 2>&1 | tee /tmp/tcpdump.log}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
