[
https://issues.apache.org/jira/browse/MESOS-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ben Whitehead updated MESOS-3562:
---------------------------------
Description:
When connecting to the new HTTP Api and attempting to {{SUBSCRIBE}} there are
some anomalous bytes contained in the chunked stream that appear to be causing
problems when I attempting to integrate.
Attached are two log files. app.log represents my application trying to connect
to mesos using RxNetty. Netty has been configured to log all data it
sends/receives over the wire this can be seen in the byte blocks in the log.
The client is constructing a protobuf in java for the subscribe call
{code:java}
final Call subscribeCall = Call.newBuilder()
.setType(Call.Type.SUBSCRIBE)
.setSubscribe(
Call.Subscribe.newBuilder()
.setFrameworkInfo(
Protos.FrameworkInfo.newBuilder()
.setUser("bill")
.setName("testing_this_shit_out")
.build()
)
)
.build();
{code}
lient sends the protobuf to mesos with the following request headers:
{code}
POST /api/v1/scheduler HTTP/1.1
Content-Type: application/x-protobuf
Accept: application/json
Content-Length: 35
Host: localhost:5050
User-Agent: RxNetty Client
{code}
The body is then serialized via protobuf and sent.
The response from the mesos master has the following headers:
{code}
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Wed, 30 Sep 2015 21:07:16 GMT
Content-Type: application/json
{code}
followed by
{code}
\r\n\r\n6c\r\n104\n{"subscribed":{"framework_id":{"value":"20150930-103028-16777343-5050-11742-0028"}},"type":"SUBSCRIBED"}
{code}
The {{\r\n\r\n}} is expected for standard http bodies, how ever {{6c\r\n}}
doesn't appear to be attached to anything. {{104}} is the correct length of the
Subscribe events JSON.
What is this extra number and why is it there?
This is not the first time confusion has come up related to the wire format for
the event stream from the new http api see
[this|http://mail-archives.apache.org/mod_mbox/mesos-user/201508.mbox/%[email protected]%3E]
message from the mailing list.
In the [Design
Doc|https://docs.google.com/document/d/1pnIY_HckimKNvpqhKRhbc9eSItWNFT-priXh_urR-T0/edit#]
there is a statement that said
{quote}
All subsequent events that are relevant to this framework generated by Mesos
are streamed on this connection. Master encodes each Event in RecordIO format,
i.e., string representation of length of the event followed by JSON or binary
Protobuf (possibly compressed) encoded event.
{quote}
There is no specification I've been able to find online that actually explains
this format. The only reference I can find to it is some sample go code.
The attached tcpdump.log contains a tcp dump between the mesos master and my
client collected using the following command {{tcpdump -xx -n -i lo "dst port
5050" or "src port 5050" 2>&1 | tee /tmp/tcpdump.log}}
was:
When connecting to the new HTTP Api and attempting to {{SUBSCRIBE}} there are
some anomalous bytes contained in the chunked stream that appear to be causing
problems when I attempting to integrate.
Attached are two log files. app.log represents my application trying to connect
to mesos using RxNetty. Netty has been configured to log all data it
sends/receives over the wire this can be seen in the byte blocks in the log.
tcpdump.log contains a tcp dump between the mesos master and my client.
The client is constructing a protobuf in java for the subscribe call
{code:java}
final Call subscribeCall = Call.newBuilder()
.setType(Call.Type.SUBSCRIBE)
.setSubscribe(
Call.Subscribe.newBuilder()
.setFrameworkInfo(
Protos.FrameworkInfo.newBuilder()
.setUser("bill")
.setName("testing_this_shit_out")
.build()
)
)
.build();
{code}
lient sends the protobuf to mesos with the following request headers:
{code}
POST /api/v1/scheduler HTTP/1.1
Content-Type: application/x-protobuf
Accept: application/json
Content-Length: 35
Host: localhost:5050
User-Agent: RxNetty Client
{code}
The body is then serialized via protobuf and sent.
The response from the mesos master has the following headers:
{code}
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Wed, 30 Sep 2015 21:07:16 GMT
Content-Type: application/json
{code}
followed by
{code}
\r\n\r\n6c\r\n104\n{"subscribed":{"framework_id":{"value":"20150930-103028-16777343-5050-11742-0028"}},"type":"SUBSCRIBED"}
{code}
The {{\r\n\r\n}} is expected for standard http bodies, how ever {{6c\r\n}}
doesn't appear to be attached to anything. {{104}} is the correct length of the
Subscribe events JSON.
What is this extra number and why is it there?
This is not the first time confusion has come up related to the wire format for
the event stream from the new http api see
[this|http://mail-archives.apache.org/mod_mbox/mesos-user/201508.mbox/%[email protected]%3E]
message from the mailing list.
In the [Design
Doc|https://docs.google.com/document/d/1pnIY_HckimKNvpqhKRhbc9eSItWNFT-priXh_urR-T0/edit#]
there is a statement that said
{quote}
All subsequent events that are relevant to this framework generated by Mesos
are streamed on this connection. Master encodes each Event in RecordIO format,
i.e., string representation of length of the event followed by JSON or binary
Protobuf (possibly compressed) encoded event.
{quote}
There is no specification I've been able to find online that actually explains
this format. The only reference I can find to it is some sample go code.
> Anomalous bytes in stream from HTTPI Api
> ----------------------------------------
>
> Key: MESOS-3562
> URL: https://issues.apache.org/jira/browse/MESOS-3562
> Project: Mesos
> Issue Type: Bug
> Components: HTTP API
> Affects Versions: 0.24.0
> Environment: Linux 3.16.7-24-desktop #1 SMP PREEMPT Mon Aug 3
> 14:37:06 UTC 2015 (ec183cc) x86_64 x86_64 GNU/Linux
> Mesos 0.24.0
> gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch revision 212064]
> Reporter: Ben Whitehead
> Priority: Blocker
> Labels: http, wireprotocol
> Attachments: app.log, tcpdump.log
>
>
> When connecting to the new HTTP Api and attempting to {{SUBSCRIBE}} there are
> some anomalous bytes contained in the chunked stream that appear to be
> causing problems when I attempting to integrate.
> Attached are two log files. app.log represents my application trying to
> connect to mesos using RxNetty. Netty has been configured to log all data it
> sends/receives over the wire this can be seen in the byte blocks in the log.
> The client is constructing a protobuf in java for the subscribe call
> {code:java}
> final Call subscribeCall = Call.newBuilder()
> .setType(Call.Type.SUBSCRIBE)
> .setSubscribe(
> Call.Subscribe.newBuilder()
> .setFrameworkInfo(
> Protos.FrameworkInfo.newBuilder()
> .setUser("bill")
> .setName("testing_this_shit_out")
> .build()
> )
> )
> .build();
> {code}
>
> lient sends the protobuf to mesos with the following request headers:
> {code}
> POST /api/v1/scheduler HTTP/1.1
> Content-Type: application/x-protobuf
> Accept: application/json
> Content-Length: 35
> Host: localhost:5050
> User-Agent: RxNetty Client
> {code}
> The body is then serialized via protobuf and sent.
> The response from the mesos master has the following headers:
> {code}
> HTTP/1.1 200 OK
> Transfer-Encoding: chunked
> Date: Wed, 30 Sep 2015 21:07:16 GMT
> Content-Type: application/json
> {code}
> followed by
> {code}
> \r\n\r\n6c\r\n104\n{"subscribed":{"framework_id":{"value":"20150930-103028-16777343-5050-11742-0028"}},"type":"SUBSCRIBED"}
> {code}
> The {{\r\n\r\n}} is expected for standard http bodies, how ever {{6c\r\n}}
> doesn't appear to be attached to anything. {{104}} is the correct length of
> the Subscribe events JSON.
> What is this extra number and why is it there?
> This is not the first time confusion has come up related to the wire format
> for the event stream from the new http api see
> [this|http://mail-archives.apache.org/mod_mbox/mesos-user/201508.mbox/%[email protected]%3E]
> message from the mailing list.
> In the [Design
> Doc|https://docs.google.com/document/d/1pnIY_HckimKNvpqhKRhbc9eSItWNFT-priXh_urR-T0/edit#]
> there is a statement that said
> {quote}
> All subsequent events that are relevant to this framework generated by Mesos
> are streamed on this connection. Master encodes each Event in RecordIO
> format, i.e., string representation of length of the event followed by JSON
> or binary Protobuf (possibly compressed) encoded event.
> {quote}
> There is no specification I've been able to find online that actually
> explains this format. The only reference I can find to it is some sample go
> code.
> The attached tcpdump.log contains a tcp dump between the mesos master and my
> client collected using the following command {{tcpdump -xx -n -i lo "dst port
> 5050" or "src port 5050" 2>&1 | tee /tmp/tcpdump.log}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
