[jira] [Commented] (MESOS-3067) Implement a streaming response decoder for events stream
[
https://issues.apache.org/jira/browse/MESOS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306785#comment-15306785
]
Anand Mazumdar commented on MESOS-3067:
---
+1 to what Vinod said.
Also, there have been other suggested alternatives like using
{{Content-Encoding}} and setting it to {{RecordIO}}. I don't quite like that
idea since {{RecordIO}} is an _application_ level detail and not how the
content was actually encoded over the wire as {{Content-Encoding}} seems to
suggest.
Since this JIRA is already resolved, I would encourage having this discussion
on MESOS-3601 that caters explicitly to formalizing such headers.
> Implement a streaming response decoder for events stream
>
>
> Key: MESOS-3067
> URL: https://issues.apache.org/jira/browse/MESOS-3067
> Project: Mesos
> Issue Type: Task
>Reporter: Anand Mazumdar
>Assignee: Benjamin Mahler
> Fix For: 0.24.0
>
>
> We need a streaming response decoder to de-serialize chunks sent from the
> master on the events stream.
> From the HTTP API design doc:
> Master encodes each Event in RecordIO format, i.e. a string representation of
> length of the event in bytes followed by JSON or binary Protobuf (possibly
> compressed) encoded event.
> As of now for getting the basic features right , this is being done in the
> test-cases:
> {code}
> auto reader = response.get().reader;
> ASSERT_SOME(reader);
> Future eventFuture = reader.get().read();
> AWAIT_READY(eventFuture);
> Event event;
> event.ParseFromString(eventFuture.get());
> {code}
> Two things need to happen:
> - We need master to emit events in RecordIO format i.e. event size followed
> by the serialized event instead of just the serialized events as is the case
> now.
> - The decoder class should then abstract away the logic of reading the
> response and de-serializing events from the stream.
> Ideally, the decoder should work with both "json" and "protobuf" responses.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3067) Implement a streaming response decoder for events stream
[
https://issues.apache.org/jira/browse/MESOS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306291#comment-15306291
]
Vinod Kone commented on MESOS-3067:
---
We chose RecordIO format because it can be applied consistently to both JSON
and serialized protobufs. As for the content type, agree that application/json
is incorrect but that is what Twitter streaming API and other APIs I have
looked around that does RecordIO streaming call it :( Maybe we should call it
application/x-json-recordio?
> Implement a streaming response decoder for events stream
>
>
> Key: MESOS-3067
> URL: https://issues.apache.org/jira/browse/MESOS-3067
> Project: Mesos
> Issue Type: Task
>Reporter: Anand Mazumdar
>Assignee: Benjamin Mahler
> Fix For: 0.24.0
>
>
> We need a streaming response decoder to de-serialize chunks sent from the
> master on the events stream.
> From the HTTP API design doc:
> Master encodes each Event in RecordIO format, i.e. a string representation of
> length of the event in bytes followed by JSON or binary Protobuf (possibly
> compressed) encoded event.
> As of now for getting the basic features right , this is being done in the
> test-cases:
> {code}
> auto reader = response.get().reader;
> ASSERT_SOME(reader);
> Future eventFuture = reader.get().read();
> AWAIT_READY(eventFuture);
> Event event;
> event.ParseFromString(eventFuture.get());
> {code}
> Two things need to happen:
> - We need master to emit events in RecordIO format i.e. event size followed
> by the serialized event instead of just the serialized events as is the case
> now.
> - The decoder class should then abstract away the logic of reading the
> response and de-serializing events from the stream.
> Ideally, the decoder should work with both "json" and "protobuf" responses.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3067) Implement a streaming response decoder for events stream
[
https://issues.apache.org/jira/browse/MESOS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306123#comment-15306123
]
David Rauschenbach commented on MESOS-3067:
---
Have you considered application/jsonstream encoding (line-separated JSON) for
the chunked response? That's easy to integrate with present Java and Node libs.
These lengths and JSON blocks aren't consistent with the Content-Type:
application/json header being returned by the interface.
> Implement a streaming response decoder for events stream
>
>
> Key: MESOS-3067
> URL: https://issues.apache.org/jira/browse/MESOS-3067
> Project: Mesos
> Issue Type: Task
>Reporter: Anand Mazumdar
>Assignee: Benjamin Mahler
> Fix For: 0.24.0
>
>
> We need a streaming response decoder to de-serialize chunks sent from the
> master on the events stream.
> From the HTTP API design doc:
> Master encodes each Event in RecordIO format, i.e. a string representation of
> length of the event in bytes followed by JSON or binary Protobuf (possibly
> compressed) encoded event.
> As of now for getting the basic features right , this is being done in the
> test-cases:
> {code}
> auto reader = response.get().reader;
> ASSERT_SOME(reader);
> Future eventFuture = reader.get().read();
> AWAIT_READY(eventFuture);
> Event event;
> event.ParseFromString(eventFuture.get());
> {code}
> Two things need to happen:
> - We need master to emit events in RecordIO format i.e. event size followed
> by the serialized event instead of just the serialized events as is the case
> now.
> - The decoder class should then abstract away the logic of reading the
> response and de-serializing events from the stream.
> Ideally, the decoder should work with both "json" and "protobuf" responses.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
