Re: protbuf to json not compatible

[James Peach]

> On Mar 23, 2017, at 7:58 PM, James Peach  wrote:
> 
>> 
>> On Mar 23, 2017, at 1:54 AM, Olivier Sallou  wrote:
>> 
>> Hi,
>> 
>> when transforming a protobug message to json with MessageToJson, the
>> json is not compatible with the json format expected by Mesos master.
> 
> This is because you generated the protobuf bindings with proto3 compiler. 
> AFAICT they made an incompatible change to the JSON wire format. This bites 
> you when using the jsonpb Go package, for example. I ended up post-processing 
> the generated Go code to correct the field names.

Sorry I forgot to mention that the other workaround is to generate the protobuf 
bindings with the proto2 compiler.

J

Re: protbuf to json not compatible

[James Peach]

> On Mar 23, 2017, at 1:54 AM, Olivier Sallou  wrote:
> 
> Hi,
> 
> when transforming a protobug message to json with MessageToJson, the
> json is not compatible with the json format expected by Mesos master.

This is because you generated the protobuf bindings with proto3 compiler. 
AFAICT they made an incompatible change to the JSON wire format. This bites you 
when using the jsonpb Go package, for example. I ended up post-processing the 
generated Go code to correct the field names.

> 
> For example, for volumes it generates
> 
> 
> volumes: [
> 
>{'hostPath': '',
> 
>  'containerPath': '...',
> 
> ...
> 
>   }
> 
> ]
> 
> 
> but HTTP API expects "source" and "container_path"
> 
> is it an expected behavior ? This prevents from "creating" a task in
> protobuf format and sending it to HTTP API with a protobug to json
> conversion.

It’s expected from a proto3 compiler. IMHO this is a breaking change and they 
should have made a fallback option, but there’s not one. There’s no good 
choices here AFAICT.

J

Re: protbuf to json not compatible

[Benjamin Mahler]

+anand, vinod

On Thu, Mar 23, 2017 at 2:09 AM, Tomek Janiszewski 
wrote:

> I have a similar problem with protobuf and json. In my case numbers were
> incorrectly unmarshaled. Here is an issue
> https://issues.apache.org/jira/browse/MESOS-970 and review
> https://reviews.apache.org/r/50851/
>
> czw., 23.03.2017, 09:54 użytkownik Olivier Sallou  >
> napisał:
>
> > Hi,
> >
> > when transforming a protobug message to json with MessageToJson, the
> > json is not compatible with the json format expected by Mesos master.
> >
> > For example, for volumes it generates
> >
> >
> > volumes: [
> >
> > {'hostPath': '',
> >
> >   'containerPath': '...',
> >
> >  ...
> >
> >}
> >
> > ]
> >
> >
> > but HTTP API expects "source" and "container_path"
> >
> > is it an expected behavior ? This prevents from "creating" a task in
> > protobuf format and sending it to HTTP API with a protobug to json
> > conversion.
> >
> > Thanks
> >
> > Olivier
> >
> > --
> > Olivier Sallou
> > IRISA / University of Rennes 1
> > Campus de Beaulieu, 35000 RENNES - FRANCE
> > Tel: 02.99.84.71.95
> >
> > gpg key id: 4096R/326D8438  (keyring.debian.org)
> > Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
> >
> >
>

Re: protbuf to json not compatible

[Benjamin Mahler]

+gilbert

On Thu, Mar 23, 2017 at 1:54 AM, Olivier Sallou 
wrote:

> Hi,
>
> when transforming a protobug message to json with MessageToJson, the
> json is not compatible with the json format expected by Mesos master.
>
> For example, for volumes it generates
>
>
> volumes: [
>
> {'hostPath': '',
>
>   'containerPath': '...',
>
>  ...
>
>}
>
> ]
>
>
> but HTTP API expects "source" and "container_path"
>
> is it an expected behavior ? This prevents from "creating" a task in
> protobuf format and sending it to HTTP API with a protobug to json
> conversion.
>
> Thanks
>
> Olivier
>
> --
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95
>
> gpg key id: 4096R/326D8438  (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
>
>

Re: [Design Doc] Improve Storage Support in Mesos using Resources Provider

[Jie Yu]

Yes, the idea is to make this general in the future. In fact, the whole
resource provider design keeps that in mind.

We could add a general "CONVERT" operation in the future with a free formed
key value pairs (as well as the source resources) as the parameters. And
it's up to the corresponding resource provider to interpret that.

- Jie

On Thu, Mar 23, 2017 at 3:50 AM, Sargun Dhillon  wrote:

> Is the intent to make this generic beyond disks? I can see the
> concepts apply beyond volumes, and blocks. Perhaps a generic
> Create{generation} -- where larger generation numbers descend from
> smaller ones?
>
> I can also see this valuable in networking. My use case is ENIs in
> AWS. I would like to have a ResourceProvider that can manipulate ENIs
> based on the invocation of the scheduler. Instead of "CREATE_BLOCK"
> it'd be CREATE_INTERFACE, with some given options about the ENI,
> giving us a raw interface. Subsequently, we would want to do a
> CREATE_IPVLAN, as a subinterface that we can assign an actual IP to.
> The IPVLAN interface is a descendant of the raw interface, just as
> volumes are descendants of block devices.
>
>
> On Sun, Mar 12, 2017 at 6:47 PM, Jie Yu  wrote:
> > Hi,
> >
> > Currently, Mesos supports both local persistent volumes as well as
> external
> > persistent volumes. However, both of them are not ideal.
> >
> > Local persistent volumes do not support offering physical or logical
> block
> > devices directly. Also, frameworks do not have choices to select
> > filesystems for their local persistent volumes. There are also some
> > usability problem with the local persistent volumes. Mesos does support
> > multiple local disks. However, it’s a big burden for operators to
> configure
> > each agent properly to be able to leverage this feature.
> >
> > External persistent volumes support in Mesos currently bypasses the
> > resource management part. In other words, using an external persistent
> > volume does not go through the usual offer cycle. Mesos doesn’t track
> > resources associated with the external volumes. This makes quota control,
> > reservation, fair sharing almost impossible to implement. Also, the
> current
> > interface Mesos uses to interact with volume providers is the Docker
> Volume
> > Driver interface (DVDI), which is very specific to operations on a
> > particular agent.
> >
> > The main problem I see currently is that we don’t have a coherent story
> for
> > storage. Yes, we have some primitives in Mesos that can support some
> > stateful services, but this is far from ideal. Some of them are just the
> > stop gap solution (e.g., the external volume support). This design tries
> to
> > tell a coherent story for supporting storage in Mesos.
> >
> > https://docs.google.com/document/d/125YWqg_
> 5BB5OY9a6M7LZcby5RSqBwo2PZzpVLuxYXh4/edit?usp=sharing
> >
> > Please feel free to reply this thread or comment on the doc if you have
> any
> > comments or suggestions! Thanks!
> >
> > - Jie
>

Re: Mesos Developer Community Meeting (Mar 23)

[Michael Park]

Alright, I still see no agenda items. We'll cancel this time around as
well. Please reach out to me if you have things you would like to present!
It can be anything from a feature announcement/walk-through, short
tech-talk, a little lesson learned, a bug you'd like to share, etc.

Thanks,

MPark

On Wed, Mar 22, 2017 at 5:29 PM, Michael Park  wrote:

> We don't have a tech talk scheduled for this week, and there are no
> agenda items on the meeting doc
> 
> .
> I'll check back tonight to see if there are any agenda items that folks
> would like to discuss, but if there's nothing on the agenda, I'll cancel
> the meeting for tomorrow as well.
>
> Thanks,
>
> MPark
>

Re: CNI: Mesos containers need to access Mesos agent

[tommy xiao]

+1

2017-03-21 7:16 GMT+08:00 Marcus Sorensen :

>
> Thanks. Good to see it is already being tracked.
>
> On 2017-03-20 14:39 (-0600), Avinash Sridharan 
> wrote:
> > Hi Marcus,
> >  The reason we need connectivity from the container's network namespace
> to
> > the host network namespace is that the Mesos executor running in the
> > container's network namespace needs to register back with the agent in
> > order to send TASK updates about the container to the agent. Without this
> > connectivity the agent will not know if the container has started
> > successfully and will simply kill the container, failing the container
> > launch.
> >
> > I know this is a restriction on some virtual networking solutions, and
> > going forward the right solution would be to support agent/executor
> > communication over domain sockets:
> > https://issues.apache.org/jira/browse/MESOS-6240
> >
> > We still need to figure out when that can be accomplished.
> >
> > In terms of the work arounds, if you can open communication to port 5051
> > between the host network namespace and the container's network namespace
> it
> > should just work.
> >
> > On Mon, Mar 20, 2017 at 9:50 AM, Marcus Sorensen 
> > wrote:
> >
> > > http://mesos.apache.org/documentation/latest/cni/
> > >
> > > "For Mesos, the executors launched as containers need to register with
> the
> > > Agent in order for a task to be successfully launched. Hence, it is
> > > imperative that the Agent IP is reachable from the container IP and
> vice
> > > versa. "
> > >
> > > Can anyone shed some light on this requirement for me?  We'd like to
> > > understand the purpose of this to determine if we can work around it or
> > > find some means of securing it.  We are really focusing on network
> security
> > > and isolation in our CNI design, we'd prefer to maintain network
> isolation
> > > between the Mesos containers and hosts.
> > >
> > > In particular, if we have to work around it, I'm wondering if there'd
> be
> > > any opportunity for the CNI plugin to open access to the port for just
> a
> > > short period until registration, then firewall it off and what the
> behavior
> > > might be if there is not continual access. Or perhaps we add a link
> local
> > > interface of some sort and a route, such that individual containers can
> > > reach their agent but the Mesos container networks don't need to be
> > > generally open to the Mesos host networks.
> > >
> >
> >
> >
> > --
> > Avinash Sridharan, Mesosphere
> > +1 (323) 702 5245 <(323)%20702-5245>
> >
>



-- 
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

Re: protbuf to json not compatible

[Tomek Janiszewski]

I have a similar problem with protobuf and json. In my case numbers were
incorrectly unmarshaled. Here is an issue
https://issues.apache.org/jira/browse/MESOS-970 and review
https://reviews.apache.org/r/50851/

czw., 23.03.2017, 09:54 użytkownik Olivier Sallou 
napisał:

> Hi,
>
> when transforming a protobug message to json with MessageToJson, the
> json is not compatible with the json format expected by Mesos master.
>
> For example, for volumes it generates
>
>
> volumes: [
>
> {'hostPath': '',
>
>   'containerPath': '...',
>
>  ...
>
>}
>
> ]
>
>
> but HTTP API expects "source" and "container_path"
>
> is it an expected behavior ? This prevents from "creating" a task in
> protobuf format and sending it to HTTP API with a protobug to json
> conversion.
>
> Thanks
>
> Olivier
>
> --
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95
>
> gpg key id: 4096R/326D8438  (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
>
>

protbuf to json not compatible

[Olivier Sallou]

Hi,

when transforming a protobug message to json with MessageToJson, the
json is not compatible with the json format expected by Mesos master.

For example, for volumes it generates


volumes: [

{'hostPath': '',

  'containerPath': '...',

 ...

   }

]


but HTTP API expects "source" and "container_path"

is it an expected behavior ? This prevents from "creating" a task in
protobuf format and sending it to HTTP API with a protobug to json
conversion.

Thanks

Olivier

-- 
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95

gpg key id: 4096R/326D8438  (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438

Re: [Design Doc] Improve Storage Support in Mesos using Resources Provider

[Sargun Dhillon]

Is the intent to make this generic beyond disks? I can see the
concepts apply beyond volumes, and blocks. Perhaps a generic
Create{generation} -- where larger generation numbers descend from
smaller ones?

I can also see this valuable in networking. My use case is ENIs in
AWS. I would like to have a ResourceProvider that can manipulate ENIs
based on the invocation of the scheduler. Instead of "CREATE_BLOCK"
it'd be CREATE_INTERFACE, with some given options about the ENI,
giving us a raw interface. Subsequently, we would want to do a
CREATE_IPVLAN, as a subinterface that we can assign an actual IP to.
The IPVLAN interface is a descendant of the raw interface, just as
volumes are descendants of block devices.


On Sun, Mar 12, 2017 at 6:47 PM, Jie Yu  wrote:
> Hi,
>
> Currently, Mesos supports both local persistent volumes as well as external
> persistent volumes. However, both of them are not ideal.
>
> Local persistent volumes do not support offering physical or logical block
> devices directly. Also, frameworks do not have choices to select
> filesystems for their local persistent volumes. There are also some
> usability problem with the local persistent volumes. Mesos does support
> multiple local disks. However, it’s a big burden for operators to configure
> each agent properly to be able to leverage this feature.
>
> External persistent volumes support in Mesos currently bypasses the
> resource management part. In other words, using an external persistent
> volume does not go through the usual offer cycle. Mesos doesn’t track
> resources associated with the external volumes. This makes quota control,
> reservation, fair sharing almost impossible to implement. Also, the current
> interface Mesos uses to interact with volume providers is the Docker Volume
> Driver interface (DVDI), which is very specific to operations on a
> particular agent.
>
> The main problem I see currently is that we don’t have a coherent story for
> storage. Yes, we have some primitives in Mesos that can support some
> stateful services, but this is far from ideal. Some of them are just the
> stop gap solution (e.g., the external volume support). This design tries to
> tell a coherent story for supporting storage in Mesos.
>
> https://docs.google.com/document/d/125YWqg_5BB5OY9a6M7LZcby5RSqBwo2PZzpVLuxYXh4/edit?usp=sharing
>
> Please feel free to reply this thread or comment on the doc if you have any
> comments or suggestions! Thanks!
>
> - Jie