In that case, I rescind my objection. Memory use as a problem and labels as an alternative work fine. Thanks! On Mon, Nov 2, 2015 at 7:03 PM Benjamin Mahler <benjamin.mah...@gmail.com> wrote:
> Sorry for the confusion, the motivation to remove 'data' is for memory > scalability reasons (the ability to express binary fields is orthogonal and > is not the reason to remove 'data'). > > We can get into a really bad state in large clusters if frameworks are > putting non-trivial amounts of 'data' in TaskInfos and ExecutorInfos. If > it's too large for the master to hold in memory, the master will > continually OOM and it becomes impossible to right your cluster. See > https://issues.apache.org/jira/browse/MESOS-1746 for some history of > stripping binary data, starting with TaskStatus. > > Labels were introduced to aid tooling, can you use labels? I realize they > are not in ExecutorInfo yet. > > On Mon, Nov 2, 2015 at 6:23 PM, David Greenberg <dsg123456...@gmail.com> > wrote: > > > Why not base64 encode the field? We use that field in our frameworks, and > > some of our platform tools would benefit from being able to read that > data. > > Base64 seems like a compromise with minimal complexity addition. It also > > removes the potential for parse errors, doesn't rule out future > > applications from using the data stored there (as specialized frameworks > > use that field), and doesn't incur a message size overhead in the (I > > presume) majority of frameworks not using that field. > > On Mon, Nov 2, 2015 at 4:28 PM Guangya Liu <gyliu...@gmail.com> wrote: > > > > > +1 to remove the field directly, one comment is that the upgrade > document > > > may need to be updated. > > > > > > From my understanding, since the data is binary data and I did not see > > too > > > much requirement on retrieving binary data. > > > > > > Thanks! > > > > > > On Sat, Oct 24, 2015 at 5:33 AM, Joseph Wu <jos...@mesosphere.io> > wrote: > > > > > > > Hello, > > > > > > > > The state endpoints, on master and agent, currently serialize two > > binary > > > > data fields in the ExecutorInfo and TaskInfo objects. These fields > are > > > set > > > > by frameworks; and Mesos does not inspect their values. > > > > > > > > The data fields can be found in the state JSON blobs: > > > > /master/state -> frameworks[*].executors[*].data > > > > /slave/state -> > > > > > > > > > > > > > > frameworks[*].(executors|completed_executors)[*].(tasks|queued_tasks|completed_tasks)[*].data > > > > > > > > *Problem:* > > > > The state endpoints are JSON-ified in a non-standard way (i.e. not > via > > > our > > > > normal Protobuf-to-json methods). When we serialize the binary > "data" > > > > fields, the binary is dumped as a string, as is. The resulting JSON > > may > > > > not be valid if the binary data includes random bytes (i.e. not > > unicode). > > > > Most JSON parsers will error on the state endpoints in this case. > > > > > > > > *Proposed solution *(and breaking change)*:* > > > > Simple -- remove the "data" fields from the state endpoints. (And > only > > > > from the state endpoints. The ExecutorInfo and TaskInfo objects will > > not > > > > change.) > > > > > > > > *Question:* > > > > We believe that frameworks/tools do not rely on retrieving the "data" > > > > fields from the state endpoints. > > > > > > > > Is there any framework/tool that retrieves the "data" field from the > > > state > > > > endpoints? > > > > And if so, is it critical to how the framework/tool works? > > > > > > > > More details here: https://issues.apache.org/jira/browse/MESOS-3771 > > > > > > > > Thanks, > > > > ~Joseph > > > > > > > > > >
