> On April 25, 2014, 6:23 a.m., Benjamin Hindman wrote:
> > This is great Tom, thanks for contributing!
> >
> > You commented on IRC that the format of MESOS_EXECUTOR_URIS is indeed a bit
> > bizarre. It's an artifact of not having JSON/protobuf parsing when this
> > code was originally written. Now that we _can_ parse JSON/protobuf,
> > however, we've been meaning to clean this up. That is, we should just send
> > JSON in the environment (and to simplify we might as well send the entire
> > CommandInfo rather than pulling out the URIs). This would boil down to just
> > the following in the containerizer:
> >
> > environment["MESOS_COMMAND_INFO"] = stringify(JSON::Protobuf(commandInfo));
> >
> >
> > And something along the lines of the following in the fetcher:
> >
> > Try<JSON::Object> parse =
> > JSON::parse<JSON::Object>(os::getenv("MESOS_COMMAND_INFO"));
> >
> > if (parse.isError()) {
> > ...
> > }
> >
> > Try<CommandInfo> commandInfo = protobuf::parse<CommandInfo>(parse.get());
> >
> > if (commandInfo.isError()) {
> > ...
> > }
> >
> > foreach (const CommandInfo::URI& uri, commandInfo.get().uris()) {
> > ...
> > }
> >
> >
> > I'm happy to accept your patch in it's current state and have you (or
> > someone else) follow up with a subsequent patch for doing the JSON/protobuf
> > parsing too.
>
> Tom Arnfeld wrote:
> That's fair enough, I was thinking about ways to improve this structure
> but wanted to keep it out of this fix. Perhaps this is something that could
> be included in the other two mesos issues about improving fetcher/executor
> caching.
Okie dokie, I added https://issues.apache.org/jira/browse/MESOS-1248 to make
sure we track this going forward.
- Benjamin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20674/#review41439
-----------------------------------------------------------
On April 25, 2014, 6:28 a.m., Tom Arnfeld wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20674/
> -----------------------------------------------------------
>
> (Updated April 25, 2014, 6:28 a.m.)
>
>
> Review request for mesos, Benjamin Hindman, Bernd Mathiske, Ian Downes,
> Niklas Nielsen, and Vinod Kone.
>
>
> Bugs: MESOS-1241
> https://issues.apache.org/jira/browse/MESOS-1241
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> This is a patch that includes a new `extract` boolean value on
> CommandInfo.URI to toggle the auto-extraction of compressed archives when
> shipping URIs with an executor or task.
>
> It's worth noting that this changes the behaviour of the MESOS_EXECUTOR_URI
> environment variable used. I've added some comments which outlines the
> expected format, instead of (e.g) hdfs://foobar/testing.tar.gz+0 (1
> signifying it's not executable) there is now an extra flag on the end (X or
> N) defining whether to auto-extract or not, resulting in
> hdfs://foobar/testing.tar.gz+0X by default.
>
>
> Diffs
> -----
>
> include/mesos/mesos.proto d7fe068
> src/launcher/fetcher.cpp 98ebc2e
> src/slave/containerizer/mesos_containerizer.cpp f5df979
> src/tests/containerizer_tests.cpp 6c48146
>
> Diff: https://reviews.apache.org/r/20674/diff/
>
>
> Testing
> -------
>
> - Tested with a single-slave mesos cluster (Mac OSX 10.9)
> - Verified `make check` after adding extra unit tests
>
>
> Thanks,
>
> Tom Arnfeld
>
>
