Re: [VOTE] Release Apache Mesos 1.9.0 (rc3)

[Gilbert Song]

+1 (binding).

Tested on our internal CI. Green on most of the platforms:
Configuration Matrix Plain SSL CMake Clang BUILD_ISOLATORS
mac
[image: Not run]
[image: Success]

[image: Not run]
[image: Not run]
[image: Not run]
mesos-ec2-centos-6
[image: Not run]
[image: Unstable]

[image: Not run]
[image: Not run]
[image: Not run]
mesos-ec2-centos-7
[image: Success]

[image: Success]

[image: Success]

[image: Not run]
[image: Not run]
mesos-ec2-debian-8
[image: Not run]
[image: Unstable]

[image: Not run]
[image: Not run]
[image: Not run]
mesos-ec2-debian-9
[image: Not run]
[image: Success]

[image: Not run]
[image: Not run]
[image: Not run]
mesos-ec2-ubuntu-14.04
[image: Not run]
[image: Success]

[image: Not run]
[image: Not run]
[image: Not run]
mesos-ec2-ubuntu-16.04
[image: Success]

[image: Success]

[image: Success]

[image: Success]

[image: Success]


The only two failed tests are known flakiness.

   - mesos-ec2-centos-6-SSL.Mesos.ExecutorAuthorizationTest.FailedSubscribe
   - mesos-ec2-debian-8-SSL.MESOS_TESTS_ABORTED.xml.[empty]

-Gilbert

On Tue, Sep 3, 2019 at 10:43 AM Vinod Kone  wrote:

> +1 (binding)
>
> Tested on ASF CI.
>
> *Revision*: 5e79a584e6ec3e9e2f96e8bf418411df9dafac2e
>
>- refs/tags/1.9.0-rc3
>
> Configuration Matrix gcc clang
> centos:7 --verbose --disable-libtool-wrappers
> --disable-parallel-test-execution --enable-libevent --enable-ssl autotools
> [image: Success]
> 
> [image: Not run]
> cmake
> [image: Success]
> 
> [image: Not run]
> --verbose --disable-libtool-wrappers --disable-parallel-test-execution
> autotools
> [image: Success]
> 
> [image: Not run]
> cmake
> [image: Success]
> 
> [image: Not run]
> ubuntu:16.04 --verbose --disable-libtool-wrappers
> --disable-parallel-test-execution --enable-libevent --enable-ssl autotools
> [image: Success]
> 

[Containerization WG] Meeting canceled today

[Gilbert Song]

Hi all,

I am in travel and we do not have any agenda for today's meeting. Let's
sync up next time.

Sorry for the short notice. Feel free to add your topic to the agenda

.

-Gilbert

Re: Subject: [VOTE] Release Apache Mesos 1.8.0 (rc1)

[Gilbert Song]

-1 for rc1.

I would like to have MESOS-9704
 (backported to 1.8.x
branch) in for 1.8.0, otherwise it could be a regression for some conner
cases.

-Gilbert

On Tue, Apr 16, 2019 at 8:40 AM Benno Evers  wrote:

> You're right, I forgot to make a 'CLI' subheading when I added the
> CLI-related highlights. I added that now that on both branches.
>
> The other items seem to be mostly editorial changes, as far as I can tell.
> If you feel anything in particular is missing, maybe you could just send me
> a list and we can make sure to have it included in the release blogpost and
> the master CHANGELOG?
>
> On Mon, Apr 15, 2019 at 11:37 PM Benjamin Mahler 
> wrote:
>
>> The CHANGELOG highlights seem a bit lacking?
>>
>> - For some reason, the task CLI command is listed in a performance
>> section?
>> - The parallel endpoint serving changes are in the longer list of items,
>> seems like we highlight them in the performance section? Maybe we could be
>> specific too about what we did additionally vs 1.7.0, since we already
>> announced parallel state reads in 1.7.0?
>> - We also have some additional allocator related performance improvements
>> in 1.8.0 that we need to add.
>> - Do we want to say something w.r.t.
>> https://issues.apache.org/jira/browse/MESOS-9211?
>>
>> Not sure to what degree we care about an accurate CHANGELOG in the 1.8.0
>> tag vs updating the 1.8.0 CHANGELOG on master?
>>
>> On Mon, Apr 15, 2019 at 2:26 PM Benno Evers 
>> wrote:
>>
>> > Hi all,
>> >
>> > Please vote on releasing the following candidate as Apache Mesos 1.8.0.
>> >
>> >
>> > 1.8.0 includes the following:
>> >
>> >
>> 
>> >  * Operation feedback for v1 schedulers.
>> >  * Per-framework minimum allocatable resources.
>> >  * New CLI subcommands `task attach` and `task exec`.
>> >  * New `linux/seccomp` isolator.
>> >  * Support for Docker v2 Schema2 manifest format.
>> >  * XFS quota for persistent volumes.
>> >  * **Experimental** Support for the new CSI v1 API.
>> >
>> > The CHANGELOG for the release is available at:
>> >
>> >
>> https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.8.0-rc1
>> >
>> >
>> 
>> >
>> > The candidate for Mesos 1.8.0 release is available at:
>> >
>> https://dist.apache.org/repos/dist/dev/mesos/1.8.0-rc1/mesos-1.8.0.tar.gz
>> >
>> > The tag to be voted on is 1.8.0-rc1:
>> > https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.8.0-rc1
>> >
>> > The SHA512 checksum of the tarball can be found at:
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/mesos/1.8.0-rc1/mesos-1.8.0.tar.gz.sha512
>> >
>> > The signature of the tarball can be found at:
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/mesos/1.8.0-rc1/mesos-1.8.0.tar.gz.asc
>> >
>> > The PGP key used to sign the release is here:
>> > https://dist.apache.org/repos/dist/release/mesos/KEYS
>> >
>> > The JAR is in a staging repository here:
>> > https://repository.apache.org/content/repositories/orgapachemesos-1251
>> >
>> > Please vote on releasing this package as Apache Mesos 1.8.0!
>> >
>> > The vote is open until Thursday, April 18 and passes if a majority of at
>> > least 3 +1 PMC votes are cast.
>> >
>> > [ ] +1 Release this package as Apache Mesos 1.8.0
>> > [ ] -1 Do not release this package because ...
>> >
>> > Thanks,
>> > Benno and Joseph
>> >
>>
>
>
> --
> Benno Evers
> Software Engineer, Mesosphere
>

Re: [MESOS-8248] - Expose information about GPU assigned to a task

[Gilbert Song]

Thanks for the feedback, BenM!

Jorge, could you mind addressing BenM's comment above and put the proposal
to a google doc?

We could discuss this proposal in next Containerization WG meeting on April
4th (please add an agenda and link your proposal):
https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit#heading=h.978qjujkxfvu

-Gilbert

On Fri, Mar 22, 2019 at 12:19 PM Benjamin Mahler 
wrote:

> Containers can be assigned multiple GPUs, so I assume you're thinking of
> putting these metrics in a repeated message? (similar to DiskStatistics)
>
> It has seemed to me we should probably make this Nvidia specific (e.g.
> NvidiaGPUStatistics). In the past we thought generalizing this would be
> good, but there's only Nvidia support at the moment and we haven't been
> able to make sure that other GPU libraries provide the same information.
>
> For each metric can you also include the relevant calls from NVML for
> obtaining the information? Can you also highlight what cadvisor provides to
> make sure we don't miss anything? From my read of their code, it seems to
> be a subset of what you listed?
>
> https://github.com/google/cadvisor/blob/e310755a36728b457fcc1de6b54bb4c6cb38f031/accelerators/nvidia.go#L216-L246
>
> On Fri, Mar 22, 2019 at 6:58 AM Jorge Machado 
> wrote:
>
> > another way would be to just use cadvisor
> >
> > > On 22 Mar 2019, at 08:35, Jorge Machado  wrote:
> > >
> > > Hi Mesos devs,
> > >
> > > In our use case from mesos we need to get gpu resource usage per task
> > and build dashboards on grafana for it.  Getting the metrics to Grafana
> we
> > will send the metrics to prometheus the main problem is how to get the
> > metrics in a reliable way.
> > > I proposing the following:
> > >
> > > Changing the mesos.proto and mesos.proto under v1 and on
> > ResourceStatistics message add:
> > >
> > > //GPU statistics for each container
> > > optional int32 gpu_idx = 50;
> > > optional string gpu_uuid = 51;
> > > optional string device_name = 52;
> > > optional uint64 gpu_memory_used_mb = 53;
> > > optional uint64 gpu_memory_total_mb = 54;
> > > optional double gpu_usage = 55;
> > > optional int32 gpu_temperature = 56;
> > > optional int32 gpu_frequency_MHz = 57;
> > > optional int32 gpu_power_used_W = 58;
> > >
> > > For starters I would like to change NvidiaGpuIsolatorProcess at
> > isolator.cpp and there get the nvml call for the usage method. As I’m new
> > to this I need some guidelines please.
> > >
> > > My questions:
> > >
> > > Does the NvidiaGpuIsolatorProcess runs already inside the container or
> > just outside in the agent ? (I’m assuming outside)
> > > From what I saw on the cpu metrics they are gathered inside the
> > container for the gpu we could do it in the NvidiaGpuIsolatorProcess and
> > get the metrics via the host.
> > > Anything more that I should check ?
> > >
> > > Thanks a lot
> > >
> > > Jorge Machado
> > > www.jmachado.me
> > >
> > >
> > >
> > >
> > >
> >
> >
>

[Containerization WG] Meeting Canceled Today

[Gilbert Song]

Hi all,

We do not have any agenda item today. Let's cancel today's meeting.

-Gilbert

[RESULT][VOTE] Release Apache Mesos 1.5.3 (rc1)

[Gilbert Song]

 Hi all,

The vote for Mesos 1.5.3 (rc1) has passed with the
following votes.

+1 (Binding)
--
Vinod Kone
Greg Mann
Gilbert Song
Chun-Hung Hsiao

+1 (Non-binding)
--
Meng Zhu

There were no 0 or -1 votes.

Please find the release at:
https://dist.apache.org/repos/dist/release/mesos/1.5.3

It is recommended to use a mirror to download the release:
http://www.apache.org/dyn/closer.cgi

The CHANGELOG for the release is available at:
https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.5.3

The mesos-1.5.3.jar has been released to:
https://repository.apache.org

The website (http://mesos.apache.org) will be updated shortly to reflect
this release.

Thanks,
Gilbert

Re: [VOTE] Release Apache Mesos 1.5.3 (rc1)

[Gilbert Song]

clang,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >
> > cmake
> > [image: Success]
> > <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/67/BUILDTOOL=cmake,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >
> > [image: Failed]
> > <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/67/BUILDTOOL=cmake,COMPILER=clang,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >
> >
> > On Wed, Mar 6, 2019 at 7:33 AM Gilbert Song  wrote:
> >
> >>  Hi all,
> >>
> >> Please vote on releasing the following candidate as Apache Mesos 1.5.3.
> >>
> >> 1.5.3 includes the following:
> >>
> >>
> 
> >> ** Bug
> >>   * [MESOS-7474] - Mesos fetcher cache doesn't retry when missed.
> >>   * [MESOS-8887] - Unreachable tasks are not GC'ed when unreachable
> agent
> >> is GC'ed.
> >>   * [MESOS-8907] - Docker image fetcher fails with HTTP/2.
> >>   * [MESOS-9210] - Mesos v1 scheduler library does not properly handle
> >> SUBSCRIBE retries
> >>   * [MESOS-9317] - Some master endpoints do not handle failed
> >> authorization
> >> properly.
> >>   * [MESOS-9332] - Nested container should run as the same user of its
> >> parent container by default.
> >>   * [MESOS-9334] - Container stuck at ISOLATING state due to libevent
> poll
> >> never returns.
> >>   * [MESOS-9362] - Test
> >> `CgroupsIsolatorTest.ROOT_CGROUPS_CreateRecursively` is flaky.
> >>   * [MESOS-9411] - Validation of JWT tokens using HS256 hashing
> algorithm
> >> is not thread safe.
> >>   * [MESOS-9492] - Persist CNI working directory across reboot.
> >>   * [MESOS-9507] - Agent could not recover due to empty docker volume
> >> checkpointed files.
> >>   * [MESOS-9518] - CNI_NETNS should not be set for orphan containers
> that
> >> do not have network namespace.
> >>   * [MESOS-9532] - ResourceOffersTest.ResourceOfferWithMultipleSlaves is
> >> flaky.
> >>   * [MESOS-9533] - CniIsolatorTest.ROOT_CleanupAfterReboot is flaky.
> >>   * [MESOS-9555] - Allocator CHECK failure:
> >> reservationScalarQuantities.contains(role).
> >>   * [MESOS-9581] - Mesos package naming appears to be undeterministic.
> >>
> >> ** Improvement:
> >>   * [MESOS-9305] - Create cgoup recursively to workaround systemd
> deleting
> >> cgroups_root.
> >>
> >> The CHANGELOG for the release is available at:
> >>
> >>
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.5.3-rc1
> >>
> >>
> 
> >>
> >> The candidate for Mesos 1.5.3 release is available at:
> >>
> https://dist.apache.org/repos/dist/dev/mesos/1.5.3-rc1/mesos-1.5.3.tar.gz
> >>
> >> The tag to be voted on is 1.5.3-rc1:
> >> https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.5.3-rc1
> >>
> >> The SHA512 checksum of the tarball can be found at:
> >>
> >>
> https://dist.apache.org/repos/dist/dev/mesos/1.5.3-rc1/mesos-1.5.3.tar.gz.sha512
> >>
> >> The signature of the tarball can be found at:
> >>
> >>
> https://dist.apache.org/repos/dist/dev/mesos/1.5.3-rc1/mesos-1.5.3.tar.gz.asc
> >>
> >> The PGP key used to sign the release is here:
> >> https://dist.apache.org/repos/dist/release/mesos/KEYS
> >>
> >> The JAR is in a staging repository here:
> >> https://repository.apache.org/content/repositories/orgapachemesos-1250
> >>
> >> Please vote on releasing this package as Apache Mesos 1.5.3!
> >>
> >> The vote is open until Fri Mar  8 22:13:20 PST 2019 and passes if a
> >> majority of at least 3 +1 PMC votes are cast.
> >>
> >> [ ] +1 Release this package as Apache Mesos 1.5.3
> >> [ ] -1 Do not release this package because ...
> >>
> >> Thanks,
> >> Gilbert
> >>
> >
>

Re: task with terminating executor stuck in queued tasks

[Gilbert Song]

Hi Eric,

What is your Mesos Version?

Did you reboot the agent machine before task getting stuck?
If yes, probably https://issues.apache.org/jira/browse/MESOS-9501

Did you enabled health check for that task?
It may increase the chance of a potential FD leak:
https://issues.apache.org/jira/browse/MESOS-9502

-Gilbert


On Fri, Mar 8, 2019 at 3:03 PM Eric Chung  wrote:

> Hello devs,
>
> We recently ran into a situation where a task's executor was killed due to
> registration timeout, but neither the executor nor the task was properly
> killed, and the task has been stuck in queued_tasks for days.
>
> The relevant log:
>
> I0305 08:43:59.069857  5215 slave.cpp:6803] Terminating executor
> '' of framework  because it did not
> register within 15mins
> I0305 09:16:28.266021  5200 slave.cpp:3644] Asked to kill task
>  of framework 
> W0305 09:16:28.266063  5200 slave.cpp:3816] Ignoring kill task
>  because the executor '' of framework
>  is terminating
>
>
> where the following just keeps repeating:
>
> I0305 09:16:28.266021  5200 slave.cpp:3644] Asked to kill task
>  of framework 
> W0305 09:16:28.266063  5200 slave.cpp:3816] Ignoring kill task
>  because the executor '' of framework
>  is terminating
>
>
> the agent state indicates that it doesn't have any active tasks but a quite
> a few queued tasks.
>
> Does anyone have any insight on why this might be happening?
>
> Thanks,
> Eric
>

[Containerization WG] Meeting Summary Today

[Gilbert Song]

Hi,

We have done some backlog bugs grooming today and come up with a list of
critical bugs that we might want to fix by Mesos 1.8.0:
https://jira.apache.org/jira/issues/?filter=12343087

Volunteers are absolutely welcome if you are interested to fix any critical
issues from this list.

Thanks,
Gilbert

[VOTE] Release Apache Mesos 1.5.3 (rc1)

[Gilbert Song]

 Hi all,

Please vote on releasing the following candidate as Apache Mesos 1.5.3.

1.5.3 includes the following:

** Bug
  * [MESOS-7474] - Mesos fetcher cache doesn't retry when missed.
  * [MESOS-8887] - Unreachable tasks are not GC'ed when unreachable agent
is GC'ed.
  * [MESOS-8907] - Docker image fetcher fails with HTTP/2.
  * [MESOS-9210] - Mesos v1 scheduler library does not properly handle
SUBSCRIBE retries
  * [MESOS-9317] - Some master endpoints do not handle failed authorization
properly.
  * [MESOS-9332] - Nested container should run as the same user of its
parent container by default.
  * [MESOS-9334] - Container stuck at ISOLATING state due to libevent poll
never returns.
  * [MESOS-9362] - Test
`CgroupsIsolatorTest.ROOT_CGROUPS_CreateRecursively` is flaky.
  * [MESOS-9411] - Validation of JWT tokens using HS256 hashing algorithm
is not thread safe.
  * [MESOS-9492] - Persist CNI working directory across reboot.
  * [MESOS-9507] - Agent could not recover due to empty docker volume
checkpointed files.
  * [MESOS-9518] - CNI_NETNS should not be set for orphan containers that
do not have network namespace.
  * [MESOS-9532] - ResourceOffersTest.ResourceOfferWithMultipleSlaves is
flaky.
  * [MESOS-9533] - CniIsolatorTest.ROOT_CleanupAfterReboot is flaky.
  * [MESOS-9555] - Allocator CHECK failure:
reservationScalarQuantities.contains(role).
  * [MESOS-9581] - Mesos package naming appears to be undeterministic.

** Improvement:
  * [MESOS-9305] - Create cgoup recursively to workaround systemd deleting
cgroups_root.

The CHANGELOG for the release is available at:
https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.5.3-rc1


The candidate for Mesos 1.5.3 release is available at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.3-rc1/mesos-1.5.3.tar.gz

The tag to be voted on is 1.5.3-rc1:
https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.5.3-rc1

The SHA512 checksum of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.3-rc1/mesos-1.5.3.tar.gz.sha512

The signature of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.3-rc1/mesos-1.5.3.tar.gz.asc

The PGP key used to sign the release is here:
https://dist.apache.org/repos/dist/release/mesos/KEYS

The JAR is in a staging repository here:
https://repository.apache.org/content/repositories/orgapachemesos-1250

Please vote on releasing this package as Apache Mesos 1.5.3!

The vote is open until Fri Mar  8 22:13:20 PST 2019 and passes if a
majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Mesos 1.5.3
[ ] -1 Do not release this package because ...

Thanks,
Gilbert

Re: [VOTE] Release Apache Mesos 1.7.2 (rc1)

[Gilbert Song]

+1 (binding).

On Fri, Feb 22, 2019 at 9:43 AM Meng Zhu  wrote:

> +1, ran through out internal CI, only flaky failures
>
> On Thu, Feb 21, 2019 at 11:41 AM Greg Mann  wrote:
>
> > +1
> >
> > Built on CentOS 7.4 and ran all tests as root. Only 3 test failures were
> > observed, all known flakes.
> >
> > Cheers,
> > Greg
> >
> > On Wed, Feb 20, 2019 at 7:12 AM Vinod Kone  wrote:
> >
> >> +1
> >>
> >> Ran this on ASF CI.
> >>
> >> The red builds are a flaky infra issue and a known flaky test
> >> .
> >>
> >> *Revision*: 58cc918e9acc2865bb07047d3d2dff156d1708b2
> >>
> >>- refs/tags/1.7.2-rc1
> >>
> >> Configuration Matrix gcc clang
> >> centos:7 --verbose --disable-libtool-wrappers
> >> --disable-parallel-test-execution --enable-libevent --enable-ssl
> autotools
> >> [image: Failed]
> >> <
> >>
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/66/BUILDTOOL=autotools,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos%3A7,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >> >
> >> [image: Not run]
> >> cmake
> >> [image: Success]
> >> <
> >>
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/66/BUILDTOOL=cmake,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos%3A7,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >> >
> >> [image: Not run]
> >> --verbose --disable-libtool-wrappers --disable-parallel-test-execution
> >> autotools
> >> [image: Success]
> >> <
> >>
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/66/BUILDTOOL=autotools,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos%3A7,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >> >
> >> [image: Not run]
> >> cmake
> >> [image: Success]
> >> <
> >>
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/66/BUILDTOOL=cmake,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos%3A7,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >> >
> >> [image: Not run]
> >> ubuntu:16.04 --verbose --disable-libtool-wrappers
> >> --disable-parallel-test-execution --enable-libevent --enable-ssl
> autotools
> >> [image: Success]
> >> <
> >>
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/66/BUILDTOOL=autotools,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >> >
> >> [image: Success]
> >> <
> >>
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/66/BUILDTOOL=autotools,COMPILER=clang,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >> >
> >> cmake
> >> [image: Success]
> >> <
> >>
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/66/BUILDTOOL=cmake,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >> >
> >> [image: Success]
> >> <
> >>
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/66/BUILDTOOL=cmake,COMPILER=clang,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >> >
> >> --verbose --disable-libtool-wrappers --disable-parallel-test-execution
> >> autotools
> >> [image: Success]
> >> <
> >>
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/66/BUILDTOOL=autotools,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >> >
> >> [image: Success]
> >> <
> >>
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/66/BUILDTOOL=autotools,COMPILER=clang,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >> >
> >> cmake
> >> [image: Success]
> >> <
> 

Re: [VOTE] Release Apache Mesos 1.4.3 (rc2)

[Gilbert Song]

+1 (binding).

On Thu, Feb 21, 2019 at 2:18 PM Gastón Kleiman  wrote:

> +1
>
> Ran it through Mesosphere's internal CI. Only 7 test failures were
> observed, all known flakes.
>
> On Fri, Feb 15, 2019 at 11:49 AM Vinod Kone  wrote:
>
>> +1 (binding)
>>
>> Tested on ASF CI. Red builds are known flaky tests or unrelated infra
>> issues.
>>
>>
>> *Revision*: 1fee9b5365bf2424e4768dc1d5209c6c78dfece6
>>
>>- refs/tags/1.4.3-rc2
>>
>> Configuration Matrix gcc clang
>> centos:7 --verbose --disable-libtool-wrappers
>> --disable-parallel-test-execution --enable-libevent --enable-ssl autotools
>> [image: Failed]
>> <
>> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/64/BUILDTOOL=autotools,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos%3A7,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
>> >
>> [image: Not run]
>> cmake
>> [image: Failed]
>> <
>> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/64/BUILDTOOL=cmake,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos%3A7,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
>> >
>> [image: Not run]
>> --verbose --disable-libtool-wrappers --disable-parallel-test-execution
>> autotools
>> [image: Failed]
>> <
>> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/64/BUILDTOOL=autotools,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos%3A7,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
>> >
>> [image: Not run]
>> cmake
>> [image: Success]
>> <
>> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/64/BUILDTOOL=cmake,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos%3A7,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
>> >
>> [image: Not run]
>> ubuntu:16.04 --verbose --disable-libtool-wrappers
>> --disable-parallel-test-execution --enable-libevent --enable-ssl autotools
>> [image: Success]
>> <
>> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/64/BUILDTOOL=autotools,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
>> >
>> [image: Success]
>> <
>> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/64/BUILDTOOL=autotools,COMPILER=clang,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
>> >
>> cmake
>> [image: Success]
>> <
>> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/64/BUILDTOOL=cmake,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
>> >
>> [image: Success]
>> <
>> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/64/BUILDTOOL=cmake,COMPILER=clang,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
>> >
>> --verbose --disable-libtool-wrappers --disable-parallel-test-execution
>> autotools
>> [image: Failed]
>> <
>> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/64/BUILDTOOL=autotools,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
>> >
>> [image: Success]
>> <
>> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/64/BUILDTOOL=autotools,COMPILER=clang,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
>> >
>> cmake
>> [image: Success]
>> <
>> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/64/BUILDTOOL=cmake,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
>> >
>> [image: Failed]
>> <
>> 

Re: [VOTE] Release Apache Mesos 1.6.2 (rc1)

[Gilbert Song]

+1 (binding).

On Wed, Feb 20, 2019 at 5:42 PM Meng Zhu  wrote:

> +1 -- ran on centos 7.4 with only known flaky tests
>
> -Meng
>
> On Wed, Feb 20, 2019 at 4:57 PM Gastón Kleiman 
> wrote:
>
>> +1 (binding) — ran the build through Mesosphere's internal CI and only
>> two known flaky tests failed.
>>
>> On Tue, Feb 19, 2019 at 11:56 AM Greg Mann  wrote:
>>
>>> Hi all,
>>>
>>> Please vote on releasing the following candidate as Apache Mesos 1.6.2.
>>>
>>>
>>> 1.6.2 includes a number of bug fixes since 1.6.1; the CHANGELOG for the
>>> release is available at:
>>>
>>> https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.6.2-rc1
>>>
>>> 
>>>
>>> The candidate for Mesos 1.6.2 release is available at:
>>> https://dist.apache.org/repos/dist/dev/mesos/1.6.2-rc1/mesos-1.6.2.tar.gz
>>>
>>> The tag to be voted on is 1.6.2-rc1:
>>> https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.6.2-rc1
>>>
>>> The SHA512 checksum of the tarball can be found at:
>>>
>>> https://dist.apache.org/repos/dist/dev/mesos/1.6.2-rc1/mesos-1.6.2.tar.gz.sha512
>>>
>>> The signature of the tarball can be found at:
>>>
>>> https://dist.apache.org/repos/dist/dev/mesos/1.6.2-rc1/mesos-1.6.2.tar.gz.asc
>>>
>>> The PGP key used to sign the release is here:
>>> https://dist.apache.org/repos/dist/release/mesos/KEYS
>>>
>>> The JAR is in a staging repository here:
>>> https://repository.apache.org/content/repositories/orgapachemesos-1246
>>>
>>> Please vote on releasing this package as Apache Mesos 1.6.2!
>>>
>>> The vote is open until Fri Feb 22 11:54 PST 2019, and passes if a
>>> majority of at least 3 +1 PMC votes are cast.
>>>
>>> [ ] +1 Release this package as Apache Mesos 1.6.2
>>> [ ] -1 Do not release this package because ...
>>>
>>> Thanks,
>>> Greg
>>>
>>

Re: [VOTE] Release Apache Mesos 1.4.3 (rc1)

[Gilbert Song]

-1

Sorry, given the fact that this is very likely to be the last point release
for Mesos 1.4, there are some security improvement patches that are very
useful for users but not included in this RC. I would down vote it for
another cut, which could include more improvements from 1.4.x branch.

-Gilbert

On Tue, Jan 29, 2019 at 6:00 PM Vinod Kone  wrote:

> +1
>
> Tested in ASF CI. Red builds are known flakes.
>
> *Revision*: fcfe1904e45726ca96fc6707d8b227a16664f4f8
>
>- refs/tags/1.4.3-rc1
>
> Configuration Matrix gcc clang
> centos:7 --verbose --disable-libtool-wrappers
> --disable-parallel-test-execution --enable-libevent --enable-ssl autotools
> [image: Failed]
> 
> [image: Not run]
> cmake
> [image: Success]
> 
> [image: Not run]
> --verbose --disable-libtool-wrappers --disable-parallel-test-execution
> autotools
> [image: Success]
> 
> [image: Not run]
> cmake
> [image: Success]
> 
> [image: Not run]
> ubuntu:16.04 --verbose --disable-libtool-wrappers
> --disable-parallel-test-execution --enable-libevent --enable-ssl autotools
> [image: Failed]
> 
> [image: Success]
> 
> cmake
> [image: Success]
> 
> [image: Success]
> 
> --verbose --disable-libtool-wrappers --disable-parallel-test-execution
> autotools
> [image: Failed]
> 
> [image: Failed]
> 
> cmake
> [image: Success]
> 
> [image: Success]
> 

[Containerization WG] Sync Agenda for January 23rd, 2018

[Gilbert Song]

Hi folks,

Tomorrow's WG meeting starts at 9 am PST. Here is the agenda
.
We will try to groom the containerization related bugs. Please add any
topic that you are interested to the agenda.

Join the Zoom Meeting: https://zoom.us/j/224160545

Thanks,
Gilbert

[RESULT][VOTE] Release Apache Mesos 1.5.2 (rc3)

[Gilbert Song]

 Hi all,

The vote for Mesos 1.5.2 (rc3) has passed with the
following votes.

+1 (Binding)
--
Jie Yu
Vinod Kone
Chun-Hung Hsiao
Gilbert Song

There were no 0 or -1 votes.

Please find the release at:
https://dist.apache.org/repos/dist/release/mesos/1.5.2

It is recommended to use a mirror to download the release:
http://www.apache.org/dyn/closer.cgi

The CHANGELOG for the release is available at:
https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.5.2

The mesos-1.5.2.jar has been released to:
https://repository.apache.org

The website (http://mesos.apache.org) will be updated shortly to reflect
this release.

Thanks,
Gilbert

Re: [mesos-mail] Re: [VOTE] Release Apache Mesos 1.5.2 (rc3)

[Gilbert Song]

1%20MESOS_VERBOSE=1,OS=ubuntu%3A14.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >
> > [image: Success]
> > <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/58/BUILDTOOL=autotools,COMPILER=clang,CONFIGURATION=--verbose,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A14.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >
> > cmake
> > [image: Success]
> > <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/58/BUILDTOOL=cmake,COMPILER=gcc,CONFIGURATION=--verbose,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A14.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >
> > [image: Success]
> > <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/58/BUILDTOOL=cmake,COMPILER=clang,CONFIGURATION=--verbose,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A14.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/
> >
> >
> >
> > On Wed, Jan 16, 2019 at 11:04 AM Jie Yu  wrote:
> >
> >> +1
> >>
> >> make dist check on macOS Mojave
> >>
> >> On Tue, Jan 15, 2019 at 12:57 AM Gilbert Song 
> wrote:
> >>
> >>>  Hi all,
> >>>
> >>> Please vote on releasing the following candidate as Apache Mesos 1.5.2.
> >>>
> >>> 1.5.2 includes the following:
> >>>
> >>>
> 
> >>> *Announce major bug fixes here*
> >>> https://jira.apache.org/jira/issues/?filter=12345443
> >>>
> >>> The CHANGELOG for the release is available at:
> >>>
> >>>
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.5.2-rc3
> >>>
> >>>
> 
> >>>
> >>> The candidate for Mesos 1.5.2 release is available at:
> >>>
> https://dist.apache.org/repos/dist/dev/mesos/1.5.2-rc3/mesos-1.5.2.tar.gz
> >>>
> >>> The tag to be voted on is 1.5.2-rc3:
> >>> https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.5.2-rc3
> >>>
> >>> The SHA512 checksum of the tarball can be found at:
> >>>
> >>>
> https://dist.apache.org/repos/dist/dev/mesos/1.5.2-rc3/mesos-1.5.2.tar.gz.sha512
> >>>
> >>> The signature of the tarball can be found at:
> >>>
> >>>
> https://dist.apache.org/repos/dist/dev/mesos/1.5.2-rc3/mesos-1.5.2.tar.gz.asc
> >>>
> >>> The PGP key used to sign the release is here:
> >>> https://dist.apache.org/repos/dist/release/mesos/KEYS
> >>>
> >>> The JAR is in a staging repository here:
> >>> https://repository.apache.org/content/repositories/orgapachemesos-1242
> >>>
> >>> Please vote on releasing this package as Apache Mesos 1.5.2!
> >>>
> >>> The vote is open until Fri Jan 18 00:52:44 PST 2019 and passes if a
> >>> majority of at least 3 +1 PMC votes are cast.
> >>>
> >>> [ ] +1 Release this package as Apache Mesos 1.5.2
> >>> [ ] -1 Do not release this package because ...
> >>>
> >>> Thanks,
> >>> Gilbert
> >>>
> >>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Apache Mesos Mail Lists" group.
> Visit this group at
> https://groups.google.com/a/mesosphere.io/group/mesos-mail/.
> For more options, visit https://groups.google.com/a/mesosphere.io/d/optout
> .
>

Re: [VOTE] Release Apache Mesos 1.7.1 (rc2)

[Gilbert Song]

+1 (binding).

All tests passed except 5 failures (known flakiness) from our internal CI:

FLAG=CMake,label=mesos-ec2-centos-7
 mesos-ec2-centos-7-CMake.Mesos.CgroupsIsolatorTest.ROOT_CGROUPS_CFS_EnableCfs

FLAG=SSL,label=mesos-ec2-centos-7
 mesos-ec2-centos-7-SSL.MESOS_TESTS_ABORTED.xml.[empty]

FLAG=SSL,label=mesos-ec2-debian-9
 
mesos-ec2-debian-9-SSL.Mesos.FetcherCacheTest.CachedCustomOutputFileWithSubdirectory

FLAG=SSL,label=mesos-ec2-ubuntu-16.04
 
mesos-ec2-ubuntu-16.04-SSL.Mesos.CniIsolatorTest.ROOT_INTERNET_CURL_LaunchCommandTask

FLAG=SSL,label=mesos-ec2-centos-6
 
mesos-ec2-centos-6-SSL.Mesos.GarbageCollectorIntegrationTest.LongLivedDefaultExecutorRestart

-Gilbert

On Wed, Jan 16, 2019 at 2:24 PM Vinod Kone  wrote:

> +1 (binding)
>
> Tested on ASF CI. Failing builds are due to missed SSL dep in the docker
> build file and a flaky test.
>
> *Revision*: d5678c3c5500cec72e22e775d9d048c55c128954
>
>- refs/tags/1.7.1-rc2
>
> Configuration Matrix gcc clang
> centos:7 --verbose --disable-libtool-wrappers
> --disable-parallel-test-execution --enable-libevent --enable-ssl autotools
> [image: Success]
> 
> [image: Not run]
> cmake
> [image: Success]
> 
> [image: Not run]
> --verbose --disable-libtool-wrappers --disable-parallel-test-execution
> autotools
> [image: Success]
> 
> [image: Not run]
> cmake
> [image: Success]
> 
> [image: Not run]
> ubuntu:16.04 --verbose --disable-libtool-wrappers
> --disable-parallel-test-execution --enable-libevent --enable-ssl autotools
> [image: Failed]
> 
> [image: Failed]
> 
> cmake
> [image: Failed]
> 
> [image: Failed]
> 
> --verbose --disable-libtool-wrappers --disable-parallel-test-execution
> autotools
> [image: Success]
> 
> [image: Failed]
> 
> 

[VOTE] Release Apache Mesos 1.5.2 (rc3)

[Gilbert Song]

 Hi all,

Please vote on releasing the following candidate as Apache Mesos 1.5.2.

1.5.2 includes the following:

*Announce major bug fixes here*
https://jira.apache.org/jira/issues/?filter=12345443

The CHANGELOG for the release is available at:
https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.5.2-rc3


The candidate for Mesos 1.5.2 release is available at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.2-rc3/mesos-1.5.2.tar.gz

The tag to be voted on is 1.5.2-rc3:
https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.5.2-rc3

The SHA512 checksum of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.2-rc3/mesos-1.5.2.tar.gz.sha512

The signature of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.2-rc3/mesos-1.5.2.tar.gz.asc

The PGP key used to sign the release is here:
https://dist.apache.org/repos/dist/release/mesos/KEYS

The JAR is in a staging repository here:
https://repository.apache.org/content/repositories/orgapachemesos-1242

Please vote on releasing this package as Apache Mesos 1.5.2!

The vote is open until Fri Jan 18 00:52:44 PST 2019 and passes if a
majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Mesos 1.5.2
[ ] -1 Do not release this package because ...

Thanks,
Gilbert

[Containerization WG] Meeting Cancelled Today

[Gilbert Song]

Hi all,

We don't have any agenda item for today's WG sync, so the meeting is
cancelled. Please add any topic you are interested in this doc

and we will discuss it on Jan 24th.

Happy New Year!

Gilbert

[Containerization WG] Sync Agenda for December 13th, 2018

[Gilbert Song]

Hi folks,

Today's WG meeting starts at 9 am PST. Here is the agenda

:

   - [gilbert, andrei]: Seccomp configurable options and default seccomp
   profile discussion.
  -
  
https://docs.google.com/document/d/146FJJ0sDi1sp_HQxVUg-vhqVSTEsdCeD4If3b1xCeec/edit
   - [gilbert, qian]: Volume Ownership potential performance impact for the
   case of Shared Persistent Volume. Cc @jpeach
  -
  
https://docs.google.com/document/d/1QyeDDX4Zr9E-0jKMoPTzsGE-v4KWwjmnCR0l8V4Tq2U/edit

Please join us via Zoom if you are interested: https://zoom.us/j/224160545

Thanks,
Gilbert

[Containerization WG] Meeting Canceled Today

[Gilbert Song]

Hi folks,

We do not have agent item today, so I cancel the meeting today.

Please add any topic that you like to discuss to the sync on Dec 13th:
https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit#heading=h.85119g508pd0

Thanks,
Gilbert

Re: [VOTE] Release Apache Mesos 1.5.2 (rc2)

[Gilbert Song]

Thanks, Chun!

I will cut rc3 sometime early next week.

On Tue, Nov 27, 2018 at 11:13 AM Chun-Hung Hsiao 
wrote:

> -1 for https://issues.apache.org/jira/browse/MESOS-8623
>
> I'm working on a fix.
>
> On Thu, Nov 22, 2018 at 1:40 PM Meng Zhu  wrote:
>
> > +1
> > make check on Ubuntu 18.04
> >
> > On Wed, Oct 31, 2018 at 4:26 PM Gilbert Song 
> > wrote:
> >
> > > Hi all,
> > >
> > > Please vote on releasing the following candidate as Apache Mesos 1.5.2.
> > >
> > > 1.5.2 includes the following:
> > >
> > >
> >
> 
> > > *Announce major bug fixes here*
> > >   * [MESOS-3790] - ZooKeeper connection should retry on `EAI_NONAME`.
> > >   * [MESOS-8128] - Make os::pipe file descriptors O_CLOEXEC.
> > >   * [MESOS-8418] - mesos-agent high cpu usage because of numerous
> > > /proc/mounts reads.
> > >   * [MESOS-8545] -
> > > AgentAPIStreamingTest.AttachInputToNestedContainerSession is flaky.
> > >   * [MESOS-8568] - Command checks should always call
> > > `WAIT_NESTED_CONTAINER` before `REMOVE_NESTED_CONTAINER`.
> > >   * [MESOS-8620] - Containers stuck in FETCHING possibly due to
> > > unresponsive server.
> > >   * [MESOS-8830] - Agent gc on old slave sandboxes could empty
> persistent
> > > volume data.
> > >   * [MESOS-8871] - Agent may fail to recover if the agent dies before
> > > image store cache checkpointed.
> > >   * [MESOS-8904] - Master crash when removing quota.
> > >   * [MESOS-8906] - `UriDiskProfileAdaptor` fails to update profile
> > > selectors.
> > >   * [MESOS-8907] - Docker image fetcher fails with HTTP/2.
> > >   * [MESOS-8917] - Agent leaking file descriptors into forked
> processes.
> > >   * [MESOS-8921] - Autotools don't work with newer OpenJDK versions.
> > >   * [MESOS-8935] - Quota limit "chopping" can lead to cpu-only and
> > > memory-only offers.
> > >   * [MESOS-8936] - Implement a Random Sorter for offer allocations.
> > >   * [MESOS-8942] - Master streaming API does not send (health) check
> > > updates for tasks.
> > >   * [MESOS-8945] - Master check failure due to CHECK_SOME(providerId).
> > >   * [MESOS-8947] - Improve the container preparing logging in
> > > IOSwitchboard and volume/secret isolator.
> > >   * [MESOS-8952] - process::await/collect n^2 performance issue.
> > >   * [MESOS-8963] - Executor crash trying to print container ID.
> > >   * [MESOS-8978] - Command executor calling setsid breaks the tty
> > support.
> > >   * [MESOS-8980] - mesos-slave can deadlock with docker pull.
> > >   * [MESOS-8986] - `slave.available()` in the allocator is expensive
> and
> > > drags down allocation performance.
> > >   * [MESOS-8987] - Master asks agent to shutdown upon auth errors.
> > >   * [MESOS-9024] - Mesos master segfaults with stack overflow under
> load.
> > >   * [MESOS-9049] - Agent GC could unmount a dangling persistent volume
> > > multiple times.
> > >   * [MESOS-9116] - Launch nested container session fails due to
> incorrect
> > > detection of `mnt` namespace of command executor's task.
> > >   * [MESOS-9125] - Port mapper CNI plugin might fail with "Resource
> > > temporarily unavailable".
> > >   * [MESOS-9127] - Port mapper CNI plugin might deadlock iptables on
> the
> > > agent.
> > >   * [MESOS-9131] - Health checks launching nested containers while a
> > > container is being destroyed lead to unkillable tasks.
> > >   * [MESOS-9142] - CNI detach might fail due to missing network config
> > > file.
> > >   * [MESOS-9144] - Master authentication handling leads to request
> > > amplification.
> > >   * [MESOS-9145] - Master has a fragile burned-in 5s authentication
> > > timeout.
> > >   * [MESOS-9146] - Agent has a fragile burn-in 5s authentication
> timeout.
> > >   * [MESOS-9147] - Agent and scheduler driver authentication retry
> > backoff
> > > time could overflow.
> > >   * [MESOS-9151] - Container stuck at ISOLATING due to FD leak.
> > >   * [MESOS-9170] - Zookeeper doesn't compile with newer gcc due to
> format
> > > error.
> > >   * [MESOS-9196] - Removing rootfs mounts may fail with EBUSY.
> > >   * [MESOS-9231] - `docker inspect` may return an unexpected result to
> > > Docker executor due to a race condition.
> > >   *

Re: Propose to run debug container as the same user of its parent container by default

[Gilbert Song]

@James Peach  agree, for debug containers, the default
user should inherit from parent, while CLI toolings (e.g., task exec)
should provide an option `--root` (by setting the commandinfo user as root).

@Qian Zhang  @Benjamin Mahler  ,
if we step back, it seems to me we should extend the user inheritance for
all nested container (instead of just for debug container). Now the default
user for nested container is from the executor (see this patch
),
which does not make sense for 3rd level nested containers or further.

I would suggest that any type of nested container (debug container, check
container, nested container etc.), its user should just inherit from its
parent's user. This would not change the behavior of default executor,
potentially change behaviors for custom executor which support 3 level or
up nested.

- Gilbert

On Thu, Oct 25, 2018 at 9:51 AM Vinod Kone  wrote:

> Sounds good to me.
>
> If I understand correctly, you want to treat this is a bug and backport it
> to previous release branches? So, you are also asking whether backporting
> this bug will be considered a breaking change for any existing users?
>
> On Thu, Oct 25, 2018 at 11:46 AM James Peach  wrote:
>
>>
>>
>> On Oct 23, 2018, at 7:47 PM, Qian Zhang  wrote:
>>
>> Hi all,
>>
>> Currently when launching a debug container (e.g., via `dcos task exec` or
>> command health check) to debug a task, by default Mesos agent will use the
>> executor's user as the debug container's user. There are actually 2 cases:
>> 1. Command task: Since the command executor's user is same with command
>> task's user, so the debug container will be launched as the same user of
>> the command task.
>> 2. The task in a task group: The default executor's user is same with the
>> framework user, so in this case the debug container will be launched as the
>> same user of the framework rather than the task.
>>
>> Basically I think the behavior of case 1 is correct. For case 2, we may
>> run into a situation that the task is run as a user (e.g., root), but the
>> debug container used to debug that task is run as another user (e.g., a
>> normal user, suppose framework is run as a normal user), this may not be
>> what user expects.
>>
>> So I created MESOS-9332
>>  and propose to run
>> debug container as the same user of its parent container (i.e., the task to
>> be debugged) by default. Please let me know if you have any comments,
>> thanks!
>>
>>
>> This sounds like a sensible default to me. I can imagine for debug use
>> cases you might want to run the debug container as root or give it elevated
>> capabilities, but that should not be the default.
>>
>> J
>>
>

Re: Welcome Meng Zhu as PMC member and committer!

[Gilbert Song]

Well deserved, Meng!

On Wed, Oct 31, 2018 at 2:36 PM Benjamin Mahler  wrote:

> Please join me in welcoming Meng Zhu as a PMC member and committer!
>
> Meng has been active in the project for almost a year and has been very
> productive and collaborative. He is now one of the few people of
> understands the allocator code well, as well as the roadmap for this area
> of the project. He has also found and fixed bugs, and helped users in slack.
>
> Thanks for all your work so far Meng, I'm looking forward to more of your
> contributions in the project.
>
> Ben
>

[VOTE] Release Apache Mesos 1.5.2 (rc2)

[Gilbert Song]

 Hi all,

Please vote on releasing the following candidate as Apache Mesos 1.5.2.

1.5.2 includes the following:

*Announce major bug fixes here*
  * [MESOS-3790] - ZooKeeper connection should retry on `EAI_NONAME`.
  * [MESOS-8128] - Make os::pipe file descriptors O_CLOEXEC.
  * [MESOS-8418] - mesos-agent high cpu usage because of numerous
/proc/mounts reads.
  * [MESOS-8545] -
AgentAPIStreamingTest.AttachInputToNestedContainerSession is flaky.
  * [MESOS-8568] - Command checks should always call
`WAIT_NESTED_CONTAINER` before `REMOVE_NESTED_CONTAINER`.
  * [MESOS-8620] - Containers stuck in FETCHING possibly due to
unresponsive server.
  * [MESOS-8830] - Agent gc on old slave sandboxes could empty persistent
volume data.
  * [MESOS-8871] - Agent may fail to recover if the agent dies before image
store cache checkpointed.
  * [MESOS-8904] - Master crash when removing quota.
  * [MESOS-8906] - `UriDiskProfileAdaptor` fails to update profile
selectors.
  * [MESOS-8907] - Docker image fetcher fails with HTTP/2.
  * [MESOS-8917] - Agent leaking file descriptors into forked processes.
  * [MESOS-8921] - Autotools don't work with newer OpenJDK versions.
  * [MESOS-8935] - Quota limit "chopping" can lead to cpu-only and
memory-only offers.
  * [MESOS-8936] - Implement a Random Sorter for offer allocations.
  * [MESOS-8942] - Master streaming API does not send (health) check
updates for tasks.
  * [MESOS-8945] - Master check failure due to CHECK_SOME(providerId).
  * [MESOS-8947] - Improve the container preparing logging in IOSwitchboard
and volume/secret isolator.
  * [MESOS-8952] - process::await/collect n^2 performance issue.
  * [MESOS-8963] - Executor crash trying to print container ID.
  * [MESOS-8978] - Command executor calling setsid breaks the tty support.
  * [MESOS-8980] - mesos-slave can deadlock with docker pull.
  * [MESOS-8986] - `slave.available()` in the allocator is expensive and
drags down allocation performance.
  * [MESOS-8987] - Master asks agent to shutdown upon auth errors.
  * [MESOS-9024] - Mesos master segfaults with stack overflow under load.
  * [MESOS-9049] - Agent GC could unmount a dangling persistent volume
multiple times.
  * [MESOS-9116] - Launch nested container session fails due to incorrect
detection of `mnt` namespace of command executor's task.
  * [MESOS-9125] - Port mapper CNI plugin might fail with "Resource
temporarily unavailable".
  * [MESOS-9127] - Port mapper CNI plugin might deadlock iptables on the
agent.
  * [MESOS-9131] - Health checks launching nested containers while a
container is being destroyed lead to unkillable tasks.
  * [MESOS-9142] - CNI detach might fail due to missing network config file.
  * [MESOS-9144] - Master authentication handling leads to request
amplification.
  * [MESOS-9145] - Master has a fragile burned-in 5s authentication timeout.
  * [MESOS-9146] - Agent has a fragile burn-in 5s authentication timeout.
  * [MESOS-9147] - Agent and scheduler driver authentication retry backoff
time could overflow.
  * [MESOS-9151] - Container stuck at ISOLATING due to FD leak.
  * [MESOS-9170] - Zookeeper doesn't compile with newer gcc due to format
error.
  * [MESOS-9196] - Removing rootfs mounts may fail with EBUSY.
  * [MESOS-9231] - `docker inspect` may return an unexpected result to
Docker executor due to a race condition.
  * [MESOS-9267] - Mesos agent crashes when CNI network is not configured
but used.
  * [MESOS-9279] - Docker Containerizer 'usage' call might be expensive if
mount table is big.
  * [MESOS-9283] - Docker containerizer actor can get backlogged with large
number of containers.
  * [MESOS-9305] - Create cgoup recursively to workaround systemd deleting
cgroups_root.
  * [MESOS-9308] - URI disk profile adaptor could deadlock.
  * [MESOS-9334] - Container stuck at ISOLATING state due to libevent poll
never returns.

The CHANGELOG for the release is available at:
https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.5.2-rc2


The candidate for Mesos 1.5.2 release is available at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.2-rc2/mesos-1.5.2.tar.gz

The tag to be voted on is 1.5.2-rc2:
https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.5.2-rc2

The SHA512 checksum of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.2-rc2/mesos-1.5.2.tar.gz.sha512

The signature of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.2-rc2/mesos-1.5.2.tar.gz.asc

The PGP key used to sign the release is here:
https://dist.apache.org/repos/dist/release/mesos/KEYS

The JAR is in a staging repository here:
https://repository.apache.org/content/repositories/orgapachemesos-1239

Please vote on releasing this package as Apache Mesos 1.5.2!

The vote is open until Mon Nov  5 16:23:11 PDT 2018 and passes 

Re: [VOTE] Release Apache Mesos 1.5.2 (rc1)

[Gilbert Song]

Ok, this issue has been fixed and backported. I will cut rc2 later today.

On Sat, Oct 27, 2018 at 9:53 PM Jie Yu  wrote:

> Gilbert, can we fix this and call another vote?
>
> Thanks,
> - Jie
>
> On Wed, Oct 24, 2018 at 12:45 PM Greg Mann  wrote:
>
>> Hmm I wonder if this is an issue on 1.5.1, or perhaps introduced by this
>> commit? https://github.com/apache/mesos/commit/902aa34b79
>>
>> On Wed, Oct 24, 2018 at 12:30 PM Vinod Kone  wrote:
>>
>>> -1
>>>
>>> Tested on ASF CI. Looks like Clang builds are failing with a build error.
>>> See example build output
>>> <
>>> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/55/BUILDTOOL=autotools,COMPILER=clang,CONFIGURATION=--verbose%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu:14.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/console
>>> >
>>> below:
>>>
>>> libtool: compile:  clang++-3.5 -DPACKAGE_NAME=\"mesos\"
>>> -DPACKAGE_TARNAME=\"mesos\" -DPACKAGE_VERSION=\"1.5.2\"
>>> "-DPACKAGE_STRING=\"mesos 1.5.2\"" -DPACKAGE_BUGREPORT=\"\"
>>> -DPACKAGE_URL=\"\" -DPACKAGE=\"mesos\" -DVERSION=\"1.5.2\"
>>> -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1
>>> -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1
>>> -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1
>>> -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\"
>>> -DHAVE_CXX11=1 -DHAVE_PTHREAD_PRIO_INHERIT=1 -DHAVE_PTHREAD=1
>>> -DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 -DHAVE_LIBAPR_1=1 -DHAVE_LIBCURL=1
>>> -DMESOS_HAS_JAVA=1 -DHAVE_EVENT2_EVENT_H=1 -DHAVE_LIBEVENT=1
>>> -DHAVE_EVENT2_THREAD_H=1 -DHAVE_LIBEVENT_PTHREADS=1 -DHAVE_LIBSASL2=1
>>> -DHAVE_OPENSSL_SSL_H=1 -DHAVE_EVENT2_BUFFEREVENT_SSL_H=1
>>> -DHAVE_LIBEVENT_OPENSSL=1 -DUSE_SSL_SOCKET=1 -DHAVE_SVN_VERSION_H=1
>>> -DHAVE_LIBSVN_SUBR_1=1 -DHAVE_SVN_DELTA_H=1 -DHAVE_LIBSVN_DELTA_1=1
>>> -DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 -DHAVE_PYTHON=\"2.7\"
>>> -DMESOS_HAS_PYTHON=1 -I. -I../../src -Werror
>>> -DLIBDIR=\"/mesos/mesos-1.5.2/_inst/lib\"
>>> -DPKGLIBEXECDIR=\"/mesos/mesos-1.5.2/_inst/libexec/mesos\"
>>> -DPKGDATADIR=\"/mesos/mesos-1.5.2/_inst/share/mesos\"
>>> -DPKGMODULEDIR=\"/mesos/mesos-1.5.2/_inst/lib/mesos/modules\"
>>> -I../../include -I../include -I../include/mesos -DPICOJSON_USE_INT64
>>> -D__STDC_FORMAT_MACROS -isystem ../3rdparty/boost-1.53.0 -isystem
>>> ../3rdparty/concurrentqueue-7b69a8f -I../3rdparty/elfio-3.2
>>> -I../3rdparty/glog-0.3.3/src -I../3rdparty/leveldb-1.19/include
>>> -I../../3rdparty/libprocess/include -I../3rdparty/nvml-352.79
>>> -I../3rdparty/picojson-1.3.0 -I../3rdparty/protobuf-3.5.0/src
>>> -I../../3rdparty/stout/include
>>> -I../3rdparty/zookeeper-3.4.8/src/c/include
>>> -I../3rdparty/zookeeper-3.4.8/src/c/generated -isystem
>>> /usr/include/subversion-1 -isystem /usr/include/apr-1 -isystem
>>> /usr/include/apr-1.0 -pthread -Wall -Wsign-compare -Wformat-security
>>> -fstack-protector-strong -fPIC -g1 -O0 -std=c++11 -MT
>>> slave/containerizer/libmesos_no_3rdparty_la-containerizer.lo -MD -MP
>>> -MF slave/containerizer/.deps/libmesos_no_3rdparty_la-containerizer.Tpo
>>> -c ../../src/slave/containerizer/containerizer.cpp  -fPIC -DPIC -o
>>> slave/containerizer/.libs/libmesos_no_3rdparty_la-containerizer.o
>>> In file included from ../../src/slave/http.cpp:30:
>>> In file included from ../../include/mesos/authorizer/authorizer.hpp:25:
>>> ../../3rdparty/libprocess/include/process/future.hpp:1089:3: error: no
>>> matching member function for call to 'set'
>>>   set(u);
>>>   ^~~
>>> ../../src/slave/http.cpp:3196:10: note: in instantiation of function
>>> template specialization
>>> 'process::Future::Future>> process::Future > >' requested here
>>>   return slave->containerizer->attach(containerId)
>>>  ^
>>> ../../3rdparty/libprocess/include/process/future.hpp:597:8: note:
>>> candidate function not viable: no known conversion from 'const
>>> process::Future >' to
>>> 'const process::http::Response' for 1st argument
>>>   bool set(const T& _t);
>>>^
>>> ../../3rdparty/libprocess/include/process/future.hpp:598:8: note:
>>> candidate function not viable: no known conversion from 'const
>&g

[VOTE] Release Apache Mesos 1.5.2 (rc1)

[Gilbert Song]

 Hi all,

Please vote on releasing the following candidate as Apache Mesos 1.5.2.

1.5.2 includes the following:

  * [MESOS-3790] - ZooKeeper connection should retry on `EAI_NONAME`.
  * [MESOS-8128] - Make os::pipe file descriptors O_CLOEXEC.
  * [MESOS-8418] - mesos-agent high cpu usage because of numerous
/proc/mounts reads.
  * [MESOS-8545] -
AgentAPIStreamingTest.AttachInputToNestedContainerSession is flaky.
  * [MESOS-8568] - Command checks should always call
`WAIT_NESTED_CONTAINER` before `REMOVE_NESTED_CONTAINER`.
  * [MESOS-8620] - Containers stuck in FETCHING possibly due to
unresponsive server.
  * [MESOS-8830] - Agent gc on old slave sandboxes could empty persistent
volume data.
  * [MESOS-8871] - Agent may fail to recover if the agent dies before image
store cache checkpointed.
  * [MESOS-8904] - Master crash when removing quota.
  * [MESOS-8906] - `UriDiskProfileAdaptor` fails to update profile
selectors.
  * [MESOS-8917] - Agent leaking file descriptors into forked processes.
  * [MESOS-8921] - Autotools don't work with newer OpenJDK versions.
  * [MESOS-8935] - Quota limit "chopping" can lead to cpu-only and
memory-only offers.
  * [MESOS-8936] - Implement a Random Sorter for offer allocations.
  * [MESOS-8942] - Master streaming API does not send (health) check
updates for tasks.
  * [MESOS-8945] - Master check failure due to CHECK_SOME(providerId).
  * [MESOS-8947] - Improve the container preparing logging in IOSwitchboard
and volume/secret isolator.
  * [MESOS-8952] - process::await/collect n^2 performance issue.
  * [MESOS-8963] - Executor crash trying to print container ID.
  * [MESOS-8978] - Command executor calling setsid breaks the tty support.
  * [MESOS-8980] - mesos-slave can deadlock with docker pull.
  * [MESOS-8986] - `slave.available()` in the allocator is expensive and
drags down allocation performance.
  * [MESOS-8987] - Master asks agent to shutdown upon auth errors.
  * [MESOS-9024] - Mesos master segfaults with stack overflow under load.
  * [MESOS-9049] - Agent GC could unmount a dangling persistent volume
multiple times.
  * [MESOS-9116] - Launch nested container session fails due to incorrect
detection of `mnt` namespace of command executor's task.
  * [MESOS-9125] - Port mapper CNI plugin might fail with "Resource
temporarily unavailable".
  * [MESOS-9127] - Port mapper CNI plugin might deadlock iptables on the
agent.
  * [MESOS-9131] - Health checks launching nested containers while a
container is being destroyed lead to unkillable tasks.
  * [MESOS-9142] - CNI detach might fail due to missing network config file.
  * [MESOS-9144] - Master authentication handling leads to request
amplification.
  * [MESOS-9145] - Master has a fragile burned-in 5s authentication timeout.
  * [MESOS-9146] - Agent has a fragile burn-in 5s authentication timeout.
  * [MESOS-9147] - Agent and scheduler driver authentication retry backoff
time could overflow.
  * [MESOS-9151] - Container stuck at ISOLATING due to FD leak.
  * [MESOS-9170] - Zookeeper doesn't compile with newer gcc due to format
error.
  * [MESOS-9196] - Removing rootfs mounts may fail with EBUSY.
  * [MESOS-9231] - `docker inspect` may return an unexpected result to
Docker executor due to a race condition.
  * [MESOS-9267] - Mesos agent crashes when CNI network is not configured
but used.
  * [MESOS-9279] - Docker Containerizer 'usage' call might be expensive if
mount table is big.
  * [MESOS-9283] - Docker containerizer actor can get backlogged with large
number of containers.
  * [MESOS-9308] - URI disk profile adaptor could deadlock.

The CHANGELOG for the release is available at:
https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.5.2-rc1


The candidate for Mesos 1.5.2 release is available at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.2-rc1/mesos-1.5.2.tar.gz

The tag to be voted on is 1.5.2-rc1:
https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.5.2-rc1

The SHA512 checksum of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.2-rc1/mesos-1.5.2.tar.gz.sha512

The signature of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.2-rc1/mesos-1.5.2.tar.gz.asc

The PGP key used to sign the release is here:
https://dist.apache.org/repos/dist/release/mesos/KEYS

The JAR is in a staging repository here:
https://repository.apache.org/content/repositories/orgapachemesos-1238

Please vote on releasing this package as Apache Mesos 1.5.2!

The vote is open until Wed Oct 24 23:59:00 PDT 2018 and passes if a
majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Mesos 1.5.2
[ ] -1 Do not release this package because ...

Thanks,
Gilbert

Re: Propose to update the minimal supported Docker version in Mesos to Docker 1.8.0

[Gilbert Song]

We are considering to backport this fix to Mesos 1.6.x, 1.5.x and 1.4.x,
which means starting from 1.4.x your minimum docker version will be bumped
to Docker 1.8.0 (an old docker daemon version).

Please reply on this thread if this is a breaking change for you. Thank you!

- Gilbert

On Fri, Sep 28, 2018 at 7:01 PM Qian Zhang  wrote:

> Hi All,
>
> When I worked on MESOS-9231
> , I found the way that
> we
> run the `docker inspect` command seems not correct, we'd better specify
> `--type=container` for it, otherwise, `docker inspect` may return an object
> which is not a container (e.g., a volume).
>
> So I posted a patch  to add
> `--type=container` to the `docker inspect` command. However I found Docker
> started to support `--type=container` from 1.8.0, but our minimal supported
> Docker version is 1.0.0 (see here
> 
> for
> details). I think it does not make sense for us to support Docker 1.0.0
> which is too old, so I propose to update the minimal supported Docker
> version in Mesos to 1.8.0.
>
> Please let me know for any comments, thanks!
>
>
> Regards,
> Qian Zhang
>

[Containerization WG] Today's WG Sync

[Gilbert Song]

Hi folks,

I apologize that I did not have a chance to confirm the agenda with
proposers yesterday. Feel free to add items that you would like to discuss
to the agenda:
https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit#heading=h.smrrkri8x6b9

The WG meeting will start at 9 am PST. See you there!

Cheers,
Gilbert

[Containerization WG] Sync Agenda for July 26th, 2018

[Gilbert Song]

Hi folks,

Tomorrow's WG meeting starts at 9 am PST. Here is the agenda:

   - [zhitao] Discussion on whether/how to retry isolator::cleanup
  - See comments in https://issues.apache.org/jira/browse/MESOS-8038

Here is the link for Containerization WG:
https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit#heading=h.9r35x5sgq5ez

Please join us via Zoom if you are interested: https://zoom.us/j/224160545

Thanks,
Gilbert

Re: Backport Policy

[Gilbert Song]

Thanks for clarifying the backporting policy, BenM!

I totally agree with the changes proposed for the backporting policy, but I
realize two more scenarios that are more clear to me yet:

   - There are some bugs that are not fixable (due to legacy technical
   decisions), and we end up with fixing the issue by a semantic/behavior
   change in a new release. Do we expect this semantic/behavior change being
   backported?
   - There might be some bugs that root cause is unknown yet, but it did
   impact on a couple releases. If we decide to add some commits for debugging
   purpose (e.g., a new debugging endpoint, or more logging), should we also
   allow these patches to be backported?

For #2, I think we should do the backporting, but for #1, maybe more
discussion is needed since it relates to whether the user has to upgrade or
not.

Cheers,
Gilbert

On Tue, Jul 17, 2018 at 4:26 PM, Lawrence Rau  wrote:

> I don’t have a big stake in, however, one opinion is if a large commercial
> enterprise was using a specific release that is working the desire is often
> to only upgrade if necessary.  Necessary can be for a number of reasons
> including new features; however if a new feature is not needed the
> compelling reason to upgrade is to fix a specific problem that is causing
> issues.  Thus keeping a maintenance release stable is very important and
> reducing the chance for, while fixing one problem, introducing another.
>
> Often a clear classification of severity of the problem would dictate the
> need to make a change. (yes these can be subjective, but some guidance
> would be better than nothing).
>
> It might be good to give committers guidance on back porting things that
> have a high impact on improving a problem.  Fixing a crashing bug, fixing a
> degenerative performance issue, etc, where these issues have no easy/viable
> work around.  Nice to have fixes aren’t, always, worth updating to.
>
> There can be an argument to respond with a “then don’t upgrade” but if
> changing the release with “nice to have’s” and several point releases later
> when a critical bug is fixed then the org if forced to accept the risk of
> the nice to have’s.
>
> just an opinion.
> …larry
>
>
> On Jul 17, 2018, at 3:00 PM, Chun-Hung Hsiao 
> wrote:
>
> I just have a comment on a special case:
> If a fix for a flaky test is easy to backport,
> IMO we probably should backport it,
> otherwise if someone backports another critical fix in the future,
> it would take them extra effort to check all CI failures.
>
> On Mon, Jul 16, 2018 at 11:39 AM Vinod Kone  wrote:
>
>> I like how you summarized it Greg and I would vote for leaving the
>> decision
>> to the committer too. In addition to what others mentioned, I think
>> committer should've the responsibility because if things break in a point
>> release (after it is released), it is the committer and contributor who
>> are
>> on the hook to triage and fix it and not the release manager.
>>
>> Having said that, if "during" the release process (i.e., cutting an RC)
>> these backports cause delays for a release manager in getting the release
>> out (e.g., CI flakiness introduced due to backports), release manager
>> could
>> be the ultimate arbiter on whether such a backport should be reverted or
>> fixed by the committer/contributor. Hopefully such issues are caught much
>> before a release process is started (e.g., CI running against release
>> branches).
>>
>> On Mon, Jul 16, 2018 at 1:28 PM Jie Yu  wrote:
>>
>> > Greg, I like your idea of adding a prescriptive "policy" when evaluating
>> > whether a bug fix should be backported, and leave the decision to
>> committer
>> > (because they have the most context, and avoid a bottleneck in the
>> > process).
>> >
>> > - Jie
>> >
>> > On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann  wrote:
>> >
>> > > My impression is that we have two opposing schools of thought here:
>> > >
>> > >1. Backport as little as possible, to avoid unforeseen consequences
>> > >2. Backport as much as proves practical, to eliminate bugs in
>> > >supported versions
>> > >
>> > > Do other people agree with this assessment?
>> > >
>> > > If so, how can we find common ground? One possible solution would be
>> to
>> > > leave the decision on backporting up to the committer, without
>> > specifying a
>> > > project-wide policy. This seems to be the status quo, and would lead
>> to
>> > > some variation across committers regarding what types of fixes are
>> > > backported. We could also choose to delegate the decision to the
>> release
>> > > manager; I favor leaving the decision with the committer, to eliminate
>> > the
>> > > burden on release managers.
>> > >
>> > > Here's a thought: rather than defining a prescriptive "policy" that we
>> > > expect committers to abide by, we could enumerate in the documentation
>> > the
>> > > competing concerns that we expect committers to consider when making
>> > > decisions on backports. The committing docs could read 

[Containerization WG] Sync Agenda for July 12th, 2018

[Gilbert Song]

Hi folks,

Tomorrow's WG meeting starts at 9 am PST. Here is the agenda:

   -

   [andrei, gilbert]: seccomp on Mesos Containerizer
   -


  
https://docs.google.com/document/d/146FJJ0sDi1sp_HQxVUg-vhqVSTEsdCeD4If3b1xCeec/edit#heading=h.9s5i7epe2uh9
  -

   [jpeach] Allow schedulers to specify allowed devices for tasks MESOS-9021
   .
   - Design doc
  

  .


Here is the link for Containerization WG:
https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit#heading=h.4r3hyrlaqiqz

Please join us via Zoom if you are interested: https://zoom.us/j/224160545

Thanks,
Gilbert

New Image Distribution Protocol and HDFS Image Fetching Design

[Gilbert Song]

Hi folks,

Here is the design doc for HDFS Image Fetching
.
It includes a custom protocol (Mesos Image Distribution Protocol
),
the corresponding Mesos support, and Image Store Deployment guide. The
Mesos Image Distribution Protocol is compatible for docker image v2 schema 1
 and schema 2
 (and potentially
compatible with OCI
).

This design doc aims for introducing an additional path to distribute
container images, compared to Docker Registry API. Please take a look and
leave comments.

Welcome to join the next Containerization WG meeting

on
July 12th, if you want to discuss in details.

Thanks,
Gilbert​​

[Containerization WG] Sync Agenda for June 28th, 2018

[Gilbert Song]

Hi folks,

Today's WG meeting starts at 9 am PST. The agenda:

   - [jpeach]: Allow schedulers to specify allowed devices for tasks
   MESOS-9021 .
  - Design doc
  

  .

Here is the link for Containerization WG:
https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit#heading=h.jhr91rmesh0a

Please join us via Zoom if you are interested: https://zoom.us/j/224160545

Thanks,
Gilbert

[Containerization WG] Sync Agenda for June 14th, 2018

[Gilbert Song]

Hi folks,

Tomorrow's WG meeting starts at 9 am PST. The agenda:

   - [gilbert]: Cgroup subsystems auto detection
   - [gilbert]: Container-specific cgroup FS mounts
  -
  
https://docs.google.com/document/d/1aaPmoKRupTSplFoQoT8op1aktm0vOmnuF5JXN0g_aTY/edit


Here is the link for Containerization WG:
https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit#heading=h.8qjdtwy10tk

Please join us via Zoom if you are interested: https://zoom.us/j/224160545

Thanks,
Gilbert

[RESULT][VOTE] Release Apache Mesos 1.5.1 (rc1)

[Gilbert Song]

 Hi all,

The vote for Mesos 1.5.1 (rc1) has passed with the
following votes.

+1 (Binding)
--
*** Greg Mann
*** Jie Yu
*** Ben Mahler

There were no 0 or -1 votes.

Please find the release at:
https://dist.apache.org/repos/dist/release/mesos/1.5.1

It is recommended to use a mirror to download the release:
http://www.apache.org/dyn/closer.cgi

The CHANGELOG for the release is available at:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.5.1

The mesos-1.5.1.jar has been released to:
https://repository.apache.org

The website (http://mesos.apache.org) will be updated shortly to reflect
this release.

Thanks,
Gilbert

[Containerization WG] Sync Agenda for May 17th, 2018

[Gilbert Song]

Hi folks,

Tomorrow's WG meeting starts at 9 am PST. The agenda:


   - [jpeach] Automatically populating device nodes allowed by the devices
   cgroup.
   -

* Container Devices
  
*
   - Command executor deprecation.
   -

* MESOS-8366 *

* - POC review chain *
   - [jie] Container logger module interface change.


Here is the link for Containerization WG:
https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit#heading=h.3fcg2lmrzman

Please join us via Zoom if you are interested: https://zoom.us/j/224160545

Thanks,
Gilbert

Re: [mesos-mail] Re: [Performance WG] Notes from meeting today

[Gilbert Song]

Do we have the recorded video for this meeting?

On Wed, May 16, 2018 at 1:54 PM, Benjamin Bannier <
benjamin.bann...@mesosphere.io> wrote:

> Hi Ben,
>
> thanks for taking the time to edit and share these detailed notes. Being
> able to asynchronously see the great work folks are doing surfaced is
> great, especially when put into context with thought like here.
>
>
> Benjamin
>
> > On May 16, 2018, at 8:06 PM, Benjamin Mahler  wrote:
> >
> > Hi folks,
> >
> > Here are some notes from the performance meeting today.
> >
> > (1) First I did a demo of flamescope, you can find it here:
> > https://github.com/Netflix/flamescope
> >
> > It's a very useful tool, hopefully we can make it easier for users to
> > generate the data that we can drop into flamescope when reporting any
> > performance issues. One of the open questions is how `perf --call-graph
> > dwarf` compares to `perf -g` but with mesos compiled with frame
> pointers. I
> > haven't had time to check this yet.
> >
> > When playing with the tool, it was easy to find some hot spots in the
> given
> > cluster I was looking at (which was not necessarily representative). For
> > the agent, jie filed:
> >
> > https://issues.apache.org/jira/browse/MESOS-8901
> >
> > And for the master, I noticed that metrics, state json generation (no
> > surprise), and a particular spot in the allocator were very expensive.
> >
> > Metrics we'd like to address via migration to push gauges (Zhitao has
> > offered to help with this effort):
> >
> > https://issues.apache.org/jira/browse/MESOS-8914
> >
> > The state generation we'd like to address via streaming state into a
> > separate actor (and providing filtering as well), this will get further
> > investigated / prioritized very soon:
> >
> > https://issues.apache.org/jira/browse/MESOS-8345
> >
> > (2) Kapil discussed benchmarks for the long standing "offer starvation"
> > issue:
> >
> > https://issues.apache.org/jira/browse/MESOS-3202
> >
> > I'll send out an email or document soon with some background on this
> issue
> > as well as our options to address it.
> >
> > Let me know if you have any questions or feedback!
> >
> > Ben
>
> --
> You received this message because you are subscribed to the Google Groups
> "Apache Mesos Mail Lists" group.
> Visit this group at https://groups.google.com/a/mesosphere.io/group/mesos-
> mail/.
> For more options, visit https://groups.google.com/a/mesosphere.io/d/optout
> .
>

[VOTE] Release Apache Mesos 1.5.1 (rc1)

[Gilbert Song]

Hi all,

Please vote on releasing the following candidate as Apache Mesos 1.5.1.

1.5.1 includes the following:

* [MESOS-1720] - Slave should send exited executor message when the
executor is never launched.
* [MESOS-7742] - Race conditions in IOSwitchboard: listening on unix socket
and premature closing of the connection.
* [MESOS-8125] - Agent should properly handle recovering an executor when
its pid is reused.
* [MESOS-8411] - Killing a queued task can lead to the command executor
never terminating.
* [MESOS-8416] - CHECK failure if trying to recover nested containers but
the framework checkpointing is not enabled.
* [MESOS-8468] - `LAUNCH_GROUP` failure tears down the default executor.
* [MESOS-8488] - Docker bug can cause unkillable tasks.
* [MESOS-8510] - URI disk profile adaptor does not consider plugin type for
a profile.
* [MESOS-8536] - Pending offer operations on resource provider resources
not properly accounted for in allocator.
* [MESOS-8550] - Bug in `Master::detected()` leads to coredump in
`MasterZooKeeperTest.MasterInfoAddress`.
* [MESOS-8552] - CGROUPS_ROOT_PidNamespaceForward and
CGROUPS_ROOT_PidNamespaceBackward tests fail.
* [MESOS-8565] - Persistent volumes are not visible in Mesos UI when
launching a pod using default executor.
* [MESOS-8569] - Allow newline characters when decoding base64 strings in
stout.
* [MESOS-8574] - Docker executor makes no progress when 'docker inspect'
hangs.
* [MESOS-8575] - Improve discard handling for 'Docker::stop' and
'Docker::pull'.
* [MESOS-8576] - Improve discard handling of 'Docker::inspect()'.
* [MESOS-8577] - Destroy nested container if
`LAUNCH_NESTED_CONTAINER_SESSION` fails.
* [MESOS-8594] - Mesos master stack overflow in libprocess socket send loop.
* [MESOS-8598] - Allow empty resource provider selector in
`UriDiskProfileAdaptor`.
* [MESOS-8601] - Master crashes during slave reregistration after failover.
* [MESOS-8604] - Quota headroom tracking may be incorrect in the presence
of hierarchical reservation.
* [MESOS-8605] - Terminal task status update will not send if 'docker
inspect' is hung.
* [MESOS-8619] - Docker on Windows uses `USERPROFILE` instead of `HOME` for
credentials.
* [MESOS-8624] - Valid tasks may be explicitly dropped by agent due to race
conditions.
* [MESOS-8631] - Agent should be able to start a task with every CPU on a
Windows machine.
* [MESOS-8641] - Event stream could send heartbeat before subscribed.
* [MESOS-8646] - Agent should be able to resolve file names on open files.
* [MESOS-8651] - Potential memory leaks in the `volume/sandbox_path`
isolator.
* [MESOS-8741] - `Add` to sequence will not run if it races with sequence
destruction.
* [MESOS-8742] - Agent resource provider config API calls should be
idempotent.
* [MESOS-8786] - CgroupIsolatorProcess accesses subsystem processes
directly.
* [MESOS-8787] - RP-related API should be experimental.
* [MESOS-8876] - Normal exit of Docker container using rexray volume
results in TASK_FAILED.
* [MESOS-8881] - Enable epoll backend in libevent integration.
* [MESOS-8885] - Disable libevent debug mode.

The CHANGELOG for the release is available at:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.5.1-rc1


The candidate for Mesos 1.5.1 release is available at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.1-rc1/mesos-1.5.1.tar.gz

The tag to be voted on is 1.5.1-rc1:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=1.5.1-rc1

The SHA512 checksum of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.1-rc1/mesos-1.5.1.tar.gz.sha512

The signature of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.1-rc1/mesos-1.5.1.tar.gz.asc

The PGP key used to sign the release is here:
https://dist.apache.org/repos/dist/release/mesos/KEYS

The JAR is in a staging repository here:
https://repository.apache.org/content/repositories/orgapachemesos-1224

Please vote on releasing this package as Apache Mesos 1.5.1!

The vote is open until Wed May 16 12:31:02 PDT 2018 and passes if a
majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Mesos 1.5.1
[ ] -1 Do not release this package because ...

Thanks,
Gilbert

[Containerization WG] Sync Agenda for May 4th, 2018

[Gilbert Song]

Hi folks,

Today's WG meeting starts at 9 am PST. The agenda:

   - Support image tar archives HDFS fetching.
   - Discussion: support multiple container runtimes on Mesos.

Here is the link for Containerization WG:
https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit#heading=h.su5jnm9qp8dx

Please join us via Zoom if you are interested: https://zoom.us/j/224160545

Thanks,
Gilbert

[Containerization WG] Cancelled Today

[Gilbert Song]

Hi folks,

Seems like we do not have any agenda item for today's meeting. I *cancel *it
this time.

Let's sync up next time on May 3rd. Please add your agenda item to the
Containerization WG Doc

.

Thanks,
Gilbert

Re: [mesos-mail] Re: Update the *Minimum Linux Kernel version* supported on Mesos

[Gilbert Song]

Hi all, FYI we landed this patch  to
avoid nested freezer cgroup support check for old kernel versions. Please
reply to this thread if you had concerns about this update.

- Gilbert

On Sun, Apr 8, 2018 at 12:18 AM, Alex Rukletsov  wrote:

> This does not seem to me as a disruptive change, so I'm +1.
>
> On Thu, Apr 5, 2018 at 6:36 PM, Jie Yu  wrote:
>
>> User namespaces require >= 3.12 (November 2013). Can we make that the
>>> minimum?
>>
>>
>> No, we need to support CentOS7 which uses 3.10 (some variant)
>>
>> - Jie
>>
>> On Thu, Apr 5, 2018 at 8:56 AM, James Peach  wrote:
>>
>>>
>>>
>>> > On Apr 5, 2018, at 5:00 AM, Andrei Budnik 
>>> wrote:
>>> >
>>> > Hi All,
>>> >
>>> > We would like to update minimum supported Linux kernel from 2.6.23 to
>>> > 2.6.28.
>>> > Linux kernel supports cgroups v1 starting from 2.6.24, but `freezer`
>>> cgroup
>>> > functionality was merged into 2.6.28, which supports nested containers.
>>>
>>> User namespaces require >= 3.12 (November 2013). Can we make that the
>>> minimum?
>>>
>>> J
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Apache Mesos Mail Lists" group.
> Visit this group at https://groups.google.com/a/mesosphere.io/group/mesos-
> mail/.
> For more options, visit https://groups.google.com/a/mesosphere.io/d/optout
> .
>

[Containerization WG] Sync Agenda for April 5th, 2018

[Gilbert Song]

Hi folks,

Tomorrow's WG meeting starts at 9 am PST. The agenda:

   - Volume Ownership use case update.
   - Mesos 1.6.0 will be cut around the end of April, let’s review our
   goals for 1.6 and update the spreadsheet.
   - Update the *Minimum Linux Kernel version* supported on Mesos to 2.6.28
   for nested freezer cgroup support.

Here is the link for Containerization WG:
https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugt
c2nHR89skFXSpU/edit#

Please join us via Zoom if you are interested: https://zoom.us/j/224160545

Thanks,
Gilbert

Re: This Month in Mesos - March 2018

[Gilbert Song]

Thanks for the awesome update, Greg!

- Gilbert

On Fri, Mar 30, 2018 at 6:20 PM, Vinod Kone  wrote:

> Thanks for the update Greg!
>
> Sent from my phone
>
> > On Mar 30, 2018, at 3:08 PM, Greg Mann  wrote:
> >
> > Oh hai there Apache Mesos Community!
> >
> > Back again with your monthly update on current events in the Mesosverse:
> >
> >
> > *Working Groups*
> >
> > Below you'll find a brief summary of the group meetings from this past
> > month, as well as some info about related work that's been happening in
> the
> > project. Working group meetings can be found on the Mesos community
> calendar
> > , and you should
> feel
> > free to add agenda items beforehand!
> >
> >
> > *API Working Group*
> >
> > [Agenda Doc
> >  IBWw1f_Ler6fLM/edit#>
> > ]
> >
> > Next Meeting: April 3 @ 11am PST
> >
> > In March we held the first two meetings of the new API working group!
> This
> > has brought about a revival of our perennial discussion on the preferred
> > Mesos release cadence; you can expect an updated release policy in our
> > documentation shortly. It's looking like the new policy will be in line
> > with what we have been doing in practice for the last few releases, so no
> > big changes there.
> >
> >
> > Zhitao also presented his ongoing work on new operations which will allow
> > the growing/shrinking of persistent volumes. You can find his design doc
> > here
> >  6EOaPzPtwYNVQUQ/edit>
> > .
> >
> >
> > *Containerization Working Group*
> >
> > [Agenda Doc
> >  c2nHR89skFXSpU/edit>
> > ]
> >
> > Next meeting: April 5 @ 9am PST
> >
> > Two big items in the containerization space this month:
> >
> >
> >   - Improvements to the Docker containerizer/executor to more gracefully
> >   handle bugs in the Docker daemon: MESOS-8572
> >   
> >   - Configurable network namespaces for nested containers: MESOS-8534
> >   
> >
> > *Community Working Group*
> >
> > [Agenda Doc
> >  3JG4f3qg-N5En-4ubg/edit#>
> > ]
> >
> > Next Meeting: April 9 @ 10:30am PST
> >
> > Community working group had a preliminary discussion about the next
> > quarterly doc-a-thon, and discussed the possibility of spinning up a new
> > Releases Working Group. We also discussed plans for the next MesosCon,
> and
> > how we may want to evolve that event going forward.
> >
> >
> > *Performance Working Group*
> >
> > [Agenda Doc
> >  4bodagrlNGCuQU/edit>
> > ]
> >
> > Next meeting: April 18 @ 10am PST
> >
> > We now have a performance dashboard
> >  rapidView=238=planning>
> > which lets you view tickets in ASF JIRA which have been marked as
> > performance-related - take a look!
> >
> >
> > Some additional copy elimination
> >  patches have been
> > merged, with more yet to come. The group also discussed the near-term
> > performance roadmap, which includes optimization of
> > authentication/authorization, master state computation, and the
> libprocess
> > HTTP code; see the agenda document for more details.
> >
> >
> >
> > Until next time,
> > -Greg
>

Mesos 1.5.1 Release

[Gilbert Song]

Folks,

I will be the release manager for Mesos 1.5.1 Release. Here is the JIRA
filter created for this release:
https://issues.apache.org/jira/issues/?filter=12343539

Please add the *JIRA target version as *1.5.1** and *mark it as *blocker**
if you have any issue that you need in 1.5.1, and reply to this thread.

Cheers,
Gilbert

Re: [Containerization WG] Call for Agenda, March 22nd, 2018

[Gilbert Song]

Seems like there is no agenda item for tomorrow's WG meeting. We will
cancel it at this time.

On Wed, Mar 21, 2018 at 10:30 AM, Gilbert Song <gilb...@apache.org> wrote:

> Folks,
>
> We are planning for a WG meeting tomorrow at 9 am PST.
>
> Please add any agenda item or topic that you like to discuss with the
> Containerization WG to the following list:
> https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugt
> c2nHR89skFXSpU/edit#heading=h.j7quoqe53vwr
>
> Thanks,
> Gilbert
>

[Containerization WG] Call for Agenda, March 22nd, 2018

[Gilbert Song]

Folks,

We are planning for a WG meeting tomorrow at 9 am PST.

Please add any agenda item or topic that you like to discuss with the
Containerization WG to the following list:
https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit#heading=h.j7quoqe53vwr

Thanks,
Gilbert

Welcome Zhitao Li as Mesos Committer and PMC Member

[Gilbert Song]

Hi,

I am excited to announce that the PMC has voted Zhitao Li as a new
committer and member of PMC for the Apache Mesos project. Please join me to
congratulate Zhitao!

Zhitao has been an active contributor to Mesos for one and a half years.
His main contributions include:

   - Designed and implemented Container Image Garbage Collection (MESOS-4945
   );
   - Designed and implemented part of the HTTP Operator API (MESOS-6007
   );
   - Reported and fixed a lot of bugs
   

   .

Zhitao spares no effort to improve the project quality and to propose
ideas. Thank you Zhitao for all contributions!

Here is his committer candidate checklist for your perusal:
https://docs.google.com/document/d/1HGz7iBdo1Q9z9c8fNRgNNLnj0XQ_PhDhjXLAfOx139s/

Congrats Zhitao!

Cheers,
Gilbert

Re: Welcome Chun-Hung Hsiao as Mesos Committer and PMC Member

[Gilbert Song]

Congrats, Chun!

It is great to have you in the community!

- Gilbert

On Sun, Mar 11, 2018 at 4:40 PM, Andrew Schwartzmeyer <
and...@schwartzmeyer.com> wrote:

> Congratulations Chun!
>
> I apologize for not also giving you a +1, as I certainly would have, but
> just discovered my mailing list isn't working. Just a heads up, don't let
> that happen to you too!
>
> I look forward to continuing to work with you.
>
> Cheers,
>
> Andy
>
>
> On 03/10/2018 9:14 pm, Jie Yu wrote:
>
>> Hi,
>>
>> I am happy to announce that the PMC has voted Chun-Hung Hsiao as a new
>> committer and member of PMC for the Apache Mesos project. Please join me
>> to
>> congratulate him!
>>
>> Chun has been an active contributor for the past year. His main
>> contributions to the project include:
>> * Designed and implemented gRPC client support to libprocess (MESOS-7749)
>> * Designed and implemented Storage Local Resource Provider (MESOS-7235,
>> MESOS-8374)
>> * Implemented part of the CSI support (MESOS-7235, MESOS-8374)
>>
>> Chun is friendly and humble, but also intelligent, insightful, and
>> opinionated. I am confident that he will be a great addition to our
>> committer pool. Thanks Chun for all your contributions to the project so
>> far!
>>
>> His committer checklist can be found here:
>> https://docs.google.com/document/d/1FjroAvjGa5NdP29zM7-2eg6t
>> LPAzQRMUmCorytdEI_U/edit?usp=sharing
>>
>> - Jie
>>
>
>

[Containerization WG] Sync Agenda for March 8th, 2018

[Gilbert Song]

Hi folks,

Tomorrow's WG meeting starts at 9 am PST. We will have 3 agenda items to
discuss:

   - [Andrei, Greg, Gilbert]: Resolve the docker daemon hanging issue and
   do backport.
   - [zhitao] Persistent volume resize support (MESOS-4965
   ).
  - WIP Design Doc
  

   - [Harold] XFS Soft Limit Kill Support (MESOS-6575
   )
  - WIP Design Doc
  


Here is the link for Containerization WG:
https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit#

Please join us via Zoom if you are interested: https://zoom.us/j/224160545

Thanks,
Gilbert

Re: Looking for a shepherd for MESOS-4965

[Gilbert Song]

Zhitao, would you mind adding an item to the agenda of Containerization WG?
https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit#heading=h.99ld98fhk6ar

On Mon, Mar 5, 2018 at 12:59 PM, Zhitao Li  wrote:

> Thanks Jie.
>
> I'll try to start with some write up on this. Can we discuss this in this
> week's container WG since it's pretty high priority to our org (some recent
> accounting shows that it's potentially a lot of saving on hardware).
>
> On Mon, Mar 5, 2018 at 11:09 AM, Jie Yu  wrote:
>
> > Zhitao, I can provide some feedback because it most likely overlap with
> the
> > Resource Provider/CSI work.
> >
> > - Jie
> >
> > On Fri, Mar 2, 2018 at 9:08 PM, Zhitao Li  wrote:
> >
> > > Hi,
> > >
> > > As our workload using persistent volume increases, the support for
> > resizing
> > > volumes  gets more
> > > important to us, especially for our space efficiency.
> > >
> > > Can someone familiar with the topic help to shepherd? I can work on
> > design
> > > and implementation.
> > >
> > > Thanks!
> > >
> > > --
> > > Cheers,
> > >
> > > Zhitao Li
> > >
> >
>
>
>
> --
> Cheers,
>
> Zhitao Li
>

Re: Tasks may be explicitly dropped by agent in Mesos 1.5

[Gilbert Song]

Meng,

Could you double check if this is really an issue in Mesos 1.5.0 release?

MESOS-1720  was resolved
after the 1.5 release (rc-2) and it seems like
it is only at the master branch and 1.5.x branch (not 1.5.0).

Did I miss anything?

- Gilbert

On Thu, Mar 1, 2018 at 4:22 PM, Benjamin Mahler  wrote:

> Put another way, we currently don't guarantee in-order task delivery to
> the executor. Due to the changes for MESOS-1720, one special case of task
> re-ordering now leads to the re-ordered task being dropped (rather than
> delivered out-of-order as before). Technically, this is strictly better.
>
> However, we'd like to start guaranteeing in-order task delivery.
>
> On Thu, Mar 1, 2018 at 2:56 PM, Meng Zhu  wrote:
>
>> Hi all:
>>
>> TLDR: In Mesos 1.5, tasks may be explicitly dropped by the agent
>> if all three conditions are met:
>> (1) Several `LAUNCH_TASK` or `LAUNCH_GROUP` calls
>>  use the same executor.
>> (2) The executor currently does not exist on the agent.
>> (3) Due to some race conditions, these tasks are trying to launch
>> on the agent in a different order from their original launch order.
>>
>> In this case, tasks that are trying to launch on the agent
>> before the first task in the original order will be explicitly dropped by
>> the agent (TASK_DROPPED` or `TASK_LOST` will be sent)).
>>
>> This bug will be fixed in 1.5.1. It is tracked in
>> https://issues.apache.org/jira/browse/MESOS-8624
>>
>> 
>>
>> In https://issues.apache.org/jira/browse/MESOS-1720, we introduced an
>> ordering dependency between two `LAUNCH`/`LAUNCH_GROUP`
>> calls to a new executor. The master would specify that the first call is
>> the
>> one to launch a new executor through the `launch_executor` field in
>> `RunTaskMessage`/`RunTaskGroupMessage`, and the second one should
>> use the existing executor launched by the first one.
>>
>> On the agent side, running a task/task group goes through a series of
>> continuations, one is `collect()` on the future that unschedule
>> frameworks from
>> being GC'ed:
>> https://github.com/apache/mesos/blob/master/src/slave/slave.cpp#L2158
>> another is `collect()` on task authorization:
>> https://github.com/apache/mesos/blob/master/src/slave/slave.cpp#L2333
>> Since these `collect()` calls run on individual actors, the futures of the
>> `collect()` calls for two `LAUNCH`/`LAUNCH_GROUP` calls may return
>> out-of-order, even if the futures these two `collect()` wait for are
>> satisfied in
>> order (which is true in these two cases).
>>
>> As a result, under some race conditions (probably under some heavy load
>> conditions), tasks rely on the previous task to launch executor may
>> get processed before the task that is supposed to launch the executor
>> first, resulting in the tasks being explicitly dropped by the agent.
>>
>> -Meng
>>
>>
>>
>

[Containerization WG] Sync Agenda Today

[Gilbert Song]

Hi folks,

Today we have two items to discuss in our agenda for the Containerization WG

meeting:






* - [Sagar]: Configurable network namespace at CNI isolator (10 ~ 15 min)-
https://issues.apache.org/jira/browse/MESOS-8534
-
https://github.com/apache/mesos/pull/263
- Why not a separate
namespace/network isolator like UTS ns to make it configurable? - [Jason,
Zhitao]: Container root filesystem space isolation- MESOS-7676
- Design Doc
Please
join us via Zoom if you are interested: https://zoom.us/j/224160545
Thanks,Gilbert*

Re: [RESULT][VOTE] Release Apache Mesos 1.5.0 (rc2)

[Gilbert Song]

Thanks to everyone who contributed to Mesos 1.5! The release blog is now up
at http://mesos.apache.org/blog/mesos-1-5-0-released/, and cross-posted on
Medium <https://medium.com/apache-mesos/apache-mesos-9c979d475d8b> in case
you prefer a shinier blog experience. Enjoy!

On Thu, Feb 8, 2018 at 8:18 AM, Gilbert Song <gilb...@apache.org> wrote:

> Hi all,
>
> The vote for Mesos 1.5.0 (rc2) has passed with the
> following votes.
>
> +1 (Binding)
> --
> Jie Yu
> Vinod Kone
> James Peach
> Andrew Schwartzmeyer
>
> +1 (Non-binding)
> --
> Zhitao Li
> Chun-Hung Hsiao
>
> There were no 0 or -1 votes.
>
> Please find the release at:
> https://dist.apache.org/repos/dist/release/mesos/1.5.0
>
> It is recommended to use a mirror to download the release:
> http://www.apache.org/dyn/closer.cgi
>
> The CHANGELOG for the release is available at:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_
> plain;f=CHANGELOG;hb=1.5.0
>
> The mesos-1.5.0.jar has been released to:
> https://repository.apache.org
>
> The website (http://mesos.apache.org) will be updated shortly to reflect
> this release.
>
> Thanks,
> Jie and Gilbert
>

[RESULT][VOTE] Release Apache Mesos 1.5.0 (rc2)

[Gilbert Song]

Hi all,

The vote for Mesos 1.5.0 (rc2) has passed with the
following votes.

+1 (Binding)
--
Jie Yu
Vinod Kone
James Peach
Andrew Schwartzmeyer

+1 (Non-binding)
--
Zhitao Li
Chun-Hung Hsiao

There were no 0 or -1 votes.

Please find the release at:
https://dist.apache.org/repos/dist/release/mesos/1.5.0

It is recommended to use a mirror to download the release:
http://www.apache.org/dyn/closer.cgi

The CHANGELOG for the release is available at:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.5.0

The mesos-1.5.0.jar has been released to:
https://repository.apache.org

The website (http://mesos.apache.org) will be updated shortly to reflect
this release.

Thanks,
Jie and Gilbert

[VOTE] Release Apache Mesos 1.5.0 (rc2)

[Gilbert Song]

Hi all,

Please vote on releasing the following candidate as Apache Mesos 1.5.0.

1.5.0 includes the following:

  * Support Container Storage Interface (CSI).
  * Agent reconfiguration policy.
  * Auto GC docker images in Mesos Containerizer.
  * Standalone containers.
  * Support gRPC client.
  * Non-leading VOTING replica catch-up.


The CHANGELOG for the release is available at:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.5.0-rc2


The candidate for Mesos 1.5.0 release is available at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.0-rc2/mesos-1.5.0.tar.gz

The tag to be voted on is 1.5.0-rc2:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=1.5.0-rc2

The MD5 checksum of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.0-rc2/mesos-1.5.0.tar.gz.md5

The signature of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.0-rc2/mesos-1.5.0.tar.gz.asc

The PGP key used to sign the release is here:
https://dist.apache.org/repos/dist/release/mesos/KEYS

The JAR is in a staging repository here:
https://repository.apache.org/content/repositories/orgapachemesos-1222

Please vote on releasing this package as Apache Mesos 1.5.0!

The vote is open until Tue Feb  6 17:35:16 PST 2018 and passes if a
majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Mesos 1.5.0
[ ] -1 Do not release this package because ...

Thanks,
Jie and Gilbert

Re: This Month in Mesos - January 2018

[Gilbert Song]

Awesome, Greg!

On Tue, Jan 30, 2018 at 10:34 AM, Vinod Kone  wrote:

> Thanks Greg for the update!
>
>
> *Medium Posts*
>> Going forward, we'll be cross-posting Apache Mesos blog posts on Medium to
>> increase exposure, get feedback on engagement, and make it easier to
>> share.
>> Check out the new Apache Mesos publication > os> on
>> Medium for the latest content!
>>
>
>
> I'm really looking forward to this. As a reminder, if any one in the
> community wants to blog (cross post) about their use of Apache Mesos please
> reach out to us. We are always looking for great content!
>

Re: [VOTE] Release Apache Mesos 1.5.0 (rc1)

[Gilbert Song]

Based on blocker issues list above and -1s, I will cut Mesos 1.5.0 rc2 once
MESOS-8481 <https://issues.apache.org/jira/browse/MESOS-8481> is resolved.

On Wed, Jan 24, 2018 at 10:40 AM, Michael Park <mp...@apache.org> wrote:

> -1 (binding) for https://issues.apache.org/jira/browse/MESOS-8481
> On Wed, Jan 24, 2018 at 10:34 AM James Peach <jor...@gmail.com> wrote:
>
> > +1
> >
> > Verified on CentOS 6 and Fedora 27
> >
> > > On Jan 22, 2018, at 7:15 PM, Gilbert Song <gilb...@apache.org> wrote:
> > >
> > > Hi all,
> > >
> > > Please vote on releasing the following candidate as Apache Mesos 1.5.0.
> > >
> > > 1.5.0 includes the following:
> > >
> > 
> 
> > >   * Support Container Storage Interface (CSI).
> > >   * Agent reconfiguration policy.
> > >   * Auto GC docker images in Mesos Containerizer.
> > >   * Standalone containers.
> > >   * Support gRPC client.
> > >   * Non-leading VOTING replica catch-up.
> > >
> > > The CHANGELOG for the release is available at:
> > >
> > https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_
> plain;f=CHANGELOG;hb=1.5.0-rc1
> > >
> > 
> 
> > >
> > > The candidate for Mesos 1.5.0 release is available at:
> > >
> > https://dist.apache.org/repos/dist/dev/mesos/1.5.0-rc1/
> mesos-1.5.0.tar.gz
> > >
> > > The tag to be voted on is 1.5.0-rc1:
> > > https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=
> commit;h=1.5.0-rc1
> > >
> > > The MD5 checksum of the tarball can be found at:
> > >
> > https://dist.apache.org/repos/dist/dev/mesos/1.5.0-rc1/
> mesos-1.5.0.tar.gz.md5
> > >
> > > The signature of the tarball can be found at:
> > >
> > https://dist.apache.org/repos/dist/dev/mesos/1.5.0-rc1/
> mesos-1.5.0.tar.gz.asc
> > >
> > > The PGP key used to sign the release is here:
> > > https://dist.apache.org/repos/dist/release/mesos/KEYS
> > >
> > > The JAR is in a staging repository here:
> > > https://repository.apache.org/content/repositories/orgapachemesos-1221
> > >
> > > Please vote on releasing this package as Apache Mesos 1.5.0!
> > >
> > > The vote is open until Thu Jan 25 18:24:36 PST 2018 and passes if a
> > majority of at least 3 +1 PMC votes are cast.
> > >
> > > [ ] +1 Release this package as Apache Mesos 1.5.0
> > > [ ] -1 Do not release this package because ...
> > >
> > > Thanks,
> > > Jie and Gilbert
> >
> >
>

[VOTE] Release Apache Mesos 1.5.0 (rc1)

[Gilbert Song]

Hi all,

Please vote on releasing the following candidate as Apache Mesos 1.5.0.

1.5.0 includes the following:

  * Support Container Storage Interface (CSI).
  * Agent reconfiguration policy.
  * Auto GC docker images in Mesos Containerizer.
  * Standalone containers.
  * Support gRPC client.
  * Non-leading VOTING replica catch-up.

The CHANGELOG for the release is available at:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.5.0-rc1


The candidate for Mesos 1.5.0 release is available at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.0-rc1/mesos-1.5.0.tar.gz

The tag to be voted on is 1.5.0-rc1:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=1.5.0-rc1

The MD5 checksum of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.0-rc1/mesos-1.5.0.tar.gz.md5

The signature of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/1.5.0-rc1/mesos-1.5.0.tar.gz.asc

The PGP key used to sign the release is here:
https://dist.apache.org/repos/dist/release/mesos/KEYS

The JAR is in a staging repository here:
https://repository.apache.org/content/repositories/orgapachemesos-1221

Please vote on releasing this package as Apache Mesos 1.5.0!

The vote is open until Thu Jan 25 18:24:36 PST 2018 and passes if a
majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Mesos 1.5.0
[ ] -1 Do not release this package because ...

Thanks,
Jie and Gilbert

Mesos 1.5.0 Release

[Gilbert Song]

Folks,

It is time for Mesos 1.5.0 release. I am the release manager.

We plan to cut the rc1 in next couple weeks. Please start to wrap up
patches if you are contributing or shepherding any issue. If you expect any
particular JIRA for this new release, please set *Target Version* as "
*1.5.0"* and mark it as "*Blocker*" priority.

The dashboard for Mesos 1.5.0 will be posted in this thread soon.

Cheers,
Gilbert

Re: Quarterly Doc-a-thon Scheduling

[Gilbert Song]

Thank you, Judith! Will definitely join!

- Gilbert

On Tue, Nov 21, 2017 at 2:07 PM, Judith Malnick <jmaln...@mesosphere.io>
wrote:

> Hi Gilbert,
>
> The date I'm suggesting is January 11th (which is the second Thursday to
> occur in January). Hope it works for you :)
>
> Cheers,
> Judith
>
> --
> Get the Boomerang Email App <http://boomerangapp.com/mobile> on your phone
>
>
> ------
> On Mon, Nov 20, 2017 at 6:17 PM, Gilbert Song <gilb...@apache.org> wrote:
>

Re: Quarterly Doc-a-thon Scheduling

[Gilbert Song]

This is great! Thank you for scheduling the docathon, Judith!

BTW, do you mean January 2nd Tuesday or January 4th Thursday (asking since
I might be on a flight on the 2nd January)?

Cheers,
Gilbert

On Mon, Nov 20, 2017 at 11:00 AM, Vinod Kone  wrote:

> SGTM!
>
> On Mon, Nov 20, 2017 at 10:13 AM, Judith Malnick 
> wrote:
>
>> Hi all,
>>
>> Since the last doc-a-thon was so great, I'd love to get another on the
>> calendar for next quarter.
>>
>> The last one was on the 2nd Thursday of October, so It might make
>> sense to have the next one on the 2nd Thursday of January. (*January
>> 11th, 3-8pm Pacific time*).
>>
>> Does this date sound good to you? If I get many nos I'll follow up with a
>> doodle poll.
>>
>> All the best!
>> Judith
>>
>> --
>> Judith Malnick
>> Community Manager
>> 310-709-1517 <(310)%20709-1517>
>>
>
>

Re: [mesos-mail] Re: Welcome Greg Mann as a new committer and PMC member!

[Gilbert Song]

Congrats, Greg!

On Wed, Jun 14, 2017 at 11:46 AM, Sam  wrote:

> Congrats, well deserved
>
>
> Regards,
> Sam Chen | APJ Country Director | DC/OS Evangelist
>
> [image: Watch the Video]
> 
>  [image:
> .]Build and run modern apps
> at scale using DC/OS
> 
>
>
> On Jun 14, 2017, at 5:44 AM, Neil Conway  wrote:
>
> Congratulations, Greg!! Very well-deserved. Looking forward to
> continuing to work with you on the project.
>
> Neil
>
>
> On Tue, Jun 13, 2017 at 2:42 PM, Vinod Kone  wrote:
>
> Hi folks,
>
>
> Please welcome Greg Mann as the newest committer and PMC member of the
>
> Apache Mesos project.
>
>
> Greg has been an active contributor to the Mesos project for close to 2
>
> years now and has made many solid contributions. His biggest source code
>
> contribution to the project has been around adding authentication support
>
> for default executor. This was a major new feature that involved quite a
> few
>
> moving parts. Additionally, he also worked on improving the scheduler and
>
> executor APIs.
>
>
> Here is his more formal checklist for your perusal.
>
>
> https://docs.google.com/document/d/1S6U5OFVrl7ySmpJsfD4fJ3_
> R8JYRRc5spV0yKrpsGBw/edit
>
>
> Thanks,
>
> Vinod
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Apache Mesos Mail Lists" group.
> Visit this group at https://groups.google.com/a/mesosphere.io/group/mesos-
> mail/.
> For more options, visit https://groups.google.com/a/mesosphere.io/d/optout
> .
>

Docker Private Registry Credential per Container

[Gilbert Song]

Hi all,

Recently, we started working on support for docker private registry
credential per container in Mesos (MESOS-7088
). This feature aims to
allow users to specify a registry credential for an image at the first
class API.

Here is the link to the design doc:
https://docs.google.com/document/d/1kMXeJEuw4_adwyxqEjoMAdxRNzYHWrT_-eGd1uQLQEg/edit?usp=sharing

The design doc may not cover some pieces of implementation details, but it
should be sufficient to cover main ideas, user-facing API, and security
concerns. Comments will be absolutely welcome.

Thanks,
Gilbert

Re: Welcome Qian Zhang as a new committer!

[Gilbert Song]

Congrats, Qian!

On Sun, Oct 9, 2016 at 10:11 AM, Zhitao Li  wrote:

> Congratulations, Qian!
>
> On Sun, Oct 9, 2016 at 6:49 AM, Yongqiao Wang 
> wrote:
>
> > Congratulations, Qian!
> >
> >
> > --
> > Regards!
> > Grady YQ. Wang（王勇桥）
> >
> >
> >
> >
> >
> >
> >
> > -- Original --
> > From:  "Jie Yu";;
> > Date:  Sun, Oct 9, 2016 02:20 AM
> > To:  "mesos"; "u...@mesos.apache.org" > mesos.apache.org>;
> >
> > Subject:  Welcome Qian Zhang as a new committer!
> >
> >
> >
> > Hi folks,
> >
> > I' happy to announce that the PMC has voted Qian Zhang as a new committer
> > and
> > member of PMC for the Apache Mesos project. Please join me to
> congratulate
> > him!
> >
> > A little more about Qian Zhang:
> >
> > Qian Zhang has been working on the Apache Mesos project for about an year
> > now. He designed and implemented the CNI
> >  (Container Network
> Interface)
> > support in Mesos with Avinash, which standardized the networking
> > integration in Mesos. He also worked with haosdent on the unified cgroups
> > isolator , which
> greatly
> > simplifies the original cgroups support in Mesos and makes extension to
> new
> > subsystems so much easier. He was also involved in discussions on quotas
> > and pods, and provided valuable feedback. He is currently working on OCI
> >  support in Mesos, trying
> to
> > enable Mesos to launch OCI containers.
> >
> > More details can be found in his committer candidate checklist
> >  > 7mjWXei9D9SaYUU/edit?usp=sharing>
> > .
> >
> > Qian, thank you for your great work to the project so far. Would love to
> > see more!
> >
> > - Jie
> >
>
>
>
> --
> Cheers,
>
> Zhitao Li
>

Re: Unified cgroups isolator

[Gilbert Song]

Awesome!

Kudos to @haosdent and @qianzhang!

On Tue, Sep 13, 2016 at 11:22 AM, haosdent  wrote:

> Really appreciate @qian and @jie's great helps on this! It makes us easier
> to add cgroups isolation for rest subsystem.
>
> Additionally, if you find any changes about unified cgroups isolator break
> your environment, please let us know. I would
> try to fix asap.
>
> On Wed, Sep 14, 2016 at 1:59 AM, Jie Yu  wrote:
>
>> Hi,
>>
>> We just merged the unified cgroups isolator. Huge shout out to @haosdent
>> and @qianzhang to make this happen!
>> https://issues.apache.org/jira/browse/MESOS-4697
>>
>> Just to give you some context. Previously, it's a huge pain to add a new
>> cgroups subsystem to Mesos because it requires creating a new isolator (a
>> lot of code duplication). Now, we merge all the subsystems into one single
>> isolator, that makes adding a new subsystem very easy.
>>
>> More importantly, the new cgroups isolator supports cgroups v2!
>>
>> - Jie
>>
>
>
>
> --
> Best Regards,
> Haosdent Huang
>

Re: Docker containerizer: override USER

[Gilbert Song]

Yeah, it is a little hard to do it because of backward compatibility.

Gilbert

On Fri, Sep 2, 2016 at 1:14 AM, Olivier Sallou <olivier.sal...@irisa.fr>
wrote:

>
>
> - Mail original -
> > De: "Gilbert Song" <gilb...@mesosphere.io>
> > À: "dev" <dev@mesos.apache.org>
> > Envoyé: Jeudi 1 Septembre 2016 19:21:06
> > Objet: Re: Docker containerizer: override USER
> >
> > We considered support --user option in docker containerizer.
> Unfortunately,
> > it would
> > potentially break some previous users in behavior. So we did not merge
> it.
> > Please
> > see this JIRA for detail:
> >
> > https://issues.apache.org/jira/browse/MESOS-5754
> >
> > However, you can still use DockerInfo::Parameter to specify your --user
> as a
> > workaround.
>
> That's what I did, but I expected a more *integrated* solution.
>
> Thanks  anyway
>
> Olivier
> >
> > Gilbert
> >
> > On Thu, Sep 1, 2016 at 9:15 AM, Olivier Sallou <olivier.sal...@irisa.fr>
> > wrote:
> >
> > >
> > >
> > > - Mail original -
> > > > De: "Qian Zhang" <zhq527...@gmail.com>
> > > > À: dev@mesos.apache.org
> > > > Envoyé: Jeudi 1 Septembre 2016 15:57:39
> > > > Objet: Re: Docker containerizer: override USER
> > > >
> > > > Hi Olivier,
> > > >
> > > > Can you try TaskInfo.CommandInfo.user?
> > >
> > > I will try but mesos.proto specifies:
> > >
> > >   // Enables executor and tasks to run as a specific user. If the user
> > >   // field is present both in FrameworkInfo and here, the CommandInfo
> > >   // user value takes precedence.
> > >
> > > FrameworkInfo.user is specified in my case and set to the expected user
> > > XX. So it does not seem that the container is executed wit the --user
> XX
> > > flag.
> > >
> > > Olivier
> > >
> > >
> > > >
> > > >
> > > > Thanks,
> > > > Qian Zhang
> > > >
> > > > On Thu, Sep 1, 2016 at 4:39 PM, Olivier Sallou <
> olivier.sal...@irisa.fr>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > If Docker image specified a USER in Dockerfile, docker will use
> this
> > > user
> > > > > when executing command in container.
> > > > > In Docker commands, it can be overriden with -u XX .
> > > > >
> > > > > I do not find however in mesos.proto a way to do so. There is the
> > > > > "arguments" of DockerInfo that I could use to append this to the
> > > executor
> > > > > command line, but I think it is not advised as it may not be
> supported
> > > in
> > > > > future.
> > > > >
> > > > > Did I miss something ?
> > > > >
> > > > > Thanks
> > > > >
> > > > > Olvier
> > > > >
> > > >
> > >
> >
>

Re: Docker containerizer: override USER

[Gilbert Song]

We considered support --user option in docker containerizer. Unfortunately,
it would
potentially break some previous users in behavior. So we did not merge it.
Please
see this JIRA for detail:

https://issues.apache.org/jira/browse/MESOS-5754

However, you can still use DockerInfo::Parameter to specify your --user as a
workaround.

Gilbert

On Thu, Sep 1, 2016 at 9:15 AM, Olivier Sallou 
wrote:

>
>
> - Mail original -
> > De: "Qian Zhang" 
> > À: dev@mesos.apache.org
> > Envoyé: Jeudi 1 Septembre 2016 15:57:39
> > Objet: Re: Docker containerizer: override USER
> >
> > Hi Olivier,
> >
> > Can you try TaskInfo.CommandInfo.user?
>
> I will try but mesos.proto specifies:
>
>   // Enables executor and tasks to run as a specific user. If the user
>   // field is present both in FrameworkInfo and here, the CommandInfo
>   // user value takes precedence.
>
> FrameworkInfo.user is specified in my case and set to the expected user
> XX. So it does not seem that the container is executed wit the --user XX
> flag.
>
> Olivier
>
>
> >
> >
> > Thanks,
> > Qian Zhang
> >
> > On Thu, Sep 1, 2016 at 4:39 PM, Olivier Sallou 
> > wrote:
> >
> > > Hi,
> > > If Docker image specified a USER in Dockerfile, docker will use this
> user
> > > when executing command in container.
> > > In Docker commands, it can be overriden with -u XX .
> > >
> > > I do not find however in mesos.proto a way to do so. There is the
> > > "arguments" of DockerInfo that I could use to append this to the
> executor
> > > command line, but I think it is not advised as it may not be supported
> in
> > > future.
> > >
> > > Did I miss something ?
> > >
> > > Thanks
> > >
> > > Olvier
> > >
> >
>

The External Containerizer is Removed

[Gilbert Song]

Hello All,

FYI, as we discussed on the mailing list in Oct 2015 (link
)
and Apr 2016 (link

),
the External Containerizer was deprecated.

Now, we removed the External Containerizer from Mesos. Please see
MESOS-3370  for details.

Gilbert

Re: How could Mesos work with docker?

[Gilbert Song]

Yu,

Mesos supports docker images without using the docker daemon. This feature
is called Unified Containerizer, which was introduced since Mesos 0.28.
Please
follow this document to launch your containers:

http://mesos.apache.org/documentation/latest/container-image/

Feel free to join our slack channel 
if you have any question.

Gilbert

On Tue, Aug 9, 2016 at 7:13 AM, Yu Wei  wrote:

> Hi Tomek,
>
>
> Good point.
> Thanks very much for your help.
>
>
> Jared, (韦煜）
> Software developer
> Interested in open source software, big data, Linux
> --
> *From:* Tomek Janiszewski 
> *Sent:* Tuesday, August 9, 2016 10:06:28 PM
> *To:* dev@mesos.apache.org; u...@mesos.apache.org
> *Subject:* Re: How could Mesos work with docker?
>
> Hi
>
> Docker support is described here http://mesos.apache.org/
> documentation/latest/docker-containerizer/
>
> Best
> Tomek
>
> wt., 9.08.2016 o 16:04 użytkownik Yu Wei  napisał:
>
>> Hi guys,
>>
>> I start learning mesos.
>>
>> How could mesos work with docker? Is there any document about this?
>>
>>
>> Any other advice?
>>
>>
>> Thx,
>>
>>
>> Jared, (韦煜）
>> Software developer
>> Interested in open source software, big data, Linux
>>
>

Re: Proposal: move content in Wiki to docs in code repo

[Gilbert Song]

+1.

At least I personally rarely touch the wiki.

Gilbert

On Mon, Jun 6, 2016 at 11:51 AM, Zhou Z Xing  wrote:

> +1
>
> It's good idea to collect and gather every thing in one single repo, would
> be easier for users to find out.
>
> Thanks & Best Wishes,
>
> Tom Xing(邢舟)
> Emerging Technology Institute, IBM China Software Development Lab
> --
> IBM China Software Development Laboratory (CSDL)
> Notes ID:Zhou Z Xing/China/IBM
> Phone :86-10-82450442
> e-Mail :xingz...@cn.ibm.com
> Address :Building No.28, ZhongGuanCun Software Park, No.8 Dong Bei Wang
> West Road, Haidian District, Beijing, P.R.China 100193
> 地址 :中国北京市海淀区东北旺西路8号 中关村软件园28号楼 100193
>
>
> [image: Inactive hide details for Jie Yu ---2016-06-06 上午 11:29:42---Hi
> folks, I am proposing moving our content in Wiki (e.g., wor]Jie Yu
> ---2016-06-06 上午 11:29:42---Hi folks, I am proposing moving our content in
> Wiki (e.g., working groups, release
>
> From: Jie Yu 
> To: mesos , "u...@mesos.apache.org" <
> u...@mesos.apache.org>
> Date: 2016-06-06 上午 11:29
> Subject: Proposal: move content in Wiki to docs in code repo
> --
>
>
>
> Hi folks,
>
> I am proposing moving our content in Wiki (e.g., working groups, release
> tracking, etc.) to our docs in the code repo. I personally found that wiki
> is hard to use and there's no reviewing process for changes in the Wiki.
> The content in Wiki historically received less attention than that in the
> docs.
>
> What do you think?
>
> - Jie
>
>
>
>

Re: Documentation about debugging mesos-master : newbie

[Gilbert Song]

For dev, I would strongly recommend using linux os (e.g., ubuntu14.04,
15.10, centos7.1, etc.). Because features are very limited in os x, most
of features like using containers, cgroup isolation etc. you will need a
linux environment.

PS: using a vagrant box may be convenient to setup env on your os x.

Gilbert

On Thu, Jun 2, 2016 at 12:24 PM, Vinit Mahedia <vinitmahe...@gmail.com>
wrote:

> I tried that, it did not help either, same error. What do you use for dev
> OS, IDE etc.
> Do you think switching on ubuntu should help? I will try that anyway. I am
> out of
> ideas at this point if switching OS does not work either.
>
> gdb) b master.cpp:2481
> Cannot access memory at address 0x714d40
>
>
> On Thu, Jun 2, 2016 at 1:56 PM, Gilbert Song <gilb...@mesosphere.io>
> wrote:
>
> > Vinit,
> >
> > Seems like your question is similar to this one:
> >
> >
> https://mail-archives.apache.org/mod_mbox/mesos-dev/201507.mbox/%3CCAOs_uxyyeJNF+CtceOd7zZabSMJTneC=yyqcpwbz-gjyesp...@mail.gmail.com%3E
> >
> > Could you verify if James' reply would help for your case?
> >
> >
> https://mail-archives.apache.org/mod_mbox/mesos-dev/201507.mbox/%3c7155f7cf-db57-4202-bc97-62e6dcfe1...@gmail.com%3E
> >
> > Gilbert
> >
> > On Thu, Jun 2, 2016 at 11:24 AM, Vinit Mahedia <vinitmahe...@gmail.com>
> > wrote:
> >
> > > Hi Gilbert,
> > >
> > > Thank you for replying.
> > >
> > > Yes, I did that.
> > >
> > >
> > >1.  ./configure --enable-debug --disable-java --disable-python
> > >2.  make
> > >3. ./bin/gdb-mesos-master.sh --ip=127.0.0.1 --work_dir=.
> > >
> > > Although even after setting source directory, I can not set breakpoint
> I
> > > get warning like this
> > >
> > > (gdb) break master.cpp:2481
> > > Cannot access memory at address 0x714d40
> > >
> > >
> > > I also tried few things, passing "static" flag to libtool, passing
> > >  "--enable-static"
> > >
> > > Although I got linker error, where I saw libtool was not using --static
> > > flag and I do
> > > not know if doing that will fix it. I forgot to mention that am
> building
> > > this on Mac OS.
> > >
> > > Thank you.
> > >
> > >
> > >
> > > On Thu, Jun 2, 2016 at 12:33 PM, Gilbert Song <gilb...@mesosphere.io>
> > > wrote:
> > >
> > > > Hi Vinit,
> > > >
> > > > Did you configure with debug mode (e.g., ../confugure
> --enable-debug)?
> > > >
> > > > Assuming you have the gdb installed, you should be able to debug
> mesos
> > > > master
> > > > in gbd:
> > > >
> > > > ./bin/gdb-mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos
> > > >
> > > >
> > > > Gilbert
> > > >
> > > > On Thu, Jun 2, 2016 at 9:30 AM, Vinit Mahedia <
> vinitmahe...@gmail.com>
> > > > wrote:
> > > >
> > > > > I have been trying to debug mesos-master using gdb-mesos-master.sh
> > > > although
> > > > > it does not load symbols or sources. I tried to set those paths as
> > well
> > > > but
> > > > > since it thinks mesos-master, libtool script, is the main binary.
> > > > >
> > > > > I just want to set the dev environment and try to fix a very stupid
> > bug
> > > > to
> > > > > learn the work flow of test/debug/commit.
> > > > >
> > > > > If I can get it working, I can help to write if such documentation
> > does
> > > > not
> > > > > exist. I also tried to set it up on eclipse CDT but it can't handle
> > > > libtool
> > > > > scripts.
> > > > >
> > > > > Thank you.
> > > > >
> > > >
> > >
> >
>

Re: Documentation about debugging mesos-master : newbie

[Gilbert Song]

Vinit,

Seems like your question is similar to this one:
https://mail-archives.apache.org/mod_mbox/mesos-dev/201507.mbox/%3CCAOs_uxyyeJNF+CtceOd7zZabSMJTneC=yyqcpwbz-gjyesp...@mail.gmail.com%3E

Could you verify if James' reply would help for your case?
https://mail-archives.apache.org/mod_mbox/mesos-dev/201507.mbox/%3c7155f7cf-db57-4202-bc97-62e6dcfe1...@gmail.com%3E

Gilbert

On Thu, Jun 2, 2016 at 11:24 AM, Vinit Mahedia <vinitmahe...@gmail.com>
wrote:

> Hi Gilbert,
>
> Thank you for replying.
>
> Yes, I did that.
>
>
>1.  ./configure --enable-debug --disable-java --disable-python
>2.  make
>3. ./bin/gdb-mesos-master.sh --ip=127.0.0.1 --work_dir=.
>
> Although even after setting source directory, I can not set breakpoint I
> get warning like this
>
> (gdb) break master.cpp:2481
> Cannot access memory at address 0x714d40
>
>
> I also tried few things, passing "static" flag to libtool, passing
>  "--enable-static"
>
> Although I got linker error, where I saw libtool was not using --static
> flag and I do
> not know if doing that will fix it. I forgot to mention that am building
> this on Mac OS.
>
> Thank you.
>
>
>
> On Thu, Jun 2, 2016 at 12:33 PM, Gilbert Song <gilb...@mesosphere.io>
> wrote:
>
> > Hi Vinit,
> >
> > Did you configure with debug mode (e.g., ../confugure --enable-debug)?
> >
> > Assuming you have the gdb installed, you should be able to debug mesos
> > master
> > in gbd:
> >
> > ./bin/gdb-mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos
> >
> >
> > Gilbert
> >
> > On Thu, Jun 2, 2016 at 9:30 AM, Vinit Mahedia <vinitmahe...@gmail.com>
> > wrote:
> >
> > > I have been trying to debug mesos-master using gdb-mesos-master.sh
> > although
> > > it does not load symbols or sources. I tried to set those paths as well
> > but
> > > since it thinks mesos-master, libtool script, is the main binary.
> > >
> > > I just want to set the dev environment and try to fix a very stupid bug
> > to
> > > learn the work flow of test/debug/commit.
> > >
> > > If I can get it working, I can help to write if such documentation does
> > not
> > > exist. I also tried to set it up on eclipse CDT but it can't handle
> > libtool
> > > scripts.
> > >
> > > Thank you.
> > >
> >
>

Re: Documentation about debugging mesos-master : newbie

[Gilbert Song]

Hi Vinit,

Did you configure with debug mode (e.g., ../confugure --enable-debug)?

Assuming you have the gdb installed, you should be able to debug mesos
master
in gbd:

./bin/gdb-mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos


Gilbert

On Thu, Jun 2, 2016 at 9:30 AM, Vinit Mahedia 
wrote:

> I have been trying to debug mesos-master using gdb-mesos-master.sh although
> it does not load symbols or sources. I tried to set those paths as well but
> since it thinks mesos-master, libtool script, is the main binary.
>
> I just want to set the dev environment and try to fix a very stupid bug to
> learn the work flow of test/debug/commit.
>
> If I can get it working, I can help to write if such documentation does not
> exist. I also tried to set it up on eclipse CDT but it can't handle libtool
> scripts.
>
> Thank you.
>

Re: Extracting a filesystem isolation utility?

[Gilbert Song]

Hi Joshua,

Thanks for pointing this out. We understand that it would be a breaking
change in the future if Aurora release the unified containerizer support.
You want the chroot::enter call to be unified in one place instead of what
we currently doing on command executor launchtask, to avoid any other
duplicate code in executor. May I ask how urgent it is for you guys in
developing unified containerizer support? We will discuss about this issue
and create a JIRA to track it.

Thanks,
Gilbert

On Thu, May 26, 2016 at 12:38 PM, Joshua Cohen  wrote:

> Hi Mesos devs,
>
> I'm working on adding unified containerizer support to Aurora, and during
> review[1] it was pointed out that the work being done to pivot into the
> task's filesystem, mount the necessary special filesystems, etc. was a
> subset (and probably an insufficient subset at that) of what Mesos does
> when isolating a container's filesystem[2]. I wanted to propose that Mesos
> creates a utility along the lines of mesos-containerizer that makes it
> possible for an executor to share Mesos's logic without having to replicate
> it wholesale.
>
> Does this sound like something that could be done?
>
> Thanks!
>
> Joshua
>
> [1] https://reviews.apache.org/r/47853/
> [2]
>
> https://github.com/apache/mesos/blob/547f4a4d122253a42819d5746cf51593923a56bc/src/linux/fs.cpp#L599-L699
>

Re: volume / mount point error with Unified Containerizer

[Gilbert Song]

@Olivier,
In mesos 0.28.1, you are supposed to be able bind mount a volume from
the host into the mesos container. Did you specify a docker image (we
determine
the mount point differently depending whether the container has a rootfs)?
How
do you specify your 'container_path' (the mount point in the container)? If
it is an
absolute path, we require that dir to be pre-existed. If it is a relative
path, we will
mkdir for it.

@Joshua,
Thank for posting your workaround on mesos. As I mentioned above, in 0.28.1
or
older, we only mkdir for container_path which is relative path (not
starting with "/").
Because if no rootfs specified for a mesos container, the container shares
the host
root filesystem. Obviously we don't want any random files to be created
implicitly
on your host fs.
>From mesos 0.29 (release by the end of this month), we will mkdir the mount
point in the container except for the command task case that specify an
absolute
container_path without a rootfs. Because we simplify the mounting logic, and
sandbox bind mount will only be done in container mount namespace instead of
host mount namespace (what we did before). Please keep tuned.

Cheers,
Gilbert

On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen  wrote:

> Hi Olivier,
>
> I touched on this issue as part of
> https://issues.apache.org/jira/browse/MESOS-5229. It would be nice if
> Mesos
> automatically created container mount points if they don't already exist.
> In the meantime, as a workaround for this, I've updated my filesystem
> images to include the path (e.g. in Dockerfile, add `RUN mkdir -p
> /some/mount/point`). Not the best solution, but the only thing I've seen
> that works at the moment.
>
> Cheers,
>
> Joshua
>
> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu  wrote:
>
> > It's pretty simple for you from scratch with source code
> >
> >
> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
> > ;-)
> >
> > Thanks,
> >
> > Guangya
> >
> > On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou  >
> > wrote:
> >
> > >
> > >
> > > On 05/18/2016 02:31 PM, Guangya Liu wrote:
> > > > Just saw that you are working with 0.28.1, the "docker volume driver"
> > > code
> > > > was not in 0.28.1, can you please have a try with mesos master branch
> > if
> > > > you are only doing some test?
> > > this is indeed test only for the moment. But I will have to
> > > recompile/install mesos  :-(  (I used packages for install).
> > >
> > > I will try when possible, but thanks for the hint.
> > > >
> > > > Thanks,
> > > >
> > > > Guangya
> > > >
> > > > On Wed, May 18, 2016 at 8:28 PM, Guangya Liu 
> > wrote:
> > > >
> > > >> Hi Olivier,
> > > >>
> > > >> I think that you need to enable "docker volume isolator" if you want
> > use
> > > >> external storage with unified container I was writing a document
> here
> > > >> https://reviews.apache.org/r/47511/, perhaps you can have a try
> > > according
> > > >> to the document and post some comments there if you find any issues.
> > > >>
> > > >> Also you can patch mesos-execute here
> > > https://reviews.apache.org/r/46762/ to
> > > >> have a try with mesos-execute.
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Guangya
> > > >>
> > > >> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
> > > olivier.sal...@irisa.fr>
> > > >> wrote:
> > > >>
> > > >>> Answering (partially) to myself.
> > > >>>
> > > >>> I seems issue is container_path does not exists inside container.
> On
> > > >>> Docker, path is created and mounted. With pure mesos,
> container_path
> > > >>> must exists.
> > > >>>
> > > >>> mesos.proto says: "If the path is an absolute path, that path must
> > > >>> already exist."
> > > >>>
> > > >>> This is an issue however, using Docker images, the path I want to
> > mount
> > > >>> does not exists, and it cannot be modified "on the fly".
> > > >>>
> > > >>> Is there a workaround for this ?
> > > >>>
> > > >>>
> > > >>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> > >  Hi,
> > >  I am trying unified containerizer on a single server
> (master/slave)
> > on
> > >  mesos 0.28.1, to switch from docker containerizer to mesos+docker
> > > image
> > >  container.
> > > 
> > >  I have setup slave config as suggested in documentation:
> > > 
> > >  containerizers=docker,mesos
> > >  image_providers=docker \
> > >  isolation=filesystem/linux,docker/runtime
> > > 
> > >  However, when I execute my task with a volume I have an error:
> > > 
> > >  
> > >  + mount -n --rbind
> > > 
> > > >>>
> > >
> >
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> > > >>>
> > >
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > >  + mount -n 

Re: Questions about Docker related tests in Mesos

[Gilbert Song]

Sorry Zhiwei, just see your email. Yes, I can do that. Just for curious,
are you using
the manifest architecture field in ppc64le?

We already have tests relying on the image created locally. Please see:
runtime_isolator_tests.cpp

What do you mean by "And could you make it create and load the image before
running Docker related tests"? The local docker image is for unified
containerizer
testing. It is irrelevant to any test for docker containerizer.

Cheers,
Gilbert

On Mon, Apr 11, 2016 at 6:27 PM, zhiwei <zhiw...@gmail.com> wrote:

> Hi Gilbert, do you have any updates on about this issue?
>
> When will you use the local build Docker image for Docker related testing?
>
> On Mon, Mar 14, 2016 at 9:44 PM, zhiwei <zhiw...@gmail.com> wrote:
>
>> Hi Gibert,
>>
>> I tested your patch, it works on ppc64le, but there is an issue that the
>> architecture in manifest file is hardcode to amd64[1].
>>
>> This field is currently not used by Docker engine, but I think you can
>> enhance it to use `os.uname().machine` [2]. Or I can submit a patch for
>> this.
>>
>> And could you make it create and load the image before running Docker
>> related tests?
>>
>> [1]:
>> https://github.com/apache/mesos/blob/69b2ad528dd79979a8ee113a8edbbab2669e32e6/src/tests/containerizer/docker_archive.hpp#L150
>> [2]:
>> https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-2.md#manifest-list-field-descriptions
>>
>>
>> On Sat, Mar 12, 2016 at 8:45 PM, zhiwei <zhiw...@gmail.com> wrote:
>>
>>> this because the architecture, alpine is x86_64, the power is ppc64le.
>>>
>>> this may help in IBM Power.
>>> On Mar 12, 2016 01:10, "Gilbert Song" <gilb...@mesosphere.io> wrote:
>>>
>>>> Hi Zhiwei,
>>>>
>>>> I am trying to understand why 'alpine' is not compatible with IBM Power
>>>> platform. Is it because of the image's rootfs?
>>>>
>>>> We currently have a JIRA(
>>>> https://issues.apache.org/jira/browse/MESOS-4684)
>>>> in progress to create a local docker image, which is used for docker
>>>> runtime isolator local tests. This test image cp the host's linux
>>>> rootfs.
>>>> Does it possibly help in your case?
>>>>
>>>> Cheers,
>>>> Gilbert
>>>>
>>>> On Fri, Mar 11, 2016 at 2:30 AM, zhiwei <zhiw...@gmail.com> wrote:
>>>>
>>>> > Yes, I prefer to create specific mesos test images and make them
>>>> > configurable in configuration file.
>>>> >
>>>> > On Fri, Mar 11, 2016 at 6:20 PM, Alex Rukletsov <a...@mesosphere.com>
>>>> > wrote:
>>>> >
>>>> > > It also looks strange to me that we use "random" containers in Mesos
>>>> > tests.
>>>> > > The proper way would be to have "mesos" or "apache" account on
>>>> docker hub
>>>> > > managed by PMC. Do you think it's worth to set up one or it's too
>>>> much
>>>> > time
>>>> > > investment?
>>>> > >
>>>> > > On Thu, Mar 10, 2016 at 12:19 PM, Jan Schlicht <j...@mesosphere.io>
>>>> > wrote:
>>>> > >
>>>> > > > Hi Zhiwei,
>>>> > > >
>>>> > > > I was thinking about this as well, but for different reasons:
>>>> Pulling
>>>> > in
>>>> > > > Docker images for tests is not the ideal solution. Sure, testing a
>>>> > `sleep
>>>> > > > 1000` should work, but testing an executor leaves some questions
>>>> on how
>>>> > > to
>>>> > > > handle changes/deprecation of the interfaces. It's not a pressing
>>>> issue
>>>> > > > right now, but might become one in the future.
>>>> > > >
>>>> > > > I think this is also what Timothy had in mind with his comment
>>>> > (Timothy,
>>>> > > > please correct me if I'm wrong): These problems can be resolved by
>>>> > using
>>>> > > > local Docker images, ideally ones that are created during `make
>>>> check`.
>>>> > > But
>>>> > > > this would create new problems. Either we would have to build
>>>> libmesos
>>>> > > > inside our local container -- to be able to build tes

Re: [DISCUSS] Fetching Docker Images Requiring User Credentials.

[Gilbert Song]

@Kevin, thanks for writing it down in detail. It sounds good that a more
concrete
schema is designed to generally solve similar auth problem.

Just have two potential issues inlined below:

On Tue, Mar 15, 2016 at 5:39 PM, Kevin Klues  wrote:

> Yeah, option 2.
>
> I was trying to expand on Avinash's suggestion and make it a bit more
> concrete in terms of what was being proposed. Needing to reload the
> agent just to update the list of credentials it accepts seems
> undesirable though.
>
> Maybe we could have a way to start the agent with a default config (by
> iterating on the schema from my previous email), but allow newly
> launched frameworks to somehow update the config on the fly through a
>

Will it be too expensive to update all agents every time a new framework
joins (handling consensus problem as well)?


> file in their sandbox that follows the same schema.
>

Does that mean the file in sandbox should be exposed to each other?


> On Tue, Mar 15, 2016 at 5:25 PM, Jie Yu  wrote:
> > Kevin, are you suggesting option 2 and having a config file like the
> above?
> >
> > I think another downside of a per-agent config is that it's hard to
> > maintain this. What if a new framework joins and has a new credential for
> > the docker images. Do we need to restart the agent to reload the config?
> >
> > - Jie
> >
> > On Tue, Mar 15, 2016 at 1:25 PM, Kevin Klues  wrote:
> >
> >> Can we be a bit more concrete here and try to build up a schema for
> this.
> >> Maybe something like:
> >>
> >> {
> >>   [
> >> {
> >>   "service" : "docker",
> >>   "registries" :
> >>   [
> >> "uri" : "",
> >> "default_credentials" :
> >> {
> >>   "type" : "",
> >>   "credential" :
> >>   {
> >>   // Custom based on type...
> >>   }
> >> },
> >> "image_credentials" :
> >> [
> >>   {
> >> "image_name" : "",
> >> "type" : "",
> >> "credential" :
> >> {
> >>   // Custom based on type...
> >> },
> >>   },
> >>   ...
> >> ],
> >> ...
> >>   ]
> >>   ...
> >> },
> >> ...
> >>   ]
> >> }
> >>
> >>
> >> On Tue, Mar 15, 2016 at 12:57 PM, Jie Yu  wrote:
> >> >>
> >> >> Yeah I was thinking having the JSON as a dictionary with keys being
> the
> >> >> registry URI (appc/docker) and the values being credentials (which
> will
> >> be
> >> >> a dictionary as well I guess).
> >> >
> >> >
> >> > Using registry URI as the key is problematic. Think about the public
> >> docker
> >> > hub. Different frameworks might want to use different credentials to
> >> access
> >> > their docker images.
> >> >
> >> > - Jie
> >> >
> >> > On Tue, Mar 15, 2016 at 11:52 AM, Avinash Sridharan <
> >> avin...@mesosphere.io
> >> >
> >> > wrote:
> >> >
> >> >> On Tue, Mar 15, 2016 at 11:43 AM, Vinod Kone 
> >> wrote:
> >> >>
> >> >> > moved core@ to *bcc*
> >> >> >
> >> >> > On Tue, Mar 15, 2016 at 11:18 AM, Avinash Sridharan <
> >> >> avin...@mesosphere.io
> >> >> > > wrote:
> >> >> >
> >> >> >> Why not follow option 2, but instead of passing the agent
> >> credentials,
> >> >> >> pass a location to the flag where credentials for the registry
> can be
> >> >> found
> >> >> >> (in JSON)? The frameworks can set credentials (maybe registry
> name or
> >> >> URL
> >> >> >> to the registry), and the credentials can be learnt from the JSON
> >> >> config.
> >> >> >>
> >> >> >
> >> >> > What if we need credentials for multiple-registries? Have a JSON
> with
> >> one
> >> >> > credential per registry I guess? But if possible, I would love to
> >> solve
> >> >> > this more generally as possible; as Gilbert mentioned, this is not
> a
> >> >> > problem just for Docker images but any URIs that need AuthN.
> >> >> >
> >> >> Yeah I was thinking having the JSON as a dictionary with keys being
> the
> >> >> registry URI (appc/docker) and the values being credentials (which
> will
> >> be
> >> >> a dictionary as well I guess).
> >> >>
> >> >>
> >> >> --
> >> >> Avinash Sridharan, Mesosphere
> >> >> +1 (323) 702 5245
> >> >>
> >>
> >>
> >>
> >> --
> >> ~Kevin
> >>
>
>
>
> --
> ~Kevin
>

[DISCUSS] Fetching Docker Images Requiring User Credentials.

[Gilbert Song]

Hi folks,

We want to raise a discussion here, seeking suggestions about passing
credentials in a secure way. This relates to the JIRA MESOS-4938
, supporting docker
private registry authentication in unified containerizer. In fact, this
problem is not limited to docker registry. For instance, how can we support
CommandInfo.URIs that need credentials?

For the docker registry problem, credentials have to be included when
communicating with the docker auth server. We have two options here:

Option 1: Passing credentials in protobuf Image::Docker.

Pros: This means supporting per-container docker registry, which is robust
because different registry can be reached by an agent, configurable by
users.

Cons: So SSL has to be enabled to encrypt the communication between master
and slave to prevent credentials from being seen by others. We also need to
make sure we don’t expose credentials in any endpoint.

Option 2: Passing credentials as an agent flag.

Pros: Not necessary to be SSL enabled.

Cons: No per-container registry support (imagine a multi-tenant cluster).

Some background: How does docker containerizer solve this issue?

In docker containerizer, we ask the framework to specify a URI for their
task/executor that points to the .dockercfg(now ~/.docker/config.json)
which contains the user and password information. The .dockercfg will be
saved in the sandbox by the fetcher. When we call docker pull, we set the
$HOME env for the subprocess to point to the sandbox so that the docker
client can pick up that .dockercfg when pulling images.

Any comment/advice will be absolutely welcome!

Thanks,
Gilbert/Jie

Re: Questions about Docker related tests in Mesos

[Gilbert Song]

Hi Zhiwei,

I am trying to understand why 'alpine' is not compatible with IBM Power
platform. Is it because of the image's rootfs?

We currently have a JIRA(https://issues.apache.org/jira/browse/MESOS-4684)
in progress to create a local docker image, which is used for docker
runtime isolator local tests. This test image cp the host's linux rootfs.
Does it possibly help in your case?

Cheers,
Gilbert

On Fri, Mar 11, 2016 at 2:30 AM, zhiwei  wrote:

> Yes, I prefer to create specific mesos test images and make them
> configurable in configuration file.
>
> On Fri, Mar 11, 2016 at 6:20 PM, Alex Rukletsov 
> wrote:
>
> > It also looks strange to me that we use "random" containers in Mesos
> tests.
> > The proper way would be to have "mesos" or "apache" account on docker hub
> > managed by PMC. Do you think it's worth to set up one or it's too much
> time
> > investment?
> >
> > On Thu, Mar 10, 2016 at 12:19 PM, Jan Schlicht 
> wrote:
> >
> > > Hi Zhiwei,
> > >
> > > I was thinking about this as well, but for different reasons: Pulling
> in
> > > Docker images for tests is not the ideal solution. Sure, testing a
> `sleep
> > > 1000` should work, but testing an executor leaves some questions on how
> > to
> > > handle changes/deprecation of the interfaces. It's not a pressing issue
> > > right now, but might become one in the future.
> > >
> > > I think this is also what Timothy had in mind with his comment
> (Timothy,
> > > please correct me if I'm wrong): These problems can be resolved by
> using
> > > local Docker images, ideally ones that are created during `make check`.
> > But
> > > this would create new problems. Either we would have to build libmesos
> > > inside our local container -- to be able to build test executors --
> which
> > > would take a long time, or we'd have to make sure that the container
> > > environment is the same as the dev environment, to be able to copy test
> > > executors into it, which isn't easy unless we'd restrict ourselves to
> > only
> > > a couple of environments.
> > >
> > > Cheers,
> > > Jan
> > >
> > > On Thu, Mar 10, 2016 at 11:25 AM, zhiwei  wrote:
> > >
> > > > Hi all,
> > > >
> > > > The Docker related test cases that hardcoded "alpine" as the Docker
> > image
> > > > which caused test cases failed on IBM Power platform, since the
> Docker
> > > > image "alpine" is not compatible with IBM Power platform.
> > > >
> > > > And I saw an inline comment by Timothy: "// TODO(tnachen): Use local
> > > image
> > > > to test if possible."
> > > >
> > > > So just wonder if someone has plan to implement this, or could you
> give
> > > me
> > > > some tips? I can implement this.
> > > >
> > > > Following are the images that used in Mesos test cases:
> > > >
> > > > 1. alpine
> > > > 2. mesosphere/alpine-expect
> > > > 3. mesosphere/inky
> > > > 4. mesosphere/test-executor
> > > > 5. tnachen/test-executor
> > > >
> > > >
> > > > Thanks,
> > > > Zhiwei
> > > >
> > >
> > >
> > >
> > > --
> > > *Jan Schlicht*
> > > Distributed Systems Engineer, Mesosphere
> > >
> >
>

Re: [VOTE] Release Apache Mesos 0.28.0 (rc1)

[Gilbert Song]

Hi Kevin,

Please remove the the patch below from the list:
Implemented runtime isolator default cmd test (still under review).
https://reviews.apache.org/r/44469/

Because the bug was fixed by patch #44468, the test should not be
considered as a block. I am updating MESOS-4888 and move the test to a
separate JIRA.

Thanks,
Gilbert

On Tue, Mar 8, 2016 at 2:43 PM, Kevin Klues  wrote:

> Here are the list of reviews/patches that have been called out in this
> thread for inclusion in 0.28.0-rc2.  Some of them are still under
> review and will need to land by Thursday to be included.
>
> Are there others?
>
> Jie's container image documentation (submitted):
> commit 7de8cdd4d8ed1d222fa03ea0d8fa6740c4a9f84b
> https://reviews.apache.org/r/44414
>
> Restore Mesos' ability to extract Docker assigned IPs (still under review):
> https://reviews.apache.org/r/43093/
>
> Fixed the logic for default docker cmd case (submitted).
> commit e42f740ccb655c0478a3002c0b6fa90c1144f41c
> https://reviews.apache.org/r/44468/
>
> Implemented runtime isolator default cmd test (still under review).
> https://reviews.apache.org/r/44469/
>
> Fixed a bug that causes the task stuck in staging state (still under
> review).
> https://reviews.apache.org/r/44435/
>
> On Tue, Mar 8, 2016 at 10:30 AM, Kevin Klues  wrote:
> > Yes, will do.
> >
> > On Tue, Mar 8, 2016 at 10:26 AM, Vinod Kone 
> wrote:
> >> +kevin klues
> >>
> >> OK. I'm cancelling this vote since there are some show stopper issues
> that
> >> we need to cherry-pick. I'll cut another RC on Thursday.
> >>
> >> @shepherds: can you please make sure the blocker tickets are marked with
> >> fix version and that they land today or tomorrow?
> >>
> >> @kevin: since you have volunteered to help with the release, can you
> make
> >> sure we have a list of commits to cherry pick for rc2?
> >>
> >> Thanks,
> >>
> >>
> >> On Tue, Mar 8, 2016 at 12:05 AM, Shuai Lin 
> wrote:
> >>
> >>> Maybe also https://issues.apache.org/jira/browse/MESOS-4877 and
> >>> https://issues.apache.org/jira/browse/MESOS-4878 ?
> >>>
> >>>
> >>> On Tue, Mar 8, 2016 at 9:13 AM, Jie Yu  wrote:
> >>>
>  I'd like to fix https://issues.apache.org/jira/browse/MESOS-4888 as
> well
>  if you guys plan to cut another RC
> 
>  On Mon, Mar 7, 2016 at 10:16 AM, Daniel Osborne <
>  daniel.osbo...@metaswitch.com> wrote:
> 
> > -1
> >
> > If it doesn’t cause too much pain, I'm hoping we can squeeze a
> > relatively small patch which restores Mesos' ability to extract
> Docker
> > assigned IPs. This has been broken with Docker 1.10's release over
> a month
> > ago, and prevents service discovery and DNS from working.
> >
> > Mesos-4370: https://issues.apache.org/jira/browse/MESOS-4370
> > RB# 43093: https://reviews.apache.org/r/43093/
> >
> > I've built 0.28.0-rc1 with this patch and can confirm that it fixes
> it
> > as expected.
> >
> > Apologies for not bringing this to attention earlier.
> >
> > Thanks all,
> > Dan
> >
> > -Original Message-
> > From: Vinod Kone [mailto:vinodk...@apache.org]
> > Sent: Thursday, March 3, 2016 5:44 PM
> > To: dev ; user 
> > Subject: [VOTE] Release Apache Mesos 0.28.0 (rc1)
> >
> > Hi all,
> >
> >
> > Please vote on releasing the following candidate as Apache Mesos
> 0.28.0.
> >
> >
> > 0.28.0 includes the following:
> >
> >
> >
> 
> >
> >   * [MESOS-4343] - A new cgroups isolator for enabling the net_cls
> > subsystem in
> >
> > Linux. The cgroups/net_cls isolator allows operators to provide
> > network
> >
> >
> > performance isolation and network segmentation for containers
> within
> > a Mesos
> >
> > cluster. To enable the cgroups/net_cls isolator, append
> > `cgroups/net_cls` to
> >
> > the `--isolation` flag when starting the slave. Please refer to
> >
> >
> > docs/mesos-containerizer.md for more details.
> >
> >
> >
> >
> >
> >   * [MESOS-4687] - The implementation of scalar resource values
> (e.g.,
> > "2.5
> >
> >
> > CPUs") has changed. Mesos now reliably supports resources with
> up to
> > three
> >
> > decimal digits of precision (e.g., "2.501 CPUs"); resources with
> > more than
> >
> > three decimal digits of precision will be rounded. Internally,
> > resource math
> >
> > is now done using a fixed-point format that supports three
> decimal
> > digits of
> >
> > precision, and then converted to/from floating point for input
> and
> > output,
> >
> > respectively. Frameworks 

Re: Executors no longer inherit environment variables from the agent

[Gilbert Song]

Yes, `LIBPROCESS_IP` will be excepted from this change. We will still have
`LIBPROCESS_IP` set and passed to executors' environment, which is for the
case that DNS is not available on the slave.

Gilbert

On Tue, Mar 8, 2016 at 11:57 AM, Zhitao Li <zhitaoli...@gmail.com> wrote:

> Is LIBPROCESS_IP going to be an exception to this? Some executors are
> using this variable as an alternative of implementing their own IP
> detection logic AFAIK so this behavior would break them.
>
> On Tue, Mar 8, 2016 at 11:33 AM, Gilbert Song <gilb...@mesosphere.io>
> wrote:
>
>> Hi,
>>
>> TL;DR Executors will no longer inherit environment variables from the
>> agent by default in 0.30.
>>
>> Currently, executors are inheriting environment variables form the agent
>> in mesos containerizer by default. This is an unfortunate legacy behavior
>> and is insecure. If you do have environment variables that you want to pass
>> to the executors, you can set it explicitly by using the
>> `--executor_environment_variables` agent flag.
>>
>> Starting from 0.30, we will no longer allow executors to inherit
>> environment variables from the agent. In other words,
>> `--executor_environment_variables` will be set to “{}” by default. If you
>> do depend on the original behavior, please set
>> `--executor_environment_variables` flag explicitly.
>>
>> Let us know if you have any comments or concerns.
>>
>> Thanks,
>> Gilbert
>>
>
>
>
> --
> Cheers,
>
> Zhitao Li
>

Executors no longer inherit environment variables from the agent

[Gilbert Song]

Hi,

TL;DR Executors will no longer inherit environment variables from the agent
by default in 0.30.

Currently, executors are inheriting environment variables form the agent in
mesos containerizer by default. This is an unfortunate legacy behavior and
is insecure. If you do have environment variables that you want to pass to
the executors, you can set it explicitly by using the
`--executor_environment_variables` agent flag.

Starting from 0.30, we will no longer allow executors to inherit
environment variables from the agent. In other words,
`--executor_environment_variables` will be set to “{}” by default. If you
do depend on the original behavior, please set
`--executor_environment_variables` flag explicitly.

Let us know if you have any comments or concerns.

Thanks,
Gilbert

Re: Making 'curl' a prerequisite for installing Mesos

[Gilbert Song]

+1

On Thu, Mar 3, 2016 at 9:13 AM, Bernd Mathiske  wrote:

> +1
>
> On Mar 3, 2016, at 6:10 PM, Jie Yu  wrote:
>
> Hi,
>
> I am proposing making 'curl' a prerequisite when installing Mesos.
> Currently, we require 'libcurl' being present when installing Mesos (
> http://mesos.apache.org/gettingstarted/). However, we found that it does
> not compose well with our asynchronous runtime environment (i.e., it'll
> block the current worker thread).
>
> Recent work on URI fetcher
>  uses 'curl' directly,
> instead of using 'libcurl' to fetch artifacts, because it composes well
> with our async runtime env. 'curl' is installed by default in most systems
> (e.g., OSX, centos, RHEL).
>
> So I am proposing adding 'curl' to our prerequisite list. Let me know if
> you have any concern on this. I'll update the Getting Started doc if you
> are OK with this change.
>
> Thanks,
> - Jie
>
>
>

Re: [3/3] mesos git commit: Plugged in docker runtime isolator.

[Gilbert Song]

Thank you guys. Should have been more careful to avoid this.

On Wed, Feb 3, 2016 at 9:17 PM, Jie Yu <yujie@gmail.com> wrote:

> Thanks James. Just committed a fix.
>
> On Wed, Feb 3, 2016 at 8:54 PM, James Peach <jor...@gmail.com> wrote:
>
> >
> > > On Feb 3, 2016, at 5:45 PM, ji...@apache.org wrote:
> > >
> > > Plugged in docker runtime isolator.
> > >
> > > Review: https://reviews.apache.org/r/43036/
> > >
> > >
> > > Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
> > > Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/0b0a3dc5
> > > Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/0b0a3dc5
> > > Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/0b0a3dc5
> > >
> > > Branch: refs/heads/master
> > > Commit: 0b0a3dc5467224511b1963dd0ac530bca7506376
> > > Parents: 2d5d14f
> > > Author: Gilbert Song <songzihao1...@gmail.com>
> > > Authored: Wed Feb 3 17:14:23 2016 -0800
> > > Committer: Jie Yu <yujie@gmail.com>
> > > Committed: Wed Feb 3 17:14:23 2016 -0800
> > >
> > > --
> > > src/slave/containerizer/mesos/containerizer.cpp | 12 +++-
> > > .../mesos/isolators/docker/runtime.cpp  | 30
> ++--
> > > 2 files changed, 39 insertions(+), 3 deletions(-)
> > > --
> > >
> > >
> > >
> >
> http://git-wip-us.apache.org/repos/asf/mesos/blob/0b0a3dc5/src/slave/containerizer/mesos/containerizer.cpp
> > > --
> > > diff --git a/src/slave/containerizer/mesos/containerizer.cpp
> > b/src/slave/containerizer/mesos/containerizer.cpp
> > > index 5f8b6c7..12294cd 100644
> > > --- a/src/slave/containerizer/mesos/containerizer.cpp
> > > +++ b/src/slave/containerizer/mesos/containerizer.cpp
> > > @@ -63,6 +63,10 @@
> > > #endif
> > >
> > > #ifdef __linux__
> > > +#include "slave/containerizer/mesos/isolators/docker/runtime.hpp"
> > > +#endif
> > > +
> > > +#ifdef __linux__
> > > #include "slave/containerizer/mesos/isolators/filesystem/linux.hpp"
> > > #endif
> > > #include "slave/containerizer/mesos/isolators/filesystem/posix.hpp"
> > > @@ -210,6 +214,7 @@ Try<MesosContainerizer*>
> MesosContainerizer::create(
> > > {"cgroups/mem", ::create},
> > > {"cgroups/net_cls", ::create},
> > > {"cgroups/perf_event", ::create},
> > > +{"docker/runtime", ::create},
> > > {"namespaces/pid", ::create},
> > > #endif
> > > #ifdef WITH_NETWORK_ISOLATOR
> > > @@ -839,7 +844,12 @@ Future<list<Option>>
> > MesosContainerizerProcess::prepare(
> > >   }
> > >
> > >   if (provisionInfo.isSome()) {
> > > -containerConfig.set_rootfs(provisionInfo.get().rootfs);
> > > +containerConfig.set_rootfs(provisionInfo->rootfs);
> > > +
> > > +if (provisionInfo->dockerManifest.isSome()) {
> > > +  ContainerConfig::Docker* docker =
> > containerConfig.mutable_docker();
> > > +
> >
> docker->mutable_manifest()->CopyFrom(provisionInfo->dockerManifest.get());
> > > +}
> > >   }
> > >
> > >   // We prepare the isolators sequentially according to their ordering
> > >
> > >
> >
> http://git-wip-us.apache.org/repos/asf/mesos/blob/0b0a3dc5/src/slave/containerizer/mesos/isolators/docker/runtime.cpp
> > > --
> > > diff --git a/src/slave/containerizer/mesos/isolators/docker/runtime.cpp
> > b/src/slave/containerizer/mesos/isolators/docker/runtime.cpp
> > > index f5f9678..5189a6d 100644
> > > --- a/src/slave/containerizer/mesos/isolators/docker/runtime.cpp
> > > +++ b/src/slave/containerizer/mesos/isolators/docker/runtime.cpp
> > > @@ -16,6 +16,10 @@
> > >
> > > #include 
> > >
> > > +#include 
> > > +
> > > +#include 
> > > +
> > > #include "slave/flags.hpp"
> > >
> > > #include "slave/containerizer/mesos/isolators/docker/runtime.hpp"
> > > @@ -48,7 +52,10 @@
> > DockerRuntimeIsolatorProcess::~DockerRuntimeIsolatorPro

Re: Fetcher refactor proposal

[Gilbert Song]

+1.
Depending on one of unified container issue MESOS-2970
, We need
to support cache for container image as our next step. It would be perfect
if we can reuse some parts of fetcher cache. It may also be helpful to
many other issues related to caching.

Thank you for writing up the proposal above!

Gilbert

On Tue, Nov 10, 2015 at 3:45 PM, Jie Yu  wrote:

> Hi,
>
> Fetcher was originally designed to fetch CommandInfo::URIs (e.g., executor
> binary) for executors/tasks. A recent refactor (MESOS-336
> ) added caching support
> to
> the fetcher. The recent work on filesystem isolation/unified containerizer
> (
> MESOS-2840 ) requires
> Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The
> natural question is: can we leverage the fetcher to fetch those filesystem
> images (and cache them accordingly)? Unfortunately, the existing fetcher
> interface is tightly coupled with CommandInfo::URIs for executors/tasks,
> making it very hard to be re-used to fetch/cache filesystem images.
>
> Another motivation for the refactor is that we want to extend the fetcher
> to support more types of schemes. For instance, we want to support magnet
> URI to enable p2p fetching. This is in fact quite important for operating a
> large cluster (MESOS-3596 <
> https://issues.apache.org/jira/browse/MESOS-3596>).
> The goal here is to allow fetcher to be extended (e.g., using modules) so
> that operators can add custom fetching support.
>
> I proposed a solution in this doc
> <
> https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVofcLWqco/edit?usp=sharing
> >.
> The main idea is to decouple artifacts fetching from artifacts cache
> management. We can make artifacts fetching extensible (e.g. to support p2p
> fetching), and solve the cache management part later.
>
> Let me know your thoughts! Thanks!
>
> - Jie
>