Re: LXC Container Support

[Tom Barber]

Thanks guys.  I do a lot of work with the juju ecosystem and lxc containers
to deploy workloads with that, for example multi node big top clusters etc.
It's far more flexible using LXC system containers than pre configured app
containers. This works on cloud and bare metal but I'd also like to be able
to deploy containers to Mesos using the same technology stack.

Cheers
Tom

On 6 Oct 2016 00:51, "Jie Yu"  wrote:

> Tom,
>
> I'm looking to add LXC/LXD support, POC initially then hopefully bake it
> > into some Mesos libraries. From what I can tell Mesos doesn't have hooks
> > for LXC in the new container setup and also the External Shim that was
> > underdevelopment is Deprecated. Sound roughly accurate so far?
>
>
> Yup. External containerizer has been deprecated and removed. The direction
> is to maintain only one containerizer in the future, and make that
> containerizer better and supports various containerization requests.
>
> If so, would extending the Unified Containerizer be the correct way to go
> > about adding LXC/LXD support?
>
>
> Yes, I would prefer exploring this direction. But even before that, it
> would be nice to understand the motivation for using lxd. I'd prefer we
> find out the feature gap and augment unified containerizer to fill the gap.
>
> - Jie
>
> On Wed, Oct 5, 2016 at 3:10 PM, Tom Barber 
> wrote:
>
> > Hello folks,
> >
> > I've been Googling Mesos container support a bit recently, I see support
> > for Mesos(AppC) and Docker in the latest versions.
> >
> > I'm looking to add LXC/LXD support, POC initially then hopefully bake it
> > into some Mesos libraries. From what I can tell Mesos doesn't have hooks
> > for LXC in the new container setup and also the External Shim that was
> > underdevelopment is Deprecated. Sound roughly accurate so far?
> >
> > If so, would extending the Unified Containerizer be the correct way to go
> > about adding LXC/LXD support? (
> > http://mesos.apache.org/documentation/latest/container-image/)
> >
> > Thanks
> >
> > Tom
> >
>

Re: 1.0.2 release

[Vinod Kone]

Release dashboard:
https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12329719

I'm waiting for 2 issues to be resolved. Once that's done, I'll start
prepping the release.

On Wed, Oct 5, 2016 at 4:11 PM, Vinod Kone  wrote:

> Hi,
>
> As the Release Manager for 1.0, I'm responsible for all subsequent patch
> releases.
>
> I'm planning to cut the next patch release (1.0.2) within a week. So, if
> you have any patches that need to get into 1.0.2 make sure that either it
> is already in the 1.0.x branch or the corresponding ticket has a target
> version set to 1.0.2.
>
> I'll send a link to the release dashboard shortly.
>
> Thanks,
> -- Vinod
>

Re: LXC Container Support

[Jie Yu]

Tom,

I'm looking to add LXC/LXD support, POC initially then hopefully bake it
> into some Mesos libraries. From what I can tell Mesos doesn't have hooks
> for LXC in the new container setup and also the External Shim that was
> underdevelopment is Deprecated. Sound roughly accurate so far?


Yup. External containerizer has been deprecated and removed. The direction
is to maintain only one containerizer in the future, and make that
containerizer better and supports various containerization requests.

If so, would extending the Unified Containerizer be the correct way to go
> about adding LXC/LXD support?


Yes, I would prefer exploring this direction. But even before that, it
would be nice to understand the motivation for using lxd. I'd prefer we
find out the feature gap and augment unified containerizer to fill the gap.

- Jie

On Wed, Oct 5, 2016 at 3:10 PM, Tom Barber  wrote:

> Hello folks,
>
> I've been Googling Mesos container support a bit recently, I see support
> for Mesos(AppC) and Docker in the latest versions.
>
> I'm looking to add LXC/LXD support, POC initially then hopefully bake it
> into some Mesos libraries. From what I can tell Mesos doesn't have hooks
> for LXC in the new container setup and also the External Shim that was
> underdevelopment is Deprecated. Sound roughly accurate so far?
>
> If so, would extending the Unified Containerizer be the correct way to go
> about adding LXC/LXD support? (
> http://mesos.apache.org/documentation/latest/container-image/)
>
> Thanks
>
> Tom
>

Re: LXC Container Support

[Vinod Kone]

Hi Tom,

Your description of the current state of art is on point.

Regarding support for LXC/LXD, what exactly does it mean? Is there an image
spec for those that you want Mesos to support? What's your use case?

Thanks,

On Wed, Oct 5, 2016 at 3:10 PM, Tom Barber  wrote:

> Hello folks,
>
> I've been Googling Mesos container support a bit recently, I see support
> for Mesos(AppC) and Docker in the latest versions.
>
> I'm looking to add LXC/LXD support, POC initially then hopefully bake it
> into some Mesos libraries. From what I can tell Mesos doesn't have hooks
> for LXC in the new container setup and also the External Shim that was
> underdevelopment is Deprecated. Sound roughly accurate so far?
>
> If so, would extending the Unified Containerizer be the correct way to go
> about adding LXC/LXD support? (
> http://mesos.apache.org/documentation/latest/container-image/)
>
> Thanks
>
> Tom
>

1.0.2 release

[Vinod Kone]

Hi,

As the Release Manager for 1.0, I'm responsible for all subsequent patch
releases.

I'm planning to cut the next patch release (1.0.2) within a week. So, if
you have any patches that need to get into 1.0.2 make sure that either it
is already in the 1.0.x branch or the corresponding ticket has a target
version set to 1.0.2.

I'll send a link to the release dashboard shortly.

Thanks,
-- Vinod

LXC Container Support

[Tom Barber]

Hello folks,

I've been Googling Mesos container support a bit recently, I see support
for Mesos(AppC) and Docker in the latest versions.

I'm looking to add LXC/LXD support, POC initially then hopefully bake it
into some Mesos libraries. From what I can tell Mesos doesn't have hooks
for LXC in the new container setup and also the External Shim that was
underdevelopment is Deprecated. Sound roughly accurate so far?

If so, would extending the Unified Containerizer be the correct way to go
about adding LXC/LXD support? (
http://mesos.apache.org/documentation/latest/container-image/)

Thanks

Tom

Notification: Community Meeting @ Thu Oct 6, 2016 3pm - 4pm PST (Apache Mesos)

[Michael Park]

[GitHub] mesos pull request #168: Update contributors.yaml for gradywang.

[vinodkone]

Github user vinodkone commented on a diff in the pull request:

https://github.com/apache/mesos/pull/168#discussion_r82063596
  
--- Diff: docs/contributors.yaml ---
@@ -402,13 +402,14 @@
   jira_user: yanyanhu
   reviewboard_user: yanyanhu
 
-- name: Yong Qiao Wang
--- End diff --

oh ok. then a single entry sounds fine.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Separate Compilation of Tests

[Joris Van Remoortere]

@mpark
Can you clarify if your goal is to:
1) refactor the source code (maybe including things like cluster.hpp) so
that general compilation of any file is faster. Examples include allowing
non-header-only stout, splitting out some files, moving some more stuff to
cpps.
or 2) splitting out the files in the cmake / autoconf so that it's possible
to compile a single test file?

Although 2) is probably the easiest way to improved performance I was under
the impression there were some arguments ~2 years ago around wanting to
have all the tests in a single binary.

I think any effort spent 1) will be valuable regardless. We've talked about
a bunch of tasks in #1 and just needed time / commitment to them.

Some people have also suggested improved tooling. For example the gold
linker.
You should also see if you are ever IO bound. Debug builds generate
binaries so large that you actually can become IO bound. There are ways to
get around that too. There have been times where an optimized build has
been faster than a debug build.

Joris

—
*Joris Van Remoortere*
Mesosphere

On Tue, Sep 27, 2016 at 6:25 PM, Michael Park  wrote:

> On Tue, Sep 27, 2016 at 12:21 PM, Alexander Rojas  >
> wrote:
>
> > The workaround I use is to go to the folder of the makefile root
> > directory, so for
> > Mesos, and then use the specific target, for example:
> >
> > For mesos-tests
> > cd ${MESOS_SRC_ROOT}/build/src
> > make -j4 mesos-tests && ./mesos-tests
> >
> > for libprocess
> > cd ${MESOS_SRC_ROOT}/build/3rdparty/libprocess
> > make -j4 libprocess-tests && ./libprocess-tests
> >
> > for stout
> > cd ${MESOS_SRC_ROOT}/build/3rdparty/stout
> > make -j4 stout-tests && ./stout-tests
> >
>
> Alexander, I'm aware of these targets, and I guess my claim is that these
> are not fine-grained enough.
> Yes, I can work on a stout change and only run stout tests to iterate
> without compiling/running all of the mesos tests.
>
> But I'm talking about times when I'm making mesos changes (let's say
> involving a stout change),
> I don't want to have to recompile basically all of the tests per iteration.
>
>
> > I don’t think it would be hard to add top level targets that just do that
> > from the top level.
> >
>
> I imagine it's probably not either, but it's not what I'm looking for in
> this thread.
>
> I however would like to have a target that compiles but not runs the tests.
> >
>
> What do you mean by this? Are you looking for `make tests`?
> As far as I'm aware, you can do `make tests`, and `make check` at every
> level,
> as opposed to the commands you're running to achieve what seems like the
> same thing to me.
>
>
> > > On 25 Sep 2016, at 15:56, Michael Park  wrote:
> > >
> > > Hello,
> > >
> > > I would like to propose a change in our build to help us improve
> > developer
> > > efficiency.
> > > The proposal is to support separate compilation of unit tests.
> > >
> > > Currently, we have the old approach of invoking `make check -j N
> > > GTEST_FILTER=""`, or a newer option of doing `make tests -j N`. From
> what
> > > I've heard the latter is slightly faster. However, when someone is
> > > developing a specific feature, it's likely that they would like to make
> > > changes and iterate on a single test file. In this workflow, having to
> > > compile (virtually) __all__ of the tests is very burdensome. This
> > situation
> > > is not so bad if you're working in a very isolated part of the
> codebase,
> > > but it gets to be pretty bad if you're experimenting with parts that
> are
> > > widely used.
> > >
> > > An example of a workflow I'm aiming for would look something like:
> > >
> > >   1. write some code...
> > >   2. `make check master_tests`  // compile and test
> > >   `src/tests/master_tests.cpp`
> > >   3. fix compilation errors...
> > >   4. `make check master_tests`  // compile and test
> > >   `src/tests/master_tests.cpp`
> > >   5. change some stuff...
> > >   6. `make check master_tests`  // compile and test
> > >   `src/tests/master_tests.cpp`
> > >   7. debug...
> > >   8. `make check master_tests`  // compile and test
> > >   `src/tests/master_tests.cpp`
> > >   9. alright, looks good. `make check`
> > >
> > > I have 0 attachment to the `make check master_tests` syntax. It'll be a
> > > different syntax for CMake anyways. I just think that the ability to
> > > perform separate compilation of tests will be immensely useful.
> > >
> > > Some numbers to justify what I'm proposing:
> > >
> > >   - `make -j 8` on my laptop takes roughly 10 mins.
> > >   - `make tests -j 8` takes about 30 mins.
> > >
> > > Of course, not every change you make triggers all of the tests to
> > > recompile. But if you change components that are widely used, it does
> end
> > > up recompiling virtually everything. Under these circumstances, I would
> > > love for each iteration to be 11 mins (10 mins + __at most__ 1 min to
> > > compile the single test), rather than 30 mins.
> > >

Re: Mapped diagnostics context - Adding internal Mesos IDs as context to the logs

[Joris Van Remoortere]

>
> This is technically feasible:
> //
> // Specify an "extension" added to the filename specified via
> // SetLogDestination.  This applies to all severity levels.  It's
> // often used to append the port we're listening on to the logfile
> // name.  Thread-safe.
> //
> GOOGLE_GLOG_DLL_DECL void SetLogFilenameExtension(
> const char* filename_extension);


@benm @vinod @benh what are your thoughts on actually doing it?

—
*Joris Van Remoortere*
Mesosphere

On Tue, Oct 4, 2016 at 2:39 PM, Zameer Manji  wrote:

> I don't know if this is feasible or not, but I would be a strong +1 to
> this.
>
> This would make tracing failures much easier.
>
> On Tue, Oct 4, 2016 at 6:12 AM, Frank Scholten 
> wrote:
>
> > Hi,
> >
> > On JIRA I found several issues about making logs less spammy as well
> > as making them easier to understand for day to day operations.
> >
> > https://issues.apache.org/jira/browse/MESOS-4432 Condense (redundant)
> > log messages related to task launch/status/finish
> > https://issues.apache.org/jira/browse/MESOS-5467 offer DECLINE /
> > ACCEPT + Recovered resource messages are spammy
> > https://issues.apache.org/jira/browse/MESOS-4430 Identify and change
> > logging level for message that don't contain specific
> > task/framework/slave info
> >
> > Besides reducing the logs would it be possible to add more context? In
> > the Java world the technique of 'Mapped diagnostic context' is used
> > with Logback where each log line contains a few fields with context.
> > See http://logback.qos.ch/manual/mdc.html
> >
> > To translate this to Mesos how about adding the internal IDs such as
> > agent, framework, and task IDs at the beginning of the log, so this
> > information is separated from the textual, human readble log message.
> > At the moment this information is tangled which makes it hard to
> > interpret, especially when there are so many logs for each task.
> >
> > For example, change this
> >
> > I1004 12:24:12.118803  3780 status_update_manager.cpp:320] Received
> > status update TASK_FAILED (UUID: a1d03948-30bf-46b3-9599-cfcfc7cbc27b)
> > for task weave-demo_database_catalogue-db.6d415c17-8a2d-11e6-90a7-
> > 0242458f9469
> > of framework f1546295-ab46-496a-8cf9-91756fece4ed-
> >
> > to
> >
> > I1004 12:24:12.118803  3780 status_update_manager.cpp:320
> > A:agent12318032910 F:f1546295-ab46-496a-8cf9-91756fece4ed-
> > T:xyz-a1d03948-30bf-46b3-9599-cfcfc7cbc27b] Received status update
> > TASK_FAILED for task 'xyz'
> >
> > In this case the header contains A:$AGENT_ID, F:$FRAMEWORK_ID,
> > T:$TASK_ID and as the context and lines with similar context can be
> > more easily correlated visually.
> >
> > Is this feasible?
> >
> > Cheers,
> >
> > Frank
> >
> > --
> > Zameer Manji
> >
>

[GitHub] mesos pull request #168: Update contributors.yaml for gradywang.

[gradywang]

Github user gradywang commented on a diff in the pull request:

https://github.com/apache/mesos/pull/168#discussion_r81929327
  
--- Diff: docs/contributors.yaml ---
@@ -402,13 +402,14 @@
   jira_user: yanyanhu
   reviewboard_user: yanyanhu
 
-- name: Yong Qiao Wang
--- End diff --

@vinodkone I also used `gradywang` as my reviewboard username  rather than 
`JamesYongQiaoWang` when I in IBM in main contributions. For my name, it does 
not changed, and just change to the recommended format. So can I only change 
the old entry?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---