Re: Separate Compilation of Tests

[Michael Park]

On Tue, Sep 27, 2016 at 12:21 PM, Alexander Rojas 
wrote:

> The workaround I use is to go to the folder of the makefile root
> directory, so for
> Mesos, and then use the specific target, for example:
>
> For mesos-tests
> cd ${MESOS_SRC_ROOT}/build/src
> make -j4 mesos-tests && ./mesos-tests
>
> for libprocess
> cd ${MESOS_SRC_ROOT}/build/3rdparty/libprocess
> make -j4 libprocess-tests && ./libprocess-tests
>
> for stout
> cd ${MESOS_SRC_ROOT}/build/3rdparty/stout
> make -j4 stout-tests && ./stout-tests
>

Alexander, I'm aware of these targets, and I guess my claim is that these
are not fine-grained enough.
Yes, I can work on a stout change and only run stout tests to iterate
without compiling/running all of the mesos tests.

But I'm talking about times when I'm making mesos changes (let's say
involving a stout change),
I don't want to have to recompile basically all of the tests per iteration.


> I don’t think it would be hard to add top level targets that just do that
> from the top level.
>

I imagine it's probably not either, but it's not what I'm looking for in
this thread.

I however would like to have a target that compiles but not runs the tests.
>

What do you mean by this? Are you looking for `make tests`?
As far as I'm aware, you can do `make tests`, and `make check` at every
level,
as opposed to the commands you're running to achieve what seems like the
same thing to me.


> > On 25 Sep 2016, at 15:56, Michael Park  wrote:
> >
> > Hello,
> >
> > I would like to propose a change in our build to help us improve
> developer
> > efficiency.
> > The proposal is to support separate compilation of unit tests.
> >
> > Currently, we have the old approach of invoking `make check -j N
> > GTEST_FILTER=""`, or a newer option of doing `make tests -j N`. From what
> > I've heard the latter is slightly faster. However, when someone is
> > developing a specific feature, it's likely that they would like to make
> > changes and iterate on a single test file. In this workflow, having to
> > compile (virtually) __all__ of the tests is very burdensome. This
> situation
> > is not so bad if you're working in a very isolated part of the codebase,
> > but it gets to be pretty bad if you're experimenting with parts that are
> > widely used.
> >
> > An example of a workflow I'm aiming for would look something like:
> >
> >   1. write some code...
> >   2. `make check master_tests`  // compile and test
> >   `src/tests/master_tests.cpp`
> >   3. fix compilation errors...
> >   4. `make check master_tests`  // compile and test
> >   `src/tests/master_tests.cpp`
> >   5. change some stuff...
> >   6. `make check master_tests`  // compile and test
> >   `src/tests/master_tests.cpp`
> >   7. debug...
> >   8. `make check master_tests`  // compile and test
> >   `src/tests/master_tests.cpp`
> >   9. alright, looks good. `make check`
> >
> > I have 0 attachment to the `make check master_tests` syntax. It'll be a
> > different syntax for CMake anyways. I just think that the ability to
> > perform separate compilation of tests will be immensely useful.
> >
> > Some numbers to justify what I'm proposing:
> >
> >   - `make -j 8` on my laptop takes roughly 10 mins.
> >   - `make tests -j 8` takes about 30 mins.
> >
> > Of course, not every change you make triggers all of the tests to
> > recompile. But if you change components that are widely used, it does end
> > up recompiling virtually everything. Under these circumstances, I would
> > love for each iteration to be 11 mins (10 mins + __at most__ 1 min to
> > compile the single test), rather than 30 mins.
> >
> > NOTE: My laptop is expensive... some people even use machines with 64
> cores
> > or whatever to compile Mesos. Not everyone has access to this kind of
> > equipment. We should strive for something better than "throw more money
> at
> > it".
> >
> > The goal of this thread for me is the following:
> >  (1) Capture whether this is something most people experience, or whether
> > I'm just doing it wrong
> >  (2) If most people do experience this inefficiency and would like this
> > change to be made,
> >   I would like to recruit 1 or 2 people to help me deliver this,
> since
> > I'm not a automake nor CMake expert.
>
>

Re: Separate Compilation of Tests

[Benjamin Bannier]

Hi,

being able to iterate more rapidly on tests sounds great.

I am slightly unsure about the cost of (i) linking even more binaries, and (ii)
the overhead of setting up the test environment for the invocations of test
binaries (I believe this was O(100ms) per `main` at some point).

I believe if one doesn't mind working with uncommitted changes one could
already now get half-way to you spot you desire by removing `SOURCE`
dependencies one doesn't care about from `mesos_tests`. At that point all one
is left with is the test case, the file containing the test `main`, and
infrastructure pieces. Since in the past we weren't super careful about cutting
these parts into components, figuring out the infrastructure code one actually
needs can be a bit tricky (and one would likely err on the side of pulling in
more than needed).

I think a slightly less disruptive and possibly incremental plan would be to
(1) clearly separate on the source level always-required infrastructure
functions and classes from more specific pieces, and (2) possibly moving
related pieces to convenience libraries. With that it should be possible to
quickly define new test programs with slimmed down dependencies while working
on a feature.  If we wanted we would still be able to add these to a single
project-wide binary like we do now when actually committing.

I think all work in this direction would be strictly cleanup, and could be done
without requiring us to change the way we perform tests, but along the way lay
the foundation for e.g., multiple test binaries, or allow us to expose parts of
the test infrastructure to outside users (e.g., tests of modules).


b.

Re: Two important tickets for improving documentation

[Timothy Anderegg]

So a few days turned into a few months, but Mesos-3011 (versioned
documentation on the website) is ready for review if anyone has time,
thanks!  https://issues.apache.org/jira/browse/MESOS-3011

-Tim

On Thu, Jun 30, 2016 at 9:43 PM Timothy Anderegg 
wrote:

> Yep, thanks Vinod!
>
> On Thu, Jun 30, 2016, 9:24 PM Benjamin Mahler  wrote:
>
>> Looks like Vinod will shepherd, thanks Vinod!
>>
>> On Tue, Jun 28, 2016 at 4:38 PM, Timothy Anderegg <
>> timothy.ander...@gmail.com> wrote:
>>
>> > I picked it up as part of the website working group, but don't have a
>> > shepherd yet.
>> >
>> > On Tue, Jun 28, 2016, 7:06 PM Benjamin Mahler 
>> wrote:
>> >
>> > > Thanks a lot Timothy!
>> > >
>> > > Do you have a shepherd or do you still need one?
>> > >
>> > > On Tue, Jun 28, 2016 at 7:19 AM, Timothy Anderegg <
>> > > timothy.ander...@gmail.com> wrote:
>> > >
>> > > > Hi Ben - I'm actually almost done with Mesos-3011, will open a
>> review
>> > > > request in the next few days.  Only a few last issues to resolve.
>> > > >
>> > > > On Mon, Jun 27, 2016, 11:20 PM Benjamin Mahler 
>> > > wrote:
>> > > >
>> > > > > Just wanted to highlight two tickets that would be of great help
>> to
>> > the
>> > > > > community.
>> > > > >
>> > > > > (1) The first is to publish documentation on the website for each
>> > > > release.
>> > > > > We currently publish periodically off of master, and so it's a
>> pain
>> > for
>> > > > > users because the documentation does not reflect the version they
>> are
>> > > > > running:
>> > > > >
>> > > > > https://issues.apache.org/jira/browse/MESOS-3011
>> > > > >
>> > > > > Once we have the tooling we can publish the documentation for
>> > previous
>> > > > > releases.
>> > > > >
>> > > > > (2) The second is to automate the generation of docs/
>> > configuration.md.
>> > > > As
>> > > > > far as I know, we're still generating this manually, which makes
>> it a
>> > > > huge
>> > > > > pain and consequently it becomes stale:
>> > > > >
>> > > > > https://issues.apache.org/jira/browse/MESOS-3019
>> > > > >
>> > > > > This would be done in a similar way to how Kevin Klues added a
>> > support
>> > > > > script for generating the help endpoint documentation.
>> > > > >
>> > > > > Vinod, looks like we need to do an update of configuration.md for
>> > 1.0?
>> > > > >
>> > > > > Ben
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: Separate Compilation of Tests

[Alexander Rojas]

The workaround I use is to go to the folder of the makefile root directory, so 
for
Mesos, and then use the specific target, for example:

For mesos-tests
cd ${MESOS_SRC_ROOT}/build/src
make -j4 mesos-tests && ./mesos-tests

for libprocess
cd ${MESOS_SRC_ROOT}/build/3rdparty/libprocess
make -j4 libprocess-tests && ./libprocess-tests

for stout
cd ${MESOS_SRC_ROOT}/build/3rdparty/stout
make -j4 stout-tests && ./stout-tests

I don’t think it would be hard to add top level targets that just do that from 
the top level.

I however would like to have a target that compiles but not runs the tests.

> On 25 Sep 2016, at 15:56, Michael Park  wrote:
> 
> Hello,
> 
> I would like to propose a change in our build to help us improve developer
> efficiency.
> The proposal is to support separate compilation of unit tests.
> 
> Currently, we have the old approach of invoking `make check -j N
> GTEST_FILTER=""`, or a newer option of doing `make tests -j N`. From what
> I've heard the latter is slightly faster. However, when someone is
> developing a specific feature, it's likely that they would like to make
> changes and iterate on a single test file. In this workflow, having to
> compile (virtually) __all__ of the tests is very burdensome. This situation
> is not so bad if you're working in a very isolated part of the codebase,
> but it gets to be pretty bad if you're experimenting with parts that are
> widely used.
> 
> An example of a workflow I'm aiming for would look something like:
> 
>   1. write some code...
>   2. `make check master_tests`  // compile and test
>   `src/tests/master_tests.cpp`
>   3. fix compilation errors...
>   4. `make check master_tests`  // compile and test
>   `src/tests/master_tests.cpp`
>   5. change some stuff...
>   6. `make check master_tests`  // compile and test
>   `src/tests/master_tests.cpp`
>   7. debug...
>   8. `make check master_tests`  // compile and test
>   `src/tests/master_tests.cpp`
>   9. alright, looks good. `make check`
> 
> I have 0 attachment to the `make check master_tests` syntax. It'll be a
> different syntax for CMake anyways. I just think that the ability to
> perform separate compilation of tests will be immensely useful.
> 
> Some numbers to justify what I'm proposing:
> 
>   - `make -j 8` on my laptop takes roughly 10 mins.
>   - `make tests -j 8` takes about 30 mins.
> 
> Of course, not every change you make triggers all of the tests to
> recompile. But if you change components that are widely used, it does end
> up recompiling virtually everything. Under these circumstances, I would
> love for each iteration to be 11 mins (10 mins + __at most__ 1 min to
> compile the single test), rather than 30 mins.
> 
> NOTE: My laptop is expensive... some people even use machines with 64 cores
> or whatever to compile Mesos. Not everyone has access to this kind of
> equipment. We should strive for something better than "throw more money at
> it".
> 
> The goal of this thread for me is the following:
>  (1) Capture whether this is something most people experience, or whether
> I'm just doing it wrong
>  (2) If most people do experience this inefficiency and would like this
> change to be made,
>   I would like to recruit 1 or 2 people to help me deliver this, since
> I'm not a automake nor CMake expert.