On Wed, Oct 5, 2016 at 9:22 AM, Joris Van Remoortere <[email protected]> wrote:
> @mpark > Can you clarify if your goal is to: > 1) refactor the source code (maybe including things like cluster.hpp) so > that general compilation of any file is faster. Examples include allowing > non-header-only stout, splitting out some files, moving some more stuff to > cpps. > or 2) splitting out the files in the cmake / autoconf so that it's possible > to compile a single test file? In this thread I was trying to capture (2). Although 2) is probably the easiest way to improved performance I was under > the impression there were some arguments ~2 years ago around wanting to > have all the tests in a single binary. > Mm... okay, I don't seem to recall :( > I think any effort spent 1) will be valuable regardless. We've talked about > a bunch of tasks in #1 and just needed time / commitment to them. > Yeah, I agree. Some people have also suggested improved tooling. For example the gold > linker. > @benjamin: I remember you were working on the gold linker stuff. As far as I remember, it didn't get committed. Are you interested in reviving that stuff? > You should also see if you are ever IO bound. Debug builds generate > binaries so large that you actually can become IO bound. There are ways to > get around that too. There have been times where an optimized build has > been faster than a debug build. > Yeah.. as far as I know optimized build builds faster. But I haven't confirmed that. Joris > > — > *Joris Van Remoortere* > Mesosphere > > On Tue, Sep 27, 2016 at 6:25 PM, Michael Park <[email protected]> wrote: > > > On Tue, Sep 27, 2016 at 12:21 PM, Alexander Rojas < > [email protected] > > > > > wrote: > > > > > The workaround I use is to go to the folder of the makefile root > > > directory, so for > > > Mesos, and then use the specific target, for example: > > > > > > For mesos-tests > > > cd ${MESOS_SRC_ROOT}/build/src > > > make -j4 mesos-tests && ./mesos-tests > > > > > > for libprocess > > > cd ${MESOS_SRC_ROOT}/build/3rdparty/libprocess > > > make -j4 libprocess-tests && ./libprocess-tests > > > > > > for stout > > > cd ${MESOS_SRC_ROOT}/build/3rdparty/stout > > > make -j4 stout-tests && ./stout-tests > > > > > > > Alexander, I'm aware of these targets, and I guess my claim is that these > > are not fine-grained enough. > > Yes, I can work on a stout change and only run stout tests to iterate > > without compiling/running all of the mesos tests. > > > > But I'm talking about times when I'm making mesos changes (let's say > > involving a stout change), > > I don't want to have to recompile basically all of the tests per > iteration. > > > > > > > I don’t think it would be hard to add top level targets that just do > that > > > from the top level. > > > > > > > I imagine it's probably not either, but it's not what I'm looking for in > > this thread. > > > > I however would like to have a target that compiles but not runs the > tests. > > > > > > > What do you mean by this? Are you looking for `make tests`? > > As far as I'm aware, you can do `make tests`, and `make check` at every > > level, > > as opposed to the commands you're running to achieve what seems like the > > same thing to me. > > > > > > > > On 25 Sep 2016, at 15:56, Michael Park <[email protected]> wrote: > > > > > > > > Hello, > > > > > > > > I would like to propose a change in our build to help us improve > > > developer > > > > efficiency. > > > > The proposal is to support separate compilation of unit tests. > > > > > > > > Currently, we have the old approach of invoking `make check -j N > > > > GTEST_FILTER=""`, or a newer option of doing `make tests -j N`. From > > what > > > > I've heard the latter is slightly faster. However, when someone is > > > > developing a specific feature, it's likely that they would like to > make > > > > changes and iterate on a single test file. In this workflow, having > to > > > > compile (virtually) __all__ of the tests is very burdensome. This > > > situation > > > > is not so bad if you're working in a very isolated part of the > > codebase, > > > > but it gets to be pretty bad if you're experimenting with parts that > > are > > > > widely used. > > > > > > > > An example of a workflow I'm aiming for would look something like: > > > > > > > > 1. write some code... > > > > 2. `make check master_tests` // compile and test > > > > `src/tests/master_tests.cpp` > > > > 3. fix compilation errors... > > > > 4. `make check master_tests` // compile and test > > > > `src/tests/master_tests.cpp` > > > > 5. change some stuff... > > > > 6. `make check master_tests` // compile and test > > > > `src/tests/master_tests.cpp` > > > > 7. debug... > > > > 8. `make check master_tests` // compile and test > > > > `src/tests/master_tests.cpp` > > > > 9. alright, looks good. `make check` > > > > > > > > I have 0 attachment to the `make check master_tests` syntax. It'll > be a > > > > different syntax for CMake anyways. I just think that the ability to > > > > perform separate compilation of tests will be immensely useful. > > > > > > > > Some numbers to justify what I'm proposing: > > > > > > > > - `make -j 8` on my laptop takes roughly 10 mins. > > > > - `make tests -j 8` takes about 30 mins. > > > > > > > > Of course, not every change you make triggers all of the tests to > > > > recompile. But if you change components that are widely used, it does > > end > > > > up recompiling virtually everything. Under these circumstances, I > would > > > > love for each iteration to be 11 mins (10 mins + __at most__ 1 min to > > > > compile the single test), rather than 30 mins. > > > > > > > > NOTE: My laptop is expensive... some people even use machines with 64 > > > cores > > > > or whatever to compile Mesos. Not everyone has access to this kind of > > > > equipment. We should strive for something better than "throw more > money > > > at > > > > it". > > > > > > > > The goal of this thread for me is the following: > > > > (1) Capture whether this is something most people experience, or > > whether > > > > I'm just doing it wrong > > > > (2) If most people do experience this inefficiency and would like > this > > > > change to be made, > > > > I would like to recruit 1 or 2 people to help me deliver this, > > > since > > > > I'm not a automake nor CMake expert. > > > > > > > > >
