Re: [Qbs] Qbs (way) faster than cmake (or benchmark issue)
On Tue, 23 Jul 2019 at 21:17, Иван Комиссаров wrote: > > Yeah, blog post would be nice. I'd wait for the product graph feature though > so we could compare cmake buildgraph and Qbs build graphs as well. Writing such an article would require a bit more work in term of collecting and analysing data. As well, one would need to make sure that the comparison is fair, which i'm not certain it currently is at this stage. As well, to be complete, i would need to measure qmake perf and cmake+make as well. I would need to check qbs with the combineMocOutput option, ... As well a comparaison when using a hot ccache could be useful to measure overhead of the build system, ... The dependency graph export i would like to write has actually nothing to do with this perf comparaison and won't be useful. The 2 graphs (product/target based) have to be identical, this is about correctness not performance. Chris ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Qbs (way) faster than cmake (or benchmark issue)
Yeah, blog post would be nice. I'd wait for the product graph feature though so we could compare cmake buildgraph and Qbs build graphs as well. Иван Комиссаров > 23 июля 2019 г., в 4:49, Vincent Hui написал(а): > > Hi Chris, > > Thank you for your benchmark. > > Would you mind writing a blog post to record your benchmark in order to > promote Qbs? > > By the way, I think we need to come up with good ideas for promoting Qbs. > > Vincent > > >> On Tue, 23 Jul 2019 at 07:31, Christian Gagneraud wrote: >> On Mon, 22 Jul 2019 at 22:10, Oswald Buddenhagen >> wrote: >> > >> > On Mon, Jul 22, 2019 at 11:00:07AM +1200, Christian Gagneraud wrote: >> > > At the 20 jobs mark, cmake start to stagnate, whereas qbs still make >> > > use of parallelism, at the 30 jobs mark, cmake completely stopped >> > > reducing overall build time, whereas qbs start to stagnate. At the 40 >> > > jobs mark, both systems are stale. >> > > >> > that means that qbs creates a build graph with fewer bottlenecks. cmake >> > certainly could do that with relatively little effort as well, at the >> > likely cost of slightly increased null rebuild times (due to the graph >> > being bigger). >> >> Should i plot qbs vs cmake null rebuild times? That would be funny if >> qbs was faster again... :P >> >> Just kidding... >> >> I think the build graph generated by cmake is too big/complex. >> By default, Qbs doesn't track dependencies on system header files for >> example. >> >> We shouldn't see such a big difference b/w cmake/ninja and qbs. >> Although i like Qbs very much, i do not believe that it implements >> revolutionary algorithms compared to ninja. >> Ninja website is actually very interesting (eg. w/ references to >> http://gittup.org/tup/ and https://github.com/apenwarr/redo) and my >> gut feeling is that this tool is really efficient in executing it's >> input. >> >> I think (on of) the problem is what cmake generates, as mentioned in >> an earlier email, I find suspicious that ninja has to call cmake to >> handle moc. And this moc aggregation is certainly one of the >> bottleneck (again, just gut feelings). >> >> > some suggestions for making a diagram where the interesting stuff isn't >> > crammed into a narrow strip at the bottom: >> > - a logarithmic job count axis >> >> Doesn't really help on it's own, but log on both axes kind of do, see >> attached document. >> >> > - build speed per job vs. job count >> > - cpu utilization per core vs. job count >> >> The main issue here is that i've used '-j' to control jobs, which is >> not good, i should have used docker --cpus instead. >> Nonetheless all the data is attached if someone fancy some data mining. >> >> > - >> >> I don't think i'll spend more time on this. But feel free to play with >> that if you (or anyone else) want. >> https://gitlab.com/chgans/cubicninjas >> >> Nothing fancy really, it's just a Dockerfile and a build script. >> >> I've attached the csv files, there's lot of potentially interesting >> data in build-qtcreator-stats.csv (see >> https://linux.die.net/man/1/time, i've dumped everything) >> The other file (build-qtcreator-dstat.csv) is data collected by dstat >> for the whole duration of the builds (see >> https://linux.die.net/man/1/dstat, i've dumped disk, load and cpu >> data). >> Again, it's full of potential useful data, it just needs some mining... >> >> Docker command: >> cd /path/to/qt-creator >> git clone https://gitlab.com/chgans/cubicninjas.git >> docker build -t cubicninjas cubicninjas >> mkdir builds >> docker run -it -v $PWD:$PWD -w $PWD --mount >> type=tmpfs,destination=/tmp --mount type=tmpfs,destination=$PWD/builds >> cubicninjas $PWD/cubicninjas/build.sh >> >> basic gnuplot command for build-qtcreator-stats.csv: >> set datafile separator "," >> set logscale x >> set logscale y >> plot 'build-qtcreator-stats.csv' every 2::1 using 2:19 with >> linespoints title 'ninja', '' every 2::0 using 2:19 with linespoints >> title 'qbs' >> >> Enjoy and my apologies for the grumpy email yesterday. >> >> Chris >> ___ >> Qbs mailing list >> Qbs@qt-project.org >> https://lists.qt-project.org/listinfo/qbs > ___ > Qbs mailing list > Qbs@qt-project.org > https://lists.qt-project.org/listinfo/qbs ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Qbs (way) faster than cmake (or benchmark issue)
Hi Chris, Thank you for your benchmark. Would you mind writing a blog post to record your benchmark in order to promote Qbs? By the way, I think we need to come up with good ideas for promoting Qbs. Vincent On Tue, 23 Jul 2019 at 07:31, Christian Gagneraud wrote: > On Mon, 22 Jul 2019 at 22:10, Oswald Buddenhagen > wrote: > > > > On Mon, Jul 22, 2019 at 11:00:07AM +1200, Christian Gagneraud wrote: > > > At the 20 jobs mark, cmake start to stagnate, whereas qbs still make > > > use of parallelism, at the 30 jobs mark, cmake completely stopped > > > reducing overall build time, whereas qbs start to stagnate. At the 40 > > > jobs mark, both systems are stale. > > > > > that means that qbs creates a build graph with fewer bottlenecks. cmake > > certainly could do that with relatively little effort as well, at the > > likely cost of slightly increased null rebuild times (due to the graph > > being bigger). > > Should i plot qbs vs cmake null rebuild times? That would be funny if > qbs was faster again... :P > > Just kidding... > > I think the build graph generated by cmake is too big/complex. > By default, Qbs doesn't track dependencies on system header files for > example. > > We shouldn't see such a big difference b/w cmake/ninja and qbs. > Although i like Qbs very much, i do not believe that it implements > revolutionary algorithms compared to ninja. > Ninja website is actually very interesting (eg. w/ references to > http://gittup.org/tup/ and https://github.com/apenwarr/redo) and my > gut feeling is that this tool is really efficient in executing it's > input. > > I think (on of) the problem is what cmake generates, as mentioned in > an earlier email, I find suspicious that ninja has to call cmake to > handle moc. And this moc aggregation is certainly one of the > bottleneck (again, just gut feelings). > > > some suggestions for making a diagram where the interesting stuff isn't > > crammed into a narrow strip at the bottom: > > - a logarithmic job count axis > > Doesn't really help on it's own, but log on both axes kind of do, see > attached document. > > > - build speed per job vs. job count > > - cpu utilization per core vs. job count > > The main issue here is that i've used '-j' to control jobs, which is > not good, i should have used docker --cpus instead. > Nonetheless all the data is attached if someone fancy some data mining. > > > - > > I don't think i'll spend more time on this. But feel free to play with > that if you (or anyone else) want. > https://gitlab.com/chgans/cubicninjas > > Nothing fancy really, it's just a Dockerfile and a build script. > > I've attached the csv files, there's lot of potentially interesting > data in build-qtcreator-stats.csv (see > https://linux.die.net/man/1/time, i've dumped everything) > The other file (build-qtcreator-dstat.csv) is data collected by dstat > for the whole duration of the builds (see > https://linux.die.net/man/1/dstat, i've dumped disk, load and cpu > data). > Again, it's full of potential useful data, it just needs some mining... > > Docker command: > cd /path/to/qt-creator > git clone https://gitlab.com/chgans/cubicninjas.git > docker build -t cubicninjas cubicninjas > mkdir builds > docker run -it -v $PWD:$PWD -w $PWD --mount > type=tmpfs,destination=/tmp --mount type=tmpfs,destination=$PWD/builds > cubicninjas $PWD/cubicninjas/build.sh > > basic gnuplot command for build-qtcreator-stats.csv: > set datafile separator "," > set logscale x > set logscale y > plot 'build-qtcreator-stats.csv' every 2::1 using 2:19 with > linespoints title 'ninja', '' every 2::0 using 2:19 with linespoints > title 'qbs' > > Enjoy and my apologies for the grumpy email yesterday. > > Chris > ___ > Qbs mailing list > Qbs@qt-project.org > https://lists.qt-project.org/listinfo/qbs > ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Qbs (way) faster than cmake (or benchmark issue)
On Mon, 22 Jul 2019 at 22:10, Oswald Buddenhagen wrote: > > On Mon, Jul 22, 2019 at 11:00:07AM +1200, Christian Gagneraud wrote: > > At the 20 jobs mark, cmake start to stagnate, whereas qbs still make > > use of parallelism, at the 30 jobs mark, cmake completely stopped > > reducing overall build time, whereas qbs start to stagnate. At the 40 > > jobs mark, both systems are stale. > > > that means that qbs creates a build graph with fewer bottlenecks. cmake > certainly could do that with relatively little effort as well, at the > likely cost of slightly increased null rebuild times (due to the graph > being bigger). > > some suggestions for making a diagram where the interesting stuff isn't > crammed into a narrow strip at the bottom: > - a logarithmic job count axis > - build speed per job vs. job count > - cpu utilization per core vs. job count > - You're on your own, you're trying to proof something everyone know your're wrong. you have to spend enregy, not us. > > ___ > Qbs mailing list > Qbs@qt-project.org > https://lists.qt-project.org/listinfo/qbs ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Qbs (way) faster than cmake (or benchmark issue)
On Mon, Jul 22, 2019 at 11:00:07AM +1200, Christian Gagneraud wrote: > At the 20 jobs mark, cmake start to stagnate, whereas qbs still make > use of parallelism, at the 30 jobs mark, cmake completely stopped > reducing overall build time, whereas qbs start to stagnate. At the 40 > jobs mark, both systems are stale. > that means that qbs creates a build graph with fewer bottlenecks. cmake certainly could do that with relatively little effort as well, at the likely cost of slightly increased null rebuild times (due to the graph being bigger). some suggestions for making a diagram where the interesting stuff isn't crammed into a narrow strip at the bottom: - a logarithmic job count axis - build speed per job vs. job count - cpu utilization per core vs. job count - ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Qbs (way) faster than cmake (or benchmark issue)
What is the reason for the difference between compiler calls with moc between Qbs and CMake/Ninja? It looks like Qbs is actually compiling the moc stuff separately while CMake/Ninja don't or rather they compile it as part of other sources somehow? It would explain both the longer time they take and the paralellism decline. At any rate that CMake/Ninja don't understand it could do it differently (more efficiently) is their flaw obviously. More generally though one could argue that the comparison is not really fair since CMake is not a build system (unlike Qbs) but just a build generator so it naturally cannot do as much in optimizing builds as Qbs (or any actual build system) can. No matter how hard you would try doing "CMake project description -> Ninja makefiles -> Ninja build" will always be inferior to "Qbs project description -> Qbs build" beacuse of that translation layer. Michael __ Od: "Christian Gagneraud" Komu: "qbs" Datum: 22.07.2019 01:00 Předmět: Re: [Qbs] Qbs (way) faster than cmake (or benchmark issue) On Tue, 16 Jul 2019 at 18:29, Christian Gagneraud wrote: > > Hi there, > > I finally found time to start my qbs vs cmake build perf. > > What i found is that qbs beats cmake 'big time' on my build machine > and i was actually surprised, so surprised that i'm trying to find > where is the mistake. Here is a graph showing qtcreator build time vs number of parallel jobs, using cmake and qbs. Builds were done in a docker container (same for both), stock Ubuntu 19.04/amd64, all dependencies (inc. qbs tool and dev libs) installed using apt. Number of compilations were measured by generating a Clang compilation database and counting the number of entries: cmake: 2881 (2869) qbs: 4362 (2848) The first number is the number of compiler calls, the second number if the same but ignoring the pattern 'moc_'. Using 2 jobs: qbs: 26:30.78 cmake: 30:11.44 =>Cmake is 15% slower than qbs Using 56 jobs: qbs: 2:35.95 cmake: 4:33.79 => CMake is 75% slower than bqs At the 20 jobs mark, cmake start to stagnate, whereas qbs still make use of parallelism, at the 30 jobs mark, cmake completely stopped reducing overall build time, whereas qbs start to stagnate. At the 40 jobs mark, both systems are stale. Chris -- ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs <https://lists.qt-project.org/listinfo/qbs> ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Qbs (way) faster than cmake (or benchmark issue)
On Tue, 16 Jul 2019 at 18:29, Christian Gagneraud wrote: > > Hi there, > > I finally found time to start my qbs vs cmake build perf. > > What i found is that qbs beats cmake 'big time' on my build machine > and i was actually surprised, so surprised that i'm trying to find > where is the mistake. Here is a graph showing qtcreator build time vs number of parallel jobs, using cmake and qbs. Builds were done in a docker container (same for both), stock Ubuntu 19.04/amd64, all dependencies (inc. qbs tool and dev libs) installed using apt. Number of compilations were measured by generating a Clang compilation database and counting the number of entries: cmake: 2881 (2869) qbs: 4362 (2848) The first number is the number of compiler calls, the second number if the same but ignoring the pattern 'moc_'. Using 2 jobs: qbs: 26:30.78 cmake: 30:11.44 =>Cmake is 15% slower than qbs Using 56 jobs: qbs: 2:35.95 cmake: 4:33.79 => CMake is 75% slower than bqs At the 20 jobs mark, cmake start to stagnate, whereas qbs still make use of parallelism, at the 30 jobs mark, cmake completely stopped reducing overall build time, whereas qbs start to stagnate. At the 40 jobs mark, both systems are stale. Chris qtcreator-build-time-cmake-vs-qbs.pdf Description: Adobe PDF document ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Qbs (way) faster than cmake (or benchmark issue)
Hi, time qbs -f ../qtcreator.qbs -j $jobs profile:qt-5-12-2 project.withAutotests:false vs cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS_DEBUG="-O0 -g" -G Ninja .. time ninja -j $jobs Is QBS's default profile is debug too? Because for CMake you set -DCMAKE_BUILD_TYPE=Debug. Maybe the use of -pipe has noticeable effects. You can play with QBS project to tyne it with same flags as are used in CMake. I meant, to modify the source qtcreator.qbs project to minimize differences. :) 17.07.2019 9:19, Christian Gagneraud пишет: On Tue, 16 Jul 2019 at 18:29, Christian Gagneraud wrote: The compiler flags are not the same, but not so far off: Qbs: -g -O0 -Wall -Wextra -m64 -pipe -fexceptions -fvisibility=hidden -fvisibility-inlines-hidden -fPIC CMake: -g -O0 -std=c++14 -fvisibility=hidden -fvisibility-inlines-hidden -fPIC -fstack-protector-strong -Wformat -Wformat-security Maybe the use of -pipe has noticeable effects. I'm currently running builds with incremental parallel jobs and i record the load average history. When building with Qbs I can clearly see "stairs" (except for high parallel jobs), with CMake it's difficult to spot as if there was a lot of noise. As well, i've now disabled building unit tests and qbs when building with qbs, the comparison should now be more fair: time qbs -f ../qtcreator.qbs -j $jobs profile:qt-5-12-2 project.withAutotests:false vs cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS_DEBUG="-O0 -g" -G Ninja .. time ninja -j $jobs Qbs builds QtCreator in 2:46, ninja in 4:55 (not counting the initial cmake call which would add 5+ seconds), the ratio is 0.56, basically Qbs is twice as fast when using all the available cores on this particular machine/env. I think i'll try to match the compiler flags exactly, and use tmpfs for /tmp and the build dir. This should remove most of the IOs "noise" What i've noticed so far is that ninja use a shell to run the compiler, and AFAIU, ninja calls "cmake -E cmake_autogen", to handle Qt's moc. Weird Chris ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Qbs (way) faster than cmake (or benchmark issue)
On Tue, 16 Jul 2019 at 18:29, Christian Gagneraud wrote: > The compiler flags are not the same, but not so far off: > Qbs: -g -O0 -Wall -Wextra -m64 -pipe -fexceptions -fvisibility=hidden > -fvisibility-inlines-hidden -fPIC > CMake: -g -O0 -std=c++14 -fvisibility=hidden > -fvisibility-inlines-hidden -fPIC -fstack-protector-strong -Wformat > -Wformat-security Maybe the use of -pipe has noticeable effects. I'm currently running builds with incremental parallel jobs and i record the load average history. When building with Qbs I can clearly see "stairs" (except for high parallel jobs), with CMake it's difficult to spot as if there was a lot of noise. As well, i've now disabled building unit tests and qbs when building with qbs, the comparison should now be more fair: time qbs -f ../qtcreator.qbs -j $jobs profile:qt-5-12-2 project.withAutotests:false vs cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS_DEBUG="-O0 -g" -G Ninja .. time ninja -j $jobs Qbs builds QtCreator in 2:46, ninja in 4:55 (not counting the initial cmake call which would add 5+ seconds), the ratio is 0.56, basically Qbs is twice as fast when using all the available cores on this particular machine/env. I think i'll try to match the compiler flags exactly, and use tmpfs for /tmp and the build dir. This should remove most of the IOs "noise" What i've noticed so far is that ninja use a shell to run the compiler, and AFAIU, ninja calls "cmake -E cmake_autogen", to handle Qt's moc. Weird Chris ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Qbs (way) faster than cmake (or benchmark issue)
On Tue, 16 Jul 2019 18:29:35 +1200 Christian Gagneraud wrote: > I finally found time to start my qbs vs cmake build perf. > > What i found is that qbs beats cmake 'big time' on my build machine > and i was actually surprised, so surprised that i'm trying to find > where is the mistake. Precompiled headers? Debug vs release? Christian ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Qbs (way) faster than cmake (or benchmark issue)
On Tue, 16 Jul 2019, 19:00 Oswald Buddenhagen, wrote: > On Tue, Jul 16, 2019 at 06:29:35PM +1200, Christian Gagneraud wrote: > > $ time qbs -f ../qtcreator.qbs profile:qt-5-12-2 > > user96m51.927s > > sys 19m52.059s > > > > $ time ninja > > user70m58.028s > > sys 23m9.238s > > > these system time figures appear *wy* too high to me. something > funny appears to be going on with your kernel. > Interesting remark, the builds were done inside docker container. Still, they might be good tho, this machine has 28 cores/56 threads. Nonetheless, wall time clearly show that qbs build way more stuff than cmake, and does this in less time. I might collect the load average while building, which will show that Qbs is more efficient. As well, the build time Vs job count graph might give clue. Chris. ___ > Qbs mailing list > Qbs@qt-project.org > https://lists.qt-project.org/listinfo/qbs > ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Qbs (way) faster than cmake (or benchmark issue)
On Tue, Jul 16, 2019 at 06:29:35PM +1200, Christian Gagneraud wrote: > $ time qbs -f ../qtcreator.qbs profile:qt-5-12-2 > user96m51.927s > sys 19m52.059s > > $ time ninja > user70m58.028s > sys 23m9.238s > these system time figures appear *wy* too high to me. something funny appears to be going on with your kernel. ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
