Re: [RFC PATCH 0/8] Meson build system [for libinput this time]

[Peter Hutterer]

On Thu, Dec 08, 2016 at 12:39:18PM +, Daniel Stone wrote:
> Hey Peter,
> 
> On 7 December 2016 at 22:33, Peter Hutterer  wrote:
> > On Tue, Nov 29, 2016 at 04:59:39PM +, Daniel Stone wrote:
> >> [meson is like totally super duper]
> >
> > I tried the same for libinput, work available in
> > https://github.com/whot/libinput/tree/wip/meson
> > I think I'm about 80% of the way there, with more testing needed and a few
> > details being different/missing.
> 
> Neat!
> 
> I think you need to filter the compiler flags list with
> cc.has_argument() to work everywhere - but this is true of
> Wayland/Weston as well. You could also replace the has_paired_device()
> check with cc.has_function('libwacom_has_paired_device', dependencies:
> dep_libwacom).
> 
> 0.36.0 also lets you get rid of some of the more ugly
> '@0@/bar'.format(foo) with join_paths(foo, '/bar') instead. I'm going
> to update Wayland/Weston to make use of this
> 
> include_directories('.') for libinput shouldn't be needed - I assume
> this is just there to pick up libinput-version.h, which you can do
> instead by adding libinput_version_h to the libinput sources list.

oh, right. just copy/pasted those I guess. There's some other polish to be
done, this will be tidied up once we get some rough plan of whether we
switch everything or not. Thanks for the review though.

> Are you sure this generates the right soversion for libinput? It looks
> like you've got args from libtool's -version-info (something I never
> quite managed to figure out), whereas Meson just takes the soversion
> directly (a la libtool -version-number).

probably not, I haven't looked at that yet. as you said I just took the
libtool version. 

> The rest looks good to me!
> 
> > benchmark results are below, but they're more limited than the weston one.
> > Running the test suite as part of the build process is pointless as it can
> > easily take half an hour. There's only one make clean/rebuild we have to
> > worry about unlike weston's multiple rebuild types.
> >
> > It's a bit apples to oranges because I opted for a single meson.build file
> > instead of the current recursive automake, so some of the speedup is likely
> > caused by that. Documentation build is currently missing, but that's just a
> > doxygen run. If anyone wants to run the numbers on different machines please
> > do so, I just ran it with JOBS=4 on my T440s, i7-4600U CPU @ 2.10GHz, 12GB
> > RAM, make -j4. I re-ran with -j8 but the numbers were almost identical
> > anyway.
> 
> Flat vs. multiple files makes no difference for Meson. Weston
> distributes it into multiple files, but they all land in one
> build.ninja file anyway.

It make a difference to automake though, so for a real comparision we'd have
to have a non-recursive automake. But I'm not really enthused about doing
*that* work ;)

Cheers,
   Peter

> > Aside from the faster build times, meson's language feels a lot cleaner than
> > autotools. I'm generally opposed to non-recursive Makefiles because they get
> > messy, I didn't mind it here. So I'd be quite ok with switching to meson
> > (after some more testing of course).
> 
> Great. :) It'll probably be another week / few days before I get to
> respin the Meson work, as I want to get atomic (+ lots of other stuff
> on top) v2 out first. But it's really neat! Glad you liked it.
> 
> Cheers,
> Daniel
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system [for libinput this time]

[Daniel Stone]

Hey Peter,

On 7 December 2016 at 22:33, Peter Hutterer  wrote:
> On Tue, Nov 29, 2016 at 04:59:39PM +, Daniel Stone wrote:
>> [meson is like totally super duper]
>
> I tried the same for libinput, work available in
> https://github.com/whot/libinput/tree/wip/meson
> I think I'm about 80% of the way there, with more testing needed and a few
> details being different/missing.

Neat!

I think you need to filter the compiler flags list with
cc.has_argument() to work everywhere - but this is true of
Wayland/Weston as well. You could also replace the has_paired_device()
check with cc.has_function('libwacom_has_paired_device', dependencies:
dep_libwacom).

0.36.0 also lets you get rid of some of the more ugly
'@0@/bar'.format(foo) with join_paths(foo, '/bar') instead. I'm going
to update Wayland/Weston to make use of this

include_directories('.') for libinput shouldn't be needed - I assume
this is just there to pick up libinput-version.h, which you can do
instead by adding libinput_version_h to the libinput sources list.

Are you sure this generates the right soversion for libinput? It looks
like you've got args from libtool's -version-info (something I never
quite managed to figure out), whereas Meson just takes the soversion
directly (a la libtool -version-number).

The rest looks good to me!

> benchmark results are below, but they're more limited than the weston one.
> Running the test suite as part of the build process is pointless as it can
> easily take half an hour. There's only one make clean/rebuild we have to
> worry about unlike weston's multiple rebuild types.
>
> It's a bit apples to oranges because I opted for a single meson.build file
> instead of the current recursive automake, so some of the speedup is likely
> caused by that. Documentation build is currently missing, but that's just a
> doxygen run. If anyone wants to run the numbers on different machines please
> do so, I just ran it with JOBS=4 on my T440s, i7-4600U CPU @ 2.10GHz, 12GB
> RAM, make -j4. I re-ran with -j8 but the numbers were almost identical
> anyway.

Flat vs. multiple files makes no difference for Meson. Weston
distributes it into multiple files, but they all land in one
build.ninja file anyway.

> Aside from the faster build times, meson's language feels a lot cleaner than
> autotools. I'm generally opposed to non-recursive Makefiles because they get
> messy, I didn't mind it here. So I'd be quite ok with switching to meson
> (after some more testing of course).

Great. :) It'll probably be another week / few days before I get to
respin the Meson work, as I want to get atomic (+ lots of other stuff
on top) v2 out first. But it's really neat! Glad you liked it.

Cheers,
Daniel
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system [for libinput this time]

[Peter Hutterer]

On Tue, Nov 29, 2016 at 04:59:39PM +, Daniel Stone wrote:
> [meson is like totally super duper]

I tried the same for libinput, work available in
https://github.com/whot/libinput/tree/wip/meson
I think I'm about 80% of the way there, with more testing needed and a few
details being different/missing.

benchmark results are below, but they're more limited than the weston one.
Running the test suite as part of the build process is pointless as it can
easily take half an hour. There's only one make clean/rebuild we have to
worry about unlike weston's multiple rebuild types.

It's a bit apples to oranges because I opted for a single meson.build file
instead of the current recursive automake, so some of the speedup is likely
caused by that. Documentation build is currently missing, but that's just a
doxygen run. If anyone wants to run the numbers on different machines please
do so, I just ran it with JOBS=4 on my T440s, i7-4600U CPU @ 2.10GHz, 12GB
RAM, make -j4. I re-ran with -j8 but the numbers were almost identical
anyway.

Aside from the faster build times, meson's language feels a lot cleaner than
autotools. I'm generally opposed to non-recursive Makefiles because they get
messy, I didn't mind it here. So I'd be quite ok with switching to meson
(after some more testing of course).

Cheers,
   Peter


Run configure stage (./autogen.sh vs. meson)
N   Min   MaxMedian   AvgStddev
x  10 12.621021 12.889188 12.693087 12.696142   0.079015141
+  10   0.49298  0.516246  0.500329 0.5013093  0.0068223524
Difference at 95.0% confidence
-12.1948 +/- 0.0526925
-96.0515% +/- 0.415028%
(Student's t, pooled s = 0.05608)

Build from clean/configured tree (make clean && make vs. ninja clean && ninja)
N   Min   MaxMedian   AvgStddev
x  10 11.095844 12.919906 12.549179 12.4522880.49349502
+  10  4.796192  4.916559  4.843162 4.8399259   0.035832531
Difference at 95.0% confidence
-7.61236 +/- 0.328739
-61.1322% +/- 2.63999%
(Student's t, pooled s = 0.349872)

Rebuild libinput from built tree (touch src/libinput.h && make vs. ninja)
N   Min   MaxMedian   AvgStddev
x  10 12.533362 20.260543 16.410773 15.298142 2.7933535
+  10  3.887209  4.814081  4.033269 4.16133430.34492342
Difference at 95.0% confidence
-11.1368 +/- 1.86998
-72.7984% +/- 12.2236%
(Student's t, pooled s = 1.9902)


___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Daniel Stone]

Hi,

On 5 December 2016 at 00:05, Peter Hutterer  wrote:
> On Fri, Dec 02, 2016 at 07:39:02PM +, Emil Velikov wrote:
>> IIRC some have the philosophical reasoning alike "thou shall not need
>> anything but autofoo and dependencies explicitly required" while
>> others run things in an (network) isolated environments where one
>> cannot pull python/others.
>>
>> One example that involved a fair bit of nitpicking: one of the guys
>> was building piglit on an older RHEL.
>> Things were failing miserably since the mako/numpy/other module
>> shipped was old. At the same time connecting to the web to pull newer
>> one (via pip) was a no-go.
>
> imo that's not any different to any other library. if e.g. cairo is too old,
> you have to update your build environment before you can build. The only
> difference is that pip and newer tools appear to make this easy, except when
> you don't have connectivity. The old-style approach of fetching the newer
> dependencies for the build root should still work.

Sure; having indeterminable/shifting build dependencies is insane. I
don't see what in Meson makes this uniquely bad, or even worse than
autotools though, especially as it has no Python dependencies beyond
the standard library. For me, it's just one more thing to have
documented along with the rest of our build dependencies. I do really
want to have at least a couple of releases as a transition period
though, during which we'd make an active effort to work with our
downstreams to get them using Meson and beat out any issues, so for
them there would be no change by the time we removed autotools.

Out of curiosity, I just checked how much it'd take to inline Meson
into a tree. Using a really lame Python zipapp, I came up with:
  - autotools: clean tree (no Meson build) 4528kB, tree with
autoreconf run (i.e. the way we dist tarballs) 9164kB
  - Meson: clean tree (no Makefile/configure.ac) 4520kB, tree with
Meson inlined 5488kB

So, if we take it as a given that we can rely on Python 3.x and its
standard library being installed, we could just ship the entirety of
Meson itself if we believe that people wouldn't be able / know how to
install Meson from pip even if we documented it. I don't think it'll
come to that, but it's maybe another interesting datapoint.

Cheers,
Daniel
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Peter Hutterer]

On Fri, Dec 02, 2016 at 07:39:02PM +, Emil Velikov wrote:
> On 2 December 2016 at 18:28, Daniel Stone  wrote:
> > Hey,
> >
> > On 2 December 2016 at 18:25, Emil Velikov  wrote:
> >> On 1 December 2016 at 15:24, Daniel Stone  wrote:
> >>> On 1 December 2016 at 14:11, Emil Velikov  
> >>> wrote:
>  This does not mitigate the fact that meson/ninja _does_ have some cool
>  features and is noticeably faster _by default_.
>  Yet there's also the fact that distributions and/or builders simply
>  cannot use python/similar tools when using distribution tarballs.
> >>>
> >>> Hm, why can't they use Python ... ? Installing Meson in a clean Debian
> >>> chroot (no Python3 installed) resulted in 5.5MB of packages being
> >>> downloaded, which expanded to 28MB on disk, much of which appears to
> >>> be Python locales, or things like expat and bzip2 which we would've
> >>> needed anyway. Fedora, Debian/Ubuntu, and OpenSUSE all have very
> >>> up-to-date versions, and I understand there's support in Yocto/BitBake
> >>> as well. It's not the sort of thing I'd expect Gentoo/Arch to have an
> >>> issue with.
> >>>
> >>> For example, Debian's Weston build depends on debhelper (including
> >>> Perl), Quilt, Pixman, libpng, libjpeg, colord, all of Mesa (with a
> >>> transient build-dependency to LLVM on x86), XCB and a few component
> >>> libraries, Xlib, udev/systemd, xkbcommon, Cairo (>4MB), Pango, (2MB),
> >>> gdk-pixbuf (?!), Xcursor, mtdev, VP8, and libinput. This pretty much
> >>> eclipses the weight of the Python stdlib subset (python3-minimal on
> >>> Debian), especially when compared to the combined 4.7MB + Perl (>
> >>> 25MB) + M4 required to rebuild the autotools files, or the 2.7MB (?!)
> >>> of shell/Makefile/M4 generated by autoreconf.
> >>
> >> The above seems like a strange mix of build tools and (runtime) 
> >> dependencies.
> >
> > I was just trying to understand your 'distributions and/or builders
> > simply cannot use python/similar tools when using distribution
> > tarballs' statement. The above is the list of build dependencies for
> > Debian's weston package: all of those things have to be installed in
> > order to compile Weston for the Debian package. Of course every
> > distribution is different, but at least with that list, installing
> > Python does not seem particularly onerous. Or did you mean something
> > by 'simply cannot use python'?
> >
> IIRC some have the philosophical reasoning alike "thou shall not need
> anything but autofoo and dependencies explicitly required" while
> others run things in an (network) isolated environments where one
> cannot pull python/others.
> 
> One example that involved a fair bit of nitpicking: one of the guys
> was building piglit on an older RHEL.
> Things were failing miserably since the mako/numpy/other module
> shipped was old. At the same time connecting to the web to pull newer
> one (via pip) was a no-go.

imo that's not any different to any other library. if e.g. cairo is too old,
you have to update your build environment before you can build. The only
difference is that pip and newer tools appear to make this easy, except when
you don't have connectivity. The old-style approach of fetching the newer
dependencies for the build root should still work.

Cheers,
   Peter

> 
> TL;DR; People have strange reasons to not use A or B. No-one can
> foresee all the corner-cases, let alone know the distribution
> policies.
> 
> -Emil
> ___
> wayland-devel mailing list
> wayland-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/wayland-devel
> 
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Jan Engelhardt]

On Friday 2016-12-02 19:28, Daniel Stone wrote:
>On 2 December 2016 at 18:25, Emil Velikov  wrote:
>>
>> The above seems like a strange mix of build tools and (runtime)
>> dependencies.
>
>I was just trying to understand your 'distributions and/or builders
>simply cannot use python/similar tools when using distribution
>tarballs' statement. The above is the list of build dependencies for
>Debian's weston package: all of those things have to be installed in
>order to compile Weston for the Debian package.



Python can be used, but the problem with any aspirational build
system is that it probably has _some_ bugs or shortcomings that make
it produce files that do not match the distro ideals (anymore), and
because of its niche, few people know how to solve it as the decisive
knob is buried inside some 20 KLOC.

(That's samba using waf, another pythonish build system, for example.)


___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Emil Velikov]

On 2 December 2016 at 18:28, Daniel Stone  wrote:
> Hey,
>
> On 2 December 2016 at 18:25, Emil Velikov  wrote:
>> On 1 December 2016 at 15:24, Daniel Stone  wrote:
>>> On 1 December 2016 at 14:11, Emil Velikov  wrote:
 This does not mitigate the fact that meson/ninja _does_ have some cool
 features and is noticeably faster _by default_.
 Yet there's also the fact that distributions and/or builders simply
 cannot use python/similar tools when using distribution tarballs.
>>>
>>> Hm, why can't they use Python ... ? Installing Meson in a clean Debian
>>> chroot (no Python3 installed) resulted in 5.5MB of packages being
>>> downloaded, which expanded to 28MB on disk, much of which appears to
>>> be Python locales, or things like expat and bzip2 which we would've
>>> needed anyway. Fedora, Debian/Ubuntu, and OpenSUSE all have very
>>> up-to-date versions, and I understand there's support in Yocto/BitBake
>>> as well. It's not the sort of thing I'd expect Gentoo/Arch to have an
>>> issue with.
>>>
>>> For example, Debian's Weston build depends on debhelper (including
>>> Perl), Quilt, Pixman, libpng, libjpeg, colord, all of Mesa (with a
>>> transient build-dependency to LLVM on x86), XCB and a few component
>>> libraries, Xlib, udev/systemd, xkbcommon, Cairo (>4MB), Pango, (2MB),
>>> gdk-pixbuf (?!), Xcursor, mtdev, VP8, and libinput. This pretty much
>>> eclipses the weight of the Python stdlib subset (python3-minimal on
>>> Debian), especially when compared to the combined 4.7MB + Perl (>
>>> 25MB) + M4 required to rebuild the autotools files, or the 2.7MB (?!)
>>> of shell/Makefile/M4 generated by autoreconf.
>>
>> The above seems like a strange mix of build tools and (runtime) dependencies.
>
> I was just trying to understand your 'distributions and/or builders
> simply cannot use python/similar tools when using distribution
> tarballs' statement. The above is the list of build dependencies for
> Debian's weston package: all of those things have to be installed in
> order to compile Weston for the Debian package. Of course every
> distribution is different, but at least with that list, installing
> Python does not seem particularly onerous. Or did you mean something
> by 'simply cannot use python'?
>
IIRC some have the philosophical reasoning alike "thou shall not need
anything but autofoo and dependencies explicitly required" while
others run things in an (network) isolated environments where one
cannot pull python/others.

One example that involved a fair bit of nitpicking: one of the guys
was building piglit on an older RHEL.
Things were failing miserably since the mako/numpy/other module
shipped was old. At the same time connecting to the web to pull newer
one (via pip) was a no-go.

TL;DR; People have strange reasons to not use A or B. No-one can
foresee all the corner-cases, let alone know the distribution
policies.

-Emil
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Daniel Stone]

Hey,

On 2 December 2016 at 18:25, Emil Velikov  wrote:
> On 1 December 2016 at 15:24, Daniel Stone  wrote:
>> On 1 December 2016 at 14:11, Emil Velikov  wrote:
>>> This does not mitigate the fact that meson/ninja _does_ have some cool
>>> features and is noticeably faster _by default_.
>>> Yet there's also the fact that distributions and/or builders simply
>>> cannot use python/similar tools when using distribution tarballs.
>>
>> Hm, why can't they use Python ... ? Installing Meson in a clean Debian
>> chroot (no Python3 installed) resulted in 5.5MB of packages being
>> downloaded, which expanded to 28MB on disk, much of which appears to
>> be Python locales, or things like expat and bzip2 which we would've
>> needed anyway. Fedora, Debian/Ubuntu, and OpenSUSE all have very
>> up-to-date versions, and I understand there's support in Yocto/BitBake
>> as well. It's not the sort of thing I'd expect Gentoo/Arch to have an
>> issue with.
>>
>> For example, Debian's Weston build depends on debhelper (including
>> Perl), Quilt, Pixman, libpng, libjpeg, colord, all of Mesa (with a
>> transient build-dependency to LLVM on x86), XCB and a few component
>> libraries, Xlib, udev/systemd, xkbcommon, Cairo (>4MB), Pango, (2MB),
>> gdk-pixbuf (?!), Xcursor, mtdev, VP8, and libinput. This pretty much
>> eclipses the weight of the Python stdlib subset (python3-minimal on
>> Debian), especially when compared to the combined 4.7MB + Perl (>
>> 25MB) + M4 required to rebuild the autotools files, or the 2.7MB (?!)
>> of shell/Makefile/M4 generated by autoreconf.
>
> The above seems like a strange mix of build tools and (runtime) dependencies.

I was just trying to understand your 'distributions and/or builders
simply cannot use python/similar tools when using distribution
tarballs' statement. The above is the list of build dependencies for
Debian's weston package: all of those things have to be installed in
order to compile Weston for the Debian package. Of course every
distribution is different, but at least with that list, installing
Python does not seem particularly onerous. Or did you mean something
by 'simply cannot use python'?

Cheers,
Daniel
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Emil Velikov]

On 1 December 2016 at 15:24, Daniel Stone  wrote:
> Hey Emil,
>
> On 1 December 2016 at 14:11, Emil Velikov  wrote:
>> On 1 December 2016 at 11:46, Daniel Stone  wrote:
>>> On Intel, the configure stage (autogen / meson -Dfoo) is 14-15x
>>> faster. A complete build is 3x faster, and rebuilding component parts
>>> of the tree is 1.5-4x faster. The 'rebuild weston from built tree'
>>> sees one huge spike on autotools, which makes me think there's some
>>> relinking going on, so the stats are potentially far worse for
>>> autotools than Meson. Running tests is marginally faster, and
>>> installation is 2-4x faster. On ARM, configure is ~20x faster,
>>> complete build and rebuilds 3-4x faster, tests not noticeably
>>> different, and install 2-4x faster (less I/O load?).
>>>
>>> I also updated the Wayland tree to support cross-compilation by
>>> building a native scanner as part of the build process, which was
>>> seriously trivial.
>>>
>> Thanks for the extensive tools - I merely had for i in `sed 1 3`; do
>> ...; done :-)
>> I'll give them a try in the next few days.
>>
>> As hinted before - in theory [at least] one should be able get
>> noticeable improvement, by polishing the old tools - convert from
>> mtime to the method used by ninja, strip down unneeded tests from
>> configure, etc.
>
> Mm, it is possible, but it still doesn't get it down to the kinds of
> runtimes which I'd find actually acceptable, especially when doing
> things like git rebase -i --exec 'make -C build-x86-64 check -j16' to
> make sure I haven't screwed anything up at any point in my tree during
> rebasing.
>
>> This does not mitigate the fact that meson/ninja _does_ have some cool
>> features and is noticeably faster _by default_.
>> Yet there's also the fact that distributions and/or builders simply
>> cannot use python/similar tools when using distribution tarballs.
>
> Hm, why can't they use Python ... ? Installing Meson in a clean Debian
> chroot (no Python3 installed) resulted in 5.5MB of packages being
> downloaded, which expanded to 28MB on disk, much of which appears to
> be Python locales, or things like expat and bzip2 which we would've
> needed anyway. Fedora, Debian/Ubuntu, and OpenSUSE all have very
> up-to-date versions, and I understand there's support in Yocto/BitBake
> as well. It's not the sort of thing I'd expect Gentoo/Arch to have an
> issue with.
>
> For example, Debian's Weston build depends on debhelper (including
> Perl), Quilt, Pixman, libpng, libjpeg, colord, all of Mesa (with a
> transient build-dependency to LLVM on x86), XCB and a few component
> libraries, Xlib, udev/systemd, xkbcommon, Cairo (>4MB), Pango, (2MB),
> gdk-pixbuf (?!), Xcursor, mtdev, VP8, and libinput. This pretty much
> eclipses the weight of the Python stdlib subset (python3-minimal on
> Debian), especially when compared to the combined 4.7MB + Perl (>
> 25MB) + M4 required to rebuild the autotools files, or the 2.7MB (?!)
> of shell/Makefile/M4 generated by autoreconf.
>
The above seems like a strange mix of build tools and (runtime) dependencies.

Not sure that comparing X and Y across distributions is wise on my
end. As a reference point here are some numbers from Arch:
 - python2 - 71MiB
 - python3 - 102MiB
 - perl - 50MiB
 - automake - 1.5MiB
 - autoconf - 2 MiB

That said, let's not dwell into this. Size and specifics of each one
is highly distro specific, not to mention that most of those things
(what to bundle, how, etc.) are governed by distribution policies and
decisions.
Since a fair few of those range from philosophical to personal
reasons, it's something I try to stay aware from, whenever possible
;-)

Thus trying to understand why X or Y do not want/like/etc to use tools
A, B or C is a going to end up... yeah.

>> Latter feature seems to be missing from meson afaict, alongside
>> cscope, ctags and a few others that some people still use (I don't
>> personally).
>
> Honestly, I'd be happy to burn distribution tarballs and just use 'git
> evtag' or similar; https://github.com/cgwalters/git-evtag has great
> discussion on the problems that reliance on tarballs causes for people
> trying to do reproducible builds. But that's a discussion for another
> day ...
>
From a quick look - both permalink and comments are broken (not found) :-(
I'll search it up at some point, thanks.

> I didn't even realise Automake had native cscope support; for me it's
> not particularly useful since I do srcdir != builddir builds, and
> placing my editor root in the builddir seems very odd, but ...
>
>> Thus do check with distro maintainers/others who ship wayland/weston,
>> _if_ in the future one decides to drop the autohell setup ;-)
>
> I'd say anyone doing cross-compilation would be very happy, since with
> the attached script, all I needed to do to make cross-compiling work,
> including generating a build-native wayland-scanner binary, was to add
> 'native: true' to a couple of places and use the below
> cross-compilation definition file; with

Re: [RFC PATCH 0/8] Meson build system

[Daniel Stone]

Hey Emil,

On 1 December 2016 at 14:11, Emil Velikov  wrote:
> On 1 December 2016 at 11:46, Daniel Stone  wrote:
>> On Intel, the configure stage (autogen / meson -Dfoo) is 14-15x
>> faster. A complete build is 3x faster, and rebuilding component parts
>> of the tree is 1.5-4x faster. The 'rebuild weston from built tree'
>> sees one huge spike on autotools, which makes me think there's some
>> relinking going on, so the stats are potentially far worse for
>> autotools than Meson. Running tests is marginally faster, and
>> installation is 2-4x faster. On ARM, configure is ~20x faster,
>> complete build and rebuilds 3-4x faster, tests not noticeably
>> different, and install 2-4x faster (less I/O load?).
>>
>> I also updated the Wayland tree to support cross-compilation by
>> building a native scanner as part of the build process, which was
>> seriously trivial.
>>
> Thanks for the extensive tools - I merely had for i in `sed 1 3`; do
> ...; done :-)
> I'll give them a try in the next few days.
>
> As hinted before - in theory [at least] one should be able get
> noticeable improvement, by polishing the old tools - convert from
> mtime to the method used by ninja, strip down unneeded tests from
> configure, etc.

Mm, it is possible, but it still doesn't get it down to the kinds of
runtimes which I'd find actually acceptable, especially when doing
things like git rebase -i --exec 'make -C build-x86-64 check -j16' to
make sure I haven't screwed anything up at any point in my tree during
rebasing.

> This does not mitigate the fact that meson/ninja _does_ have some cool
> features and is noticeably faster _by default_.
> Yet there's also the fact that distributions and/or builders simply
> cannot use python/similar tools when using distribution tarballs.

Hm, why can't they use Python ... ? Installing Meson in a clean Debian
chroot (no Python3 installed) resulted in 5.5MB of packages being
downloaded, which expanded to 28MB on disk, much of which appears to
be Python locales, or things like expat and bzip2 which we would've
needed anyway. Fedora, Debian/Ubuntu, and OpenSUSE all have very
up-to-date versions, and I understand there's support in Yocto/BitBake
as well. It's not the sort of thing I'd expect Gentoo/Arch to have an
issue with.

For example, Debian's Weston build depends on debhelper (including
Perl), Quilt, Pixman, libpng, libjpeg, colord, all of Mesa (with a
transient build-dependency to LLVM on x86), XCB and a few component
libraries, Xlib, udev/systemd, xkbcommon, Cairo (>4MB), Pango, (2MB),
gdk-pixbuf (?!), Xcursor, mtdev, VP8, and libinput. This pretty much
eclipses the weight of the Python stdlib subset (python3-minimal on
Debian), especially when compared to the combined 4.7MB + Perl (>
25MB) + M4 required to rebuild the autotools files, or the 2.7MB (?!)
of shell/Makefile/M4 generated by autoreconf.

> Latter feature seems to be missing from meson afaict, alongside
> cscope, ctags and a few others that some people still use (I don't
> personally).

Honestly, I'd be happy to burn distribution tarballs and just use 'git
evtag' or similar; https://github.com/cgwalters/git-evtag has great
discussion on the problems that reliance on tarballs causes for people
trying to do reproducible builds. But that's a discussion for another
day ...

I didn't even realise Automake had native cscope support; for me it's
not particularly useful since I do srcdir != builddir builds, and
placing my editor root in the builddir seems very odd, but ...

> Thus do check with distro maintainers/others who ship wayland/weston,
> _if_ in the future one decides to drop the autohell setup ;-)

I'd say anyone doing cross-compilation would be very happy, since with
the attached script, all I needed to do to make cross-compiling work,
including generating a build-native wayland-scanner binary, was to add
'native: true' to a couple of places and use the below
cross-compilation definition file; with Autotools, you end up doing
two separate builds, or insane gymnastics.

> Please consider using a low traffic medium (hint not wayland-devel)
> since the question _will_ get lost.

Yeah, I'd be fine with pinging them all individually. Like I say, a
long-ish transition period to get everyone adjusted and make sure we
don't have any issues with downstreams or individual users, is
something we absolutely need. I don't want to get rid of Autotools
overnight for everyone, but I do want to only ever have to use it
again myself when I change the build, to make sure I haven't broken
it. :)

Cheers,
Daniel

[0]: Using a Debian chroot generated by qemu-bootstrap, symlinks -r,
and installing the build deps, plus a small wrapper around pkg-config
to set PKG_CONFIG_LIBDIR: cat >meson-cross-stretch-armhf.txt 

Re: [RFC PATCH 0/8] Meson build system

[Emil Velikov]

On 1 December 2016 at 11:46, Daniel Stone  wrote:
> Hi,
>
> On 30 November 2016 at 10:00, Daniel Stone  wrote:
>> On 30 November 2016 at 01:02, Emil Velikov  wrote:
>>> NB: Everything but the Install test, varies ±0.2s across 3 consecutive
>>> runs, thus it's been rounded to the closest 0.5s on average.
>>>
>>>  |  Laptop |   Laptop 2 |
>>> -+-++
>>> Full | 27.78s / 11.24s |   ~21s /  ~10s |
>>> Rebuild  | 13.80s /  9.63s | ~12.5s / ~8.5s |
>>> Check| 13.22s /  8.80s |  ~7.5s / ~5.5s |
>>> Install  |  0.47s /  0.14s |  0.47s / 0.14s |
>>> -+-++
>>
>> Ha, this isn't Skylake laptop vs. Broadwell laptop, this is my 14-core
>> Xeon vs. your Broadwell laptop! That they're so different suggests
>> that our configuration is pretty wildly different - are you perhaps
>> not building a load of stuff which configure autodetects? That's the
>> only way I can imagine how your laptop would outpace a £4000
>> workstation. :\
>
> Having talked it through later, the conclusion was that the Xeon's
> single-core performance isn't nearly as good as the Skylake-U's.
> Various articles seemed to confirm that.
>
Ack. Just as a confirmation the above discrepancy is between XPS 13 +
F25 vs X1C3 + Arch.
After double-checking - there's nothing that gets disabled during
configure here. Thus the ~15 vs ~10s runtime between our setups is
very disturbing.

>>> So if we are to ignore configure time, numbers are comparable.
>>> And yes, that's a very big if to ask for.
>>
>> Interesting indeed. I do wonder what is going on there. Either way,
>> it's still a 1.5-2x speed win for you ...
>>
>> As another data point, when I'm rebasing, I make very heavy use of git
>> rebase -i --exec right down to the parent tree, to make sure it
>> compiles/works the whole way through. Across my compositor-drm tree
>> (now up to 60 patches; most only touching compositor-drm.c, but some
>> touching wider parts of the source), this was 260.0s for make -j8, and
>> 19.3sec for ninja. There were three points at which autotools re-run
>> and Meson didn't; if we assume we have to take the full penalty for a
>> rebuild, then we end up on ~50s vs. ~260s.
>
> So, on this note, you guilted me into doing some more serious
> benchmarking of the results, which can be found:
> https://people.collabora.com/~daniels/wayland-meson-20161201/benchmark.py
> (modified to exclude simple-dmabuf-intel on ARM systems)
> https://people.collabora.com/~daniels/wayland-meson-20161201/weston.rpi2
> https://people.collabora.com/~daniels/wayland-meson-20161201/weston.rock2
> https://people.collabora.com/~daniels/wayland-meson-20161201/weston.laptop
> https://people.collabora.com/~daniels/wayland-meson-20161201/weston.xeon
>
> All the files are the result of running benchmark.py in the Weston
> tree, with ministat installed from FreeBSD (cf.
> http://anholt.net/compare-perf/). These do 10 runs and take a
> statistically-sound average. I'd also earlier tried to split the runs
> into common tasks people might perform, but it's probably easier for
> clarity to show the individual steps.
>
> On Intel, the configure stage (autogen / meson -Dfoo) is 14-15x
> faster. A complete build is 3x faster, and rebuilding component parts
> of the tree is 1.5-4x faster. The 'rebuild weston from built tree'
> sees one huge spike on autotools, which makes me think there's some
> relinking going on, so the stats are potentially far worse for
> autotools than Meson. Running tests is marginally faster, and
> installation is 2-4x faster. On ARM, configure is ~20x faster,
> complete build and rebuilds 3-4x faster, tests not noticeably
> different, and install 2-4x faster (less I/O load?).
>
> I also updated the Wayland tree to support cross-compilation by
> building a native scanner as part of the build process, which was
> seriously trivial.
>
Thanks for the extensive tools - I merely had for i in `sed 1 3`; do
...; done :-)
I'll give them a try in the next few days.

As hinted before - in theory [at least] one should be able get
noticeable improvement, by polishing the old tools - convert from
mtime to the method used by ninja, strip down unneeded tests from
configure, etc.

This does not mitigate the fact that meson/ninja _does_ have some cool
features and is noticeably faster _by default_.
Yet there's also the fact that distributions and/or builders simply
cannot use python/similar tools when using distribution tarballs.

Latter feature seems to be missing from meson afaict, alongside
cscope, ctags and a few others that some people still use (I don't
personally).
Thus do check with distro maintainers/others who ship wayland/weston,
_if_ in the future one decides to drop the autohell setup ;-)

Please consider using a low traffic medium (hint not wayland-devel)
since the question _will_ get lost.

Thanks
Emil
P.S. Speaking of which do you have some time to set this [1] up ?
https://bugs.freedesktop.org/sho

Re: [RFC PATCH 0/8] Meson build system

[Daniel Stone]

Hi,

On 30 November 2016 at 10:00, Daniel Stone  wrote:
> On 30 November 2016 at 01:02, Emil Velikov  wrote:
>> NB: Everything but the Install test, varies ±0.2s across 3 consecutive
>> runs, thus it's been rounded to the closest 0.5s on average.
>>
>>  |  Laptop |   Laptop 2 |
>> -+-++
>> Full | 27.78s / 11.24s |   ~21s /  ~10s |
>> Rebuild  | 13.80s /  9.63s | ~12.5s / ~8.5s |
>> Check| 13.22s /  8.80s |  ~7.5s / ~5.5s |
>> Install  |  0.47s /  0.14s |  0.47s / 0.14s |
>> -+-++
>
> Ha, this isn't Skylake laptop vs. Broadwell laptop, this is my 14-core
> Xeon vs. your Broadwell laptop! That they're so different suggests
> that our configuration is pretty wildly different - are you perhaps
> not building a load of stuff which configure autodetects? That's the
> only way I can imagine how your laptop would outpace a £4000
> workstation. :\

Having talked it through later, the conclusion was that the Xeon's
single-core performance isn't nearly as good as the Skylake-U's.
Various articles seemed to confirm that.

>> So if we are to ignore configure time, numbers are comparable.
>> And yes, that's a very big if to ask for.
>
> Interesting indeed. I do wonder what is going on there. Either way,
> it's still a 1.5-2x speed win for you ...
>
> As another data point, when I'm rebasing, I make very heavy use of git
> rebase -i --exec right down to the parent tree, to make sure it
> compiles/works the whole way through. Across my compositor-drm tree
> (now up to 60 patches; most only touching compositor-drm.c, but some
> touching wider parts of the source), this was 260.0s for make -j8, and
> 19.3sec for ninja. There were three points at which autotools re-run
> and Meson didn't; if we assume we have to take the full penalty for a
> rebuild, then we end up on ~50s vs. ~260s.

So, on this note, you guilted me into doing some more serious
benchmarking of the results, which can be found:
https://people.collabora.com/~daniels/wayland-meson-20161201/benchmark.py
(modified to exclude simple-dmabuf-intel on ARM systems)
https://people.collabora.com/~daniels/wayland-meson-20161201/weston.rpi2
https://people.collabora.com/~daniels/wayland-meson-20161201/weston.rock2
https://people.collabora.com/~daniels/wayland-meson-20161201/weston.laptop
https://people.collabora.com/~daniels/wayland-meson-20161201/weston.xeon

All the files are the result of running benchmark.py in the Weston
tree, with ministat installed from FreeBSD (cf.
http://anholt.net/compare-perf/). These do 10 runs and take a
statistically-sound average. I'd also earlier tried to split the runs
into common tasks people might perform, but it's probably easier for
clarity to show the individual steps.

On Intel, the configure stage (autogen / meson -Dfoo) is 14-15x
faster. A complete build is 3x faster, and rebuilding component parts
of the tree is 1.5-4x faster. The 'rebuild weston from built tree'
sees one huge spike on autotools, which makes me think there's some
relinking going on, so the stats are potentially far worse for
autotools than Meson. Running tests is marginally faster, and
installation is 2-4x faster. On ARM, configure is ~20x faster,
complete build and rebuilds 3-4x faster, tests not noticeably
different, and install 2-4x faster (less I/O load?).

I also updated the Wayland tree to support cross-compilation by
building a native scanner as part of the build process, which was
seriously trivial.

Cheers,
Daniel
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Kristian Høgsberg]

Hi Daniel,

Thanks for doing this!

I share a lot of the same background - I've worked with autotools for many
years, know them well and have also been hesitant to move away. I've
recently ported a small personal project from autotools to meson and I
found it very nice and easy to work with. It feels like a better fit for
the kind of workflow that I'm used to with autotools than many of the other
newer build systems out there.

In terms of speed, it's certainly nice that meson ends up using ninja, but
the much faster configure step is what I'm excited about. Especially on a
multicore system where the configure step runs single-threaded and
bottlenecks the rest of the otherwise parallelizable build process. And the
fact that meson is fast enough that there's no point in a separate
autogen.sh step, make the overall system much simpler and faster.

Of course, one of the points of the generate step in autotools is that you
ship the generated files with your tarballs and the end results only
requires standard sh and make. I don't think meson has a similar feature,
but I'd argue that today requiring python isn't going to limit you any more
than requiring sh and make.

I think having both meson and autotools side by side for a bit would be a
good way to start and I'd be in favor of moving to meson exclusively a few
releases down the road.

Kristian

On Tue, Nov 29, 2016 at 9:00 AM Daniel Stone  wrote:

> Hi all,
> Whilst working on the atomic series, and the recent review spree, I
> started to really lose patience with how incredibly slow our build
> system is.
>
> This patchset provides a working port to Meson, a Python-based build
> system with a Ninja backend. Whilst autotools runs a combination of
> shell and Perl (aclocal/automake/autoconf) to generate the build files
> and Make and shell to run it, Meson runs its configuration step in
> Python, and outputs Ninja build files.
>
> (tl;dr: smaller platforms 3-6x quicker, bigger platforms 2-4x quicker)
>
> There are a few things which attracted me to Meson. It's really,
> mindblowingly, fast. There's no artificial split between configure and
> build stages (i.e. configure.ac vs. Makefile.am). Its dependencies are
> quite lightweight. It has first-class support for pkg-config, and I've
> not had a problem with executables, shared libraries, modules, static
> helper libraries, binaries to install to libexec, tests[0], etc etc.
>
> I've been keeping an eye on it for a while as it matures; one of the
> things which interested me was seeing GStreamer and GNOME start to
> migrate some of their projects towards it. Not because they're
> incredible and we should blindly follow everything they do, but because
> they have a very similar set of build-system requirements to us, and
> they have also been very conservative in migrating away from autotools
> to date.
>
> I'm incredibly familiar with autotools, dating back to 2004 when I
> started experimenting with converting the X server to use it, which
> later grew into the xserver module we have today. As new build systems
> have come around, I've made a point of looking into them, and trying to
> use them.
>
> SCons and waf were immediately disqualified, because I think writing
> your entire build in actual Python is insane, and too large a barrier to
> entry. CMake is brutally slow[1], I find its interface very unintuitive,
> and cmake --help-full splits out a 37,399-line Markup file to stdout. GN
> (used for Chrome) is very nice, and I prefer its system of 'configs' to
> Meson's per-target flags, but isn't really portable. Most of the rest
> focus on languages we don't use, and/or bring massive dependencies.
>
> Meson's the first one I've found that strikes the right balance (not
> perfect, but within striking distance) of lightweight DSL vs. useful
> programming language. And, it's really incredibly fast. It has far
> better support for cross-compilation and sysroots than autotools can
> ever hope to have. And it's got quite a good upstream community behind
> it, and solid documentation.
>
> I've done a port for Meson, with the full range of options and
> configuration available in autotools today. Currently we're missing
> Doxygen support (because I haven't got around to it) in Wayland, plus I
> haven't been able to test the Weston RDP backend (because it doesn't
> build in Fedora 25) or the fbdev backend (because it's 2016), but they
> are there. There's also currently no support for 'auto' build options; I
> need to look into how best to do that. A couple of the tests in Weston
> are missing, and I think also PAM support for weston-launch.
>
> There are also some inefficiencies: currently, for some reason, Meson
> will run wayland-scanner once for every generated file _for every
> target_. So xdg-shell-unstable-v5-client-protocol.h, for example, gets
> built a bazillion times. Fixing that upstream would actually visibly cut
> the build times. We could also get a speed boost by moving away from
> config.h and using 

Re: [RFC PATCH 0/8] Meson build system

[Silvan Jegen]

On Wed, Nov 30, 2016 at 11:58 AM, Daniel Stone  wrote:
> Hi Silvan,
>
> On 30 November 2016 at 10:51, Silvan Jegen  wrote:
>> On Wed, Nov 30, 2016 at 11:00 AM, Daniel Stone  wrote:
>>> Ha, this isn't Skylake laptop vs. Broadwell laptop, this is my 14-core
>>> Xeon vs. your Broadwell laptop! That they're so different suggests
>>> that our configuration is pretty wildly different - are you perhaps
>>> not building a load of stuff which configure autodetects? That's the
>>> only way I can imagine how your laptop would outpace a £4000
>>> workstation. :\
>>
>> I wonder how much of the difference is due to the speed of the
>> NVME/PCI-E SSD Emil is using.
>>
>> In my limited experience, the Xeons on my server did not perform as
>> well as the (higher tacted) i5 CPU on my desktop for some simple XML
>> processing task (provided the degree of parallelism is the same of
>> course).
>
> Apparently I need more coffee, because I was reading the wrong table,
> and this is about Wayland rather than Weston builds. Oops. Either way,
> the Xeon also has a NVME/PCI-E SSD, which seems pretty zippy; it also
> has 128GB of RAM, so it shouldn't be touching the SSD anyway.

Ah, I must have missed that in your description. I thought your Xeon
would use a regular SSD and that you are only measuring the first run
(cold page cache). Benchmarks are always tricky.


Cheers,

Silvan
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Daniel Stone]

Hi Silvan,

On 30 November 2016 at 10:51, Silvan Jegen  wrote:
> On Wed, Nov 30, 2016 at 11:00 AM, Daniel Stone  wrote:
>> Ha, this isn't Skylake laptop vs. Broadwell laptop, this is my 14-core
>> Xeon vs. your Broadwell laptop! That they're so different suggests
>> that our configuration is pretty wildly different - are you perhaps
>> not building a load of stuff which configure autodetects? That's the
>> only way I can imagine how your laptop would outpace a £4000
>> workstation. :\
>
> I wonder how much of the difference is due to the speed of the
> NVME/PCI-E SSD Emil is using.
>
> In my limited experience, the Xeons on my server did not perform as
> well as the (higher tacted) i5 CPU on my desktop for some simple XML
> processing task (provided the degree of parallelism is the same of
> course).

Apparently I need more coffee, because I was reading the wrong table,
and this is about Wayland rather than Weston builds. Oops. Either way,
the Xeon also has a NVME/PCI-E SSD, which seems pretty zippy; it also
has 128GB of RAM, so it shouldn't be touching the SSD anyway.

Cheers,
Daniel
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Silvan Jegen]

Hi

On Wed, Nov 30, 2016 at 11:00 AM, Daniel Stone  wrote:
> Hey Emil,
> Thanks for the detailed reply! :) It's really interesting to hear your
> perspective, especially with Mesa also using SCons.
>
> On 30 November 2016 at 01:02, Emil Velikov  wrote:
>> On 29 November 2016 at 20:50, Daniel Stone  wrote:
 As you know better than me the actual speed increase isn't in using
 Meson, it's due to ninja.

 If one is to use (write?) make backend for Meson the results wouldn't
 be that different.
>>>
>>> I disagree. Using Ninja is nice, and a huge win, but there is already
>>> a massive, massive, massive, win before you invoke either Make or
>>> Ninja; just generating the configuration (autoreconf + ./configure vs.
>>> meson) is 16.75 seconds on my laptop (again, current-generation Intel
>>> with 16GB of RAM) for autotools, vs. 1.24 seconds for Meson. If
>>> they're equal, we'd have 53.96s vs. 28.0sec for the total build,
>>> rather than 53.96 vs. 12.47.
>>>
>> Thanks for the reminder how much configure sucks and how much faster
>> non-recursive makefiles are.
>
> Heh, Wayland and Weston both (apart from docs) use non-recursive
> Makefiles! We took the readability hit of flattening everything into
> one Makefile so we could get a speed/parallelism boost.
>
>> Numbers seems far better over here...
>>
>> $ git clean -fXd -- .. && rm -rf * && time (../autogen.sh
>> --prefix=/tmp/wayland-autotools --disable-documentation)
>> ~10 real
>>
>> The rest of the number also seems to vary noticeably, despite having a
>> lower spec machine.
>> Using stock Arch here, should distro of choice matter that much ?
>>
>> System 3 ('Laptop'): Dell XPS 13 9350, 2.5GHz dual-core Intel Skylake
>> i7-6500U, 4MB cache, 16GB RAM, storage on NVME/PCI-E, make -j8
>>
>> System X ('Laptop 2'): Lenovo X1C3, 245GHz dual-core Intel Broadwell
>> i7-5500U, 4MB cache, 8GB RAM, storage on NVME/PCI-E, make -j8
>>
>>
>> NB: Everything but the Install test, varies ±0.2s across 3 consecutive
>> runs, thus it's been rounded to the closest 0.5s on average.
>>
>>  |  Laptop |   Laptop 2 |
>> -+-++
>> Full | 27.78s / 11.24s |   ~21s /  ~10s |
>> Rebuild  | 13.80s /  9.63s | ~12.5s / ~8.5s |
>> Check| 13.22s /  8.80s |  ~7.5s / ~5.5s |
>> Install  |  0.47s /  0.14s |  0.47s / 0.14s |
>> -+-++
>
> Ha, this isn't Skylake laptop vs. Broadwell laptop, this is my 14-core
> Xeon vs. your Broadwell laptop! That they're so different suggests
> that our configuration is pretty wildly different - are you perhaps
> not building a load of stuff which configure autodetects? That's the
> only way I can imagine how your laptop would outpace a £4000
> workstation. :\

I wonder how much of the difference is due to the speed of the
NVME/PCI-E SSD Emil is using.

In my limited experience, the Xeons on my server did not perform as
well as the (higher tacted) i5 CPU on my desktop for some simple XML
processing task (provided the degree of parallelism is the same of
course).


Cheers,

Silvan
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Daniel Stone]

Hi,

On 30 November 2016 at 10:28, Pekka Paalanen  wrote:
> On Wed, 30 Nov 2016 10:00:40 + Daniel Stone  wrote:
>> Yeah, generally I agree. That was my point about GNOME/GStreamer also
>> being conservative with build systems: neither of us have jumped over
>> to any new and fancy build systems before, because the benefits just
>> didn't outweigh the disadvantages of doing so. And I think that makes
>> total sense, since switching does have a very large cost. For all that
>> autotools is a Kafkaesque nightmare, and no-one actually knows how M4
>> quoting works, it's well understood by end users who build, by
>> distributors and integrators who build, and by developers who work on
>> the code. That's a huge thing to be throwing away, and not to be done
>> lightly.
>>
>> Meson for me is the first system whose advantages actually outweigh
>> the disadvantage of losing that. But that being said, I would like the
>> two to sit in parallel for a couple of releases; I don't think it
>> would make much sense to remove autotools support before 1.16-1.17,
>> when we'll have a useful version of Meson in the released version of
>> Debian (stretch) and Ubuntu, soon before that also becomes the new
>> Ubuntu LTS version. That would give us as developers plenty of time to
>> get used to things, and downstream plenty of time to adjust and help
>> us discover problems with it.
>
> That's a nice plan - how's the CI for all this? :-)
> The very least to point out buggy master even if it can't yet test
> patches before they land.

Ha ha. That's step two. :)

Cheers,
Daniel
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Pekka Paalanen]

On Wed, 30 Nov 2016 10:00:40 +
Daniel Stone  wrote:

> Hey Emil,
> Thanks for the detailed reply! :) It's really interesting to hear your
> perspective, especially with Mesa also using SCons.
> 
> On 30 November 2016 at 01:02, Emil Velikov  wrote:

> > Please don't get me wrong, autohell^Wautotools is pretty annoying/etc
> > at times. Yet again I'm more of a person who would aim to improve
> > things before going to plan B.  
> 
> Yeah, generally I agree. That was my point about GNOME/GStreamer also
> being conservative with build systems: neither of us have jumped over
> to any new and fancy build systems before, because the benefits just
> didn't outweigh the disadvantages of doing so. And I think that makes
> total sense, since switching does have a very large cost. For all that
> autotools is a Kafkaesque nightmare, and no-one actually knows how M4
> quoting works, it's well understood by end users who build, by
> distributors and integrators who build, and by developers who work on
> the code. That's a huge thing to be throwing away, and not to be done
> lightly.
> 
> Meson for me is the first system whose advantages actually outweigh
> the disadvantage of losing that. But that being said, I would like the
> two to sit in parallel for a couple of releases; I don't think it
> would make much sense to remove autotools support before 1.16-1.17,
> when we'll have a useful version of Meson in the released version of
> Debian (stretch) and Ubuntu, soon before that also becomes the new
> Ubuntu LTS version. That would give us as developers plenty of time to
> get used to things, and downstream plenty of time to adjust and help
> us discover problems with it.

That's a nice plan - how's the CI for all this? :-)
The very least to point out buggy master even if it can't yet test
patches before they land.


Thanks,
pq


pgpYsk3dBYUO2.pgp
Description: OpenPGP digital signature
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Pekka Paalanen]

On Wed, 30 Nov 2016 10:54:11 +0100 (CET)
Jan Engelhardt  wrote:

> That is where modern build systems could improve on. But instead they
> throw *everything* overboard, even though Makefile.am is reusable:
> minus custom rules, it is just a set of key-value pairs and
> lazy-evaluated variables.

Hi,

if we look at that in context of Wayland, every project will have
custom rules for generating sources with wayland-scanner. Often, to
save typing, they are written in a very complicated way as pattern
rules or similar.


Thanks,
pq


pgpavqO73hC_K.pgp
Description: OpenPGP digital signature
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Daniel Stone]

Hey Emil,
Thanks for the detailed reply! :) It's really interesting to hear your
perspective, especially with Mesa also using SCons.

On 30 November 2016 at 01:02, Emil Velikov  wrote:
> On 29 November 2016 at 20:50, Daniel Stone  wrote:
>>> As you know better than me the actual speed increase isn't in using
>>> Meson, it's due to ninja.
>>>
>>> If one is to use (write?) make backend for Meson the results wouldn't
>>> be that different.
>>
>> I disagree. Using Ninja is nice, and a huge win, but there is already
>> a massive, massive, massive, win before you invoke either Make or
>> Ninja; just generating the configuration (autoreconf + ./configure vs.
>> meson) is 16.75 seconds on my laptop (again, current-generation Intel
>> with 16GB of RAM) for autotools, vs. 1.24 seconds for Meson. If
>> they're equal, we'd have 53.96s vs. 28.0sec for the total build,
>> rather than 53.96 vs. 12.47.
>>
> Thanks for the reminder how much configure sucks and how much faster
> non-recursive makefiles are.

Heh, Wayland and Weston both (apart from docs) use non-recursive
Makefiles! We took the readability hit of flattening everything into
one Makefile so we could get a speed/parallelism boost.

> Numbers seems far better over here...
>
> $ git clean -fXd -- .. && rm -rf * && time (../autogen.sh
> --prefix=/tmp/wayland-autotools --disable-documentation)
> ~10 real
>
> The rest of the number also seems to vary noticeably, despite having a
> lower spec machine.
> Using stock Arch here, should distro of choice matter that much ?
>
> System 3 ('Laptop'): Dell XPS 13 9350, 2.5GHz dual-core Intel Skylake
> i7-6500U, 4MB cache, 16GB RAM, storage on NVME/PCI-E, make -j8
>
> System X ('Laptop 2'): Lenovo X1C3, 245GHz dual-core Intel Broadwell
> i7-5500U, 4MB cache, 8GB RAM, storage on NVME/PCI-E, make -j8
>
>
> NB: Everything but the Install test, varies ±0.2s across 3 consecutive
> runs, thus it's been rounded to the closest 0.5s on average.
>
>  |  Laptop |   Laptop 2 |
> -+-++
> Full | 27.78s / 11.24s |   ~21s /  ~10s |
> Rebuild  | 13.80s /  9.63s | ~12.5s / ~8.5s |
> Check| 13.22s /  8.80s |  ~7.5s / ~5.5s |
> Install  |  0.47s /  0.14s |  0.47s / 0.14s |
> -+-++

Ha, this isn't Skylake laptop vs. Broadwell laptop, this is my 14-core
Xeon vs. your Broadwell laptop! That they're so different suggests
that our configuration is pretty wildly different - are you perhaps
not building a load of stuff which configure autodetects? That's the
only way I can imagine how your laptop would outpace a £4000
workstation. :\

> Having a close look at the functional discrepancies:
>  - meson/ninja does not create static libraries, like autotools
> Haven't tried unifying/checking.

Indeed. To be clear, Meson itself can, but I'm just not doing that
yet. I think we could get a further boost, especially on the slower
machines, by generating those intermediate static libraries. That
would be a reasonable next step, but I wanted to get this out there
before -ETIME, given that this has been a spare-time project for me.

>  - meson checks for build_docs programs (toggle build_docs to false in
> meson_options.txt)
> Changing it makes no noticeable difference.
>  - autotools builds fixed-benchmark
> Removing it makes no noticeable difference.

Oh, yes. There are also, as mentioned, a couple of tests which aren't
yet built in Meson.

> So if we are to ignore configure time, numbers are comparable.
> And yes, that's a very big if to ask for.

Interesting indeed. I do wonder what is going on there. Either way,
it's still a 1.5-2x speed win for you ...

As another data point, when I'm rebasing, I make very heavy use of git
rebase -i --exec right down to the parent tree, to make sure it
compiles/works the whole way through. Across my compositor-drm tree
(now up to 60 patches; most only touching compositor-drm.c, but some
touching wider parts of the source), this was 260.0s for make -j8, and
19.3sec for ninja. There were three points at which autotools re-run
and Meson didn't; if we assume we have to take the full penalty for a
rebuild, then we end up on ~50s vs. ~260s.

>> Ninja works on cores*2 by default AIUI, to match either the number of
>> threads (relevant for my Intel machines). Even in the RPi case where
>> it's not matching threads, it at least allows you to hide I/O latency
>> a little bit, which is huge on an SD card ...
>>
>>> At the same time, -jJOBS seems to be missing in some of the make
>>> invocations. Is that just in the summary here in the actual test runs
>>> as well ?
>>
>> It's missing in make install, because parallelising
>>
> $ make -j100 check && make -j100 install
> ... works like a charm.

Hm, I'm sure that didn't work when I tried it on the Xeon at least,
but either way, on my laptop it drops install time from ~1.7s to
~1.4s, stable over a few runs.

> I've never tried doing both check and install targets at the same time
> 

Re: [RFC PATCH 0/8] Meson build system

[Pekka Paalanen]

On Wed, 30 Nov 2016 09:21:29 +
Daniel Stone  wrote:

> Hi Pekka,
> 
> On 30 November 2016 at 08:54, Pekka Paalanen  wrote:
> > On Wed, 30 Nov 2016 01:02:13 + Emil Velikov  
> > wrote:  
> >> As we build (simple `make') binaries are linked against the in-tree
> >> DSO(s). Upon `make install' autotools/libtool relinks each binary. The
> >> latter of which can be quite costly.
> >> One way around it is to hack into libtool with another to use a
> >> project called slibtool (I've mentioned it a while back). The latter
> >> links only once, thus in theory (at least) the whole thing could be
> >> noticeably faster. For some reason normal linking with slibtool can be
> >> a bit slow :-(  
> >
> > I haven't yet looked at what Meson actually generates, but the
> > documentation mentioned that one can just run the binaries from the
> > build tree and they will use the libraries from the build tree
> > automatically. Would that not imply that the install step must massage
> > all binaries to have them use installed libs instead of leaving
> > left-overs from the build setup?
> >
> > Autotools does it worse: every binary is actually a shell script with
> > the real binary in a hidder directory, which makes using gdb and
> > valgrind require a magic libtool incantation, *and* (you say) it also
> > relinks on install. What's up with that?
> >
> > How does slibtool achieve the "easy" running of binaries in the build
> > dir if it does not relink on install?  
> 
> I'm not sure what slibtool does, but at least for Meson, it's a pretty
> easy answer. The binaries get a DT_RPATH ELF entry for the build tree,
> which makes them look there for libraries first. On install, you don't
> need to relink, but just strip the rpath entry from the binary as you
> install.

Hi,

Ok, so it's massaging all binaries but not relinking. Cool.

> > One huge convenience of Meson is (the documentation claims) that you
> > only need the one file 'meson.py' to use it, which makes the dependency
> > to meson a very easy one. When I look at the list of files installed by
> > e.g. CMake or autoconf...  
> 
> I would like to agree and say that's great, but I think it's either
> slightly misunderstood or only about half true. Meson itself _can_ be
> 'zipped' into a single Python file indeed, though I haven't actually
> tried this myself: I either use distro packages or use a local install
> via pip.

Yeah, my confusion. It was actually about running from an extracted
tar-ball or git which is practically as good.

https://github.com/mesonbuild/meson/wiki/FAQ#how-to-use-meson-on-a-host-where-it-is-not-available-in-system-packages

> That being said, its effect on the source tree is far better
> than autotools: it generates build.ninja & compile_commands.json
> inside the build directory, and leaves the source directory completely
> untouched. This is markedly better than autotools, which generates a
> huge amount of stuff inside the source directory. From that
> perspective at least, it's far more clean.

Indeed, even if autotools was used for an out-of-tree build.

I'm still in the middle or reading through Meson documentation, will
probably ask more once I finish.


Thanks,
pq


pgpQvSqNZUWpu.pgp
Description: OpenPGP digital signature
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Jan Engelhardt]

On Wednesday 2016-11-30 09:54, Pekka Paalanen wrote:
>I'm looking forward to hearing your war stories on trying to beat sense
>into autotools. I would never dare even approach it. Maybe it's FUD you
>can show to be false, but I kind of doubt it.
>
>One huge convenience of Meson is (the documentation claims) that you
>only need the one file 'meson.py' to use it, which makes the dependency
>to meson a very easy one.

I think they got it the wrong way around:
configure only needs sh+sed, while meson needs python.

The slowness of configure comes from mostly from the use of sh, and
then specifically bash, and also because you have to create new
processes for every other trivial thing like grepping in output.

That is where modern build systems could improve on. But instead they
throw *everything* overboard, even though Makefile.am is reusable:
minus custom rules, it is just a set of key-value pairs and
lazy-evaluated variables.
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Daniel Stone]

Hi Pekka,

On 30 November 2016 at 08:54, Pekka Paalanen  wrote:
> On Wed, 30 Nov 2016 01:02:13 + Emil Velikov  
> wrote:
>> As we build (simple `make') binaries are linked against the in-tree
>> DSO(s). Upon `make install' autotools/libtool relinks each binary. The
>> latter of which can be quite costly.
>> One way around it is to hack into libtool with another to use a
>> project called slibtool (I've mentioned it a while back). The latter
>> links only once, thus in theory (at least) the whole thing could be
>> noticeably faster. For some reason normal linking with slibtool can be
>> a bit slow :-(
>
> I haven't yet looked at what Meson actually generates, but the
> documentation mentioned that one can just run the binaries from the
> build tree and they will use the libraries from the build tree
> automatically. Would that not imply that the install step must massage
> all binaries to have them use installed libs instead of leaving
> left-overs from the build setup?
>
> Autotools does it worse: every binary is actually a shell script with
> the real binary in a hidder directory, which makes using gdb and
> valgrind require a magic libtool incantation, *and* (you say) it also
> relinks on install. What's up with that?
>
> How does slibtool achieve the "easy" running of binaries in the build
> dir if it does not relink on install?

I'm not sure what slibtool does, but at least for Meson, it's a pretty
easy answer. The binaries get a DT_RPATH ELF entry for the build tree,
which makes them look there for libraries first. On install, you don't
need to relink, but just strip the rpath entry from the binary as you
install.

> One huge convenience of Meson is (the documentation claims) that you
> only need the one file 'meson.py' to use it, which makes the dependency
> to meson a very easy one. When I look at the list of files installed by
> e.g. CMake or autoconf...

I would like to agree and say that's great, but I think it's either
slightly misunderstood or only about half true. Meson itself _can_ be
'zipped' into a single Python file indeed, though I haven't actually
tried this myself: I either use distro packages or use a local install
via pip. That being said, its effect on the source tree is far better
than autotools: it generates build.ninja & compile_commands.json
inside the build directory, and leaves the source directory completely
untouched. This is markedly better than autotools, which generates a
huge amount of stuff inside the source directory. From that
perspective at least, it's far more clean.

Cheers,
Daniel
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Daniel Stone]

Hi Jan,

On 29 November 2016 at 21:42, Jan Engelhardt  wrote:
> On Tuesday 2016-11-29 17:59, Daniel Stone wrote:
>>This patchset provides a working port to Meson, a Python-based build
>>system with a Ninja backend.
>
> Do you have this available in a convenient-to-download git repo
> somewhere?

I do! Should've said in the announcement.

git://git.collabora.com/git/user/daniels/wayland#wip/2016-11/meson
git://git.collabora.com/git/user/daniels/weston#wip/2016-11/meson

Cheers,
Daniel
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Pekka Paalanen]

On Wed, 30 Nov 2016 01:02:13 +
Emil Velikov  wrote:

> On 29 November 2016 at 20:50, Daniel Stone  wrote:
> > Hey Emil,
> >
> > On 29 November 2016 at 20:41, Emil Velikov  
> > wrote:  
> >> My voice doesn't carry much weight on wayland-devel still I think it
> >> will bring some nice food for thought.
> >>
> >> As you know better than me the actual speed increase isn't in using
> >> Meson, it's due to ninja.
> >>
> >> If one is to use (write?) make backend for Meson the results wouldn't
> >> be that different.  
> >
> > I disagree. Using Ninja is nice, and a huge win, but there is already
> > a massive, massive, massive, win before you invoke either Make or
> > Ninja; just generating the configuration (autoreconf + ./configure vs.
> > meson) is 16.75 seconds on my laptop (again, current-generation Intel
> > with 16GB of RAM) for autotools, vs. 1.24 seconds for Meson. If
> > they're equal, we'd have 53.96s vs. 28.0sec for the total build,
> > rather than 53.96 vs. 12.47.
> >  
> Thanks for the reminder how much configure sucks and how much faster
> non-recursive makefiles are.

> As we build (simple `make') binaries are linked against the in-tree
> DSO(s). Upon `make install' autotools/libtool relinks each binary. The
> latter of which can be quite costly.
> One way around it is to hack into libtool with another to use a
> project called slibtool (I've mentioned it a while back). The latter
> links only once, thus in theory (at least) the whole thing could be
> noticeably faster. For some reason normal linking with slibtool can be
> a bit slow :-(

Hi,

I haven't yet looked at what Meson actually generates, but the
documentation mentioned that one can just run the binaries from the
build tree and they will use the libraries from the build tree
automatically. Would that not imply that the install step must massage
all binaries to have them use installed libs instead of leaving
left-overs from the build setup?

Autotools does it worse: every binary is actually a shell script with
the real binary in a hidder directory, which makes using gdb and
valgrind require a magic libtool incantation, *and* (you say) it also
relinks on install. What's up with that?

How does slibtool achieve the "easy" running of binaries in the build
dir if it does not relink on install?

> >> Last but not least:
> >> Any ideas (did you see any threads) why so many people go and create
> >> their own build system/tools rather than "beating some sense" into the
> >> existing ones ? I'm yet to look into either ones of the autotools
> >> components, having only a cutesy skim through libtool.  

> Please don't get me wrong, autohell^Wautotools is pretty annoying/etc
> at times. Yet again I'm more of a person who would aim to improve
> things before going to plan B.

I'm looking forward to hearing your war stories on trying to beat sense
into autotools. I would never dare even approach it. Maybe it's FUD you
can show to be false, but I kind of doubt it.

One huge convenience of Meson is (the documentation claims) that you
only need the one file 'meson.py' to use it, which makes the dependency
to meson a very easy one. When I look at the list of files installed by
e.g. CMake or autoconf...


Thanks,
pq


pgpPSzlRTKX_2.pgp
Description: OpenPGP digital signature
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Emil Velikov]

On 29 November 2016 at 20:50, Daniel Stone  wrote:
> Hey Emil,
>
> On 29 November 2016 at 20:41, Emil Velikov  wrote:
>> My voice doesn't carry much weight on wayland-devel still I think it
>> will bring some nice food for thought.
>>
>> As you know better than me the actual speed increase isn't in using
>> Meson, it's due to ninja.
>>
>> If one is to use (write?) make backend for Meson the results wouldn't
>> be that different.
>
> I disagree. Using Ninja is nice, and a huge win, but there is already
> a massive, massive, massive, win before you invoke either Make or
> Ninja; just generating the configuration (autoreconf + ./configure vs.
> meson) is 16.75 seconds on my laptop (again, current-generation Intel
> with 16GB of RAM) for autotools, vs. 1.24 seconds for Meson. If
> they're equal, we'd have 53.96s vs. 28.0sec for the total build,
> rather than 53.96 vs. 12.47.
>
Thanks for the reminder how much configure sucks and how much faster
non-recursive makefiles are.
Numbers seems far better over here...

$ git clean -fXd -- .. && rm -rf * && time (../autogen.sh
--prefix=/tmp/wayland-autotools --disable-documentation)
~10 real

The rest of the number also seems to vary noticeably, despite having a
lower spec machine.
Using stock Arch here, should distro of choice matter that much ?

System 3 ('Laptop'): Dell XPS 13 9350, 2.5GHz dual-core Intel Skylake
i7-6500U, 4MB cache, 16GB RAM, storage on NVME/PCI-E, make -j8

System X ('Laptop 2'): Lenovo X1C3, 245GHz dual-core Intel Broadwell
i7-5500U, 4MB cache, 8GB RAM, storage on NVME/PCI-E, make -j8


NB: Everything but the Install test, varies ±0.2s across 3 consecutive
runs, thus it's been rounded to the closest 0.5s on average.

 |  Laptop |   Laptop 2 |
-+-++
Full | 27.78s / 11.24s |   ~21s /  ~10s |
Rebuild  | 13.80s /  9.63s | ~12.5s / ~8.5s |
Check| 13.22s /  8.80s |  ~7.5s / ~5.5s |
Install  |  0.47s /  0.14s |  0.47s / 0.14s |
-+-++

Having a close look at the functional discrepancies:
 - meson/ninja does not create static libraries, like autotools
Haven't tried unifying/checking.
 - meson checks for build_docs programs (toggle build_docs to false in
meson_options.txt)
Changing it makes no noticeable difference.
 - autotools builds fixed-benchmark
Removing it makes no noticeable difference.

So if we are to ignore configure time, numbers are comparable.
And yes, that's a very big if to ask for.

>> Writing an ninja backend for autotools on the other hand might be a
>> little bit harder.
>
> 'A little bit'.
>
Precisely. Thanks!

>> Last time I've looked neither make nor ninja scaled if JOB > actual
>> cores. Thus pushing -j8 on a RPi2 might be an overkill ;-)
>
> Ninja works on cores*2 by default AIUI, to match either the number of
> threads (relevant for my Intel machines). Even in the RPi case where
> it's not matching threads, it at least allows you to hide I/O latency
> a little bit, which is huge on an SD card ...
>
>> At the same time, -jJOBS seems to be missing in some of the make
>> invocations. Is that just in the summary here in the actual test runs
>> as well ?
>
> It's missing in make install, because parallelising
>
$ make -j100 check && make -j100 install
... works like a charm.

I've never tried doing both check and install targets at the same time
and I'm not sure if its legal.

>> As you know separate make and make install/check stages does force the
>> costly relink. slibtool [libtool rewrite in C] does not do that
>> although it does seem a bit slower in some of my testing. I'll need to
>> check with the author about addressing that.
>
> Slower again? How?!
>
Not sure what you mean here.

As we build (simple `make') binaries are linked against the in-tree
DSO(s). Upon `make install' autotools/libtool relinks each binary. The
latter of which can be quite costly.
One way around it is to hack into libtool with another to use a
project called slibtool (I've mentioned it a while back). The latter
links only once, thus in theory (at least) the whole thing could be
noticeably faster. For some reason normal linking with slibtool can be
a bit slow :-(

>> Last but not least:
>> Any ideas (did you see any threads) why so many people go and create
>> their own build system/tools rather than "beating some sense" into the
>> existing ones ? I'm yet to look into either ones of the autotools
>> components, having only a cutesy skim through libtool.
>
> Having worked with autotools quite a lot for the last 12 years, I
> can't say I was ever tempted to send a patch for it. Even just working
> around some of its more esoteric quirks and deficiencies (cf. magic
> relinking rules which kill parallelism) makes me lose the will to
> live. On the other hand, I found Meson very easy and accessible to
> work with, and had the pull request for skipping tests accepted today.
> I just can't ever imagine bringing myself to work on the beast which
> is autot

Re: [RFC PATCH 0/8] Meson build system

[Jan Engelhardt]

On Tuesday 2016-11-29 17:59, Daniel Stone wrote:

>This patchset provides a working port to Meson, a Python-based build
>system with a Ninja backend.

Do you have this available in a convenient-to-download git repo 
somewhere?
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Daniel Stone]

Hey Emil,

On 29 November 2016 at 20:41, Emil Velikov  wrote:
> My voice doesn't carry much weight on wayland-devel still I think it
> will bring some nice food for thought.
>
> As you know better than me the actual speed increase isn't in using
> Meson, it's due to ninja.
>
> If one is to use (write?) make backend for Meson the results wouldn't
> be that different.

I disagree. Using Ninja is nice, and a huge win, but there is already
a massive, massive, massive, win before you invoke either Make or
Ninja; just generating the configuration (autoreconf + ./configure vs.
meson) is 16.75 seconds on my laptop (again, current-generation Intel
with 16GB of RAM) for autotools, vs. 1.24 seconds for Meson. If
they're equal, we'd have 53.96s vs. 28.0sec for the total build,
rather than 53.96 vs. 12.47.

> Writing an ninja backend for autotools on the other hand might be a
> little bit harder.

'A little bit'.

> Last time I've looked neither make nor ninja scaled if JOB > actual
> cores. Thus pushing -j8 on a RPi2 might be an overkill ;-)

Ninja works on cores*2 by default AIUI, to match either the number of
threads (relevant for my Intel machines). Even in the RPi case where
it's not matching threads, it at least allows you to hide I/O latency
a little bit, which is huge on an SD card ...

> At the same time, -jJOBS seems to be missing in some of the make
> invocations. Is that just in the summary here in the actual test runs
> as well ?

It's missing in make install, because parallelising

> As you know separate make and make install/check stages does force the
> costly relink. slibtool [libtool rewrite in C] does not do that
> although it does seem a bit slower in some of my testing. I'll need to
> check with the author about addressing that.

Slower again? How?!

> Last but not least:
> Any ideas (did you see any threads) why so many people go and create
> their own build system/tools rather than "beating some sense" into the
> existing ones ? I'm yet to look into either ones of the autotools
> components, having only a cutesy skim through libtool.

Having worked with autotools quite a lot for the last 12 years, I
can't say I was ever tempted to send a patch for it. Even just working
around some of its more esoteric quirks and deficiencies (cf. magic
relinking rules which kill parallelism) makes me lose the will to
live. On the other hand, I found Meson very easy and accessible to
work with, and had the pull request for skipping tests accepted today.
I just can't ever imagine bringing myself to work on the beast which
is autotools.

Plus, what are you realistically going to do with it which would
result in a 2-6x speed increase? It's used so widely that I cannot for
the life of me imagine that it's only because of no-one working on it
that it's this slow. I'm going to ascribe it more to being enormous
piles of shell and Perl, compatible all the way back to some ancient
AIX sh variant. Might be wrong, but ...

Point is, autotools has literal decades to be this fast, and still
isn't. To me, that points to deeper issues.

Cheers,
Daniel
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [RFC PATCH 0/8] Meson build system

[Emil Velikov]

Hi Dan,

My voice doesn't carry much weight on wayland-devel still I think it
will bring some nice food for thought.

As you know better than me the actual speed increase isn't in using
Meson, it's due to ninja.

If one is to use (write?) make backend for Meson the results wouldn't
be that different.
Writing an ninja backend for autotools on the other hand might be a
little bit harder.

Last time I've looked neither make nor ninja scaled if JOB > actual
cores. Thus pushing -j8 on a RPi2 might be an overkill ;-)
At the same time, -jJOBS seems to be missing in some of the make
invocations. Is that just in the summary here in the actual test runs
as well ?

As you know separate make and make install/check stages does force the
costly relink. slibtool [libtool rewrite in C] does not do that
although it does seem a bit slower in some of my testing. I'll need to
check with the author about addressing that.

Last but not least:
Any ideas (did you see any threads) why so many people go and create
their own build system/tools rather than "beating some sense" into the
existing ones ? I'm yet to look into either ones of the autotools
components, having only a cutesy skim through libtool.

Thanks
Emil
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel