Parallel building of nss with gyp+ninja (Re: Tracing where build time is spent)

[Luboš Luňák]

On Monday 17 of February 2020, Luboš Luňák wrote:
> On Monday 17 of February 2020, Jan-Marek Glogowski wrote:
> > From
> > https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Building:
> >
> > "Ideally, also install gyp and ninja and put them on your path. This is
> > recommended, as the build is faster and more reliable."
> >
> > I had the impression, that just the gyp and ninja build supports
> > parallel build currently.
>
>  I've tried it, and yes, it builds so much faster. On Linux it's easy, on
> Windows it seems to require something called MozillaBuild, which is an .exe
> installer of all kinds of Unix tools. I've eventually managed to get nss to
> build with gyp+ninja even on Windows with this, but I'll need to look more
> into it to get it to build with gbuild. Still, I think this is the way to
> go.


 FWIW, this is now in gerrit as https://gerrit.libreoffice.org/c/core/+/90115 
and the follow-up patches. It currently doesn't build in Jenkins, as it is 
still pending on buildbots getting updated to the latest LODE. 

-- 
 Luboš Luňák
 l.lu...@collabora.com
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Luboš Luňák]

On Wednesday 19 of February 2020, Stephan Bergmann wrote:
> On 19/02/2020 09:51, Luboš Luňák wrote:
> > On Tuesday 18 of February 2020, Eike Rathke wrote:
> >> On Monday, 2020-02-17 19:06:23 +0100, Luboš Luňák wrote:
> >>>   And is there any worthwhile gain in insisting on using upstream
> >>> tarballs?
> >>
> >> Reliable checksums and reproducible packaging.
...
> (That doesn't mean that at least I suggest repackaging is something we
> should avoid at all cost.  IMO, it may be an option if it can be done in
> an accountable way and has some clear benefit.)

 The patch at https://gerrit.libreoffice.org/c/core/+/89825 includes a script 
to do the boost repacking, so I hope that's sufficient in this case.

-- 
 Luboš Luňák
 l.lu...@collabora.com
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Thorsten Behrens]

Stephan Bergmann wrote:
> Still, I would prefer it if we keep the differences between Gerrit Jenkins
> and "official" TDF builds as meaningfully small as possible. The motivation
> for --enable-dbgutil deviation is certainly different from the motivation
> for --enable-python=system deviation.  Deviation motivated by build
> performance should IMO only be a last resort.
> 
Seconded.

Cheers,

-- Thorsten


signature.asc
Description: PGP signature
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Stephan Bergmann]

On 19/02/2020 09:51, Luboš Luňák wrote:

On Tuesday 18 of February 2020, Eike Rathke wrote:

On Monday, 2020-02-17 19:06:23 +0100, Luboš Luňák wrote:

  And is there any worthwhile gain in insisting on using upstream
tarballs?


Reliable checksums and reproducible packaging.

A responsible developer introducing a new tarball on the download server
a) checks it against the official checksum after download
b) creates the SHA256SUM of the file to use in download.lst

Any repacking invalidates that, specifically on a developer's machine
could introduce omissions or additions.


  That is the theory, but the reality is that we already do have some tarballs
that do not have any matching upstream tarballs (e.g. because do not exist),
so I think that point is moot.


But the theory is definitely something worth aiming for, IMO.  Ever so 
often have I been frustrated in trying to track down the origins of some 
artifact at .  (And I still think 
we should put that site under some kind of version control, with a 
journal detailing how exactly each individual artifact was obtained. 
But that's a somewhat different issue.)


(That doesn't mean that at least I suggest repackaging is something we 
should avoid at all cost.  IMO, it may be an option if it can be done in 
an accountable way and has some clear benefit.)


___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Luboš Luňák]

On Tuesday 18 of February 2020, Eike Rathke wrote:
> Hi,
>
> On Monday, 2020-02-17 19:06:23 +0100, Luboš Luňák wrote:
> >  And is there any worthwhile gain in insisting on using upstream
> > tarballs?
>
> Reliable checksums and reproducible packaging.
>
> A responsible developer introducing a new tarball on the download server
> a) checks it against the official checksum after download
> b) creates the SHA256SUM of the file to use in download.lst
>
> Any repacking invalidates that, specifically on a developer's machine
> could introduce omissions or additions.

 That is the theory, but the reality is that we already do have some tarballs 
that do not have any matching upstream tarballs (e.g. because do not exist), 
so I think that point is moot.

-- 
 Luboš Luňák
 l.lu...@collabora.com
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Eike Rathke]

Hi,

On Monday, 2020-02-17 19:06:23 +0100, Luboš Luňák wrote:

>  And is there any worthwhile gain in insisting on using upstream tarballs? 

Reliable checksums and reproducible packaging.

A responsible developer introducing a new tarball on the download server
a) checks it against the official checksum after download
b) creates the SHA256SUM of the file to use in download.lst

Any repacking invalidates that, specifically on a developer's machine
could introduce omissions or additions.

  Eike

-- 
GPG key 0x6A6CD5B765632D3A - 2265 D7F3 A7B0 95CC 3918  630B 6A6C D5B7 6563 2D3A


signature.asc
Description: PGP signature
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Stephan Bergmann]

On 18/02/2020 13:17, Luboš Luňák wrote:

On Tuesday 18 of February 2020, Stephan Bergmann wrote:

You left it somewhat unclear what the target audiences for your various
performance improvement proposals are (local builds, Gerrit Jenkins
builds, other tinderbox builds, "official" TDF release builds, ...).


  I assumed it was implied where it mattered. Using --enable-debug/dbgutil is
for developer builds, isn't it?


...but then you mentioned "Windows Jenkins builds" with regard to 
--enable-python=system in your previous response.  Probably better to 
spell things out explicitly.



Using --enable-python=system for Gerrit Jenkins builds would trade
significance of those builds ("will a release build with this change be
good?") for build performance.


  Strictly speaking, no Jenkins build does that except for the Linux GCC
release one, as all the others build with --enable-dbgutil, so I find this
argument weak. In practice it seems it's the tinderboxes that check this (if
at all).


Still, I would prefer it if we keep the differences between Gerrit 
Jenkins and "official" TDF builds as meaningfully small as possible. 
The motivation for --enable-dbgutil deviation is certainly different 
from the motivation for --enable-python=system deviation.  Deviation 
motivated by build performance should IMO only be a last resort.


___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Luboš Luňák]

On Tuesday 18 of February 2020, Stephan Bergmann wrote:
> You left it somewhat unclear what the target audiences for your various
> performance improvement proposals are (local builds, Gerrit Jenkins
> builds, other tinderbox builds, "official" TDF release builds, ...).

 I assumed it was implied where it mattered. Using --enable-debug/dbgutil is 
for developer builds, isn't it?

> Using --enable-python=system for Gerrit Jenkins builds would trade
> significance of those builds ("will a release build with this change be
> good?") for build performance.

 Strictly speaking, no Jenkins build does that except for the Linux GCC 
release one, as all the others build with --enable-dbgutil, so I find this 
argument weak. In practice it seems it's the tinderboxes that check this (if 
at all).

-- 
 Luboš Luňák
 l.lu...@collabora.com
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Stephan Bergmann]

On 18/02/2020 12:44, Luboš Luňák wrote:

On Tuesday 18 of February 2020, Michael Stahl wrote:

On 17.02.20 22:03, Luboš Luňák wrote:

   But the same can be said about system Python on Linux, no? So where's
the difference?


--enable-python=system is a configuration that only distro packagers
should use, after verifying that the system Python does provide
everything LO requires.


  I've been using it for ages too, and it just works for me. And I'm willing to
bet it'll just work for Windows Jenkins builds too.


it's not possible to distribute such LO builds


  This is not about distributing anything:

On Sunday 16 of February 2020, Luboš Luňák wrote:

It would make sense to have some --with-system-libs=auto which would try to
use as much system stuff as possible and print a summary,
and --enable-debug/dbgutil could default to it. Even on Windows, e.g.

   ^^

python3 is an optional component installed by MSVC, I don't know why we
don't even try to use it.


You left it somewhat unclear what the target audiences for your various 
performance improvement proposals are (local builds, Gerrit Jenkins 
builds, other tinderbox builds, "official" TDF release builds, ...).


Using --enable-python=system for Gerrit Jenkins builds would trade 
significance of those builds ("will a release build with this change be 
good?") for build performance.


___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Luboš Luňák]

On Tuesday 18 of February 2020, Michael Stahl wrote:
> On 17.02.20 22:03, Luboš Luňák wrote:
> >   But the same can be said about system Python on Linux, no? So where's
> > the difference?
>
> --enable-python=system is a configuration that only distro packagers
> should use, after verifying that the system Python does provide
> everything LO requires.

 I've been using it for ages too, and it just works for me. And I'm willing to 
bet it'll just work for Windows Jenkins builds too.

> it's not possible to distribute such LO builds 

 This is not about distributing anything:

On Sunday 16 of February 2020, Luboš Luňák wrote:
> It would make sense to have some --with-system-libs=auto which would try to
> use as much system stuff as possible and print a summary,
> and --enable-debug/dbgutil could default to it. Even on Windows, e.g.
  ^^
> python3 is an optional component installed by MSVC, I don't know why we
> don't even try to use it. 

-- 
 Luboš Luňák
 l.lu...@collabora.com
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Michael Stahl]

On 17.02.20 22:03, Luboš Luňák wrote:

On Monday 17 of February 2020, Michael Stahl wrote:

the problem is that the MSVC's python binaries won't contain any patches
required for LO, and may be configured with a different set of optional
modules enabled...  probably it's good enough to pass all the unit
tests, but i wouldn't bet on it.  oh, and will it be built with debug
runtimes for --enable-dbgutil?


  But the same can be said about system Python on Linux, no? So where's the
difference?


--enable-python=system is a configuration that only distro packagers 
should use, after verifying that the system Python does provide 
everything LO requires.  it's not possible to distribute such LO builds 
as due to the lack of ABI stability in CPython it would require the same 
specific Python major.minor version on the system.


the same applies for many of the bundled libraries on Linux.

the only exception i'm aware of where we likely could be using the 
system library on Linux but don't currently is NSS (there used to be a 
patch to add some PEM parsing feature to it but i removed it years ago 
and nobody complained).



Norbert once added some support to gbuild for downloading "binary
tarballs" but i don't think anybody used it in years.


  Interesting, I've assumed that it was a decision that everything would be
always built from source.


i don't remember any details about it, but maybe building the binary 
tarballs took more time than it saved.


___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Luboš Luňák]

On Monday 17 of February 2020, Stephan Bergmann wrote:
> On 17/02/2020 15:15, Michael Stahl wrote:
> > jenkins would very likely benefit from -Og with few if any downside.
>
> Optimization is already orthogonal to the generation of debug
> information in the build system, so enabling it for Jenkins builds
> should just be a matter of adding --enable-optimized to the appropriate
> files in distro-configs/Jenkins/.

 https://gerrit.libreoffice.org/c/core/+/88917 and follow-ups.

 Since -Og was enabled for debug builds at one point in the 
past, --enable-optimized=debug could be also for those who are willing to try 
it for their builds. Seeing how much longer that makes things build here with 
Clang+PCH (+70% on starmath), I think I do not see it worth it myself.

-- 
 Luboš Luňák
 l.lu...@collabora.com
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Luboš Luňák]

On Tuesday 18 of February 2020, Jan-Marek Glogowski wrote:
> Am 17.02.20 um 22:02 schrieb Luboš Luňák:
> > $ time tar xf boost_1_71_0.tar.bz2
> >
> > real4m43.122s
> > user0m25.077s
> > sys 1m5.514s
>
> Ughh.
>
> Still - the point I was trying to make is that the unpack with your
> setup just hits one core and nobody is actually waiting in that
> seemingly half-empty timespan in the log. So the build would save ~10s,
> if you can cut this time in half.

 Not everyone has 16 cores. Moreover, even if it saves just a little, the cost 
of the repacking is tiny and unpacking happens often, so it's still worth it.

-- 
 Luboš Luňák
 l.lu...@collabora.com
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Jan-Marek Glogowski]

Am 17.02.20 um 22:02 schrieb Luboš Luňák:
> On Monday 17 of February 2020, Jan-Marek Glogowski wrote:
>>> - It takes 3 minutes to unpack the 100MiB bzip2 tarball of boost, which
>>> unpacks to about 0.5GiB of stuff including generated html docs. It would
>>> be a cheap gain to get rid of all doc/ example/ test/ and repack it as
>>> .xz .
>>
>> I don't think this will help and it's a "false positive".
> 
> $ time tar xf boost_1_71_0.tar.bz2
> 
> real4m43.122s
> user0m25.077s
> sys 1m5.514s

Ughh.

Still - the point I was trying to make is that the unpack with your
setup just hits one core and nobody is actually waiting in that
seemingly half-empty timespan in the log. So the build would save ~10s,
if you can cut this time in half. While firebird and nss really are much
slower, due to the non-parallel make.

> (I have no idea what the Ryzen 7 Windows machine needs all that time for.)

My impression is that general IO operation in Cygwin is slow. We already
know the Win32 make is much faster. Same is documented for git-win in
the wiki for "git status" (native at least a magnitude faster then
Cygwin git -
https://wiki.documentfoundation.org/Development/BuildingOnWindows#Cygwin_and_git).

No idea if native unpack tools could help here and if that is the real
problem, just as for make and seemingly git.
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Luboš Luňák]

On Monday 17 of February 2020, Michael Stahl wrote:
> the problem is that the MSVC's python binaries won't contain any patches
> required for LO, and may be configured with a different set of optional
> modules enabled...  probably it's good enough to pass all the unit
> tests, but i wouldn't bet on it.  oh, and will it be built with debug
> runtimes for --enable-dbgutil?

 But the same can be said about system Python on Linux, no? So where's the 
difference?

> Norbert once added some support to gbuild for downloading "binary
> tarballs" but i don't think anybody used it in years.

 Interesting, I've assumed that it was a decision that everything would be 
always built from source.

-- 
 Luboš Luňák
 l.lu...@collabora.com
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Luboš Luňák]

On Monday 17 of February 2020, Jan-Marek Glogowski wrote:
> Nice tooling. Debian Chromium doesn't have about:tracing, so I used
> https://chromium.googlesource.com/catapult/+/refs/heads/master/tracing/
> with ./bin/trace2html trace.json --output trace.html to analyze the trace.

 There's also https://www.speedscope.app/ , although I have no idea how to 
actually use that.

> From
> https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Building:
>
> "Ideally, also install gyp and ninja and put them on your path. This is
> recommended, as the build is faster and more reliable."
>
> I had the impression, that just the gyp and ninja build supports
> parallel build currently.

 I've tried it, and yes, it builds so much faster. On Linux it's easy, on 
Windows it seems to require something called MozillaBuild, which is an .exe 
installer of all kinds of Unix tools. I've eventually managed to get nss to 
build with gyp+ninja even on Windows with this, but I'll need to look more 
into it to get it to build with gbuild. Still, I think this is the way to go.

> > - It takes 3 minutes to unpack the 100MiB bzip2 tarball of boost, which
> > unpacks to about 0.5GiB of stuff including generated html docs. It would
> > be a cheap gain to get rid of all doc/ example/ test/ and repack it as
> > .xz .
>
> I don't think this will help and it's a "false positive".

$ time tar xf boost_1_71_0.tar.bz2

real4m43.122s
user0m25.077s
sys 1m5.514s

(I have no idea what the Ryzen 7 Windows machine needs all that time for.)

-- 
 Luboš Luňák
 l.lu...@collabora.com
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Luboš Luňák]

On Monday 17 of February 2020, Caolán McNamara wrote:
> On Mon, 2020-02-17 at 11:51 +0100, Luboš Luňák wrote:
> > On Sunday 16 of February 2020, Noel Grandin wrote:
> > > On Sun, 16 Feb 2020 at 16:33, Luboš Luňák 
> > >
> > > wrote:
> > > > - It takes 3 minutes to unpack the 100MiB bzip2 tarball of boost,
> > > > which
> > > > unpacks to about 0.5GiB of stuff including generated html docs.
> > > > It would be a cheap gain to get rid of all doc/ example/ test/
> > > > and repack it as .xz .
> > > >
> > > > Or maybe teach the unpacker to skip those?
> >
> >  I think you cannot teach unpacker to skip parts of .tar.XYZ , it
> > pretty much always has to unpack the whole thing to get the .tar.
>
> could use the upstream zips instead, that would allow selectively
> skipping decompression, though the zips are correspondingly larger than
> the tar.xz

 And is there any worthwhile gain in insisting on using upstream tarballs? 
Simply repacking seems better to me in all regards except for requiring few 
seconds to run a script whenever the tarball is updated, which is way less 
often then the archive gets used.

-- 
 Luboš Luňák
 l.lu...@collabora.com
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Jan-Marek Glogowski]

I have to drop the test keys from xmlsecurity/qa from my keyring... so
once more:

Am 16.02.20 um 15:33 schrieb Luboš Luňák:
>  I've pushed https://gerrit.libreoffice.org/c/core/+/88754 , which allows 
> viewing what actually happens during building. Exact instructions are in 
> https://cgit.freedesktop.org/libreoffice/core/tree/solenv/gbuild/Trace.mk . 
> 
>  At http://ge.tt/9z4s0J13 I've uploaded a trace 
> of './autogen.sh --enable-dbgutil --disable-java && make' Windows build on 16 
> HT core Ryzen 7 that takes about 70 minutes, viewable using the 
> chrome://tracing URL in Chromium.

Nice tooling. Debian Chromium doesn't have about:tracing, so I used
https://chromium.googlesource.com/catapult/+/refs/heads/master/tracing/
with ./bin/trace2html trace.json --output trace.html to analyze the trace.

>  A couple of observations:
> 
> - About 9 minutes are spent building only nss and firebird, both apparently 
> doing -j1 builds, and nss blocks most of LO sources, and nss itself is 
> blocked by python3. IIRC somebody has already tried to do something about 
> these and didn't manage. Nss's build page says that the recommended build is 
> actually using some Mozilla build tool instead of autotools, has somebody 
> tried that?

I fixed larger parts of the NSS parallel build in here:
https://gerrit.libreoffice.org/c/core/+/74751

My originally small patch turned out to become a whole patchset over
time. It's still there and was working for the latest NSS upload. As you
can see from the patches, I handle NSS in a separate git repo. I'm
normally building my local Windows and Linux builds with it. But that's
not a good indicator, as the build isn't that parallel here.

From https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Building:

"Ideally, also install gyp and ninja and put them on your path. This is
recommended, as the build is faster and more reliable."

I had the impression, that just the gyp and ninja build supports
parallel build currently. I also found these upstream bugs:

Upstream also has multiple bugs:
- https://bugzilla.mozilla.org/show_bug.cgi?id=290526
- https://bugzilla.mozilla.org/show_bug.cgi?id=836220

I didn't yet try to upstream the patches. On NSS IRC nobody replied when
I asked about it.

> - It takes 3 minutes to unpack the 100MiB bzip2 tarball of boost, which 
> unpacks to about 0.5GiB of stuff including generated html docs. It would be a 
> cheap gain to get rid of all doc/ example/ test/ and repack it as .xz .

I don't think this will help and it's a "false positive".

In the graph this is ~3m on one of the 16 threads of that build. All the
other threads are busy doing other LO build stuff in parallel most of
the time. And 5 are building externals during the whole UPK gap.

The main problem of the graph: you don't see the parallel external
builds spreading out jobs. IMHO as long as there is one parallel
external build running, the machine is actually busy parallel building
that external.

Which AFAIK leaves the NSS and firebird builds, as you already observed.

Jan-Marek
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Stephan Bergmann]

On 17/02/2020 15:15, Michael Stahl wrote:

On 16.02.20 15:33, Luboš Luňák wrote:
- Unittests in dbgutil take as much as half the time it takes to 
compile .cxx
sources (not counting externals not built by gbuild). It could make 
sense to
revisit using -O1/-Og, at least for Jenkins builds. Also, now that we 
do have


jenkins would very likely benefit from -Og with few if any downside.


Optimization is already orthogonal to the generation of debug 
information in the build system, so enabling it for Jenkins builds 
should just be a matter of adding --enable-optimized to the appropriate 
files in distro-configs/Jenkins/.


___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Jan-Marek Glogowski]

binj3rv9MLYzu.bin
Description: PGP/MIME version identification


encrypted.asc
Description: OpenPGP encrypted message
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Michael Stahl]

On 16.02.20 15:33, Luboš Luňák wrote:


  Hello,

  I've pushed https://gerrit.libreoffice.org/c/core/+/88754 , which allows
viewing what actually happens during building. Exact instructions are in
https://cgit.freedesktop.org/libreoffice/core/tree/solenv/gbuild/Trace.mk .

  At http://ge.tt/9z4s0J13 I've uploaded a trace
of './autogen.sh --enable-dbgutil --disable-java && make' Windows build on 16
HT core Ryzen 7 that takes about 70 minutes, viewable using the
chrome://tracing URL in Chromium.

  A couple of observations:

- The build gets to building actual LO sources only after half the build time.
This means that the default of building most externals ourselves is kind of
silly, especially on Windows without ccache. It would make sense to have
some --with-system-libs=auto which would try to use as much system stuff as
possible and print a summary, and --enable-debug/dbgutil could default to it.
Even on Windows, e.g. python3 is an optional component installed by MSVC, I
don't know why we don't even try to use it.


the problem is that the MSVC's python binaries won't contain any patches 
required for LO, and may be configured with a different set of optional 
modules enabled...  probably it's good enough to pass all the unit 
tests, but i wouldn't bet on it.  oh, and will it be built with debug 
runtimes for --enable-dbgutil?


Norbert once added some support to gbuild for downloading "binary 
tarballs" but i don't think anybody used it in years.



- About 9 minutes are spent building only nss and firebird, both apparently
doing -j1 builds, and nss blocks most of LO sources, and nss itself is
blocked by python3. IIRC somebody has already tried to do something about
these and didn't manage. Nss's build page says that the recommended build is
actually using some Mozilla build tool instead of autotools, has somebody
tried that?


there's a humongous patch in our gerrit to make it build in parallel 
with make that really should be upstreamed because it would be quite 
unmaintainable as LO downstream patch... and also NSS has grown a 2nd 
build system using "gyp" in the meantime, which i don't know anything 
about but would presumably introduce new build dependencies.


https://gerrit.libreoffice.org/c/core/+/74751/70


- Unittests in dbgutil take as much as half the time it takes to compile .cxx
sources (not counting externals not built by gbuild). It could make sense to
revisit using -O1/-Og, at least for Jenkins builds. Also, now that we do have


jenkins would very likely benefit from -Og with few if any downside.


Jenkins builds, maybe finally plain 'make' could be sensible and not run
tests all the time.


i have never liked the number of tests that are run via "make", 
particularly since the vast majority of them are not unit tests, but 
integration tests.  i'd like to move all integration tests to "make 
subsequentcheck", but a previous attempt at that found 2 problems:


* most jenkins builders only run "make" and not "make check"

* it introduced an additional delay while linking the serialized large 
libraries where currently some integration tests run in parallel (this 
would require some tweaks to avoid)


https://gerrit.libreoffice.org/c/core/+/31075
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Caolán McNamara]

On Mon, 2020-02-17 at 11:51 +0100, Luboš Luňák wrote:
> On Sunday 16 of February 2020, Noel Grandin wrote:
> > On Sun, 16 Feb 2020 at 16:33, Luboš Luňák 
> > wrote:
> > > - It takes 3 minutes to unpack the 100MiB bzip2 tarball of boost,
> > > which
> > > unpacks to about 0.5GiB of stuff including generated html docs.
> > > It would be a cheap gain to get rid of all doc/ example/ test/
> > > and repack it as .xz .
> > > 
> > > Or maybe teach the unpacker to skip those?
> 
>  I think you cannot teach unpacker to skip parts of .tar.XYZ , it
> pretty much always has to unpack the whole thing to get the .tar.

could use the upstream zips instead, that would allow selectively
skipping decompression, though the zips are correspondingly larger than
the tar.xz

___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Luboš Luňák]

On Sunday 16 of February 2020, Noel Grandin wrote:
> On Sun, 16 Feb 2020 at 16:33, Luboš Luňák  wrote:
> > - It takes 3 minutes to unpack the 100MiB bzip2 tarball of boost, which
> > unpacks to about 0.5GiB of stuff including generated html docs. It would
> > be a
> > cheap gain to get rid of all doc/ example/ test/ and repack it as .xz .
> >
> > Or maybe teach the unpacker to skip those?

 I think you cannot teach unpacker to skip parts of .tar.XYZ , it pretty much 
always has to unpack the whole thing to get the .tar . Besides, a small 
script in external/boost should be simple and do savings everywhere, and it's 
not like we always use genuine upstream tarballs.

-- 
 Luboš Luňák
 l.lu...@collabora.com
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Mike Kaganski]

On 2020-02-17 11:52, Stephan Bergmann wrote:

On 16/02/2020 15:33, Luboš Luňák wrote:

Also, now that we do have Jenkins builds, maybe finally plain 'make'
could be sensible and not run tests all the time.
I'm all for that.  I think our default make target behavior and the 
build-nocheck target, both non-standard, are not too helpful overall. 
(Gerrit Jenkins builds that today use the default make target should 
then make sure they still run the tests they run today, by making 
whatever appropriate target instead.)


My opinion, too :-)

--
Best regards,
Mike Kaganski
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Stephan Bergmann]

On 16/02/2020 15:33, Luboš Luňák wrote:

Also, now that we do have Jenkins builds, maybe finally plain 'make'
could be sensible and not run tests all the time.
I'm all for that.  I think our default make target behavior and the 
build-nocheck target, both non-standard, are not too helpful overall. 
(Gerrit Jenkins builds that today use the default make target should 
then make sure they still run the tests they run today, by making 
whatever appropriate target instead.)


___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Tracing where build time is spent

[Noel Grandin]

On Sun, 16 Feb 2020 at 16:33, Luboš Luňák  wrote:

>  I've pushed https://gerrit.libreoffice.org/c/core/+/88754 , which allows
> viewing what actually happens during building. Exact instructions are in
> https://cgit.freedesktop.org/libreoffice/core/tree/solenv/gbuild/Trace.mk
> .
>
> Nice!


> - It takes 3 minutes to unpack the 100MiB bzip2 tarball of boost, which
> unpacks to about 0.5GiB of stuff including generated html docs. It would
> be a
> cheap gain to get rid of all doc/ example/ test/ and repack it as .xz .
>
> Or maybe teach the unpacker to skip those?
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Tracing where build time is spent

[Luboš Luňák]

 Hello,

 I've pushed https://gerrit.libreoffice.org/c/core/+/88754 , which allows 
viewing what actually happens during building. Exact instructions are in 
https://cgit.freedesktop.org/libreoffice/core/tree/solenv/gbuild/Trace.mk . 

 At http://ge.tt/9z4s0J13 I've uploaded a trace 
of './autogen.sh --enable-dbgutil --disable-java && make' Windows build on 16 
HT core Ryzen 7 that takes about 70 minutes, viewable using the 
chrome://tracing URL in Chromium.

 A couple of observations:

- The build gets to building actual LO sources only after half the build time. 
This means that the default of building most externals ourselves is kind of 
silly, especially on Windows without ccache. It would make sense to have 
some --with-system-libs=auto which would try to use as much system stuff as 
possible and print a summary, and --enable-debug/dbgutil could default to it. 
Even on Windows, e.g. python3 is an optional component installed by MSVC, I 
don't know why we don't even try to use it.

- About 9 minutes are spent building only nss and firebird, both apparently 
doing -j1 builds, and nss blocks most of LO sources, and nss itself is 
blocked by python3. IIRC somebody has already tried to do something about 
these and didn't manage. Nss's build page says that the recommended build is 
actually using some Mozilla build tool instead of autotools, has somebody 
tried that?

- Unittests in dbgutil take as much as half the time it takes to compile .cxx 
sources (not counting externals not built by gbuild). It could make sense to 
revisit using -O1/-Og, at least for Jenkins builds. Also, now that we do have 
Jenkins builds, maybe finally plain 'make' could be sensible and not run 
tests all the time.

- It takes 3 minutes to unpack the 100MiB bzip2 tarball of boost, which 
unpacks to about 0.5GiB of stuff including generated html docs. It would be a 
cheap gain to get rid of all doc/ example/ test/ and repack it as .xz .

-- 
 Luboš Luňák
 l.lu...@collabora.com
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice