On Fri, Mar 19, 2021 at 2:39 PM <erik.joels...@oracle.com> wrote:
> On 2021-03-19 03:14, Thomas Stüfe wrote:
> > On Fri, Mar 19, 2021 at 11:04 AM Andrew Haley <a...@redhat.com> wrote:
> >
> >> On 3/19/21 9:22 AM, Thomas Stüfe wrote:
> >>>> 2. More choices to actually build the project: Use integrated build
> >>>> tools of IDEs (Visual Studio, Xcode) or use Ninja, which is faster
> than
> >>>> gmake
> >>>>
> >>> Is gmake really where we lose time? Did you analyze this or is this
> just
> >> an
> >>> assumption? I would have thought it's things like single threaded jmod,
> >>> jlink, and subprocess spawning.
> >> I'm sure it is. The other slow thing is linking HotSpot.
> >>
> > What is so slow with gmake? Rule processing?
> >
> > It also depends on the platform, I guess. Eg on Cygwin, the fork
> emulation
> > is extremely slow.
> >
> I have done pretty extensive work optimizing our build's performance
> over the years. There are many ways to measure performance. First we
> need to establish what kind of build we are even measuring.
>
> For a full images build ("make images"), on a reasonably sized machine
> (8-16 HV threads), we scale pretty well and use most CPUs most of the
> time. There isn't much additional concurrency to gain here. Obvious
> single threaded steps are hotspot linking and jlink. In such a build,
> Hotspot is mostly linked in parallel with all the Java compilation, so
> not an issue. Jlinking the JDK image does stick out as something we
> can't do much in parallel with, unless we also build the test or docs
> image. For a hotspot only build ("make hotspot"), then the hotspot
> linking will stick out as a single threaded step. Note that cmake/ninja
> will not help with any of this. Potential speed up from ninja is also
> very limited as the rules processing of our make scripts does not amount
> to any significant part of a full build.
>
> On Windows specifically, we do have an issue with fork being
> inefficient. We also have a less efficient file system making file
> operations more expensive in general. I have two big (though old)
> identical workstations, one with Windows and one Linux. Very rough
> numbers are 5 minutes for "make images" on the Linux machine and 10 on
> the Windows machine. These differences vary wildly on different hardware
> though. Using Windows native tools here would certainly help to some
> extent. OTOH, we have WSL, which is already considerably more performant
> than Cygwin (very rough numbers, maybe 8-9 minutes for the same build).
> The setup is a bit trickier than Cygwin, but once set up, it works
> really well in my experience.
>
> The area where ninja would provide the most benefit is for incremental
> builds, especially when very little work is actually needed, as it
> processes rules much faster than make. We have worked hard at making
> incremental builds as efficient and fast as possible, but our build is
> also pretty big so the time it takes is still noticeable, especially on
> Windows.
>
> All this said, when picking a build system, compatibility issues are the
> number one concern. If the support matrix of CMake does not completely
> cover the support matrix of OpenJDK, it's a no go to me. I would also be
> hesitant to be at the mercy of the platform support of a 3rd party when
> a new port of OpenJDK needs to be made.
>
> Regarding IDE integration, our build system is able to produce
> compile-commands.json which several IDEs know how to consume.
>
> Another big objection I have to this is the amount work required to
> rewrite the build system (again). I would expect a rewrite like this to
> be several man months, just for the OpenJDK (not counting forced
> downstream work for custom extensions to the build as well as all system
> currently interacting with the build, which I'm sure exist in more
> places than just Oracle).
>
> /Erik
>
Thanks for that extensive explanation! I have nothing to contribute here.
But would like to thank you guys for the work on the build system. It's
exceptionally stable and works very well.
Cheers, Thomas
