Re: My build work and goals
Julian Elischer wrote: > > Some improvements I have made recently: > > - WITH_FAST_DEPEND: Replacing the antiquated 'make depend'/'mkdep' with > > compiler dependency flags to generate the .depend files as a side-effect > > of compiling. This saves tremendous time in buildworld and buildkernel. > > Mach used to have this through their entire (bsd 4.3) tree. I forget Junos built this way for over 10 years but it only helps for the C/C++ code. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: My build work and goals
On 11/12/2015 3:56 AM, Bryan Drewery wrote: I think it is fair to say that most people don't understand how any of this works. [...] - No one really was trying to improve it head-on and focused on FreeBSD's general audience needs. The maintenance problems come down to expertise. Most developers are not experts in the build and don't have time or interest to become so. These three points are all pretty much the same thing. I agree that it definitely needs to have a fairy godfather. Some improvements I have made recently: - WITH_FAST_DEPEND: Replacing the antiquated 'make depend'/'mkdep' with compiler dependency flags to generate the .depend files as a side-effect of compiling. This saves tremendous time in buildworld and buildkernel. Mach used to have this through their entire (bsd 4.3) tree. I forget how they bootstrapped it. Gcc would generate added dep files which were included into the static files. Without the dep files, every thing was assumed to be dirty. The new dep files only kicked in on the second build. It worked very very well. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: My build work and goals
Firstly, nice writup - a few extra blank lines might have helped my eyes
though.
Bryan Drewery wrote:
> This mail is to outline my recent work, goals and motivations. This is
> long. This is not really a architectural review or discussion mail. I
...
> Some problems with the combined DIRDEPS_BUILD build:
>
> It is very hard to maintain, prone to massive churn in directories you
> did and did not touch, requires checking in fickle generated files, has
> a chicken-and-egg problem for adding new things, does not respect
> WITH/WITHOUT options or ARCH-dependent SUBDIR, has subtle problems with
> a static dependencies list around options, and has no 'installworld'
> support.
Dealing with all the optional cases is definitely a problem to be
solved.
I've a few ideas on that front, but haven't had time to experiment.
FWIW we (Juniper) see zero "churn" in our dependencies
a/ we use a fixed set of options
b/ we've fixed the bugs in makefiles that were causing churn.
Ignoring the WITH/WITHOUT options stuff for a sec, dependency churn is
usually a result of bugs in the leaf makefiles.
These are often easy to spot, sometimes quite hard.
The bootstrap issue and even just the need to checkin "generated" files
is a head scratcher for a surprising number of people - even after 4+
years of building this way.
> The lack of 'installworld' is a deal breaker, albeit probably trivial to
> fix. The reason for this is that the system is used to generate
> packages (not pkgng style) at Juniper. We have our own packaging effort
> occurring in projects/release-pkg though and won't likely use the same
> methods that the DIRDEPS_BUILD does (of which none of the support is
> really in the src tree).
Though AFAIK all the tools we use are, so its really just a matter of
adding some suitable makefiles and manifest - to say build a bootable vm
image - I probably would have done that by now if bhyve supported vmdk
or qcow2 ;-)
But that's likely orthogonal to your point.
Some form of "installworld" is probably needed.
> This build does not use recursive SUBDIR walking (by design). It uses a
> static list of directories to build in the
> targets/pseudo/userland/*/Makefile.depend files.
That's mostly just for the purpose of demonstration/example.
If/when FreeBSD has some form of manifest/list for what goes into its
"packages" (of whatever form they might be,) then the necessary data
to drive what needs to be built could be obtained from them - as we do
in the Junos build.
> The checked-in Makefile.depend files have a flaw (feature for static
> builds perhaps) that the invoking the linker will cause non-direct
> dependencies to show up in Makefile.depend. This leads to situations
> such as https://svnweb.freebsd.org/changeset/base/291558. There is also
> a situation where a local build may set MK_SSP=no and yet need to build
> a dependency that wants MK_SSP=yes (libssp dependency) but because of
Yes, options are a pain.
You can either hard code them via local.dirdeps.mk but it is probably
better to do something a bit like what the current intermediate
makefiles do (one of the idea I mentioned above).
Eg. if the captured dependencies had something like ${lib_libssp} rather than
lib/libssp. If MK_SSP=no it expands to nothing.
I can elaborate in separate thread if you prefer.
> than upstream, is an exhaustive chicken-and-egg problem. These problems
> make checked-in Makefile.depend infeasible. "But wait, can you just not
> check them in?" Not really.
You can, but you lose virtually all the benefits.
> recently added support to local.dirdeps.mk to make an educated guess on
> what DIRDEPS should be even if there is no Makefile.depend. This is
> based on what is being built (C and C++ objects have common
> dependencies) and what DPADD/LIBADD there are. In most cases this is
> enough. It has greatly helped when bootstrapping Makefile.depend.
You still have the issue of local depenenencies.
Sure, only a small percentage of makefiles have problems in this regard,
but humans are generally bad at expressing the dependencies that result
from using things like yacc - get it wrong about 80% of the time.
> Going further though I have written a script that just recursively walks
> the tree to generate the DIRDEPS graph from the first make process
> rather than including the Makefile.depend files. This removes the need
That's what we used to do, but was deemed too expensive.
Using that to generate Makefile.depend though for your tree, could save
you some overhead on subsequent builds, while allowing tailorying tree
to your set of options.
> P.S. Working on this stuff can be exhausting. Mistakes are easy.
Thanks for your efforts btw.
--sjg
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
My build work and goals
This mail is to outline my recent work, goals and motivations. This is long. This is not really a architectural review or discussion mail. I am leaving those and details for their own threads where needed, so please do not start discussions about any of the details here. We will have them later. No one has really objected to my work but many have asked what my goals are and if the churn is worth it. A lot of my work lately has been to remove hacks in the build and use normal mechanisms or framework to achieve the same thing. Some of this is to remove hacks from the META build. Some of this is just natural evolution of the framework since we haven't had a real maintainer for so long. That may be putting it wrong but I think it is fair to say we're all a little scared of touching share/mk and try to do so as little as possible. Consider bsd.files.mk/bsd.incs.mk/bsd.confs.mk(new) are all 95% the same. I have not yet combined them but plan to. Why is SCRIPTS only in bsd.prog.mk? There's many problems in these files that need fixing. The vast majority of my work and "churn" lately has been improving the META mode build which is not the default build. So it may appear that I am churning the build a lot when I am really not. More on the META build later. First an introductory and background. Who am I? I came to the FreeBSD community as a Ports committer who was primarily focused on building up the Ports framework and its build tools. I tookover upstream maintenance of Portupgrade and then got involved with Poudriere in its early stages and helped bring it to what it is today. I think it is fair to say I have been the maintainer of Poudriere for some time now. I am one of the maintainers of the pkg.FreeBSD.org package builds and oversee its automation. I've written a few FreeBSD Journal articles about that. 99% of this was not sponsored though. For my day job at Isilon I recently moved my efforts to the base build. In many people's eyes the build system, 'buildworld', mostly "just works". The problems come in when "works" to you is time and productivity. At Isilon we have a range of development efforts that span from developers only caring about the kernel to ones who care about 1 userland tool. All of us expect that we should be able to just build our 1 thing rather than everything. Some of these 1 userland tool cases though have hundreds of dependencies. Most developers instinctively try building manually rather than Jenkins as it feels like it should be quicker. This leads to grief. The problem comes down to productivity. I've been given a great opportunity to address these problems and am running with it. Isilon has a quite convoluted build. Our product has its own ELF brand/ABI/KBI and cannot run on native FreeBSD systems. The build is done from a FreeBSD system for reasons, so is entirely a cross-build by expectations. We have a buildworld, ports phase, and then we have a buildworld-type thing of stuff that depends on ports! Both the ports and ports-dependent pieces are built in a jail using a special hack of a kernel module that provides KBI compatibility to the native FreeBSD system so we can run target binaries. QEMU does not apply here as it's not an architecture problem, it's a syscall/KBI problem. Solving ports cross-building will remove the need for this. Stepping back to pure FreeBSD now. Some various problems I've observed in the build: - Building clang twice. We build clang from the source tree so we can build everything with it even if /usr/bin/cc qualifies as "new enough" and "capable of cross-building the target". We later build clang for the target as well. Try doing this for 'make universe' for N architectures and you build clang N*2 times rather than N times. - Not being highly parallel. - Requiring building everything to build anything (without being an expert on manually building dependencies). AKA, no reliable sub-directory builds. - Incremental builds don't work reliably. We have stealth dependencies that are not tracked such as csu, libgcc, compiler_rt, compiler and Makefile (CFLAGS/LDFLAGS,build command). - There is gross under-linkage in libraries and over-linkage in binaries. - No foreign OS cross-build support. You must build from FreeBSD 8.0+. This is a problem when people prefer to run OSX or Linux for their primary system. - No cross-build support in Ports. - share/mk and Makefile.inc1 has a ton of bitrot. Some various problems I've observed with maintaining the build: - Adding new libraries into the build usually results in doing it wrong in Makefile.inc1's handling of 'make libraries'. I think it is fair to say that most people don't understand how any of this works. Just yesterday I discovered more of how it works that surprised me. - Adding new libraries into the build is usually done wrong in terms of the new framework. - There are little build framework sanity checks. The ports build has grown a large array of sanity checks o
