Re: 'make depend' or 'make' bug on recent --current
On 01.06.2016 23:04, Bryan Drewery wrote: > No. The build and how dependencies are generated and handled is still > fundamentally the same. Open the .depend.* files and see. It is only > simple dependencies for the target built. There's nothing new about its > content. If foo.c includes stdlib.h then it includes sys/_types.h which > includes sys/cdefs.h, etc. This graph is in the old (mkdep) and new > .depend* content. This is easily provable by just comparing mkdep > output to the new versions. > > Without specific evidence of a bug I cannot help. Thanx for pointing me to MD *div sources, I overlook them. About 'make depend', I don't notice anything suspicious during simple tests and don't have time or energy to look deeper in that area. Probably several clang libs rebuilds during the single day while none usually expected cause my overreaction. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: 'make depend' or 'make' bug on recent --current
On 6/1/16 12:58 PM, Andrey Chernov wrote: > On 01.06.2016 22:36, Bryan Drewery wrote: The graph in the original commit for WITH_FAST_DEPEND disagrees. https://svnweb.freebsd.org/base?view=revision=290433 We run the preprocessor once now, not twice. >>> >>> It sounds good, just implemented bad. You measure some spherical chicken >>> in vacuum, not what really happens. In the old times I almost never have >>> clang libs rebuild (few files from there max when FreeBSD_version is >>> increased), but now I got them fully rebuilt with any header change. >>> That is the biggest slowdown and not what you try to measure. >>> Don't ever use 'make world'. Try to rebuild the system incrementally and >>> you'll see. >> >> I cannot argue with a lack of solid evidence. >> >> The old 'make depend' ran 'mkdep' which ran 'cc -E -M' which produces >> *the same output* as 'cc foo.c -o foo.o -MD -MF .depend.foo.o -MT >> foo.o'. There's nothing different in the actual .depend file >> implementation/content. Clang rebuilds often because it is changing >> often! Just look at recent commit logs and you'll see r300984 which >> will cause a rebuild of clang. Or r301011 which modified >> sys/sys/param.h which will rebuild just about everything. These are >> normal and how the old system worked as well. >> >> There certainly are some issues with the new system. >> 1. Processing the split files can be slow over NFS >> 2. make cleandepend - will cause the next build to make a lot of guesses >> and not be very efficient. >> 3. make cleandepend - There is no way to generate the .depend* files >> again without rebuilding. >> 4. It doesn't fix all of the missing dependencies that have been missing >> forever such as crt, csu, libgcc, etc for static builds. >> 5. the csu builds don't use it yet but have workarounds for it >> 6. removing the 'make depend' tree-walk can hurt downstream builds which >> relied on having a multi-phase build to generate a .mk file to include >> (odd case I hit at work). No such case exists in the upstream FreeBSD tree. >> > > There are two different things: pure recursive dependency (it is how > make depends works now) and just nested include graph with dependencies > marked directly by Makefile's rules (it is how old system works). Now it > is enough to touch some low level include to rebuild everything, before > only files which include it directly or have make rules for it are rebuilt. No. The build and how dependencies are generated and handled is still fundamentally the same. Open the .depend.* files and see. It is only simple dependencies for the target built. There's nothing new about its content. If foo.c includes stdlib.h then it includes sys/_types.h which includes sys/cdefs.h, etc. This graph is in the old (mkdep) and new .depend* content. This is easily provable by just comparing mkdep output to the new versions. Without specific evidence of a bug I cannot help. -- Regards, Bryan Drewery signature.asc Description: OpenPGP digital signature
Re: 'make depend' or 'make' bug on recent --current
On 01.06.2016 22:36, Bryan Drewery wrote: >>> The graph in the original commit for WITH_FAST_DEPEND disagrees. >>> https://svnweb.freebsd.org/base?view=revision=290433 >>> >>> We run the preprocessor once now, not twice. >> >> It sounds good, just implemented bad. You measure some spherical chicken >> in vacuum, not what really happens. In the old times I almost never have >> clang libs rebuild (few files from there max when FreeBSD_version is >> increased), but now I got them fully rebuilt with any header change. >> That is the biggest slowdown and not what you try to measure. >> Don't ever use 'make world'. Try to rebuild the system incrementally and >> you'll see. > > I cannot argue with a lack of solid evidence. > > The old 'make depend' ran 'mkdep' which ran 'cc -E -M' which produces > *the same output* as 'cc foo.c -o foo.o -MD -MF .depend.foo.o -MT > foo.o'. There's nothing different in the actual .depend file > implementation/content. Clang rebuilds often because it is changing > often! Just look at recent commit logs and you'll see r300984 which > will cause a rebuild of clang. Or r301011 which modified > sys/sys/param.h which will rebuild just about everything. These are > normal and how the old system worked as well. > > There certainly are some issues with the new system. > 1. Processing the split files can be slow over NFS > 2. make cleandepend - will cause the next build to make a lot of guesses > and not be very efficient. > 3. make cleandepend - There is no way to generate the .depend* files > again without rebuilding. > 4. It doesn't fix all of the missing dependencies that have been missing > forever such as crt, csu, libgcc, etc for static builds. > 5. the csu builds don't use it yet but have workarounds for it > 6. removing the 'make depend' tree-walk can hurt downstream builds which > relied on having a multi-phase build to generate a .mk file to include > (odd case I hit at work). No such case exists in the upstream FreeBSD tree. > There are two different things: pure recursive dependency (it is how make depends works now) and just nested include graph with dependencies marked directly by Makefile's rules (it is how old system works). Now it is enough to touch some low level include to rebuild everything, before only files which include it directly or have make rules for it are rebuilt. signature.asc Description: OpenPGP digital signature
Re: 'make depend' or 'make' bug on recent --current
On 6/1/16 11:49 AM, Andrey Chernov wrote: > On 01.06.2016 21:18, Bryan Drewery wrote: >> On 6/1/2016 6:11 AM, Andrey Chernov wrote: >>> Steps to reproduce: >>> >>> cd /usr/src/lib/libc/stdlib >>> touch *div*.c >>> cd .. >>> make depend >>> make >>> >>> And see how imaxdiv.o only is recompiled. >>> No div.o ldiv.o lldiv.o are recompiled. Because they never were compiled! lib/libc/stdlib/Makefile.inc MISRCS has div.c ldiv.c lldiv.c However, for at least amd64 it also includes: lib/libc/amd64/stdlib/Makefile.inc which has MDSRCS= div.S ldiv.S lldiv.S Note the .depend.lldiv.o file: > ~/git/freebsd/lib/libc/stdlib # cat > /usr/obj/root/git/freebsd/lib/libc/.depend.lldiv.o > lldiv.o: /root/git/freebsd/lib/libc/amd64/stdlib/lldiv.S \ > /usr/obj/root/git/freebsd/tmp/usr/include/machine/asm.h \ > /usr/obj/root/git/freebsd/tmp/usr/include/sys/cdefs.h It's built from lldiv.S, not lldiv.c. No bug here. proof: > cc -O2 -pipe -I/root/git/freebsd/lib/libc/include > -I/root/git/freebsd/lib/libc/../../include -I/root/git/freebsd/lib/libc/amd64 > -DNLS -D__DBINTERFACE_PRIVATE -I/root/git/freebsd/lib/libc/../../contrib/ > gdtoa -I/root/git/freebsd/lib/libc/../../contrib/libc-vis -DINET6 > -I/usr/obj/root/git/freebsd/lib/libc -I/root/git/freebsd/lib/libc/resolv > -D_ACL_PRIVATE -DPOSIX_MISTAKE -I/root/git/freebsd/lib/libc/../li > bmd -I/root/git/freebsd/lib/libc/../../contrib/jemalloc/include > -DMALLOC_PRODUCTION -I/root/git/freebsd/lib/libc/../../contrib/tzcode/stdtime > -I/root/git/freebsd/lib/libc/stdtime -I/root/git/freebsd/lib/l > ibc/locale -DBROKEN_DES -DPORTMAP -DDES_BUILTIN > -I/root/git/freebsd/lib/libc/rpc -DYP -DNS_CACHING -DSYMBOL_VERSIONING -MD > -MF.depend.lldiv.o -MTlldiv.o -std=gnu99 -fstack-protector-strong > -Wsystem-heade > rs -Werror -Wall -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign > -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable > -Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality > -Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef > -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -fcolor-diagnostics > -Qunused-arguments -I/root/git/freebsd/lib/libutil -I/roo > t/git/freebsd/lib/msun/amd64 -I/root/git/freebsd/lib/msun/x86 > -I/root/git/freebsd/lib/msun/src -c > /root/git/freebsd/lib/libc/amd64/stdlib/lldiv.S -o lldiv.o > cc -fpic -DPIC -O2 -pipe -I/root/git/freebsd/lib/libc/include > -I/root/git/freebsd/lib/libc/../../include -I/root/git/freebsd/lib/libc/amd64 > -DNLS -D__DBINTERFACE_PRIVATE -I/root/git/freebsd/lib/libc/.. > /../contrib/gdtoa -I/root/git/freebsd/lib/libc/../../contrib/libc-vis -DINET6 > -I/usr/obj/root/git/freebsd/lib/libc -I/root/git/freebsd/lib/libc/resolv > -D_ACL_PRIVATE -DPOSIX_MISTAKE -I/root/git/freebsd/li > b/libc/../libmd -I/root/git/freebsd/lib/libc/../../contrib/jemalloc/include > -DMALLOC_PRODUCTION -I/root/git/freebsd/lib/libc/../../contrib/tzcode/stdtime > -I/root/git/freebsd/lib/libc/stdtime -I/root/git/f > reebsd/lib/libc/locale -DBROKEN_DES -DPORTMAP -DDES_BUILTIN > -I/root/git/freebsd/lib/libc/rpc -DYP -DNS_CACHING -DSYMBOL_VERSIONING -MD > -MF.depend.lldiv.So -MTlldiv.So -std=gnu99 -fstack-protector-strong > -Wsystem-headers -Werror -Wall -Wno-format-y2k -Wno-uninitialized > -Wno-pointer-sign -Wno-empty-body -Wno-string-plus-int > -Wno-unused-const-variable -Wno-tautological-compare -Wno-unused-value > -Wno-parenth > eses-equality -Wno-unused-function -Wno-enum-conversion > -Wno-unused-local-typedef -Wno-switch -Wno-switch-enum > -Wno-knr-promoted-parameter -fcolor-diagnostics -Qunused-arguments > -I/root/git/freebsd/lib/ > libutil -I/root/git/freebsd/lib/msun/amd64 -I/root/git/freebsd/lib/msun/x86 > -I/root/git/freebsd/lib/msun/src-c > /root/git/freebsd/lib/libc/amd64/stdlib/lldiv.S -o lldiv.So note lldiv.S in both. >> >> My dev system is busy at the moment. I'll test it and get back to you. > > I need to add that I do 'make cleandepend/cleandir/cleanobj' + 'make > obj' again and full rebuild with no old files, but the bug repeated again. > >>> P.S. new make depend is simple disgusting. It tends to recompile >>> everything in the system if some minor header file is touched, but >> >> If the header is used by all source files then that is expected. >> >> However if you do not have a .depend.obj.o file then it is quite >> aggressive with building. If you touch any header it will rebuild >> everything. But you shouldn't get into that situation unless you rm -f >> .depend* first. >> >>> completely forget to recompile source code changes. I suggest to back >>> out all AI in that area. >>> 'make depend' is not time-consuming task and good old way never made >>> mistakes. >> >> The graph in the original commit for WITH_FAST_DEPEND disagrees. >> https://svnweb.freebsd.org/base?view=revision=290433 >> >> We run the preprocessor once now, not twice. > > It sounds good, just implemented bad. You measure some spherical chicken > in
Re: 'make depend' or 'make' bug on recent --current
On 01.06.2016 21:18, Bryan Drewery wrote: > On 6/1/2016 6:11 AM, Andrey Chernov wrote: >> Steps to reproduce: >> >> cd /usr/src/lib/libc/stdlib >> touch *div*.c >> cd .. >> make depend >> make >> >> And see how imaxdiv.o only is recompiled. >> No div.o ldiv.o lldiv.o are recompiled. > > My dev system is busy at the moment. I'll test it and get back to you. I need to add that I do 'make cleandepend/cleandir/cleanobj' + 'make obj' again and full rebuild with no old files, but the bug repeated again. >> P.S. new make depend is simple disgusting. It tends to recompile >> everything in the system if some minor header file is touched, but > > If the header is used by all source files then that is expected. > > However if you do not have a .depend.obj.o file then it is quite > aggressive with building. If you touch any header it will rebuild > everything. But you shouldn't get into that situation unless you rm -f > .depend* first. > >> completely forget to recompile source code changes. I suggest to back >> out all AI in that area. >> 'make depend' is not time-consuming task and good old way never made >> mistakes. > > The graph in the original commit for WITH_FAST_DEPEND disagrees. > https://svnweb.freebsd.org/base?view=revision=290433 > > We run the preprocessor once now, not twice. It sounds good, just implemented bad. You measure some spherical chicken in vacuum, not what really happens. In the old times I almost never have clang libs rebuild (few files from there max when FreeBSD_version is increased), but now I got them fully rebuilt with any header change. That is the biggest slowdown and not what you try to measure. Don't ever use 'make world'. Try to rebuild the system incrementally and you'll see. signature.asc Description: OpenPGP digital signature
Re: 'make depend' or 'make' bug on recent --current
On 6/1/2016 6:11 AM, Andrey Chernov wrote: > Steps to reproduce: > > cd /usr/src/lib/libc/stdlib > touch *div*.c > cd .. > make depend > make > > And see how imaxdiv.o only is recompiled. > No div.o ldiv.o lldiv.o are recompiled. My dev system is busy at the moment. I'll test it and get back to you. > > P.S. new make depend is simple disgusting. It tends to recompile > everything in the system if some minor header file is touched, but If the header is used by all source files then that is expected. However if you do not have a .depend.obj.o file then it is quite aggressive with building. If you touch any header it will rebuild everything. But you shouldn't get into that situation unless you rm -f .depend* first. > completely forget to recompile source code changes. I suggest to back > out all AI in that area. > 'make depend' is not time-consuming task and good old way never made > mistakes. The graph in the original commit for WITH_FAST_DEPEND disagrees. https://svnweb.freebsd.org/base?view=revision=290433 We run the preprocessor once now, not twice. -- Regards, Bryan Drewery signature.asc Description: OpenPGP digital signature
'make depend' or 'make' bug on recent --current
Steps to reproduce: cd /usr/src/lib/libc/stdlib touch *div*.c cd .. make depend make And see how imaxdiv.o only is recompiled. No div.o ldiv.o lldiv.o are recompiled. P.S. new make depend is simple disgusting. It tends to recompile everything in the system if some minor header file is touched, but completely forget to recompile source code changes. I suggest to back out all AI in that area. 'make depend' is not time-consuming task and good old way never made mistakes. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
