[qubes-users] Re: How to ccache kernel compilations
On Friday, August 24, 2018 at 1:14:09 PM UTC+2, Marcus Linsner wrote: > For posterity, the modifications (applied on top of 'qubes-linux-kernel' > repo's tag 'v4.14.57-2') that I used to achieve the above, are here: > https://github.com/constantoverride/qubes-linux-kernel/commit/ac9a975512bdc67dc12c948355b14dfdcc229b1a > (also attached just in case github goes away, somehow) The way I tried to compile kernel in this thread was wrong(because installing it in dom0 would fail due to compilation VM being Fedora 28 instead of 25 and thus missing some new libs; on a Fedora 25 VM compilation would fail). The right way to compile a VM (and dom0?) kernel is by using qubes-builder (which chroots to a fc25(aka Fedora 25, which is what dom0 is on) even though we're running inside a Fedora 28 VM): thanks to fepitre for telling me the steps here https://github.com/QubesOS/qubes-linux-kernel/pull/22#issuecomment-415453140 I'll keep track of my kernel compilation progress here: https://gist.github.com/constantoverride/825717e0136f804aa6ebf66293234b57 (like making ccache work for this version of compilation steps) -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/ab3b33ab-5519-4c68-95bb-18aee89b8731%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[qubes-users] Re: How to ccache kernel compilations
On Friday, August 24, 2018 at 12:13:01 PM UTC+2, Marcus Linsner wrote: > On Friday, August 24, 2018 at 11:24:27 AM UTC+2, Marcus Linsner wrote: > > This is how a full(well, slightly modified) kernel compilation looks like > > now, with ccache working: > > ie. `time make rpms` > > real7m47.483s > > user9m2.507s > > sys 6m47.245s > > > > cache directory /home/user/.ccache > > primary config /home/user/.ccache/ccache.conf > > secondary config (readonly)/etc/ccache.conf > > stats zero time Fri Aug 24 11:09:03 2018 > > cache hit (direct) 14047 > > cache hit (preprocessed) 1 > > cache miss 8 > > cache hit rate 99.94 % > > called for link 47 > > called for preprocessing 21125 > > unsupported code directive 4 > > no input file 1092 > > cleanups performed 0 > > files in cache 42606 > > cache size 865.4 MB > > max cache size 20.0 GB > > > > The build phase actually takes only 2min (for 14k files): > > real2m1.674s > > user5m28.075s > > sys 4m50.768s > > > > > > cache directory /home/user/.ccache > > primary config /home/user/.ccache/ccache.conf > > secondary config (readonly)/etc/ccache.conf > > stats zero time Fri Aug 24 11:17:37 2018 > > cache hit (direct) 14011 > > cache hit (preprocessed) 0 > > cache miss 5 > > cache hit rate 99.96 % > > called for link 28 > > called for preprocessing 21069 > > unsupported code directive 4 > > no input file342 > > cleanups performed 0 > > files in cache 42616 > > cache size 865.6 MB > > max cache size 20.0 GB For posterity, the modifications (applied on top of 'qubes-linux-kernel' repo's tag 'v4.14.57-2') that I used to achieve the above, are here: https://github.com/constantoverride/qubes-linux-kernel/commit/ac9a975512bdc67dc12c948355b14dfdcc229b1a (also attached just in case github goes away, somehow) -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/4806bbad-04a5-4463-95a6-d0b8c485267f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. commit ac9a975512bdc67dc12c948355b14dfdcc229b1a Author: constantoverride Date: Fri Aug 24 12:49:27 2018 +0200 made ccache work; personalized .config a lil also applied missing patches to avoid compilation errors when gcc plugins are enabled diff --git a/.gitignore b/.gitignore index deea910..99f843f 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,4 @@ linux-*.tar.bz2 linux-*.tar.xz linux-*.sign kernel-*/ +u2mfn/ diff --git a/Makefile b/Makefile index 25f00e0..6853ea9 100644 --- a/Makefile +++ b/Makefile @@ -93,9 +93,17 @@ rpms-dom0: get-sources $(SPECFILE) rpms-nobuild: $(RPM_WITH_DIRS) --nobuild -bb $(SPECFILE) -rpms-just-build: +rpms-just-cleanbuild: + make clean -C kernel-4.14.57/linux-obj + +rpms-just-build-clean: rpms-just-cleanbuild + +rpms-just-build: $(RPM_WITH_DIRS) --short-circuit -bc $(SPECFILE) +rpms-just-install: + $(RPM_WITH_DIRS) --short-circuit -bi $(SPECFILE) + rpms-install: $(RPM_WITH_DIRS) -bi $(SPECFILE) @@ -110,8 +118,8 @@ verrel: # mop up, printing out exactly what was mopped. -.PHONY : clean -clean :: +.PHONY : rpmclean +rpmclean :: @echo "Running the %clean script of the rpmbuild..." $(RPM_WITH_DIRS) --clean --nodeps $(SPECFILE) diff --git a/config-qubes-me b/config-qubes-me new file mode 100644 index 000..6e46fae --- /dev/null +++ b/config-qubes-me @@ -0,0 +1,43 @@ +## comments need doube # like ## !!! +## single # are not comments! + +## remove AMD stuff: +CONFIG_PROCESSOR_SELECT=y +CONFIG_CPU_SUP_INTEL=y +# CONFIG_CPU_SUP_AMD is not set +# CONFIG_CPU_SUP_CENTAUR is not set +## CONFIG_GART_IOMMU is not set +## CONFIG_X86_MCE_AMD is not set +## CONFIG_PERF_EVENTS_AMD_POWER is not set +# CONFIG_MICROCODE_AMD is not set +## CONFIG_AMD_MEM_ENCRYPT is not set +## CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT is not set +## CONFIG_ARCH_USE_MEMREMAP_PROT is not set +# CONFIG_AMD_NUMA is not set +## CONFIG_X86_AMD_FREQ_SENSITIVITY is not set +## CONFIG_AMD_NB is not set +## CONFIG_SENSORS_FAM15H_POWER is not set +## CONFIG_AGP_AMD64 is not set +## CONFIG_EDAC_DECODE_MCE is not set +##
[qubes-users] Re: How to ccache kernel compilations
On Friday, August 24, 2018 at 11:24:27 AM UTC+2, Marcus Linsner wrote: > This is how a full(well, slightly modified) kernel compilation looks like > now, with ccache working: > ie. `time make rpms` > real 7m47.483s > user 9m2.507s > sys 6m47.245s > > cache directory /home/user/.ccache > primary config /home/user/.ccache/ccache.conf > secondary config (readonly)/etc/ccache.conf > stats zero time Fri Aug 24 11:09:03 2018 > cache hit (direct) 14047 > cache hit (preprocessed) 1 > cache miss 8 > cache hit rate 99.94 % > called for link 47 > called for preprocessing 21125 > unsupported code directive 4 > no input file 1092 > cleanups performed 0 > files in cache 42606 > cache size 865.4 MB > max cache size 20.0 GB > > The build phase actually takes only 2min (for 14k files): > real 2m1.674s > user 5m28.075s > sys 4m50.768s > > > cache directory /home/user/.ccache > primary config /home/user/.ccache/ccache.conf > secondary config (readonly)/etc/ccache.conf > stats zero time Fri Aug 24 11:17:37 2018 > cache hit (direct) 14011 > cache hit (preprocessed) 0 > cache miss 5 > cache hit rate 99.96 % > called for link 28 > called for preprocessing 21069 > unsupported code directive 4 > no input file342 > cleanups performed 0 > files in cache 42616 > cache size 865.6 MB > max cache size 20.0 GB And for comparison, a full %build phase when CONFIG_GCC_PLUGINS is untouched(aka set): real17m19.746s user125m44.920s sys 17m9.877s cache directory /home/user/.ccache primary config /home/user/.ccache/ccache.conf secondary config (readonly)/etc/ccache.conf stats zero time Fri Aug 24 11:27:18 2018 cache hit (direct)28 cache hit (preprocessed) 133 cache miss 13857 cache hit rate 1.15 % called for link 30 called for preprocessing 21075 unsupported code directive 4 no input file348 cleanups performed 0 files in cache 84685 cache size 1.7 GB max cache size 20.0 GB So you see, 15 more minutes than with ccache. Ok, maybe let's say that that was the first compilation with CONFIG_GCC_PLUGINS set (ie. cold cache?), so redoing it(make prep; ccache -z; time make rpms-just-build) means it should make use of the now primed ccache (ie. hot cache?): real18m34.318s user122m23.001s sys 17m7.478s cache directory /home/user/.ccache primary config /home/user/.ccache/ccache.conf secondary config (readonly)/etc/ccache.conf stats zero time Fri Aug 24 11:46:30 2018 cache hit (direct) 160 cache hit (preprocessed) 2 cache miss 13856 cache hit rate 1.16 % called for link 30 called for preprocessing 21075 unsupported code directive 4 no input file348 cleanups performed 0 files in cache126746 cache size 2.6 GB max cache size 20.0 GB It probably took one minute longer than before because I was using the other VMs for browsing (also started a few) But you get the point, 1.2% ccache hit rate. Appalling! :D On Friday, August 24, 2018 at 11:51:45 AM UTC+2, awokd wrote: > Any idea what those GCC plugins are for? Seems like it's usually a hassle > to track them down on distro version updates too. According to 'config-qubes' file [1] they help "Enable some more hardening options" According to '/home/user/qubes-linux-kernel/kernel-4.14.57/linux-4.14.57/arch/Kconfig' [2]: menuconfig GCC_PLUGINS bool "GCC plugins" depends on HAVE_GCC_PLUGINS depends on !COMPILE_TEST help GCC plugins are loadable modules that provide extra features to the compiler. They are useful for runtime instrumentation and static analysis. See Documentation/gcc-plugins.txt for details. (see url [3] at the end, for this gcc-plugins.txt) config GCC_PLUGIN_LATENT_ENTROPY bool "Generate some entropy during boot and runtime" depends on GCC_PLUGINS help By saying Y here the
Re: [qubes-users] Re: How to ccache kernel compilations
On Fri, August 24, 2018 9:24 am, Marcus Linsner wrote: > This is how a full(well, slightly modified) kernel compilation looks like > now, with ccache working: ie. `time make rpms` real 7m47.483s user 9m2.507s > sys 6m47.245s Any idea what those GCC plugins are for? Seems like it's usually a hassle to track them down on distro version updates too. -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/8af436925fcfaeafbf788ff4fe73179f.squirrel%40tt3j2x4k5ycaa5zt.onion. For more options, visit https://groups.google.com/d/optout.
[qubes-users] Re: How to ccache kernel compilations
This is how a full(well, slightly modified) kernel compilation looks like now, with ccache working: ie. `time make rpms` real7m47.483s user9m2.507s sys 6m47.245s cache directory /home/user/.ccache primary config /home/user/.ccache/ccache.conf secondary config (readonly)/etc/ccache.conf stats zero time Fri Aug 24 11:09:03 2018 cache hit (direct) 14047 cache hit (preprocessed) 1 cache miss 8 cache hit rate 99.94 % called for link 47 called for preprocessing 21125 unsupported code directive 4 no input file 1092 cleanups performed 0 files in cache 42606 cache size 865.4 MB max cache size 20.0 GB The build phase actually takes only 2min (for 14k files): real2m1.674s user5m28.075s sys 4m50.768s cache directory /home/user/.ccache primary config /home/user/.ccache/ccache.conf secondary config (readonly)/etc/ccache.conf stats zero time Fri Aug 24 11:17:37 2018 cache hit (direct) 14011 cache hit (preprocessed) 0 cache miss 5 cache hit rate 99.96 % called for link 28 called for preprocessing 21069 unsupported code directive 4 no input file342 cleanups performed 0 files in cache 42616 cache size 865.6 MB max cache size 20.0 GB -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/d462f7f0-bf71-4a5f-b91a-69f68d803be4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[qubes-users] Re: How to ccache kernel compilations
On Thursday, August 23, 2018 at 5:21:48 PM UTC+2, Marcus Linsner wrote:
> On Thursday, August 23, 2018 at 4:27:13 PM UTC+2, Marcus Linsner wrote:
> > I'm trying to use ccache to compile kernel(s) but after 1k files of cache
> > miss, I see only 8% hit rate, even if I keep the compilation dir
> > ('linux-obj'), do a clean (kernel Makefile's clean, not rpm's clean) and
> > re-issue the just-build again after a ccache -z (to clear ccache stats).
> ok I'm on to something: it's the .config !
> If I'm using default .config (aka in the source folder `make mrproper; make
> menuconfig`, Save then Exit) then copy that .config into ../linux-obj/ and
> then execute this twice:
> $ time make clean;ccache -z; time make -j18
> I get ccache hit direct over 90% ! which is how it should be.
>
> I'll post again if I find out exactly which options in the .config are the
> ccache-busting culprits
Well, it's CONFIG_GCC_PLUGINS, it has to be unset for ccache to work. (else you
get well under 8% hit rate, instead of like almost 100%)
and it's being set in the file `config-qubes` like so:
CONFIG_GCC_PLUGINS=y
CONFIG_GCC_PLUGIN_LATENT_ENTROPY=y
CONFIG_GCC_PLUGIN_STRUCTLEAK=y
it was also the reason these two patches were needed here:
https://groups.google.com/forum/#!topic/qubes-devel/Q3cdQKQS4Tk
to avoid compilation failure when compiling kernel 4.14.57
Great, now ccache works even with `make rpms` !!
If anyone knows another way or why I should keep those gcc plugins, lemme know?
:D
cache directory /home/user/.ccache
primary config /home/user/.ccache/ccache.conf
secondary config (readonly)/etc/ccache.conf
stats zero time Thu Aug 23 17:57:29 2018
cache hit (direct) 1029
cache hit (preprocessed) 8
cache miss32
cache hit rate 97.01 %
called for link 37
called for preprocessing1900
cache file missing 1
unsupported code directive 3
no input file704
cleanups performed 0
files in cache 73615
cache size 1.8 GB
max cache size 20.0 GB
--
You received this message because you are subscribed to the Google Groups
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to qubes-users+unsubscr...@googlegroups.com.
To post to this group, send email to qubes-users@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/qubes-users/765a1348-d849-43e9-aa93-2fd0dbf2cb26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[qubes-users] Re: How to ccache kernel compilations
On Thursday, August 23, 2018 at 4:27:13 PM UTC+2, Marcus Linsner wrote:
> I'm trying to use ccache to compile kernel(s) but after 1k files of cache
> miss, I see only 8% hit rate, even if I keep the compilation dir
> ('linux-obj'), do a clean (kernel Makefile's clean, not rpm's clean) and
> re-issue the just-build again after a ccache -z (to clear ccache stats).
ok I'm on to something: it's the .config !
If I'm using default .config (aka in the source folder `make mrproper; make
menuconfig`, Save then Exit) then copy that .config into ../linux-obj/ and then
execute this twice:
$ time make clean;ccache -z; time make -j18
I get ccache hit direct over 90% ! which is how it should be.
I'll post again if I find out exactly which options in the .config are the
ccache-busting culprits
--
You received this message because you are subscribed to the Google Groups
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to qubes-users+unsubscr...@googlegroups.com.
To post to this group, send email to qubes-users@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/qubes-users/87524cfd-1603-4269-a177-9525f0e6e9fe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[qubes-users] Re: How to ccache kernel compilations
On Thursday, August 23, 2018 at 4:36:51 PM UTC+2, Marcus Linsner wrote: > Quick question: > If kernel-4.14.57/linux-obj/Makefile is being regenerated on every build > (even if `linux-obj` dir is being kept between successing builds) does that > automatically cause `make` to rebuild everything and somehow invalidate > ccache? to answer my own question: no effect even if I stop it from being regenerated (eg. touched) -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/7b1bd6d3-78ae-4a8f-8366-015b138a9b68%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
