Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
On Wed, Apr 03, 2019 at 10:46:01AM -0700, Daniel Colascione wrote: > On Wed, Apr 3, 2019 at 10:20 AM Joel Fernandes wrote: > > > > On Wed, Apr 03, 2019 at 04:48:37PM +0900, Masahiro Yamada wrote: > > > On Thu, Mar 28, 2019 at 2:32 AM Joel Fernandes > > > wrote: > > > > > > > > On Mon, Mar 25, 2019 at 09:49:47AM -0400, Joel Fernandes wrote: > > > > > On Thu, Feb 14, 2019 at 07:19:29PM -0800, Alexei Starovoitov wrote: > > > > > > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) > > > > > > wrote: > > > > > > > Introduce in-kernel headers and other artifacts which are made > > > > > > > available > > > > > > > as an archive through proc (/proc/kheaders.txz file). This > > > > > > > archive makes > > > > > > > it possible to build kernel modules, run eBPF programs, and other > > > > > > > tracing programs that need to extend the kernel for tracing > > > > > > > purposes > > > > > > > without any dependency on the file system having headers and build > > > > > > > artifacts. > > > > > > > > > > > > > > On Android and embedded systems, it is common to switch kernels > > > > > > > but not > > > > > > > have kernel headers available on the file system. Raw kernel > > > > > > > headers > > > > > > > also cannot be copied into the filesystem like they can be on > > > > > > > other > > > > > > > distros, due to licensing and other issues. There's no > > > > > > > linux-headers > > > > > > > package on Android. Further once a different kernel is booted, any > > > > > > > headers stored on the file system will no longer be useful. By > > > > > > > storing > > > > > > > the headers as a compressed archive within the kernel, we can > > > > > > > avoid these > > > > > > > issues that have been a hindrance for a long time. > > > > > > > > > > > > The set looks good to me and since the main use case is building > > > > > > bpf progs > > > > > > I can route it via bpf-next tree if there are no objections. > > > > > > Masahiro, could you please ack it? > > > > > > > > > > FYI, Masahiro's comments were all address by v5: > > > > > https://lore.kernel.org/patchwork/project/lkml/list/?series=387311 > > > > > > > > > > I believe aren't more outstanding concerns. Could we consider it for > > > > > v5.2? > > > > > > > > Just to highlight the problem, today I booted v5.0 on an emulated > > > > Android > > > > device for some testing, that didn't have a set of prebuilt headers > > > > that we > > > > have been packaging on well known kernels, to get around this issue. > > > > This > > > > caused great pain and issues with what I was doing. I know me and many > > > > others > > > > really want this. With this I can boot an emulated Android device with > > > > IKCONFIG_PROC=y and run BCC with that that. Also I want to do the BCC > > > > side of > > > > the development, but first want to know if we can merge this upstream. > > > > > > > > Masahiro, I believe due diligence has been done in solidifying it as > > > > posted > > > > in the v5. Anything else we need to do here? Are you with the patches? > > > > > > > > > As you said, these updates are "cosmetic". > > > Nobody is worried about (or interested in) them. > > > Even if we miss something, they are fixable later. > > > > > > I accept embedding headers in the kernel, > > > but I do not like the part about external module build. > > > - kernel embeds build artifacts that depend on a > > >particular host-arch > > > - users do not know which compiler to use > > > Perhaps, you may remember my concerns. > > > https://lore.kernel.org/patchwork/patch/1046307/#1239501 > > > > Masahiro, > > I think we already discussed this right. The compiler issue is a user-issue > > - > > it is not a problem that has arisen because this patch. Anyone can build a > > kernel module today without this patch - using a compiler that is different > > from the running kernel. So I did not fully understand your concern. This > > patch does not ship a compiler in the archive. The compiler is upto the user > > based on user environment. They can already shoot themself in foot without > > this patch. > > > > Greg, > > Would be great to get your thoughts here as well about Masami's concern. > > > > Honestly, the "kernel module building artifacts" bit is a bonus with this > > patch. The main thing we are adding or that we need is really the headers > > (for BCC). I just thought shipping the kernel build artifacts would also be > > awesome because people can now build kernel modules without needing a kernel > > tree at all. > > > > But I also want this stuff merged so if folks are really unhappy with the > > module build artifacts and only want headers - then that's also fine with me > > and I can yank them - that would also mean though that this work cannot be a > > replacement for linux-headers package on distros, so that would be sad. > > > > thanks! > > IMHO, including the module build stuff is fine, and the stuff that > Masahiro mentions doesn't matter much. To be spe
Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
On Thu, Feb 21, 2019 at 11:34:41PM +0900, Masahiro Yamada wrote: > On Wed, Feb 20, 2019 at 12:17 AM Joel Fernandes > wrote: > > > > > Firstly, I want to apologize for not testing this and other corner cases you > > brought up. I should have known better. Since my build was working, I > > assumed > > that the feature is working. For that, I am very sorry. > > > You do not need to apologize. 0day bot usually catches build errors. > I guess 0day bot performs compile-tests only incrementally > and that is why we did not get any report. Oh ok :) thanks. > > Secondly, it turns out Module.symvers circularly dependency problem also > > exists with another use case. > > If one does 'make modules_prepare' in a base kernel tree and then tries to > > build modules with that tree, a warning like this is printed but the module > > still gets built: > > > > WARNING: Symbol version dump ./Module.symvers > >is missing; modules will have no dependencies and modversions. > > > > CC [M] /tmp/testmod/test.o > > Building modules, stage 2. > > MODPOST 1 modules > > CC /tmp/testmod/test.mod.o > > LD [M] /tmp/testmod/test.ko > > > > So, I am thinking that at least for first pass I will just drop the > > inclusion > > of Module.symvers in the archive and allow any modules built using > > /proc/kheaders.tar.xz to not use it. > > > > Kbuild will print a warning anyway when anyone tries to build using > > /proc/kheaders.tar.xz, so if the user really wants module symbol versioning > > then they should probably use a full kernel source tree with Module.symvers > > available. For our usecase, kernel symbol versioning is a bit useless when > > using /proc/kheaders.tar.gz because the proc file is generated with the same > > kernel that the module is being built against, and subsequently loaded into > > the kernel. So it is not likely that the CRC of a kernel symbol will be > > different from what the module expects. > > > Without Module.symver, modpost cannot check whether references are > resolvable or not. > > You will see "WARNING ... undefined" for every symbol referenced from > the module. > > > I am not an Android developer. > So, I will leave this judge to other people. IMO I don't see a way around this limiation but it would be nice if there was a way to make it work. Since the kernel modules being built by this mechanism are for tracing/debugging purposes, it is not a major concern for us. > One more request if you have a chance to submit the next version. > Please do not hide error messages. Actually it was intended to suppress noise, not hide errors as such. I have fixed all the errors in the next version and will be submitting it soon. Thanks a lot for the review! - Joel > I wondered why you redirected stdout/stderr from the script. > > I applied the following patch, and I tested. Then I see why. > > Please fix your code instead of hiding underlying problems. > > > diff --git a/kernel/Makefile b/kernel/Makefile > index 1d13a7a..a76ccbd 100644 > --- a/kernel/Makefile > +++ b/kernel/Makefile > @@ -148,7 +148,7 @@ $(obj)/kheaders.o: $(obj)/kheaders_data.h > targets += kheaders_data.txz > > quiet_cmd_genikh = GEN $(obj)/kheaders_data.txz > -cmd_genikh = $(srctree)/scripts/gen_ikh_data.sh $@ $^ >/dev/null 2>&1 > +cmd_genikh = $(srctree)/scripts/gen_ikh_data.sh $@ $^ > $(obj)/kheaders_data.txz: $(ikh_file_list) FORCE > $(call cmd,genikh) > > > > > > > masahiro@grover:~/workspace/linux-yamada$ make > CALLscripts/checksyscalls.sh > DESCEND objtool > CHK include/generated/compile.h > GEN kernel/kheaders_data.txz > find: ‘FORCE’: No such file or directory > 70106 blocks > Can't do inplace edit: kernel/kheaders_data.txz.tmp is not a regular file. > Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch is not a regular > file. > Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86 is not a > regular file. > Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include > is not a regular file. > Can't do inplace edit: > kernel/kheaders_data.txz.tmp/arch/x86/include/uapi is not a regular > file. > Can't do inplace edit: > kernel/kheaders_data.txz.tmp/arch/x86/include/uapi/asm is not a > regular file. > Can't do inplace edit: > kernel/kheaders_data.txz.tmp/arch/x86/include/generated is not a > regular file. > Can't do inplace edit: > kernel/kheaders_data.txz.tmp/arch/x86/include/generated/uapi is not a > regular file. > Can't do inplace edit: > kernel/kheaders_data.txz.tmp/arch/x86/include/generated/uapi/asm is > not a regular file. > Can't do inplace edit: > kernel/kheaders_data.txz.tmp/arch/x86/include/generated/asm is not a > regular file. > Can't do inplace edit: > kernel/kheaders_data.txz.tmp/arch/x86/include/asm is not a regular > file. > Can't do inplace edit: > kernel/kheaders_data.txz.tmp/arch/x86/include/asm/xen is not a regular > file. > Can't do inplace edit: > kernel/kheaders_data.txz.tmp/arch/x86/include/asm/u
Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
On Wed, Feb 20, 2019 at 12:17 AM Joel Fernandes wrote: > > Firstly, I want to apologize for not testing this and other corner cases you > brought up. I should have known better. Since my build was working, I assumed > that the feature is working. For that, I am very sorry. You do not need to apologize. 0day bot usually catches build errors. I guess 0day bot performs compile-tests only incrementally and that is why we did not get any report. > Secondly, it turns out Module.symvers circularly dependency problem also > exists with another use case. > If one does 'make modules_prepare' in a base kernel tree and then tries to > build modules with that tree, a warning like this is printed but the module > still gets built: > > WARNING: Symbol version dump ./Module.symvers >is missing; modules will have no dependencies and modversions. > > CC [M] /tmp/testmod/test.o > Building modules, stage 2. > MODPOST 1 modules > CC /tmp/testmod/test.mod.o > LD [M] /tmp/testmod/test.ko > > So, I am thinking that at least for first pass I will just drop the inclusion > of Module.symvers in the archive and allow any modules built using > /proc/kheaders.tar.xz to not use it. > > Kbuild will print a warning anyway when anyone tries to build using > /proc/kheaders.tar.xz, so if the user really wants module symbol versioning > then they should probably use a full kernel source tree with Module.symvers > available. For our usecase, kernel symbol versioning is a bit useless when > using /proc/kheaders.tar.gz because the proc file is generated with the same > kernel that the module is being built against, and subsequently loaded into > the kernel. So it is not likely that the CRC of a kernel symbol will be > different from what the module expects. Without Module.symver, modpost cannot check whether references are resolvable or not. You will see "WARNING ... undefined" for every symbol referenced from the module. I am not an Android developer. So, I will leave this judge to other people. One more request if you have a chance to submit the next version. Please do not hide error messages. I wondered why you redirected stdout/stderr from the script. I applied the following patch, and I tested. Then I see why. Please fix your code instead of hiding underlying problems. diff --git a/kernel/Makefile b/kernel/Makefile index 1d13a7a..a76ccbd 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -148,7 +148,7 @@ $(obj)/kheaders.o: $(obj)/kheaders_data.h targets += kheaders_data.txz quiet_cmd_genikh = GEN $(obj)/kheaders_data.txz -cmd_genikh = $(srctree)/scripts/gen_ikh_data.sh $@ $^ >/dev/null 2>&1 +cmd_genikh = $(srctree)/scripts/gen_ikh_data.sh $@ $^ $(obj)/kheaders_data.txz: $(ikh_file_list) FORCE $(call cmd,genikh) masahiro@grover:~/workspace/linux-yamada$ make CALLscripts/checksyscalls.sh DESCEND objtool CHK include/generated/compile.h GEN kernel/kheaders_data.txz find: ‘FORCE’: No such file or directory 70106 blocks Can't do inplace edit: kernel/kheaders_data.txz.tmp is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86 is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include/uapi is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include/uapi/asm is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include/generated is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include/generated/uapi is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include/generated/uapi/asm is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include/generated/asm is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include/asm is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include/asm/xen is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include/asm/uv is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include/asm/numachip is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include/asm/e820 is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include/asm/fpu is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include/asm/crypto is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include/asm/trace is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts is not a regular file. Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/genksyms is not a regular file. Can't do inplace edit: kernel/k
Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
On Tue, Feb 19, 2019 at 09:05:31AM +0300, Alexey Dobriyan wrote: > > /proc/kheaders.txz > > This is gross. It is a bit gross, but it solves a long standing problem we have nicely. And there's also /proc/config.gz. > > The feature is also buildable as a module just in case the user desires > > it not being part of the kernel image. This makes it possible to load > > and unload the headers on demand. A tracing program, or a kernel module > > builder can load the module, do its operations, and then unload the > > module to save kernel memory. > > Please explain how keeping headers on the filesystem is not OK due > to "licensing and other issues" but keeping a module on the filesystem > is OK. In the Android ecosystem, the Android teams only provide a "userspace system image" which goes on the system partition of the flash. This image is not GPL and doesn't contain anything GPL. It is thus not possible to put kernel headers on the system image and I already had many discussions on the subject with the teams, it is something that is just not done. Now for kernel modules, there's another image called the "vendor image" which is flashed onto the vendor parition, this is where kernel modules go. This vendor image is not provided by Google for non-Pixel devices. So we have no control over what goes there. We do know that kernel modules that are enabled will go there, and we do have control over enforcing that certain kernel modules should be built and available as they are mandatory for Android to function properly (and we can enforce this as a case of Android device compliance). So this solves the problem rather nicely. It is also useful for non-Android usecases where a linux-headers package needs to be installed for tracing programs. With this feature, such a package install step is not needed. > > > I can route it via bpf-next tree if there are no objections. > > Please don't. Why not? You do know that eBPF based compilation needs to process kernel headers to build the eBPF program? This was primarily inspired by problems with getting eBPF to work, but it also applies to other trace tools like systemtap that build for a module backend. In fact for Android, I am planning to have facilities to compile code for a module backend as well. > IKHD_ST IKHD_ED are bogus artifacts as others mentioned. > proc_create(S_IFREG) is redundant. > seq_file.h is not needed as is THIS_MODULE. > I'd say such data should live in their own section for easy extraction > with "objdump -j", something /proc/config.gz never did. Thanks for the suggestions, I will look into these and refresh my series. My first priority right now is to make incremental builds work. Other than these, other changes are mostly cosmetic so I can address them after. Of course I will address everything before posting again. - Joel
Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
On Tue, Feb 19, 2019 at 01:42:13PM +0900, Masahiro Yamada wrote: > On Tue, Feb 19, 2019 at 1:14 PM Masahiro Yamada > wrote: > > > > On Fri, Feb 15, 2019 at 11:48 PM Alexei Starovoitov > > wrote: > > > > > > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote: > > > > Introduce in-kernel headers and other artifacts which are made available > > > > as an archive through proc (/proc/kheaders.txz file). > > > > > > The extension '.txz' is not used in kernel code. > > > > > > '.tar.xz' is used for 'tarxz-pkg', 'perf-tarxz-src-pkg' etc. > > > > > > $ git grep '\.txz' > > $ git grep '\.tar\.xz' > > Documentation/admin-guide/README.rst: xz -cd linux-4.X.tar.xz | tar xvf > > - > > arch/x86/crypto/camellia-aesni-avx-asm_64.S: * > > http://koti.mbnet.fi/axh/crypto/camellia-BSD-1.2.0-aesni1.tar.xz > > crypto/testmgr.h: * > > https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > > crypto/testmgr.h: * > > https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > > crypto/testmgr.h: * > > https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > > crypto/testmgr.h: * > > https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > > crypto/testmgr.h: * > > https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > > scripts/package/Makefile: @echo ' perf-tarxz-src-pkg - Build > > $(perf-tar).tar.xz source tarball' > > tools/testing/selftests/gen_kselftest_tar.sh: > > ext=".tar.xz" > > > > > > > > I prefer '.tar.xz' for consistency. > > > > > > > > > > > > > > BTW, have you ever looked at scripts/extract-ikconfig? > > > > You added IKHD_ST and IKHD_ED > > just to mimic kernel/configs.c > > > > It is currently pointless without the extracting tool, > > but you might think it is useful to extract headers > > from vmlinux or the module without mounting procfs. > > > > > > > > > > > > This archive makes > > > > it possible to build kernel modules, run eBPF programs, and other > > > > tracing programs that need to extend the kernel for tracing purposes > > > > without any dependency on the file system having headers and build > > > > artifacts. > > > > > > > > On Android and embedded systems, it is common to switch kernels but not > > > > have kernel headers available on the file system. Raw kernel headers > > > > also cannot be copied into the filesystem like they can be on other > > > > distros, due to licensing and other issues. There's no linux-headers > > > > package on Android. Further once a different kernel is booted, any > > > > headers stored on the file system will no longer be useful. By storing > > > > the headers as a compressed archive within the kernel, we can avoid > > > > these > > > > issues that have been a hindrance for a long time. > > > > > > The set looks good to me and since the main use case is building bpf progs > > > I can route it via bpf-next tree if there are no objections. > > > Masahiro, could you please ack it? > > > > > > Honestly, I was not tracking this thread > > since I did not know I was responsible for this. > > > > > > I just started to take a closer look, then immediately got scared. > > > > This version is not mature enough for the merge. > > > > > > > > First of all, this patch cannot be compiled out-of-tree (O= option). > > > > > > I do not know why 0-day bot did not catch this apparent breakage. > > > > > > $ make -j8 O=hoge > > make[1]: Entering directory '/home/masahiro/workspace/bsp/linux/hoge' > > GEN Makefile > > Using .. as source for kernel > > DESCEND objtool > > CALL../scripts/checksyscalls.sh > > CHK include/generated/compile.h > > make[2]: *** No rule to make target 'Module.symvers', needed by > > 'kernel/kheaders_data.txz'. Stop. > > make[2]: *** Waiting for unfinished jobs > > /home/masahiro/workspace/bsp/linux/Makefile:1043: recipe for target > > 'kernel' failed > > make[1]: *** [kernel] Error 2 > > make[1]: *** Waiting for unfinished jobs > > make[1]: Leaving directory '/home/masahiro/workspace/bsp/linux/hoge' > > Makefile:152: recipe for target 'sub-make' failed > > make: *** [sub-make] Error 2 > > > > > I saw this build error for in-tree building as well. > > We cannot build this from a pristine source tree. > > For example, I observed the build error in the following procedure. > > $ make mrproper > $ make defconfig > Set CONFIG_IKHEADERS_PROC=y > $ make > > > > > Module.symvers is generated in the modpost stage > (the very last stage of build). > > But, vmlinux depends on kernel/kheaders_data.txz, > which includes Module.symvers. Firstly, I want to apologize for not testing this and other corner cases you brought up. I should have known better. Since my build was working, I assumed that the feature is working. For that, I am very sorry. Secondly, it turns out Module.symvers circularly dependency problem also exists with another use case. If one does 'make modules_prepare' in a base kernel tree and then tries to build modules with that tree, a warning like th
Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
> /proc/kheaders.txz This is gross. > The feature is also buildable as a module just in case the user desires > it not being part of the kernel image. This makes it possible to load > and unload the headers on demand. A tracing program, or a kernel module > builder can load the module, do its operations, and then unload the > module to save kernel memory. Please explain how keeping headers on the filesystem is not OK due to "licensing and other issues" but keeping a module on the filesystem is OK. > > I can route it via bpf-next tree if there are no objections. Please don't. IKHD_ST IKHD_ED are bogus artifacts as others mentioned. proc_create(S_IFREG) is redundant. seq_file.h is not needed as is THIS_MODULE. I'd say such data should live in their own section for easy extraction with "objdump -j", something /proc/config.gz never did.
Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
On Tue, Feb 19, 2019 at 01:42:13PM +0900, Masahiro Yamada wrote: [..] > > > > This archive makes > > > > it possible to build kernel modules, run eBPF programs, and other > > > > tracing programs that need to extend the kernel for tracing purposes > > > > without any dependency on the file system having headers and build > > > > artifacts. > > > > > > > > On Android and embedded systems, it is common to switch kernels but not > > > > have kernel headers available on the file system. Raw kernel headers > > > > also cannot be copied into the filesystem like they can be on other > > > > distros, due to licensing and other issues. There's no linux-headers > > > > package on Android. Further once a different kernel is booted, any > > > > headers stored on the file system will no longer be useful. By storing > > > > the headers as a compressed archive within the kernel, we can avoid > > > > these > > > > issues that have been a hindrance for a long time. > > > > > > The set looks good to me and since the main use case is building bpf progs > > > I can route it via bpf-next tree if there are no objections. > > > Masahiro, could you please ack it? > > > > > > Honestly, I was not tracking this thread > > since I did not know I was responsible for this. > > > > > > I just started to take a closer look, then immediately got scared. > > > > This version is not mature enough for the merge. > > > > > > > > First of all, this patch cannot be compiled out-of-tree (O= option). > > > > > > I do not know why 0-day bot did not catch this apparent breakage. > > > > > > $ make -j8 O=hoge > > make[1]: Entering directory '/home/masahiro/workspace/bsp/linux/hoge' > > GEN Makefile > > Using .. as source for kernel > > DESCEND objtool > > CALL../scripts/checksyscalls.sh > > CHK include/generated/compile.h > > make[2]: *** No rule to make target 'Module.symvers', needed by > > 'kernel/kheaders_data.txz'. Stop. > > make[2]: *** Waiting for unfinished jobs > > /home/masahiro/workspace/bsp/linux/Makefile:1043: recipe for target > > 'kernel' failed > > make[1]: *** [kernel] Error 2 > > make[1]: *** Waiting for unfinished jobs > > make[1]: Leaving directory '/home/masahiro/workspace/bsp/linux/hoge' > > Makefile:152: recipe for target 'sub-make' failed > > make: *** [sub-make] Error 2 > > > > > I saw this build error for in-tree building as well. > > We cannot build this from a pristine source tree. > > For example, I observed the build error in the following procedure. > > $ make mrproper > $ make defconfig > Set CONFIG_IKHEADERS_PROC=y > $ make > > > > > Module.symvers is generated in the modpost stage > (the very last stage of build). > > But, vmlinux depends on kernel/kheaders_data.txz, > which includes Module.symvers. > > So, this is not so simple since it is a circular dependency... I guess I was not building a pristine tree either and missed this circular dependency :-/ . Any ideas on how we can fix the Module.symvers issue? One idea is to reserve the space in the binaries, but only populate the space reserved in the binary *after* the modpost stage, once the archive is ready.. thanks, - Joel
Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
On Tue, Feb 19, 2019 at 1:14 PM Masahiro Yamada wrote: > > On Fri, Feb 15, 2019 at 11:48 PM Alexei Starovoitov > wrote: > > > > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote: > > > Introduce in-kernel headers and other artifacts which are made available > > > as an archive through proc (/proc/kheaders.txz file). > > > The extension '.txz' is not used in kernel code. > > > '.tar.xz' is used for 'tarxz-pkg', 'perf-tarxz-src-pkg' etc. > > > $ git grep '\.txz' > $ git grep '\.tar\.xz' > Documentation/admin-guide/README.rst: xz -cd linux-4.X.tar.xz | tar xvf - > arch/x86/crypto/camellia-aesni-avx-asm_64.S: * > http://koti.mbnet.fi/axh/crypto/camellia-BSD-1.2.0-aesni1.tar.xz > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > scripts/package/Makefile: @echo ' perf-tarxz-src-pkg - Build > $(perf-tar).tar.xz source tarball' > tools/testing/selftests/gen_kselftest_tar.sh: > ext=".tar.xz" > > > > I prefer '.tar.xz' for consistency. > > > > > > > BTW, have you ever looked at scripts/extract-ikconfig? > > You added IKHD_ST and IKHD_ED > just to mimic kernel/configs.c > > It is currently pointless without the extracting tool, > but you might think it is useful to extract headers > from vmlinux or the module without mounting procfs. > > > > > > > This archive makes > > > it possible to build kernel modules, run eBPF programs, and other > > > tracing programs that need to extend the kernel for tracing purposes > > > without any dependency on the file system having headers and build > > > artifacts. > > > > > > On Android and embedded systems, it is common to switch kernels but not > > > have kernel headers available on the file system. Raw kernel headers > > > also cannot be copied into the filesystem like they can be on other > > > distros, due to licensing and other issues. There's no linux-headers > > > package on Android. Further once a different kernel is booted, any > > > headers stored on the file system will no longer be useful. By storing > > > the headers as a compressed archive within the kernel, we can avoid these > > > issues that have been a hindrance for a long time. > > > > The set looks good to me and since the main use case is building bpf progs > > I can route it via bpf-next tree if there are no objections. > > Masahiro, could you please ack it? > > > Honestly, I was not tracking this thread > since I did not know I was responsible for this. > > > I just started to take a closer look, then immediately got scared. > > This version is not mature enough for the merge. > > > > First of all, this patch cannot be compiled out-of-tree (O= option). > > > I do not know why 0-day bot did not catch this apparent breakage. > > > $ make -j8 O=hoge > make[1]: Entering directory '/home/masahiro/workspace/bsp/linux/hoge' > GEN Makefile > Using .. as source for kernel > DESCEND objtool > CALL../scripts/checksyscalls.sh > CHK include/generated/compile.h > make[2]: *** No rule to make target 'Module.symvers', needed by > 'kernel/kheaders_data.txz'. Stop. > make[2]: *** Waiting for unfinished jobs > /home/masahiro/workspace/bsp/linux/Makefile:1043: recipe for target > 'kernel' failed > make[1]: *** [kernel] Error 2 > make[1]: *** Waiting for unfinished jobs > make[1]: Leaving directory '/home/masahiro/workspace/bsp/linux/hoge' > Makefile:152: recipe for target 'sub-make' failed > make: *** [sub-make] Error 2 I saw this build error for in-tree building as well. We cannot build this from a pristine source tree. For example, I observed the build error in the following procedure. $ make mrproper $ make defconfig Set CONFIG_IKHEADERS_PROC=y $ make Module.symvers is generated in the modpost stage (the very last stage of build). But, vmlinux depends on kernel/kheaders_data.txz, which includes Module.symvers. So, this is not so simple since it is a circular dependency... -- Best Regards Masahiro Yamada
Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
On Tue, Feb 19, 2019 at 01:14:11PM +0900, Masahiro Yamada wrote: > On Fri, Feb 15, 2019 at 11:48 PM Alexei Starovoitov > wrote: > > > > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote: > > > Introduce in-kernel headers and other artifacts which are made available > > > as an archive through proc (/proc/kheaders.txz file). > > > The extension '.txz' is not used in kernel code. > > > '.tar.xz' is used for 'tarxz-pkg', 'perf-tarxz-src-pkg' etc. > > > $ git grep '\.txz' > $ git grep '\.tar\.xz' > Documentation/admin-guide/README.rst: xz -cd linux-4.X.tar.xz | tar xvf - > arch/x86/crypto/camellia-aesni-avx-asm_64.S: * > http://koti.mbnet.fi/axh/crypto/camellia-BSD-1.2.0-aesni1.tar.xz > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz > scripts/package/Makefile: @echo ' perf-tarxz-src-pkg - Build > $(perf-tar).tar.xz source tarball' > tools/testing/selftests/gen_kselftest_tar.sh: > ext=".tar.xz" > > > > I prefer '.tar.xz' for consistency. Ok, I will change it to that. > BTW, have you ever looked at scripts/extract-ikconfig? > > You added IKHD_ST and IKHD_ED > just to mimic kernel/configs.c > > It is currently pointless without the extracting tool, > but you might think it is useful to extract headers > from vmlinux or the module without mounting procfs. I am planing add to extraction support in the future, so I prefer to leave the markers as it is for now. Hope that's Ok with you. > > > This archive makes > > > it possible to build kernel modules, run eBPF programs, and other > > > tracing programs that need to extend the kernel for tracing purposes > > > without any dependency on the file system having headers and build > > > artifacts. > > > > > > On Android and embedded systems, it is common to switch kernels but not > > > have kernel headers available on the file system. Raw kernel headers > > > also cannot be copied into the filesystem like they can be on other > > > distros, due to licensing and other issues. There's no linux-headers > > > package on Android. Further once a different kernel is booted, any > > > headers stored on the file system will no longer be useful. By storing > > > the headers as a compressed archive within the kernel, we can avoid these > > > issues that have been a hindrance for a long time. > > > > The set looks good to me and since the main use case is building bpf progs > > I can route it via bpf-next tree if there are no objections. > > Masahiro, could you please ack it? > > > Honestly, I was not tracking this thread > since I did not know I was responsible for this. > > > I just started to take a closer look, then immediately got scared. > > This version is not mature enough for the merge. > First of all, this patch cannot be compiled out-of-tree (O= option). Oh, ok. This is not how I build the kernel. So I am sorry for missing that. I will try this out-of-tree build option and fix it. > I do not know why 0-day bot did not catch this apparent breakage. > > > $ make -j8 O=hoge > make[1]: Entering directory '/home/masahiro/workspace/bsp/linux/hoge' > GEN Makefile > Using .. as source for kernel > DESCEND objtool > CALL../scripts/checksyscalls.sh > CHK include/generated/compile.h > make[2]: *** No rule to make target 'Module.symvers', needed by > 'kernel/kheaders_data.txz'. Stop. > make[2]: *** Waiting for unfinished jobs > /home/masahiro/workspace/bsp/linux/Makefile:1043: recipe for target > 'kernel' failed > make[1]: *** [kernel] Error 2 > make[1]: *** Waiting for unfinished jobs > make[1]: Leaving directory '/home/masahiro/workspace/bsp/linux/hoge' > Makefile:152: recipe for target 'sub-make' failed > make: *** [sub-make] Error 2 > > > > > > I was able to compile it in-tree > but it makes the incremental build extremely slow. > > (Here, the incremental build means > "make" without changing any code after the full build.) > > > > > Before this patch, "make -j8" took 11 sec on my machine. > > real 0m11.777s > user 0m16.608s > sys 0m5.164s > > > > After this patch, x86_64_defconfig + CONFIG_IKHEADERS_PROC=y > takes 53 sec for me since kernel/kheaders_data.txz is regenerated > every time even when you did not touch any source file. Hmm, true. Let me know look into that as well.. > $ time make -j8 > DESCEND objtool > CALLscripts/checksyscalls.sh > CHK include/generated/compile.h > GEN kernel/kheaders_data.txz > UPD kernel/kheaders_data.h > CC kernel/kheaders.o > AR kernel/built-in.a > GEN .version > CHK include/generated/compile.h > UPD include/generated/compile.h
Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
On Tue, Feb 19, 2019 at 01:14:11PM +0900, Masahiro Yamada wrote: > > I was able to compile it in-tree > but it makes the incremental build extremely slow. > > (Here, the incremental build means > "make" without changing any code after the full build.) > > Before this patch, "make -j8" took 11 sec on my machine. > > After this patch, x86_64_defconfig + CONFIG_IKHEADERS_PROC=y > takes 53 sec for me since kernel/kheaders_data.txz is regenerated > every time even when you did not touch any source file. > > We should take time for careful review and test. > > Please give me more time for thorough review. Thanks for taking a look. All are very valid points. Incremental build is must have.
Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
On Fri, Feb 15, 2019 at 11:48 PM Alexei Starovoitov wrote: > > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote: > > Introduce in-kernel headers and other artifacts which are made available > > as an archive through proc (/proc/kheaders.txz file). The extension '.txz' is not used in kernel code. '.tar.xz' is used for 'tarxz-pkg', 'perf-tarxz-src-pkg' etc. $ git grep '\.txz' $ git grep '\.tar\.xz' Documentation/admin-guide/README.rst: xz -cd linux-4.X.tar.xz | tar xvf - arch/x86/crypto/camellia-aesni-avx-asm_64.S: * http://koti.mbnet.fi/axh/crypto/camellia-BSD-1.2.0-aesni1.tar.xz crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz scripts/package/Makefile: @echo ' perf-tarxz-src-pkg - Build $(perf-tar).tar.xz source tarball' tools/testing/selftests/gen_kselftest_tar.sh: ext=".tar.xz" I prefer '.tar.xz' for consistency. BTW, have you ever looked at scripts/extract-ikconfig? You added IKHD_ST and IKHD_ED just to mimic kernel/configs.c It is currently pointless without the extracting tool, but you might think it is useful to extract headers from vmlinux or the module without mounting procfs. > > This archive makes > > it possible to build kernel modules, run eBPF programs, and other > > tracing programs that need to extend the kernel for tracing purposes > > without any dependency on the file system having headers and build > > artifacts. > > > > On Android and embedded systems, it is common to switch kernels but not > > have kernel headers available on the file system. Raw kernel headers > > also cannot be copied into the filesystem like they can be on other > > distros, due to licensing and other issues. There's no linux-headers > > package on Android. Further once a different kernel is booted, any > > headers stored on the file system will no longer be useful. By storing > > the headers as a compressed archive within the kernel, we can avoid these > > issues that have been a hindrance for a long time. > > The set looks good to me and since the main use case is building bpf progs > I can route it via bpf-next tree if there are no objections. > Masahiro, could you please ack it? Honestly, I was not tracking this thread since I did not know I was responsible for this. I just started to take a closer look, then immediately got scared. This version is not mature enough for the merge. First of all, this patch cannot be compiled out-of-tree (O= option). I do not know why 0-day bot did not catch this apparent breakage. $ make -j8 O=hoge make[1]: Entering directory '/home/masahiro/workspace/bsp/linux/hoge' GEN Makefile Using .. as source for kernel DESCEND objtool CALL../scripts/checksyscalls.sh CHK include/generated/compile.h make[2]: *** No rule to make target 'Module.symvers', needed by 'kernel/kheaders_data.txz'. Stop. make[2]: *** Waiting for unfinished jobs /home/masahiro/workspace/bsp/linux/Makefile:1043: recipe for target 'kernel' failed make[1]: *** [kernel] Error 2 make[1]: *** Waiting for unfinished jobs make[1]: Leaving directory '/home/masahiro/workspace/bsp/linux/hoge' Makefile:152: recipe for target 'sub-make' failed make: *** [sub-make] Error 2 I was able to compile it in-tree but it makes the incremental build extremely slow. (Here, the incremental build means "make" without changing any code after the full build.) Before this patch, "make -j8" took 11 sec on my machine. real 0m11.777s user 0m16.608s sys 0m5.164s After this patch, x86_64_defconfig + CONFIG_IKHEADERS_PROC=y takes 53 sec for me since kernel/kheaders_data.txz is regenerated every time even when you did not touch any source file. $ time make -j8 DESCEND objtool CALLscripts/checksyscalls.sh CHK include/generated/compile.h GEN kernel/kheaders_data.txz UPD kernel/kheaders_data.h CC kernel/kheaders.o AR kernel/built-in.a GEN .version CHK include/generated/compile.h UPD include/generated/compile.h CC init/version.o AR init/built-in.a AR built-in.a LD vmlinux.o MODPOST vmlinux.o KSYM.tmp_kallsyms1.o KSYM.tmp_kallsyms2.o LD vmlinux SORTEX vmlinux SYSMAP System.map Building modules, stage 2. CC arch/x86/boot/version.o MODPOST 17 modules VOFFSET arch/x86/boot/compressed/../voffset.h OBJCOPY arch/x86/boot/compressed/vmlinux.bin RELOCS arch/x86/boot/compressed/vmlinux.relocs CC arch/x86/boot/compressed/kaslr.o GZIParch/x86/boot/compressed/vmlinux.bin.gz CC arch/x86/boot/compressed/misc.o MKPIGGY arch/x86/boot/compressed/piggy.S AS arch
Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
Joel Fernandes writes: > On Thu, Feb 14, 2019 at 07:19:29PM -0800, Alexei Starovoitov wrote: >> On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote: >> > Introduce in-kernel headers and other artifacts which are made available >> > as an archive through proc (/proc/kheaders.txz file). This archive makes >> > it possible to build kernel modules, run eBPF programs, and other >> > tracing programs that need to extend the kernel for tracing purposes >> > without any dependency on the file system having headers and build >> > artifacts. >> > >> > On Android and embedded systems, it is common to switch kernels but not >> > have kernel headers available on the file system. Raw kernel headers >> > also cannot be copied into the filesystem like they can be on other >> > distros, due to licensing and other issues. There's no linux-headers >> > package on Android. Further once a different kernel is booted, any >> > headers stored on the file system will no longer be useful. By storing >> > the headers as a compressed archive within the kernel, we can avoid these >> > issues that have been a hindrance for a long time. >> >> The set looks good to me and since the main use case is building bpf progs >> I can route it via bpf-next tree if there are no objections. >> Masahiro, could you please ack it? >> > > Yes, eBPF is one of the usecases. After this, I am also planning to send > patches to BCC so that it can use this feature when compiling C to eBPF. > Tested-by: Manoj Rao I think this can definitely make it easier to use eBPF on Android. Thanks for initiating this. > Thanks! > > - Joel -- Manoj http://www.mycpu.org
Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
On Thu, Feb 14, 2019 at 07:19:29PM -0800, Alexei Starovoitov wrote: > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote: > > Introduce in-kernel headers and other artifacts which are made available > > as an archive through proc (/proc/kheaders.txz file). This archive makes > > it possible to build kernel modules, run eBPF programs, and other > > tracing programs that need to extend the kernel for tracing purposes > > without any dependency on the file system having headers and build > > artifacts. > > > > On Android and embedded systems, it is common to switch kernels but not > > have kernel headers available on the file system. Raw kernel headers > > also cannot be copied into the filesystem like they can be on other > > distros, due to licensing and other issues. There's no linux-headers > > package on Android. Further once a different kernel is booted, any > > headers stored on the file system will no longer be useful. By storing > > the headers as a compressed archive within the kernel, we can avoid these > > issues that have been a hindrance for a long time. > > The set looks good to me and since the main use case is building bpf progs > I can route it via bpf-next tree if there are no objections. > Masahiro, could you please ack it? > Yes, eBPF is one of the usecases. After this, I am also planning to send patches to BCC so that it can use this feature when compiling C to eBPF. Thanks! - Joel
Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote: > Introduce in-kernel headers and other artifacts which are made available > as an archive through proc (/proc/kheaders.txz file). This archive makes > it possible to build kernel modules, run eBPF programs, and other > tracing programs that need to extend the kernel for tracing purposes > without any dependency on the file system having headers and build > artifacts. > > On Android and embedded systems, it is common to switch kernels but not > have kernel headers available on the file system. Raw kernel headers > also cannot be copied into the filesystem like they can be on other > distros, due to licensing and other issues. There's no linux-headers > package on Android. Further once a different kernel is booted, any > headers stored on the file system will no longer be useful. By storing > the headers as a compressed archive within the kernel, we can avoid these > issues that have been a hindrance for a long time. The set looks good to me and since the main use case is building bpf progs I can route it via bpf-next tree if there are no objections. Masahiro, could you please ack it?
Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
On 2/11/19 9:35 AM, Joel Fernandes (Google) wrote: Introduce in-kernel headers and other artifacts which are made available as an archive through proc (/proc/kheaders.txz file). This archive makes it possible to build kernel modules, run eBPF programs, and other tracing programs that need to extend the kernel for tracing purposes without any dependency on the file system having headers and build artifacts. On Android and embedded systems, it is common to switch kernels but not have kernel headers available on the file system. Raw kernel headers also cannot be copied into the filesystem like they can be on other distros, due to licensing and other issues. There's no linux-headers package on Android. Further once a different kernel is booted, any headers stored on the file system will no longer be useful. By storing the headers as a compressed archive within the kernel, we can avoid these issues that have been a hindrance for a long time. The feature is also buildable as a module just in case the user desires it not being part of the kernel image. This makes it possible to load and unload the headers on demand. A tracing program, or a kernel module builder can load the module, do its operations, and then unload the module to save kernel memory. The total memory needed is 3.8MB. The code to read the headers is based on /proc/config.gz code and uses the same technique to embed the headers. To build a module, the below steps have been tested on an x86 machine: modprobe kheaders rm -rf $HOME/headers mkdir -p $HOME/headers tar -xvf /proc/kheaders.txz -C $HOME/headers >/dev/null cd my-kernel-module make -C $HOME/headers M=$(pwd) modules rmmod kheaders Signed-off-by: Joel Fernandes (Google) Acked-by: Karim Yaghmour --- Changes since v1: - removed IKH_EXTRA variable, not needed (Masahiro Yamada) - small fix ups to selftest - added target to main Makefile etc - added MODULE_LICENSE to test module - made selftest more quiet Changes since RFC: Both changes bring size down to 3.8MB: - use xz for compression - strip comments except SPDX lines - Call out the module name in Kconfig - Also added selftests in second patch to ensure headers are always working. Documentation/dontdiff| 1 + init/Kconfig | 11 ++ kernel/.gitignore | 2 ++ kernel/Makefile | 27 ++ kernel/kheaders.c | 74 +++ scripts/gen_ikh_data.sh | 19 ++ scripts/strip-comments.pl | 8 + 7 files changed, 142 insertions(+) create mode 100644 kernel/kheaders.c create mode 100755 scripts/gen_ikh_data.sh create mode 100755 scripts/strip-comments.pl [snip] -- Karim Yaghmour CEO - Opersys inc. / www.opersys.com http://twitter.com/karimyaghmour
[PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
Introduce in-kernel headers and other artifacts which are made available as an archive through proc (/proc/kheaders.txz file). This archive makes it possible to build kernel modules, run eBPF programs, and other tracing programs that need to extend the kernel for tracing purposes without any dependency on the file system having headers and build artifacts. On Android and embedded systems, it is common to switch kernels but not have kernel headers available on the file system. Raw kernel headers also cannot be copied into the filesystem like they can be on other distros, due to licensing and other issues. There's no linux-headers package on Android. Further once a different kernel is booted, any headers stored on the file system will no longer be useful. By storing the headers as a compressed archive within the kernel, we can avoid these issues that have been a hindrance for a long time. The feature is also buildable as a module just in case the user desires it not being part of the kernel image. This makes it possible to load and unload the headers on demand. A tracing program, or a kernel module builder can load the module, do its operations, and then unload the module to save kernel memory. The total memory needed is 3.8MB. The code to read the headers is based on /proc/config.gz code and uses the same technique to embed the headers. To build a module, the below steps have been tested on an x86 machine: modprobe kheaders rm -rf $HOME/headers mkdir -p $HOME/headers tar -xvf /proc/kheaders.txz -C $HOME/headers >/dev/null cd my-kernel-module make -C $HOME/headers M=$(pwd) modules rmmod kheaders Signed-off-by: Joel Fernandes (Google) --- Changes since v1: - removed IKH_EXTRA variable, not needed (Masahiro Yamada) - small fix ups to selftest - added target to main Makefile etc - added MODULE_LICENSE to test module - made selftest more quiet Changes since RFC: Both changes bring size down to 3.8MB: - use xz for compression - strip comments except SPDX lines - Call out the module name in Kconfig - Also added selftests in second patch to ensure headers are always working. Documentation/dontdiff| 1 + init/Kconfig | 11 ++ kernel/.gitignore | 2 ++ kernel/Makefile | 27 ++ kernel/kheaders.c | 74 +++ scripts/gen_ikh_data.sh | 19 ++ scripts/strip-comments.pl | 8 + 7 files changed, 142 insertions(+) create mode 100644 kernel/kheaders.c create mode 100755 scripts/gen_ikh_data.sh create mode 100755 scripts/strip-comments.pl diff --git a/Documentation/dontdiff b/Documentation/dontdiff index 2228fcc8e29f..05a2319ee2a2 100644 --- a/Documentation/dontdiff +++ b/Documentation/dontdiff @@ -151,6 +151,7 @@ int8.c kallsyms kconfig keywords.c +kheaders_data.h* ksym.c* ksym.h* kxgettext diff --git a/init/Kconfig b/init/Kconfig index c9386a365eea..9fbf4f73d98c 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -563,6 +563,17 @@ config IKCONFIG_PROC This option enables access to the kernel configuration file through /proc/config.gz. +config IKHEADERS_PROC + tristate "Enable kernel header artifacts through /proc/kheaders.txz" + select BUILD_BIN2C + depends on PROC_FS + help + This option enables access to the kernel header and other artifacts that + are generated during the build process. These can be used to build kernel + modules, and other in-kernel programs such as those generated by eBPF + and systemtap tools. If you build the headers as a module, a module + called kheaders.ko is built which can be loaded to get access to them. + config LOG_BUF_SHIFT int "Kernel log buffer size (16 => 64KB, 17 => 128KB)" range 12 25 diff --git a/kernel/.gitignore b/kernel/.gitignore index b3097bde4e9c..6acf71acbdcb 100644 --- a/kernel/.gitignore +++ b/kernel/.gitignore @@ -3,5 +3,7 @@ # config_data.h config_data.gz +kheaders_data.h +kheaders_data.txz timeconst.h hz.bc diff --git a/kernel/Makefile b/kernel/Makefile index 6aa7543bcdb2..1d13a7a6c537 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -70,6 +70,7 @@ obj-$(CONFIG_UTS_NS) += utsname.o obj-$(CONFIG_USER_NS) += user_namespace.o obj-$(CONFIG_PID_NS) += pid_namespace.o obj-$(CONFIG_IKCONFIG) += configs.o +obj-$(CONFIG_IKHEADERS_PROC) += kheaders.o obj-$(CONFIG_SMP) += stop_machine.o obj-$(CONFIG_KPROBES_SANITY_TEST) += test_kprobes.o obj-$(CONFIG_AUDIT) += audit.o auditfilter.o @@ -130,3 +131,29 @@ filechk_ikconfiggz = \ targets += config_data.h $(obj)/config_data.h: $(obj)/config_data.gz FORCE $(call filechk,ikconfiggz) + +# Build a list of in-kernel headers for building kernel modules +ikh_file_list := include/ +ikh_file_list += arch/$(ARCH)/Makefile +ikh_file_list += arch/$(ARCH)/include/ +ikh_file_list += scripts/ +ikh_file_list += Makefile +ikh_file_list += Module.symvers +ifeq ($(CONFIG_STACK_VALIDATION), y) +i
