Re: [WIP] Create kernel-core and kernel-drivers subpackages
On Wednesday, March 19, 2014, 2:09:15 PM, Josh Boyer wrote: On Wed, Mar 19, 2014 at 2:02 PM, Bill Nottingham nott...@splat.cc wrote: Josh Boyer (jwbo...@fedoraproject.org) said: 2) A per-arch filter list, because the existing one that works on x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad. OK, I sorted this out this week. I believe the only arch left to do is s390x and that's only because I forgot about it. Oops. Is this even needed on s390 for reasons other than consistency? Similarly with power, is the idea to have a core kernel for running on an LPAR and then -drivers for the rest of it? Needed? Probably not. At the moment it's not possible to build a normal kernel on one arch and the split on another. If we're going to go off and make changes to anaconda and yum and dnf to cope with this, consistency on what is shipped is probably a good thing. That being said, it is flexible in terms of the content of those packages. So ppc64 could do what you suggest. s390x would arguably just shove almost everything in -drivers. In reality, I expect most arches to just install both packages anyway. If you update the ppc64 kernel package, please also do the same for the ppc 32-bit kernel. There are a few of us out there with vintage PPC hardware. This week, I picked up an IBM 7046-B50 (32-bit only, CHRP), to go with the PPC Macs (G5, G4, G3) that I've been gathering with the intent of recreating a minimal vintage PPC spin (likely based on LXDE). This would include support for other non-Mac 32-bit PPC systems. While most of the official effort is 64-bit (BE and now LE), the PPC arch still builds 32-bit userspace packages, and has stated that in their wiki that they have no intent to discontinue 32-bit kernels. Thanks! Al ___ kernel mailing list kernel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/kernel
Re: [WIP] Create kernel-core and kernel-drivers subpackages
On Fri, Mar 21, 2014 at 7:54 AM, Al Dunsmuir al.dunsm...@sympatico.ca wrote: On Wednesday, March 19, 2014, 2:09:15 PM, Josh Boyer wrote: On Wed, Mar 19, 2014 at 2:02 PM, Bill Nottingham nott...@splat.cc wrote: Josh Boyer (jwbo...@fedoraproject.org) said: 2) A per-arch filter list, because the existing one that works on x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad. OK, I sorted this out this week. I believe the only arch left to do is s390x and that's only because I forgot about it. Oops. Is this even needed on s390 for reasons other than consistency? Similarly with power, is the idea to have a core kernel for running on an LPAR and then -drivers for the rest of it? Needed? Probably not. At the moment it's not possible to build a normal kernel on one arch and the split on another. If we're going to go off and make changes to anaconda and yum and dnf to cope with this, consistency on what is shipped is probably a good thing. That being said, it is flexible in terms of the content of those packages. So ppc64 could do what you suggest. s390x would arguably just shove almost everything in -drivers. In reality, I expect most arches to just install both packages anyway. If you update the ppc64 kernel package, please also do the same for the ppc 32-bit kernel. I did. I have to adapt for all architectures we build for, and ppc is one of those. You can find it in the scratch build I pointed to earlier in the thread. josh ___ kernel mailing list kernel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/kernel
Re: [WIP] Create kernel-core and kernel-drivers subpackages
On Mon, Mar 17, 2014 at 03:13:54PM -0400, Josh Boyer wrote:
TODO:
Right now this is what has been building in my COPR for the past week.
It works well on x86_64 (which is all I've tested), and i686 builds but
I know it's broken on ARM. To finish it up, I need:
1) Review from you all.
I took a stab at this.
Let's start with the dumb question first because I couldn't find the
obvious answer, why kernel-driver? and not just kernel and
kernel-modules-extras? IOW what does kernel-core buy us except for spec
churn. I am sure yum is invovled in that answer.
Second, the whole mv .. restore/ and back again dance made me cry. Can we
just hack the scripts/Makefile.modinst file to pass a filter argument such
that the installs look like
#kernel core
make modules_install INSTALL_MOD_DIR=core INSTALL_MOD_FILTER=modules.list
#kernel module extras
make modules_install INSTALL_MOD_DIR=suppl INSTALL_MOD_FILTER_INV=modules.list
then you have two directories core/ and suppl/ that you can run depmod on
individually and for filelist purposes. Then you can combine it later for
packaging purposes I think. Something like that. Heck you might be able
to run a depmod check on the filtered files and fail if a dependency isn't
copied over during install (or just do it automagically to keep
modules.list smaller).
Just a thought.
Most of this is just moving and renaming so that isn't too bad. I didn't
focus on the small subtle changes on the %post stuff.
I keep telling myself I would love to rework the spec file to simplify
again by moving some of the junk into rpm properly. The number of people
who can fully understand the spec file is becoming a dying breed. :-(
So all this churn doesn't make it any better. Hence why I was looking for
kernel Makefile solutions or other rpm solutions to simplify the quirks.
2) A per-arch filter list, because the existing one that works on
x86_64 leaves modules in kernel-core on ARM that lack their
dependencies. Bad.
Much like the config files?
3) Further testing on all arches.
4) To make the depmod call fatal to the build. Without making this
fatal we run the risk of pushing kernel-core packages that aren't
self-contained and that seems pretty bad to me. It shouldn't be a huge
issue as I would think most of the breakage is going to come from merge
window stuff and settle down after that.
I was trying to envision a 'make depcheck' or something similar to a 'make
configs' that fails early as opposed to an hour after the build completes.
But everytime I dive to deep into the Makefile I get lost, try a quick
hack and make things worse (I still dream of a 'make kabi' like command
for rhel reasons...).
Cheers,
Don
josh
diff --git a/kernel.spec b/kernel.spec
index e055bb2..4ffa1b6 100644
--- a/kernel.spec
+++ b/kernel.spec
@@ -34,7 +34,7 @@ Summary: The Linux kernel
# For non-released -rc kernels, this will be appended after the rcX and
# gitX tags, so a 3 here would become part of release 0.rcX.gitX.3
#
-%global baserelease 1
+%global baserelease 8
%global fedora_build %{baserelease}
# base_sublevel is the kernel version we're starting with and patching
@@ -385,29 +385,6 @@ Summary: The Linux kernel
%define kernel_prereq fileutils, systemd = 203-2
%define initrd_prereq dracut = 027
-#
-# This macro does requires, provides, conflicts, obsoletes for a kernel
package.
-#%%kernel_reqprovconf subpackage
-# It uses any kernel_subpackage_conflicts and kernel_subpackage_obsoletes
-# macros defined above.
-#
-%define kernel_reqprovconf \
-Provides: kernel = %{rpmversion}-%{pkg_release}\
-Provides: kernel-%{_target_cpu} = %{rpmversion}-%{pkg_release}%{?1:+%{1}}\
-Provides: kernel-drm-nouveau = 16\
-Provides: kernel-uname-r = %{KVERREL}%{?1:+%{1}}\
-Requires(pre): %{kernel_prereq}\
-Requires(pre): %{initrd_prereq}\
-Requires(pre): linux-firmware = 20130724-29.git31f6b30\
-Requires(preun): systemd = 200\
-%{expand:%%{?kernel%{?1:_%{1}}_conflicts:Conflicts:
%%{kernel%{?1:_%{1}}_conflicts}}}\
-%{expand:%%{?kernel%{?1:_%{1}}_obsoletes:Obsoletes:
%%{kernel%{?1:_%{1}}_obsoletes}}}\
-%{expand:%%{?kernel%{?1:_%{1}}_provides:Provides:
%%{kernel%{?1:_%{1}}_provides}}}\
-# We can't let RPM do the dependencies automatic because it'll then pick up\
-# a correct but undesirable perl dependency from the module headers which\
-# isn't required for the kernel proper to function\
-AutoReqProv: no\
-%{nil}
Name: kernel%{?variant}
Group: System Environment/Kernel
@@ -419,8 +396,9 @@ Release: %{pkg_release}
# SET %%nobuildarches (ABOVE) INSTEAD
ExclusiveArch: noarch %{all_x86} x86_64 ppc ppc64 ppc64p7 s390 s390x %{arm}
aarch64 ppc64le
ExclusiveOS: Linux
+Requires: kernel-%{?variant:%{variant}-}core-uname-r = %{KVERREL}%{?variant}
+Requires: kernel-%{?variant:%{variant}-}drivers-uname-r =
%{KVERREL}%{?variant}
-%kernel_reqprovconf
#
# List the packages used during the kernel build
@@
Re: [WIP] Create kernel-core and kernel-drivers subpackages
On Thu, Mar 20, 2014 at 02:51:13PM -0400, Josh Boyer wrote: On Thu, Mar 20, 2014 at 02:13:20PM -0400, Don Zickus wrote: On Mon, Mar 17, 2014 at 03:13:54PM -0400, Josh Boyer wrote: TODO: Right now this is what has been building in my COPR for the past week. It works well on x86_64 (which is all I've tested), and i686 builds but I know it's broken on ARM. To finish it up, I need: 1) Review from you all. I took a stab at this. Thanks! Let's start with the dumb question first because I couldn't find the obvious answer, why kernel-driver? and not just kernel and kernel-modules-extras? IOW what does kernel-core buy us except for spec churn. I am sure yum is invovled in that answer. kernel - metapackage the requires the equivalent of what kernel is today. kernel-core - the tiny thing Cloud wants. vmlinux + a small set of drivers kernel-drivers - the other half of what kernel is today kernel-modules-extra - modules that really aren't needed anywhere by default and I wish we could just turn off entirely. So kernel-modules-extra already exists today and repurposing it to be the equivalent of kernel-drivers could happen I guess. However, just having kernel be the tiny thing is probably broken from a conceptual standpoint because just installing kernel isn't enough to fully boot on most bare-metal machines. If we made it so, then it would probably be large enough to just not bother with this at all. Hence kernel-core. Right, I guess I don't understand how kernel-drivers magically installs/not installs. I had assumed some of this was hardcoded in some install file anaconda reads. So my initial assumption was kernel and kernel-drivers would auto install on a normal desktop. I didn't think far enough to understand how an update would work.. So in this case, I suppose cloud only installs kernel-core whereas desktop installs kernel (which installs both -core and -driver)? Second, the whole mv .. restore/ and back again dance made me cry. Can we just hack the scripts/Makefile.modinst file to pass a filter argument such that the installs look like #kernel core make modules_install INSTALL_MOD_DIR=core INSTALL_MOD_FILTER=modules.list #kernel module extras make modules_install INSTALL_MOD_DIR=suppl INSTALL_MOD_FILTER_INV=modules.list then you have two directories core/ and suppl/ that you can run depmod on individually and for filelist purposes. Then you can combine it later for packaging purposes I think. Something like that. Heck you might be able to run a depmod check on the filtered files and fail if a dependency isn't copied over during install (or just do it automagically to keep modules.list smaller). Er... no? I mean, we could install to core and suppl directories if we had modules.list. But if we had modules.list already then we wouldn't need to generate the %files list, which is what this hunk is doing. Essentially this is avoiding hardcoded, one line per .ko lists by going through and removing entire directories of things and saving off the list as the contents of kernel-core and kernel-drivers. Then we fix it up since we actually want kernel-drivers to install to /lib/modules/`uname -r`/kernel/ also. It's probably clearer if you have filter-modules.sh to look at. I've attached it below. Apologies for not including it to begin with. Ah, yes, now things are clearer. I had assumed you were listing everything one by one (sorta akin to what is done with the config options). I could argue you would have better control if you mimic'd the config options and created a modules.list. But I am not maintaining this, so I won't. :-) Most of this is just moving and renaming so that isn't too bad. I didn't focus on the small subtle changes on the %post stuff. Yeah, that's fairly trivial. I keep telling myself I would love to rework the spec file to simplify again by moving some of the junk into rpm properly. The number of people who can fully understand the spec file is becoming a dying breed. :-( So all this churn doesn't make it any better. Hence why I was looking for kernel Makefile solutions or other rpm solutions to simplify the quirks. Except the complexities don't come from the kernel makefiles. They come from RPM :\. It needs a file with a list of files to include for the %files -f directive of both kernel-core and kernel-drivers to work. If we don't use %files -f, we're stuck listing out individual directories and .ko files in the %files section for each. That would be a nightmare to maintain, so I've automated it with %files -f. I wasn't referring to the %files stuff. More like all the shell scripting prep work for restore/ and friends. Just trying to think of ways to simplify things. Some of this could be avoided (not a whole lot) if we invented /lib/modules/`uname -r`/suppl and installed the kernel-drivers content there, but
Re: [WIP] Create kernel-core and kernel-drivers subpackages
On Thu, Mar 20, 2014 at 03:47:19PM -0400, Josh Boyer wrote: I believe anaconda does install kernel today. So since the kernel metapackage Requires:kernel-drivers, it will bring that in automatically. Updates from existing installs would pick up kernel, kernel-core, and kernel-drivers based on the Requires: present. It looks like 3 packages instead of one, but the overall installed size is literally the same as the existing kernel package today. It's a split to keep things working, while allowing new approaches at the same time. So in this case, I suppose cloud only installs kernel-core whereas desktop installs kernel (which installs both -core and -driver)? Yes, exactly. That would be done via kickstart (and is extendable to anyone that wants to avoid kernel-drivers but Cloud is the main user.) Thanks! Second, the whole mv .. restore/ and back again dance made me cry. Can we just hack the scripts/Makefile.modinst file to pass a filter argument such that the installs look like #kernel core make modules_install INSTALL_MOD_DIR=core INSTALL_MOD_FILTER=modules.list #kernel module extras make modules_install INSTALL_MOD_DIR=suppl INSTALL_MOD_FILTER_INV=modules.list then you have two directories core/ and suppl/ that you can run depmod on individually and for filelist purposes. Then you can combine it later for packaging purposes I think. Something like that. Heck you might be able to run a depmod check on the filtered files and fail if a dependency isn't copied over during install (or just do it automagically to keep modules.list smaller). Er... no? I mean, we could install to core and suppl directories if we had modules.list. But if we had modules.list already then we wouldn't need to generate the %files list, which is what this hunk is doing. Essentially this is avoiding hardcoded, one line per .ko lists by going through and removing entire directories of things and saving off the list as the contents of kernel-core and kernel-drivers. Then we fix it up since we actually want kernel-drivers to install to /lib/modules/`uname -r`/kernel/ also. It's probably clearer if you have filter-modules.sh to look at. I've attached it below. Apologies for not including it to begin with. Ah, yes, now things are clearer. I had assumed you were listing everything one by one (sorta akin to what is done with the config options). I could argue you would have better control if you mimic'd the config options and created a modules.list. But I am not maintaining this, so I won't. :-) I might be able to generate a list of modules (.ko files) based on the config settings. That would basically be the equivalent of the first find we do in the spec file change. Not sure it would simplify anything else, but I'll think about it a bit and see if there's further optimizations to be done. It was just a different point of view. Not saying it is any better. Just thought it might simplify the spec file changes and control the kernel-core bloat over a number of releases. The thought was if you the average desktop user was install kenrel-core and kernel-driver, then it doesn't matter where the .kos are because in the end, all the .kos end up on the machine. So the kernel-core module.list would really be for cloud. And someone could keep trimming that list until the cloud folks had something super small that made them excited. If the list was too small it didn't impact the desktop folks (not that your approach causes any negative impact either). Again it was just solving a problem from a different perspective. If you are happy with your approach continue with it. I'll have to see how it works in RHEL in a couple of years. :-) Most of this is just moving and renaming so that isn't too bad. I didn't focus on the small subtle changes on the %post stuff. Yeah, that's fairly trivial. I keep telling myself I would love to rework the spec file to simplify again by moving some of the junk into rpm properly. The number of people who can fully understand the spec file is becoming a dying breed. :-( So all this churn doesn't make it any better. Hence why I was looking for kernel Makefile solutions or other rpm solutions to simplify the quirks. Except the complexities don't come from the kernel makefiles. They come from RPM :\. It needs a file with a list of files to include for the %files -f directive of both kernel-core and kernel-drivers to work. If we don't use %files -f, we're stuck listing out individual directories and .ko files in the %files section for each. That would be a nightmare to maintain, so I've automated it with %files -f. I wasn't referring to the %files stuff. More like all the shell scripting prep work for restore/ and
Re: [WIP] Create kernel-core and kernel-drivers subpackages
On Mon, Mar 17, 2014 at 03:13:54PM -0400, Josh Boyer wrote: TODO: Right now this is what has been building in my COPR for the past week. It works well on x86_64 (which is all I've tested), and i686 builds but I know it's broken on ARM. To finish it up, I need: 1) Review from you all. Ahem. 2) A per-arch filter list, because the existing one that works on x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad. OK, I sorted this out this week. I believe the only arch left to do is s390x and that's only because I forgot about it. Oops. 3) Further testing on all arches. http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=1711902 http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=2255745 http://koji.fedoraproject.org/koji/taskinfo?taskID=6646945 The last link for i686 wasn't really needed as both i686 and x86_64 are covered by the COPR builds. I did it anyway because it was easy to kick off. 4) To make the depmod call fatal to the build. Without making this fatal we run the risk of pushing kernel-core packages that aren't self-contained and that seems pretty bad to me. It shouldn't be a huge issue as I would think most of the breakage is going to come from merge window stuff and settle down after that. After I get s390x sorted I'll probably start making depmod fail the build locally. josh ___ kernel mailing list kernel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/kernel
Re: [WIP] Create kernel-core and kernel-drivers subpackages
Josh Boyer (jwbo...@fedoraproject.org) said: 2) A per-arch filter list, because the existing one that works on x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad. OK, I sorted this out this week. I believe the only arch left to do is s390x and that's only because I forgot about it. Oops. Is this even needed on s390 for reasons other than consistency? Similarly with power, is the idea to have a core kernel for running on an LPAR and then -drivers for the rest of it? Bill ___ kernel mailing list kernel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/kernel
