Re: [OE-core] [PATCH v3] kernel-module-split: Append KERNEL_VERSION string to kernel module name

[Ola Redell]

On 01/12/2017 02:44 PM, Bruce Ashfield wrote:



Interesting. I think we've probably fallen off the radar of the casual 
reader that may have
more bitbake knowledge to explain why the virtual package 
(kernel-modules) provides
works with the same name and different versions, while a package with 
the version postfix

that rprovides the same name doesn't work with different versions.

I think I have a clue on this so let me try to straight it out as much 
as I can in a somewhat more structured manner than my earlier attempt, 
and hopefully with a readable formatting this time... I'll do a 
background and overview too to let new readers in. I am sorry about this 
being lengthy. I'll post a patch v4 based on the end conclusion.


Anyone with a better knowledge in package management than me (that means 
very many of you) might want to fill in or correct the below. And Bruce, 
I kept #1 and #3 separate since I see quite a difference between "meta 
packages" and "virtual packages".


In my current system I use deb packages and apt-get as package manager 
so the discussion below is apt-get centric. The *assumption* here is 
that package managers in general behave in a similar way to apt-get, 
when it comes to the details below.



Background

My use case is that I want a kernel module packaging that allows me to 
upgrade my kernel and kernel modules using a package manager such as 
apt-get without the kernel modules for the older kernel being removed. 
When I perform an upgrade today (either dist-upgrade or upgrade on the 
package "kernel-modules") the old modules are removed. My motivation is 
that I want to be able to fall back to the older kernel and its modules 
if necessary.


The "kernel-modules" package is called a "meta package" in the code. 
Even though it is an empty package (therefore "meta"), it is a package 
in its own right with a package version based on the kernel version used 
to build it. It has dependencies to all kernel module packages (e.g. 
kernel-module-foo) that are built by bitbake. Hence when the 
"kernel-modules" package is installed or upgraded it pulls in all those 
modules that it depends on.


Side note: If apt-get finds many "kernel-modules" packages available it 
simply picks the one with the latest version (unless given a specific 
version). So having many "meta packages" of different versions available 
at the same time is not a problem - apt-get can distinguish between them.


Today, the kernel module packages like kernel-module-foo are named the 
same regardless of what kernel they belong to. Hence kernel-module-foo 
for kernel-x is the same package as kernel-module-foo for kernel-y, just 
with another version. Therefore, when the "kernel-modules" package is 
upgraded, the new version of kernel-module-foo is installed while the 
older version is removed (you cannot have two versions of the same 
package installed at once). That is the problem.



Alternative solutions

My original suggestion (#0) was to just postfix the kernel module 
package names with a kernel version string (creating 
kernel-module-foo-) just like what is done with the kernel 
itself already today. Note that this  has very little to do 
with the actual version of the package.  is just 
KERNEL_VERSION, nothing else. This solves my use case but causes other 
problems as Bruce pointed out. Existing recipes with dependencies to 
kernel-module-foo will break, and even worse, will have to be updated 
for every new kernel version.


One alternative (#1) would be to add the postfix and also add empty 
"meta packages" for all kernel-module packages, such that the meta 
package "kernel-module-foo" would depend on 
"kernel-module-foo-". That would work great (presumably - 
untested), just like the "kernel-modules" meta package, but would add a 
lot of new packages to the already time consuming build. It would also 
make the kernel-module-split class even more complex.


Another alternative (#2) is to simply make the postfix configurable, on 
or off. This would work fine but it would also add to the complexity of 
the kernel-module-split code as well as adding another (rarely used) 
code path, with the effects that has on stability.


A third alternative (#3) is to add the postfix and let the kernel module 
packages RPROVIDE the shorter named package. Hence 
kernel-module-foo- would then provide kernel-module-foo such 
that recipes can still depend on the kernel-module-foo which is now a 
"virtual package". A "virtual package" is more like an alias for the 
real package and is not a package in its own right (unlike the meta 
package). With this solution one would also be able to install 
kernel-module-foo using "apt-get install kernel-module-foo" but only if 
there is only one provider available. Hence, it would not be possible to 
"apt-get upgrade kernel-module-foo" to get 
kernel-module-foo- since the original package 
(kernel-module-foo-) is always available in an OE system(?) 
(afaik). This would not be a 

Re: [OE-core] [PATCH v3] kernel-module-split: Append KERNEL_VERSION string to kernel module name

[Bruce Ashfield]

On Wed, Jan 11, 2017 at 9:43 AM, Ola Redell  wrote:

>
>
> 2017-01-10 15:45 GMT+01:00 Bruce Ashfield :
>
>>
>>
>>>
>>> So basically, as I understand it, this may be a blocker for this change.
>>> Alternatives I can come up with are:
>>>
>>> 1) For all packaged kernel modules, add meta packages that depend on the
>>> correct version of the kernel module itself. Then "kernel-module-foo" would
>>> depend on "kernel-module-foo-". The package "kernel-modules" would
>>> then still be made depending on "kernel-module-foo-" and the likes.
>>>
>>
>>
>> Wouldn't this break your primary use case ? i.e. if you had two kernels
>> in the image or
>> when you install the 2nd newer kernel, the meta packages would conflict
>> and hence couldn't
>> be around at the same time ?
>>
>>
> No, it would not break as far as my tests show. I use the meta package
> "kernel-modules" today and since it is versioned (not the version string we
> are talking about adding here - but regular package version) it all works
> fine. apt-get knows about the versions and installs the latest one, with
> dependencies, if I don't say anything else. I don't have any problems
> having many kernel-modules packages available for installation in my repos
> simultaneously. The fine thing with this is that I can perform an "apt-get
> dist-upgrade" (or if I'd prefer apt-get upgrade kernel-modules) to install
> all new versions of the packages. When I have done that, apt-get finds out
> that the older packages are no longer needed and allows me to remove them
> (when I want to remove them - i.e. when I know my new kernel works fine)
> using "apt-get autoremove". (A note here: After my first kernel upgrade
> this is not the case because apt-get considers the original kernel and all
> original kernel-modules to not having been installed "automatically", that
> is not as dependencies to other packages. But after my second dist-upgrade
> the autoremove option allows me to remove unused packages)
>


Aha. Good to know. I wasn't sure, not having tried this myself.


>
>
>> Rather than a meta-package, I think it would be feasible to just have the
>> generated split
>> modules RPROVIDE both a PN- and PN. i.e. kernel-module-foo and
>> kernel-module-foo-
>> would be RPROVIDED by the packages. I'd still consider this the same
>> solution as #1, just
>> with a different implementation, but one that wouldn't generate more
>> packages and slow
>> the install/generation phase even more.
>>
>
>
>>
>> I'm not sure what happens if two packages can have a different on-disk
>> name (i.e. kernel-module-foo-1.2
>> and kernel-module-foo-1.3) and both RPROVIDE "kernel-module-foo". Only
>> one would be installed,
>> but if someone did an apt-get or a smart install of kernel-module-foo ..
>> chaos may erupt.
>>
>>
> That is an interesting alternative and I looked in to that a bit by doing
> the necessary hack and testing it out. Your earlier concerns are solved
> with this - you can do RDEPENDS_x = "kernel-module-foo" and I can install
> that virtual package using "apt-get install kernel-module-foo" as long as I
> only have one package providing that virtual package available.  But as
> soon as I have more than one package providing that virtual package,
> apt-get refuses to install it without me stating exactly what package I
> want. Hence I then need to point out kernel-module-foo-. The
> reason for this seems to be that virtual packages are not versioned (again
> package version - not the postfix string we discuss here). This is not a
> huge problem for me with my current use case, but if I'd only install a
> couple of kernel modules with bitbake and then later would want to upgrade
> them using e.g. "apt-get dist-upgrade" or "apt-get upgrade", that would be
> a problem.
>
>

Interesting. I think we've probably fallen off the radar of the casual
reader that may have
more bitbake knowledge to explain why the virtual package (kernel-modules)
provides
works with the same name and different versions, while a package with the
version postfix
that rprovides the same name doesn't work with different versions.



> I agree adding a huge number of more meta packages should be avoided if
> possible.
>

Agreed again. If possible, taking a first step that doesn't break the
existing use cases,
but only adds a minor extra step in your transition case (individual module
installs) is
probably the least controversial route to take.

It would also be something that could evolve once in the tree.


>
>
>>
>>
>>>
>>> 2) Make the version postfix configurable with a
>>> KERNEL_MODULE_PACKAGE_POSTFIX, exactly like the prefix (which was added
>>> recently). The default would be an empty postfix, but it could be used to
>>> achieve this functionality for those who need it.
>>>
>>> No. 2 is a very simple change. No. 1 is more uncertain (to me at least).
>>>
>>
>> I was discussing this change with a few people in person this morning and
>> we came up with 

Re: [OE-core] [PATCH v3] kernel-module-split: Append KERNEL_VERSION string to kernel module name

[Ola Redell]

2017-01-10 15:45 GMT+01:00 Bruce Ashfield :

>
>
>>
>> So basically, as I understand it, this may be a blocker for this change.
>> Alternatives I can come up with are:
>>
>> 1) For all packaged kernel modules, add meta packages that depend on the
>> correct version of the kernel module itself. Then "kernel-module-foo" would
>> depend on "kernel-module-foo-". The package "kernel-modules" would
>> then still be made depending on "kernel-module-foo-" and the likes.
>>
>
>
> Wouldn't this break your primary use case ? i.e. if you had two kernels in
> the image or
> when you install the 2nd newer kernel, the meta packages would conflict
> and hence couldn't
> be around at the same time ?
>
>
No, it would not break as far as my tests show. I use the meta package
"kernel-modules" today and since it is versioned (not the version string we
are talking about adding here - but regular package version) it all works
fine. apt-get knows about the versions and installs the latest one, with
dependencies, if I don't say anything else. I don't have any problems
having many kernel-modules packages available for installation in my repos
simultaneously. The fine thing with this is that I can perform an "apt-get
dist-upgrade" (or if I'd prefer apt-get upgrade kernel-modules) to install
all new versions of the packages. When I have done that, apt-get finds out
that the older packages are no longer needed and allows me to remove them
(when I want to remove them - i.e. when I know my new kernel works fine)
using "apt-get autoremove". (A note here: After my first kernel upgrade
this is not the case because apt-get considers the original kernel and all
original kernel-modules to not having been installed "automatically", that
is not as dependencies to other packages. But after my second dist-upgrade
the autoremove option allows me to remove unused packages)


> Rather than a meta-package, I think it would be feasible to just have the
> generated split
> modules RPROVIDE both a PN- and PN. i.e. kernel-module-foo and
> kernel-module-foo-
> would be RPROVIDED by the packages. I'd still consider this the same
> solution as #1, just
> with a different implementation, but one that wouldn't generate more
> packages and slow
> the install/generation phase even more.
>


>
> I'm not sure what happens if two packages can have a different on-disk
> name (i.e. kernel-module-foo-1.2
> and kernel-module-foo-1.3) and both RPROVIDE "kernel-module-foo". Only one
> would be installed,
> but if someone did an apt-get or a smart install of kernel-module-foo ..
> chaos may erupt.
>
>
That is an interesting alternative and I looked in to that a bit by doing
the necessary hack and testing it out. Your earlier concerns are solved
with this - you can do RDEPENDS_x = "kernel-module-foo" and I can install
that virtual package using "apt-get install kernel-module-foo" as long as I
only have one package providing that virtual package available.  But as
soon as I have more than one package providing that virtual package,
apt-get refuses to install it without me stating exactly what package I
want. Hence I then need to point out kernel-module-foo-. The
reason for this seems to be that virtual packages are not versioned (again
package version - not the postfix string we discuss here). This is not a
huge problem for me with my current use case, but if I'd only install a
couple of kernel modules with bitbake and then later would want to upgrade
them using e.g. "apt-get dist-upgrade" or "apt-get upgrade", that would be
a problem.

I agree adding a huge number of more meta packages should be avoided if
possible.


>
>
>>
>> 2) Make the version postfix configurable with a
>> KERNEL_MODULE_PACKAGE_POSTFIX, exactly like the prefix (which was added
>> recently). The default would be an empty postfix, but it could be used to
>> achieve this functionality for those who need it.
>>
>> No. 2 is a very simple change. No. 1 is more uncertain (to me at least).
>>
>
> I was discussing this change with a few people in person this morning and
> we came up with making this
> optional as well (your #2), which would ensure that existing use cases are
> not broken, but on the
> flip side it makes already complex code more complex and would leave some
> paths more tested
> than others.
>
> If #1 is feasible, I wonder if a variant of both 1 an 2 are viable ? i.e.
> make the generation of the
> packages always have a version in the name, but make the generation of a
> "versionless" meta
> package be optional (or a versionless RPROVIDE). In the default case
> they'd be enabled, and no
> one would notice that anything had changed but the module-split code would
> have a single path.
> In a case where multiple kernels are to be installed, you'd have to
> disable the versionless package
> generation.
>

Given my tests described above, this may be the best solution, yes.


>
> I mentioned wildcards above, I'm not sure if anyone else reading this
> thread knows if 

Re: [OE-core] [PATCH v3] kernel-module-split: Append KERNEL_VERSION string to kernel module name

[Bruce Ashfield]

On Tue, Jan 10, 2017 at 3:44 AM, Ola Redell  wrote:

>
>
> On 01/09/2017 06:20 PM, Bruce Ashfield wrote:
>
>
> The functionality you describe is a good thing, but I have a few questions
> to make
> sure I understand the change, and confirm that existing functionality
> isn't broken ..
> and that the internals of the package management do the right thing (I'm
> not an
> expert on those internals .. so bear with me).
>
> I am not an expert either so I thought I'd suggest this since it works for
> me and see how it is received.
>


A good plan. It is always better to spur discussion with a tangible patch,
which it has done!


>
> With this change, do you know if RDEPENDS="kernel-module-foo" still works
> ? or
> does it now have to be RDEPENDS="kernel-module-foo-" ?
>
> No, the old writing would not work. You would need to add the version as
> in your second example. That will break things, true.
>


ok. That was what I thought as well. As you noted, that is a problem since
it would now
mean that any recipe that wants to rrecommend or rdepend on a package no
longer has
kernel version flexibility (unless wildcards are in play) and not the
virtual/kernel type
binding that it may have had before.


> On a similar thought, "smart install kernel-module-foo" would also work ?
> or does it
> now require a version ?
>
> Also in this example you would need to add the version, even though I have
> only used apt-get and not smart.
>
> The meta package "kernel-modules" will work as before, without a version
> postfix.
>

As it turns out, I normally use the meta package myself, but I do know of
others that
don't use it, but do reference individual module packages.


>
> So basically, as I understand it, this may be a blocker for this change.
> Alternatives I can come up with are:
>
> 1) For all packaged kernel modules, add meta packages that depend on the
> correct version of the kernel module itself. Then "kernel-module-foo" would
> depend on "kernel-module-foo-". The package "kernel-modules" would
> then still be made depending on "kernel-module-foo-" and the likes.
>

Wouldn't this break your primary use case ? i.e. if you had two kernels in
the image or
when you install the 2nd newer kernel, the meta packages would conflict and
hence couldn't
be around at the same time ?

Rather than a meta-package, I think it would be feasible to just have the
generated split
modules RPROVIDE both a PN- and PN. i.e. kernel-module-foo and
kernel-module-foo-
would be RPROVIDED by the packages. I'd still consider this the same
solution as #1, just
with a different implementation, but one that wouldn't generate more
packages and slow
the install/generation phase even more.

I'm not sure what happens if two packages can have a different on-disk name
(i.e. kernel-module-foo-1.2
and kernel-module-foo-1.3) and both RPROVIDE "kernel-module-foo". Only one
would be installed,
but if someone did an apt-get or a smart install of kernel-module-foo ..
chaos may erupt.



>
> 2) Make the version postfix configurable with a
> KERNEL_MODULE_PACKAGE_POSTFIX, exactly like the prefix (which was added
> recently). The default would be an empty postfix, but it could be used to
> achieve this functionality for those who need it.
>
> No. 2 is a very simple change. No. 1 is more uncertain (to me at least).
>

I was discussing this change with a few people in person this morning and
we came up with making this
optional as well (your #2), which would ensure that existing use cases are
not broken, but on the
flip side it makes already complex code more complex and would leave some
paths more tested
than others.

If #1 is feasible, I wonder if a variant of both 1 an 2 are viable ? i.e.
make the generation of the
packages always have a version in the name, but make the generation of a
"versionless" meta
package be optional (or a versionless RPROVIDE). In the default case they'd
be enabled, and no
one would notice that anything had changed but the module-split code would
have a single path.
In a case where multiple kernels are to be installed, you'd have to disable
the versionless package
generation.

I mentioned wildcards above, I'm not sure if anyone else reading this
thread knows if RDEPENDS
can use wildcards ? i.e. if RDEPENDS="kernel-module-foo-4%"  or
"kernel-module-foo-%" worked,
then I could see the version string requirement being less of an issue and
not something that would
require constant bumping as the kernel version changed.

Bruce


>
> /Ola
>
>


-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end"
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v3] kernel-module-split: Append KERNEL_VERSION string to kernel module name

[Ola Redell]

On 01/09/2017 06:20 PM, Bruce Ashfield wrote:


The functionality you describe is a good thing, but I have a few 
questions to make
sure I understand the change, and confirm that existing functionality 
isn't broken ..
and that the internals of the package management do the right thing 
(I'm not an

expert on those internals .. so bear with me).
I am not an expert either so I thought I'd suggest this since it works 
for me and see how it is received.


With this change, do you know if RDEPENDS="kernel-module-foo" still 
works ? or

does it now have to be RDEPENDS="kernel-module-foo-" ?

No, the old writing would not work. You would need to add the version as 
in your second example. That will break things, true.
On a similar thought, "smart install kernel-module-foo" would also 
work ? or does it

now require a version ?

Also in this example you would need to add the version, even though I 
have only used apt-get and not smart.


The meta package "kernel-modules" will work as before, without a version 
postfix.


So basically, as I understand it, this may be a blocker for this change. 
Alternatives I can come up with are:


1) For all packaged kernel modules, add meta packages that depend on the 
correct version of the kernel module itself. Then "kernel-module-foo" 
would depend on "kernel-module-foo-". The package 
"kernel-modules" would then still be made depending on 
"kernel-module-foo-" and the likes.


2) Make the version postfix configurable with a 
KERNEL_MODULE_PACKAGE_POSTFIX, exactly like the prefix (which was added 
recently). The default would be an empty postfix, but it could be used 
to achieve this functionality for those who need it.


No. 2 is a very simple change. No. 1 is more uncertain (to me at least).

/Ola

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v3] kernel-module-split: Append KERNEL_VERSION string to kernel module name

[Bruce Ashfield]

On Mon, Jan 9, 2017 at 4:54 AM,  wrote:

> From: Ola Redell 
>
> The KERNEL_VERSION string is added to kernel module package names in order
> to
> make the kernel modules for different kernel versions distinct packages
> instead
> of different versions of the same package. With this change, when a new
> kernel
> is installed together with its kernel modules (e.g. by upgrade of the
> packages
> kernel and kernel-modules) using some package manager such as apt-get or
> rpm,
> the kernel modules for the older kernel will not be removed. This enables a
> fall back to the older kernel if the new one fails.
>


The functionality you describe is a good thing, but I have a few questions
to make
sure I understand the change, and confirm that existing functionality isn't
broken ..
and that the internals of the package management do the right thing (I'm
not an
expert on those internals .. so bear with me).

With this change, do you know if RDEPENDS="kernel-module-foo" still works ?
or
does it now have to be RDEPENDS="kernel-module-foo-" ?

On a similar thought, "smart install kernel-module-foo" would also work ?
or does it
now require a version ?

Bruce


>
> Signed-off-by: Ola Redell 
> ---
>  meta/classes/kernel-module-split.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes/kernel-module-split.bbclass
> b/meta/classes/kernel-module-split.bbclass
> index efe1b42..13cff51 100644
> --- a/meta/classes/kernel-module-split.bbclass
> +++ b/meta/classes/kernel-module-split.bbclass
> @@ -122,7 +122,7 @@ python split_kernel_module_packages () {
>  module_regex = '^(.*)\.k?o$'
>
>  module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
> -module_pattern = module_pattern_prefix + 'kernel-module-%s'
> +module_pattern = module_pattern_prefix + 'kernel-module-%s-' +
> d.getVar("KERNEL_VERSION", True)
>
>  postinst = d.getVar('pkg_postinst_modules')
>  postrm = d.getVar('pkg_postrm_modules')
> --
> 1.9.1
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end"
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3] kernel-module-split: Append KERNEL_VERSION string to kernel module name

[ola . redell]

From: Ola Redell 

The KERNEL_VERSION string is added to kernel module package names in order to
make the kernel modules for different kernel versions distinct packages instead
of different versions of the same package. With this change, when a new kernel
is installed together with its kernel modules (e.g. by upgrade of the packages
kernel and kernel-modules) using some package manager such as apt-get or rpm,
the kernel modules for the older kernel will not be removed. This enables a
fall back to the older kernel if the new one fails.

Signed-off-by: Ola Redell 
---
 meta/classes/kernel-module-split.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/kernel-module-split.bbclass 
b/meta/classes/kernel-module-split.bbclass
index efe1b42..13cff51 100644
--- a/meta/classes/kernel-module-split.bbclass
+++ b/meta/classes/kernel-module-split.bbclass
@@ -122,7 +122,7 @@ python split_kernel_module_packages () {
 module_regex = '^(.*)\.k?o$'
 
 module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
-module_pattern = module_pattern_prefix + 'kernel-module-%s'
+module_pattern = module_pattern_prefix + 'kernel-module-%s-' + 
d.getVar("KERNEL_VERSION", True)
 
 postinst = d.getVar('pkg_postinst_modules')
 postrm = d.getVar('pkg_postrm_modules')
-- 
1.9.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core