Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf
Dne 9.8.2018 v 07:34 Neal Gompa napsal(a): > On Wed, Aug 8, 2018 at 7:09 PM Pascal Terjan wrote: >> On 7 August 2018 at 09:50, Michael Schroeder wrote: >>> On Mon, Aug 06, 2018 at 04:36:07PM +, Zbigniew J??drzejewski-Szmek >>> wrote: this mail is a continuation of an FPC [1] and a FESCo [2] tickets. A proposal was made is to disallow packages in Fedora from using file deps, and to optimize dnf to not load filelists.xml. File deps would still be supported, because external packages and users want to use them, but they would not be allowed for distro packages. Not downloading or loading filelists.xml which are required for file deps would provide significant bandwidth savings (~47 MB compressed) and noticeable runtime savings (~10s at dnf startup) in many common cases. So this is something that is worth exploring, but it's not clear if it is at all feasible. >>> There's also something that can easily be done and would make >>> loading the filelist unneeded in most of the cases: extend the >>> primary filelist to include some whitelist of files. The whitelist >>> must also be stored in the primary data, so that the solver knows >>> what to expect. >> That's what Mandrake/Mandriva/Mageia/... has been doing for many >> years, there is a small file-deps file containing the ones we end up >> generating, mostly from scriptlets IIRC, and we end up with provides >> added for those in the main metadata when generating it. Then file >> lists are lazily loaded when people want to query them but not used >> for dependency resolution. >> >> $ GET >> http://ftp.free.fr/mirrors/mageia.org/distrib/cauldron/x86_64/media/media_info/file-deps >> /bin/csh >> /bin/grep >> /bin/perl >> /usr/bin/ln >> /usr/bin/rm >> /sbin/service >> /usr/bin/chattr >> /usr/bin/guile >> /usr/bin/openssl >> /usr/bin/pear >> /usr/bin/texhash >> /usr/bin/tr >> /usr/bin/which >> /usr/sbin/groupadd >> /usr/sbin/groupdel >> /usr/sbin/useradd >> /usr/sbin/userdel >> > So the primary.xml already includes all that. If you actually look in > the primary.xml.gz files in the Mageia rpm-md data, those are already > there. The problem is that there are people who actually request files > outside of the base whitelist as a means to be able to request > "things" without knowing how they are packaged, because the file path > is the consistent thing across distros. So couldn't be createrepo actually extended in a way that if it identifies package, which has "Requires: /some/random/path" and at the same time, the "/some/random/path" is actually included in the repository, such file/package would be included in primary.xml.gz? This would help with huge repositories, since there is the highest cost of downloading filelist.xml. V. ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] RFC #417 %optional file attribute
Dne 24.7.2018 v 20:41 Jeff Johnson napsal(a): > >> On Jul 24, 2018, at 7:42 AM, Vít Ondruch wrote: >> >> I would have use for %optional: >> >> https://src.fedoraproject.org/rpms/scl-utils/pull-request/1 >> >> Let me explain. scl-utils 2.x introduced support for environment >> modules. They are now enforcing existence of modulefile. However, the >> modulefile is not available in SCLs prepared for scl-utils 1.x and >> moreover, it is not needed, because scl-utils 2.x should be able to >> execute the old collections in backward compatible way just fine. If >> there was %optional macro used at this line [1], it would allow to >> include the module file if the collection is prepared for scl-utils 2.x, >> while it would allow older collections prepared for scl-utils 1.x to be >> still built. >> >> I am about the merge the PR above, but that means that whoever is going >> to create the modulefile now will have to explicitly list it in files >> section. I don't see any better option ATM. >> > The ability to imagine a usage case for %optional does not establish a sound > basis for adding the feature. > > There are many existing ways in rpm to create a package that handles your > scl-utils packaging problem that do not require editing %files for an > optional file. > > Perhaps the best way is to use a manifest in %files and add/delete the > additional file based on whatever logic you wish (like testing whether a 2.x > or a 1.x build of scl-utils is being attempted). > > Knowing the conditions for including the file is what is important for > determining whether a package has been built correctly and reproducibly. > > Having a %optional marker reveals nothing about the circumstances where the > file SHOULD exist necessary to determine whether a package has been built > correctly. I don't disagree. However, this is a bit about being explicit or implicit. Should user who adds modulefile, which is not supported by scl-utils 1.x and is optional for scl-utils 2.x list the file somewhere explicitly or should it be included implicitly? Frankly, the same question is if I should better add directory including its whole content into file section or should I add dir explicitly and the files contained one by one. As long as RPM supports former, then there is no reason to not support %optional, because RPM generally does not care whether file should exits. V. > > 73 de Jeff > > >> Vít >> >> >> [1] >> https://src.fedoraproject.org/rpms/scl-utils/blob/273a583584bffa3c5121043f24cc98f2e2bb99da/f/macros.scl-filesystem#_7 >> >> >> Dne 26.3.2018 v 12:43 Florian Festi napsal(a): >>> Hi! >>> >>> We are currently pondering about #417 [1]. For adding a %optional file >>> attribute that would allow adding file to to %files sections that may >>> not be built under some circumstances (e.g. some architectures). >>> >>> It is already perfectly legal to have files not listed explicitly if >>> they are within a directory that is included in the %file section. But >>> some packages (examples wanted) may have trouble using this due to the >>> way the package files are laid out. >>> >>> Otoh %optional would be another spec language key word that packagers >>> have to deal with and we as RPM upstream developers have a hard time >>> judging whether the benefit of the attribute really out weights the cost >>> of bloating the spec language. >>> >>> Any input - especially with real world packages that would benefit such >>> an addition - is welcome. >>> >>> Florian >>> >>> >>> [1] https://github.com/rpm-software-management/rpm/pull/417 >> >> ___ >> Rpm-ecosystem mailing list >> Rpm-ecosystem@lists.rpm.org >> http://lists.rpm.org/mailman/listinfo/rpm-ecosystem > ___ > Rpm-ecosystem mailing list > Rpm-ecosystem@lists.rpm.org > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] RFC #417 %optional file attribute
I would have use for %optional: https://src.fedoraproject.org/rpms/scl-utils/pull-request/1 Let me explain. scl-utils 2.x introduced support for environment modules. They are now enforcing existence of modulefile. However, the modulefile is not available in SCLs prepared for scl-utils 1.x and moreover, it is not needed, because scl-utils 2.x should be able to execute the old collections in backward compatible way just fine. If there was %optional macro used at this line [1], it would allow to include the module file if the collection is prepared for scl-utils 2.x, while it would allow older collections prepared for scl-utils 1.x to be still built. I am about the merge the PR above, but that means that whoever is going to create the modulefile now will have to explicitly list it in files section. I don't see any better option ATM. Vít [1] https://src.fedoraproject.org/rpms/scl-utils/blob/273a583584bffa3c5121043f24cc98f2e2bb99da/f/macros.scl-filesystem#_7 Dne 26.3.2018 v 12:43 Florian Festi napsal(a): > Hi! > > We are currently pondering about #417 [1]. For adding a %optional file > attribute that would allow adding file to to %files sections that may > not be built under some circumstances (e.g. some architectures). > > It is already perfectly legal to have files not listed explicitly if > they are within a directory that is included in the %file section. But > some packages (examples wanted) may have trouble using this due to the > way the package files are laid out. > > Otoh %optional would be another spec language key word that packagers > have to deal with and we as RPM upstream developers have a hard time > judging whether the benefit of the attribute really out weights the cost > of bloating the spec language. > > Any input - especially with real world packages that would benefit such > an addition - is welcome. > > Florian > > > [1] https://github.com/rpm-software-management/rpm/pull/417 > ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Is there anything I can do to help zchunk reviews along?
Dne 28.6.2018 v 12:43 Jonathan Dieter napsal(a): > Pull requests to enable zchunk support in librepo, libsolv, dnf, libdnf > and createrepo_c are at: > > https://github.com/rpm-software-management/librepo/pull/127 > https://github.com/openSUSE/libsolv/pull/270 > https://github.com/rpm-software-management/dnf/pull/1107 > https://github.com/rpm-software-management/libdnf/pull/478 > https://github.com/rpm-software-management/createrepo_c/pull/92 > > I'd love to get these merged and properly tested in time for Fedora > 29's change code complete deadline (August 28), but I'm not sure how > reasonable that is. If it is achievable, I need to get the change > proposal done in the next few days, otherwise I'll punt to Fedora 30. You can do the change proposal anyway. In worst case it will be postponed ... V. > > Thanks Neal for looking at all of these patches, and Colin and Igor for > your comments and suggestions. Is there anything I can do to help > these PRs reach a mergeable state, or do I just need to wait for > further reviews? > > Jonathan > > > ___ > Rpm-ecosystem mailing list > Rpm-ecosystem@lists.rpm.org > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Fwd: [Rpm-maint] Fixing macro scoping
Dne 6.2.2017 v 16:00 Panu Matilainen napsal(a): > > One open question I have is what to do with %undefine's: currently rpm > allows %undefining anything from any scope, and that is at odds with > any attempt to rationalize and formalize the scoping to something > actually comprehensible. A simple approach is that you can only > undefine something from your local scope or the global scope. But what > if there's something by the same name in between? > What is wrong with this? You always undefine the value from the top of the stack ... Vít ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Writing a dnf plugin to better deal with out of tree kernel modules
Dne 6.10.2016 v 12:26 Florian Festi napsal(a): > On 10/06/2016 12:16 PM, Vít Ondruch wrote: >> Actually is this really DNF issue? >> >> DNF resolves the dependencies properly and installs what can be >> installed. E.g. it keeps the kernel-core-4.6.1-1 which is compatible >> with the module on the system as long as the module is installed, while >> nothing prohibits installation of more recent kernel. >> >> The problem is that the default boot loader entry is modified. Or that >> kernel is blindly trying to load inappropriate modules. > I disagree. It is the job of dnf and rpm to make sure all the desired > packages get installed and having the matching modules installed with > the new kernel is clearly what is the right thing to do here. And I have to disagree here. If there was only possible to install single kernel at the time, everything would work. E.g. kernel won't be updated as long as there is no updated module. Everybody would be happy. The kernel maintainers claims that kernels can be installed side by side, but apparently this is not fully true. If there are kernel modules build against specific version of kernel, they should be probably installed into some versioned folder, so kernel 4.6.1 loads just modules for kernel 4.6.1 and does not try to load modules for kernel 4.6.2. But this is probably not true. Of course part of the equation (at least for Fedora) is that 3rd party kernel modules are not supported by kernel maintainers, so this is non-issue for them. Vít > Arguing > from an "but rpm (and dnf) follow the rules of the dependencies" POV > does not cut it here. The question is how to craft the rules in a way > they produce the desired result. > > If the module is not installed with the kernel no one is going to > install it later, so having a smarter script for updating the boot > loader won't cut it either. > > Florian > ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Writing a dnf plugin to better deal with out of tree kernel modules
Actually is this really DNF issue? DNF resolves the dependencies properly and installs what can be installed. E.g. it keeps the kernel-core-4.6.1-1 which is compatible with the module on the system as long as the module is installed, while nothing prohibits installation of more recent kernel. The problem is that the default boot loader entry is modified. Or that kernel is blindly trying to load inappropriate modules. Vít Dne 6.10.2016 v 11:28 Hans de Goede napsal(a): > Hi All, > > There has been some discussion about giving end users a smoother > nvidia binary driver experience (or a smoother experience with > any out of tree kernel module they depend on really). > > The problem with out of tree kernel modules installed via > a repo is that as soon as the Fedora kernel pkg gets updated, > the module needs to be rebuild and typically there will be > some lag between the repo hosting the module doing that > rebuild and the user receiving the kernel update. > > akmods, or dkms are 2 solutions to this, but we really do > not want end users to depend on automatically building > stuff from source on their system, this is undesirable > in general, and is esp. problematic since this may > break if the (glue) code needs some minor fix because > of e.g. a kernel function getting an extra argument. > > When things break this way, then the user may end up > with an unusable system (e.g. black screen after boot), > and this is something which we ought to fix. > > I believe that a possible solution for this updates > problem is a dnf plugin which knows about kernel module > packages and which will consider the following scenario > broken deps and thus will skip the kernel update: > > installed > kernel-core-4.6.1-1 > nvidia-kernel-module-4.6.1-1 (requiring kernel-core=4.6.1-1) > nvidia-xorg-driver (requiring nvidia-kernel-module) > > available as update: > kernel-core-4.6.2-1 > > Currently dnf will happily install the update > and then without dkms / akmod on the next reboot the nvidia > binary driver will not work since there is no > nvidia-kernel-module available for the running kernel, > typically resulting in a black screen, which is a quite > bad user experience. And as said we really want to avoid > dkms / akmod. > > So the idea is to have a new dnf-plugin which understands > that if an extra kernel-module is installed that then not > having that module available for a newer kernel should > be treated as that kernel having broken deps causing dnf > to skip the kerel update, unless --best is used in which > case this will be an error. > > So 2 questions: > > 1) Any better ideas to delay a kernel update until > matching extra-module pkgs are available for all > extra-modules the user has installed ? > > 2) Would it be feasible to write such a dnf plugin? > > Regards, > > Hans > ___ > Rpm-ecosystem mailing list > Rpm-ecosystem@lists.rpm.org > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
