Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-28 Thread Vít Ondruch
Dne 28.8.2018 v 13:56 Nicolas Mailhot napsal(a): > Le 2018-08-08 22:48, Jeff Johnson a écrit : >> My issue is the misdirection discussing lazy filelist downloading as a >> "solution" to the "problem" of huge amounts of data that is forced to >> be downloaded and loaded. >> >> The issue has been

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-28 Thread Nicolas Mailhot
Le 2018-08-28 15:00, Nico Kadel-Garcia a écrit : And no, I'm afraid it does *not* scale since the exclusions from another repository may not be appropriate to the repository one is currently building, and automatically including that whitelist is automatically mismatching the content database

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-28 Thread Nico Kadel-Garcia
On Tue, Aug 28, 2018 at 7:57 AM Nicolas Mailhot wrote: > > Le 2018-08-08 22:48, Jeff Johnson a écrit : > > My issue is the misdirection discussing lazy filelist downloading as a > > "solution" to the "problem" of huge amounts of data that is forced to > > be downloaded and loaded. > > > > The

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-28 Thread Nicolas Mailhot
Le 2018-08-08 22:48, Jeff Johnson a écrit : My issue is the misdirection discussing lazy filelist downloading as a "solution" to the "problem" of huge amounts of data that is forced to be downloaded and loaded. The issue has been discussed repeatedly without a solution. Adding -- and

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-09 Thread Zbigniew Jędrzejewski-Szmek
> On Wed, Aug 8, 2018 at 7:09 PM Pascal Terjan wrote: > > $ GET > > http://ftp.free.fr/mirrors/mageia.org/distrib/cauldron/x86_64/media/media_info/file-deps > > /bin/csh > > /bin/grep > > /bin/perl > > /usr/bin/ln > > /usr/bin/rm > > /sbin/service > > /usr/bin/chattr > > /usr/bin/guile > >

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-09 Thread Vít Ondruch
Dne 9.8.2018 v 07:34 Neal Gompa napsal(a): > On Wed, Aug 8, 2018 at 7:09 PM Pascal Terjan wrote: >> On 7 August 2018 at 09:50, Michael Schroeder wrote: >>> On Mon, Aug 06, 2018 at 04:36:07PM +, Zbigniew J??drzejewski-Szmek >>> wrote: this mail is a continuation of an FPC [1] and a

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-09 Thread Jeff Johnson
There is a further optimization for file dependencies not in some whitelist that might be undertaken by depsolvers like dnf: lookup the path in the already installed packages. The fundamental need for depsolvers resolving a dependency is the ability to traverse a mapping from path to package

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-09 Thread Florian Weimer
On 08/08/2018 11:45 PM, Samuel Sieb wrote: Except that it doesn't quite solve it.  What happens if a third party repo package depends on a file that is not included in the Fedora whitelist?  It doesn't help that the other repo has that file path in its whitelist, dnf won't be able to resolve

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-09 Thread Jeff Johnson
Perhaps you do not understand the implications of "maintained" wrto a whitelist. A cross repository file dependency should lead to a request to add a path or pattern to a whitelist for the other repository to provide a file path, not to a lazy download of megabytes of mostly unused data "just

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-08 Thread Neal Gompa
On Wed, Aug 8, 2018 at 7:09 PM Pascal Terjan wrote: > > On 7 August 2018 at 09:50, Michael Schroeder wrote: > > On Mon, Aug 06, 2018 at 04:36:07PM +, Zbigniew J??drzejewski-Szmek > > wrote: > >> this mail is a continuation of an FPC [1] and a FESCo [2] tickets. > >> > >> A proposal was made

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-08 Thread Samuel Sieb
On 08/08/2018 01:48 PM, Jeff Johnson wrote: Adding -- and maintaining -- patterns or whitelist exceptions, which moves file dependencies into primary.xml is actually an approach that solves the problem, and scales to multiple repos as well, each of which also will have their own patterns and

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-08 Thread Jeff Johnson
My issue is the misdirection discussing lazy filelist downloading as a "solution" to the "problem" of huge amounts of data that is forced to be downloaded and loaded. The issue has been discussed repeatedly without a solution. Adding -- and maintaining -- patterns or whitelist exceptions,

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-08 Thread Samuel Sieb
On 08/08/2018 11:04 AM, Jeff Johnson wrote: But "occasionally" downloading file lists does not solve the problems stated, either then, or now. I haven't been able to figure out what your problem is with this. Do you think that "occasionally" is incorrect and it will be "usually"? Or you

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-08 Thread Adam Williamson
On Wed, 2018-08-08 at 16:43 +, Zbigniew Jędrzejewski-Szmek wrote: > That's a good question. The time window where this can happen is not > be that big, because without filelists the loading of metadata is > quicker. But it's still non-zero, so given enough machines and enough > updates, it'll

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-08 Thread Jeff Johnson
> On Tue, Aug 07, 2018 at 08:50:25AM +, Michael Schroeder wrote: > > Yep, that sounds like an excellent idea. > > > That's a good question. The time window where this can happen is not > be that big, because without filelists the loading of metadata is > quicker. But it's still non-zero, so

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-08 Thread Zbigniew Jędrzejewski-Szmek
On Tue, Aug 07, 2018 at 08:50:25AM +, Michael Schroeder wrote: > On Mon, Aug 06, 2018 at 04:36:07PM +, Zbigniew J??drzejewski-Szmek wrote: > > this mail is a continuation of an FPC [1] and a FESCo [2] tickets. > > > > A proposal was made is to disallow packages in Fedora from using file >

Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

2018-08-07 Thread Jonathan Dieter
On Mon, 2018-08-06 at 16:36 +, Zbigniew Jędrzejewski-Szmek wrote: > Hi dnf and libsolv developers, > > this mail is a continuation of an FPC [1] and a FESCo [2] tickets. > > A proposal was made is to disallow packages in Fedora from using file > deps, and to optimize dnf to not load