I discussed this a little bit on the #rpm.org channel. Here is the gist of
that discussion
- The metadata is "crazy, but technically valid"
- "the entire SUSE ecosystem tends to do this a lot, anything using OBS,
including nvidia and dell and friends"
- "also, SUSE packages can have the same NEVRA with being completely
different packages because of how their build system makes packages"
I'm not sure what the best means to fix it would be. Perhaps the
uniqueness constraint should be on the location_href, instead of on the
NEVRA? Or on NEVRA + location_href?
On Wed, Mar 18, 2020 at 9:47 AM Ina Panova wrote:
> Pavel,
> I meant to say, that pulp3 does not have such limitation as pulp2 had (
> saving rpms on the filesystem with same nevra).
> The error is raised in pulp3 [0] when a repo version is created, because
> of the repo key[1], we cannot have 2 rpms with save NEVRA.
>
> We can enable that, if we decide to, by adding location_href to the
> repo_key, *but* this needs to be evaluated, it can have side effects and we
> should involve our stakeholders to weigh in.
>
> [0]
> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570
> [1]
> https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/package.py#L188
>
>
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
> go instead where there is no path and leave a trail."
>
>
> On Wed, Mar 18, 2020 at 2:24 PM Pavel Picka wrote:
>
>> True in opensuse repository there are two possibilities 'src' and 'nosrc'
>> (this one should be legacy without source code), both are recognized by
>> createrepo_c as arch 'src'.
>>
>> To point the pulp2 code I mentioned I found here [0] (base rpm package
>> what I understood).
>>
>> The rise of error in pulp3 happening here [1] in pulpcore when adding
>> packages to repository version.
>> So as Ina mentioned it doesn't have to be an issue with packages itself
>> than the logic in sync.
>>
>> [0]
>> https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L779
>> [1]
>> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570
>>
>> On Wed, Mar 18, 2020 at 1:55 PM Ina Panova wrote:
>>
>>> Tanya and Pavel,
>>> in this issue it is explained why we cannot keep 2 packages with same
>>> NEVRA but different checksums within a repo
>>> https://pulp.plan.io/issues/494
>>>
>>> Pulp2 had a limitation where it was not able to save on the filesystem 2
>>> rpms with same filename, it lead to the primary.xml that could have pointed
>>> to the rpm that did not actually get saved.
>>> I believe in Pulp3 we could allow having rpm with same NEVRA if they
>>> have different location_href within a repo.
>>>
>>>
>>> Regards,
>>>
>>> Ina Panova
>>> Senior Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>> go instead where there is no path and leave a trail."
>>>
>>>
>>> On Wed, Mar 18, 2020 at 10:47 AM Tatiana Tereshchenko <
>>> ttere...@redhat.com> wrote:
>>>
Hi Pavel,
On Tue, Mar 17, 2020 at 7:31 PM Pavel Picka wrote:
> Hello, would like to ask you how to proceed with issue with duplicate
> (but not really) packages.
>
> I am syncing suse repository (opensuse42 and SLE12) and get and
> duplicate error. But when checking the packages [0](from primary.xml)
> glibc
> and glibc they got same nevra but different checksum (and a few more as
> size..) so doesn't look like real duplicates.
>
Those are weird, the have the same nevra but see the location_href, one
is src and the other one is nosrc! :/ :
It looks like something OpenSUSE specific. I'm not sure if it's a valid
way to create a repo with such metadata, we need to figure it out at some
point.
> I've checked Pulp2 and there is used nevra+sum for repository
> uniqueness. In pulp3 we use only nevra.
>
Why do you think that in pulp 2 we use NEVRA + checksum? have you
tested it? please point to the code.
I believe in Pulp 2 as well as in Pulp 3 we allow to have packages with
different checksums in Pulp storage.
I don't think we allow having the same packages with different
checksums in the same repo.
FWIW, in pulp 2 the most recently added package is chosen to stay in a
repo, no packages with duplicate NEVRA left after sync, see
https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/purge.py#L285-L333
>
> My suggestion is to extend repo_key_fields for rpm package as is in
> pulp2 with pkgId (checksum). As I don't think they are really duplicates
> and other software can rely on specific version of package.
>
Unfortunately, I don't remember the main reason to remove duplicates
based on nevra. Was it because some tooling will complain, or
