I have now flushed and migrated a couple of times and tried to sync my migrated
'appstream' repo against the migrated 'appstream' remote. Both times I've done
this the result is:
{
"child_tasks": [],
"created_resources": [],
"error": {
"description": "Incoming and existing advisories have the same id and
timestamp but different and intersecting package lists. At least one of them is
wrong. Advisory id: RHBA-2019:2723",
"traceback": " File \"/usr/lib/python3.6/site-packages/rq/worker.py\",
line 883, in perform_job\n rv = job.perform()\n File
\"/usr/lib/python3.6/site-packages/rq/job.py\", line 657, in perform\n
self._result = self._execute()\n File
\"/usr/lib/python3.6/site-packages/rq/job.py\", line 663, in _execute\n
return self.func(*self.args, **self.kwargs)\n File
\"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line
264, in synchronize\n dv.create()\n File
\"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\",
line 148, in create\n loop.run_until_complete(pipeline)\n File
\"/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py\", line
776, in __exit__\n repository.finalize_new_version(self)\n File
\"/usr/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py\", line
151, in finalize_new_version\n resolve_advisories(new_version,
previous_version)\n File
\"/usr/lib/python3.6/site-packages/pulp_rpm/app/advisory.py\", line 79, in
resolve_advisories\n previous_advisory, added_advisory\n File
\"/usr/lib/python3.6/site-packages/pulp_rpm/app/advisory.py\", line 154, in
resolve_advisory_conflict\n raise AdvisoryConflict(_('Incoming and existing
advisories have the same id and '\n"
},
"finished_at": "2020-10-02T14:16:17.768388Z",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"parent_task": null,
"progress_reports": [
{
"code": "parsing.modulemds",
"done": 206,
"message": "Parsed Modulemd",
"state": "completed",
"suffix": null,
"total": 206
},
{
"code": "parsing.modulemd_defaults",
"done": 42,
"message": "Parsed Modulemd-defaults",
"state": "completed",
"suffix": null,
"total": 42
},
{
"code": "parsing.comps",
"done": 67,
"message": "Parsed Comps",
"state": "completed",
"suffix": null,
"total": 67
},
{
"code": "parsing.advisories",
"done": 784,
"message": "Parsed Advisories",
"state": "completed",
"suffix": null,
"total": 784
},
{
"code": "parsing.packages",
"done": 12185,
"message": "Parsed Packages",
"state": "completed",
"suffix": null,
"total": 12185
},
{
"code": "downloading.metadata",
"done": 5,
"message": "Downloading Metadata Files",
"state": "completed",
"suffix": null,
"total": null
},
{
"code": "downloading.artifacts",
"done": 64,
"message": "Downloading Artifacts",
"state": "completed",
"suffix": null,
"total": null
},
{
"code": "associating.content",
"done": 975,
"message": "Associating Content",
"state": "completed",
"suffix": null,
"total": null
}
],
"pulp_created": "2020-10-02T14:14:07.970174Z",
"pulp_finished_at": "2020-10-02T14:16:17.768388Z",
"pulp_href": "/pulp/api/v3/tasks/0eb7d3a2-c8f4-42bd-af52-b25ccc6a8015/",
"reserved_resources_record": [
"/pulp/api/v3/repositories/rpm/rpm/44baa281-e85f-40cc-ab76-9a91e45237cf/",
"/pulp/api/v3/remotes/rpm/rpm/e48a62af-bf49-4632-8a57-2f0c66a82be9/"
],
"started_at": "2020-10-02T14:14:08.531173Z",
"state": "failed",
"task_group": null,
"worker": "/pulp/api/v3/workers/9e02cddf-f703-41f3-8c84-6b03b3592887/"
}
So there is a lot of content being pulled in even though the repo should be
complete from the migration. We do however have 'Retain Old Count' set in our
pulp2 env, and I dont know if that setting is available in pulp 3.4 which I am
running (looks like it's added in 3.5), which might explain that. Regarding the
error there is a closed issue about this:
https://pulp.plan.io/issues/6531
I have been able to reproduce it twice now so that issue might be worth
re-opening. Whether this has anything at all to do with the multi-repo copy
behaviour (which I have also tested after each migration and it still does not
produce any dependencies) I don't know.
//Adam
________________________________
From: [email protected] <[email protected]> on behalf of
Winberg Adam <[email protected]>
Sent: 30 September 2020 09:11
To: Daniel Alley
Cc: [email protected]
Subject: Re: [Pulp-list] rpm plugin: multirepo copy of errata not working
>Ok, tried that now, synced a total of 72 advisories in 3-4 copy operations.
This was a bit unclear I realize, but I created new pulp3 repos for appstream
and baseos and synced them with our redhat cdn remote as source. Then I
_copied_ a total of 72 advisories from this new appstream repo to a new empty
repo, while using the new baseos repo as a 'dependency source' paired with
another new empty repo.
//Adam
________________________________
From: [email protected] <[email protected]> on behalf of
Winberg Adam <[email protected]>
Sent: 30 September 2020 09:06
To: Daniel Alley
Cc: [email protected]
Subject: Re: [Pulp-list] rpm plugin: multirepo copy of errata not working
We're glad to help, Pulp is extremely useful for us so if we can contribute we
are happy to do so. I created an issue for this:
https://pulp.plan.io/issues/7625
> If you sync the Pulp 3 repository using the migrated remote, what happens?
Do you mean syncing the 'my-new-repo1' repo with the migrated appstream remote?
> Are the Pulp 2 repositories you're migrating modified significantly from when
> they were originally synced?
Well, in the case of the appstream repo it has been synced against the Red Hat
source every night, so the content has certainly been modified. Otherwise than
that, no.
> If you make brand new Pulp 3 repositories and try the same copy operation,
> does the same weird copy behavior occur?
Do you mean new pulp3 repos synced using the appstream and baseos remotes (w.
content from redhat cdn) and then using that as the source of the copy? Ok,
tried that now, synced a total of 72 advisories in 3-4 copy operations. The
copy itself between my new pulp3 appstream repo and the new empty repo works
well and 610 rpm packages has been copied. But there are no dependencies copied
from the new baseos repo, no content at all.
//Adam
________________________________
From: Daniel Alley <[email protected]>
Sent: 29 September 2020 20:28
To: Winberg Adam
Cc: [email protected]
Subject: Re: [Pulp-list] rpm plugin: multirepo copy of errata not working
Hi Adam,
Thank you for providing feedback on some of these rough edges, it is extremely
helpful.
There's a couple of things you can try that would give us some useful
information. If you sync the Pulp 3 repository using the migrated remote, what
happens? Does the repository change or does it stay the same?
Are the Pulp 2 repositories you're migrating modified significantly from when
they were originally synced? If you make brand new Pulp 3 repositories and try
the same copy operation, does the same weird copy behavior occur?
Also, if it wouldn't be too much trouble, could you file this as an issue on
our bug tracker (https://pulp.plan.io/) with the details? We can continue
discussion here but if there is a lot of information it's better to keep it in
one place on the issue.
Thanks again,
Daniel
On Tue, Sep 29, 2020 at 8:21 AM Winberg Adam
<[email protected]<mailto:[email protected]>> wrote:
I applied the patch from
https://github.com/pulp/pulp_rpm/commit/712abdf1abb95c969b54fd2968a573189b77bcba
and the the copy then went through without errors.
I'm a bit confused by the result however. I copied 16 advisories from the
appstreams repo to my new empty repo and the copy ended up with copying all
modulemds and almost all packages (11466 of 12053). That doesnt seem right to
me. And my other new repo ('my-new-repo2') is still empty, meaning that of
those 11000 packages there were none that had any dependencies from the baseos
repo which also strikes me as odd. If i set 'dependency_solving=True' (contrary
to the documentation) I end up with the same amount of packages in my
'my-new-repo1' and 2 packages in 'my-new-repo2'.
Am I misunderstanding something about this functionality?
//Adam
________________________________
From: Winberg Adam
Sent: 28 September 2020 16:44
To: [email protected]<mailto:[email protected]>
Subject: rpm plugin: multirepo copy of errata not working
I have succeeded in migrating my pulp2 content to pulp3 and all repos look
complete (i've reran the migration and no new repoversions are generated).
In an attempt to test the multirepo copy functionality described at
https://pulp-rpm.readthedocs.io/en/latest/workflows/copy.html#recipes
, I created a couple of new, empty repos and tried to copy errata from my
migrated RHEL8-appstream repo to them:
POST /pulp/api/v3/rpm/copy/
config:=[
{"source_repo_version": "<rhel8-appstream-version-href>", "dest_repo":
"my-new-repo1", "content": [$ADVISORY_HREF1]},
{"source_repo_version": "<rhel8-baseos-version-href>", "dest_repo":
"my-new-repo2", "content": []},
]
dependency_solving=False
All looks correct, but the operation ultimately fails with the following error:
"description": "Modulemd matching query does not exist.",
"traceback": " File \"/usr/lib/python3.6/site-packages/rq/worker.py\",
line 883, in perform_job\n rv = job.perform()\n File
\"/usr/lib/python3.6/site-packages/rq/job.py\", line 657, in perform\n
self._result = self._execute()\n File
\"/usr/lib/python3.6/site-packages/rq/job.py\", line 663, in _execute\n
return self.func(*self.args, **self.kwargs)\n File
\"/usr/lib64/python3.6/contextlib.py\", line 52, in inner\n return
func(*args, **kwds)\n File
\"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/copy.py\", line 167, in
copy_content\n content_to_copy |= find_children_of_content(content_to_copy,
source_repo_version)\n File
\"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/copy.py\", line 74, in
find_children_of_content\n name=name, stream=stream, version=version,
context=context, arch=arch)\n File
\"/usr/lib/python3.6/site-packages/django/db/models/query.py\", line 408, in
get\n self.model._meta.object_name\n"
},
Any ideas why this happens?
Regards,
Adam
_______________________________________________
Pulp-list mailing list
[email protected]<mailto:[email protected]>
https://www.redhat.com/mailman/listinfo/pulp-list
_______________________________________________
Pulp-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/pulp-list
