Re: [Pulp-list] rpm plugin: multirepo copy of errata not working
I have now flushed and migrated a couple of times and tried to sync my migrated
'appstream' repo against the migrated 'appstream' remote. Both times I've done
this the result is:
{
"child_tasks": [],
"created_resources": [],
"error": {
"description": "Incoming and existing advisories have the same id and
timestamp but different and intersecting package lists. At least one of them is
wrong. Advisory id: RHBA-2019:2723",
"traceback": " File \"/usr/lib/python3.6/site-packages/rq/worker.py\",
line 883, in perform_job\nrv = job.perform()\n File
\"/usr/lib/python3.6/site-packages/rq/job.py\", line 657, in perform\n
self._result = self._execute()\n File
\"/usr/lib/python3.6/site-packages/rq/job.py\", line 663, in _execute\n
return self.func(*self.args, **self.kwargs)\n File
\"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line
264, in synchronize\ndv.create()\n File
\"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\",
line 148, in create\nloop.run_until_complete(pipeline)\n File
\"/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py\", line
776, in __exit__\nrepository.finalize_new_version(self)\n File
\"/usr/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py\", line
151, in finalize_new_version\nresolve_advisories(new_version,
previous_version)\n File
\"/usr/lib/python3.6/site-packages/pulp_rpm/app/advisory.py\", line 79, in
resolve_advisories\nprevious_advisory, added_advisory\n File
\"/usr/lib/python3.6/site-packages/pulp_rpm/app/advisory.py\", line 154, in
resolve_advisory_conflict\nraise AdvisoryConflict(_('Incoming and existing
advisories have the same id and '\n"
},
"finished_at": "2020-10-02T14:16:17.768388Z",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"parent_task": null,
"progress_reports": [
{
"code": "parsing.modulemds",
"done": 206,
"message": "Parsed Modulemd",
"state": "completed",
"suffix": null,
"total": 206
},
{
"code": "parsing.modulemd_defaults",
"done": 42,
"message": "Parsed Modulemd-defaults",
"state": "completed",
"suffix": null,
"total": 42
},
{
"code": "parsing.comps",
"done": 67,
"message": "Parsed Comps",
"state": "completed",
"suffix": null,
"total": 67
},
{
"code": "parsing.advisories",
"done": 784,
"message": "Parsed Advisories",
"state": "completed",
"suffix": null,
"total": 784
},
{
"code": "parsing.packages",
"done": 12185,
"message": "Parsed Packages",
"state": "completed",
"suffix": null,
"total": 12185
},
{
"code": "downloading.metadata",
"done": 5,
"message": "Downloading Metadata Files",
"state": "completed",
"suffix": null,
"total": null
},
{
"code": "downloading.artifacts",
"done": 64,
"message": "Downloading Artifacts",
"state": "completed",
"suffix": null,
"total": null
},
{
"code": "associating.content",
"done": 975,
"message": "Associating Content",
"state": "completed",
"suffix": null,
"total": null
}
],
"pulp_created": "2020-10-02T14:14:07.970174Z",
"pulp_finished_at": "2020-10-02T14:16:17.768388Z",
"pulp_href": "/pulp/api/v3/tasks/0eb7d3a2-c8f4-42bd-af52-b25ccc6a8015/",
"reserved_resources_record": [
"/pulp/api/v3/repositories/rpm/rpm/44baa281-e85f-40cc-ab76-9a91e45237cf/",
"/pulp/api/v3/remotes/rpm/rpm/e48a62af-bf49-4632-8a57-2f0c66a82be9/"
],
"started_at": "2020-10-02T14:14:08.531173Z",
"state": "failed",
"task_group": null,
"worker": "/pulp/api/v3/workers/9e02cddf-f703-41f3-8c84-6b03b3592887/"
Re: [Pulp-list] rpm plugin: multirepo copy of errata not working
>Ok, tried that now, synced a total of 72 advisories in 3-4 copy operations. This was a bit unclear I realize, but I created new pulp3 repos for appstream and baseos and synced them with our redhat cdn remote as source. Then I _copied_ a total of 72 advisories from this new appstream repo to a new empty repo, while using the new baseos repo as a 'dependency source' paired with another new empty repo. //Adam From: [email protected] on behalf of Winberg Adam Sent: 30 September 2020 09:06 To: Daniel Alley Cc: [email protected] Subject: Re: [Pulp-list] rpm plugin: multirepo copy of errata not working We're glad to help, Pulp is extremely useful for us so if we can contribute we are happy to do so. I created an issue for this: https://pulp.plan.io/issues/7625 > If you sync the Pulp 3 repository using the migrated remote, what happens? Do you mean syncing the 'my-new-repo1' repo with the migrated appstream remote? > Are the Pulp 2 repositories you're migrating modified significantly from when > they were originally synced? Well, in the case of the appstream repo it has been synced against the Red Hat source every night, so the content has certainly been modified. Otherwise than that, no. > If you make brand new Pulp 3 repositories and try the same copy operation, > does the same weird copy behavior occur? Do you mean new pulp3 repos synced using the appstream and baseos remotes (w. content from redhat cdn) and then using that as the source of the copy? Ok, tried that now, synced a total of 72 advisories in 3-4 copy operations. The copy itself between my new pulp3 appstream repo and the new empty repo works well and 610 rpm packages has been copied. But there are no dependencies copied from the new baseos repo, no content at all. //Adam From: Daniel Alley Sent: 29 September 2020 20:28 To: Winberg Adam Cc: [email protected] Subject: Re: [Pulp-list] rpm plugin: multirepo copy of errata not working Hi Adam, Thank you for providing feedback on some of these rough edges, it is extremely helpful. There's a couple of things you can try that would give us some useful information. If you sync the Pulp 3 repository using the migrated remote, what happens? Does the repository change or does it stay the same? Are the Pulp 2 repositories you're migrating modified significantly from when they were originally synced? If you make brand new Pulp 3 repositories and try the same copy operation, does the same weird copy behavior occur? Also, if it wouldn't be too much trouble, could you file this as an issue on our bug tracker (https://pulp.plan.io/) with the details? We can continue discussion here but if there is a lot of information it's better to keep it in one place on the issue. Thanks again, Daniel On Tue, Sep 29, 2020 at 8:21 AM Winberg Adam mailto:[email protected]>> wrote: I applied the patch from https://github.com/pulp/pulp_rpm/commit/712abdf1abb95c969b54fd2968a573189b77bcba and the the copy then went through without errors. I'm a bit confused by the result however. I copied 16 advisories from the appstreams repo to my new empty repo and the copy ended up with copying all modulemds and almost all packages (11466 of 12053). That doesnt seem right to me. And my other new repo ('my-new-repo2') is still empty, meaning that of those 11000 packages there were none that had any dependencies from the baseos repo which also strikes me as odd. If i set 'dependency_solving=True' (contrary to the documentation) I end up with the same amount of packages in my 'my-new-repo1' and 2 packages in 'my-new-repo2'. Am I misunderstanding something about this functionality? //Adam From: Winberg Adam Sent: 28 September 2020 16:44 To: [email protected]<mailto:[email protected]> Subject: rpm plugin: multirepo copy of errata not working I have succeeded in migrating my pulp2 content to pulp3 and all repos look complete (i've reran the migration and no new repoversions are generated). In an attempt to test the multirepo copy functionality described at https://pulp-rpm.readthedocs.io/en/latest/workflows/copy.html#recipes , I created a couple of new, empty repos and tried to copy errata from my migrated RHEL8-appstream repo to them: POST /pulp/api/v3/rpm/copy/ config:=[ {"source_repo_version": "", "dest_repo": "my-new-repo1", "content": [$ADVISORY_HREF1]}, {"source_repo_version": "", "dest_repo": "my-new-repo2", "content": []}, ] dependency_solving=False All looks correct, but the operation ultimately fails with the following error: "description": "Modulemd matching query does n
Re: [Pulp-list] rpm plugin: multirepo copy of errata not working
We're glad to help, Pulp is extremely useful for us so if we can contribute we are happy to do so. I created an issue for this: https://pulp.plan.io/issues/7625 > If you sync the Pulp 3 repository using the migrated remote, what happens? Do you mean syncing the 'my-new-repo1' repo with the migrated appstream remote? > Are the Pulp 2 repositories you're migrating modified significantly from when > they were originally synced? Well, in the case of the appstream repo it has been synced against the Red Hat source every night, so the content has certainly been modified. Otherwise than that, no. > If you make brand new Pulp 3 repositories and try the same copy operation, > does the same weird copy behavior occur? Do you mean new pulp3 repos synced using the appstream and baseos remotes (w. content from redhat cdn) and then using that as the source of the copy? Ok, tried that now, synced a total of 72 advisories in 3-4 copy operations. The copy itself between my new pulp3 appstream repo and the new empty repo works well and 610 rpm packages has been copied. But there are no dependencies copied from the new baseos repo, no content at all. //Adam From: Daniel Alley Sent: 29 September 2020 20:28 To: Winberg Adam Cc: [email protected] Subject: Re: [Pulp-list] rpm plugin: multirepo copy of errata not working Hi Adam, Thank you for providing feedback on some of these rough edges, it is extremely helpful. There's a couple of things you can try that would give us some useful information. If you sync the Pulp 3 repository using the migrated remote, what happens? Does the repository change or does it stay the same? Are the Pulp 2 repositories you're migrating modified significantly from when they were originally synced? If you make brand new Pulp 3 repositories and try the same copy operation, does the same weird copy behavior occur? Also, if it wouldn't be too much trouble, could you file this as an issue on our bug tracker (https://pulp.plan.io/) with the details? We can continue discussion here but if there is a lot of information it's better to keep it in one place on the issue. Thanks again, Daniel On Tue, Sep 29, 2020 at 8:21 AM Winberg Adam mailto:[email protected]>> wrote: I applied the patch from https://github.com/pulp/pulp_rpm/commit/712abdf1abb95c969b54fd2968a573189b77bcba and the the copy then went through without errors. I'm a bit confused by the result however. I copied 16 advisories from the appstreams repo to my new empty repo and the copy ended up with copying all modulemds and almost all packages (11466 of 12053). That doesnt seem right to me. And my other new repo ('my-new-repo2') is still empty, meaning that of those 11000 packages there were none that had any dependencies from the baseos repo which also strikes me as odd. If i set 'dependency_solving=True' (contrary to the documentation) I end up with the same amount of packages in my 'my-new-repo1' and 2 packages in 'my-new-repo2'. Am I misunderstanding something about this functionality? //Adam From: Winberg Adam Sent: 28 September 2020 16:44 To: [email protected]<mailto:[email protected]> Subject: rpm plugin: multirepo copy of errata not working I have succeeded in migrating my pulp2 content to pulp3 and all repos look complete (i've reran the migration and no new repoversions are generated). In an attempt to test the multirepo copy functionality described at https://pulp-rpm.readthedocs.io/en/latest/workflows/copy.html#recipes , I created a couple of new, empty repos and tried to copy errata from my migrated RHEL8-appstream repo to them: POST /pulp/api/v3/rpm/copy/ config:=[ {"source_repo_version": "", "dest_repo": "my-new-repo1", "content": [$ADVISORY_HREF1]}, {"source_repo_version": "", "dest_repo": "my-new-repo2", "content": []}, ] dependency_solving=False All looks correct, but the operation ultimately fails with the following error: "description": "Modulemd matching query does not exist.", "traceback": " File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 883, in perform_job\nrv = job.perform()\n File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 657, in perform\n self._result = self._execute()\n File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 663, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/usr/lib64/python3.6/contextlib.py\", line 52, in inner\nreturn func(*args, **kwds)\n File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/copy.py\", line 167, in copy_content\ncontent_to_co
Re: [Pulp-list] rpm plugin: multirepo copy of errata not working
Hi Adam,
Thank you for providing feedback on some of these rough edges, it is
*extremely* helpful.
There's a couple of things you can try that would give us some useful
information. If you sync the Pulp 3 repository using the migrated remote,
what happens? Does the repository change or does it stay the same?
Are the Pulp 2 repositories you're migrating modified significantly from
when they were originally synced? If you make brand new Pulp 3 repositories
and try the same copy operation, does the same weird copy behavior occur?
Also, if it wouldn't be too much trouble, could you file this as an issue
on our bug tracker (https://pulp.plan.io/) with the details? We can
continue discussion here but if there is a lot of information it's better
to keep it in one place on the issue.
Thanks again,
Daniel
On Tue, Sep 29, 2020 at 8:21 AM Winberg Adam wrote:
> I applied the patch from
>
>
> https://github.com/pulp/pulp_rpm/commit/712abdf1abb95c969b54fd2968a573189b77bcba
>
>
> and the the copy then went through without errors.
>
>
> I'm a bit confused by the result however. I copied 16 advisories from the
> appstreams repo to my new empty repo and the copy ended up with copying all
> modulemds and almost all packages (11466 of 12053). That doesnt seem right
> to me. And my other new repo ('my-new-repo2') is still empty, meaning that
> of those 11000 packages there were none that had any dependencies from the
> baseos repo which also strikes me as odd. If i set
> 'dependency_solving=True' (contrary to the documentation) I end up with the
> same amount of packages in my 'my-new-repo1' and 2 packages in
> 'my-new-repo2'.
>
>
> Am I misunderstanding something about this functionality?
>
>
> //Adam
>
>
>
>
>
> --
> *From:* Winberg Adam
> *Sent:* 28 September 2020 16:44
> *To:* [email protected]
> *Subject:* rpm plugin: multirepo copy of errata not working
>
>
> I have succeeded in migrating my pulp2 content to pulp3 and all repos look
> complete (i've reran the migration and no new repoversions are generated).
>
>
> In an attempt to test the multirepo copy functionality described at
>
> https://pulp-rpm.readthedocs.io/en/latest/workflows/copy.html#recipes
>
>
> , I created a couple of new, empty repos and tried to copy errata from my
> migrated RHEL8-appstream repo to them:
>
> POST /pulp/api/v3/rpm/copy/
> config:=[
> {"source_repo_version": "",
> "dest_repo": "my-new-repo1", "content": [$ADVISORY_HREF1]},
> {"source_repo_version": "", "dest_repo":
> "my-new-repo2", "content": []},
> ]
> dependency_solving=False
>
>
> All looks correct, but the operation ultimately fails with the following
> error:
>
> "description": "Modulemd matching query does not exist.",
> "traceback": " File
> \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 883, in
> perform_job\nrv = job.perform()\n File
> \"/usr/lib/python3.6/site-packages/rq/job.py\", line 657, in perform\n
> self._result = self._execute()\n File
> \"/usr/lib/python3.6/site-packages/rq/job.py\", line 663, in _execute\n
> return self.func(*self.args, **self.kwargs)\n File
> \"/usr/lib64/python3.6/contextlib.py\", line 52, in inner\nreturn
> func(*args, **kwds)\n File
> \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/copy.py\", line 167,
> in copy_content\ncontent_to_copy |=
> find_children_of_content(content_to_copy, source_repo_version)\n File
> \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/copy.py\", line 74,
> in find_children_of_content\nname=name, stream=stream, version=version,
> context=context, arch=arch)\n File
> \"/usr/lib/python3.6/site-packages/django/db/models/query.py\", line 408,
> in get\nself.model._meta.object_name\n"
> },
>
> Any ideas why this happens?
>
>
> Regards,
>
> Adam
>
>
>
>
> ___
> Pulp-list mailing list
> [email protected]
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/pulp-list
Re: [Pulp-list] rpm plugin: multirepo copy of errata not working
I applied the patch from
https://github.com/pulp/pulp_rpm/commit/712abdf1abb95c969b54fd2968a573189b77bcba
and the the copy then went through without errors.
I'm a bit confused by the result however. I copied 16 advisories from the
appstreams repo to my new empty repo and the copy ended up with copying all
modulemds and almost all packages (11466 of 12053). That doesnt seem right to
me. And my other new repo ('my-new-repo2') is still empty, meaning that of
those 11000 packages there were none that had any dependencies from the baseos
repo which also strikes me as odd. If i set 'dependency_solving=True' (contrary
to the documentation) I end up with the same amount of packages in my
'my-new-repo1' and 2 packages in 'my-new-repo2'.
Am I misunderstanding something about this functionality?
//Adam
From: Winberg Adam
Sent: 28 September 2020 16:44
To: [email protected]
Subject: rpm plugin: multirepo copy of errata not working
I have succeeded in migrating my pulp2 content to pulp3 and all repos look
complete (i've reran the migration and no new repoversions are generated).
In an attempt to test the multirepo copy functionality described at
https://pulp-rpm.readthedocs.io/en/latest/workflows/copy.html#recipes
, I created a couple of new, empty repos and tried to copy errata from my
migrated RHEL8-appstream repo to them:
POST /pulp/api/v3/rpm/copy/
config:=[
{"source_repo_version": "", "dest_repo":
"my-new-repo1", "content": [$ADVISORY_HREF1]},
{"source_repo_version": "", "dest_repo":
"my-new-repo2", "content": []},
]
dependency_solving=False
All looks correct, but the operation ultimately fails with the following error:
"description": "Modulemd matching query does not exist.",
"traceback": " File \"/usr/lib/python3.6/site-packages/rq/worker.py\",
line 883, in perform_job\nrv = job.perform()\n File
\"/usr/lib/python3.6/site-packages/rq/job.py\", line 657, in perform\n
self._result = self._execute()\n File
\"/usr/lib/python3.6/site-packages/rq/job.py\", line 663, in _execute\n
return self.func(*self.args, **self.kwargs)\n File
\"/usr/lib64/python3.6/contextlib.py\", line 52, in inner\nreturn
func(*args, **kwds)\n File
\"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/copy.py\", line 167, in
copy_content\ncontent_to_copy |= find_children_of_content(content_to_copy,
source_repo_version)\n File
\"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/copy.py\", line 74, in
find_children_of_content\nname=name, stream=stream, version=version,
context=context, arch=arch)\n File
\"/usr/lib/python3.6/site-packages/django/db/models/query.py\", line 408, in
get\nself.model._meta.object_name\n"
},
Any ideas why this happens?
Regards,
Adam
___
Pulp-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/pulp-list
[Pulp-list] rpm plugin: multirepo copy of errata not working
I have succeeded in migrating my pulp2 content to pulp3 and all repos look
complete (i've reran the migration and no new repoversions are generated).
In an attempt to test the multirepo copy functionality described at
https://pulp-rpm.readthedocs.io/en/latest/workflows/copy.html#recipes
, I created a couple of new, empty repos and tried to copy errata from my
migrated RHEL8-appstream repo to them:
POST /pulp/api/v3/rpm/copy/
config:=[
{"source_repo_version": "", "dest_repo":
"my-new-repo1", "content": [$ADVISORY_HREF1]},
{"source_repo_version": "", "dest_repo":
"my-new-repo2", "content": []},
]
dependency_solving=False
All looks correct, but the operation ultimately fails with the following error:
"description": "Modulemd matching query does not exist.",
"traceback": " File \"/usr/lib/python3.6/site-packages/rq/worker.py\",
line 883, in perform_job\nrv = job.perform()\n File
\"/usr/lib/python3.6/site-packages/rq/job.py\", line 657, in perform\n
self._result = self._execute()\n File
\"/usr/lib/python3.6/site-packages/rq/job.py\", line 663, in _execute\n
return self.func(*self.args, **self.kwargs)\n File
\"/usr/lib64/python3.6/contextlib.py\", line 52, in inner\nreturn
func(*args, **kwds)\n File
\"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/copy.py\", line 167, in
copy_content\ncontent_to_copy |= find_children_of_content(content_to_copy,
source_repo_version)\n File
\"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/copy.py\", line 74, in
find_children_of_content\nname=name, stream=stream, version=version,
context=context, arch=arch)\n File
\"/usr/lib/python3.6/site-packages/django/db/models/query.py\", line 408, in
get\nself.model._meta.object_name\n"
},
Any ideas why this happens?
Regards,
Adam
___
Pulp-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/pulp-list
