Re: [Pulp-list] Error on Azure storage

[Dennis Kliban]

I recommend filing an issue here:
https://github.com/Azure/azure-sdk-for-python/issues

It seems like that issue tracker has a few related issues in a closed
state. Looks like they were resolved with a change on the Azure side.

On Wed, May 25, 2022 at 2:19 PM Briand, Sheldon <
sheldon.bri...@nrc-cnrc.gc.ca> wrote:

> I’m currently using:
>
> Django Storages: 1.11.1.
>
> Pulp core: 3.17.6
>
> Pulp rpm: 3.17.4
>
>
>
> *From:* Dennis Kliban [mailto:dkli...@redhat.com]
> *Sent:* May 25, 2022 3:10 PM
> *To:* Briand, Sheldon 
> *Cc:* pulp-list@redhat.com
> *Subject:* Re: [Pulp-list] Error on Azure storage
>
>
>
> ATTENTION*** This email originated from outside of the NRC.
> ***ATTENTION*** Ce courriel provient de l'extérieur du CNRC*
>
> What version of django-storages do you have installed? Could you try
> downgrading it to 1.11.1? Don't forget to restart pulpcore-content services
> after.
>
>
>
> I found a not completely related issue[0] with django-storages newer than
> 1.11.1.
>
>
>
> [0] https://github.com/jschneier/django-storages/issues/1116
>
>
>
> On Wed, May 25, 2022 at 1:47 PM Briand, Sheldon <
> sheldon.bri...@nrc-cnrc.gc.ca> wrote:
>
> Both VMs are synced to the same time server using chronyd.
>
>
>
> *From:* Dennis Kliban [mailto:dkli...@redhat.com]
> *Sent:* May 25, 2022 2:46 PM
> *To:* Briand, Sheldon 
> *Cc:* pulp-list@redhat.com
> *Subject:* Re: [Pulp-list] Error on Azure storage
>
>
>
> ATTENTION*** This email originated from outside of the NRC.
> ***ATTENTION*** Ce courriel provient de l'extérieur du CNRC*
>
> Perhaps the time is wrong on the machine trying to download the content?
>
>
>
> On Wed, May 25, 2022 at 1:33 PM Briand, Sheldon <
> sheldon.bri...@nrc-cnrc.gc.ca> wrote:
>
> Thanks for getting back to me.  The time on the server is correct (it is
> to the eastern time zone).
>
>
>
> *From:* Dennis Kliban [mailto:dkli...@redhat.com]
> *Sent:* May 25, 2022 2:29 PM
> *To:* Briand, Sheldon 
> *Cc:* pulp-list@redhat.com
> *Subject:* Re: [Pulp-list] Error on Azure storage
>
>
>
> ATTENTION*** This email originated from outside of the NRC.
> ***ATTENTION*** Ce courriel provient de l'extérieur du CNRC*
>
> I don't have much experience with S3, but is it possible that the
> date/time on your Pulp server is not correct?
>
>
>
> On Wed, May 25, 2022 at 7:58 AM Briand, Sheldon <
> sheldon.bri...@nrc-cnrc.gc.ca> wrote:
>
> Hi,
>
>
>
> I’m getting an error when I’m accessing Azure storage for a AlmaLinux 8
> pulp repository.
>
>
>
> From the dnf.log:
>
> error: Status code: 403 for
> https://xx/pulp-storage/artifact/37/e8d520f7f51a02b3530d1e51
>
>
> cc1b74e9b9f10c46c358ed78cca91ddba0?se=2022-05-23T20%3A01%3A52Z=rt=2021-04-
>
> 10
>
>
>
> The above error doesn’t show it but it includes a signature (I have
> purposely put xxx in for the storage account address).
>
>
>
> If I try to manually run the above query with curl and the URL in quotes I
> get:
>
> ERROR 403: Server failed to authenticate the request. Make sure the value
> of Authorization header is formed correctly including the signature..
>
> Signed expiry time [Wed, 25 May 2022
> 09:51:09 GMT] must be after signed start time [Wed, 25 May 2022 10:00:16
> GMT]
>
>
>
> I ran a cron job which generated this error in the logs.  The cron job ran
> at Wed, 25 May 2022 09:56:00 GMT.  So to me it looks like the expiry time
> is set to a time before the request is even made and that gives me the 403
> error.  Is this a possible bug?
>
>
>
> Thanks,
>
> -Sheldon
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-list
>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Error on Azure storage

[Dennis Kliban]

What version of django-storages do you have installed? Could you try
downgrading it to 1.11.1? Don't forget to restart pulpcore-content services
after.

I found a not completely related issue[0] with django-storages newer than
1.11.1.

[0] https://github.com/jschneier/django-storages/issues/1116

On Wed, May 25, 2022 at 1:47 PM Briand, Sheldon <
sheldon.bri...@nrc-cnrc.gc.ca> wrote:

> Both VMs are synced to the same time server using chronyd.
>
>
>
> *From:* Dennis Kliban [mailto:dkli...@redhat.com]
> *Sent:* May 25, 2022 2:46 PM
> *To:* Briand, Sheldon 
> *Cc:* pulp-list@redhat.com
> *Subject:* Re: [Pulp-list] Error on Azure storage
>
>
>
> ATTENTION*** This email originated from outside of the NRC.
> ***ATTENTION*** Ce courriel provient de l'extérieur du CNRC*
>
> Perhaps the time is wrong on the machine trying to download the content?
>
>
>
> On Wed, May 25, 2022 at 1:33 PM Briand, Sheldon <
> sheldon.bri...@nrc-cnrc.gc.ca> wrote:
>
> Thanks for getting back to me.  The time on the server is correct (it is
> to the eastern time zone).
>
>
>
> *From:* Dennis Kliban [mailto:dkli...@redhat.com]
> *Sent:* May 25, 2022 2:29 PM
> *To:* Briand, Sheldon 
> *Cc:* pulp-list@redhat.com
> *Subject:* Re: [Pulp-list] Error on Azure storage
>
>
>
> ATTENTION*** This email originated from outside of the NRC.
> ***ATTENTION*** Ce courriel provient de l'extérieur du CNRC*
>
> I don't have much experience with S3, but is it possible that the
> date/time on your Pulp server is not correct?
>
>
>
> On Wed, May 25, 2022 at 7:58 AM Briand, Sheldon <
> sheldon.bri...@nrc-cnrc.gc.ca> wrote:
>
> Hi,
>
>
>
> I’m getting an error when I’m accessing Azure storage for a AlmaLinux 8
> pulp repository.
>
>
>
> From the dnf.log:
>
> error: Status code: 403 for
> https://xx/pulp-storage/artifact/37/e8d520f7f51a02b3530d1e51
>
>
> cc1b74e9b9f10c46c358ed78cca91ddba0?se=2022-05-23T20%3A01%3A52Z=rt=2021-04-
>
> 10
>
>
>
> The above error doesn’t show it but it includes a signature (I have
> purposely put xxx in for the storage account address).
>
>
>
> If I try to manually run the above query with curl and the URL in quotes I
> get:
>
> ERROR 403: Server failed to authenticate the request. Make sure the value
> of Authorization header is formed correctly including the signature..
>
> Signed expiry time [Wed, 25 May 2022
> 09:51:09 GMT] must be after signed start time [Wed, 25 May 2022 10:00:16
> GMT]
>
>
>
> I ran a cron job which generated this error in the logs.  The cron job ran
> at Wed, 25 May 2022 09:56:00 GMT.  So to me it looks like the expiry time
> is set to a time before the request is even made and that gives me the 403
> error.  Is this a possible bug?
>
>
>
> Thanks,
>
> -Sheldon
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-list
>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Error on Azure storage

[Dennis Kliban]

Perhaps the time is wrong on the machine trying to download the content?

On Wed, May 25, 2022 at 1:33 PM Briand, Sheldon <
sheldon.bri...@nrc-cnrc.gc.ca> wrote:

> Thanks for getting back to me.  The time on the server is correct (it is
> to the eastern time zone).
>
>
>
> *From:* Dennis Kliban [mailto:dkli...@redhat.com]
> *Sent:* May 25, 2022 2:29 PM
> *To:* Briand, Sheldon 
> *Cc:* pulp-list@redhat.com
> *Subject:* Re: [Pulp-list] Error on Azure storage
>
>
>
> ATTENTION*** This email originated from outside of the NRC.
> ***ATTENTION*** Ce courriel provient de l'extérieur du CNRC*
>
> I don't have much experience with S3, but is it possible that the
> date/time on your Pulp server is not correct?
>
>
>
> On Wed, May 25, 2022 at 7:58 AM Briand, Sheldon <
> sheldon.bri...@nrc-cnrc.gc.ca> wrote:
>
> Hi,
>
>
>
> I’m getting an error when I’m accessing Azure storage for a AlmaLinux 8
> pulp repository.
>
>
>
> From the dnf.log:
>
> error: Status code: 403 for
> https://xx/pulp-storage/artifact/37/e8d520f7f51a02b3530d1e51
>
>
> cc1b74e9b9f10c46c358ed78cca91ddba0?se=2022-05-23T20%3A01%3A52Z=rt=2021-04-
>
> 10
>
>
>
> The above error doesn’t show it but it includes a signature (I have
> purposely put xxx in for the storage account address).
>
>
>
> If I try to manually run the above query with curl and the URL in quotes I
> get:
>
> ERROR 403: Server failed to authenticate the request. Make sure the value
> of Authorization header is formed correctly including the signature..
>
> Signed expiry time [Wed, 25 May 2022
> 09:51:09 GMT] must be after signed start time [Wed, 25 May 2022 10:00:16
> GMT]
>
>
>
> I ran a cron job which generated this error in the logs.  The cron job ran
> at Wed, 25 May 2022 09:56:00 GMT.  So to me it looks like the expiry time
> is set to a time before the request is even made and that gives me the 403
> error.  Is this a possible bug?
>
>
>
> Thanks,
>
> -Sheldon
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-list
>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Error on Azure storage

[Dennis Kliban]

I don't have much experience with S3, but is it possible that the date/time
on your Pulp server is not correct?

On Wed, May 25, 2022 at 7:58 AM Briand, Sheldon <
sheldon.bri...@nrc-cnrc.gc.ca> wrote:

> Hi,
>
>
>
> I’m getting an error when I’m accessing Azure storage for a AlmaLinux 8
> pulp repository.
>
>
>
> From the dnf.log:
>
> error: Status code: 403 for
> https://xx/pulp-storage/artifact/37/e8d520f7f51a02b3530d1e51
>
>
> cc1b74e9b9f10c46c358ed78cca91ddba0?se=2022-05-23T20%3A01%3A52Z=rt=2021-04-
>
> 10
>
>
>
> The above error doesn’t show it but it includes a signature (I have
> purposely put xxx in for the storage account address).
>
>
>
> If I try to manually run the above query with curl and the URL in quotes I
> get:
>
> ERROR 403: Server failed to authenticate the request. Make sure the value
> of Authorization header is formed correctly including the signature..
>
> Signed expiry time [Wed, 25 May 2022
> 09:51:09 GMT] must be after signed start time [Wed, 25 May 2022 10:00:16
> GMT]
>
>
>
> I ran a cron job which generated this error in the logs.  The cron job ran
> at Wed, 25 May 2022 09:56:00 GMT.  So to me it looks like the expiry time
> is set to a time before the request is even made and that gives me the 403
> error.  Is this a possible bug?
>
>
>
> Thanks,
>
> -Sheldon
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-list
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] T/EULA for Pulp 3

[Dennis Kliban]

Pulp 3 is licensed as GPLv2[0].

[0] https://github.com/pulp/pulpcore/blob/main/LICENSE

On Tue, May 24, 2022 at 3:28 AM Crawley, Brigitte <
brigitte.craw...@insight.com> wrote:

> Hi team,
>
>
> Can you please confirm if there are T/EULA for Pulp 3?
>
>
>
> Thank you
>
>
>
> Brigitte
>
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-list
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] redis sentinel

[Dennis Kliban]

This is not currently on our road map. Now that we are no longer using RQ
for the tasking system, we are in a much better position to add support for
redis sentinel. Please file a story at https://pulp.plan.io. This is an
important feature for making Pulp fully HA.

On Wed, Oct 20, 2021 at 11:26 AM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Just wondering if there is a plan to support external redis sentinel in
> the future?
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] pulpcore 3.16.0 has been released

[Dennis Kliban]

pulpcore 3.16.0 has been released to PyPI. For more information check out
the post on Discourse.

https://discourse.pulpproject.org/t/pulpcore-3-16-0-is-generally-available/184
___
Pulp-list mailing list
Pulp-list@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulp3 migration directory

[Dennis Kliban]

All the migrations are shipped with pulpcore[0] and will be installed when
you install pulpcore 3.15 in the new virtual env. The database has a table
that keeps track of which migrations have already been applied to it. The
installer will apply only the new migrations. Did I understand your
question correctly?

[0] https://github.com/pulp/pulpcore/tree/master/pulpcore/app/migrations


On Mon, Sep 20, 2021 at 4:01 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Thanks Mike. What you described is what we plan to do. I do see the
> migration historical files. They are under ../venv/pulp/3.7 in our case. It
> is a virtual environment used by the pulp_installer. We set the
> pulp_install_dir to this directory. Since we are upgrading python to 3.8
> for pulp 3.15. We set the pulp_install_dir to a new ../venv/pulp/3.8.
> Because this is a new python install dir, I am not sure how pulp_installer
> know where to find the historical file under the old ../venv/pulp/3.7.
> Could this be an issue for the migration?
>
> From: mikedep...@redhat.com At: 09/20/21 12:21:22 UTC-4:00
> To: Bin Li (BLOOMBERG/ 120 PARK ) 
> Cc: pulp-list@redhat.com
> Subject: Re: [Pulp-list] pulp3 migration directory
>
> Hi Bin,
>
> We are trying to understand what you are asking exactly.
>
> It sounds like:
> 1. you have a test environment with a copy of the production database.
> 2. You upgraded pulp on the test environment. During this, the migrations
> were ran by pulp_installer against the test database copy.
> 3. You will run the upgrade pulp on the production environment, which will
> upgrade the production database.
>
> So we have all the possibly needed migration files on disk. They are
> static, and are part of the pypi packages (and RPM packages too.) This is a
> change from the pulpcore 3.0 alpha/beta/rc behavior, when we generated the
> migrations.
>
> If you run `find /usr/local/lib/pulp | grep migrations` , you will see all
> of them. You'll see both compiled .pyc files and the original .py files.
>
> When the installer runs, it runs all the needed migrations. This is part
> of the pulp_database_config role, the task is named "Run database
> migrations".
>
> roles/pulp_database_config/README.md still refers to the pulpcore 3.0
> alpha/beta/rc behavior when it says that it "Create and run migrations." It
> only runs them now. I will fix this doc.
>
> Does this answer your questions?
>
> -Mike
>
> On Thu, Sep 16, 2021 at 4:52 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> The 3.15 release requires python 3.8. We are currently running pulpcore
>> 3.7 with python 3.7 and an external db. When we use the pulp_installer to
>> migrate the older release to 3.15 with new python 3.8 venv,it run
>> successfully in a test environment. I am wondering how the new version
>> knows the the historical migration file. Should I copy the historical
>> migration files from older python 3.7 venv to the new 3.8 venv first?
>> ___
>> Pulp-list mailing list
>> Pulp-list@redhat.com
>> https://listman.redhat.com/mailman/listinfo/pulp-list
>
>
>
> --
>
> Mike DePaulo
>
> He / Him / His
>
> Service Reliability Engineer, Pulp
>
> Red Hat 
>
> IM: mikedep333
>
> GPG: 51745404
> 
>
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] content only server

[Dennis Kliban]

On Tue, Sep 21, 2021 at 4:29 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Does anyone know if there is an option to only enable the content server
> in the pulp_installer? I think I could skip pulp_database and
> pulp_database_config and disable systemd api, worker, redis service once it
> is installed. Just wondering if there is a better way to do this. The
> content only server will share the storage and db with the primary server.
> We are planning to have multiple servers to distribute the load and achieve
> high availability.
>

You should follow the instructions here[0]. There is an example playbook
that only runs the `pulp_content` role[1] (which depends on the
`pulp_common` role).

[0]
https://docs.pulpproject.org/pulp_installer/customizing/#separate-servers-for-each-and-every-service
[1] https://docs.pulpproject.org/pulp_installer/roles/pulp_content/


> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] Pulp File 1.8.1 is Generally Available

[Dennis Kliban]

pulp_file 1.8.1 is now available on PyPI.

This release contains a fix related to packaging.

This release is compatible with pulpcore 3.14.

PyPI: https://pypi.org/project/pulp-file/1.8.1/
Changelog: https://docs.pulpproject.org/pulp_file/en/1.8.1/changes.html
Docs: https://docs.pulpproject.org/pulp_file/en/1.8.1/
Python bindings: https://pypi.org/project/pulp-file-client/1.8.1/
Ruby bindings: https://rubygems.org/gems/pulp_file_client/versions/1.8.1/
___
Pulp-list mailing list
Pulp-list@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulp 3.7.3 aarch64 repo

[Dennis Kliban]

I don't see anything special about these repos. Are you still experiencing
this problem?


On Wed, Jun 16, 2021 at 5:00 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Tried to download aarch64 repo and got the errors below:
>
> "description": "",
> "traceback": " File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\",
> line 886, in perform_job\n rv = job.perform()\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\",
> line 664, in perform\n self._result = self._execute()\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\",
> line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py\",
> line 266, in synchronize\n dv.create()\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py\",
> line 148, in create\n loop.run_until_complete(pipeline)\n File
> \"/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py\", line 584, in
> run_until_complete\n return future.result()\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\",
> line 225, in create_pipeline\n await asyncio.gather(*futures)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\",
> line 43, in __call__\n await self.run()\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
> line 152, in run\n pb.done += task.result() # download_count\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
> line 178, in _handle_content_unit\n await
> asyncio.gather(*downloaders_for_content)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py\",
> line 88, in download\n download_result = await
> downloader.run(extra_data=self.extra_data)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py\",
> line 227, in run\n return await self._run(extra_data=extra_data)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py\",
> line 87, in _run\n async with self.session.get(url, proxy=self.proxy,
> auth=self.auth) as response:\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\",
> line 1012, in __aenter__\n self._resp = await self._coro\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\",
> line 504, in _request\n await resp.start(conn)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py\",
> line 847, in start\n message, payload = await self._protocol.read() # type:
> ignore # noqa\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py\",
> line 591, in read\n await self._waiter\n
>
> The same error happened on both remote
> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/baseos/os and
> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os
>
> Please advise.
>
> Thanks
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] pulpcore 3.9.1 is generally available

[Dennis Kliban]

Pulpcore 3.9.1 has been released to PyPI[0]. Client libraries are available
on PyPI[1] and rubygems.org[2]. This release addresses a single bug that
was encountered by users that set a custom CHUNKED_UPLOAD_DIR setting. The
value for this setting must now be a relative path inside MEDIA_ROOT[3].

# Installation and Upgrade

Users should use the 3.9.1 release of pulp_installer[4] to install or
upgrade their installations. This version of the installer will check
compatibility of all installed plugins with pulpcore 3.9. The installer
will abort if any plugin is incompatible.

The pulp_installer collection can be installed from Ansible Galaxy with the
following command:

ansible-galaxy collection install --force pulp.pulp_installer

The --force flag will upgrade the collection if you had a previous version
installed.

[0] https://pypi.org/project/pulpcore/3.9.1/
[1] https://pypi.org/project/pulpcore-client/3.9.1/
[2] https://rubygems.org/gems/pulpcore_client/versions/3.9.1
[3] https://docs.pulpproject.org/pulpcore/en/3.9.1/changes.html#id1
[4] https://galaxy.ansible.com/pulp/pulp_installer
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] mirrored sync in 3.7.3

[Dennis Kliban]

We have historically treated the revision number as required. Please file
an issue at https://pulp.plan.io/issues/new/ and we will investigate what
the behavior should be in this situation.

On Wed, Jan 13, 2021 at 6:06 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> I noticed the repomd.xml on artifactory is a little different (see below).
> This is why pulp doesn't sync when optimize is true. Is the revision
> optional field in repomd.xml?
>
> 
> http://linux.duke.edu/metadata/repo; xmlns:rpm="
> http://linux.duke.edu/metadata/rpm;>
> 
>  href="repodata/deddaac9159b0dd137637e3f054699ed835c2188-other.xml.gz"/>
>  pkgid="YES">deddaac9159b0dd137637e3f054699ed835c2188
> 55565
> 1610569598
>  pkgid="YES">440b3887bd333a9f198daf8d11d883c39b9cfa6b
> 163364
> 
> 
> 
>  href="repodata/2ab424ed2e5cb27ded110002678a64efe61fdc6c-filelists.xml.gz"/>
>  pkgid="YES">2ab424ed2e5cb27ded110002678a64efe61fdc6c
> 722
> 1610569598
>  pkgid="YES">7b21b424589b3d0fcbfb5e78b5eb21441a739dd5
> 3064
> 
> 
> 
>  href="repodata/18af240dc9319adc1fc746da197f195d35d3fca6-primary.xml.gz"/>
>  pkgid="YES">18af240dc9319adc1fc746da197f195d35d3fca6
> 1724
> 1610569599
>  pkgid="YES">49ad32d347e2bde51dd070085cbe145b0146faf4
> 9708
> 
> 
>
>
> From: dkli...@redhat.com At: 01/13/21 14:36:27
> To: Bin Li (BLOOMBERG/ 120 PARK ) 
> Cc: pulp-list@redhat.com
> Subject: Re: [Pulp-list] mirrored sync in 3.7.3
>
> The sync task examines the repomd.xml and the revision number in it. All
> the logic for optimizing the sync is here[0].
>
> [0]
> https://github.com/pulp/pulp_rpm/blob/3.7/pulp_rpm/app/tasks/synchronizing.py#L169-L175
>
> On Wed, Jan 13, 2021 at 1:45 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> Happy new year.
>> I run a mirrored sync with upstream. The sync didn't pick up new rpms in
>> upstream repo. I rerun sync with optimize=false and it synced all rpms. I
>> do see the file names from new rpms are in the filelists.xml.gz of the
>> upstream repodata. Just wondering which repo data does pulp check before
>> starting a optimized mirror sync and why the optimized sync didn't work?
>> ___
>> Pulp-list mailing list
>> Pulp-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] mirrored sync in 3.7.3

[Dennis Kliban]

The sync task examines the repomd.xml and the revision number in it. All
the logic for optimizing the sync is here[0].

[0]
https://github.com/pulp/pulp_rpm/blob/3.7/pulp_rpm/app/tasks/synchronizing.py#L169-L175

On Wed, Jan 13, 2021 at 1:45 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Happy new year.
> I run a mirrored sync with upstream. The sync didn't pick up new rpms in
> upstream repo. I rerun sync with optimize=false and it synced all rpms. I
> do see the file names from new rpms are in the filelists.xml.gz of the
> upstream repodata. Just wondering which repo data does pulp check before
> starting a optimized mirror sync and why the optimized sync didn't work?
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] How to create issues?

[Dennis Kliban]

Thank you for your feedback about the experience for someone trying to
participate in the Pulp community for the first time. I also appreciate
your offer to help with integrating our Redmine instance with GitHub
issues. Instead of keeping issues in two places, we would like to
eventually migrate to using GitHub issues exclusively.

With regard to the issue that you discovered with pulp_container: anyone
can create an account on https://pulp.plan.io. It would be great if you
could open an account and file an issue there[0]. I promise to prioritize a
fix for it.

[0] https://pulp.plan.io/projects/pulp_container/issues/new

Thank you,
Dennis


On Mon, Dec 21, 2020 at 2:12 PM Tobias Hochgürtel <
tobias.hochguer...@googlemail.com> wrote:

> Hey David,
>
> Yeah I tried pulp 3 - and tried to create a container Registry with adding
> a remote to the repository which created a new repository version which I
> distributed.
>
> But I wasn’t able to pull any image over this distributed - remote
> repository to docker hub. I tried to figure out why and what is wrong for a
> couple of hours but I wasn’t able to see the origin of the issue in this
> time.
> I decided then to give it up to work with pulp 3. I tried them at least to
> create an issue but I wasn’t really clear where and how ...
>
> I figured out - over a open pull request, that the stories are planned in
> plan.io. To find the plan.io instance for pulp over a pull request ... is
> also a bit “hidden” and not so really easy to find for some one - I think
> in terms of that it’s like - yeah we are open for the community but - it’s
> difficult.
>
> I wasn’t sure if I can register there for free and then create an issue -
> it was also looking more internally as that this project is open for the
> community yet. I think there is a company behind pulp 3 which powers the
> development.
>
> But here later on the day... I played around with “Harbor” (a competitiver
>  and had the same issue in the end with Harbor as I had with Pulp 3 and the
> pulp_container plugin.
>
> I found also a solution / or explaining for the issue which I had with
> pulp3 and harbor. The origin is that docker hub has changed something in
> their api, and that isn’t yet implemented in pulp 3 and harbor (and I thing
> also in other container Registry solutions which are open source...)
>
> I want to share for today the issue link to GitHub, where I added a
> comment and a reference to pulp_container project on GitHub.
>
> https://github.com/goharbor/harbor/issues/13740
>
>
>
>
> The issue is “closed” - but that isn’t reflecting the current state of
> work - because you see there that they working on a solution after clothing
> the ticket - I think this is a mistake. And they had not the time to open
> it again. But check this and speak internally about it? Or maybe it’s
> already a known issue?
>
> I think what you need in this project - when the discussion ended with no
> result / decision, is someone who develops a bridge between GitHub issues
> and plan.io (what is originally Redmine). I know redmine, because I
> worked with it in my career.
>
> I develop myself often missing integration “bridge” tools to combine two
> tools to a great solution.
>
> Hope this helps? I would like to have a better integration of GitHub
> issues for pulp3 because that makes it bit more friendly and warm welcome
> for contributions from the outside.
>
> Speak to me when a integration bridge between plan.io and GitHub sounds
> as a interesting solution. I mean when your devs can stay at plan.io and
> get the things from GitHub synced - also when they add in plan.io a
> comment to the mirrored issues from GitHub, these comments get synced back
> to GitHub... then nobody will really have a gap - and not  notice from
> where the issue is, but both worlds can communicate.
>
> Kind regards,
> Tobias
>
>
>
>
>
>
> David Davis  schrieb am Mo. 21. Dez. 2020 um 14:24:
>
>> Tobias,
>>
>> Glad to hear you are enjoying Pulp 3. We use https://pulp.plan.io/ to
>> track our issues. For the pulp_container project, you can report an issue
>> at https://pulp.plan.io/projects/pulp_container/issues/new (must be
>> signed up and logged in).
>>
>> Your feedback regarding using Github to track issues is helpful. We
>> discussed doing so a few months ago but the discussion died due to other
>> higher priorities. I'm hopeful though we may have time to look at using
>> Github Issues again in 2021.
>>
>> Thank you.
>>
>> David
>>
>>
>> On Sun, Dec 20, 2020 at 10:00 PM Tobias Hochgürtel <
>> tobias.hochguer...@googlemail.com> wrote:
>>
>>> I tried Pulp 3 via the container which the project provides. But I  got
>>> some issues and tried to post a bug report like always on GitHub - but I
>>> can’t.
>>>
>>> Is there a different location?
>>>
>>> I have issues with the pulp_container plugin.
>>>
>>> I like the thin, and clear architecture of pulp 3 ;) the docs are very
>>> great written.
>>>
>>> But the development is not so transparent for 

[Pulp-list] pulp_installer 3.9.0-1 is available

[Dennis Kliban]

pulp_installer 3.9.0-1 is available on Ansible Galaxy[0]. It can be
installed with the following command

ansible-galaxy collection install pulp.pulp_installer:3.9.0-1

This release contains a bug fix that prevented users from installing Pulp
on CentOS 8.3 and CentOS Stream[1].

[0] https://galaxy.ansible.com/pulp/pulp_installer
[1] https://pulp-installer.readthedocs.io/en/latest/CHANGES/#bugfixes
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] pulp_file 1.4.0 and 1.5.0 are now available

[Dennis Kliban]

pulp_file 1.4.0 is compatible with pulpcore 3.7 - 3.9

pulp_file 1.5.0 is compatible with pulpcore 3.7 - 3.10 (when 3.10 is
released)

There are no significant changes[0]. These releases are mostly
compatibility releases for pulpcore 3.9 and 3.10.

[0] https://docs.pulpproject.org/pulp_file/changes.html
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] sync issue on 3.7.3

[Dennis Kliban]

I suspect that the proxy is what is making the difference here.

However, it does look like the two systems are using different versions of
'aiohttp', the library Pulp uses to perform downloads. Try downgrading
'aiohttp' to 3.6.2 on your 3.7 system and see if that resolves the issue.

On Mon, Nov 2, 2020 at 2:18 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Thanks. Here is the info
>
> On Test Server with 3.7.3
> # pip freeze
> aiodns==2.0.0
> aiofiles==0.6.0
> aiohttp==3.7.2
> ansible==2.10.1
> ansible-base==2.10.2
> async-timeout==3.0.1
> attrs==20.2.0
> backoff==1.10.0
> certifi==2020.6.20
> cffi==1.14.3
> chardet==3.0.4
> click==7.1.2
> coreapi==2.3.3
> coreschema==0.0.4
> createrepo-c==0.16.0.post1
> cryptography==3.2.1
> defusedxml==0.6.0
> diff-match-patch==20200713
> distro==1.5.0
> Django==2.2.16
> django-currentuser==0.5.1
> django-extensions==3.0.9
> django-filter==2.3.0
> django-guardian==2.3.0
> django-import-export==2.3.0
> django-lifecycle==0.8.0
> django-readonly-field==1.0.5
> djangorestframework==3.11.2
> djangorestframework-queryfields==1.0.0
> drf-access-policy==0.7.0
> drf-nested-routers==0.91
> drf-spectacular==0.9.13
> drf-yasg==1.17.1
> dynaconf==3.1.2
> et-xmlfile==1.0.1
> gunicorn==20.0.4
> idna==2.10
> importlib-metadata==2.0.0
> inflection==0.5.1
> itypes==1.2.0
> jdcal==1.4.1
> Jinja2==2.11.2
> jsonschema==3.2.0
> libcomps==0.1.15.post1
> MarkupPy==1.14
> MarkupSafe==1.1.1
> modulemd==1.3.3
> multidict==5.0.0
> nose==1.3.7
> odfpy==1.4.1
> openpyxl==3.0.5
> packaging==20.4
> pip-tools==5.3.1
> productmd==1.28
> psycopg2==2.8.6
> pulp-file==1.3.0
> pulp-file-client==1.3.0
> pulp-rpm==3.7.0
> pulp-rpm-client==3.7.0
> pulpcore==3.7.3
> pulpcore-client==3.7.3
> pyasn1==0.4.8
> pyasn1-modules==0.2.8
> pycairo==1.20.0
> pycares==3.1.1
> pycparser==2.20
> PyGObject==3.38.0
> pygtrie==2.3.3
> pyparsing==2.4.7
> pyrsistent==0.17.3
> python-box==5.2.0
> python-dateutil==2.8.1
> python-dotenv==0.15.0
> python-gnupg==0.4.6
> python-ldap==3.3.1
> pytz==2020.1
> PyYAML==5.3.1
> redis==3.5.3
> requests==2.24.0
> rq==1.3.0
> ruamel.yaml==0.16.12
> ruamel.yaml.clib==0.2.2
> scikit-build==0.11.1
> six==1.15.0
> solv==0.7.14
> sqlparse==0.4.1
> tablib==2.0.0
> toml==0.10.1
> typing-extensions==3.7.4.3
> uritemplate==3.0.1
> urllib3==1.25.11
> urlman==1.3.0
> whitenoise==5.0.1
> xlrd==1.2.0
> xlwt==1.3.0
> yarl==1.6.2
> zipp==3.4.0
>
> On Prod server running 3.3
> aiodns==2.0.0
> aiofiles==0.5.0
> aiohttp==3.6.2
> ansible==2.9.9
> async-timeout==3.0.1
> attrs==19.3.0
> autopep8==1.5.3
> backoff==1.10.0
> certifi==2020.4.5.1
> cffi==1.14.0
> chardet==3.0.4
> click==7.1.2
> coreapi==2.3.3
> coreschema==0.0.4
> createrepo-c==0.15.10
> cryptography==2.9.2
> defusedxml==0.6.0
> diff-match-patch==2018
> distro==1.5.0
> Django==2.2.12
> django-filter==2.2.0
> django-import-export==2.0.2
> djangorestframework==3.10.3
> djangorestframework-queryfields==1.0.0
> drf-nested-routers==0.91
> drf-yasg==1.17.1
> dynaconf==3.0.0rc1
> et-xmlfile==1.0.1
> gunicorn==20.0.4
> idna==2.9
> importlib-metadata==1.6.0
> inflection==0.4.0
> itypes==1.2.0
> jdcal==1.4.1
> Jinja2==2.11.2
> jsonschema==3.2.0
> libcomps==0.1.14.post1
> MarkupPy==1.14
> MarkupSafe==1.1.1
> modulemd==1.3.3
> multidict==4.7.6
> nose==1.3.7
> odfpy==1.4.1
> openpyxl==3.0.3
> packaging==20.4
> pip-tools==5.2.0
> productmd==1.26
> psycopg2==2.8.5
> pulp-file==0.3.0
> pulp-file-client==0.3.0
> pulp-rpm==3.3.2
> pulp-rpm-client==3.3.2
> pulpcore==3.3.1
> pulpcore-client==3.3.1
> pyasn1==0.4.8
> pyasn1-modules==0.2.8
> pycairo==1.19.1
> pycares==3.1.1
> pycodestyle==2.6.0
> pycparser==2.20
> PyGObject==3.36.1
> pygtrie==2.3.3
> pyparsing==2.4.7
> pyrsistent==0.16.0
> python-box==4.2.3
> python-dateutil==2.8.1
> python-dotenv==0.13.0
> python-gnupg==0.4.6
> python-ldap==3.2.0
> pytz==2020.1
> PyYAML==5.3.1
> redis==3.5.2
> requests==2.23.0
> rq==1.3.0
> ruamel.yaml==0.16.10
> ruamel.yaml.clib==0.2.0
> scikit-build==0.11.1
> six==1.15.0
> solv==0.7.11
> sqlparse==0.3.1
> tablib==2.0.0
> toml==0.10.1
> uritemplate==3.0.1
> urllib3==1.25.9
> whitenoise==5.0.1
> xlrd==1.2.0
> xlwt==1.3.0
> yarl==1.4.2
> zipp==3.1.0
>
> They are on different proxy host. I am asking network team to check proxy
> also.
>
>
> From: dkli...@redhat.com At: 11/02/20 14:10:53
> To: Bin Li (BLOOMBERG/ 120 PARK ) 
> Cc: pulp-list@redhat.com
> Subject: Re: [Pulp-list] sync issue on 3.7.3
>
> Are both systems downloading via the same proxy?
>
> Could you share the output of 'pip freeze' from your 3.7.3 install and one
> from your 3.3 install?
>
> On Mon, Nov 2, 2020 at 11:13 AM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> We have been seeing a few sync issues during test of 3.7.3. Below is a
>> list of summary of repo name and errors. We were able to resolve some
>> issues by rerun the sync but most of errors seems to be consistent. The
>> test was done on a 4cpu 64GB host with 2 workers configured. We are running
>> 3.3 in our production 

Re: [Pulp-list] sync issue on 3.7.3

[Dennis Kliban]

Are both systems downloading via the same proxy?

Could you share the output of 'pip freeze' from your 3.7.3 install and one
from your 3.3 install?

On Mon, Nov 2, 2020 at 11:13 AM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> We have been seeing a few sync issues during test of 3.7.3. Below is a
> list of summary of repo name and errors. We were able to resolve some
> issues by rerun the sync but most of errors seems to be consistent. The
> test was done on a 4cpu 64GB host with 2 workers configured. We are running
> 3.3 in our production without any of these issue. Just wondering if anyone
> experienced something similar and how we troubleshoot.
>
>
> rhel-7-server-optional-rpms
> Response payload is not completed
> =
> rhel-7-server-rpms
> Cannot connect to host cdn.redhat.com:443 ssl:default [Connection reset
> by peer]
> =
> rhel-7-server-debug-rpms
> Cannot connect to host cdn.redhat.com:443 ssl:default [None]
> =
> rhel-7-server-source-rpms
> [Errno 104] Connection reset by peer
> =
> rhel-server-rhscl-7-rpms
> Response payload is not completed
> =
> rhel-7-server-optional-debug-rpms
> Response payload is not completed
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] pulpcore 3.7.3 and pulp-2to3-migration 0.5.1 are generally available

[Dennis Kliban]

pulpcore 3.7.3 is available on PyPI[0]. This release includes a fix for a
bug that could silently delete an Artifact from the filesystem[1]. This bug
was introduced in 3.7.0 and was most prominently experienced during
migration of content from Pulp 2 to 3.

pulp-2to3-migration 0.5.1 is available on PyPI[2]. This release includes a
fix for a serious issue related to migration of RPM content[3]. Prior to
this release, the RPM content metadata was migrated incorrectly. Users that
have migrated RPM content using pulp-2to3-migration prior to 0.5.1 will
need to remove the migrated RPM content from their Pulp 3 system and run
the migration again using pulp-2to3-migration 0.5.1.

pulp_installer 3.7.3 should be used to upgrade to pulpcore 3.7.3[4].

[0] https://pypi.org/project/pulpcore/3.7.3/
[1] https://docs.pulpproject.org/pulpcore/en/3.7/changes.html#id1
[2] https://pypi.org/project/pulp-2to3-migration/0.5.1/
[3] https://pulp-2to3-migration.readthedocs.io/en/latest/changes.html#id1
[4] https://galaxy.ansible.com/pulp/pulp_installer
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Pulp3 published content change

[Dennis Kliban]

This is currently not possible with Pulp 3. Please file a feature request
at https://pulp.plan.io/issues/new/.

I believe such a feature is possible and not very difficult to implement.
The hardest part would be deciding which date to show.

Date when the Content was created in Pulp
Date when the Content was added to a Repository Version.
Date when a Publication was created from the Repository Version.
Date for when the Publication was associated with the Distribution.

I believe the most appropriate date would be the creation date for the
Publication.

On Tue, Oct 27, 2020 at 3:21 PM Chris Taylor (chtaylo2) 
wrote:

> In Pulp2, yum_distributor created a directory tree structure under
> “/var/lib/pulp/published/yum/https/repos/”.   I exposed this to clients via
> Apache, using directory indexing and while this allowed for custom pages,
> more importantly, allowed easy page navigation and viewing file dates.
>
>
>
> In Pulp3, it appears manual https navigation of content is dynamically
> generated using gunicorn and only shows the file names.
>
>
>
> Any recommendations to have a similar look/feel for clients manually
> navigating content in a browser?
>
>
>
> Thanks
>
> Chris
>
>
>
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] 2to3migration fails on 'Package' object has no attribute '_remote_artifact_saver_cas'

[Dennis Kliban]

I continued the investigation and ended up filing a bug against Django[0].
I updated my initial patch to reflect the deeper understanding of the
problem. The latest patch should generate fewer queries to the DB[1] during
migration from Pulp 2 to 3.

[0] https://code.djangoproject.com/ticket/32089
[1] https://github.com/pulp/pulpcore/pull/937

On Tue, Sep 29, 2020 at 9:58 AM Daniel Alley  wrote:

> The createrepo_c issue is a packaging problem, our Python packages include
> the correct patch but the RPMs don't have the patch applied.  We're going
> to get this fixed within the next day or two.
>
> On Mon, Sep 28, 2020 at 3:00 AM Winberg Adam  wrote:
>
>> Thanks! That's a very nice patch, after trying for a week to get the
>> migration to work and continously searching for and deleting duplicate
>> pulp2 content, the migration worked right away after this patch. :)
>>
>>
>> Or, at least regarding the duplicate content errors, however I still have
>> problems with a createrepo_c error, same as described in
>> https://pulp.plan.io/issues/7193 (huge input lookup).
>>
>>
>> That error only arise for a few packages so that is manageable for me to
>> purge from pulp2 before migration, but it would be nice to not have to do
>> that..
>>
>>
>> //Adam
>>
>>
>>
>>
>>
>> --
>> *From:* Dennis Kliban 
>> *Sent:* 27 September 2020 17:35
>> *To:* Winberg Adam
>> *Cc:* Ina Panova; pulp-list@redhat.com
>> *Subject:* Re: [Pulp-list] 2to3migration fails on 'Package' object has
>> no attribute '_remote_artifact_saver_cas'
>>
>> Here is a patch for pulpcore that fixes this problem[0].
>>
>> [0] https://github.com/pulp/pulpcore/pull/937
>>
>> On Mon, Sep 21, 2020 at 7:17 AM Winberg Adam 
>> wrote:
>>
>>> Thank you for your reply - yes I did clean orphans after i removed the
>>> pulp2 repos that i suspected might be the cause. But as you say, there
>>> might be some other cause for this in my case.
>>>
>>>
>>> At the moment I have adjusted the code so the iteration only runs if the
>>> '_remote_artifact_saver_cas' attribute is present. Don't know if the
>>> result will be any good though, running the migration right now.
>>>
>>>
>>> //Adam
>>>
>>>
>>> --
>>> *From:* Ina Panova 
>>> *Sent:* 21 September 2020 13:09
>>> *To:* Winberg Adam
>>> *Cc:* pulp-list@redhat.com
>>> *Subject:* Re: [Pulp-list] 2to3migration fails on 'Package' object has
>>> no attribute '_remote_artifact_saver_cas'
>>>
>>> Hi,
>>> the provided steps in the mentioned issue are the steps to reproduce the
>>> issue, however, unfortunately, this does not necessarily mean that this is
>>> the root cause of the manifested problem.
>>> Apparently we need to find a fix to properly handle duplicated
>>> declarative content in a batch.
>>>
>>> Looking at the steps you have tried to bypass the issue, have you run
>>> orphan clean up after pulp2 repos removal?
>>>
>>>
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Senior Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>>
>>>
>>> On Sun, Sep 20, 2020 at 9:05 AM Winberg Adam 
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>> When running the 2to3migration for the 'rpm' plugin, I get the
>>>> following error:
>>>>
>>>>
>>>> AttributeError: 'Package' object has no attribute
>>>> '_remote_artifact_saver_cas'
>>>>
>>>>
>>>>
>>>> This is the same as specified in https://pulp.plan.io/issues/7147, and
>>>> I actually had a couple of repos in pulp2 which shared the same feed. I
>>>> removed the redundant repos, flushed the pulp3 db and reran the
>>>> 2to3-migration but still got stuck on the same error.
>>>>
>>>>
>>>> Anyone got any pointers how to resolve this?
>>>>
>>>>
>>>> //Adam
>>>> ___
>>>> Pulp-list mailing list
>>>> Pulp-list@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>
>>> ___
>>> Pulp-list mailing list
>>> Pulp-list@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>> ___
>> Pulp-list mailing list
>> Pulp-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] namespacing base_path of Distributions

[Dennis Kliban]

The container plugin is introducing a 'namespace' resource that will be
associated with Distributions. The name of the namespace will be used as
the first part of the base_path for the ContainerDistribution. Role based
access control will allow users to create content under their own namespace
and any other namespace that they have permission to push container images
to.

Would a similar feature be useful in any other plugins?
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Can't get CentOS 7 Updates repo published

[Dennis Kliban]

Filelists are stored in the database in a compressed form[0]. During
publishing of metadata, each fillist is decompressed before being written
to a file[1]. It's hard to speculate when data corruption occurred. It's
possible that the data was corrupted in the remote repository the first
time that this RPM was downloaded. That metadata could have been fixed
since then, but Pulp thinks it already has the information and is not
trying to re-download.

[0]
https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L1051
[1]
https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L1064

On Fri, Oct 2, 2020 at 3:27 AM Konstantin M. Khankin <
khankin.konstan...@gmail.com> wrote:

> Sorry for reviving this thread but I think I found something.
>
> Today I tried to install CentOS 7 and got an error:
> "TypeError: Parsing filelists.xml error: expected '>'"
>
> So I did this:
>
> >>> import xml.etree.ElementTree as ET
> >>> tree =
> ET.parse('/mnt/sysimage/var/tmp/yum.cache/centos_updates/gen/filelists.xml')
> ...
>   File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1506, in
> _raiseerror
> raise err
> xml.etree.ElementTree.ParseError: not well-formed (invalid token): line
> 1884123, column 136
>
> I opened filelists.xml and found that line 1884123 is indeed corrupt and
> looks like this:
>
> /usr/share/javadoc/java-1.8.0-openjdk-1.8.0.111-1.b15.el7_2-debug/api/javax/sql/rowset/serial/compact3-package-frame.html   !,^@ 0
> 480!share/javadoc/java-1.8.0-openjdk-1.8.0.111-1.b15.el7_2-debug/api/javax/sql/rowset/serial/compact3-package-summary.html
>
> I verified another consumer has exactly the same issue with this file.
>
> Do I understand correctly Pulp generates filelists.xml? If so, can it be
> because of a bug in Pulp or should I look for a silent data corruption
> issue on my pulp server?
>
> Thanks!
>
>
> вт, 1 сент. 2020 г. в 17:14, Dennis Kliban :
>
>> The repoview feature requires the sqlite db to be generated. This
>> feature generates HTML pages for browsing the repository in a web browser.
>> I believe that users are still able to browse the repository without it,
>> but in that case the HTML listing of directories is generated by the web
>> server each time a user requests it.
>>
>> On Tue, Sep 1, 2020 at 4:53 AM Konstantin M. Khankin <
>> khankin.konstan...@gmail.com> wrote:
>>
>>> Thank you, that obviously helped. Still would be useful to know the root
>>> cause.
>>>
>>> OTOH, if sqlite creation functionality is not that critical - why would
>>> we ever enable something which only consumes time during sync for no
>>> benefit?
>>>
>>> вт, 1 сент. 2020 г. в 00:26, Dennis Kliban :
>>>
>>>> Looks like everything is up to date. I have no idea what the root cause
>>>> is, but according to this comment[0], you can work around the problem by
>>>> disabling the generation of sqlite db. I am not sure what the exact effect
>>>> of this will be for the clients that are consuming this content, but the
>>>> repository will be usable.
>>>>
>>>> [0] https://pulp.plan.io/issues/2019#note-2
>>>>
>>>> On Mon, Aug 31, 2020 at 4:10 PM Konstantin M. Khankin <
>>>> khankin.konstan...@gmail.com> wrote:
>>>>
>>>>> # rpm -qa | grep pulp | sort
>>>>> pulp-admin-client-2.21.3-1.el7.noarch
>>>>> pulp-agent-2.21.3-1.el7.noarch
>>>>> pulp-consumer-client-2.21.3-1.el7.noarch
>>>>> pulp-deb-admin-extensions-1.10.1-1.el7.noarch
>>>>> pulp-deb-plugins-1.10.1-1.el7.noarch
>>>>> pulp-docker-admin-extensions-3.2.6-1.el7.noarch
>>>>> pulp-docker-plugins-3.2.6-1.el7.noarch
>>>>> pulp-puppet-consumer-extensions-2.21.3-1.el7.noarch
>>>>> pulp-puppet-handlers-2.21.3-1.el7.noarch
>>>>> pulp-python-admin-extensions-2.0.4-1.el7.noarch
>>>>> pulp-python-plugins-2.0.4-1.el7.noarch
>>>>> pulp-rpm-admin-extensions-2.21.3-1.el7.noarch
>>>>> pulp-rpm-consumer-extensions-2.21.3-1.el7.noarch
>>>>> pulp-rpm-handlers-2.21.3-1.el7.noarch
>>>>> pulp-rpm-plugins-2.21.3-1.el7.noarch
>>>>> pulp-rpm-yumplugins-2.21.3-1.el7.noarch
>>>>> pulp-selinux-2.21.3-1.el7.noarch
>>>>> pulp-server-2.21.3-1.el7.noarch
>>>>> python-pulp-agent-lib-2.21.3-1.el7.noarch
>>>>> python-pulp-bindings-2.21.3-1.el7.noarch
>>>>> python-pulp-client-lib-2.21.3-1.el7.noarch
>

[Pulp-list] pulpcore 3.7.1 is Generally Available

[Dennis Kliban]

Pulpcore 3.7.1 is now available[0]. This release contains a bug fix related
to packaging. For a list of all changes, please check the changelog for
pulpcore[1].

# Installation and Upgrade

Users should use the 3.7.1 release of pulp_installer[2] to install or
upgrade their installations. This version of the installer will check
compatibility of all installed plugins with pulpcore 3.7. The installer
will abort if any plugin is incompatible.

The pulp_installer collection can be installed from Ansible Galaxy with the
following command:

ansible-galaxy collection  install --force pulp.pulp_installer

The --force flag will upgrade the collection if you had a previous version
installed.


[0] https://pypi.org/project/pulpcore/3.7.1/
[1] https://docs.pulpproject.org/pulpcore/en/3.7.1/changes.html#id1
[2] https://galaxy.ansible.com/pulp/pulp_installer
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Pulp3 Automation

[Dennis Kliban]

On Tue, Sep 29, 2020 at 2:26 PM Dennis Kliban  wrote:

> On Tue, Sep 29, 2020 at 1:37 PM Maarten Beeckmans 
> wrote:
>
>> On 2020-09-29 18:22, Dennis Kliban wrote:
>>
>> On Tue, Sep 29, 2020 at 10:30 AM Maarten Beeckmans 
>> wrote:
>>
>>> We are trying to automate our Pulp3 setup the same way as our Pulp2
>>> setup.
>>>
>>> We have the following use cases in our Pulp2 setup:
>>>
>>> a) We create mirrors and repositories automatically for every
>>> environment (using puppet and the API). The repositories are defined in a
>>> yaml configuration file (Puppet Hiera).
>>>   - We have several problems for doing this in Pulp3. It's very complex
>>> to automatically create mirrors (ex. centos-8-appstream) for the
>>> development environment (that are in sync with the upstream repositories).
>>> A possible solution for this problem would be to add a 'latest' flag to
>>> publications and distributions so that they are always pointing to the
>>> latest version of the repository.
>>>   - It is possible to create the repositories with scripts, but that's
>>> specific and quite hacky. It's also not possible to automatically update
>>> the publications and distributions this way. We would need to keep a local
>>> copy of what distribution maps to what repository, but that's not a good
>>> idea. Why keep a local copy of the mapping if that's present in pulp.
>>>
>>
>> We are already planning to improve this workflow. A single API call will
>> create a task that will sync a repo, create a new publication, and update
>> the distribution. We are first going to introduce the change in pulp_file
>> and then other plugins will follow. https://pulp.plan.io/issues/7469
>>
>> This issue seems to be solving my issue.
>>
>>
>>
>>> b) We upload packages from one repository (that is not served to
>>> clients) to another repository based on lists in yaml (cherry picking), or
>>> straight from a (jenkins) Pipeline. This is done by the pulp-admin commands.
>>>   - cherry picking could be done by copying content from one repository
>>> to another using the copy API call, but we want to automatically expose
>>> those changes to the development environment (promotion can be done as in
>>> c).
>>>   - Uploading content isn't an issue because this is possible by
>>> converting the pulp-admin commands to api calls.
>>>
>>
>> You want to be able to upload a package and have that operation create a
>> new publication and update the distribution. This would be similar to the
>> story that I linked above for syncing - except for the upload use case.  Am
>> I understanding correctly?
>>
>> Yes, that's what we want. If we create an rpm itself, we want that's it
>> directly available in the (development) repository so we don't have to
>> create a new publication and update the distribution. For adding it to
>> production, we would promote the repository as described in c. Same when
>> copying packages between repositories.
>>
>
> I definitely see the utility in making the content available to the users
> right away. Please file a story at https://pulp.plan.io/issues/new/ so we
> have this use captured. This use case should be considered also when
> implementing #7469.
>
>
>>
>>
>>> c) We want to be able to 'promote' a set of repositories from one
>>> environment to another. We are using a script that is generated by puppet
>>> when creating the repositories in a).
>>>   For 'promoting' a set of repositories to one environment to another
>>> environment, this is explained
>>> https://docs.pulpproject.org/pulpcore/workflows/promotion.html.
>>>
>>>
>> I don't understand what challenge you are describing in C.
>>
>> Now we have different repositories for development and production. We
>> want to be able to easily promote repositories from development to
>> production (with archiving, the production repository keeps X older
>> versions of packages), this may be done with one command in the cli
>> interface or different commands that can be scripted. This is somewhat
>> explained in the docs, but it's very high level. Maybe it's a good idea to
>> add an example in the rpm docs (with the promoting of an CentOS repository)?
>>
>
> I agree that we need more docs around this. Can you please file an issue
> about this also?
>
> There are two ways to accomplish this.
>
> 1) Create a distribution for each environment. In this case whenever you
>

Re: [Pulp-list] Pulp3 Automation

[Dennis Kliban]

On Tue, Sep 29, 2020 at 1:37 PM Maarten Beeckmans 
wrote:

> On 2020-09-29 18:22, Dennis Kliban wrote:
>
> On Tue, Sep 29, 2020 at 10:30 AM Maarten Beeckmans 
> wrote:
>
>> We are trying to automate our Pulp3 setup the same way as our Pulp2 setup.
>>
>> We have the following use cases in our Pulp2 setup:
>>
>> a) We create mirrors and repositories automatically for every environment
>> (using puppet and the API). The repositories are defined in a yaml
>> configuration file (Puppet Hiera).
>>   - We have several problems for doing this in Pulp3. It's very complex
>> to automatically create mirrors (ex. centos-8-appstream) for the
>> development environment (that are in sync with the upstream repositories).
>> A possible solution for this problem would be to add a 'latest' flag to
>> publications and distributions so that they are always pointing to the
>> latest version of the repository.
>>   - It is possible to create the repositories with scripts, but that's
>> specific and quite hacky. It's also not possible to automatically update
>> the publications and distributions this way. We would need to keep a local
>> copy of what distribution maps to what repository, but that's not a good
>> idea. Why keep a local copy of the mapping if that's present in pulp.
>>
>
> We are already planning to improve this workflow. A single API call will
> create a task that will sync a repo, create a new publication, and update
> the distribution. We are first going to introduce the change in pulp_file
> and then other plugins will follow. https://pulp.plan.io/issues/7469
>
> This issue seems to be solving my issue.
>
>
>
>> b) We upload packages from one repository (that is not served to clients)
>> to another repository based on lists in yaml (cherry picking), or straight
>> from a (jenkins) Pipeline. This is done by the pulp-admin commands.
>>   - cherry picking could be done by copying content from one repository
>> to another using the copy API call, but we want to automatically expose
>> those changes to the development environment (promotion can be done as in
>> c).
>>   - Uploading content isn't an issue because this is possible by
>> converting the pulp-admin commands to api calls.
>>
>
> You want to be able to upload a package and have that operation create a
> new publication and update the distribution. This would be similar to the
> story that I linked above for syncing - except for the upload use case.  Am
> I understanding correctly?
>
> Yes, that's what we want. If we create an rpm itself, we want that's it
> directly available in the (development) repository so we don't have to
> create a new publication and update the distribution. For adding it to
> production, we would promote the repository as described in c. Same when
> copying packages between repositories.
>

I definitely see the utility in making the content available to the users
right away. Please file a story at https://pulp.plan.io/issues/new/ so we
have this use captured. This use case should be considered also when
implementing #7469.


>
>
>> c) We want to be able to 'promote' a set of repositories from one
>> environment to another. We are using a script that is generated by puppet
>> when creating the repositories in a).
>>   For 'promoting' a set of repositories to one environment to another
>> environment, this is explained
>> https://docs.pulpproject.org/pulpcore/workflows/promotion.html.
>>
>>
> I don't understand what challenge you are describing in C.
>
> Now we have different repositories for development and production. We want
> to be able to easily promote repositories from development to production
> (with archiving, the production repository keeps X older versions of
> packages), this may be done with one command in the cli interface or
> different commands that can be scripted. This is somewhat explained in the
> docs, but it's very high level. Maybe it's a good idea to add an example in
> the rpm docs (with the promoting of an CentOS repository)?
>

I agree that we need more docs around this. Can you please file an issue
about this also?

There are two ways to accomplish this.

1) Create a distribution for each environment. In this case whenever you
are ready to promote a publication from the Dev distribution to the
Production distribution you just simply look at which publication is
currently assigned to Dev and assign it to Production.

When you want to roll back to a previous publication you need to query for
a publication that is associated with a specific repository version (or
publication date of creation) that you are interested in. Then you need to
update the distribution 

Re: [Pulp-list] Pulp3 Automation

[Dennis Kliban]

On Tue, Sep 29, 2020 at 10:30 AM Maarten Beeckmans 
wrote:

> We are trying to automate our Pulp3 setup the same way as our Pulp2 setup.
>
> We have the following use cases in our Pulp2 setup:
>
> a) We create mirrors and repositories automatically for every environment
> (using puppet and the API). The repositories are defined in a yaml
> configuration file (Puppet Hiera).
>   - We have several problems for doing this in Pulp3. It's very complex to
> automatically create mirrors (ex. centos-8-appstream) for the development
> environment (that are in sync with the upstream repositories). A possible
> solution for this problem would be to add a 'latest' flag to publications
> and distributions so that they are always pointing to the latest version of
> the repository.
>   - It is possible to create the repositories with scripts, but that's
> specific and quite hacky. It's also not possible to automatically update
> the publications and distributions this way. We would need to keep a local
> copy of what distribution maps to what repository, but that's not a good
> idea. Why keep a local copy of the mapping if that's present in pulp.
>

We are already planning to improve this workflow. A single API call will
create a task that will sync a repo, create a new publication, and update
the distribution. We are first going to introduce the change in pulp_file
and then other plugins will follow. https://pulp.plan.io/issues/7469


> b) We upload packages from one repository (that is not served to clients)
> to another repository based on lists in yaml (cherry picking), or straight
> from a (jenkins) Pipeline. This is done by the pulp-admin commands.
>   - cherry picking could be done by copying content from one repository to
> another using the copy API call, but we want to automatically expose those
> changes to the development environment (promotion can be done as in c).
>   - Uploading content isn't an issue because this is possible by
> converting the pulp-admin commands to api calls.
>

You want to be able to upload a package and have that operation create a
new publication and update the distribution. This would be similar to the
story that I linked above for syncing - except for the upload use case.  Am
I understanding correctly?


> c) We want to be able to 'promote' a set of repositories from one
> environment to another. We are using a script that is generated by puppet
> when creating the repositories in a).
>   For 'promoting' a set of repositories to one environment to another
> environment, this is explained
> https://docs.pulpproject.org/pulpcore/workflows/promotion.html.
>
>
I don't understand what challenge you are describing in C.


> Thanks for all the work
>
> Maarten
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] 2to3migration fails on 'Package' object has no attribute '_remote_artifact_saver_cas'

[Dennis Kliban]

Here is a patch for pulpcore that fixes this problem[0].

[0] https://github.com/pulp/pulpcore/pull/937

On Mon, Sep 21, 2020 at 7:17 AM Winberg Adam  wrote:

> Thank you for your reply - yes I did clean orphans after i removed the
> pulp2 repos that i suspected might be the cause. But as you say, there
> might be some other cause for this in my case.
>
>
> At the moment I have adjusted the code so the iteration only runs if the 
> '_remote_artifact_saver_cas'
> attribute is present. Don't know if the result will be any good though,
> running the migration right now.
>
>
> //Adam
>
>
> --
> *From:* Ina Panova 
> *Sent:* 21 September 2020 13:09
> *To:* Winberg Adam
> *Cc:* pulp-list@redhat.com
> *Subject:* Re: [Pulp-list] 2to3migration fails on 'Package' object has no
> attribute '_remote_artifact_saver_cas'
>
> Hi,
> the provided steps in the mentioned issue are the steps to reproduce the
> issue, however, unfortunately, this does not necessarily mean that this is
> the root cause of the manifested problem.
> Apparently we need to find a fix to properly handle duplicated declarative
> content in a batch.
>
> Looking at the steps you have tried to bypass the issue, have you run
> orphan clean up after pulp2 repos removal?
>
>
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Sun, Sep 20, 2020 at 9:05 AM Winberg Adam  wrote:
>
>> Hi,
>>
>>
>> When running the 2to3migration for the 'rpm' plugin, I get the following
>> error:
>>
>>
>> AttributeError: 'Package' object has no attribute
>> '_remote_artifact_saver_cas'
>>
>>
>>
>> This is the same as specified in https://pulp.plan.io/issues/7147, and I
>> actually had a couple of repos in pulp2 which shared the same feed. I
>> removed the redundant repos, flushed the pulp3 db and reran the
>> 2to3-migration but still got stuck on the same error.
>>
>>
>> Anyone got any pointers how to resolve this?
>>
>>
>> //Adam
>> ___
>> Pulp-list mailing list
>> Pulp-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] pulp_container 2.1.0 is Generally Available

[Dennis Kliban]

pulp_container 2.1.0 is now available on PyPI[0]. This release is
compatible with pulpcore 3.7.0. It contains documentation updates and an
improvement for how content is served when S3 is used as a storage backend.

PyPI: https://pypi.org/project/pulp-container/2.1.0/
Changelog: https://pulp-container.readthedocs.io/en/2.1.0/changes.html#id1
Docs: https://pulp-container.readthedocs.io/
Python bindings: https://pypi.org/project/pulp-container-client/2.1.0/
Ruby bindings:
https://rubygems.org/gems/pulp_container_client/versions/2.1.0/
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] pulp_container 2.0.1 is Generally Available

[Dennis Kliban]

pulp_container 2.0.1 is now available on PyPI[0]. This release contains an
important bug fix for deployments behind an HTTPS reverse proxy.

PyPI: https://pypi.org/project/pulp-container/2.0.1/
Changelog: https://pulp-container.readthedocs.io/en/2.0.1/changes.html#id1
Docs: https://pulp-container.readthedocs.io/
Python bindings: https://pypi.org/project/pulp-container-client/2.0.1/
Ruby bindings:
https://rubygems.org/gems/pulp_container_client/versions/2.0.1/
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] pulpcore 3.6.3 is generally available

[Dennis Kliban]

pulpcore 3.6.3 is available on PyPI[0]. This release fixes a packaging
bug[1].

# Installation and Upgrade

Users should use the 3.6.3 release of pulp_installer[2] to install or
upgrade their installations. This version of the installer will check
compatibility of all installed plugins with pulpcore 3.6. The installer
will abort if any plugin is incompatible.

The pulp_installer collection can be installed from Ansible Galaxy with the
following command:

ansible-galaxy collection  install --force pulp.pulp_installer

The --force flag will upgrade the collection if you had a previous version
installed.

# Plugin API

There are no plugin API changes.

# Client libraries

Both the Python[3] and Ruby[4] clients are available also.

[0] https://pypi.org/project/pulpcore/3.6.3/
[1] https://docs.pulpproject.org/pulpcore/en/3.6.3/changes.html#id1
[2] https://galaxy.ansible.com/pulp/pulp_installer
[3] https://pypi.org/project/pulpcore-client/3.6.3/
[4] https://rubygems.org/gems/pulpcore_client/versions/3.6.3
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] pulpcore 3.6.1 and 3.6.2 are Generally Available

[Dennis Kliban]

Pulpcore 3.6.1 was released yesterday, but a bug with the OpenAPI schema
was discovered before this announcement could be sent out. This bug has
been fixed and pulpcore 3.6.2 is now available also[0]. These releases
contain three bug fixes and improvements to documentation. For a list of
all changes, please check the changelog for pulpcore[1].

# Installation and Upgrade

Users should use the 3.6.2 release of pulp_installer[2] to install or
upgrade their installations. This version of the installer will check
compatibility of all installed plugins with pulpcore 3.6. The installer
will abort if any plugin is incompatible.

The pulp_installer collection can be installed from Ansible Galaxy with the
following command:

ansible-galaxy collection  install --force pulp.pulp_installer

The --force flag will upgrade the collection if you had a previous version
installed.

# Plugin API

The plugin API changes were all related to problems with the OpenAPI
schema. These bugs were all introduced in 3.6.0 when pulpcore upgraded to
OpenAPI version 3. The OpenAPI schema improvements are also evident in the
both the Python[3] and Ruby[4] clients. For the full list of changes,
please check the plugin API changelog[5].

[0] https://pypi.org/project/pulpcore/3.6.2/
[1] https://docs.pulpproject.org/pulpcore/en/3.6.2/changes.html#id1
[2] https://galaxy.ansible.com/pulp/pulp_installer
[3] https://pypi.org/project/pulpcore-client/3.6.2/
[4] https://rubygems.org/gems/pulpcore_client/versions/3.6.2
[5] https://docs.pulpproject.org/en/3.6.2/changes.html#plugin-api
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Can't get CentOS 7 Updates repo published

[Dennis Kliban]

The repoview feature requires the sqlite db to be generated. This  feature
generates HTML pages for browsing the repository in a web browser. I
believe that users are still able to browse the repository without it, but
in that case the HTML listing of directories is generated by the web server
each time a user requests it.

On Tue, Sep 1, 2020 at 4:53 AM Konstantin M. Khankin <
khankin.konstan...@gmail.com> wrote:

> Thank you, that obviously helped. Still would be useful to know the root
> cause.
>
> OTOH, if sqlite creation functionality is not that critical - why would we
> ever enable something which only consumes time during sync for no benefit?
>
> вт, 1 сент. 2020 г. в 00:26, Dennis Kliban :
>
>> Looks like everything is up to date. I have no idea what the root cause
>> is, but according to this comment[0], you can work around the problem by
>> disabling the generation of sqlite db. I am not sure what the exact effect
>> of this will be for the clients that are consuming this content, but the
>> repository will be usable.
>>
>> [0] https://pulp.plan.io/issues/2019#note-2
>>
>> On Mon, Aug 31, 2020 at 4:10 PM Konstantin M. Khankin <
>> khankin.konstan...@gmail.com> wrote:
>>
>>> # rpm -qa | grep pulp | sort
>>> pulp-admin-client-2.21.3-1.el7.noarch
>>> pulp-agent-2.21.3-1.el7.noarch
>>> pulp-consumer-client-2.21.3-1.el7.noarch
>>> pulp-deb-admin-extensions-1.10.1-1.el7.noarch
>>> pulp-deb-plugins-1.10.1-1.el7.noarch
>>> pulp-docker-admin-extensions-3.2.6-1.el7.noarch
>>> pulp-docker-plugins-3.2.6-1.el7.noarch
>>> pulp-puppet-consumer-extensions-2.21.3-1.el7.noarch
>>> pulp-puppet-handlers-2.21.3-1.el7.noarch
>>> pulp-python-admin-extensions-2.0.4-1.el7.noarch
>>> pulp-python-plugins-2.0.4-1.el7.noarch
>>> pulp-rpm-admin-extensions-2.21.3-1.el7.noarch
>>> pulp-rpm-consumer-extensions-2.21.3-1.el7.noarch
>>> pulp-rpm-handlers-2.21.3-1.el7.noarch
>>> pulp-rpm-plugins-2.21.3-1.el7.noarch
>>> pulp-rpm-yumplugins-2.21.3-1.el7.noarch
>>> pulp-selinux-2.21.3-1.el7.noarch
>>> pulp-server-2.21.3-1.el7.noarch
>>> python-pulp-agent-lib-2.21.3-1.el7.noarch
>>> python-pulp-bindings-2.21.3-1.el7.noarch
>>> python-pulp-client-lib-2.21.3-1.el7.noarch
>>> python-pulp-common-2.21.3-1.el7.noarch
>>> python-pulp-deb-common-1.10.1-1.el7.noarch
>>> python-pulp-docker-common-3.2.6-1.el7.noarch
>>> python-pulp-oid_validation-2.21.3-1.el7.noarch
>>> python-pulp-puppet-common-2.21.3-1.el7.noarch
>>> python-pulp-python-common-2.0.4-1.el7.noarch
>>> python-pulp-repoauth-2.21.3-1.el7.noarch
>>> python-pulp-rpm-common-2.21.3-1.el7.noarch
>>>
>>> # rpm -qa | grep createrepo
>>> createrepo-0.9.9-28.el7.noarch
>>> createrepo_c-0.10.0-20.el7.x86_64
>>> createrepo_c-libs-0.10.0-20.el7.x86_64
>>>
>>> # cat /etc/redhat-release
>>> CentOS Linux release 7.8.2003 (Core)
>>>
>>> пн, 31 авг. 2020 г. в 22:48, Dennis Kliban :
>>>
>>>> This looks exactly like the issue that was reported here[0].
>>>>
>>>> What version of pulp are you using? What version of createrepo_c is
>>>> installed?
>>>>
>>>> [0] https://pulp.plan.io/issues/2019
>>>>
>>>> On Mon, Aug 31, 2020 at 3:16 PM Konstantin M. Khankin <
>>>> khankin.konstan...@gmail.com> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> The issue still persists. Could someone take a look please?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> ср, 19 авг. 2020 г., 13:15 Konstantin M. Khankin <
>>>>> khankin.konstan...@gmail.com>:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> I found that my pulp2-managed mirror of
>>>>>> http://mirror.yandex.ru/centos/7/updates/x86_64 has not been
>>>>>> successfully published since April 30. I ran publish manually and 
>>>>>> received
>>>>>> an error:
>>>>>>
>>>>>> '''
>>>>>> Generating sqlite files
>>>>>> [/]
>>>>>> ... failed
>>>>>> Error occurred during 'sqliterepo_c' execution: Preparing sqlite
>>>>>> DBs
>>>>>>
>>>>>> ::
>>>>>> C_CREATEREPOLIB: Critical: cr_xml_parser_generic: parsing error
>>>>>> '/var/cache/pulp/reserved_resource_worke...@hive.gsk.loc
>>>

Re: [Pulp-list] Pulp 3.3 sync issue from artifactory

[Dennis Kliban]

Sounds like you are experiencing a bug that was fixed in 3.5.1. You can
work around this bug in older versions of pulp_rpm by setting
optimize=False.

https://pulp.plan.io/issues/7228

On Mon, Aug 31, 2020 at 3:42 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Yes, we have 'mirror' = True. The remote repository has not changed. This
> should not delete all packages from repo after sync. Version 2 repo in pulp
> shows all packages are deleted while remote repo has not changed from
> version 1.
>
> From: dkli...@redhat.com At: 08/31/20 15:35:06
> To: Bin Li (BLOOMBERG/ 120 PARK ) 
> Cc: pulp-list@redhat.com
> Subject: Re: [Pulp-list] Pulp 3.3 sync issue from artifactory
>
> I suspect that you have 'mirror' = True in your 'sync' request[0]. With
> this option set to True, the sync creates an exact copy of the remote
> repository. If no packages are found, a new version is created without any
> packages. When a subsequent sync occurs, no changes are detected, so a new
> version is not created.
>
> [0]
> https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/repositories_rpm_rpm_sync
>
> On Mon, Aug 31, 2020 at 3:12 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> We created a rpm repos on artifactory and sync the repo to pulp 3.3. All
>> packages are synced to pulp as version 1. If we run sync again, pulp could
>> not find any packages and remove all previous packages in version 2.
>> Subsequent sync also could not find any packages and the repo version stay
>> as version 2. So far we only experience the issue when syncing from
>> artifactory. Please advise.
>>
>> Thanks
>> ___
>> Pulp-list mailing list
>> Pulp-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Can't get CentOS 7 Updates repo published

[Dennis Kliban]

Looks like everything is up to date. I have no idea what the root cause is,
but according to this comment[0], you can work around the problem by
disabling the generation of sqlite db. I am not sure what the exact effect
of this will be for the clients that are consuming this content, but the
repository will be usable.

[0] https://pulp.plan.io/issues/2019#note-2

On Mon, Aug 31, 2020 at 4:10 PM Konstantin M. Khankin <
khankin.konstan...@gmail.com> wrote:

> # rpm -qa | grep pulp | sort
> pulp-admin-client-2.21.3-1.el7.noarch
> pulp-agent-2.21.3-1.el7.noarch
> pulp-consumer-client-2.21.3-1.el7.noarch
> pulp-deb-admin-extensions-1.10.1-1.el7.noarch
> pulp-deb-plugins-1.10.1-1.el7.noarch
> pulp-docker-admin-extensions-3.2.6-1.el7.noarch
> pulp-docker-plugins-3.2.6-1.el7.noarch
> pulp-puppet-consumer-extensions-2.21.3-1.el7.noarch
> pulp-puppet-handlers-2.21.3-1.el7.noarch
> pulp-python-admin-extensions-2.0.4-1.el7.noarch
> pulp-python-plugins-2.0.4-1.el7.noarch
> pulp-rpm-admin-extensions-2.21.3-1.el7.noarch
> pulp-rpm-consumer-extensions-2.21.3-1.el7.noarch
> pulp-rpm-handlers-2.21.3-1.el7.noarch
> pulp-rpm-plugins-2.21.3-1.el7.noarch
> pulp-rpm-yumplugins-2.21.3-1.el7.noarch
> pulp-selinux-2.21.3-1.el7.noarch
> pulp-server-2.21.3-1.el7.noarch
> python-pulp-agent-lib-2.21.3-1.el7.noarch
> python-pulp-bindings-2.21.3-1.el7.noarch
> python-pulp-client-lib-2.21.3-1.el7.noarch
> python-pulp-common-2.21.3-1.el7.noarch
> python-pulp-deb-common-1.10.1-1.el7.noarch
> python-pulp-docker-common-3.2.6-1.el7.noarch
> python-pulp-oid_validation-2.21.3-1.el7.noarch
> python-pulp-puppet-common-2.21.3-1.el7.noarch
> python-pulp-python-common-2.0.4-1.el7.noarch
> python-pulp-repoauth-2.21.3-1.el7.noarch
> python-pulp-rpm-common-2.21.3-1.el7.noarch
>
> # rpm -qa | grep createrepo
> createrepo-0.9.9-28.el7.noarch
> createrepo_c-0.10.0-20.el7.x86_64
> createrepo_c-libs-0.10.0-20.el7.x86_64
>
> # cat /etc/redhat-release
> CentOS Linux release 7.8.2003 (Core)
>
> пн, 31 авг. 2020 г. в 22:48, Dennis Kliban :
>
>> This looks exactly like the issue that was reported here[0].
>>
>> What version of pulp are you using? What version of createrepo_c is
>> installed?
>>
>> [0] https://pulp.plan.io/issues/2019
>>
>> On Mon, Aug 31, 2020 at 3:16 PM Konstantin M. Khankin <
>> khankin.konstan...@gmail.com> wrote:
>>
>>> Hi!
>>>
>>> The issue still persists. Could someone take a look please?
>>>
>>> Thanks!
>>>
>>> ср, 19 авг. 2020 г., 13:15 Konstantin M. Khankin <
>>> khankin.konstan...@gmail.com>:
>>>
>>>> Hi!
>>>>
>>>> I found that my pulp2-managed mirror of
>>>> http://mirror.yandex.ru/centos/7/updates/x86_64 has not been
>>>> successfully published since April 30. I ran publish manually and received
>>>> an error:
>>>>
>>>> '''
>>>> Generating sqlite files
>>>> [/]
>>>> ... failed
>>>> Error occurred during 'sqliterepo_c' execution: Preparing sqlite
>>>> DBs
>>>>
>>>> ::
>>>> C_CREATEREPOLIB: Critical: cr_xml_parser_generic: parsing error
>>>> '/var/cache/pulp/reserved_resource_worke...@hive.gsk.loc
>>>> /12257118-33e7-4294-ad68
>>>>
>>>> -98e0515f2627/repodata/9e958c09c7880d130ef3321332000f43a4e4e701e416801a0f54313f5
>>>> 766d03b-filelists.xml.gz': not well-formed (invalid token)
>>>> Parse error
>>>> '/var/cache/pulp/reserved_resource_worke...@hive.gsk.loc
>>>> /12257118-33e7-4294-ad68
>>>>
>>>> -98e0515f2627/repodata/9e958c09c7880d130ef3321332000f43a4e4e701e416801a0f54313f5
>>>> 766d03b-filelists.xml.gz' at line: 1884123 (not well-formed (invalid
>>>> token))
>>>> '''
>>>>
>>>> I can't open .sqlite files in /tmp from CLI either:
>>>>
>>>> '''
>>>> -rw---. 1 apache apache  54066176 Aug 19 15:12
>>>> filelists.211JP0.sqlite
>>>> -rw---. 1 apache apache 157097984 Aug 19 15:12 primary.PQ3JP0.sqlite
>>>> -rw---. 1 apache apache 0 Aug 19 15:12 other.X201JP0.sqlite
>>>>
>>>> [root@hive tmp]# sqlite3 primary.PQ3JP0.sqlite
>>>> SQLite version 3.7.17 2013-05-20 00:56:22
>>>> Enter ".help" for instructions
>>>> Enter SQL statements terminated with a ";"
>>>> sqlite> .databases
>>>> Error: file is encrypted or is not a database
>>>> '''
>>>>
>>>> I have multiple repos managed by pulp, some of which originate also
>>>> from mirror.yandex.ru, and they are synced and published normally.
>>>> --force-full doesn't help.
>>>>
>>>> What issue could be here?
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>> Konstantin Khankin
>>>>
>>> ___
>>> Pulp-list mailing list
>>> Pulp-list@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>>
>
> --
> Ханкин Константин
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Can't get CentOS 7 Updates repo published

[Dennis Kliban]

This looks exactly like the issue that was reported here[0].

What version of pulp are you using? What version of createrepo_c is
installed?

[0] https://pulp.plan.io/issues/2019

On Mon, Aug 31, 2020 at 3:16 PM Konstantin M. Khankin <
khankin.konstan...@gmail.com> wrote:

> Hi!
>
> The issue still persists. Could someone take a look please?
>
> Thanks!
>
> ср, 19 авг. 2020 г., 13:15 Konstantin M. Khankin <
> khankin.konstan...@gmail.com>:
>
>> Hi!
>>
>> I found that my pulp2-managed mirror of
>> http://mirror.yandex.ru/centos/7/updates/x86_64 has not been
>> successfully published since April 30. I ran publish manually and received
>> an error:
>>
>> '''
>> Generating sqlite files
>> [/]
>> ... failed
>> Error occurred during 'sqliterepo_c' execution: Preparing sqlite
>> DBs
>>
>> ::
>> C_CREATEREPOLIB: Critical: cr_xml_parser_generic: parsing error
>> '/var/cache/pulp/reserved_resource_worke...@hive.gsk.loc
>> /12257118-33e7-4294-ad68
>>
>> -98e0515f2627/repodata/9e958c09c7880d130ef3321332000f43a4e4e701e416801a0f54313f5
>> 766d03b-filelists.xml.gz': not well-formed (invalid token)
>> Parse error
>> '/var/cache/pulp/reserved_resource_worke...@hive.gsk.loc
>> /12257118-33e7-4294-ad68
>>
>> -98e0515f2627/repodata/9e958c09c7880d130ef3321332000f43a4e4e701e416801a0f54313f5
>> 766d03b-filelists.xml.gz' at line: 1884123 (not well-formed (invalid
>> token))
>> '''
>>
>> I can't open .sqlite files in /tmp from CLI either:
>>
>> '''
>> -rw---. 1 apache apache  54066176 Aug 19 15:12 filelists.211JP0.sqlite
>> -rw---. 1 apache apache 157097984 Aug 19 15:12 primary.PQ3JP0.sqlite
>> -rw---. 1 apache apache 0 Aug 19 15:12 other.X201JP0.sqlite
>>
>> [root@hive tmp]# sqlite3 primary.PQ3JP0.sqlite
>> SQLite version 3.7.17 2013-05-20 00:56:22
>> Enter ".help" for instructions
>> Enter SQL statements terminated with a ";"
>> sqlite> .databases
>> Error: file is encrypted or is not a database
>> '''
>>
>> I have multiple repos managed by pulp, some of which originate also from
>> mirror.yandex.ru, and they are synced and published normally.
>> --force-full doesn't help.
>>
>> What issue could be here?
>>
>> Thanks!
>>
>> --
>> Konstantin Khankin
>>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Pulp 3.3 sync issue from artifactory

[Dennis Kliban]

I suspect that you have 'mirror' = True in your 'sync' request[0]. With
this option set to True, the sync creates an exact copy of the remote
repository. If no packages are found, a new version is created without any
packages. When a subsequent sync occurs, no changes are detected, so a new
version is not created.

[0]
https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/repositories_rpm_rpm_sync

On Mon, Aug 31, 2020 at 3:12 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> We created a rpm repos on artifactory and sync the repo to pulp 3.3. All
> packages are synced to pulp as version 1. If we run sync again, pulp could
> not find any packages and remove all previous packages in version 2.
> Subsequent sync also could not find any packages and the repo version stay
> as version 2. So far we only experience the issue when syncing from
> artifactory. Please advise.
>
> Thanks
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Pulp RHEL8 rpm installation

[Dennis Kliban]

The pulp_installer supports installing from RPMs. Currently we don't
publish the RPMs, but the Katello project does include such packages in its
repository[0]. However, this repository only includes the plugins that are
used by Katello - pulp_file, pulp_rpm, pulp_deb, pulp_container,
pulp-certguard, and pulp-2to3-migration.

There is definitely a desire to eventually provide official Pulp RPMs in
our own repositories, however, that is not a focus at this time.

[0] https://fedorapeople.org/groups/katello/releases/yum/3.16/pulpcore/

On Thu, Aug 27, 2020 at 2:52 PM Winberg Adam  wrote:

> Hi,
>
>
> We are currently running pulp2 on RHEL7 and I was looking to upgrade to
> pulp3 and move this over to our RHEL8 environment. To my surprise I
> discovered that rpm as installation method is no longer an option with
> pulp3 and there are no rhel8 rpm builds of either pulp2 or pulp3.
>
>
> Our organization is relying quite heavily on RPM for CI/CD and automatic
> rebuilds of server. Ansible/pypi installations require Internet access and
> external docker/container images can also be considered a security
> liability (is in my org anyway). Sure, we can have local pypi indexes and
> such but Ansible/pypi as deployment method to me just does not feel very
> 'enterprisey'.
>
>
> So I wanted to open this up for discussion. Anyone else out there hoping
> for rpm builds of Pulp for RHEL8 (or other dists)?
>
>
> Regards,
>
> Adam
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulpcore-3.6.0 and TaskResponse.reserved_resources_record bindings

[Dennis Kliban]

This is not an intended change. Please file a bug at
https://pulp.plan.io/issues/new/ and we will try to address it as soon as
possible.


On Tue, Aug 18, 2020 at 7:31 AM Sebastian Skracic 
wrote:

>
>   Howdi all!
>
>   I have just upgraded the backend to pulpcore-3.6.0 and
> pulp-rpm-3.6.0, and then bumped the corresponding Python bindings to
> pulpcore-client==3.6.0 and pulp-rpm-client==3.6.0.
>
>   To my surprise, I've discovered that 'reserved_resources_record'
> for tasks is no longer present in the bindings, and is omitted from the
> OpenAPI specs, cf: https://pulp3.skracic.net/pulp/api/v3/docs/api.json
> (search for TaskResponse)
>
>   What's puzzling me is that the field is still present in the REST
> responses, so it's technically possible to fetch it using
> pulpcore-client==3.5.0 bindings.
>
>   Is this an intended change or maybe a glitch in OpenAPI generation
> process?
>
>   Thanks,
>
> --
> Sebastian Skracic   -  Red Hat  -   sskra...@redhat.com
> GPG: 9D5B 2B87 908E 85B6 5D1E  3527 76B7 6594 08A5 A206
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] pulpcore 3.6.0 and pulp_file 1.2.0 are generally available

[Dennis Kliban]

Pulpcore 3.6.0[0] and pulp_file 1.2.0[1] have been released.

Details of the most important changes are available in our blog[2]. For a
full list of changes, please check the changelog for pulpcore[3] and
pulp_file[4].

There are breaking changes for users of the client bindings and OpenAPI
schema users.

# Installation and Upgrade

Users should use the 3.6.0 release of pulp_installer[5] to install or
upgrade their installations. This version of the installer will check
compatibility of all installed plugins with pulpcore 3.6. The installer
will abort if any plugin is incompatible.

The pulp_installer collection can be installed from Ansible Galaxy with the
following command:

ansible-galaxy collection  install --force pulp.pulp_installer

The --force flag will upgrade the collection if you had a previous version
installed.

# Plugin API

Plugin writers can see the API changes here[6].  Some highlights:

 * Role Based Access Control support.

And in keeping with the recommended strategy to pin plugins to a 3.y
version of pulpcore, plugins should release compatibility releases with 3.5
as soon as they can. We recommend using "pulpcore>=3.6,<3.7".

[0] https://pypi.org/project/pulpcore/3.6.0/
[1] https://pypi.org/project/pulp-file/1.2.0/
[2] https://pulpproject.org/2020/08/13/pulp-3.6-is-generally-available/
[3] https://docs.pulpproject.org/en/3.6/changes.html#id1
[4] https://pulp-file.readthedocs.io/en/latest/changes.html#id1
[5] https://galaxy.ansible.com/pulp/pulp_installer
[6] https://docs.pulpproject.org/en/3.6/changes.html#plugin-api
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] 502 Bad Gateway error connecting to new pulp instance installed with pulp_installer

[Dennis Kliban]

t;>>   stdout_lines: 
>>>>>>
>>>>>> PLAY RECAP
>>>>>> *pulpcentos
>>>>>> : ok=33   changed=14   unreachable=0failed=1
>>>>>>  skipped=16   rescued=0ignored=0
>>>>>>
>>>>>> I believe this means that the version of pulp_installer role(s) I
>>>>>> have/had installed have become broken bc of compatibility changes made to
>>>>>> one or more versions they were referencing. This seems bad, 
>>>>>> nevertheless, I
>>>>>> went ahead and updated my pulp_installer to a newer tag (from 3.4.1 to
>>>>>> 3.5.0), and reran the pulp.yml playbook, with the following results:
>>>>>>
>>>>>> With 3.5.0 pulp_installer, running against fresh new centos 8
>>>>>> machine, it got past the pulpcore/plugin version check, but failed here, 
>>>>>> in
>>>>>> pulp_common's check for required variables. This worked fine before (on 
>>>>>> my
>>>>>> debian-based machine) as you can see in my playbook I'm using an
>>>>>> ansible-vault encrypted string as the secret_key.
>>>>>>
>>>>>> TASK [pulp_common : Check if required variables are set]
>>>>>> ***Monday
>>>>>> 27 July 2020  14:34:27 -0700 (0:00:00.024)   0:00:19.821 ***
>>>>>>
>>>>>>   ok: [pulpcentos] => (item=pulp_settings.content_origin) =>
>>>>>> changed=false
>>>>>>ansible_loop_var: item
>>>>>>
>>>>>>item:
>>>>>> pulp_settings.content_origin
>>>>>>
>>>>>>msg: All assertions passed
>>>>>>
>>>>>>  fatal: [pulpcentos]: FAILED! =>
>>>>>>   msg: 'The conditional check ''pulp_settings.secret_key |
>>>>>> default(, true) | length > 0'' failed. The error was: Unexpected
>>>>>> templating type error occurred on ({% if pulp_settings.secret_key |
>>>>>> default(, true) | length > 0 %} True {% else %} False {% endif %}):
>>>>>> object of type ''AnsibleVaultEncryptedUnicode'' has no len()'
>>>>>>
>>>>>> Not sure what's up, but at the very least so far it's not working any
>>>>>> better with CentOS. I'm all ears for suggestions.
>>>>>>
>>>>>> Does anyone have a turnkey, fully-automated solution they can share,
>>>>>> like a vagrant box that brings up a pulp instance from scratch? Seems 
>>>>>> like
>>>>>> I'm doing a lot more work here than should be required to bring this 
>>>>>> thing
>>>>>> up. Thanks.
>>>>>>
>>>>>> On Sat, Jul 11, 2020 at 1:49 PM Dennis Kliban 
>>>>>> wrote:
>>>>>>
>>>>>>> I would recommend re-running the installer on a fresh VM that is
>>>>>>> running CentOS 7.7+. I've experienced this problem before when the
>>>>>>> installer had to be run multiple times due to various failures. In my 
>>>>>>> case,
>>>>>>> the database migrations had not been run and the output of "systemctl
>>>>>>> status pulpcore*" showed that Pulp services were failing to start due to
>>>>>>> database issues. I suspected it was due to permissions problems with
>>>>>>> /etc/pulp/settings.py, however, I never confirmed this by actually 
>>>>>>> fixing
>>>>>>> the install. I've always just reprovisioned on a new VM.
>>>>>>>
>>>>>>> If you can reproduce this issue again on a new VM, I would recommend
>>>>>>> filing an issue at https://pulp.plan.io/issues/new/. The installer
>>>>>>> is definitely doing something wrong, but I am not sure how to reproduce 
>>>>>>> the
>>>>>>> issue consistently.
>>>>>>>

Re: [Pulp-list] Pulp 3 distribution trees

[Dennis Kliban]

Distribution trees are currently considered a tech preview feature. There
are multiple bugs that are actively being fixed at this time[0,1,2]. There
is currently no way for users to create new Distribution Trees. They can
only be synced from remote repositories.

[0] https://pulp.plan.io/issues/7150
[1] https://pulp.plan.io/issues/7115
[2] https://pulp.plan.io/issues/7046


On Mon, Jul 20, 2020 at 4:21 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Hi,
> I am trying to understand how to create a distribution tree. I was able to
> create a repo and sync a kickstart tree to this repo. Should I create a
> publication and a distribution, then use the distribution base_url to
> access the the tree just as a regular repo? Is there anything else we need
> add to distribution tree? Can we create a distribution tree from a iso or
> local directory?
>
> Thanks
> Bin
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] 502 Bad Gateway error connecting to new pulp instance installed with pulp_installer

[Dennis Kliban]

I would recommend re-running the installer on a fresh VM that is running
CentOS 7.7+. I've experienced this problem before when the installer had to
be run multiple times due to various failures. In my case, the database
migrations had not been run and the output of "systemctl status pulpcore*"
showed that Pulp services were failing to start due to database issues. I
suspected it was due to permissions problems with /etc/pulp/settings.py,
however, I never confirmed this by actually fixing the install. I've always
just reprovisioned on a new VM.

If you can reproduce this issue again on a new VM, I would recommend filing
an issue at https://pulp.plan.io/issues/new/. The installer is definitely
doing something wrong, but I am not sure how to reproduce the issue
consistently.


On Fri, Jul 10, 2020 at 11:12 PM Tim Black  wrote:

> Thanks Matthias. I get 502 at http://pulp.my.domain/pulp/api/v3/status/
> as well. Below is my nginx.conf, pulled from my freshly provisioned pulp
> server. My skills are a little weak on the webserver side of things so I'm
> open to suggestions for any simplifications I can make to my config to get
> this working. I'm not trying to do anything fancy here.
>
> /etc/nginx/nginx.conf:
>
> # TODO: Support IPv6.
> # TODO: Configure SSL certificates.
> # TODO: Maybe serve multiple `location`s, not just one.
>
> # Gunicorn docs suggest this value.
> worker_processes 1;
> events {
> worker_connections 1024;  # increase if you have lots of clients
> accept_mutex off;  # set to 'on' if nginx worker_processes > 1
> }
>
> http {
> include mime.types;
> # fallback in case we can't determine a type
> default_type application/octet-stream;
> sendfile on;
>
> # If left at the default of 1024, nginx emits a warning about being
> unable
> # to build optimal hash types.
> types_hash_max_size 4096;
>
> upstream pulp-content {
>  server 127.0.0.1:24816;
> }
>
> upstream pulp-api {
>  server 127.0.0.1:24817;
> }
>
> server {
> # Gunicorn docs suggest the use of the "deferred" directive on
> Linux.
> listen 80 default_server deferred;
> server_name $hostname;
>
> # The default client_max_body_size is 1m. Clients uploading
> # files larger than this will need to chunk said files.
>
> # Gunicorn docs suggest this value.
> keepalive_timeout 5;
>
> location /pulp/content/ {
> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> proxy_set_header X-Forwarded-Proto $scheme;
> proxy_set_header Host $http_host;
> # we don't want nginx trying to do something clever with
> # redirects, we set the Host: header above already.
> proxy_redirect off;
> proxy_pass http://pulp-content;
> }
>
> location /pulp/api/v3/ {
> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> proxy_set_header X-Forwarded-Proto $scheme;
> proxy_set_header Host $http_host;
> # we don't want nginx trying to do something clever with
> # redirects, we set the Host: header above already.
> proxy_redirect off;
> proxy_pass http://pulp-api;
> }
>
> location /auth/login/ {
> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> proxy_set_header X-Forwarded-Proto $scheme;
> proxy_set_header Host $http_host;
> # we don't want nginx trying to do something clever with
> # redirects, we set the Host: header above already.
> proxy_redirect off;
> proxy_pass http://pulp-api;
> }
>
> include pulp/*.conf;
>
> location / {
> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> proxy_set_header X-Forwarded-Proto $scheme;
> proxy_set_header Host $http_host;
> # we don't want nginx trying to do something clever with
> # redirects, we set the Host: header above already.
> proxy_redirect off;
> proxy_pass http://pulp-api;
> # static files are served through whitenoise -
> http://whitenoise.evans.io/en/stable/
> }
> }
> }
>
> On Tue, Jul 7, 2020 at 11:56 PM Matthias Dellweg 
> wrote:
>
>> The only thing that sticks out to me is `content_origin: "http://{{
>> ansible_fqdn }}:8080"`. This is the address seen from the outside, and
>> since both content and api are subject to the same reverse proxy and
>> so should be available on port 80 (and 443 soon). But that is for sure
>> not the problem you have with the API.
>> Can you, however, try `http
>> http://pulp.my.domain/pulp/api/v3/status/`
>> ? And if it still didn't
>> produce a result, provide the content of /etc/nginx/nginx.conf ?
>>
>> On Tue, Jul 7, 2020 at 11:18 PM Tim Black  wrote:
>> >
>> 

[Pulp-list] pulpcore 3.5.0 and pulp_file 1.1.0 are Generally Available

[Dennis Kliban]

Pulpcore 3.5.0[0] and pulp_file 1.1.0[1] have been released.

It should be noted the installer has breaking changes. Upgrades will
require updating existing playbooks. Details of these changes are available
in our blog.[2] For a full list of changes, please check the changelog for
pulpcore[3] and pulp_file[4].

Here are some of the highlights:

 * Added start_versions= to export to allow for arbitrary incremental
exports.
 * Added GroupProgressReport to track progress in a TaskGroup.
 * Provide a user agent string with all aiohttp requests by default.

# Installation and Upgrade

Users should use the 3.5.0 release of pulp_installer[5] to install or
upgrade their installations. This version of the installer will check
compatibility of all installed plugins with pulpcore 3.5. The installer
will abort if any plugin is incompatible.

The pulp_installer collection can be installed from Ansible Galaxy with the
following command:

ansible-galaxy collection  install --force pulp.pulp_installer

The --force flag will upgrade the collection if you had a previous version
installed.

# Plugin API

Plugin writers can see the API changes here[6].  Some highlights:

 * Views can specify the tag name with pulp_tag_name
 * Added GroupProgressReport to track progress in a TaskGroup.
 * Exported the symbols serializers.SingleContentArtifactField and
files.PulpTemporaryUploadedFile.

And in keeping with the recommended strategy to pin plugins to a 3.y
version of pulpcore, plugins should release compatibility releases with 3.5
as soon as they can. We recommend using "pulpcore>=3.5,<3.5". Though most
plugins should continue to be compatible with 3.4 and in those cases will
need to only update the maximum version of pulpcore.

[0] https://pypi.org/project/pulpcore/3.5.0/
[1] https://pypi.org/project/pulp-file/1.1.0/
[2] https://pulpproject.org/2020/07/09/pulp-3.5-installer-roles/
[3] https://docs.pulpproject.org/en/3.5/changes.html#id1
[4] https://pulp-file.readthedocs.io/en/latest/changes.html#id1
[5] https://galaxy.ansible.com/pulp/pulp_installer
[6] https://docs.pulpproject.org/en/3.5/changes.html#plugin-api
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Jenkins pipeline integration

[Dennis Kliban]

There is no Jenkins plugin for Pulp. However, each plugin publishes a
Python client library to PyPI. I would recommend using those libraries to
publish content into Pulp. The documentation for the clients is not
currently published, but can be generated using the instruction for
generating a client library[0]. There is also a script that shows how to
use the clients[1].

The example scripts use bash because we wanted to provide examples that can
be executed from the command line. Once we have a working CLI, the examples
will be converted to that.

[0] https://docs.pulpproject.org/client_bindings.html#other-languages
[1] https://gist.github.com/dkliban/384e5a3c20005e777ef2a2fce9c2a6ba

On Tue, Jul 7, 2020 at 1:28 PM Tim Black  wrote:

> Anyone using pulp for artifact management with Jenkins Pipelines, aside
> from building and testing pulp project itself? I didn't find anything on
> the pulp mailing list archives about Jenkins integration (and the #pulp IRC
> channel has been a desert) and am looking for clarification and
> suggestions.
>
> I see in each plugin's Workflows documentation example bash scripts for
> doing the basic mechanics of creating and syncing repos, publishing and
> retrieving content, and I assume that's the intended starting point for
> newcomers like myself. I just wanted to confirm this and wondered if anyone
> knows of any prior work to wrap this up for easy integration into Jenkins
> pipelines, e.g. Groovy wrappers around your scripts.
>
> FWIW, my use case for pulp is to manage all intermediate and final
> artifacts generated by several Jenkins pipelines for complex embedded,
> desktop and web software projects. Our pipelines would be using the
> following content plugins to retrieve and/or publish all of the (versioned)
> artifacts that our pipelines build and/or use:
>
>1. pulp-deb for .debs
>2. pulp-python for python packages
>3. pulp-container for Docker images
>4. pulp-ansible for Ansible roles and collections
>5. pulp-file for everything else (this is actually the majority of our
>content)
>
> We will be using pulp to manage not only the "final" pipeline output, i.e.
> artifacts consumed by customers, but also intermediate build artifacts
> which are used as a sort of build cache to speed up our development
> process. Any other info or anecdotal reports related to these uses would be
> interesting to me.
>
> In case the above is not clear, what I'm looking for is the existence of
> anything like this
> for
> pulp.
>
> Thanks,
> Tim Black
>
> p.s. I'm curious why the pulp devs chose to write the usage scripts in
> bash when the rest of their application was in python.
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Replication between Pulp installation

[Dennis Kliban]

Pulp 3 does not have a concept of "nodes". This feature was deprecated in
later releases of Pulp 2. Users of Pulp 3 should create normal Pulp
installations anywhere they wish to have a mirror of their master. These
mirrors should have Remotes with a URL pointing to a Distribution hosted by
the master Pulp. These remotes should then be used to sync content from the
master Pulp. The current limitation of this approach is that you must
create a new Publication in each Pulp. In case of RPM repositories, that
means that metadata will be slightly different on each server. We are
working to resolve this issue by adding an ability to mirror metadata[0].
The goal is to deliver this feature in the Fall of 2020.

[0] https://pulp.plan.io/issues/6353

On Mon, Jun 29, 2020 at 11:29 AM Pablo Martinez Schroder <
pa...@docecosas.com> wrote:

> Hi, I cannot find an automatic way to implement replication between
> different Pulp installations. My use case is I have a data center (let's
> call it master) where all the RPM repositories are stored. We also have
> some other data centers where we want to have a replica of those
> repositories but right now I cannot see a way to achieve it with Pulp 3. Am
> I missing something?
>
> With Pulp 2 I could use the Children/Parent nodes relationship, is there
> anything similar or some suggestion of what approach would be better with
> Pulp 3?
>
> Thanks in advance.
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] pulp_container 2.0.0 beta 1 is available

[Dennis Kliban]

pulp_container 2.0.0b1 is available on PyPI[0]. It is accompanied by a
Python client[1] and a Ruby client[2]. This is the first in a series of
betas for the 2.0 release of the Container plugin. This release is
compatible with pulpcore 3.4.

Please note that all container registry authentication has been disabled in
this beta. However, we intend to add token auth back before 2.0.0 is
generally available.

This release contains two major features:

* Ability to use docker or podman to push images into the registry.
* Ability to use S3 as the storage backend

The new REST APIs for supporting push required us to move the registry API
from port 24816 to port 24817. This is the reason for the major version
being incremented from 1 to 2. Just as on the 1.y line, the Python package
provides nginx and apache configs to configure those servers as reverse
proxies so that your clients don't actually have to use either of those
port numbers.

Documentation along with a full changelog can be found on readthedocs.org
[3].

## Installation and upgrades

You can use the latest release of pulp_installer to install or upgrade[4].
The installer will symlink the nginx or apache configs so that the
webserver can reverse proxy the requests to the appropriate service.


[0] https://pypi.org/project/pulp-container/2.0.0b1/
[1] https://pypi.org/project/pulp-container-client/2.0.0b1/
[2] https://rubygems.org/gems/pulp_container_client/versions/2.0.0b1
[3] https://pulp-container.readthedocs.io/en/2.0.0b1/
[4] https://github.com/pulp/pulp_installer/releases/tag/3.4.0
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] pulp_rpm 3.4.0 is Generally Available

[Dennis Kliban]

pulp_rpm 3.4.0 is available on PyPI[0]. This release is compatible with
pulpcore 3.4.0. It is also accompanied by a Python client[1] and a Ruby
client[2].

Please note that upgrades from 3.2.z to 3.3.z were broken due to a
migration bug. This release addresses that problem.

This release contains a new feature, several bug fixes, and documentation
improvements. The changelog contains the full list of changes[3], but here
are some highlights:

## Features

* Distributions now serves a config.repo, and when signing is enabled
also a public.key, in the base_path. #5356

## Bugfixes

* Fixed the duplicated advisory case when only auxiliary fields were
updated but not any timestamp or version. #6604
* Fixed dependency solving issue where not all RPM dependencies were
coped. #6820
* Make ‘last_sync_revision_number’ nullable in all migrations. #6861
* Fixed a bug where the behavior of RPM advanced copy with dependency
solving differed depending on the order of the source-destination
repository pairs provided by the user. #6868

## Installation and upgrades

You can use the pulp_installer version 3.4.0 to perform an installation or
upgrade[4]. The installer has its own documentation[5].

[0] https://pypi.org/project/pulp-rpm/3.4.0/
[1] https://pypi.org/project/pulp-rpm-client/3.4.0/
[2] https://rubygems.org/gems/pulp_rpm_client/versions/3.4.0
[3] https://pulp-rpm.readthedocs.io/en/3.4/changes.html#id1
[4] https://github.com/pulp/pulp_installer/releases/tag/3.4.0
[5] https://pulp-installer.readthedocs.io/
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] pulp-python 3.0.0b9 is available

[Dennis Kliban]

The latest beta release of the Python plugin is available on PyPI[0]. This
version is compatible with pulpcore 3.4.0.

This is mainly a pulpcore 3.4.0 compatibility release. In addition to that,
this release contains a new feature that allows users to upload Python
packages directly into a repository[1]. The changelog contains the full
list of changes[2].

Python bindings[3] and Ruby[4] bindings are available.

### Installation

Please use the 3.4.0 release of pulp_installer to install this plugin[5].

[0] https://pypi.org/project/pulp-python/3.0.0b9/
[1] https://pulp.plan.io/issues/5464
[2] https://pulp-python.readthedocs.io/en/latest/changes.html#b9-2020-06-01
[3] https://pypi.org/project/pulp-python-client/3.0.0b9/
[4] https://rubygems.org/gems/pulp_python_client/versions/3.0.0b9
[5] https://github.com/pulp/pulp_installer/releases/tag/3.4.0
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulp 3.3 migration error

[Dennis Kliban]

Are you upgrading to pulp-rpm 3.3.2?

On Thu, May 28, 2020 at 5:08 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Hi All,
> Getting a new error when migrate from 3.2 to 3.3. Please advise if this
> can be fixed manually.
>
> TASK [pulp_database : Run database migrations]
> 
> fatal: [pulpp-ob-581]: FAILED! => {"changed": true, "cmd":
> ["/opt/utils/venv/pulp/3.7.3/bin/django-admin", "migrate", "--no-input"],
> "delta": "0:00:01.768909", "end": "2020-05-28 17:02:22.996242", "msg":
> "non-zero return code", "rc": 1, "start": "2020-05-28 17:02:21.227333",
> "stderr": "Traceback (most recent call last):\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py\",
> line 84, in _execute\n return self.cursor.execute(sql,
> params)\npsycopg2.errors.NotNullViolation: column
> \"last_sync_revision_number\" contains null values\n\n\nThe above exception
> was the direct cause of the following exception:\n\nTraceback (most recent
> call last):\n File \"/opt/utils/venv/pulp/3.7.3/bin/django-admin\", line 8,
> in \n sys.exit(execute_from_command_line())\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/management/__init__.py\",
> line 381, in execute_from_command_line\n utility.execute()\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/management/__init__.py\",
> line 375, in execute\n
> self.fetch_command(subcommand).run_from_argv(self.argv)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/management/base.py\",
> line 323, in run_from_argv\n self.execute(*args, **cmd_options)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/management/base.py\",
> line 364, in execute\n output = self.handle(*args, **options)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/management/base.py\",
> line 83, in wrapped\n res = handle_func(*args, **kwargs)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/management/commands/migrate.py\",
> line 234, in handle\n fake_initial=fake_initial,\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/migrations/executor.py\",
> line 117, in migrate\n state = self._migrate_all_forwards(state, plan,
> full_plan, fake=fake, fake_initial=fake_initial)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/migrations/executor.py\",
> line 147, in _migrate_all_forwards\n state = self.apply_migration(state,
> migration, fake=fake, fake_initial=fake_initial)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/migrations/executor.py\",
> line 245, in apply_migration\n state = migration.apply(state,
> schema_editor)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/migrations/migration.py\",
> line 124, in apply\n operation.database_forwards(self.app_label,
> schema_editor, old_state, project_state)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/migrations/operations/fields.py\",
> line 112, in database_forwards\n field,\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/base/schema.py\",
> line 447, in add_field\n self.execute(sql, params)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/base/schema.py\",
> line 137, in execute\n cursor.execute(sql, params)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py\",
> line 67, in execute\n return self._execute_with_wrappers(sql, params,
> many=False, executor=self._execute)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py\",
> line 76, in _execute_with_wrappers\n return executor(sql, params, many,
> context)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py\",
> line 84, in _execute\n return self.cursor.execute(sql, params)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/utils.py\",
> line 89, in __exit__\n raise dj_exc_value.with_traceback(traceback) from
> exc_value\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py\",
> line 84, in _execute\n return self.cursor.execute(sql,
> params)\ndjango.db.utils.IntegrityError: column
> \"last_sync_revision_number\" contains null values", "stderr_lines":
> ["Traceback (most recent call last):", " File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py\",
> line 84, in _execute", " return self.cursor.execute(sql, params)",
> "psycopg2.errors.NotNullViolation: column \"last_sync_revision_number\"
> contains null values", "", "", "The above exception was the direct cause of
> the following exception:", "", "Traceback 

[Pulp-list] pulpcore 3.4.0 and pulp_file 1.0.0 are Generally Available

[Dennis Kliban]

Pulpcore 3.4.0[0] and pulp_file 1.0.0 [1] have been released. For a list of
changes, please check the changelogs for pulpcore[2] and pulp_file[3].

It should be noted that this release removes SecretCharFields and therefore
until Role-Based Access Control is added to Pulp, REST API is not safe for
multi-user use. Sensitive credentials can be read by any user, e.g.
Remote.password, Remote.client_key.

Here are some of the other highlights:

* Implemented incremental-exporting for PulpExport. #6136
* Added support for S3 and other non-filesystem storage options to pulp
import/export functionality. #6456
* Optimized imports by having repository versions processed using child
tasks. #6484
* Added repository type check during Pulp imports. #6532
* Added version checking to import process. #6558
* Taught PulpExport to export by RepositoryVersions if specified. #6566
* Task groups now have an ‘all_tasks_dispatched’ field which denotes that
no more tasks will spawn as part of this group. #6591
* Taught export how to split export-file into chunk_size bytes. #6736

The only backwards incompatible change is the one mentioned in the notice
above:

* Content of ssl certificates and keys changed to be return their full
value instead of sha256 through REST API. #6691

# Installation and Upgrade

Users should use the 3.4.0 release of pulp_installer (previously known as
ansible-pulp) [5] to install or upgrade their installations. This version
of the installer will check compatibility of all installed plugins with
pulpcore 3.4. The installer will abort if any plugin is incompatible.

# Plugin API

Plugin writers can see the plugin API changes here[4]. Some highlights:

* Added new NoArtifactContentUploadSerializer and
NoArtifactContentUploadViewSet to enable plugin writers to upload content
without storing an Artifact #6281
* Added view_name_pattern to DetailRelatedField and DetailIdentityField to
properly identify wrong resource types. #6521
* Added support for Distributions to provide non-Artifact content via a
content_handler. #6570
* Added constants to the plugin API at pulpcore.plugin.constants. #6579
* TaskGroups now have an ‘all_tasks_dispatched’ field that can be used to
notify systems that no further tasks will be dispatched for a TaskGroup.
Plugin writers should call “.finish()” on all TaskGroups created once they
are done using them to set this field. #6591

And in keeping with the recommended strategy to pin plugins to a 3.y
version of pulpcore, plugins should release compatibility releases with 3.4
as soon as they can. We recommend using "pulpcore>=3.4,<3.5".

[0] https://pypi.org/project/pulpcore/3.4.0/
[1] https://pypi.org/project/pulp-file/1.0.0/
[2] https://docs.pulpproject.org/en/3.4/changes.html#id1
[3] https://pulp-file.readthedocs.io/en/latest/changes.html#id1
[4] https://docs.pulpproject.org/en/3.4/changes.html#plugin-api
[5] https://github.com/pulp/pulp_installer/releases/tag/3.4.0
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] How to disambiguate different platforms?

[Dennis Kliban]

I am not very familiar with Debian/Ubuntu packages but I can speak for
RPMs. When you list or read a single RPM package[0], you can look at the
'release' field to determine which distribution it was built for. The
release is usually an integer followed by the platform. e.g. "1.el7" or
"3.f24".

[0]
https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/content_rpm_packages_read


On Tue, May 19, 2020 at 4:07 AM Christoph Höger <
christoph.hoe...@celeraone.com> wrote:

> Dear all,
>
> I just finished setting up a prototype apt repository using pulp_deb on
> pulpcore 3.3. Now If I understand the workflow documentation correctly, to
> publish a package one has to do these steps:
>
> 1. Upload content
> 2. Create a new repository version (can be done automatically on upload)
> 3. Create a new publication
> 4. Update the distribution to the new publication
> 5. Delete the old publication (here I am not sure if and when this makes
> sense)
>
> While this is somewhat involved, I guess it will work just fine for a
> single repository. But we support at least two different versions of Ubuntu
> (and similarly Redhat) so I wonder how I would disambiguate these different
> platforms.
>
> In particular, when uploading a package the build job knows it is package
> foo, version x.y for platform Z. Package name and version are obviously
> encoded in the package itself, but I see no way to disambiguate the same
> package for Ubuntu 18.04 and 20.04, like a subfolder or some kind of tag.
>
> Is there a typical way to disambiguate these packages by the platforms
> they were built for?
>
> Thanks,
>
> Christoph
> --
> Christoph Höger
>
> CeleraOne GmbH
> Usedomer Straße 4
> 13355 Berlin
>
> Email: christoph.hoe...@celeraone.com 
> Web: www.celeraone.com
>
> Sitz der Gesellschaft: Berlin
> AG Berlin-Charlottenburg HRB 142747
> Geschäftsführer: Moritz Hilger, York Walterscheid
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Bug triage is moving to #pulp-meeting on Freenode

[Dennis Kliban]

The decision to do this was actually made a while ago when we noticed that
holding meetings in #pulp-dev interrupted discussions that were happening
in that channel around the same time as the meeting. However, I did not
follow through to actually announce this decision and update the website.
@bmbouter brought this up again on Friday during open floor.

On Mon, May 18, 2020 at 8:35 AM David Davis  wrote:

> Any chance we can get some background on how, why, where this decision was
> made? I'm not opposed to it but having some more information in this
> announcement would be helpful.
>
> David
>
>
> On Fri, May 15, 2020 at 11:21 AM Dennis Kliban  wrote:
>
>> Starting on May 19th, bug triage will be held in #pulp-meeting on
>> Freenode[0].
>>
>> [0] https://pulpproject.org/get_involved/#meetings
>> ___
>> Pulp-list mailing list
>> Pulp-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] Bug triage is moving to #pulp-meeting on Freenode

[Dennis Kliban]

Starting on May 19th, bug triage will be held in #pulp-meeting on
Freenode[0].

[0] https://pulpproject.org/get_involved/#meetings
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulp 3 live-live setup

[Dennis Kliban]

On Wed, May 13, 2020 at 12:40 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Is it true that both host should have it's own rq in this case?
>
> If the systems are read only, you don't actually need rq. rq is only used
for performing tasks that modify the system. However, you could have
pulpcore-workers (rq) on both systems or any additional systems (to
increase throughput even more). All pulpcore workers need to use the same
settings so that they are connected to the same database and the same
instance of redis. We currently don't support clustered redis.


> Is it true we don't need add cert on both pulp hosts if we'd like enable
> ssl and install cert on load balancer which will forward traffic from 443
> to pulp port 80?
>

That's right. If you are able to configure your load balancer with similar
configs as nginx on your Pulp server, you could remove nginx that's acting
as a reverse proxy from your pulp servers and directly forward traffic to
ports 24816 (content) and 24817 (rest api).


>
> From: dkli...@redhat.com At: 05/13/20 12:28:30
> To: Bin Li (BLOOMBERG/ 120 PARK ) 
> Cc: pulp-list@redhat.com
> Subject: Re: [Pulp-list] pulp 3 live-live setup
>
> On Wed, May 13, 2020 at 12:21 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> Hi All,
>> If we share s3 and external postgres between two pulp host, is it
>> possible to have both up for read only purpose behind a load balancer?
>>
>
> Yes, you just want to make sure that the CONTENT_ORIGIN setting reflects
> the hostname of the load balancer.
>
>
>> ___
>> Pulp-list mailing list
>> Pulp-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulp 3.3.1 server error

[Dennis Kliban]

Could you create a clone of your production system and try upgrading the
cloned environment? I suspect that some problems may have been created from
the extra migration being created in between all the upgrade attempts on
your test system.

Could you provide the SQL schema (without data) from the database of the
system that is experiencing the problem? We want to compare to a clean
installed pulp 3.3.1 schema.

Could you provide the SQL schema (without data) from the database of the
3.2 production system?

On Wed, May 13, 2020 at 11:52 AM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Here is what I have
> # django-admin showmigrations
> admin
> [X] 0001_initial
> [X] 0002_logentry_remove_auto_add
> [X] 0003_logentry_add_action_flag_choices
> auth
> [X] 0001_initial
> [X] 0002_alter_permission_name_max_length
> [X] 0003_alter_user_email_max_length
> [X] 0004_alter_user_username_opts
> [X] 0005_alter_user_last_login_null
> [X] 0006_require_contenttypes_0002
> [X] 0007_alter_validators_add_error_messages
> [X] 0008_alter_user_username_max_length
> [X] 0009_alter_user_last_name_max_length
> [X] 0010_alter_group_name_max_length
> [X] 0011_update_proxy_permissions
> contenttypes
> [X] 0001_initial
> [X] 0002_remove_content_type_name
> core
> [X] 0001_initial
> [X] 0002_increase_artifact_size_field
> [X] 0003_remove_upload_completed
> [X] 0004_add_duplicated_reserved_resources
> [X] 0005_progressreport_code
> [X] 0006_repository_plugin_managed
> [X] 0007_delete_progress_proxies
> [X] 0008_published_metadata_as_content
> [X] 0009_remove_task_non_fatal_errors
> [X] 0010_pulp_fields
> [X] 0011_relative_path
> [X] 0012_auto_20191104_2000
> [X] 0013_repository_pulp_type
> [X] 0014_remove_repository_plugin_managed
> [X] 0015_auto_20191112_1426
> [X] 0016_charfield_to_textfield
> [X] 0017_remove_task_parent
> [X] 0018_auto_20191127_2350
> [X] 0019_add_signing_service_model
> [X] 0020_change_publishedartifact_constraints
> [X] 0021_add_signing_service_foreign_key
> [X] 0022_rename_last_version
> [X] 0023_change_exporter_models
> [X] 0024_use_local_storage_for_uploads
> [X] 0025_task_parent_task
> [X] 0026_task_group
> [X] 0027_export_backend
> [X] 0028_import_importer_pulpimporter_pulpimporterrepository
> file
> [X] 0001_initial
> [X] 0002_file_related_names
> [X] 0003_auto_20191014_1721
> [X] 0004_filefilesystemexporter
> [X] 0005_filerepository
> [X] 0006_delete_filefilesystemexporter
> [X] 0007_filefilesystemexporter
> rpm
> [X] 0001_initial
> [X] 0002_updaterecord_reboot_suggested
> [X] 0003_DATA_incorrect_json
> [X] 0004_add_metadata_signing_service_fk
> [X] 0005_optimize_sync
> [X] 0006_opensuse_support
> [X] 0007_checksum_types
> [X] 0008_advisory_pkg_sumtype_as_int
> [X] 0009_revision_null
> sessions
> [X] 0001_initial
>
>
> From: ipan...@redhat.com At: 05/13/20 11:41:13
> To: Bin Li (BLOOMBERG/ 120 PARK ) 
> Cc: ttere...@redhat.com, pulp-list@redhat.com
> Subject: Re: [Pulp-list] pulp 3.3.1 server error
>
> Can you confirm that migration 0009_revision_null was applied?
>
> 
>
> You can see the migrations with this command 'django-admin showmigrations'
>
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Wed, May 13, 2020 at 4:09 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> Hi Ina,
>>
>> The error was not from migration. I see the error when I try to create a
>> new repo. My pulp_installer is on the latest version. Rerun the installer
>> doesn't solve the issue. What should I do to fix this? We cannot reset
>> database from scratch since we already have a lot data.
>>
>> From: ipan...@redhat.com At: 05/13/20 09:43:45
>> To: Bin Li (BLOOMBERG/ 120 PARK ) 
>> Cc: ttere...@redhat.com, pulp-list@redhat.com
>> Subject: Re: [Pulp-list] pulp 3.3.1 server error
>>
>> This issue was fixed in the 3.3.1 release [0][1]
>>
>> [0] https://pulp-rpm.readthedocs.io/en/latest/changes.html#bugfixes
>> [1] https://pulp.plan.io/issues/6662
>>
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Senior Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Tue, May 12, 2020 at 11:30 PM Bin Li (BLOOMBERG/ 120 PARK) <
>> bli...@bloomberg.net> wrote:
>>
>>> I removed the 0009_auto.. and rerun the installer. The upgrade was
>>> successful.
>>> Below is error log.
>>>
>>> May 12 17:25:05 myhost gunicorn[55426]: pulp: django.request:ERROR:
>>> Internal Server Error: /pulp/api/v3/repositories/rpm/rpm/
>>> May 12 17:25:05 myhost gunicorn[55426]: Traceback (most recent call
>>> last):
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py",
>>> line 84, in _execute
>>> May 12 17:25:05 myhost gunicorn[55426]: 

Re: [Pulp-list] pulp 3 live-live setup

[Dennis Kliban]

On Wed, May 13, 2020 at 12:21 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Hi All,
> If we share s3 and external postgres between two pulp host, is it possible
> to have both up for read only purpose behind a load balancer?
>

Yes, you just want to make sure that the CONTENT_ORIGIN setting reflects
the hostname of the load balancer.


> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] 3.3.1 migration error

[Dennis Kliban]

I filed the issue about the requirements.in file[0]. In the meantime you
could manually install pulp_rpm using pip. Then run 'pulpcore-manager
migrate'.


[0] https://pulp.plan.io/issues/6697

On Fri, May 8, 2020 at 5:48 PM Dennis Kliban  wrote:

> This looks like a bug in pulp_installer. There is supposed to be a newline
> between pulp-rpm-client and pulp-file-client. I'll file the issue and we'll
> try to get a fix out on Monday.
>
> On Fri, May 8, 2020 at 5:35 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> After remove entire pulp installation directory. Now I am getting an error
>>
>> fatal: [pulp3-2]: FAILED! => {"changed": false, "cmd":
>> ["/opt/utils/venv/pulp/3.7.3/bin/pip-compile"], "delta": "0:00:00.778728",
>> "end": "2020-05-08 17:26:32.680701", "failed_when_result": true, "msg":
>> "non-zero return code", "rc": 2, "start": "2020-05-08 17:26:31.901973",
>> "stderr": "Could not find a version that matches
>> pulp-rpm-clientpulp-file-client (from -r requirements.in (line 6))\nNo
>> versions found\nWas
>> https://artifactory.inf.bloomberg.com/artifactory/api/pypi/bloomberg-pypi-ose/simple
>> reachable?", "stderr_lines": ["Could not find a version that matches
>> pulp-rpm-clientpulp-file-client (from -r requirements.in (line 6))", "No
>> versions found", "Was
>> https://artifactory.inf.bloomberg.com/artifactory/api/pypi/bloomberg-pypi-ose/simple
>> reachable?"], "stdout": "", "stdout_lines": []}
>>
>> It looks like this file is wrong
>> # more /opt/utils/venv/pulp/3.7.3/requirements.in
>> pulpcore==3.3.1
>> pulp-file==0.3.0
>> pulp-rpm==3.3.1
>> # Any plugins listed below were already installed but not specified in
>> # pulp_install_plugins
>> pulp-rpm-clientpulp-file-client
>>
>>
>> From: Bin Li (BLOOMBERG/ 120 PARK) At: 05/08/20 17:15:02
>> To: ggai...@redhat.com
>> Cc: pulp-list@redhat.com
>> Subject: Re: [Pulp-list] 3.3.1 migration error
>>
>> No idea where it came from. The timestamp is from 4/25. I run pip
>> uninstall pulp-rpm and it didn't remove it.
>>
>> # ls -ld
>> /opt/utils/venv/pulp/3.7.3/lib/python3.7/site-packages/pulp_rpm/app/migrations/0009_auto_20200425_1932.py
>> -rw-rw-r-- 1 pulp mse-python 492 Apr 25 15:32
>> /opt/utils/venv/pulp/3.7.3/lib/python3.7/site-packages/pulp_rpm/app/migrations/0009_auto_20200425_1932.py
>>
>> # more
>> /opt/utils/venv/pulp/3.7.3/lib/python3.7/site-packages/pulp_rpm/app/migrations/0009_auto_20200425_1932.py
>> # Generated by Django 2.2.12 on 2020-04-25 19:32
>>
>> from django.db import migrations, models
>>
>>
>> class Migration(migrations.Migration):
>>
>> dependencies = [
>> ('rpm', '0008_advisory_pkg_sumtype_as_int'),
>> ]
>>
>> operations = [
>> migrations.AlterField(
>> model_name='updatecollectionpackage',
>> name='sum_type',
>> field=models.PositiveIntegerField(choices=[(0, 0), (1, 1), (2, 2), (3,
>> 3), (4, 4), (5, 5), (6, 6), (7, 7)], null=True),
>> ),
>>
>>
>> From: ggai...@redhat.com At: 05/08/20 16:21:45
>> To: Bin Li (BLOOMBERG/ 120 PARK ) 
>> Cc: pulp-list@redhat.com
>> Subject: Re: [Pulp-list] 3.3.1 migration error
>>
>> Hey there!
>>
>> On Fri, May 8, 2020 at 3:53 PM Bin Li (BLOOMBERG/ 120 PARK) <
>> bli...@bloomberg.net> wrote:
>>
>>> Hi All,
>>> Getting an error to upgrade from 3.3. run 'python manage.py
>>> makemigrations --merge' gave more errors. Anyidea how this can be fixed?
>>>
>>> TASK [pulp-database : Run database auth migrations]
>>> **
>>> fatal: [pulp3-2]: FAILED! => {"changed": true, "cmd":
>>> ["/opt/utils/venv/pulp/3.7.3/bin/django-admin", "migrate", "auth",
>>> "--no-input"], "delta": "0:00:03.534845", "end": "2020-05-08
>>> 15:44:20.362670", "msg": "non-zero return code", "rc": 1, "start":
>>> "2020-05-08 15:44:16.827825", "stderr": "CommandError: Conflicting
>>> migrations detected; multiple leaf nodes in the migration graph:
>>> (0009_revision_null, 0009_auto_20200425_1932 in rpm).\nTo fix them run
>>> 'python manage.py makemigrations --merge'", "stderr_lines": ["CommandError:
>>> Conflicting migrations detected; multiple leaf nodes in the migration
>>> graph: (0009_revision_null, 0009_auto_20200425_1932 in rpm).", "To fix them
>>> run 'python manage.py makemigrations --merge'"], "stdout": "",
>>> "stdout_lines": []}
>>>
>>
>> Where did migration "0009_auto_20200425_1932" come from? Looks like it's
>> conflicting with the delivered 0009_revision_null and causing your problem.
>>
>> G
>> --
>> Grant Gainey
>> Principal Software Engineer, Red Hat System Management Engineering
>>
>>
>> ___
>> Pulp-list mailing list
>> Pulp-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] 3.3.1 migration error

[Dennis Kliban]

This looks like a bug in pulp_installer. There is supposed to be a newline
between pulp-rpm-client and pulp-file-client. I'll file the issue and we'll
try to get a fix out on Monday.

On Fri, May 8, 2020 at 5:35 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> After remove entire pulp installation directory. Now I am getting an error
>
> fatal: [pulp3-2]: FAILED! => {"changed": false, "cmd":
> ["/opt/utils/venv/pulp/3.7.3/bin/pip-compile"], "delta": "0:00:00.778728",
> "end": "2020-05-08 17:26:32.680701", "failed_when_result": true, "msg":
> "non-zero return code", "rc": 2, "start": "2020-05-08 17:26:31.901973",
> "stderr": "Could not find a version that matches
> pulp-rpm-clientpulp-file-client (from -r requirements.in (line 6))\nNo
> versions found\nWas
> https://artifactory.inf.bloomberg.com/artifactory/api/pypi/bloomberg-pypi-ose/simple
> reachable?", "stderr_lines": ["Could not find a version that matches
> pulp-rpm-clientpulp-file-client (from -r requirements.in (line 6))", "No
> versions found", "Was
> https://artifactory.inf.bloomberg.com/artifactory/api/pypi/bloomberg-pypi-ose/simple
> reachable?"], "stdout": "", "stdout_lines": []}
>
> It looks like this file is wrong
> # more /opt/utils/venv/pulp/3.7.3/requirements.in
> pulpcore==3.3.1
> pulp-file==0.3.0
> pulp-rpm==3.3.1
> # Any plugins listed below were already installed but not specified in
> # pulp_install_plugins
> pulp-rpm-clientpulp-file-client
>
>
> From: Bin Li (BLOOMBERG/ 120 PARK) At: 05/08/20 17:15:02
> To: ggai...@redhat.com
> Cc: pulp-list@redhat.com
> Subject: Re: [Pulp-list] 3.3.1 migration error
>
> No idea where it came from. The timestamp is from 4/25. I run pip
> uninstall pulp-rpm and it didn't remove it.
>
> # ls -ld
> /opt/utils/venv/pulp/3.7.3/lib/python3.7/site-packages/pulp_rpm/app/migrations/0009_auto_20200425_1932.py
> -rw-rw-r-- 1 pulp mse-python 492 Apr 25 15:32
> /opt/utils/venv/pulp/3.7.3/lib/python3.7/site-packages/pulp_rpm/app/migrations/0009_auto_20200425_1932.py
>
> # more
> /opt/utils/venv/pulp/3.7.3/lib/python3.7/site-packages/pulp_rpm/app/migrations/0009_auto_20200425_1932.py
> # Generated by Django 2.2.12 on 2020-04-25 19:32
>
> from django.db import migrations, models
>
>
> class Migration(migrations.Migration):
>
> dependencies = [
> ('rpm', '0008_advisory_pkg_sumtype_as_int'),
> ]
>
> operations = [
> migrations.AlterField(
> model_name='updatecollectionpackage',
> name='sum_type',
> field=models.PositiveIntegerField(choices=[(0, 0), (1, 1), (2, 2), (3, 3),
> (4, 4), (5, 5), (6, 6), (7, 7)], null=True),
> ),
>
>
> From: ggai...@redhat.com At: 05/08/20 16:21:45
> To: Bin Li (BLOOMBERG/ 120 PARK ) 
> Cc: pulp-list@redhat.com
> Subject: Re: [Pulp-list] 3.3.1 migration error
>
> Hey there!
>
> On Fri, May 8, 2020 at 3:53 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> Hi All,
>> Getting an error to upgrade from 3.3. run 'python manage.py
>> makemigrations --merge' gave more errors. Anyidea how this can be fixed?
>>
>> TASK [pulp-database : Run database auth migrations]
>> **
>> fatal: [pulp3-2]: FAILED! => {"changed": true, "cmd":
>> ["/opt/utils/venv/pulp/3.7.3/bin/django-admin", "migrate", "auth",
>> "--no-input"], "delta": "0:00:03.534845", "end": "2020-05-08
>> 15:44:20.362670", "msg": "non-zero return code", "rc": 1, "start":
>> "2020-05-08 15:44:16.827825", "stderr": "CommandError: Conflicting
>> migrations detected; multiple leaf nodes in the migration graph:
>> (0009_revision_null, 0009_auto_20200425_1932 in rpm).\nTo fix them run
>> 'python manage.py makemigrations --merge'", "stderr_lines": ["CommandError:
>> Conflicting migrations detected; multiple leaf nodes in the migration
>> graph: (0009_revision_null, 0009_auto_20200425_1932 in rpm).", "To fix them
>> run 'python manage.py makemigrations --merge'"], "stdout": "",
>> "stdout_lines": []}
>>
>
> Where did migration "0009_auto_20200425_1932" come from? Looks like it's
> conflicting with the delivered 0009_revision_null and causing your problem.
>
> G
> --
> Grant Gainey
> Principal Software Engineer, Red Hat System Management Engineering
>
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] 3.3.1 migration error

[Dennis Kliban]

I suspect that migration got created when I suggested doing a manual
workaround and setting the null=True on the migration. I actually don't
know how that would have happened because I suggested modifying an existing
migration. Anyway, I now realize that was bad advice.

On Fri, May 8, 2020 at 4:22 PM Grant Gainey  wrote:

> Hey there!
>
> On Fri, May 8, 2020 at 3:53 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> Hi All,
>> Getting an error to upgrade from 3.3. run 'python manage.py
>> makemigrations --merge' gave more errors. Anyidea how this can be fixed?
>>
>> TASK [pulp-database : Run database auth migrations]
>> **
>> fatal: [pulp3-2]: FAILED! => {"changed": true, "cmd":
>> ["/opt/utils/venv/pulp/3.7.3/bin/django-admin", "migrate", "auth",
>> "--no-input"], "delta": "0:00:03.534845", "end": "2020-05-08
>> 15:44:20.362670", "msg": "non-zero return code", "rc": 1, "start":
>> "2020-05-08 15:44:16.827825", "stderr": "CommandError: Conflicting
>> migrations detected; multiple leaf nodes in the migration graph:
>> (0009_revision_null, 0009_auto_20200425_1932 in rpm).\nTo fix them run
>> 'python manage.py makemigrations --merge'", "stderr_lines": ["CommandError:
>> Conflicting migrations detected; multiple leaf nodes in the migration
>> graph: (0009_revision_null, 0009_auto_20200425_1932 in rpm).", "To fix them
>> run 'python manage.py makemigrations --merge'"], "stdout": "",
>> "stdout_lines": []}
>>
>
> Where did migration "0009_auto_20200425_1932" come from? Looks like it's
> conflicting with the delivered 0009_revision_null and causing your problem.
>
> G
> --
> Grant Gainey
> Principal Software Engineer, Red Hat System Management Engineering
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] pulp_rpm 3.3.1 GA

[Dennis Kliban]

pulp_rpm 3.3.1 has been released. It is compatible with pulpcore 3.3.

This bugfix release includes:

   - Taught copy to always include specified packages. #6519
   
   - Fixed the upgrade issue, revision number can be empty now. #6662
   


PyPI: https://pypi.org/project/pulp-rpm/3.3.1/
Changelog: https://pulp-rpm.readthedocs.io/en/3.3/changes.html#id1
Docs: https://pulp-rpm.readthedocs.io/
Python bindings: https://pypi.org/project/pulp-rpm-client/3.3.1/
Ruby bindings: https://rubygems.org/gems/pulp_rpm_client/versions/3.3.1/
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] [Pulp-dev] Pulp 3 CLI MVP

[Dennis Kliban]

On Thu, May 7, 2020 at 7:13 AM Tatiana Tereshchenko 
wrote:

> +1 to `pulp` command.
>
> I think for me as a user, the most logical would be to have a plugin name
> first and then follow the URL pattern.
> The majority of commands are plugin specific. If I work with multiple
> plugins, it also makes it easy for me as a user to just change the plugin
> name in front for the commands which have the same structure in different
> plugins.
> It also makes it visually clear that I work with a specific plugin, in
> comparison to when the plugin name is somewhere in the 3rd/4th place.
> It will also allow not to guess in commands like the `pulp repositories
> rpm rpm`  which one is the plugin name and which one is a repo type.
>
> I agree that this would make much more clear to the user which 'rpm' is
the plugin type and which 'rpm' is the resource type.


> +1 for
> pulp rpm content packages
> pulp rpm repositories rpm
> pulp rpm repositories mirror
> ...
>
> On Wed, May 6, 2020 at 7:58 PM Dennis Kliban  wrote:
>
>> On Wed, May 6, 2020 at 12:30 PM David Davis 
>> wrote:
>>
>>> Matthias and I met today to go over some plans for a prototype. I wrote
>>> some notes[0] down. As part of the prototype, we'd propose two deliverables
>>> (one this week and one next week):
>>>
>>> 1. A set of ~2-3 click commands that use the bindings to interact with
>>> Pulp
>>> 2. Some openapi-generator templates that will be able to generate such
>>> commands from the schema
>>>
>>> There is a question we had about how the commands for typed resources
>>> will be structured in the CLI. To illustrate with two endpoints:
>>>
>>>
>> We should call the command 'pulp' instead of pulp-cli.
>>
>>
>>> # rpm.package content (/pulp/api/v3/content/rpm/packages/):
>>> - pulp-cli rpm content packages ...
>>> - pulp-cli content rpm packages ...
>>> - pulp-cli rpm packages content ...
>>> - ???
>>>
>>
>> I was thinking we should structure the commands similar to the URLs in
>> the REST API.
>>
>> pulp content rpm packages
>>
>>
>>>
>>> # file.file repositories (/pulp/api/v3/repositories/file/file/):
>>> - pulp-cli file repositories file ...
>>> - pulp-cli repositories file file ...
>>> - pulp-cli file file repositories ...
>>> - ???
>>>
>>> pulp repositories file
>>
>> Plugins that provide multiple types of the same resource would need to be
>> more descriptive. Though I can see a practical reason for requiring all
>> resources to be that descriptive.
>>
>> pulp repositories rpm rpm
>> pulp repositories rpm mirror
>>
>>
>>
>>> [0] https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig?view#Prototype
>>>
>>> David
>>>
>>>
>>> On Thu, Apr 30, 2020 at 1:42 PM David Davis 
>>> wrote:
>>>
>>>> Today we met to discuss some ideas for a technical design for how the
>>>> CLI would work. Here's a copy of our notes:
>>>>
>>>> https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig#Technical-discussion
>>>>
>>>> And there is a rough design in the document as well:
>>>>
>>>> https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig#Design
>>>>
>>>> I have also entered the CLI user stories from our meeting last week
>>>> into redmine under the Pulp CLI project:
>>>>
>>>> https://pulp.plan.io/versions/93
>>>>
>>>> And I've filed a user story that we talked about today that would
>>>> handle sync, publish, and distribution of repos. Feedback welcome:
>>>>
>>>> https://pulp.plan.io/issues/6626
>>>>
>>>> Matthias and I are planning to meet next week to look at creating a
>>>> proof of concept that would provide 2-3 commands. If anyone is interested
>>>> in joining us, please let me know and I can add you.
>>>>
>>>> David
>>>>
>>>>
>>>> On Tue, Apr 28, 2020 at 8:06 AM David Davis 
>>>> wrote:
>>>>
>>>>> I've also started working on some questions about how the CLI will
>>>>> work. Feel free to add some of your own:
>>>>>
>>>>> https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig?view#Technical-discussion
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>> On Tue, Apr 28, 2020 at 8:05 AM David Davis 
>>>>> wrote:
>>>>>
>&g

Re: [Pulp-list] [Pulp-dev] Pulp 3 CLI MVP

[Dennis Kliban]

On Wed, May 6, 2020 at 12:30 PM David Davis  wrote:

> Matthias and I met today to go over some plans for a prototype. I wrote
> some notes[0] down. As part of the prototype, we'd propose two deliverables
> (one this week and one next week):
>
> 1. A set of ~2-3 click commands that use the bindings to interact with Pulp
> 2. Some openapi-generator templates that will be able to generate such
> commands from the schema
>
> There is a question we had about how the commands for typed resources will
> be structured in the CLI. To illustrate with two endpoints:
>
>
We should call the command 'pulp' instead of pulp-cli.


> # rpm.package content (/pulp/api/v3/content/rpm/packages/):
> - pulp-cli rpm content packages ...
> - pulp-cli content rpm packages ...
> - pulp-cli rpm packages content ...
> - ???
>

I was thinking we should structure the commands similar to the URLs in the
REST API.

pulp content rpm packages


>
> # file.file repositories (/pulp/api/v3/repositories/file/file/):
> - pulp-cli file repositories file ...
> - pulp-cli repositories file file ...
> - pulp-cli file file repositories ...
> - ???
>
> pulp repositories file

Plugins that provide multiple types of the same resource would need to be
more descriptive. Though I can see a practical reason for requiring all
resources to be that descriptive.

pulp repositories rpm rpm
pulp repositories rpm mirror



> [0] https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig?view#Prototype
>
> David
>
>
> On Thu, Apr 30, 2020 at 1:42 PM David Davis  wrote:
>
>> Today we met to discuss some ideas for a technical design for how the CLI
>> would work. Here's a copy of our notes:
>>
>> https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig#Technical-discussion
>>
>> And there is a rough design in the document as well:
>>
>> https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig#Design
>>
>> I have also entered the CLI user stories from our meeting last week into
>> redmine under the Pulp CLI project:
>>
>> https://pulp.plan.io/versions/93
>>
>> And I've filed a user story that we talked about today that would handle
>> sync, publish, and distribution of repos. Feedback welcome:
>>
>> https://pulp.plan.io/issues/6626
>>
>> Matthias and I are planning to meet next week to look at creating a proof
>> of concept that would provide 2-3 commands. If anyone is interested in
>> joining us, please let me know and I can add you.
>>
>> David
>>
>>
>> On Tue, Apr 28, 2020 at 8:06 AM David Davis 
>> wrote:
>>
>>> I've also started working on some questions about how the CLI will work.
>>> Feel free to add some of your own:
>>>
>>> https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig?view#Technical-discussion
>>>
>>> David
>>>
>>>
>>> On Tue, Apr 28, 2020 at 8:05 AM David Davis 
>>> wrote:
>>>
 I have set up a meeting to discuss the CLI technical design. Below are
 the details. I think a video conference might be easier for technical
 discussion but am open to consider meeting on #pulp-meeting again.

 URL: https://meet.google.com/vgx-bzbb-wnh
 Date/time: April 30, 2020 at 9:00am ET (1pm UTC)

 David


 On Fri, Apr 24, 2020 at 10:29 AM David Davis 
 wrote:

> Today we met in #pulp-meeting on freenode to discuss the user stories
> for a Pulp 3 CLI MVP. The document with the user stories is available
> below. I'd like to ask for any feedback from users or plugin writers.
>
> The goal of the CLI MVP is to cover the pulp_file happy path (sync,
> publish, distribute) and make it possible for plugin writers to generate
> and write their own commands. I'm imagining that plugins will release 
> their
> own sets of CLI commands after we complete the initial MVP.
>
> https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig
>
> Feedback is welcome. I plan to enter these user stories into redmine
> next week.
>
> David
>
 ___
> Pulp-dev mailing list
> pulp-...@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Create distribution issues

[Dennis Kliban]

On Tue, May 5, 2020 at 10:40 AM Heide, Todd 
wrote:

> New install of 3.3.0 and I can create the repo,
>
> # http POST http://localhost:24817/pulp/api/v3/repositories/file/file/
> name=foo
> HTTP/1.1 201 Created
> Allow: GET, POST, HEAD, OPTIONS
> Connection: close
> Content-Length: 376
> Content-Type: application/json
> Date: Tue, 05 May 2020 14:04:24 GMT
> Location:
> /pulp/api/v3/repositories/file/file/7885baed-b707-4002-b1e1-a98924e39ee5/
> Server: gunicorn/20.0.4
> Vary: Accept, Cookie
> X-Frame-Options: SAMEORIGIN
>
> {
> "description": null,
> "latest_version_href":
> "/pulp/api/v3/repositories/file/file/7885baed-b707-4002-b1e1-a98924e39ee5/versions/0/",
> "name": "foo",
> "pulp_created": "2020-05-05T14:04:24.441705Z",
> "pulp_href":
> "/pulp/api/v3/repositories/file/file/7885baed-b707-4002-b1e1-a98924e39ee5/",
> "versions_href":
> "/pulp/api/v3/repositories/file/file/7885baed-b707-4002-b1e1-a98924e39ee5/versions/"
> }
>
> Export the repo
> # export REPO_HREF=$(http :24817/pulp/api/v3/repositories/file/file/ | jq
> -r '.results[] | select(.name == "foo") | .pulp_href')
>
> But when I try to make the distribution it errors 404
>
> # http POST :24817/pulp/api/v3/distributions/file/file/ name='baz'
> base_path='mypath' repository=$REPO_HREF``
> HTTP/1.1 400 Bad Request
> Allow: GET, POST, HEAD, OPTIONS
> Connection: close
> Content-Length: 35
> Content-Type: application/json
> Date: Tue, 05 May 2020 14:05:41 GMT
> Server: gunicorn/20.0.4
> Vary: Accept, Cookie
> X-Frame-Options: SAMEORIGIN
>
> {
> "repository": [
> "Unexpected field"
> ]
> }
>
>
FileDsitribution requires a publication not a repository. Instructions for
that are here:
https://pulp-file.readthedocs.io/en/latest/workflows/publish-host.html#publish-and-host


> Did something get missed during the install?   I also tried manual
> distribution, fails at AnsibleRemote
> http POST :24817/pulp/api/v3/remotes/ansible/ansible/ name=bar url='
> https://galaxy.ansible.com/api/v1/roles/?
> > namespace__name=elastic'
> HTTP/1.1 404 Not Found
> Connection: close
> Content-Length: 77
> Content-Type: text/html
> Date: Tue, 05 May 2020 14:23:20 GMT
> Server: gunicorn/20.0.4
> X-Frame-Options: SAMEORIGIN
>
> Not FoundThe requested resource was not found on this
> server.
>
> # http POST :24817/pulp/api/v3/distributions/file/file/ name='baz'
> base_path='dev' repository_version=REPO_VERSION_HREF
> HTTP/1.1 400 Bad Request
> Allow: GET, POST, HEAD, OPTIONS
> Connection: close
> Content-Length: 43
> Content-Type: application/json
> Date: Tue, 05 May 2020 14:24:42 GMT
> Server: gunicorn/20.0.4
> Vary: Accept, Cookie
> X-Frame-Options: SAMEORIGIN
>
> {
> "repository_version": [
> "Unexpected field"
> ]
> }
>
> The only one that seemed to have done anything is manual distribution.
>
> # http POST :24817/pulp/api/v3/distributions/file/file/ name='baz'
> base_path='bar' publication=$PUBLICATION_HREF
> HTTP/1.1 202 Accepted
> Allow: GET, POST, HEAD, OPTIONS
> Connection: close
> Content-Length: 67
> Content-Type: application/json
> Date: Tue, 05 May 2020 14:35:20 GMT
> Server: gunicorn/20.0.4
> Vary: Accept, Cookie
> X-Frame-Options: SAMEORIGIN
>
> {
> "task": "/pulp/api/v3/tasks/4c752d6b-9102-4105-b84c-7b523439914a/"
> }
>
>
> So out of the three on this page,
> https://docs.pulpproject.org/workflows/exposing-content.html the only one
> to work is manual distribution of a publication. I can view and download
> from the web page.   Did I miss a plugin or a configuration somewhere?
>
>
A File repository version contains all the files. However, File
publications always include a manifest file that lists all the files and
checksums for those files. The manifest is generated only when a
publication is created from a repository version.

RPM repositories work the same way and always require generating a
publication first also.



>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulp 3.3 db migration

[Dennis Kliban]

Another user filed an issue about this problem[0].

[0] https://pulp.plan.io/issues/6662

On Mon, May 4, 2020 at 4:29 PM Dennis Kliban  wrote:

> You could manually add null=True (keyword argument) to the definition of
> the last_sync_revision_number here[0].
>
> [0]
> https://github.com/pulp/pulp_rpm/blob/3.3.0/pulp_rpm/app/migrations/0005_optimize_sync.py#L28
>
> On Mon, May 4, 2020 at 4:19 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> Is there a workaround or I cannot upgrade before this is fixed?
>>
>> Thanks
>>
>> From: dkli...@redhat.com At: 05/04/20 15:52:16
>> To: Bin Li (BLOOMBERG/ 120 PARK ) 
>> Cc: pulp-list@redhat.com
>> Subject: Re: [Pulp-list] pulp 3.3 db migration
>>
>> Please file a bug.
>>
>> This is a problem with a migration that adds the ability to track when an
>> RPM repository can skip syncing from the remote repository because the
>> metadata is the same as it was during the previous sync. The migration is
>> failing to provide a default value for databases that are being upgraded.
>>
>> On Mon, May 4, 2020 at 3:09 PM Bin Li (BLOOMBERG/ 120 PARK) <
>> bli...@bloomberg.net> wrote:
>>
>>> Hi,
>>>
>>> Keep getting this error from ansible_install to upgrade from 3.2 to 3.3.
>>> What could cause this and how can we fix it?
>>>
>>> Thanks
>>>
>>>
>>> TASK [pulp-database : Run database migrations]
>>> **
>>> fatal: [pulp3-1]: FAILED! => {"changed": true, "cmd":
>>> ["/opt/utils/venv/pulp/3.7.3/bin/django -admin", "migrate", "--no-input"],
>>> "delta": "0:00:03.879347", "end": "2020-05-04 14:54:18.580 273", "msg":
>>> "non-zero return code", "rc": 1, "start": "2020-05-04 14:54:14.700926",
>>> "stderr" : "Traceback (most recent call last):\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/si
>>> te-packages/django/db/backends/utils.py\", line 84, in _execute\n return
>>> self.cursor.execu te(sql, params)\npsycopg2.errors.NotNullViolation: column
>>> \"last_sync_revision_number\" conta ins null values\n\n\nThe above
>>> exception was the direct cause of the following exception:\n\n Traceback
>>> (most recent call last):\n File
>>> \"/opt/utils/venv/pulp/3.7.3/bin/django-admin\", l ine 8, in \n
>>> sys.exit(execute_from_command_line())\n File \"/opt/utils/venv/pulp/
>>> 3.7.3/lib64/python3.7/site-packages/django/core/management/__init__.py\",
>>> line 381, in execut e_from_command_line\n utility.execute()\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3
>>> .7/site-packages/django/core/management/__init__.py\", line 375, in
>>> execute\n self.fetch_c ommand(subcommand).run_from_argv(self.argv)\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python
>>> 3.7/site-packages/django/core/management/base.py\", line 323, in
>>> run_from_argv\n self.exec ute(*args, **cmd_options)\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/
>>> django/core/management/base.py\", line 364, in execute\n output =
>>> self.handle(*args, **opt ions)\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/manageme
>>> nt/base.py\", line 83, in wrapped\n res = handle_func(*args, **kwargs)\n
>>> File \"/opt/util
>>> s/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/management/commands/migrate.py\",
>>> line 234, in handle\n fake_initial=fake_initial,\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib
>>> 64/python3.7/site-packages/django/db/migrations/executor.py\", line 117, in
>>> migrate\n stat e = self._migrate_all_forwards(state, plan, full_plan,
>>> fake=fake, fake_initial=fake_initial)\ n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/migrations/execu
>>> tor.py\", line 147, in _migrate_all_forwards\n state =
>>> self.apply_migration(state, migrati on, fake=fake,
>>> fake_initial=fake_initial)\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.
>>> 7/site-packages/django/db/migrations/executor.py\", line 245, in
>>> apply_migration\n state = migration.apply(state, schema_editor)\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/s
>>> ite-packages/django/db/migrati

Re: [Pulp-list] pulp 3.3 db migration

[Dennis Kliban]

You could manually add null=True (keyword argument) to the definition of
the last_sync_revision_number here[0].

[0]
https://github.com/pulp/pulp_rpm/blob/3.3.0/pulp_rpm/app/migrations/0005_optimize_sync.py#L28

On Mon, May 4, 2020 at 4:19 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Is there a workaround or I cannot upgrade before this is fixed?
>
> Thanks
>
> From: dkli...@redhat.com At: 05/04/20 15:52:16
> To: Bin Li (BLOOMBERG/ 120 PARK ) 
> Cc: pulp-list@redhat.com
> Subject: Re: [Pulp-list] pulp 3.3 db migration
>
> Please file a bug.
>
> This is a problem with a migration that adds the ability to track when an
> RPM repository can skip syncing from the remote repository because the
> metadata is the same as it was during the previous sync. The migration is
> failing to provide a default value for databases that are being upgraded.
>
> On Mon, May 4, 2020 at 3:09 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> Hi,
>>
>> Keep getting this error from ansible_install to upgrade from 3.2 to 3.3.
>> What could cause this and how can we fix it?
>>
>> Thanks
>>
>>
>> TASK [pulp-database : Run database migrations]
>> **
>> fatal: [pulp3-1]: FAILED! => {"changed": true, "cmd":
>> ["/opt/utils/venv/pulp/3.7.3/bin/django -admin", "migrate", "--no-input"],
>> "delta": "0:00:03.879347", "end": "2020-05-04 14:54:18.580 273", "msg":
>> "non-zero return code", "rc": 1, "start": "2020-05-04 14:54:14.700926",
>> "stderr" : "Traceback (most recent call last):\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/si
>> te-packages/django/db/backends/utils.py\", line 84, in _execute\n return
>> self.cursor.execu te(sql, params)\npsycopg2.errors.NotNullViolation: column
>> \"last_sync_revision_number\" conta ins null values\n\n\nThe above
>> exception was the direct cause of the following exception:\n\n Traceback
>> (most recent call last):\n File
>> \"/opt/utils/venv/pulp/3.7.3/bin/django-admin\", l ine 8, in \n
>> sys.exit(execute_from_command_line())\n File \"/opt/utils/venv/pulp/
>> 3.7.3/lib64/python3.7/site-packages/django/core/management/__init__.py\",
>> line 381, in execut e_from_command_line\n utility.execute()\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3
>> .7/site-packages/django/core/management/__init__.py\", line 375, in
>> execute\n self.fetch_c ommand(subcommand).run_from_argv(self.argv)\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python
>> 3.7/site-packages/django/core/management/base.py\", line 323, in
>> run_from_argv\n self.exec ute(*args, **cmd_options)\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/
>> django/core/management/base.py\", line 364, in execute\n output =
>> self.handle(*args, **opt ions)\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/manageme
>> nt/base.py\", line 83, in wrapped\n res = handle_func(*args, **kwargs)\n
>> File \"/opt/util
>> s/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/management/commands/migrate.py\",
>> line 234, in handle\n fake_initial=fake_initial,\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib
>> 64/python3.7/site-packages/django/db/migrations/executor.py\", line 117, in
>> migrate\n stat e = self._migrate_all_forwards(state, plan, full_plan,
>> fake=fake, fake_initial=fake_initial)\ n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/migrations/execu
>> tor.py\", line 147, in _migrate_all_forwards\n state =
>> self.apply_migration(state, migrati on, fake=fake,
>> fake_initial=fake_initial)\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.
>> 7/site-packages/django/db/migrations/executor.py\", line 245, in
>> apply_migration\n state = migration.apply(state, schema_editor)\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/s
>> ite-packages/django/db/migrations/migration.py\", line 124, in apply\n
>> operation.database_ forwards(self.app_label, schema_editor, old_state,
>> project_state)\n File \"/opt/utils/venv/p
>> ulp/3.7.3/lib64/python3.7/site-packages/django/db/migrations/operations/fields.py\",
>> line 112 , in database_forwards\n field,\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-
>> packages/django/db/backends/base/schema.py\", line 447, in add_field\n
>> self.execute(sql, p arams)\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/
>> base/schema.py\", line 137, in execute\n cursor.execute(sql, params)\n File
>> \"/opt/utils/
>> venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py\",
>> line 67, in exec ute\n return self._execute_with_wrappers(sql, params,
>> many=False, executor=self._execute)\ n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.p
>> y\", line 76, in _execute_with_wrappers\n return executor(sql, params,
>> many, context)\n F ile
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py\",
>> line 84, 

Re: [Pulp-list] pulp 3.3 db migration

[Dennis Kliban]

Please file a bug.

This is a problem with a migration that adds the ability to track when an
RPM repository can skip syncing from the remote repository because the
metadata is the same as it was during the previous sync. The migration is
failing to provide a default value for databases that are being upgraded.

On Mon, May 4, 2020 at 3:09 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Hi,
>
> Keep getting this error from ansible_install to upgrade from 3.2 to 3.3.
> What could cause this and how can we fix it?
>
> Thanks
>
>
> TASK [pulp-database : Run database migrations]
> **
> fatal: [pulp3-1]: FAILED! => {"changed": true, "cmd":
> ["/opt/utils/venv/pulp/3.7.3/bin/django -admin", "migrate", "--no-input"],
> "delta": "0:00:03.879347", "end": "2020-05-04 14:54:18.580 273", "msg":
> "non-zero return code", "rc": 1, "start": "2020-05-04 14:54:14.700926",
> "stderr" : "Traceback (most recent call last):\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/si
> te-packages/django/db/backends/utils.py\", line 84, in _execute\n return
> self.cursor.execu te(sql, params)\npsycopg2.errors.NotNullViolation: column
> \"last_sync_revision_number\" conta ins null values\n\n\nThe above
> exception was the direct cause of the following exception:\n\n Traceback
> (most recent call last):\n File
> \"/opt/utils/venv/pulp/3.7.3/bin/django-admin\", l ine 8, in \n
> sys.exit(execute_from_command_line())\n File \"/opt/utils/venv/pulp/
> 3.7.3/lib64/python3.7/site-packages/django/core/management/__init__.py\",
> line 381, in execut e_from_command_line\n utility.execute()\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3
> .7/site-packages/django/core/management/__init__.py\", line 375, in
> execute\n self.fetch_c ommand(subcommand).run_from_argv(self.argv)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python
> 3.7/site-packages/django/core/management/base.py\", line 323, in
> run_from_argv\n self.exec ute(*args, **cmd_options)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/
> django/core/management/base.py\", line 364, in execute\n output =
> self.handle(*args, **opt ions)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/manageme
> nt/base.py\", line 83, in wrapped\n res = handle_func(*args, **kwargs)\n
> File \"/opt/util
> s/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/management/commands/migrate.py\",
> line 234, in handle\n fake_initial=fake_initial,\n File
> \"/opt/utils/venv/pulp/3.7.3/lib
> 64/python3.7/site-packages/django/db/migrations/executor.py\", line 117, in
> migrate\n stat e = self._migrate_all_forwards(state, plan, full_plan,
> fake=fake, fake_initial=fake_initial)\ n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/migrations/execu
> tor.py\", line 147, in _migrate_all_forwards\n state =
> self.apply_migration(state, migrati on, fake=fake,
> fake_initial=fake_initial)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.
> 7/site-packages/django/db/migrations/executor.py\", line 245, in
> apply_migration\n state = migration.apply(state, schema_editor)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/s
> ite-packages/django/db/migrations/migration.py\", line 124, in apply\n
> operation.database_ forwards(self.app_label, schema_editor, old_state,
> project_state)\n File \"/opt/utils/venv/p
> ulp/3.7.3/lib64/python3.7/site-packages/django/db/migrations/operations/fields.py\",
> line 112 , in database_forwards\n field,\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-
> packages/django/db/backends/base/schema.py\", line 447, in add_field\n
> self.execute(sql, p arams)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/
> base/schema.py\", line 137, in execute\n cursor.execute(sql, params)\n File
> \"/opt/utils/
> venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py\",
> line 67, in exec ute\n return self._execute_with_wrappers(sql, params,
> many=False, executor=self._execute)\ n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.p
> y\", line 76, in _execute_with_wrappers\n return executor(sql, params,
> many, context)\n F ile
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py\",
> line 84, in _execute\n return self.cursor.execute(sql, params)\n File
> \"/opt/utils/venv/
> pulp/3.7.3/lib64/python3.7/site-packages/django/db/utils.py\", line 89, in
> __exit__\n rais e dj_exc_value.with_traceback(traceback) from exc_value\n
> File \"/opt/utils/venv/pulp/3.7.3/
> lib64/python3.7/site-packages/django/db/backends/utils.py\", line 84, in
> _execute\n return self.cursor.execute(sql,
> params)\ndjango.db.utils.IntegrityError: column \"last_sync_revisio
> n_number\" contains null values", "stderr_lines": ["Traceback (most recent
> call last):", " F ile
> 

Re: [Pulp-list] pulp 3.2.1 duplicate key error

[Dennis Kliban]

I accidently reproduced the issue[0]. We should now be able to fix it.

[0] https://pulp.plan.io/issues/6463#note-5


On Mon, Apr 20, 2020 at 11:25 AM Dennis Kliban  wrote:

> The issue does not have any steps to reproduce it. Have you been able to
> consistently reproduce the issue with a specific repository? Does it go
> away the next time you perform the sync?
>
> On Wed, Apr 8, 2020 at 9:03 AM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> Thanks Brian. I filed an issue https://pulp.plan.io/issues/6463
>>
>> From: bmbou...@redhat.com At: 04/07/20 15:59:11
>> To: Bin Li (BLOOMBERG/ 120 PARK ) 
>> Cc: pulp-list@redhat.com
>> Subject: Re: [Pulp-list] pulp 3.2.1 duplicate key error
>>
>> I heard another developer report a similar issue, but we couldn't
>> reproduce it. Could you file this as an issue also please?
>>
>> On Tue, Apr 7, 2020 at 3:42 PM Bin Li (BLOOMBERG/ 120 PARK) <
>> bli...@bloomberg.net> wrote:
>>
>>> Noticed we have a few errors when running sync.
>>>
>>> "error": {
>>> "description": "duplicate key value violates unique constraint
>>> \"core_repositoryversion_repository_id_number_3c54ce50_uniq\"\nDETAIL: Key
>>> (repository_id, number)=(59eb02b1-edab-46e3-a69b-d69a8b314f20, 2) already
>>> exists.\n",
>>>
>>> What could be the cause of this? How can we resolve it?
>>> ___
>>> Pulp-list mailing list
>>> Pulp-list@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>>
>> ___
>> Pulp-list mailing list
>> Pulp-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] upgrade to 3.3

[Dennis Kliban]

It seems like you are using a very old version of pulp_installer (formerly
known as ansible-pulp). What commit are you at?

You should be using pulp_installer 3.3.0 to do the install[0].

[0] https://github.com/pulp/pulp_installer/releases/tag/3.3.0

On Wed, Apr 29, 2020 at 8:12 AM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Hi Fabricio,
>
> I got the errors below. It looks it only update pulpcore but not the
> plugins.
>
> RUNNING HANDLER [pulp : Collect static content]
> 
> fatal: [pulp3]: FAILED! => {"changed": true, "cmd":
> ["/opt/utils/venv/pulp/3.7.3/bin/django-admin", "collectstatic",
> "--noinput", "--link"], "delta": "0:00:00.250852", "end": "2020-04-29
> 08:08:45.007362", "msg": "non-zero return code", "rc": 1, "start":
> "2020-04-29 08:08:44.756510", "stderr": "Traceback (most recent call
> last):\n File \"/opt/utils/venv/pulp/3.7.3/bin/django-admin\", line 8, in
> \n sys.exit(execute_from_command_line())\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/management/__init__.py\",
> line 381, in execute_from_command_line\n utility.execute()\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/management/__init__.py\",
> line 325, in execute\n settings.INSTALLED_APPS\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/conf/__init__.py\",
> line 79, in __getattr__\n self._setup(name)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/conf/__init__.py\",
> line 66, in _setup\n self._wrapped = Settings(settings_module)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/conf/__init__.py\",
> line 157, in __init__\n mod =
> importlib.import_module(self.SETTINGS_MODULE)\n File
> \"/opt/python/3.7.3/lib64/python3.7/importlib/__init__.py\", line 127, in
> import_module\n return _bootstrap._gcd_import(name[level:], package,
> level)\n File \"\", line 1006, in
> _gcd_import\n File \"\", line 983, in
> _find_and_load\n File \"\", line 967, in
> _find_and_load_unlocked\n File \"\", line 677,
> in _load_unlocked\n File \"\", line
> 728, in exec_module\n File \"\", line 219, in
> _call_with_frames_removed\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/settings.py\",
> line 73, in \n plugin_app_config = entry_point.load()\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pkg_resources/__init__.py\",
> line 2410, in load\n self.require(*args, **kwargs)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pkg_resources/__init__.py\",
> line 2433, in require\n items = working_set.resolve(reqs, env, installer,
> extras=self.extras)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pkg_resources/__init__.py\",
> line 791, in resolve\n raise VersionConflict(dist,
> req).with_context(dependent_req)\npkg_resources.VersionConflict: (pulpcore
> 3.3.0 (/opt/utils/venv/pulp/3.7.3/lib/python3.7/site-packages),
> Requirement.parse('pulpcore<3.2,>=3.1'))", "stderr_lines": ["Traceback
> (most recent call last):", " File
> \"/opt/utils/venv/pulp/3.7.3/bin/django-admin\", line 8, in ", "
> sys.exit(execute_from_command_line())", " File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/management/__init__.py\",
> line 381, in execute_from_command_line", " utility.execute()", " File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/management/__init__.py\",
> line 325, in execute", " settings.INSTALLED_APPS", " File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/conf/__init__.py\",
> line 79, in __getattr__", " self._setup(name)", " File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/conf/__init__.py\",
> line 66, in _setup", " self._wrapped = Settings(settings_module)", " File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/conf/__init__.py\",
> line 157, in __init__", " mod =
> importlib.import_module(self.SETTINGS_MODULE)", " File
> \"/opt/python/3.7.3/lib64/python3.7/importlib/__init__.py\", line 127, in
> import_module", " return _bootstrap._gcd_import(name[level:], package,
> level)", " File \"\", line 1006, in
> _gcd_import", " File \"\", line 983, in
> _find_and_load", " File \"\", line 967, in
> _find_and_load_unlocked", " File \"\", line
> 677, in _load_unlocked", " File \"\",
> line 728, in exec_module", " File \"\", line
> 219, in _call_with_frames_removed", " File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/settings.py\",
> line 73, in ", " plugin_app_config = entry_point.load()", " File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pkg_resources/__init__.py\",
> line 2410, in load", " self.require(*args, **kwargs)", " File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pkg_resources/__init__.py\",
> line 2433, in require", " items = working_set.resolve(reqs, env, installer,
> 

Re: [Pulp-list] pulpcore-client 3.2 ldap authentication

[Dennis Kliban]

cify admin:pwd
>>>>
>>>> https://docs.pulpproject.org/installation/authentication.html#webserver-auth-with-reverse-proxy
>>>>
>>>> Adding the below to settings.py doesn't seem to work.
>>>> REMOTE_USER_ENVIRON_NAME = 'HTTP_REMOTE_USER'
>>>> AUTHENTICATION_BACKENDS =
>>>> ['pulpcore.app.authentication.PulpNoCreateRemoteUserBackend']
>>>> REST_FRAMEWORK['DEFAULT_AUTHENTICATION_CLASSES'] = (
>>>> 'rest_framework.authentication.SessionAuthentication',
>>>> 'pulpcore.app.authentication.PulpRemoteUserAuthentication'
>>>>
>>>> I am a little confused what need to be added for this setup.
>>>> nginx <---http---> gunicorn <WSGI> pulpcore.app.wsgi
>>>> application
>>>>
>>>> Please advise
>>>> Thanks
>>>>
>>>>
>>>> From: dkli...@redhat.com At: 04/17/20 10:45:31
>>>> To: Bin Li (BLOOMBERG/ 120 PARK ) 
>>>> Cc: pulp-list@redhat.com
>>>> Subject: Re: [Pulp-list] pulpcore-client 3.2 ldap authentication
>>>>
>>>> Theoretically you should be able to use pulpcore-client even with LDAP
>>>> authentication in the web server. However, I have not tested this. I've
>>>> only helped users that use certificate authentication in the webserver.
>>>> What error are you seeing on the client side? Do you see any errors in pulp
>>>> logs?
>>>>
>>>> On Fri, Apr 17, 2020 at 10:20 AM Bin Li (BLOOMBERG/ 120 PARK) <
>>>> bli...@bloomberg.net> wrote:
>>>>
>>>>> Thanks Dennis.
>>>>>
>>>>> We use pulpcore python client to interact with api. Once we enable
>>>>> ldap on nginx, the below code that pulpcore-client authenticate will not
>>>>> work any more. I am wonder if we are still be able to use pulpcore-client?
>>>>> or we have to rewrite the client code. This sounds too much work for us 
>>>>> for
>>>>> now.
>>>>> configuration = pulpcore.Configuration()
>>>>> configuration.host = 'http://localhost'
>>>>> configuration.username = 'admin'
>>>>> configuration.password = 'pwd'
>>>>> rpm_client = pulp_rpm.ApiClient(configuration)
>>>>>
>>>>> From: dkli...@redhat.com At: 04/16/20 08:38:38
>>>>> To: Bin Li (BLOOMBERG/ 120 PARK ) 
>>>>> Cc: pulp-list@redhat.com
>>>>> Subject: Re: [Pulp-list] pulpcore-client 3.2 ldap authentication
>>>>>
>>>>> Please be aware that there is a bug in dynaconf 2.2 with how settings
>>>>> are merged[0]. I recommend upgrading it to dynaconf 3.0.0rc1 for best
>>>>> results when configuring authentication backends in pulp.
>>>>>
>>>>> [0] https://pulp.plan.io/issues/6244
>>>>> [1] https://pypi.org/project/dynaconf/3.0.0rc1/
>>>>>
>>>>>
>>>>> On Wed, Apr 15, 2020 at 7:02 PM Dennis Kliban 
>>>>> wrote:
>>>>>
>>>>>> Pulp 3 does not currently support multiple users. We are planning to
>>>>>> add support for RBAC in the near future. However, I don't have a concrete
>>>>>> timeline for that. With all that said, you still can configure the web
>>>>>> server to perform authentication[0]. In this case Pulp will stop 
>>>>>> performing
>>>>>> authentication and will simply look for a WSGI environment variable that
>>>>>> contains the username.
>>>>>>
>>>>>> [0]
>>>>>> https://docs.pulpproject.org/installation/authentication.html#webserver-auth
>>>>>> [1]
>>>>>> https://docs.pulpproject.org/settings.html?highlight=remote_user#remote-user-environ-name
>>>>>>
>>>>>> On Wed, Apr 15, 2020 at 3:19 PM Bin Li (BLOOMBERG/ 120 PARK) <
>>>>>> bli...@bloomberg.net> wrote:
>>>>>>
>>>>>>>
>>>>>>> I am thinking to configure nginx with ldap authentication, but I
>>>>>>> couldn't find a way to interact with the api. Does pulpcore-client work
>>>>>>> with ldap authentication? Has anyone made httpie work with ldap?
>>>>>>>
>>>>>>> Thanks
>>>>>>> ___
>>>>>>> Pulp-list mailing list
>>>>>>> Pulp-list@redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulpcore-client 3.2 ldap authentication

[Dennis Kliban]

 I have not tested this. I've
>>> only helped users that use certificate authentication in the webserver.
>>> What error are you seeing on the client side? Do you see any errors in pulp
>>> logs?
>>>
>>> On Fri, Apr 17, 2020 at 10:20 AM Bin Li (BLOOMBERG/ 120 PARK) <
>>> bli...@bloomberg.net> wrote:
>>>
>>>> Thanks Dennis.
>>>>
>>>> We use pulpcore python client to interact with api. Once we enable ldap
>>>> on nginx, the below code that pulpcore-client authenticate will not work
>>>> any more. I am wonder if we are still be able to use pulpcore-client? or we
>>>> have to rewrite the client code. This sounds too much work for us for now.
>>>> configuration = pulpcore.Configuration()
>>>> configuration.host = 'http://localhost'
>>>> configuration.username = 'admin'
>>>> configuration.password = 'pwd'
>>>> rpm_client = pulp_rpm.ApiClient(configuration)
>>>>
>>>> From: dkli...@redhat.com At: 04/16/20 08:38:38
>>>> To: Bin Li (BLOOMBERG/ 120 PARK ) 
>>>> Cc: pulp-list@redhat.com
>>>> Subject: Re: [Pulp-list] pulpcore-client 3.2 ldap authentication
>>>>
>>>> Please be aware that there is a bug in dynaconf 2.2 with how settings
>>>> are merged[0]. I recommend upgrading it to dynaconf 3.0.0rc1 for best
>>>> results when configuring authentication backends in pulp.
>>>>
>>>> [0] https://pulp.plan.io/issues/6244
>>>> [1] https://pypi.org/project/dynaconf/3.0.0rc1/
>>>>
>>>>
>>>> On Wed, Apr 15, 2020 at 7:02 PM Dennis Kliban 
>>>> wrote:
>>>>
>>>>> Pulp 3 does not currently support multiple users. We are planning to
>>>>> add support for RBAC in the near future. However, I don't have a concrete
>>>>> timeline for that. With all that said, you still can configure the web
>>>>> server to perform authentication[0]. In this case Pulp will stop 
>>>>> performing
>>>>> authentication and will simply look for a WSGI environment variable that
>>>>> contains the username.
>>>>>
>>>>> [0]
>>>>> https://docs.pulpproject.org/installation/authentication.html#webserver-auth
>>>>> [1]
>>>>> https://docs.pulpproject.org/settings.html?highlight=remote_user#remote-user-environ-name
>>>>>
>>>>> On Wed, Apr 15, 2020 at 3:19 PM Bin Li (BLOOMBERG/ 120 PARK) <
>>>>> bli...@bloomberg.net> wrote:
>>>>>
>>>>>>
>>>>>> I am thinking to configure nginx with ldap authentication, but I
>>>>>> couldn't find a way to interact with the api. Does pulpcore-client work
>>>>>> with ldap authentication? Has anyone made httpie work with ldap?
>>>>>>
>>>>>> Thanks
>>>>>> ___
>>>>>> Pulp-list mailing list
>>>>>> Pulp-list@redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>>>
>>>>>
>>>>
>>>
>>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulpcore-client 3.2 ldap authentication

[Dennis Kliban]

You need to replace

REST_FRAMEWORK['DEFAULT_AUTHENTICATION_CLASSES'] =

with

REST_FRAMEWORK__DEFAULT_AUTHENTICATION_CLASSES =

On Tue, Apr 21, 2020 at 10:09 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> This setting actually failed to restart pulp. See errors below.
>
> Apr 21 21:56:27 ip-1-76-158-49 gunicorn[24414]: NameError: name
> 'REST_FRAMEWORK' is not defined
> Apr 21 21:56:27 ip-1-76-158-49 gunicorn[24414]: [2020-04-21 21:56:27
> -0400] [24417] [INFO] Worker exiting (pid: 24417)
> Apr 21 21:56:27 ip-1-76-158-49 gunicorn[24414]: [2020-04-21 21:56:27
> -0400] [24414] [INFO] Shutting down: Master
> Apr 21 21:56:27 ip-1-76-158-49 gunicorn[24414]: [2020-04-21 21:56:27
> -0400] [24414] [INFO] Reason: Worker failed to boot.
> Apr 21 21:56:27 ip-1-76-158-49 systemd[1]: pulpcore-api.service: main
> process exited, code=exited, status=3/NOTIMPLEMENTED
> Apr 21 21:56:27 ip-1-76-158-49 systemd[1]: Unit pulpcore-api.service
> entered failed state.
> Apr 21 21:56:27 ip-1-76-158-49 systemd[1]: pulpcore-api.service failed.
> Apr 21 21:56:27 ip-1-76-158-49 systemd[1]:
> pulpcore-resource-manager.service holdoff time over, scheduling restart.
>
>
> From: Bin Li (BLOOMBERG/ 120 PARK) At: 04/21/20 21:32:49
> To: dkli...@redhat.com
> Cc: pulp-list@redhat.com
> Subject: Re: [Pulp-list] pulpcore-client 3.2 ldap authentication
>
> Yes, I did
> # pip list |grep dynaconf
> dynaconf 3.0.0rc1
>
>
> From: dkli...@redhat.com At: 04/21/20 20:01:00
> To: Bin Li (BLOOMBERG/ 120 PARK ) 
> Cc: pulp-list@redhat.com
> Subject: Re: [Pulp-list] pulpcore-client 3.2 ldap authentication
>
> Did you update dynaconf to 3.0.0rc1? There was a bug that caused the
> settings to get merged instead of overwritten.
>
> [0] https://pulp.plan.io/issues/6244
> [1] https://pypi.org/project/dynaconf/3.0.0rc1/
>
> On Tue, Apr 21, 2020 at 5:59 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> I have followed the setup
>> https://www.nginx.com/blog/nginx-plus-authenticate-users/ to setup nginx
>> LDAP authentication.
>>
>> This command works "http -a admin:password GET
>> localhost/pulp/api/v3/repositories/rpm/rpm/ Cookie:nginxauth=XXX". The
>> Cookie is the base64 encoded ldap username and password.
>>
>> I assume I should follow the below so I don't have to specify admin:pwd
>>
>> https://docs.pulpproject.org/installation/authentication.html#webserver-auth-with-reverse-proxy
>>
>> Adding the below to settings.py doesn't seem to work.
>> REMOTE_USER_ENVIRON_NAME = 'HTTP_REMOTE_USER'
>> AUTHENTICATION_BACKENDS =
>> ['pulpcore.app.authentication.PulpNoCreateRemoteUserBackend']
>> REST_FRAMEWORK['DEFAULT_AUTHENTICATION_CLASSES'] = (
>> 'rest_framework.authentication.SessionAuthentication',
>> 'pulpcore.app.authentication.PulpRemoteUserAuthentication'
>>
>> I am a little confused what need to be added for this setup.
>> nginx <---http---> gunicorn <WSGI> pulpcore.app.wsgi application
>>
>> Please advise
>> Thanks
>>
>>
>> From: dkli...@redhat.com At: 04/17/20 10:45:31
>> To: Bin Li (BLOOMBERG/ 120 PARK ) 
>> Cc: pulp-list@redhat.com
>> Subject: Re: [Pulp-list] pulpcore-client 3.2 ldap authentication
>>
>> Theoretically you should be able to use pulpcore-client even with LDAP
>> authentication in the web server. However, I have not tested this. I've
>> only helped users that use certificate authentication in the webserver.
>> What error are you seeing on the client side? Do you see any errors in pulp
>> logs?
>>
>> On Fri, Apr 17, 2020 at 10:20 AM Bin Li (BLOOMBERG/ 120 PARK) <
>> bli...@bloomberg.net> wrote:
>>
>>> Thanks Dennis.
>>>
>>> We use pulpcore python client to interact with api. Once we enable ldap
>>> on nginx, the below code that pulpcore-client authenticate will not work
>>> any more. I am wonder if we are still be able to use pulpcore-client? or we
>>> have to rewrite the client code. This sounds too much work for us for now.
>>> configuration = pulpcore.Configuration()
>>> configuration.host = 'http://localhost'
>>> configuration.username = 'admin'
>>> configuration.password = 'pwd'
>>> rpm_client = pulp_rpm.ApiClient(configuration)
>>>
>>> From: dkli...@redhat.com At: 04/16/20 08:38:38
>>> To: Bin Li (BLOOMBERG/ 120 PARK ) 
>>> Cc: pulp-list@redhat.com
>>> Subject: Re: [Pulp-list] pulpcore-client 3.2 ldap authentication
>>>
>>> Please be aware that there is a bug in dynaconf 2.2 with how settings
>>> are m

Re: [Pulp-list] pulpcore-client 3.2 ldap authentication

[Dennis Kliban]

Did you update dynaconf to 3.0.0rc1? There was a bug that caused the
settings to get merged instead of overwritten.

[0] https://pulp.plan.io/issues/6244
[1] https://pypi.org/project/dynaconf/3.0.0rc1/

On Tue, Apr 21, 2020 at 5:59 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> I have followed the setup
> https://www.nginx.com/blog/nginx-plus-authenticate-users/ to setup nginx
> LDAP authentication.
>
> This command works "http -a admin:password GET
> localhost/pulp/api/v3/repositories/rpm/rpm/ Cookie:nginxauth=XXX". The
> Cookie is the base64 encoded ldap username and password.
>
> I assume I should follow the below so I don't have to specify admin:pwd
>
> https://docs.pulpproject.org/installation/authentication.html#webserver-auth-with-reverse-proxy
>
> Adding the below to settings.py doesn't seem to work.
> REMOTE_USER_ENVIRON_NAME = 'HTTP_REMOTE_USER'
> AUTHENTICATION_BACKENDS =
> ['pulpcore.app.authentication.PulpNoCreateRemoteUserBackend']
> REST_FRAMEWORK['DEFAULT_AUTHENTICATION_CLASSES'] = (
> 'rest_framework.authentication.SessionAuthentication',
> 'pulpcore.app.authentication.PulpRemoteUserAuthentication'
>
> I am a little confused what need to be added for this setup.
> nginx <---http---> gunicorn <WSGI> pulpcore.app.wsgi application
>
> Please advise
> Thanks
>
>
> From: dkli...@redhat.com At: 04/17/20 10:45:31
> To: Bin Li (BLOOMBERG/ 120 PARK ) 
> Cc: pulp-list@redhat.com
> Subject: Re: [Pulp-list] pulpcore-client 3.2 ldap authentication
>
> Theoretically you should be able to use pulpcore-client even with LDAP
> authentication in the web server. However, I have not tested this. I've
> only helped users that use certificate authentication in the webserver.
> What error are you seeing on the client side? Do you see any errors in pulp
> logs?
>
> On Fri, Apr 17, 2020 at 10:20 AM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> Thanks Dennis.
>>
>> We use pulpcore python client to interact with api. Once we enable ldap
>> on nginx, the below code that pulpcore-client authenticate will not work
>> any more. I am wonder if we are still be able to use pulpcore-client? or we
>> have to rewrite the client code. This sounds too much work for us for now.
>> configuration = pulpcore.Configuration()
>> configuration.host = 'http://localhost'
>> configuration.username = 'admin'
>> configuration.password = 'pwd'
>> rpm_client = pulp_rpm.ApiClient(configuration)
>>
>> From: dkli...@redhat.com At: 04/16/20 08:38:38
>> To: Bin Li (BLOOMBERG/ 120 PARK ) 
>> Cc: pulp-list@redhat.com
>> Subject: Re: [Pulp-list] pulpcore-client 3.2 ldap authentication
>>
>> Please be aware that there is a bug in dynaconf 2.2 with how settings are
>> merged[0]. I recommend upgrading it to dynaconf 3.0.0rc1 for best results
>> when configuring authentication backends in pulp.
>>
>> [0] https://pulp.plan.io/issues/6244
>> [1] https://pypi.org/project/dynaconf/3.0.0rc1/
>>
>>
>> On Wed, Apr 15, 2020 at 7:02 PM Dennis Kliban  wrote:
>>
>>> Pulp 3 does not currently support multiple users. We are planning to add
>>> support for RBAC in the near future. However, I don't have a concrete
>>> timeline for that. With all that said, you still can configure the web
>>> server to perform authentication[0]. In this case Pulp will stop performing
>>> authentication and will simply look for a WSGI environment variable that
>>> contains the username.
>>>
>>> [0]
>>> https://docs.pulpproject.org/installation/authentication.html#webserver-auth
>>> [1]
>>> https://docs.pulpproject.org/settings.html?highlight=remote_user#remote-user-environ-name
>>>
>>> On Wed, Apr 15, 2020 at 3:19 PM Bin Li (BLOOMBERG/ 120 PARK) <
>>> bli...@bloomberg.net> wrote:
>>>
>>>>
>>>> I am thinking to configure nginx with ldap authentication, but I
>>>> couldn't find a way to interact with the api. Does pulpcore-client work
>>>> with ldap authentication? Has anyone made httpie work with ldap?
>>>>
>>>> Thanks
>>>> ___
>>>> Pulp-list mailing list
>>>> Pulp-list@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>
>>>
>>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] bindings users will need to use a new model when creating content

[Dennis Kliban]

Starting with pulpcore 3.4, the bindings for resources that inherit from
Content will contain 2 separate models: one for reading and one for
writing. e.g. FileFileContentRead and FileFileContentWrite.

This is a backwards incompatible change to fix a bug that was preventing
users from properly using the existing models for creating Content
resources in a single upload request.
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulp 3.2.1 duplicate key error

[Dennis Kliban]

The issue does not have any steps to reproduce it. Have you been able to
consistently reproduce the issue with a specific repository? Does it go
away the next time you perform the sync?

On Wed, Apr 8, 2020 at 9:03 AM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Thanks Brian. I filed an issue https://pulp.plan.io/issues/6463
>
> From: bmbou...@redhat.com At: 04/07/20 15:59:11
> To: Bin Li (BLOOMBERG/ 120 PARK ) 
> Cc: pulp-list@redhat.com
> Subject: Re: [Pulp-list] pulp 3.2.1 duplicate key error
>
> I heard another developer report a similar issue, but we couldn't
> reproduce it. Could you file this as an issue also please?
>
> On Tue, Apr 7, 2020 at 3:42 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> Noticed we have a few errors when running sync.
>>
>> "error": {
>> "description": "duplicate key value violates unique constraint
>> \"core_repositoryversion_repository_id_number_3c54ce50_uniq\"\nDETAIL: Key
>> (repository_id, number)=(59eb02b1-edab-46e3-a69b-d69a8b314f20, 2) already
>> exists.\n",
>>
>> What could be the cause of this? How can we resolve it?
>> ___
>> Pulp-list mailing list
>> Pulp-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulp 3 publication required

[Dennis Kliban]

Some content types require publications and others don't. It all depends on
the requirements of the client that consumes content. For example, yum/dnf
expects to find metadata about all the packages in the repository. Since a
RepositoryVersion only contains the packages, you must create a Publication
to generate the metadata. On the other hand 'podman' and 'docker' interact
with a REST API that serves content without any special metadata. So
Container repositories and repository versions can be directly associated
with ContainerDistributions.

On Fri, Apr 17, 2020 at 9:59 AM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> I was under impression that the publication has to be created before
> creating a distribution. It looks like now I can create a distribution by
> specify a repository or repository version. Is the publication created
> automatically in this case? If not, in what scenario we need a create a
> publication before create a distribution?
>
> Thanks
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulpcore-client 3.2 ldap authentication

[Dennis Kliban]

Theoretically you should be able to use pulpcore-client even with LDAP
authentication in the web server. However, I have not tested this. I've
only helped users that use certificate authentication in the webserver.
What error are you seeing on the client side? Do you see any errors in pulp
logs?

On Fri, Apr 17, 2020 at 10:20 AM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Thanks Dennis.
>
> We use pulpcore python client to interact with api. Once we enable ldap on
> nginx, the below code that pulpcore-client authenticate will not work any
> more. I am wonder if we are still be able to use pulpcore-client? or we
> have to rewrite the client code. This sounds too much work for us for now.
> configuration = pulpcore.Configuration()
> configuration.host = 'http://localhost'
> configuration.username = 'admin'
> configuration.password = 'pwd'
> rpm_client = pulp_rpm.ApiClient(configuration)
>
> From: dkli...@redhat.com At: 04/16/20 08:38:38
> To: Bin Li (BLOOMBERG/ 120 PARK ) 
> Cc: pulp-list@redhat.com
> Subject: Re: [Pulp-list] pulpcore-client 3.2 ldap authentication
>
> Please be aware that there is a bug in dynaconf 2.2 with how settings are
> merged[0]. I recommend upgrading it to dynaconf 3.0.0rc1 for best results
> when configuring authentication backends in pulp.
>
> [0] https://pulp.plan.io/issues/6244
> [1] https://pypi.org/project/dynaconf/3.0.0rc1/
>
>
> On Wed, Apr 15, 2020 at 7:02 PM Dennis Kliban  wrote:
>
>> Pulp 3 does not currently support multiple users. We are planning to add
>> support for RBAC in the near future. However, I don't have a concrete
>> timeline for that. With all that said, you still can configure the web
>> server to perform authentication[0]. In this case Pulp will stop performing
>> authentication and will simply look for a WSGI environment variable that
>> contains the username.
>>
>> [0]
>> https://docs.pulpproject.org/installation/authentication.html#webserver-auth
>> [1]
>> https://docs.pulpproject.org/settings.html?highlight=remote_user#remote-user-environ-name
>>
>> On Wed, Apr 15, 2020 at 3:19 PM Bin Li (BLOOMBERG/ 120 PARK) <
>> bli...@bloomberg.net> wrote:
>>
>>>
>>> I am thinking to configure nginx with ldap authentication, but I
>>> couldn't find a way to interact with the api. Does pulpcore-client work
>>> with ldap authentication? Has anyone made httpie work with ldap?
>>>
>>> Thanks
>>> ___
>>> Pulp-list mailing list
>>> Pulp-list@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulpcore-client 3.2 ldap authentication

[Dennis Kliban]

Please be aware that there is a bug in dynaconf 2.2 with how settings are
merged[0]. I recommend upgrading it to dynaconf 3.0.0rc1 for best results
when configuring authentication backends in pulp.

[0] https://pulp.plan.io/issues/6244
[1] https://pypi.org/project/dynaconf/3.0.0rc1/


On Wed, Apr 15, 2020 at 7:02 PM Dennis Kliban  wrote:

> Pulp 3 does not currently support multiple users. We are planning to add
> support for RBAC in the near future. However, I don't have a concrete
> timeline for that. With all that said, you still can configure the web
> server to perform authentication[0]. In this case Pulp will stop performing
> authentication and will simply look for a WSGI environment variable that
> contains the username.
>
> [0]
> https://docs.pulpproject.org/installation/authentication.html#webserver-auth
> [1]
> https://docs.pulpproject.org/settings.html?highlight=remote_user#remote-user-environ-name
>
> On Wed, Apr 15, 2020 at 3:19 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>>
>> I am thinking to configure nginx with ldap authentication, but I couldn't
>> find a way to interact with the api. Does pulpcore-client work with ldap
>> authentication? Has anyone made httpie work with ldap?
>>
>> Thanks
>> ___
>> Pulp-list mailing list
>> Pulp-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulpcore-client 3.2 ldap authentication

[Dennis Kliban]

Pulp 3 does not currently support multiple users. We are planning to add
support for RBAC in the near future. However, I don't have a concrete
timeline for that. With all that said, you still can configure the web
server to perform authentication[0]. In this case Pulp will stop performing
authentication and will simply look for a WSGI environment variable that
contains the username.

[0]
https://docs.pulpproject.org/installation/authentication.html#webserver-auth
[1]
https://docs.pulpproject.org/settings.html?highlight=remote_user#remote-user-environ-name

On Wed, Apr 15, 2020 at 3:19 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

>
> I am thinking to configure nginx with ldap authentication, but I couldn't
> find a way to interact with the api. Does pulpcore-client work with ldap
> authentication? Has anyone made httpie work with ldap?
>
> Thanks
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulp 3.2.1 sync error

[Dennis Kliban]

Could you please file a bug at https://pulp.plan.io/issues/new/ ? Please
include the version of pulpcore you upgraded from. Thanks!

On Wed, Apr 1, 2020 at 9:49 AM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Hi All,
> We recently upgraded the pulp to 3.2.1. Syncing to redhat repo now gives
> us the following error:
> "error": {
> "traceback": " File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\",
> line 884, in perform_job\n rv = job.perform()\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\",
> line 664, in perform\n self._result = self._execute()\n File
> \"/opt/utils/venv/pulp/3.7.3/l
> ib64/python3.7/site-packages/rq/job.py\", line 670, in _execute\n return
> self.func(*self.args, **self.kwargs)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/si
> te-packages/pulp_rpm/app/tasks/synchronizing.py\", line 126, in
> synchronize\n treeinfo = get_treeinfo_data(remote)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3
> .7/site-packages/pulp_rpm/app/kickstart/treeinfo.py\", line 21, in
> get_treeinfo_data\n downloader =
> remote.get_downloader(url=urljoin(remote_url, namespace))\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/repository.py\",
> line 287, in get_downloader\n return self.download_factory.build(url
> te-packages/pulp_rpm/app/tasks/synchronizing.py\", line 126, in
> synchronize\n treeinfo = get_treeinfo_data(remote)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3
> .7/site-packages/pulp_rpm/app/kickstart/treeinfo.py\", line 21, in
> get_treeinfo_data\n downloader =
> remote.get_downloader(url=urljoin(remote_url, namespace))\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/repository.py\",
> line 287, in get_downloader\n return self.download_factory.build(url
> , **kwargs)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/repository.py\",
> line 245, in download_factory\n self._download_fa
> ctory = DownloaderFactory(self)\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/factory.py\",
> line 68, in __init__\n self._sessi
> on = self._make_aiohttp_session_from_remote()\n File
> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/factory.py\",
> line 85, in _make_aioht
> tp_session_from_remote\n sslcontext =
> ssl.create_default_context(cadata=self._remote.ca_cert)\n File
> \"/opt/python/3.7.3/lib64/python3.7/ssl.py\", line 573, in crea
> te_default_context\n context.load_verify_locations(cafile, capath,
> cadata)\n",
> "description": "[PEM: NO_START_LINE] no start line (_ssl.c:3947)"
>
> We had no issues before we upgraded. Please advise how this can be fixed.
>
>
> Thanks
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Red Hat Repo Sync , pulp 2.21

[Dennis Kliban]

Glad to hear you got it figured out!

On Tue, Mar 31, 2020 at 1:22 PM Gravel Bone  wrote:

> Ugh, I didn't realize that rhsmcertd was turned back on for my pulp server
> so the certs were rotating.   I figured it was going to end up being
> something I forgot to check.
>
> On Tue, Mar 31, 2020 at 10:32 AM Dennis Kliban  wrote:
>
>> Please open a support case with Red Hat. It seem like a problem with the
>> CDN.
>>
>> On Tue, Mar 31, 2020 at 10:17 AM Gravel Bone 
>> wrote:
>>
>>> I've been syncing Red Hat repos, however, almost weekly they start
>>> failing with the dreaded "Error retrieving metadata: Not found" or "Error
>>> retrieving metadata: Forbidden"
>>>
>>> If I delete the repo and re-create with same entitlements, it starts
>>> working again.  If I update the entitlement with the same entitlement, it
>>> sometimes starts working again, if I just go get a new entitlement key it
>>> will generally work.
>>>
>>> I've been trying various things like --force option, republishing.
>>> Don't know if it's the server configuration or a pulp thing at this point.
>>>
>>> I've searched, but almost all the help is referring to Foreman now and
>>> refreshing a manifest or disabling proxy, neither of which I think I'm
>>> using.
>>> ___
>>> Pulp-list mailing list
>>> Pulp-list@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Red Hat Repo Sync , pulp 2.21

[Dennis Kliban]

Please open a support case with Red Hat. It seem like a problem with the
CDN.

On Tue, Mar 31, 2020 at 10:17 AM Gravel Bone  wrote:

> I've been syncing Red Hat repos, however, almost weekly they start failing
> with the dreaded "Error retrieving metadata: Not found" or "Error
> retrieving metadata: Forbidden"
>
> If I delete the repo and re-create with same entitlements, it starts
> working again.  If I update the entitlement with the same entitlement, it
> sometimes starts working again, if I just go get a new entitlement key it
> will generally work.
>
> I've been trying various things like --force option, republishing.  Don't
> know if it's the server configuration or a pulp thing at this point.
>
> I've searched, but almost all the help is referring to Foreman now and
> refreshing a manifest or disabling proxy, neither of which I think I'm
> using.
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] RHEL 8 rpm repo sync errors

[Dennis Kliban]

I was referring to the certificate and key configured on the importer
associated with the repository. Our docs explain how to configure a new
repository to sync from the Red Hat servers[0].

If your other repositories are able to sync Red Hat content, I would
compare the configuration of those repositories with the broken one.

[0]
https://docs.pulpproject.org/en/2.21/plugins/pulp_rpm/user-guide/recipes.html#sync-a-protected-repo

On Wed, Mar 18, 2020 at 3:43 PM Venkataramana Bora 
wrote:

> Hi Dennis , Thanks a lot for your reply.
> Sorry, for this question . I'm not well versed with Puppet configuration ,
> some one else did the config part here, no longer with company.
>  You said  , we should have the correct client certificate and client key
> configured.
> Is that  SSLCertificateFile  and SSLCertificateKeyFile in
> /etc/httpd/conf.d/ssl.conf ?
> Can you please let me know patch to check?
>
>
>
> Sincerely,
> Ramana Bora
>
>
>
> - Original message -
> From: Dennis Kliban 
> To: Venkataramana Bora 
> Cc: pulp-list 
> Subject: [EXTERNAL] Re: [Pulp-list] RHEL 8 rpm repo sync errors
> Date: Wed, Mar 18, 2020 7:25 PM
>
> This looks like a problem with the client certificate. The very first
> request for a  file that pulp tries to download is receiving a 403 response
> from the CDN. Please make sure you have the correct client certificate and
> client key configured.
>
> On Tue, Mar 17, 2020 at 5:53 PM Venkataramana Bora 
> wrote:
>
>
>
> Hi Teamn ,
> Getting errors as shown here when we are trying to sync RHEL 8 repos (Base
> OS and Appstream) creatd on Pulp master. The same Pulp master we already
> using for RHEL 6 and 7 repos with out any issues. Strugling with RHEL 8
> repos creation,
> 1.Could you please let us know whethere RHEL 8 rpm repos sync works on
> Pulp master version  2.16 or not ?
> 2.feed_ca_cert "redhat-uep.pem" and feed_cert "rhel-identity.pem" are
> there  . We have no issues with RHEL 6 and 7 repo syncs, those created some
> years ago and working well.
>   Is there any thing specifically need to do for RHEL 8 repos in terms of
> ca cert/feed cert ? We see "failed with code 403 Forbiddenin" in var log
> messages.
> 3.pulp-admin tasks details Traceback shows "/usr/lib/python2.7/..."
>   For this added these 3 lines in
> /etc/pulp/server/plugins.conf.d/yum_importer.json as recommended here
> https://pulp.plan.io/issues/6327
>  {
>"validate":true
> }
> After adding that json , I see that RHEL 8 repodata xml.gz  in
> rhel8/x86_64/baseos/os/repodata/  but not "packages" folder created yet ,
> checked that thru pulp web access .
> Please help  resolving this issue .
>
> Feed url:
> https://cdn.redhat.com/content/dist/rhel8/$releasever/x86_64/baseos/os
> Feed url:
> https://cdn.redhat.com/content/dist/rhel8/$releasever/x86_64/appstream/os
> Repoid for these urls are :
> rhel-8-server-baseos-x86_64
> rhel-8-server-appstream-x86_64
> Note: Tried with "8" in place "$releasever" in feed but same error . Our
> Pulp master ver.2.16 is on CentOS 7.7.
>
> --
> Tasks performed:
>
> [root@swy01opplppr01 ~]#  pulp-admin rpm repo sync run --force-full
> --repo-id rhel-8-server-baseos-x86_64
> +--+
>  Synchronizing Repository [rhel-8-server-baseos-x86_64]
> +--+
> The following sync configuration options will be used:
> Force Full:  True
>
> This command may be exited via ctrl+c without affecting the request.
>
> Downloading metadata...
> [\]
> ... completed
> Downloading repository content...
> [-]
> [==] 100%
> RPMs:   0/0 items
> Delta RPMs: 0/0 items
> ... completed
> Downloading distribution files...
> [==] 100%
> Distributions: 0/0 items
> Task Failed
> Error retrieving metadata: Forbidden
>
>
> -
> From /var/log/messages:
>
> Mar 17 17:11:23 swy01opplppr01 pulp: celery.worker.strategy:INFO: Received
> task:
> pulp.server.async.tasks._release_resource[c5b01662-a20a-4c44-b114-b1ba595f05fc]
> Mar 17 17:11:23 swy01opplppr01 pulp: celery.app.trace:INFO: [55ba0954]
> Task
> pulp.server.async.tasks._queue_reserved_task[55ba0954-1a18-410a-9507-f61d74c4776f]
> succeeded

Re: [Pulp-list] RHEL 8 rpm repo sync errors

[Dennis Kliban]

This looks like a problem with the client certificate. The very first
request for a  file that pulp tries to download is receiving a 403 response
from the CDN. Please make sure you have the correct client certificate and
client key configured.

On Tue, Mar 17, 2020 at 5:53 PM Venkataramana Bora 
wrote:

>
>
> Hi Teamn ,
> Getting errors as shown here when we are trying to sync RHEL 8 repos (Base
> OS and Appstream) creatd on Pulp master. The same Pulp master we already
> using for RHEL 6 and 7 repos with out any issues. Strugling with RHEL 8
> repos creation,
> 1.Could you please let us know whethere RHEL 8 rpm repos sync works on
> Pulp master version  2.16 or not ?
> 2.feed_ca_cert "redhat-uep.pem" and feed_cert "rhel-identity.pem" are
> there  . We have no issues with RHEL 6 and 7 repo syncs, those created some
> years ago and working well.
>   Is there any thing specifically need to do for RHEL 8 repos in terms of
> ca cert/feed cert ? We see "failed with code 403 Forbiddenin" in var log
> messages.
> 3.pulp-admin tasks details Traceback shows "/usr/lib/python2.7/..."
>   For this added these 3 lines in
> /etc/pulp/server/plugins.conf.d/yum_importer.json as recommended here
> https://pulp.plan.io/issues/6327
>  {
>"validate":true
> }
> After adding that json , I see that RHEL 8 repodata xml.gz  in
> rhel8/x86_64/baseos/os/repodata/  but not "packages" folder created yet ,
> checked that thru pulp web access .
> Please help  resolving this issue .
>
> Feed url:
> https://cdn.redhat.com/content/dist/rhel8/$releasever/x86_64/baseos/os
> Feed url:
> https://cdn.redhat.com/content/dist/rhel8/$releasever/x86_64/appstream/os
> Repoid for these urls are :
> rhel-8-server-baseos-x86_64
> rhel-8-server-appstream-x86_64
> Note: Tried with "8" in place "$releasever" in feed but same error . Our
> Pulp master ver.2.16 is on CentOS 7.7.
>
> --
> Tasks performed:
>
> [root@swy01opplppr01 ~]#  pulp-admin rpm repo sync run --force-full
> --repo-id rhel-8-server-baseos-x86_64
> +--+
>  Synchronizing Repository [rhel-8-server-baseos-x86_64]
> +--+
> The following sync configuration options will be used:
> Force Full:  True
>
> This command may be exited via ctrl+c without affecting the request.
>
> Downloading metadata...
> [\]
> ... completed
> Downloading repository content...
> [-]
> [==] 100%
> RPMs:   0/0 items
> Delta RPMs: 0/0 items
> ... completed
> Downloading distribution files...
> [==] 100%
> Distributions: 0/0 items
> Task Failed
> Error retrieving metadata: Forbidden
>
>
> -
> From /var/log/messages:
>
> Mar 17 17:11:23 swy01opplppr01 pulp: celery.worker.strategy:INFO: Received
> task:
> pulp.server.async.tasks._release_resource[c5b01662-a20a-4c44-b114-b1ba595f05fc]
> Mar 17 17:11:23 swy01opplppr01 pulp: celery.app.trace:INFO: [55ba0954]
> Task
> pulp.server.async.tasks._queue_reserved_task[55ba0954-1a18-410a-9507-f61d74c4776f]
> succeeded in 0.115055091001s: None
> Mar 17 17:11:23 swy01opplppr01 pulp:
> pulp_rpm.plugins.importers.yum.sync:INFO: [fbbc36d3] Downloading metadata
> from https://cdn.redhat.com/content/dist/rhel8//x86_64/baseos/os/.
> Mar 17 17:11:23 swy01opplppr01 pulp: urllib3.connectionpool:INFO: Starting
> new HTTPS connection (1): cdn.redhat.com
> Mar 17 17:11:23 swy01opplppr01 pulp: nectar.downloaders.threaded:INFO:
> Download failed: Download of
> https://cdn.redhat.com/content/dist/rhel8//x86_64/baseos/os/repodata/repomd.xml
> failed with code 403: Forbidden
> Mar 17 17:11:24 swy01opplppr01 pulp: urllib3.connectionpool:INFO:
> [fbbc36d3] Starting new HTTPS connection (1): cdn.redhat.com
> Mar 17 17:11:24 swy01opplppr01 pulp: nectar.downloaders.threaded:INFO:
> [fbbc36d3] Download failed: Download of
> https://cdn.redhat.com/content/dist/rhel8//x86_64/baseos/os failed with
> code 403: Forbidden
> Mar 17 17:11:24 swy01opplppr01 pulp:
> pulp_rpm.plugins.importers.yum.sync:INFO: [fbbc36d3] Downloading metadata
> from https://cdn.redhat.com/content/dist/rhel8//x86_64/baseos/os/.
> Mar 17 17:11:24 swy01opplppr01 pulp: urllib3.connectionpool:INFO: Starting
> new HTTPS connection (1): cdn.redhat.com
> Mar 17 17:11:24 swy01opplppr01 pulp: nectar.downloaders.threaded:INFO:
> Download failed: Download of
> https://cdn.redhat.com/content/dist/rhel8//x86_64/baseos/os/repodata/repomd.xml
> failed with code 403: Forbidden
> Mar 17 17:11:25 swy01opplppr01 pulp: urllib3.connectionpool:INFO:
> [fbbc36d3] Starting new HTTPS connection (1): cdn.redhat.com
> Mar 17 17:11:25 

[Pulp-list] pulpcore 3.2.1 is Generally Available

[Dennis Kliban]

Pulpcore 3.2.1 is available[0]. This is a bug fix release. There are 2
changes in this release:

 - dynaconf required version has been bumped to include the upcoming 3.0.0
release of dynaconf which will resolve a problem with nested settings
 - bug fix related to removing duplicates from a repository version when no
previous version existed

## New Installations

If you are using a new install, you can use the latest ansible installer[1]
and you'll receive 3.2.1. The quick start guides below are useful for that.

* pulp_rpm installation --
https://pulp-rpm.readthedocs.io/en/latest/installation.html
* pulp_container installation --
https://pulp-container.readthedocs.io/en/latest/installation.html
* pulp_file installation --
https://pulp-file.readthedocs.io/en/latest/installation.html

## Upgrading from 3.y.z

You should first upgrade your installer to the latest one from GitHub and
re-run it. If you get the new installer, you'll get a newer pulpcore
release. If you rerun an older version of the installer, you'll get an
older version (downgrading is not supported).

[0] https://pypi.org/project/pulpcore/3.2.1/
[1] https://github.com/pulp/ansible-pulp/releases/tag/3.2.1
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] Pulp 3 in a single container

[Dennis Kliban]

I just published a container image of Pulp 3. I described how to use it on
our blog.

https://pulpproject.org/2020/03/15/pulp-fedora31-single-container/

Thanks to everyone at CfgMgmtCamp that suggested this approach for
deploying Pulp.
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulpcore 3.2.0 and pulp_file 0.2.0 releases

[Dennis Kliban]

pulp_container 1.2.0 has been released[0]. It is compatible with pulpcore
3.2. For a list of all changes please check the changelog[1].

[0]: https://pypi.org/project/pulp-container/1.2.0/
[1] https://pulp-container.readthedocs.io/en/latest/changes.html#id1

On Thu, Feb 27, 2020 at 4:02 PM David Davis  wrote:

> pulpcore 3.2.0[0] and pulp_file 0.2.0[1] have been released. For a list of
> all changes, please check the changelogs for pulpcore[2] and pulp_file[3].
>
> # Installation and Upgrade
>
> Users should use the 3.2.0 release of ansible-pulp installer[4] to install
> or upgrade their installations. This version of the installer will check
> compatibility of all installed plugins with pulpcore 3.2. The installer
> will abort if any plugin is incompatible.
>
> # Plugin API
>
> Plugin writers can see the plugin API changelog here (
> https://docs.pulpproject.org/en/3.2.0/changes.html#plugin-api ). There
> was only one backwards incompatible change[5], and in keeping with the
> recommended strategy to pin plugins to a 3.y version of pulpcore, plugins
> should release compatibility releases with 3.2 as soon as they can. We
> recommend using "pulpcore>=3.0,<3.3".
>
> The installer also has a backwards incompatible change in 3.2.0 where it
> no longer attempts to handle custom URLs for plugins automatically. If your
> plugin uses URLs outside of /pulp/api/v3/ or /pulp/content/ you will need
> to add a snippet. See these docs[6] for more.
>
> [0]: https://pypi.org/project/pulpcore/3.2.0/
> [1]: https://pypi.org/project/pulp-file/0.2.0/
> [2]: https://docs.pulpproject.org/changes.html#id1
> [3]: https://pulp-file.readthedocs.io/en/0.2.0/changes.html#id1
> [4]: https://github.com/pulp/ansible-pulp/releases/tag/3.2.0
> [5]: https://github.com/pulp/pulpcore/pull/555
> [6]: https://pulp.plan.io/issues/6057
>
> David
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Pulp v3 Installation and testing

[Dennis Kliban]

On Thu, Feb 27, 2020 at 7:13 AM Sathasivam, Pradeep <
pradeep.sathasi...@hpe.com> wrote:

> Hi All,
>
>
>
> I am not able to find a proper installation document for pulp v3. Using
> the current documentation at -
> https://docs.pulpproject.org/installation/instructions.html#ansible-installation-recommended
> I am not able to use REST APIs.
>
Did you use ansible-pulp? Which release?
What kind of errors are you seeing in journalctl -l -f?

It's possible that your settings file has the wrong permissions on it and
we are failing to raise an error due to a recently discovered bug[0]. It
should be owned by pulp:pulp.

[0] https://github.com/rochacbruno/dynaconf/issues/310


>
> My use case for using Pulp is:
>
> 1.   Handle security patches for CentOS
>
> 2.   Handle CentOS updates (7.3 à 7.4)
>
>
>
> Can anybody point me to a proper documentation?
>
>
You need to install pulp_rpm plugin. Instructions for that are here[1].

[1]
https://pulp-rpm.readthedocs.io/en/latest/installation.html#install-with-ansible-pulp-recommended


>
>
> Also I am not sure if ‘*pulp-admin*’ and ‘*pulp-consumer*’ exists in Pulp
> v3. Please clarify.
>
>
>
> My understanding is that all operations like:
>
> 1.   Registering consumer
>
> 2.   Creating repo
>
> 3.   Syncing repo
>
> 4.   Installing RPM package on a consumer
>
> 5.   Updating a consumer
>
Pulp 3 does not manage consumers. You can only sync external repositories.
You can upload your own content. You can create custom repository versions
using a mix of content from external repos and uploaded.


> Is all via REST API. Please confirm.
>
>
>
Yes, right now you have to use the REST API. We provide a Python and Ruby
clients[2]. We also provide documentation on how to generate a client in
any language supported by openapi-generator[3]. There is an effort to
create ansible modules[4].

[2] https://docs.pulpproject.org/client_bindings.html
[3] https://docs.pulpproject.org/client_bindings.html#other-languages
[4] https://github.com/mdellweg/ansible_modules_pulp


Regards.
>
> Pradeep. S
>
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] CentOS 8 Repositories module metadata

[Dennis Kliban]

This does sound like a bug to me. Could you please file it at
https://pulp.plan.io/projects/pulp_rpm/issues/new ? Thank you!

On Tue, Feb 18, 2020 at 9:33 AM Payne, Lee (London) 
wrote:

> Hi,
>
>
>
> Thanks for the reply. I have just managed to fix the issue.
>
> When creating the repository within pulp I was using the following
> additional options which I had set for all my RHEL/CentOS 7 repos.
>
> --generate-sqlite true
>
> --checksum-type sha256
>
>
>
> If I create the repositories without these options my client is able to
> see and use the module metadata.
>
> I know it’s probably only one of the additional parameters that causing
> the issue I just haven’t had a chance to test.
>
>
>
>
>
>
>
>
>
>
>
> *From:* Dennis Kliban 
> *Sent:* 18 February 2020 14:13
> *To:* Payne, Lee (London) 
> *Cc:* Pulp-list@redhat.com
> *Subject:* Re: [Pulp-list] CentOS 8 Repositories module metadata
>
>
>
> Could you try to reproduce the problem with Pulp 2.21.0? If the problem
> persists with 2.21.0, please file an issue at
> https://pulp.plan.io/issues/new/
>
>
>
> On Wed, Feb 12, 2020 at 4:44 AM Payne, Lee (London) 
> wrote:
>
> Hi,
>
>
>
> I have pulp 2.18 installed and am trying to setup CentOS 8 / AppStream
> repositories.
>
> I have created the new repos and synchronised the contents down from “
> mirror.centos.org” and all appears to be fine, however when I try to use
> the repositories I get errors trying to install certain packages such as
> python36..
>
> *Running transaction check*
>
> *No available modular metadata for modular package
> 'python36-3.6.8-2.module_el8.1.0+245+c39af44f.x86_64', it cannot be
> installed on the system*
>
>
>
> When the repository is synchronised from upstream it doesn’t appear to be
> importing the module metadata.
>
> All other package installs work fine apart from the module extension ones.
>
>
>
> Is there anything I’m missing when creating repositories for CentOS 8?
>
>
>
> Thanks
>
> Lee
>
>
>
> This email has been sent by a member of the Man group (“Man”). Man's
> parent company, Man Group plc, is registered in Jersey (company number
> 127570) with its registered office at 22 Grenville Street, St Helier,
> Jersey, JE4 8PX. The contents of this email are for the named addressee(s)
> only. It contains information which may be confidential and privileged. If
> you are not the intended recipient, please notify the sender immediately,
> destroy this email and any attachments and do not otherwise disclose or use
> them. Email transmission is not a secure method of communication and Man
> cannot accept responsibility for the completeness or accuracy of this email
> or any attachments. Whilst Man makes every effort to keep its network free
> from viruses, it does not accept responsibility for any computer virus
> which might be transferred by way of this email or any attachments. This
> email does not constitute a request, offer, recommendation or solicitation
> of any kind to buy, subscribe, sell or redeem any investment instruments or
> to perform other such transactions of any kind. Man reserves the right to
> monitor, record and retain all electronic and telephone communications
> through its network in accordance with applicable laws and regulations.
>
> During the course of our business relationship with you, we may process
> your personal data, including through the monitoring of electronic
> communications. We will only process your personal data to the extent
> permitted by laws and regulations; for the purposes of ensuring compliance
> with our legal and regulatory obligations and internal policies; and for
> managing client relationships. For further information please see our
> Privacy Notice: https://www.man.com/privacy-policy
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
>
> This email has been sent by a member of the Man group (“Man”). Man's
> parent company, Man Group plc, is registered in Jersey (company number
> 127570) with its registered office at 22 Grenville Street, St Helier,
> Jersey, JE4 8PX. The contents of this email are for the named addressee(s)
> only. It contains information which may be confidential and privileged. If
> you are not the intended recipient, please notify the sender immediately,
> destroy this email and any attachments and do not otherwise disclose or use
> them. Email transmission is not a secure method of communication and Man
> cannot accept responsibility for the completeness or accuracy of this email
> or any attachments. Whilst Man makes every

Re: [Pulp-list] pulp3 High availability and disaster recovery

[Dennis Kliban]

The redis cluster support needs to be added to rq actually[0,1]. Looks like
there is an open PR but it hasn't moved forward in a long time[2].

[0] https://github.com/rq/rq/issues/862
[1] https://github.com/rq/rq/issues/1048
[2] https://github.com/rq/rq/pull/942

On Tue, Feb 18, 2020 at 9:07 AM Dennis Kliban  wrote:

> How many instances of Redis are involved? Is every pulpcore-api instance
> and pulpcore-worker instance pointing to the same redis instance? This is
> necessary for the work to be routed correctly.
>
> Pulpcore currently uses redis-py, which does not support connecting to a
> Redis Cluster[0]. However, we should investigate if it's viable to switch
> to using redis-py-cluster[1].
>
> [0] https://github.com/andymccurdy/redis-py/issues/931
> [1] https://github.com/Grokzen/redis-py-cluster
>
> On Thu, Feb 13, 2020, 6:28 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> Hi Brian,
>> I did a quick test on a active passive pulp 3.1 setup. Two pulp servers
>> are pointing to the same external postgres database. Only one server is
>> active at any time. Redis queue resides on the localhost. The /var/lib/pulp
>> are synced from primary to the contingency host.
>> After I shutdown primary host, I was able to bring up the contingency
>> pulp server and created a repo. Deleting any repo stuck in a waiting state.
>> Then I started primary host and shutdown contingency host, I was able to
>> delete repos I created on the contingency host but all previous delete job
>> continually stuck in the waiting state.
>> I am wonder if anything I could do to make this work on contingency host
>> or this setup is not going to work?
>>
>> Thanks
>>
>>
>> From: pulp-list@redhat.com At: 01/03/20 12:01:44
>> To: pulp-list@redhat.com
>> Subject: Pulp-list Digest, Vol 122, Issue 1
>>
>> Send Pulp-list mailing list submissions to
>> pulp-list@redhat.com
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://www.redhat.com/mailman/listinfo/pulp-list
>> or, via email, send a message with subject or body 'help' to
>> pulp-list-requ...@redhat.com
>>
>> You can reach the person managing the list at
>> pulp-list-ow...@redhat.com
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Pulp-list digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Re: pulp3 High availability and disaster recovery (Brian Bouterse)
>>
>>
>> --
>>
>> Message: 1
>> Date: Thu, 2 Jan 2020 16:10:29 -0500
>> From: Brian Bouterse 
>> To: JASON STELZER 
>> Cc: pulp-list 
>> Subject: Re: [Pulp-list] pulp3 High availability and disaster recovery
>> Message-ID:
>> 
>> Content-Type: text/plain; charset="utf-8"
>>
>> Sorry for the late reply. Each component of Pulp itself can be deployed in
>> HA configurations. Of the services Pulp's processes depend on, Redis is
>> the
>> one service that can't run as a full cluster because RQ doesn't support
>> that yet, so the best you can do is a hot-spare Redis that auto-fails
>> over.
>> That isn't graceful failover so when traffic routes to your hot-spare
>> Redis
>> it has to data and doesn't have the tasking system's data. Those Pulp
>> tasks
>> would be cancelled, and Pulp would be immediately ready to accept new
>> tasks
>> so they could be resubmitted, e.g. Katello resubmits some job failures I
>> believe.
>>
>> More docs about this are here:
>> https://docs.pulpproject.org/components.html#architecture-and-deploying
>> More questions are welcome; sorry for the slow response. If you can see
>> any
>> way to improve the docs and want to get involved, PRs are welcome!
>>
>> -Brian
>>
>>
>> On Mon, Nov 18, 2019 at 7:37 AM JASON STELZER 
>> wrote:
>>
>> > For what it is worth, at heart pulp3 is a django app. So, following the
>> > advice for HA and django apps generally works. A lot of it is driven by
>> the
>> > particulars of your use case.
>> >
>> > My use case is a little different than yours I'm sure. But in terms of
>> HA
>> > for now I'm good with a balancer and nodes in multiple azs, an RDS db
>> with
>> > failover, and regular db backups.
>> >
>> > In my case, the pulp3 server is far enough behind the scenes that even
>> if
>> > there were to be a several hour outage, the impact would be minimal.
>> YMMV.
>> >

Re: [Pulp-list] pulp3 High availability and disaster recovery

[Dennis Kliban]

How many instances of Redis are involved? Is every pulpcore-api instance
and pulpcore-worker instance pointing to the same redis instance? This is
necessary for the work to be routed correctly.

Pulpcore currently uses redis-py, which does not support connecting to a
Redis Cluster[0]. However, we should investigate if it's viable to switch
to using redis-py-cluster[1].

[0] https://github.com/andymccurdy/redis-py/issues/931
[1] https://github.com/Grokzen/redis-py-cluster

On Thu, Feb 13, 2020, 6:28 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Hi Brian,
> I did a quick test on a active passive pulp 3.1 setup. Two pulp servers
> are pointing to the same external postgres database. Only one server is
> active at any time. Redis queue resides on the localhost. The /var/lib/pulp
> are synced from primary to the contingency host.
> After I shutdown primary host, I was able to bring up the contingency pulp
> server and created a repo. Deleting any repo stuck in a waiting state. Then
> I started primary host and shutdown contingency host, I was able to delete
> repos I created on the contingency host but all previous delete job
> continually stuck in the waiting state.
> I am wonder if anything I could do to make this work on contingency host
> or this setup is not going to work?
>
> Thanks
>
>
> From: pulp-list@redhat.com At: 01/03/20 12:01:44
> To: pulp-list@redhat.com
> Subject: Pulp-list Digest, Vol 122, Issue 1
>
> Send Pulp-list mailing list submissions to
> pulp-list@redhat.com
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://www.redhat.com/mailman/listinfo/pulp-list
> or, via email, send a message with subject or body 'help' to
> pulp-list-requ...@redhat.com
>
> You can reach the person managing the list at
> pulp-list-ow...@redhat.com
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Pulp-list digest..."
>
>
> Today's Topics:
>
> 1. Re: pulp3 High availability and disaster recovery (Brian Bouterse)
>
>
> --
>
> Message: 1
> Date: Thu, 2 Jan 2020 16:10:29 -0500
> From: Brian Bouterse 
> To: JASON STELZER 
> Cc: pulp-list 
> Subject: Re: [Pulp-list] pulp3 High availability and disaster recovery
> Message-ID:
> 
> Content-Type: text/plain; charset="utf-8"
>
> Sorry for the late reply. Each component of Pulp itself can be deployed in
> HA configurations. Of the services Pulp's processes depend on, Redis is the
> one service that can't run as a full cluster because RQ doesn't support
> that yet, so the best you can do is a hot-spare Redis that auto-fails over.
> That isn't graceful failover so when traffic routes to your hot-spare Redis
> it has to data and doesn't have the tasking system's data. Those Pulp tasks
> would be cancelled, and Pulp would be immediately ready to accept new tasks
> so they could be resubmitted, e.g. Katello resubmits some job failures I
> believe.
>
> More docs about this are here:
> https://docs.pulpproject.org/components.html#architecture-and-deploying
> More questions are welcome; sorry for the slow response. If you can see any
> way to improve the docs and want to get involved, PRs are welcome!
>
> -Brian
>
>
> On Mon, Nov 18, 2019 at 7:37 AM JASON STELZER 
> wrote:
>
> > For what it is worth, at heart pulp3 is a django app. So, following the
> > advice for HA and django apps generally works. A lot of it is driven by
> the
> > particulars of your use case.
> >
> > My use case is a little different than yours I'm sure. But in terms of HA
> > for now I'm good with a balancer and nodes in multiple azs, an RDS db
> with
> > failover, and regular db backups.
> >
> > In my case, the pulp3 server is far enough behind the scenes that even if
> > there were to be a several hour outage, the impact would be minimal.
> YMMV.
> >
> > Others can chime in with pulp3 specifics.
> >
> > On Fri, Nov 15, 2019 at 11:41 AM Bin Li (BLOOMBERG/ 120 PARK) <
> > bli...@bloomberg.net> wrote:
> >
> >> Does pulp3 support active/active or active/passive configuration? What
> is
> >> the strategy to restore the pulp3 service on a different server if the
> >> primary is down? Do we have any documentation on this topic?
> >>
> >> Thanks
> >> ___
> >> Pulp-list mailing list
> >> Pulp-list@redhat.com
> >> https://www.redhat.com/mailman/listinfo/pulp-list
> >
> >
> >
> > --
> > J.
> > ___
> > Pulp-list mailing list
> > Pulp-list@redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-list
> -- next part --
> An HTML attachment was scrubbed...
> URL:
> <
> https://www.redhat.com/archives/pulp-list/attachments/20200102/4cc40982/atta
> chment.html>
>
> --
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>
> End of 

Re: [Pulp-list] pulpcore 3.1 is Generally Available

[Dennis Kliban]

pulp_maven 0.1.0 are now generally available. It is compatible with
pulpcore 3.1.0. pulp_maven-client is also available.

https://pypi.org/project/pulp-maven/
https://pypi.org/project/pulp-maven-client/
https://pulp-maven.readthedocs.io/en/latest/




On Fri, Jan 31, 2020 at 7:12 AM Brian Bouterse  wrote:

> Pulpcore 3.1.0 is the first feature release of Pulp 3 and contains some
> significant features and fixes. See the release notes for all the details.
> https://docs.pulpproject.org/en/3.1.0/changes.html Here are some
> highlights:
>
> * Azure support
> * signing services for plugins to use
> * restriction of file:// paths and a new setting to mark filesystem areas
> as safe to import from
> * improved webserver auth support
> * various bugfixes including resolution of a task hanging issue, S3
> filename fix, and more
>
> Pulpcore 3.1.0 contains no user API breaking changes (ever), but plugins
> will require a compatibility release due to the plugin API bringing
> breaking changes in with each Y-release. Users should watch for
> announcements from plugins soon and upgrade when all of your plugins are
> compatible. Plugin writers can see their changelog to see what has changed
> in the plugin API here:
>
> ## New Installations
>
> If you are using a new install, you can use the ansible-installer directly
> and you'll receive 3.1.0. The quick start guides below are useful for
> that.
>
> * pulp_rpm installation -- https://pulp-rpm.readthedocs.io/en/3.0
> /installation.html
> * pulp_container installation -- https://pulp-container.readthedocs.io/en/
> 1.0/installation.html
> * pulp_file installation -- https://pulp-file.readthedocs.io/en/0.1
> .0/installation.html
>
> ## Updating from 3.0.z
>
> You should first upgrade your installer to the latest one from PyPI and
> re-run it. Starting with 3.1.0 each release will push a new version of the
> installer. If you get the new installer, you'll get a newer pulpcore
> release.
>
> ## Quickstart
>
> Pulp_rpm offers a quickstart guide (link below) which uses pulplift (link
> below) to create a not-long-living installation just to try it out. These
> same instructions can be adapted to the other plugins also.
>
> https://pulp-rpm.readthedocs.io/en/3.0/quickstart.html
> https://github.com/pulp/pulplift
>
> ## Migrating from Pulp2 -> Pulp3
>
> A Pulp 2 -> Pulp 3 migration tool is now in beta and can migrate docker
> and iso content ( https://github.com/pulp/pulp-2to3-migration ). RPM
> migrations are coming soon, but are not ready yet.
>
> ## Getting Help or Reporting Bugs
>
> Find where to get help or report bugs via the help page on the website (
> https://pulpproject.org/help/ ). If you're unsure where to start, we
> recommend emailing your question to the Pulp user mailing list (
> https://www.redhat.com/mailman/listinfo/pulp-list ).
>
> ## Feedback
>
> Let us know what you think! Please send feedback to either:
>
> * Pulp user mailing list --
> https://www.redhat.com/mailman/listinfo/pulp-list
> * Pulp developer mailing list --
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
> ## Thanks!
>
> Thank you to all the developers, integrators, testers, and users who have
> contributed to this feature release.
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulp 3 list packages

[Dennis Kliban]

This can be modified using the systemd unit file[0]. Gunicorn takes a -t
argument. It defaults to 30 seconds.

[0]
https://github.com/pulp/ansible-pulp/blob/master/roles/pulp/templates/pulpcore-api.service.j2#L13


On Fri, Jan 31, 2020 at 9:26 AM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> Thanks Brian.
>
> It looks like the worker timeout.
>
> Jan 31 09:19:30 pulpp-ob-581 gunicorn[147883]: [2020-01-31 09:19:30 -0500]
> [147883] [CRITICAL] WORKER TIMEOUT (pid:147898)
> Jan 31 09:19:30 pulpp-ob-581 gunicorn[147883]: [2020-01-31 14:19:30 +]
> [147898] [INFO] Worker exiting (pid: 147898)
> Jan 31 09:19:31 pulpp-ob-581 gunicorn[147883]: [2020-01-31 09:19:31 -0500]
> [161199] [INFO] Booting worker with pid: 161199
>
> How do I change the timeout setting in settings.py?
>
>
> From: bmbou...@redhat.com At: 01/31/20 04:18:14
> To: Bin Li (BLOOMBERG/ 120 PARK ) 
> Cc: pulp-list@redhat.com
> Subject: Re: [Pulp-list] pulp 3 list packages
>
> What do the logs say about why the gunicorn process serving pulp-api is
> dying? Would you want to file an issue https://pulp.plan.io/issues/new so
> we can do some testing?
>
> As an aside, I recommend using paging when pulling so many items from an
> API. You could decompose your large request to more, smaller requests like:
>
> http GET localhost/pulp/api/v3/content/rpm/packages/ offset=0 limit==1
> repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/
> --timeout=200
> http GET localhost/pulp/api/v3/content/rpm/packages/ offset=1
> limit==1
> repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/
> --timeout=200
>
> What's interesting about more, smaller requests is you can likely get the
> data out of Pulp a lot faster since you can engage more gunicorn processes
> in parallel. Conceptually one large query is attractive though, so maybe we
> could improve that if you file it.
>
> Another idea is to limit which fields are being returned to get at the
> data you need faster.
>
> All the best,
> Brian
>
>
>
> On Thu, Jan 30, 2020 at 2:46 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli...@bloomberg.net> wrote:
>
>> The rhel 7 servers rpm repo has more than 26k packages. I got an "502 Bad
>> Gateway" error if I tried to list all of them
>>
>> http GET localhost/pulp/api/v3/content/rpm/packages/ limit==2
>> repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/
>> --timeout=200
>>
>> What could cause this? Is there a fix?
>>
>> Thanks
>> ___
>> Pulp-list mailing list
>> Pulp-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Content Signing for Pulp 3.y -- Use Cases

[Dennis Kliban]

I've written 4 stories to start the implementation on the use cases we have
come up with so far. Do these accurately capture what was discussed?

[0] https://pulp.plan.io/issues/5943
[1] https://pulp.plan.io/issues/5944
[2] https://pulp.plan.io/issues/5945
[3] https://pulp.plan.io/issues/5946

On Mon, Dec 9, 2019 at 4:40 PM Dennis Kliban  wrote:

> I've updated the document[0] with the meeting minutes from December 5. The
> full chat logs are here[1]. I would like to host another meeting at 9:30 AM
> EST (in your time zone
> <https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-12-10=9.5-10.5>)
> on Tuesday, December 10, 2019 (tomorrow). During this meeting we will focus
> on the details of the following 2 use cases:
>
>
>-
>
>As an administrator, I can use django-admin to add/remove a Signing
>Service.
>-
>
>As a REST API user (repo admin), I can assign a Signing Service to a
>repository and provide a key signature as additional configuration.
>
>
>
> [0]
> https://docs.google.com/document/d/1HSrRduFYuidhbc8ISjUnkb3IXk6FnJ2iXEqa4vDC-pA/edit#
> [1]
> https://pulpadmin.fedorapeople.org/triage/pulp-meeting/2019/pulp-meeting.2019-12-05-14.00.log.html
>
>
> On Tue, Nov 26, 2019 at 4:44 PM Dennis Kliban  wrote:
>
>> Thank you everyone who participated in the discussion earlier today. I
>> have moved the content to a google doc[0]. You can find the summary of
>> today's discussion below the use cases. The full log of the discussion is
>> here[1].
>>
>> I would like to continue the discussion at 9 AM EST (in your time zone
>> <https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-12-5=9-10>)
>> on Thursday, December 5, 2019 in #pulp-meeting on IRC. Your
>>
>> [0]
>> https://docs.google.com/document/d/1HSrRduFYuidhbc8ISjUnkb3IXk6FnJ2iXEqa4vDC-pA/edit#
>> [1]
>> https://pulpadmin.fedorapeople.org/triage/pulp-dev/2019/pulp-dev.2019-11-26-14.30.log.html
>>
>> On Thu, Nov 21, 2019 at 8:27 AM Dennis Kliban  wrote:
>>
>>> On Thu, Nov 21, 2019 at 7:59 AM Dennis Kliban 
>>> wrote:
>>>
>>>> I will be hosting a chat meeting to discuss use cases for signing in
>>>> Pulp 3.y.
>>>> The meeting details and agenda link are below. Both users and plugin
>>>> developers are invited. Please join if you're interested!
>>>>
>>>> When: Wednesday, November 26 9:30 – 10:30am EST or in your time zone
>>>> <https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-11-26=9.5-10.5>
>>>> .
>>>>
>>>
>>> Correction: this will take place on Tuesday. The link is correct.
>>>
>>>
>>>> Where: #pulp-dev on freenode
>>>> Agenda: https://etherpad.net/p/Pulp_-_Signing_Use_Cases
>>>>
>>>> Questions and feedback are also welcome ahead of time.
>>>>
>>>> Thanks!
>>>>
>>>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Content Signing for Pulp 3.y -- Use Cases

[Dennis Kliban]

I've updated the document[0] with the meeting minutes from December 5. The
full chat logs are here[1]. I would like to host another meeting at 9:30 AM
EST (in your time zone
<https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-12-10=9.5-10.5>)
on Tuesday, December 10, 2019 (tomorrow). During this meeting we will focus
on the details of the following 2 use cases:


   -

   As an administrator, I can use django-admin to add/remove a Signing
   Service.
   -

   As a REST API user (repo admin), I can assign a Signing Service to a
   repository and provide a key signature as additional configuration.



[0]
https://docs.google.com/document/d/1HSrRduFYuidhbc8ISjUnkb3IXk6FnJ2iXEqa4vDC-pA/edit#
[1]
https://pulpadmin.fedorapeople.org/triage/pulp-meeting/2019/pulp-meeting.2019-12-05-14.00.log.html


On Tue, Nov 26, 2019 at 4:44 PM Dennis Kliban  wrote:

> Thank you everyone who participated in the discussion earlier today. I
> have moved the content to a google doc[0]. You can find the summary of
> today's discussion below the use cases. The full log of the discussion is
> here[1].
>
> I would like to continue the discussion at 9 AM EST (in your time zone
> <https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-12-5=9-10>)
> on Thursday, December 5, 2019 in #pulp-meeting on IRC. Your
>
> [0]
> https://docs.google.com/document/d/1HSrRduFYuidhbc8ISjUnkb3IXk6FnJ2iXEqa4vDC-pA/edit#
> [1]
> https://pulpadmin.fedorapeople.org/triage/pulp-dev/2019/pulp-dev.2019-11-26-14.30.log.html
>
> On Thu, Nov 21, 2019 at 8:27 AM Dennis Kliban  wrote:
>
>> On Thu, Nov 21, 2019 at 7:59 AM Dennis Kliban  wrote:
>>
>>> I will be hosting a chat meeting to discuss use cases for signing in
>>> Pulp 3.y.
>>> The meeting details and agenda link are below. Both users and plugin
>>> developers are invited. Please join if you're interested!
>>>
>>> When: Wednesday, November 26 9:30 – 10:30am EST or in your time zone
>>> <https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-11-26=9.5-10.5>
>>> .
>>>
>>
>> Correction: this will take place on Tuesday. The link is correct.
>>
>>
>>> Where: #pulp-dev on freenode
>>> Agenda: https://etherpad.net/p/Pulp_-_Signing_Use_Cases
>>>
>>> Questions and feedback are also welcome ahead of time.
>>>
>>> Thanks!
>>>
>>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Content Signing for Pulp 3.y -- Use Cases

[Dennis Kliban]

Thank you everyone who participated in the discussion earlier today. I have
moved the content to a google doc[0]. You can find the summary of today's
discussion below the use cases. The full log of the discussion is here[1].

I would like to continue the discussion at 9 AM EST (in your time zone
<https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-12-5=9-10>)
on Thursday, December 5, 2019 in #pulp-meeting on IRC. Your

[0]
https://docs.google.com/document/d/1HSrRduFYuidhbc8ISjUnkb3IXk6FnJ2iXEqa4vDC-pA/edit#
[1]
https://pulpadmin.fedorapeople.org/triage/pulp-dev/2019/pulp-dev.2019-11-26-14.30.log.html

On Thu, Nov 21, 2019 at 8:27 AM Dennis Kliban  wrote:

> On Thu, Nov 21, 2019 at 7:59 AM Dennis Kliban  wrote:
>
>> I will be hosting a chat meeting to discuss use cases for signing in Pulp
>> 3.y.
>> The meeting details and agenda link are below. Both users and plugin
>> developers are invited. Please join if you're interested!
>>
>> When: Wednesday, November 26 9:30 – 10:30am EST or in your time zone
>> <https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-11-26=9.5-10.5>
>> .
>>
>
> Correction: this will take place on Tuesday. The link is correct.
>
>
>> Where: #pulp-dev on freenode
>> Agenda: https://etherpad.net/p/Pulp_-_Signing_Use_Cases
>>
>> Questions and feedback are also welcome ahead of time.
>>
>> Thanks!
>>
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] pulp3 rc8 ssl_client_key

[Dennis Kliban]

You are right. We fixed this issue in the source code[0]. The fix will be
available in RC 9 that will ship next week.


[0] https://pulp.plan.io/issues/5770

On Tue, Nov 26, 2019 at 1:20 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli...@bloomberg.net> wrote:

> It looks rc8 is still looking from ssl_client_key attribute to sync the
> remote.
>
> I got the following error:
>
> "description": "'RpmRemote' object has no attribute 'ssl_client_key'",
> "traceback": " File
> \"/opt/utils/venv/pulp/3.6.5/lib64/python3.6/site-packages/rq/worker.py\",
> line 822, in perform_job\n rv = job.perform()\n File
> \"/opt/utils/venv/pulp/3.6.5/lib64/python3.6/site-packages/rq/job.py\",
> line 605, in perform\n self._result = self._execute()\n File
> \"/opt/utils/venv/pulp/3.6.5/lib64/python3.6/site-packages/rq/job.py\",
> line 611, in _execute\n return self.func(*self.args, **self.kwargs)\n File
> \"/opt/utils/venv/pulp/3.6.5/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\",
> line 122, in synchronize\n treeinfo = get_treeinfo_data(remote)\n File
> \"/opt/utils/venv/pulp/3.6.5/lib64/python3.6/site-packages/pulp_rpm/app/kickstart/treeinfo.py\",
> line 19, in get_treeinfo_data\n downloader =
> remote.get_downloader(url=urljoin(remote_url, namespace))\n File
> \"/opt/utils/venv/pulp/3.6.5/lib64/python3.6/site-packages/pulpcore/app/models/repository.py\",
> line 261, in get_downloader\n return self.download_factory.build(url,
> **kwargs)\n File
> \"/opt/utils/venv/pulp/3.6.5/lib64/python3.6/site-packages/pulpcore/app/models/repository.py\",
> line 219, in download_factory\n self._download_factory =
> DownloaderFactory(self)\n File
> \"/opt/utils/venv/pulp/3.6.5/lib64/python3.6/site-packages/pulpcore/download/factory.py\",
> line 68, in __init__\n self._session =
> self._make_aiohttp_session_from_remote()\n File
> \"/opt/utils/venv/pulp/3.6.5/lib64/python3.6/site-packages/pulpcore/download/factory.py\",
> line 90, in _make_aiohttp_session_from_remote\n
> key_file.write(bytes(self._remote.ssl_client_key, 'utf-8'))\n"
> },
>
> The remote already have the updated attribute client_key:
> {
> "ca_cert":
> "b8bd944ff40f1756c08743800453b724f133029725dc762ed9ce6504a828a5ec",
> "client_cert":
> "3f54707a201ac8a03484c7db9d8e804983bb7aa3a5864a30b946efae325002b0",
> "client_key":
> "5df5b2961f6c6ceca347eb25e645bb2e5702324f4c6e31fcfc6b36281e5e3e32",
> "download_concurrency": 5,
> "name": "rhel-x86_64-server-7",
> "policy": "immediate",
> "proxy_url": "http://proxy/;,
> "pulp_created": "2019-11-26T17:41:55.986500Z",
> "pulp_href":
> "/pulp/api/v3/remotes/rpm/rpm/3dbbd2f9-65d6-4d40-bd3a-64ace3071a01/",
> "pulp_last_updated": "2019-11-26T17:41:55.986524Z",
> "tls_validation": true,
> "url": "
> https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/;
> }
>
> Version that I am running is:
> "versions": [
> {
> "component": "pulpcore",
> "version": "3.0.0rc8"
> },
> {
> "component": "pulp_rpm",
> "version": "3.0.0rc1"
> },
> {
> "component": "pulp_file",
> "version": "0.1.0rc1"
> }
>
>
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Content Signing for Pulp 3.y -- Use Cases

[Dennis Kliban]

On Thu, Nov 21, 2019 at 7:59 AM Dennis Kliban  wrote:

> I will be hosting a chat meeting to discuss use cases for signing in Pulp
> 3.y.
> The meeting details and agenda link are below. Both users and plugin
> developers are invited. Please join if you're interested!
>
> When: Wednesday, November 26 9:30 – 10:30am EST or in your time zone
> <https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-11-26=9.5-10.5>
> .
>

Correction: this will take place on Tuesday. The link is correct.


> Where: #pulp-dev on freenode
> Agenda: https://etherpad.net/p/Pulp_-_Signing_Use_Cases
>
> Questions and feedback are also welcome ahead of time.
>
> Thanks!
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] Content Signing for Pulp 3.y -- Use Cases

[Dennis Kliban]

I will be hosting a chat meeting to discuss use cases for signing in Pulp
3.y.
The meeting details and agenda link are below. Both users and plugin
developers are invited. Please join if you're interested!

When: Wednesday, November 26 9:30 – 10:30am EST or in your time zone

.
Where: #pulp-dev on freenode
Agenda: https://etherpad.net/p/Pulp_-_Signing_Use_Cases

Questions and feedback are also welcome ahead of time.

Thanks!
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list