Re: [Pulp-list] pulpcore 3.1 is Generally Available
pulp_container 1.1.0 is GA with the set of new features and bugfixes. It is also compatible with pulpcore 3.1. https://pulp-container.readthedocs.io/en/1.1/changes.html https://pypi.org/project/pulp-container/1.1.0/ Regards, Ina Panova Senior Software Engineer| Pulp| Red Hat Inc. "Do not go where the path may lead, go instead where there is no path and leave a trail." On Fri, Jan 31, 2020 at 1:12 PM Brian Bouterse wrote: > Pulpcore 3.1.0 is the first feature release of Pulp 3 and contains some > significant features and fixes. See the release notes for all the details. > https://docs.pulpproject.org/en/3.1.0/changes.html Here are some > highlights: > > * Azure support > * signing services for plugins to use > * restriction of file:// paths and a new setting to mark filesystem areas > as safe to import from > * improved webserver auth support > * various bugfixes including resolution of a task hanging issue, S3 > filename fix, and more > > Pulpcore 3.1.0 contains no user API breaking changes (ever), but plugins > will require a compatibility release due to the plugin API bringing > breaking changes in with each Y-release. Users should watch for > announcements from plugins soon and upgrade when all of your plugins are > compatible. Plugin writers can see their changelog to see what has changed > in the plugin API here: > > ## New Installations > > If you are using a new install, you can use the ansible-installer directly > and you'll receive 3.1.0. The quick start guides below are useful for > that. > > * pulp_rpm installation -- https://pulp-rpm.readthedocs.io/en/3.0 > /installation.html > * pulp_container installation -- https://pulp-container.readthedocs.io/en/ > 1.0/installation.html > * pulp_file installation -- https://pulp-file.readthedocs.io/en/0.1 > .0/installation.html > > ## Updating from 3.0.z > > You should first upgrade your installer to the latest one from PyPI and > re-run it. Starting with 3.1.0 each release will push a new version of the > installer. If you get the new installer, you'll get a newer pulpcore > release. > > ## Quickstart > > Pulp_rpm offers a quickstart guide (link below) which uses pulplift (link > below) to create a not-long-living installation just to try it out. These > same instructions can be adapted to the other plugins also. > > https://pulp-rpm.readthedocs.io/en/3.0/quickstart.html > https://github.com/pulp/pulplift > > ## Migrating from Pulp2 -> Pulp3 > > A Pulp 2 -> Pulp 3 migration tool is now in beta and can migrate docker > and iso content ( https://github.com/pulp/pulp-2to3-migration ). RPM > migrations are coming soon, but are not ready yet. > > ## Getting Help or Reporting Bugs > > Find where to get help or report bugs via the help page on the website ( > https://pulpproject.org/help/ ). If you're unsure where to start, we > recommend emailing your question to the Pulp user mailing list ( > https://www.redhat.com/mailman/listinfo/pulp-list ). > > ## Feedback > > Let us know what you think! Please send feedback to either: > > * Pulp user mailing list -- > https://www.redhat.com/mailman/listinfo/pulp-list > * Pulp developer mailing list -- > https://www.redhat.com/mailman/listinfo/pulp-dev > > ## Thanks! > > Thank you to all the developers, integrators, testers, and users who have > contributed to this feature release. > ___ > Pulp-list mailing list > Pulp-list@redhat.com > https://www.redhat.com/mailman/listinfo/pulp-list ___ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list
Re: [Pulp-list] pulp 3 list packages
This can be modified using the systemd unit file[0]. Gunicorn takes a -t argument. It defaults to 30 seconds. [0] https://github.com/pulp/ansible-pulp/blob/master/roles/pulp/templates/pulpcore-api.service.j2#L13 On Fri, Jan 31, 2020 at 9:26 AM Bin Li (BLOOMBERG/ 120 PARK) < bli...@bloomberg.net> wrote: > Thanks Brian. > > It looks like the worker timeout. > > Jan 31 09:19:30 pulpp-ob-581 gunicorn[147883]: [2020-01-31 09:19:30 -0500] > [147883] [CRITICAL] WORKER TIMEOUT (pid:147898) > Jan 31 09:19:30 pulpp-ob-581 gunicorn[147883]: [2020-01-31 14:19:30 +] > [147898] [INFO] Worker exiting (pid: 147898) > Jan 31 09:19:31 pulpp-ob-581 gunicorn[147883]: [2020-01-31 09:19:31 -0500] > [161199] [INFO] Booting worker with pid: 161199 > > How do I change the timeout setting in settings.py? > > > From: bmbou...@redhat.com At: 01/31/20 04:18:14 > To: Bin Li (BLOOMBERG/ 120 PARK ) > Cc: pulp-list@redhat.com > Subject: Re: [Pulp-list] pulp 3 list packages > > What do the logs say about why the gunicorn process serving pulp-api is > dying? Would you want to file an issue https://pulp.plan.io/issues/new so > we can do some testing? > > As an aside, I recommend using paging when pulling so many items from an > API. You could decompose your large request to more, smaller requests like: > > http GET localhost/pulp/api/v3/content/rpm/packages/ offset=0 limit==1 > repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/ > --timeout=200 > http GET localhost/pulp/api/v3/content/rpm/packages/ offset=1 > limit==1 > repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/ > --timeout=200 > > What's interesting about more, smaller requests is you can likely get the > data out of Pulp a lot faster since you can engage more gunicorn processes > in parallel. Conceptually one large query is attractive though, so maybe we > could improve that if you file it. > > Another idea is to limit which fields are being returned to get at the > data you need faster. > > All the best, > Brian > > > > On Thu, Jan 30, 2020 at 2:46 PM Bin Li (BLOOMBERG/ 120 PARK) < > bli...@bloomberg.net> wrote: > >> The rhel 7 servers rpm repo has more than 26k packages. I got an "502 Bad >> Gateway" error if I tried to list all of them >> >> http GET localhost/pulp/api/v3/content/rpm/packages/ limit==2 >> repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/ >> --timeout=200 >> >> What could cause this? Is there a fix? >> >> Thanks >> ___ >> Pulp-list mailing list >> Pulp-list@redhat.com >> https://www.redhat.com/mailman/listinfo/pulp-list > > > ___ > Pulp-list mailing list > Pulp-list@redhat.com > https://www.redhat.com/mailman/listinfo/pulp-list ___ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list
[Pulp-list] pulp 3 django.request:WARNING
Hi All, Just noticed we continues get the following messages in the logs. Has anyone seen this before? How can we fix this? Jan 31 09:26:01 pulp gunicorn[147915]: 127.0.0.1 [31/Jan/2020:14:26:01 +] "GET / HTTP/1.0" 404 172 "-" "-" Jan 31 09:26:01 pulp gunicorn[147883]: pulp: django.request:WARNING: Not Found: / Jan 31 09:26:01 pulp gunicorn[147883]: 127.0.0.1 - - [31/Jan/2020:14:26:01 +] "GET / HTTP/1.0" 404 77 "-" "-" Jan 31 09:26:03 pulp gunicorn[147883]: pulp: django.request:WARNING: Not Found: / Jan 31 09:26:03 pulp gunicorn[147883]: 127.0.0.1 - - [31/Jan/2020:14:26:03 +] "GET / HTTP/1.0" 404 77 "-" "-" Jan 31 09:26:03 pulp gunicorn[147915]: 127.0.0.1 [31/Jan/2020:14:26:03 +] "GET / HTTP/1.0" 404 172 "-" "-" Jan 31 09:26:06 pulpp-ob-581 gunicorn[147915]: 127.0.0.1 [31/Jan/2020:14:26:06 +] "GET / HTTP/1.0" 404 172 "-" "-" Jan 31 09:26:06 pulp gunicorn[147883]: pulp: django.request:WARNING: Not Found: / Jan 31 09:26:06 pulp gunicorn[147883]: 127.0.0.1 - - [31/Jan/2020:14:26:06 +] "GET / HTTP/1.0" 404 77 "-" "-" Jan 31 09:26:08 pulp gunicorn[147883]: pulp: django.request:WARNING: Not Found: / Jan 31 09:26:08 pulp gunicorn[147883]: 127.0.0.1 - - [31/Jan/2020:14:26:08 +] "GET / HTTP/1.0" 404 77 "-" "-" Jan 31 09:26:08 pulp gunicorn[147915]: 127.0.0.1 [31/Jan/2020:14:26:08 +] "GET / HTTP/1.0" 404 172 "-" "-" Jan 31 09:26:11 pulp gunicorn[147915]: 127.0.0.1 [31/Jan/2020:14:26:11 +] "GET / HTTP/1.0" 404 172 "-" "-" Jan 31 09:26:11 pulp gunicorn[147883]: pulp: django.request:WARNING: Not Found: / Jan 31 09:26:11 pulp gunicorn[147883]: 127.0.0.1 - - [31/Jan/2020:14:26:11 +] "GET / HTTP/1.0" 404 77 "-" "-" Jan 31 09:26:13 pulp gunicorn[147883]: pulp: django.request:WARNING: Not Found: / ___ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list
Re: [Pulp-list] pulp 3 list packages
Thanks Brian. It looks like the worker timeout. Jan 31 09:19:30 pulpp-ob-581 gunicorn[147883]: [2020-01-31 09:19:30 -0500] [147883] [CRITICAL] WORKER TIMEOUT (pid:147898) Jan 31 09:19:30 pulpp-ob-581 gunicorn[147883]: [2020-01-31 14:19:30 +] [147898] [INFO] Worker exiting (pid: 147898) Jan 31 09:19:31 pulpp-ob-581 gunicorn[147883]: [2020-01-31 09:19:31 -0500] [161199] [INFO] Booting worker with pid: 161199 How do I change the timeout setting in settings.py? From: bmbou...@redhat.com At: 01/31/20 04:18:14To: Bin Li (BLOOMBERG/ 120 PARK ) Cc: pulp-list@redhat.com Subject: Re: [Pulp-list] pulp 3 list packages What do the logs say about why the gunicorn process serving pulp-api is dying? Would you want to file an issue https://pulp.plan.io/issues/new so we can do some testing? As an aside, I recommend using paging when pulling so many items from an API. You could decompose your large request to more, smaller requests like: http GET localhost/pulp/api/v3/content/rpm/packages/ offset=0 limit==1 repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/ --timeout=200 http GET localhost/pulp/api/v3/content/rpm/packages/ offset=1 limit==1 repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/ --timeout=200 What's interesting about more, smaller requests is you can likely get the data out of Pulp a lot faster since you can engage more gunicorn processes in parallel. Conceptually one large query is attractive though, so maybe we could improve that if you file it. Another idea is to limit which fields are being returned to get at the data you need faster. All the best, Brian On Thu, Jan 30, 2020 at 2:46 PM Bin Li (BLOOMBERG/ 120 PARK) wrote: The rhel 7 servers rpm repo has more than 26k packages. I got an "502 Bad Gateway" error if I tried to list all of them http GET localhost/pulp/api/v3/content/rpm/packages/ limit==2 repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/ --timeout=200 What could cause this? Is there a fix? Thanks___ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list ___ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list
[Pulp-list] pulpcore 3.1 is Generally Available
Pulpcore 3.1.0 is the first feature release of Pulp 3 and contains some significant features and fixes. See the release notes for all the details. https://docs.pulpproject.org/en/3.1.0/changes.html Here are some highlights: * Azure support * signing services for plugins to use * restriction of file:// paths and a new setting to mark filesystem areas as safe to import from * improved webserver auth support * various bugfixes including resolution of a task hanging issue, S3 filename fix, and more Pulpcore 3.1.0 contains no user API breaking changes (ever), but plugins will require a compatibility release due to the plugin API bringing breaking changes in with each Y-release. Users should watch for announcements from plugins soon and upgrade when all of your plugins are compatible. Plugin writers can see their changelog to see what has changed in the plugin API here: ## New Installations If you are using a new install, you can use the ansible-installer directly and you'll receive 3.1.0. The quick start guides below are useful for that. * pulp_rpm installation -- https://pulp-rpm.readthedocs.io/en/3.0 /installation.html * pulp_container installation -- https://pulp-container.readthedocs.io/en/1 .0/installation.html * pulp_file installation -- https://pulp-file.readthedocs.io/en/0.1 .0/installation.html ## Updating from 3.0.z You should first upgrade your installer to the latest one from PyPI and re-run it. Starting with 3.1.0 each release will push a new version of the installer. If you get the new installer, you'll get a newer pulpcore release. ## Quickstart Pulp_rpm offers a quickstart guide (link below) which uses pulplift (link below) to create a not-long-living installation just to try it out. These same instructions can be adapted to the other plugins also. https://pulp-rpm.readthedocs.io/en/3.0/quickstart.html https://github.com/pulp/pulplift ## Migrating from Pulp2 -> Pulp3 A Pulp 2 -> Pulp 3 migration tool is now in beta and can migrate docker and iso content ( https://github.com/pulp/pulp-2to3-migration ). RPM migrations are coming soon, but are not ready yet. ## Getting Help or Reporting Bugs Find where to get help or report bugs via the help page on the website ( https://pulpproject.org/help/ ). If you're unsure where to start, we recommend emailing your question to the Pulp user mailing list ( https://www.redhat.com/mailman/listinfo/pulp-list ). ## Feedback Let us know what you think! Please send feedback to either: * Pulp user mailing list -- https://www.redhat.com/mailman/listinfo/pulp-list * Pulp developer mailing list -- https://www.redhat.com/mailman/listinfo/pulp-dev ## Thanks! Thank you to all the developers, integrators, testers, and users who have contributed to this feature release. ___ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list
Re: [Pulp-list] pulp 3 list packages
What do the logs say about why the gunicorn process serving pulp-api is dying? Would you want to file an issue https://pulp.plan.io/issues/new so we can do some testing? As an aside, I recommend using paging when pulling so many items from an API. You could decompose your large request to more, smaller requests like: http GET localhost/pulp/api/v3/content/rpm/packages/ offset=0 limit==1 repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/ --timeout=200 http GET localhost/pulp/api/v3/content/rpm/packages/ offset=1 limit==1 repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/ --timeout=200 What's interesting about more, smaller requests is you can likely get the data out of Pulp a lot faster since you can engage more gunicorn processes in parallel. Conceptually one large query is attractive though, so maybe we could improve that if you file it. Another idea is to limit which fields are being returned to get at the data you need faster. All the best, Brian On Thu, Jan 30, 2020 at 2:46 PM Bin Li (BLOOMBERG/ 120 PARK) < bli...@bloomberg.net> wrote: > The rhel 7 servers rpm repo has more than 26k packages. I got an "502 Bad > Gateway" error if I tried to list all of them > > http GET localhost/pulp/api/v3/content/rpm/packages/ limit==2 > repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/ > --timeout=200 > > What could cause this? Is there a fix? > > Thanks > ___ > Pulp-list mailing list > Pulp-list@redhat.com > https://www.redhat.com/mailman/listinfo/pulp-list ___ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list
