Re: [Pulp-dev] How to list orphan content in pulp3 before remocing them ?
individual content types, There were endpoints available as well to see >> what are those contents. >> >> +--+ >> Summary >> +--+ >> >> Distribution:0 >> Docker Blob: 0 >> Docker Image:0 >> Docker Manifest: 0 >> Docker Manifest List:0 >> Docker Tag: 0 >> Drpm:0 >> Erratum: 105 >> Iso: 0 >> Modulemd:2 >> Modulemd Defaults: 0 >> Ostree: 0 >> Package Category:0 >> Package Environment: 0 >> Package Group: 0 >> Package Langpacks: 0 >> Puppet Module: 0 >> Rpm: 700 >> Srpm:0 >> Yum Repo Metadata File: 0 >> Total: 807 >> >> >> So, I guess, It makes sense to have something similar in Pulp3 as well. >> >> I will try out the method you have mentioned and see what else can be >> done with it. >> >> >> Thanks & Regards, >> >> Sayan das >> >> *S*enior* T*echnical *S*upport *E*ngineer, RHCE >> >> Red Hat India >> <https://www.redhat.com/> >> >> Red Hat India Pvt. Ltd, Level-5, Tower-10, Cyber City >> >> Magarpatta City Hadapsar, Pune-411013, Maharashtra, India. >> >> say...@redhat.comM: +91-7890892756 IRC: Sayan >> <https://red.ht/sig> >> >> >> On Mon, Jan 31, 2022 at 5:59 PM Grant Gainey wrote: >> >>> Hey Sayan, >>> >>> The following dumps all the Content-objects that are orphaned (ie, "not >>> assigned to a repository and not touched in the last >>> "minutes_since_touched" minutes). Needs some expansion, but that depends on >>> what you want to get out of it. Artifact checksums? Content "name" >>> (whatever that means)? UUIDs? >>> >>> $ pulpcore-manager shell >>> from pulpcore.app.models.content import Content >>> minutes_since_touched = 1 >>> print(Content.objects.orphaned(minutes_since_touched).all()) >>> >> pk=2aee273b-3c1d-493a-9849-5ea717076795>, >> (pulp_type=rpm.distribution_tree): >>> pk=7aa4d756-fb5b-4630-bfe6-9e9c4f7461ee>, >> (pulp_type=rpm.packagegroup): shark>, >> (pulp_type=rpm.packagecategory): all>, >> (pulp_type=rpm.packageenvironment): SharkEnvironment>]> >>> >>> G >>> >>> On Mon, Jan 31, 2022 at 3:33 AM Sayan Das wrote: >>> >>>> Hello All, >>>> >>>> I hope everyone is doing great. >>>> >>>> I am reaching out to the developers here as I wanted to find out *How >>>> I can list all orphan contents in Pulp 3* before deleting them? >>>> >>>> I have checked in Pulpcore API as well as Pulp cli but The only option >>>> I see is to remove the orphan contents directly. >>>> >>>> There are many situations where due to customer demand or our own >>>> understanding we may need to understand what all orphan data is present >>>> before we clear them up. >>>> >>>> Keeping that in mind, can anyone please suggest or confirm if we have >>>> any way to simply list the orphan content details in Pulp 3 similar to how >>>> we were able to do with Pulp 2? >>>> >>>> >>>> >>>> Thanks & Regards, >>>> >>>> Sayan das >>>> >>>> *S*enior* T*echnical *S*upport *E*ngineer, RHCE >>>> >>>> Red Hat India >>>> <https://www.redhat.com/> >>>> >>>> Red Hat India Pvt. Ltd, Level-5, Tower-10, Cyber City >>>> >>>> Magarpatta City Hadapsar, Pune-411013, Maharashtra, India. >>>> >>>> say...@redhat.comM: +91-7890892756 IRC: Sayan >>>> <https://red.ht/sig> >>>> ___ >>>> Pulp-dev mailing list >>>> Pulp-dev@redhat.com >>>> https://listman.redhat.com/mailman/listinfo/pulp-dev >>>> >>> >>> >>> -- >>> Grant Gainey >>> Principal Software Engineer, Red Hat System Management Engineering >>> >> -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] How to list orphan content in pulp3 before remocing them ?
Hey Sayan, The following dumps all the Content-objects that are orphaned (ie, "not assigned to a repository and not touched in the last "minutes_since_touched" minutes). Needs some expansion, but that depends on what you want to get out of it. Artifact checksums? Content "name" (whatever that means)? UUIDs? $ pulpcore-manager shell from pulpcore.app.models.content import Content minutes_since_touched = 1 print(Content.objects.orphaned(minutes_since_touched).all()) , , , , ]> G On Mon, Jan 31, 2022 at 3:33 AM Sayan Das wrote: > Hello All, > > I hope everyone is doing great. > > I am reaching out to the developers here as I wanted to find out *How I > can list all orphan contents in Pulp 3* before deleting them? > > I have checked in Pulpcore API as well as Pulp cli but The only option I > see is to remove the orphan contents directly. > > There are many situations where due to customer demand or our own > understanding we may need to understand what all orphan data is present > before we clear them up. > > Keeping that in mind, can anyone please suggest or confirm if we have any > way to simply list the orphan content details in Pulp 3 similar to how we > were able to do with Pulp 2? > > > > Thanks & Regards, > > Sayan das > > *S*enior* T*echnical *S*upport *E*ngineer, RHCE > > Red Hat India > <https://www.redhat.com/> > > Red Hat India Pvt. Ltd, Level-5, Tower-10, Cyber City > > Magarpatta City Hadapsar, Pune-411013, Maharashtra, India. > > say...@redhat.comM: +91-7890892756 IRC: Sayan > <https://red.ht/sig> > _______ > Pulp-dev mailing list > Pulp-dev@redhat.com > https://listman.redhat.com/mailman/listinfo/pulp-dev > -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] pulp-certguard 1.5.2 is Generally Available
Details can be found in Discourse: https://discourse.pulpproject.org/t/pulp-certguard-1-5-2-is-generally-available/284 Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp Integration Meeting
Minutes posted to Discourse: https://discourse.pulpproject.org/t/katello-pulp3-integration-meeting/37/21 Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] pulp-certguard 1.5.1 is Generally Available
Details can be found in Discourse : https://discourse.pulpproject.org/t/pulp-certguard-1-5-1-is-generally-available/182 Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Status
2021-09-29 * rebased PR pulpcore/pull/1653 <https://github.com/pulp/pulpcore/pull/1653> for 3.14 after major changes merged into 3.14 - fun! * 2006307 <https://bugzilla.redhat.com/show_bug.cgi?id=2006307> (2to3 OOM) - investigating w/ QE - unconvinced it's a Pulp issue * 9464 <https://pulp.plan.io/issues/9464> / 2009049 <https://bugzilla.redhat.com/show_bug.cgi?id=2009049> (basic-auth on remote-URL broken) - investigation, discussion of options w/ dralley/bmbouter/daviddavis - submitted PR pulp_rpm/pull/2138 <https://github.com/pulp/pulp_rpm/pull/2138>, merged - backport request/PR : 9472 <https://pulp.plan.io/issues/9472> / pulp_rpm/pull/2139 <https://github.com/pulp/pulp_rpm/pull/2139> , merged * reviewed and/or merged PRs - pulp-cli/pull/369 <https://github.com/pulp/pulp-cli/pull/369> - pulp-cli/pull/353 <https://github.com/pulp/pulp-cli/pull/353> * discussion w/ jsherrill/dsingh RE 2004397 <https://bugzilla.redhat.com/show_bug.cgi?id=2004397> * CLI Mtg * Katello/Pulp Integration G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp Integration Meeting
Minutes posted to Discourse: https://discourse.pulpproject.org/t/katello-pulp3-integration-meeting/37/20 Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp Integration Meeting
Minutes posted to Discourse: https://discourse.pulpproject.org/t/katello-pulp3-integration-meeting/37/19 Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp Integration Meeting
Minutes posted to Discourse: https://discourse.pulpproject.org/t/katello-pulp3-integration-meeting/37/18 Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp Integration Meeting
Minutes posted to discourse: https://discourse.pulpproject.org/t/katello-pulp3-integration-meeting/37/16 Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp/Katello Integration meeting 11-AUG (moving to Discourse!)
Hey folks, If you haven't seen David Davis' post <https://listman.redhat.com/archives/pulp-dev/2021-August/msg8.html> from earlier today, Pulp is moving to its own Discourse server. Once we've pulled the data from github.com/pulp/community, we'll decommission that site. Therefore - please find minutes from today's integration-meeting in their new home: https://discourse.pulpproject.org/t/katello-pulp3-integration-meeting/37/13 <https://discourse.pulpproject.org/t/katello-pulp3-integration-meeting/37/13> Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp/Katello Integration meeting 04-AUG
Minutes posted : https://github.com/pulp/community/discussions/4#discussioncomment-1135820 Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] pulp-certguard 1.5.0 is Generally Available
pulp-certguard 1.5.0 has been released. This version is compatible with pulpcore 3.10 and the upcoming 3.15. This release adds python-3.8 support and drops python 3.6/3.7 support. PyPI: https://pypi.org/project/pulp-certguard/1.5.0/ Changelog: https://docs.pulpproject.org/pulp_certguard/en/1.5.0/changes.html#id1 Docs: https://docs.pulpproject.org/pulp_certguard/ Python bindings: https://pypi.org/project/pulp-certguard-client/1.5.0/ Ruby bindings: https://rubygems.org/gems/pulp_certguard_client/versions/1.5.0 Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Today's Discussions topic: Deadlocks!
Hullo pulp-devs, I have been investigating some deadlocks we've seen in pulp3 code, and want to share the Fun! Please check out the discussion at https://github.com/pulp/community/discussions/77 , and feel free to chime in with ideas, objections, sympathies, or even just to point and laugh :) G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp/Katello Integration meeting 21-JUL
Minutes posted : https://github.com/pulp/community/discussions/4#discussioncomment-1083364 Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp/Katello Integration meeting 14-JUL
Minutes posted : https://github.com/pulp/community/discussions/4#discussioncomment-1033849 Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp/Katello Integration meeting 14-JUL
Minutes posted : https://github.com/pulp/community/discussions/4#discussioncomment-1004869 Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp/Katello Integration meeting 07-JUL
Minutes posted : https://github.com/pulp/community/discussions/4#discussioncomment-976730 Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp/Katello Integration meeting 30-JUN
Minutes posted : https://github.com/pulp/community/discussions/4#discussioncomment-944705 Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] [Pulp-list] Github Discussions
There are a few places in the docs where we reference pulp-dev@ , that should be changed as part of/before we pull the plug. contributing/index.rst: * through the developer mailing list (`` pulp-dev@redhat.com``) plugins/plugin-writer/concepts/index.rst:maintainers either through the developer mailing list (``pulp-dev@redhat.com``) or on Freenode in plugins/index.rst: Are we missing a plugin? Let us know via the pulp-dev@redhat.com mailing list. On Wed, Jun 16, 2021 at 11:55 AM David Davis wrote: > Yesterday at open floor, we discussed decommissioning pulp-dev list in > favor of of using Github Discussions[0] for developer discussions. > > If there are no objections, I plan to decommission the pulp-dev list next > week. > > [0] https://github.com/pulp/community/discussions > > David > > > On Thu, Jun 3, 2021 at 11:01 AM Brian Bouterse > wrote: > >> I did this also for the pulpcore meeting: >> https://github.com/pulp/community/discussions/8 >> >> My format was a little different, but the same idea. >> >> On Thu, Jun 3, 2021 at 10:13 AM Grant Gainey wrote: >> >>> On Thu, Jun 3, 2021 at 9:40 AM David Davis >>> wrote: >>> >>>> Based on feedback, I've moved discussions to its own repo: >>>> https://github.com/pulp/community/discussions. >>>> >>> >>> Brillliant! >>> >>> One discovery I made this week - for 'Meetings' threads that exist to >>> have meeting-minutes posted, the first entry should be a description of >>> what the meeting you're recording is for, and each set of minutes should be >>> a comment. This lets the reader sort by "Newest" and get >>> most-recent-minutes-first.The initial message in a discussion is always at >>> the top, no matter how you sort - so if it's your first meeting-minutes, >>> they'll always be first. >>> >>> I redid the katello/pulp and community/pulp integration >>> discussion-threads (in their new location) in light of this, apologies to >>> anyone who got some notification-spam as a result this morning. >>> >>>- https://github.com/pulp/community/discussions/7 >>>- https://github.com/pulp/community/discussions/4 >>> >>> G >>> >>>> >>>> David >>>> >>>> >>>> On Tue, May 25, 2021 at 1:49 PM David Davis >>>> wrote: >>>> >>>>> We've heard from the community about the amount of friction involved >>>>> in getting help with Pulp and one of the areas I think we could improve is >>>>> user communications. We currently run two mailing lists: pulp-list and >>>>> pulp-dev. >>>>> >>>>> At today's open floor meeting, we talked about using Github's new >>>>> Discussions feature[0] to host these conversations instead. I've set up a >>>>> Discussion against pulpcore[1] for us to try but here's also an example of >>>>> a project that has a lot of threads[2]. >>>>> >>>>> I think the consensus was that we'd just keep pulpcore as our one and >>>>> only Github Discussions instance, which would serve as a replacement for >>>>> pulp-list and pulp-dev. I'd propose that we try this out for a bit and >>>>> eventually decommission our mailing lists. >>>>> >>>>> [0] https://docs.github.com/en/discussions >>>>> [1] https://github.com/pulp/pulpcore/discussions >>>>> [2] https://github.com/vercel/next.js/discussions >>>>> >>>>> David >>>>> >>>> ___ >>>> Pulp-list mailing list >>>> pulp-l...@redhat.com >>>> https://listman.redhat.com/mailman/listinfo/pulp-list >>> >>> >>> >>> -- >>> Grant Gainey >>> Principal Software Engineer, Red Hat System Management Engineering >>> ___ >>> Pulp-dev mailing list >>> Pulp-dev@redhat.com >>> https://listman.redhat.com/mailman/listinfo/pulp-dev >>> >> -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp-Community Integration meeting minutes
Minutes can be found in the GitHub Discussion thread: https://github.com/pulp/community/discussions/4?sort=new#discussioncomment-846700 <https://github.com/pulp/community/discussions/4?sort=new#discussioncomment-846700> G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] [Pulp-list] Github Discussions
On Thu, Jun 3, 2021 at 9:40 AM David Davis wrote: > Based on feedback, I've moved discussions to its own repo: > https://github.com/pulp/community/discussions. > Brillliant! One discovery I made this week - for 'Meetings' threads that exist to have meeting-minutes posted, the first entry should be a description of what the meeting you're recording is for, and each set of minutes should be a comment. This lets the reader sort by "Newest" and get most-recent-minutes-first.The initial message in a discussion is always at the top, no matter how you sort - so if it's your first meeting-minutes, they'll always be first. I redid the katello/pulp and community/pulp integration discussion-threads (in their new location) in light of this, apologies to anyone who got some notification-spam as a result this morning. - https://github.com/pulp/community/discussions/7 - https://github.com/pulp/community/discussions/4 G > > David > > > On Tue, May 25, 2021 at 1:49 PM David Davis wrote: > >> We've heard from the community about the amount of friction involved in >> getting help with Pulp and one of the areas I think we could improve is >> user communications. We currently run two mailing lists: pulp-list and >> pulp-dev. >> >> At today's open floor meeting, we talked about using Github's new >> Discussions feature[0] to host these conversations instead. I've set up a >> Discussion against pulpcore[1] for us to try but here's also an example of >> a project that has a lot of threads[2]. >> >> I think the consensus was that we'd just keep pulpcore as our one and >> only Github Discussions instance, which would serve as a replacement for >> pulp-list and pulp-dev. I'd propose that we try this out for a bit and >> eventually decommission our mailing lists. >> >> [0] https://docs.github.com/en/discussions >> [1] https://github.com/pulp/pulpcore/discussions >> [2] https://github.com/vercel/next.js/discussions >> >> David >> > ___ > Pulp-list mailing list > pulp-l...@redhat.com > https://listman.redhat.com/mailman/listinfo/pulp-list -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration meeting
Hey folks, Pulp is experimenting with using Github Discussions for communications instead of email lists, as a way of keeping discussions closer to the code. You will find the meeting-minutes for $TOPIC here: https://github.com/pulp/pulpcore/discussions/1366#discussioncomment-816813 Let us know what you think! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp-Community Integration meeting minutes
We're experimenting with GitHub discussions - find the meeting-minutes in this thread: https://github.com/pulp/pulpcore/discussions/1381 G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration meeting
Hey folks, Pulp is experimenting with using Github Discussions for communications instead of email lists, as a way of keeping discussions closer to the code. You will find the meeting-minutes for $TOPIC here: https://github.com/pulp/pulpcore/discussions/1366 Let us know what you think! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp Certguard 1.3.0 is Generally Available
pulp-certguard 1.3.0 has been released. It is compatible with pulpcore versions 3.10 through future version 3.13. This is only a compatibility release. PyPi: https://pypi.org/project/pulp-certguard/1.3.0/ Docs: https://docs.pulpproject.org/pulp_certguard/ Python bindings: https://pypi.org/project/pulp-certguard-client/1.3.0/ Ruby: https://rubygems.org/gems/pulp_certguard_client/versions/1.3.0/ Enjoy! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration meeting
May 19, 2021 Overview - Katello Schedule - 4.1 branching ~May 2021 - pulpcore 3.11 (or newer) - 4.2 branching ~August 2021 - Pulpcore 3.13 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.13 now scheduled for 25-May - Dalley is release-nanny - 3.11.2 in-progress - 4 requested backports - Requires django-2.2.23, pins click to 7.1.2 - Is there work needed RE cleaning up media-dirs? Let bmbouter know! - Logging suggestion/concern - https://listman.redhat.com/archives/pulp-dev/2021-May/msg00052.html - Already an existing issue RE correlation-id-log-chattiness - Feedback needed https://pulp.plan.io/issues/8459 reclaim disk space story - RPM - 3.11 is released - Adds support for modularity changes (static_context) - Last Y release compatible with pulpcore < 3.12 - Consider upgrading pulp_rpm in Katello 3.18.z - 3.12 should be out this week - Auto-publish support - ULN (Oracle) support - Repodata mirroring is close to completion, will be released in core-3.13 - Migration - Ansible - 0.8.0 will be released for pulpcore 3.13 compatibility - Add dependency syncing flag on CollectionRemote sync_dependencies - Now has label support - Pulp Container - Upcoming 2.5.3 and compatibility release 2.6.0 with core-3.13 - Pulp CLI - 0.9.0 released <https://listman.redhat.com/archives/pulp-list/2021-May/msg00035.html> - Pins click to 7.1.2 - Adds --dry-run flag - Pulp-certguard - Requests from Eric RE chained-CAs - we’re using the wrong CA into certguard - Would be nice to pass a bundle to certguard and have it Do The Right Thing - Issue filed - https://pulp.plan.io/issues/8783 - Compatibility-release coming out #soon - Release-automation-improvements happening - Tasking-system-regression in Pulp2 - solved - Discussion around odd Pulp3-tasking-weirdness - if/when encountered locally, hop into #pulp-dev and let us know! - Should Pulp2 discussions/issues come here? - When wider comms needed, probably a good place for it Katello - Working on upgrading to pulp-rpm 3.11 - Nightly to start - If All Goes Well, will look at release-cadence for 3.18/4.0 - Ryan our intern started, will be working on ‘Generic content’ and python content support - Katello-4.2 will prob be compatible with core-3.13 QE - latest-z-release migration & bz testing is done - Start looking at next y-release features and test pre-snap G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp Integration meeting
May 12, 2021 Overview - Katello Schedule - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - Django 2.2.23 expected May 13th - https://github.com/django/django/pull/14372/files#diff-4e258c842491991e7d5af289ca7f850661292ab721c4ff3ee1c6db52434b21daR5 - Required pulp change will be released into 3.7, 3.9, and 3.11 - Also backporting this change https://pulp.plan.io/issues/8712 - Pulpcore 3.13 expected tentatively May 18th. - Alternate Content Source to be delivered in Julyish - New Pulp tasking system - Off by default, eventually will become default - Available in 3.14 - Does not use Redis, but we’re keeping Redis in the stack for now - Higher throughput, makes Pulp fully HA, no resource manager - https://pulp.plan.io/issues/8495 - Need to change how you run the workers and set a setting to opt-in - RPM - Working on a 3.7-thru-3.11 compatible release (pulp-rpm-3.11) - Includes modulemd static_context change - Might affect applicability in katello - Requires some orchestration/rework in git - in-progress - Very very few changes from 3.10 - A few known issues with BZs - “Soon” ™ (as priorities allow) - https://pulp.plan.io/issues/8722 - https://pulp.plan.io/issues/8713 - Migration - Investigating post-migration capsule (pulp3->pulp2) sync - Ansible - No updates - Pulp Container - 2.5.3 incoming for katello-4.1 but depends on the Django fix - Assisting with BZ verification - Pulp CLI - New version of click (8.0) causes the CLI to break - https://github.com/clarkperkins/click-shell/pull/29 - If ^ isn’t merged/released soon, we’ll pin and release - What about pulp_python? - Very possibly - stay tuned Katello - Finished discussing alternate content sources, ostree, metadata mirroring this week for estimates - Katello 4.1 RC soon - Container Gateway “done” - Removing more Pulp 2 code, applicability related code is now gone QE - Finishing up remaining BZs for migration - Looking into katello-4.1 features - Automation failures -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] pulp_rpm and current backwards-compatibility problems
I'm sorry, this is going to be a noisy thread for a while - but this is Complicated, and I want to make sure everyone stays on the same page as we get ourselves out of this thicket... On Tue, May 11, 2021 at 4:20 PM Grant Gainey wrote: > Whew. So anyway, the sequence to unsnarl this goes something like so: > > 1. get pulp_rpm CI building against pulpcore-3.12 > - PR#1985 <https://github.com/pulp/pulp_rpm/pull/1985> > - figure out how to pin django-2.2.20 before pip tries to install 2.2.21 > This step is *completed*. PR#1985 <https://github.com/pulp/pulp_rpm/pull/1985> was modified and built successfully! fao89 points out that the trick we're doing w/ requirements.txt won't work for release-jobs because they install pulpcore from pypi. Suggestion: an alternative would be pinning django on the container image - https://github.com/pulp/pulp_rpm/blob/master/.ci/ansible/Containerfile.j2#L13-L22 fao891: yeah that's what we need right there So that can be something to look at going forward. Next up is "revert The Things", but I'm not going to do that at the end of a long day. G 2. Revert the following commits in pulp_rpm: > - d0c9badd Refactor distribution migration 0032 > - fbaadaca Add support for automatic publishing and distributing > - 3157ad Oracle ULN > 3. get the static_context change updated (since a migration will have Left > the Building) and get it merged > 4. merge any other fixes that won't break 3.7-compat [OPTIONAL], and THEN > 5. cut pulp-rpm/3.11 as compatible with pulpcore/3.7-thru-3.10 > > Once 3.11 is in the can, then we : > 1. Re-apply the reverted PRs and resolve conflicts (there will be some) > 2. remove the dependency on enqueue_with_reservation() > 3. Unpin pulp_rpm from 3.12/django-2.2.20 > 4 Release a pulp_rpm/3.12 that will be ready for pulppcore/3.13 to be > released > > I feel like I've missed a step somewhere - dalley, bmbouter, was there > something else form the call we just had? Or is this It? > > Whew! > > OK, I am off to adjust PR-1985 to do step-1 above. > > G > > On Tue, May 11, 2021 at 11:49 AM Grant Gainey wrote: > >> Hey folks, >> >> We've been talking about how we need a pulpcore/3.7-to-3.11-compatible >> release of pulp_rpm. The static_context change requires a schema-change, >> and it has to be available to katello-3.18 (and hence pulpcore-3.7) >> >> The static_context change is PR#1984 >> <https://github.com/pulp/pulp_rpm/pull/1984> >> >> Right now, pulp-rpm/master has changes that require pulpcore/3.12 or >> later. Those changes are: >> >> d0c9badd Refactor distribution migration 0032 >> fbaadaca Add support for automatic publishing and distributing >> >> In addition, pulp_rpm/master is *currently broken* because it still >> references the deprecated enqueue_with_reservation(), that just got removed >> from pulpcore/master. >> >> As I understand it, what needs to happen #SOON, is the following: >> >>1. revert the two commits above and merge, >>2. get the static_context change updated (since a migration will have >>Left the Building) and get it merged, >>3. merge any other fixes that won't break 3.7-compat [OPTIONAL], and >>THEN >>4. cut 3.11 as compatible with 3.7-thru-3.10 pulpcore >> >> Once pulp_rpm/3.11 is released, we can then: >> >> 1. re-apply the auto-pub/dist-schema changes, >>2. fix enqueue-problem, >> 3. mark pulp_rpm/master as 3.12+ compat, and finally >>4. release pulp_rpm/3.12 to be ready for pulpcore-3.13 >> >> And this all needs to happen by next week? >> >> Is there anything I'm missing here? >> >> G >> -- >> Grant Gainey >> Principal Software Engineer, Red Hat System Management Engineering >> > > > -- > Grant Gainey > Principal Software Engineer, Red Hat System Management Engineering > -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] pulp_rpm and current backwards-compatibility problems
Hey gang - After learning of some other Complications, the timeline/steps I've listed need to be a little more complete. The key additional points are - there's a third pulp_rpm commit that needs post-core-3.12 code (3157ad <https://github.com/pulp/pulp_rpm/commit/3157ad10abf777009620ffa421b3eab45d85a5bb> Oracle ULN) - pulpcore PR 1318 <https://github.com/pulp/pulpcore/pull/1318> breaks pulp_rpm because we can't move to using dispatch() until after the dance we're doing right now; - pulp_rpm can't pin against an earlier version of pulpcore, because they all will break due to the Django Problem - Once Django releases their fix, fixing the Django Problem for us requires bmbouter's patch Whew. So anyway, the sequence to unsnarl this goes something like so: 1. get pulp_rpm CI building against pulpcore-3.12 - PR#1985 <https://github.com/pulp/pulp_rpm/pull/1985> - figure out how to pin django-2.2.20 before pip tries to install 2.2.21 2. Revert the following commits in pulp_rpm: - d0c9badd Refactor distribution migration 0032 - fbaadaca Add support for automatic publishing and distributing - 3157ad Oracle ULN 3. get the static_context change updated (since a migration will have Left the Building) and get it merged 4. merge any other fixes that won't break 3.7-compat [OPTIONAL], and THEN 5. cut pulp-rpm/3.11 as compatible with pulpcore/3.7-thru-3.10 Once 3.11 is in the can, then we : 1. Re-apply the reverted PRs and resolve conflicts (there will be some) 2. remove the dependency on enqueue_with_reservation() 3. Unpin pulp_rpm from 3.12/django-2.2.20 4 Release a pulp_rpm/3.12 that will be ready for pulppcore/3.13 to be released I feel like I've missed a step somewhere - dalley, bmbouter, was there something else form the call we just had? Or is this It? Whew! OK, I am off to adjust PR-1985 to do step-1 above. G On Tue, May 11, 2021 at 11:49 AM Grant Gainey wrote: > Hey folks, > > We've been talking about how we need a pulpcore/3.7-to-3.11-compatible > release of pulp_rpm. The static_context change requires a schema-change, > and it has to be available to katello-3.18 (and hence pulpcore-3.7) > > The static_context change is PR#1984 > <https://github.com/pulp/pulp_rpm/pull/1984> > > Right now, pulp-rpm/master has changes that require pulpcore/3.12 or > later. Those changes are: > > d0c9badd Refactor distribution migration 0032 > fbaadaca Add support for automatic publishing and distributing > > In addition, pulp_rpm/master is *currently broken* because it still > references the deprecated enqueue_with_reservation(), that just got removed > from pulpcore/master. > > As I understand it, what needs to happen #SOON, is the following: > >1. revert the two commits above and merge, >2. get the static_context change updated (since a migration will have >Left the Building) and get it merged, >3. merge any other fixes that won't break 3.7-compat [OPTIONAL], and >THEN >4. cut 3.11 as compatible with 3.7-thru-3.10 pulpcore > > Once pulp_rpm/3.11 is released, we can then: > >1. re-apply the auto-pub/dist-schema changes, >2. fix enqueue-problem, >3. mark pulp_rpm/master as 3.12+ compat, and finally >4. release pulp_rpm/3.12 to be ready for pulpcore-3.13 > > And this all needs to happen by next week? > > Is there anything I'm missing here? > > G > -- > Grant Gainey > Principal Software Engineer, Red Hat System Management Engineering > -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] pulp_rpm and current backwards-compatibility problems
On Tue, May 11, 2021 at 1:36 PM Grant Gainey wrote: > > I'm going to submit a PR to get pulp_rpm's CI unbroken, that has to happen > before we do anything else. THEN I will respond to final review-comments on > PR 1984. Once that passes, THEN we can talk about what happens next. (That > PR, for example, is affected by the proposed reverts, because its migration > is currently built on-top-of the one(s) being reverted and will need to be > renamed. So we/I probably don't want to merge it, until post-revert, and an > update to match the new State Of The World at that point.) > > Anyway, first things first... > Update: no, I'm not. Marking pulp_rpm as "requires pulpcore/3.12" fails to build. Because core-3.12 depends on django~=2.2.20, which pulls in 2.2.21, which is the Broken Django, and Everything Is Sad. (Mostly me, to be honest) Right this second, we can't build against core-master, and we can't build against anything other than -master because of the Django Problem. PR#1985 <https://github.com/pulp/pulp_rpm/pull/1985> is waiting while I think about how to make progress. G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] pulp_rpm and current backwards-compatibility problems
On Tue, May 11, 2021 at 1:18 PM Tanya Tereshchenko wrote: > Hi Grant, > > Thanks for putting this together. > It sounds about right, as a general idea of what needs to happen. > > I would leave it to a person who performs all the git fu to figure out > exact commits and details and to the reviewer of all those changes, when PR > 1984 is ready to be merged and releases can be done. > To say yes, or no, to the plan would mean to check properly all the > commits/details and it likely would be more time effective to do it when > the time comes to perform all those steps. > Concur. > Off the top of my head: > - pulp_rpm 3.11 needs to be compatible with pulpcore >=3.7,<3.13 as 3.10 > does. > - `re-apply the auto-pub/dist-schema changes` should include a migration > update to adjust the dependency in it. > > David, > All the rush and hassle is happening because of the changes with > migrations and compatibility with an older release. Some commits which > need to be reapplied have migrations. > Whichever way a person who does it decides to do, the migrations order > should be the same on 3.11 branch and on the master. > I'm going to submit a PR to get pulp_rpm's CI unbroken, that has to happen before we do anything else. THEN I will respond to final review-comments on PR 1984. Once that passes, THEN we can talk about what happens next. (That PR, for example, is affected by the proposed reverts, because its migration is currently built on-top-of the one(s) being reverted and will need to be renamed. So we/I probably don't want to merge it, until post-revert, and an update to match the new State Of The World at that point.) Anyway, first things first... G > > Tanya > > On Tue, May 11, 2021 at 7:02 PM David Davis wrote: > >> What if you create a 3.11 release branch and then revert the commits on >> the 3.11 branch? That would save you from having to reapply the two >> commits. >> >> You could also pin to pulpcore < 3.12 on the 3.11 branch to get the >> branch passing while you work on fixing the enqueue problem on master. >> >> David >> >> >> On Tue, May 11, 2021 at 11:49 AM Grant Gainey wrote: >> >>> Hey folks, >>> >>> We've been talking about how we need a pulpcore/3.7-to-3.11-compatible >>> release of pulp_rpm. The static_context change requires a schema-change, >>> and it has to be available to katello-3.18 (and hence pulpcore-3.7) >>> >>> The static_context change is PR#1984 >>> <https://github.com/pulp/pulp_rpm/pull/1984> >>> >>> Right now, pulp-rpm/master has changes that require pulpcore/3.12 or >>> later. Those changes are: >>> >>> d0c9badd Refactor distribution migration 0032 >>> fbaadaca Add support for automatic publishing and distributing >>> >>> In addition, pulp_rpm/master is *currently broken* because it still >>> references the deprecated enqueue_with_reservation(), that just got removed >>> from pulpcore/master. >>> >>> As I understand it, what needs to happen #SOON, is the following: >>> >>>1. revert the two commits above and merge, >>>2. get the static_context change updated (since a migration will >>>have Left the Building) and get it merged, >>>3. merge any other fixes that won't break 3.7-compat [OPTIONAL], and >>>THEN >>>4. cut 3.11 as compatible with 3.7-thru-3.10 pulpcore >>> >>> Once pulp_rpm/3.11 is released, we can then: >>> >>>1. re-apply the auto-pub/dist-schema changes, >>>2. fix enqueue-problem, >>>3. mark pulp_rpm/master as 3.12+ compat, and finally >>>4. release pulp_rpm/3.12 to be ready for pulpcore-3.13 >>> >>> And this all needs to happen by next week? >>> >>> Is there anything I'm missing here? >>> >>> G >>> -- >>> Grant Gainey >>> Principal Software Engineer, Red Hat System Management Engineering >>> ___ >>> Pulp-dev mailing list >>> Pulp-dev@redhat.com >>> https://listman.redhat.com/mailman/listinfo/pulp-dev >>> >> -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] pulp_rpm and current backwards-compatibility problems
On Tue, May 11, 2021 at 1:02 PM David Davis wrote: > What if you create a 3.11 release branch and then revert the commits on > the 3.11 branch? That would save you from having to reapply the two > commits. > But then we'd have to cherry-pick lots more, one of which would be a schema-change :( I'm doing an experiment right now, and the reverts aren't too bad (altho I still have one problem to flatten around the gpgcheck fields that are part of the reverts...) G You could also pin to pulpcore < 3.12 on the 3.11 branch to get the branch > passing while you work on fixing the enqueue problem on master. > > David > > > On Tue, May 11, 2021 at 11:49 AM Grant Gainey wrote: > >> Hey folks, >> >> We've been talking about how we need a pulpcore/3.7-to-3.11-compatible >> release of pulp_rpm. The static_context change requires a schema-change, >> and it has to be available to katello-3.18 (and hence pulpcore-3.7) >> >> The static_context change is PR#1984 >> <https://github.com/pulp/pulp_rpm/pull/1984> >> >> Right now, pulp-rpm/master has changes that require pulpcore/3.12 or >> later. Those changes are: >> >> d0c9badd Refactor distribution migration 0032 >> fbaadaca Add support for automatic publishing and distributing >> >> In addition, pulp_rpm/master is *currently broken* because it still >> references the deprecated enqueue_with_reservation(), that just got removed >> from pulpcore/master. >> >> As I understand it, what needs to happen #SOON, is the following: >> >>1. revert the two commits above and merge, >>2. get the static_context change updated (since a migration will have >>Left the Building) and get it merged, >>3. merge any other fixes that won't break 3.7-compat [OPTIONAL], and >>THEN >>4. cut 3.11 as compatible with 3.7-thru-3.10 pulpcore >> >> Once pulp_rpm/3.11 is released, we can then: >> >>1. re-apply the auto-pub/dist-schema changes, >>2. fix enqueue-problem, >>3. mark pulp_rpm/master as 3.12+ compat, and finally >>4. release pulp_rpm/3.12 to be ready for pulpcore-3.13 >> >> And this all needs to happen by next week? >> >> Is there anything I'm missing here? >> >> G >> -- >> Grant Gainey >> Principal Software Engineer, Red Hat System Management Engineering >> ___ >> Pulp-dev mailing list >> Pulp-dev@redhat.com >> https://listman.redhat.com/mailman/listinfo/pulp-dev >> > -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] pulp_rpm and current backwards-compatibility problems
Hey folks, We've been talking about how we need a pulpcore/3.7-to-3.11-compatible release of pulp_rpm. The static_context change requires a schema-change, and it has to be available to katello-3.18 (and hence pulpcore-3.7) The static_context change is PR#1984 <https://github.com/pulp/pulp_rpm/pull/1984> Right now, pulp-rpm/master has changes that require pulpcore/3.12 or later. Those changes are: d0c9badd Refactor distribution migration 0032 fbaadaca Add support for automatic publishing and distributing In addition, pulp_rpm/master is *currently broken* because it still references the deprecated enqueue_with_reservation(), that just got removed from pulpcore/master. As I understand it, what needs to happen #SOON, is the following: 1. revert the two commits above and merge, 2. get the static_context change updated (since a migration will have Left the Building) and get it merged, 3. merge any other fixes that won't break 3.7-compat [OPTIONAL], and THEN 4. cut 3.11 as compatible with 3.7-thru-3.10 pulpcore Once pulp_rpm/3.11 is released, we can then: 1. re-apply the auto-pub/dist-schema changes, 2. fix enqueue-problem, 3. mark pulp_rpm/master as 3.12+ compat, and finally 4. release pulp_rpm/3.12 to be ready for pulpcore-3.13 And this all needs to happen by next week? Is there anything I'm missing here? G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Pulp on Openshift
Hey Bob! pulp-operator is under active development here : https://github.com/pulp/pulp-operator . I don't think there's an OpenShift "how to" (yet), but they've been making great progress! G On Fri, May 7, 2021 at 10:09 AM Bob Fahr wrote: > Is there a formal or informal doc on installing and setting up a new pulp > instance on Openshift? > > -- > > Bob Fahr > > Principal Software Engineer, CEE Insights > > Red Hat <https://www.redhat.com/> > > bf...@redhat.comT: 256-217-0125 > M: 501-733-2516 > <https://red.ht/sig> > ___ > Pulp-dev mailing list > Pulp-dev@redhat.com > https://listman.redhat.com/mailman/listinfo/pulp-dev > -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Status
May 5, 2021 Overview - Katello Schedule - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.11.1 released - https://docs.pulpproject.org/pulpcore/en/3.11.1/changes.html#id1 - 3.7.6 released with fixes for HTB - https://docs.pulpproject.org/pulpcore/en/3.7.6/changes.html#id1 - 3.13 release upcoming - Tentatively scheduled for May 12 - dalley to release - Django 2.2.21 CVE update broke pulpcore - working on a fix - https://pulp.plan.io/issues/8691 - No migration required - Task debug/cleanup notes (WIP) - https://hackmd.io/@pulp/reserved_resource_debugging - Bmbouter will take over reporting to this mtg - RPM - Appstream pulp3->pulp2 issue - https://bugzilla.redhat.com/show_bug.cgi?id=1954839 - Prob w/ comps.xml with a group w/ empty-package-list - Metadata mirroring support in-progress - https://pulp.plan.io/issues/6353#note-26 - More feedback would be great! - RHEL 9 content support in-progress soon - https://pulp.plan.io/issues/8638 - Migration - 0.11.1 is out with fixes for HTB - Appstream migration issues to be filed - Ansible - import/export and Deprecated model (v3) - https://pulp.plan.io/issues/8204 - What version of katello must-have this deprecation model supported for ansible-import-export? - Not immediately critical - Check on this at pulp_ansible call - 0.5.9 released - 0.7.3 released - Pulp Container - 2.1.2 is out with fixes for HTB - Meeting with QE to give a braindump on pulp_docker/pulp_container and what to test - 2.5.3 waiting on django-resolution before releasing w/ cnx-backport - Pulp CLI - 0.8.0 released Katello - Adapting to new proxy attributes post-3.11 - Removing pulp2 applicability code - Discussing alternate content sources, ostree, metadata mirroring this week for estimates - Katello-4.1 is branching (has branched?) this week! QE - Waiting for next snap to do ON_QA - Manual testing for these in progress - Test-automation happening as well -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp-Community Integration meeting minutes
Our second open integration meeting. Good questions, good discussion! We've decided to hold these monthly, adjusting as we go. Next meeting is 1000 EDT 01-JUN - anyone who's interested in attending, let me know at ggai...@redhat.com and I'll add you to the invite! Happy reading, G https://hackmd.io/@pulp/pulp-deb-katello-integration ## 2021-05-04 1000-1031 ### Attendees: quba42, mbucher, ggainey, jsherrill ### Regrets: ### Agenda: * Taking stock: pulp_deb updates are now planned for Katello 4.0.1 and 3.18.4 (complete for devel) * Remaining open [packaging PRs]( https://github.com/theforeman/foreman-packaging/pull/6587) are red. * re-running job should fix * [rubygem-katello.spec changes]( https://github.com/theforeman/foreman-packaging/blob/3a0e355a86196383225b9f3d08d0852c7c501dd9/packages/katello/rubygem-katello/rubygem-katello.spec#L59) are still missing. * will happen "automatically" for release (but not nightly) * When will the next pulpcore version branch be created for pulpcore-packaging? * please invite/notify quba42 next time we do this so he can learn How It All Works * can we add pulp-dev@ , foreman-dev@ to the email to build-team requesting next branch? * AI: jsherrill to add to wiki, heads-up to Evgeni * Is Katello 3.18 the last version on which 2to3 migrations are possible? * correct - by the time you're on katello-4.0, you're done with 2to3 * What about pulp_deb "cassette test" coverage? * AI: jsherril to sched 30 min "How to do VCR tests" pair-programming exercise * Are there plans for Katello to adopt the new (pulpcore 3.13) Pulp "autopublish" feature? * katello will prob not take advantage of this * Can version level dependencies from Katello to Pulp components be made explicit? * Goal: A new package in a pulpcore-packaging repo should never break existing releases. * Consider a multi node scenario where Pulp is on a different node from Katello. * This is a hard problem, and we need to think about how we can solve it * Multiple stakeholders on multiple schedules with multiple needs makes everything Fun! ### Action Items: * AI: jsherrill to update release-wiki with a note to notify pulp-dev@ when starting a release-branch * AI: jsherrill to schedule a 30-min "How to VCR" exercise w/ quba42, mbucher * AI: ggainey to schedule next meeting for beginning of June [done, 01-JUN] * AI: ggainey to send minutes to pulp-dev@ -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Status
April 28, 2021 Overview - Katello Schedule - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - sha256/remote-artifacts problem has a fix available - https://pulp.plan.io/issues/8625 - PR opened for tasking-race-condition reported by katello - https://pulp.plan.io/issues/8603 - Backport to 3.11? - Katello only seeing it running on el8/3.11 - SoS plugin for pulp3 needs review - https://github.com/sosreport/sos/pull/2512 - Backports to 3.11, 3.7, 3.12 - RPM - RHEL 9 module changes (“static_context”) to be worked on soon - next 2 weeks hopefully - https://pulp.plan.io/issues/8638 - Would be good if this could get into downstream - 3.18 compatible - Some ideas about metadata mirroring user experience being tossed around - https://pulp.plan.io/issues/6353#note-26 - Migration - Non-sha256 repo migration issues is fixed, see pulpcore section - Kickstart migration issue https://pulp.plan.io/issues/8566 - Likely centos8 specific - Migration plugin maintenance/support for the version compatible with pulpcore 3.7 - 0.12 - early next week - Ansible - Roadmap mtg last week - Token-work starting #soon - Pulp Container - Pulp 3 to Pulp 2 sync issues (smart proxy use case): - https://bugzilla.redhat.com/show_bug.cgi?id=1953982 - https://bugzilla.redhat.com/show_bug.cgi?id=1953984 - There’s a related katello issue - Container release incoming as well - Pulp CLI - 0.8.0 to be released #soon Katello - Adding checksums configuration to settings.py - merged, will support md5 - Turning on ansible-collection-support for 4.1 - merged - Auth’d podman pull support on container gateway - merged - Pipeline for live vcr testing with latest bindings - merged - Now runs nightly - someone needs to pay attention to results - Deb migration support merged, will appear in 3.18.3 - Brad hit this testing - bz incoming: https://community.theforeman.org/t/bug-smart-proxies-do-not-sync-katello-3-15-through-3-18-rc2/21654/3 - Need to get a reading on priority/urgency - QE to check on state of capsule-sync-scenarios tested QE - downstream migration testing done - BZ found for non-sha-256: https://bugzilla.redhat.com/show_bug.cgi?id=1953106 - Finishing up dogfood testing - bz verification when snap hits today -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Tracking issues for plugin_template
On Thu, Apr 22, 2021 at 6:48 AM Ina Panova wrote: > > On Wed, Apr 21, 2021 at 7:55 PM David Davis wrote: > >> Wanted to bump this to hopefully get some feedback. Also, today during >> our CI/CD meeting we discussed also tracking issues for pulp-oci-images on >> github as well. >> >> If there are no objections by next week April 26, I'll assume I can >> proceed with moving these projects' issues over to Github. >> >> David >> >> > I have no objections regarding moving plugin_template and pulp-oci-images > to github. > > The question I have is - what is our long term goal? Do we aim to > eventually move all of the projects from pulp.plan.io? I know there were > discussions in the past but we have not found a solution on how to connect > issues with downstream trackers. The concern I have, if we don't have such > plan, then we might have plugins' issue tracking scattered between plan.io > and github where users, as a result, will file issues in the redmine and > then we will need to whether ask them to open a github issue or do it > ourselves. > We just had a python issue filed in the redmine last week. > This is a good point. I just looked at bugzilla, and 'Github' is listed as one of the Systems you can "Add Link" to. We'd (obviously) need to modify the linking/state-change-scripts. The complicating issues I can see there, are a) not having moved *all* the projects to github, so some BZs would want links to redmine and some to github, and b) being able to get "all" the issues - in redmine, I can go to pulp.plan.io/issues and see all the issues for every project, not sure we can get something equivalent from github. G > >> On Fri, Apr 16, 2021 at 8:38 AM David Davis >> wrote: >> >>> I've always felt that tracking plugin_template issues under the main >>> pulp project in plan.io was suboptimal and with other repos such as >>> pulp-cli moving to github issues, I feel that it might make sense for the >>> plugin_template to move to github issues as well. >>> >>> There's only 11 open issues right now in pulp.plan.io? for the >>> plugin_template so I think it would be an easy move. I'd propose we also >>> remove the plugin_template tag from the list of options (but keep it on old >>> issues for history). >>> >>> Thoughts? >>> >>> David >>> >> _______ >> Pulp-dev mailing list >> Pulp-dev@redhat.com >> https://listman.redhat.com/mailman/listinfo/pulp-dev >> > ___ > Pulp-dev mailing list > Pulp-dev@redhat.com > https://listman.redhat.com/mailman/listinfo/pulp-dev > -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration meeting
April 21, 2021 Overview - Katello Schedule - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.12.1 - Plugin-required API needed to be added - Should Katello use auto-publish/distribute feature? - https://bugzilla.redhat.com/show_bug.cgi?id=1844634 - RPM - Modular static_context in Pulp2? - Would need work to start *now* - Migration - Docker migration fix, in review - https://github.com/pulp/pulp-2to3-migration/pull/352 - Needs feedback from pulp_deb (ATIX)? - Deb-migration needs work to fix reported issue - Working in the background on the zchunk issue https://pulp.plan.io/issues/8400 - Prob useful to get onto a z-stream release for katello - Need to look into https://pulp.plan.io/issues/8566 - Ansible - No updates - Pulp Container - 2.5.2 is out - Apache webserver snippet fixed - Prob doesn’t impact katello - Pulp CLI - No updates Katello - Dogfood testing - Duplicate file issue filed https://pulp.plan.io/issues/8582 - Workaround: remove dup-files? - Adding checksums configuration to settings.py - PR open - Turning on ansible-collection-support for 4.1 - Auth’d podman pull support on container gateway (close to merge) - Pipeline for live vcr testing with latest bindings #soon - Evgeni Hit this on container 2.2.1: https://pulp.plan.io/issues/8350, will ask him to file an issue - Is this a 4.0 blocker? - AI: Jsherrill to check on related-katello-fix - is it in the snap? - Yes (alas) QE - Continue to test latest downstream z-stream - all green so far! - Waiting on dogfood upgrade to run migration - Customer DB - Migration completed - Need help with change hostname on satlab side to run switchover -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Thoughts needed on Yet Another Advisory-Merge Oddness
On Wed, Apr 21, 2021 at 8:03 AM Tanya Tereshchenko wrote: > Hey Grant, > [brilliant investigation happens] And this is exactly what I needed - thanks Tanya, that clears up All The Things for me. I'm going to amend the PR to use Pulp2's naming convention, and add some commentary in the collection-merge code so the next person won't have to go through this exercise. This was a great help, thanks so much! G > > Let's start with thoughts without modularity :) > I think the main reasons for the existing behavior are semantical and also > aim to preserve collections as they are. > The situation you describe (when I change only pkglist and nothing else at > all) usually means one of the following: > 1. A user wants to combine the same advisory which came from different > repos, e.g RHEL base and RHEL debuginfo. > 2. A user tries to fix an advisory in a bad way. This should never > rarely happen. If they change advisory, they need to update the `updated` > timestamp and or version, and no merge will be done. > > For situation #1, if it happens that collection names are the same (which > is often not the case in the wild), we are not modifying the existing one, > and you can clearly see that it was combined, it also preserves grouping of > packages in it which might be relevant, e.g. debuginfo vs others. > For situation #2, I agree it looks a bit weird but they should not do it > in the first place. > > Having said that, there is no problem functionality wise, one or two > collections will work fine if their names are different. > I personally would leave it as is. It gives me more confidence that we are > not modifying a collection package list but just adding a new one and > feeding it to createrepo_c as is. Plus if there is a problem, it's easier > to see that it was a merge. > > Now the modularity part :) > This might require some experimenting with dnf or just asking them how the > situation with multiple collections for the same modules is handled. > Here is how modular collection looks: > > Collection > - name_0 > - module NSVCA_0 > - list of packages > > If we look at 2 modular collections with the same name but different > modules, those can not be merged. > Current behaviour covers that case. If we choose to put all packages into > one collection at the merge time, we need to check the modularity aspect of > the collection. > > If we look at 2 modular collections with the same name and same modules, > we technically can merge them. > And here is the unknown part for me, if we leave 2 collections with the > same modules and make names different, will dnf handle it correctly, or > does it expect only one collection per module. > I'd think that dnf doesn't care for module uniqueness if the collection > names are different but it's worth checking. > > Thanks for sorting it all out, > Tanya > > P.S. +1 to give a better name to the unnamed collections, fwiw, in pulp 2 > it was something like "collection_0", collection_1", etc. > > > On Tue, Apr 20, 2021 at 3:03 PM Grant Gainey wrote: > >> Hey gang, >> >> In the process of fixing issue 7282 <https://pulp.plan.io/issues/7282>, >> I have encountered a behavior from advisory-merge that appears to be the >> deliberately-desired behavior - and I'm not sure it's really what we want >> to do? I need some thoughts on this. Figured I'd start in email, because >> it's advisory-merge, and complicated, and trying to follow a verbal >> description is confusing. More confusing. >> >> The PR with the proposed fix is >> https://github.com/pulp/pulp_rpm/pull/1973 >> >> The scenario is as follows: >> >> 1) Upload an advisory that has a package-list with one RPM, say >> 'bear', to a repository. >> 2) Upload an *identical* advisory, except for having a different package >> list with one RPM, say 'camel', to the same repository. >> 3) The resulting advisory-conflict-resolution falls into the "date and >> version identical, pkg-list disjoint" case, which calls for an advisory >> merge here: >> >> https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/advisory.py#L222 >> >> 4) advisory_merge() finds two 'collections' - one for each >> pkglist.packages in each advisory, neither of which is named. >> 5) advisory_merge() results in an advisory with two collections. One has >> no name, and is a package-list containing 'bear'. The other is named >> "None_0"[0], and is a package-list containing "camel". >> >> You can see where this decision is made here: >> >> https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/advisory.py#L273-L
[Pulp-dev] Thoughts needed on Yet Another Advisory-Merge Oddness
Hey gang, In the process of fixing issue 7282 <https://pulp.plan.io/issues/7282>, I have encountered a behavior from advisory-merge that appears to be the deliberately-desired behavior - and I'm not sure it's really what we want to do? I need some thoughts on this. Figured I'd start in email, because it's advisory-merge, and complicated, and trying to follow a verbal description is confusing. More confusing. The PR with the proposed fix is https://github.com/pulp/pulp_rpm/pull/1973 The scenario is as follows: 1) Upload an advisory that has a package-list with one RPM, say 'bear', to a repository. 2) Upload an *identical* advisory, except for having a different package list with one RPM, say 'camel', to the same repository. 3) The resulting advisory-conflict-resolution falls into the "date and version identical, pkg-list disjoint" case, which calls for an advisory merge here: https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/advisory.py#L222 4) advisory_merge() finds two 'collections' - one for each pkglist.packages in each advisory, neither of which is named. 5) advisory_merge() results in an advisory with two collections. One has no name, and is a package-list containing 'bear'. The other is named "None_0"[0], and is a package-list containing "camel". You can see where this decision is made here: https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/advisory.py#L273-L284 That result was surprising to me. My assumption going in was, if we were merging two advisories, we would merge their same-named collections into one collection with elements from both. But we're very carefully choosing to not-do that. So, my question here is, can someone explain to me the rationale for the current behavior? I'm sure it has something to do with modularity, because half of the weird things I run into in RPM-land are because of modularity :) But I'd like to understand why we're doing what we're doing, so I can add some commentary in the test or in the merge_advisories() code itself. If this turns out to be undesirable emergent behavior, then I can address it as part of this PR. The test I added for all this fun is here: https://github.com/pulp/pulp_rpm/pull/1973/files#diff-781abba15eb7e61b9ae644ae112bf1292dfd7eb78e7162b74830d4ddb43e1868R274 Thoughts very much appreciated! G [0] "None_0" is Rude. Even if this is the right behavior, we prob could stand to notice 'unnamed' collections and do something a little less painful... -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration meeting
April 14, 2021 Overview - Katello Schedule - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.7.5 release - https://docs.pulpproject.org/pulpcore/en/3.7.5/changes.html#id1 - Pyyaml CVE update as well - 3.12 release - https://docs.pulpproject.org/pulpcore/changes.html#id1 - Deprecation warnings are now logged at WARNING level - https://pulp.plan.io/issues/8499 - Can be disabled by adjusting log level for `pulpcore.deprecation` (see last paragraph <https://docs.pulpproject.org/pulpcore/plugins/plugin-writer/concepts/index.html#plugin-api-stability-and-deprecation-policy> ) - Created pulp_hashlib patch for pulpcore 3.7 - https://hackmd.io/@pulp/Pulp3-FIPS-matrix?both#What-about-37 - ‘dynaconf/env issue” resolved in katello-installer ‘for now’ - Should investigate dynaconf/pulp-installer - RPM - Memory consumption fix (for huge repos) - https://pulp.plan.io/issues/8467 - Backport to 3.10? - 3.11 release Soon™ - Oracle ULN repo support, auto-publish support. pulpcore 3.12+ only - New Bodhi released - will fix (EPEL ‘always shows as updated’ problem) - When it’s pushed to production - not sure when that happens - “Some time after the Fedora 34 release” is what I’m being told - Is this happening on RHEL repos? - “Is RHEL using Bodhi” is a good question to know the answer to regardless - Migration - 0.11.0 release - https://docs.pulpproject.org/pulp_2to3_migration/changes.html#id - FIPS and 3.7 investigation completed - https://hackmd.io/@pulp/Pulp3-FIPS-matrix?view#What-about-37 - Migration of corrupted docker content - PR incoming “soon” - work w/ ipanova if emergency need - Ansible - No update - Pulp Container - 2.5.0, 2.5.1 release - https://docs.pulpproject.org/pulp_container/en/2.5.1/changes.html - Pulp CLI - Added auto-update support Katello - Dogfood testing - Switchover complete, test file issue? - 3.11 upgrade complete in master - Pulp-rpm 3.10 upgraded in 3.18, 4.0, master - Auth’d podman pull support on container gateway - Pipeline for live vcr testing with latest bindings #soon - Slew of downstream fixes QE - Finished migration testing; cheering ensues! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Community Integration meeting
Hey folks - quba42 raised some questions around backports and katello integration and pulpcore and plugins (oh my!), and we had a quick conference call to discuss. Below please find minutes therefrom. This may be something that's useful going forward, and perhaps to more than just pulp_deb. I've scheduled another for 4-MAY - if anyone is interested, let me know and I will add you to the invite! Happy reading, G ## 2021-04-13 1000-1030 ### Attendees: ggainey, quba42, jsherrill ### Regrets: mbucher ### Agenda: * Katello's dependency-requirements thought process * What version of pulpcore goes/will go with which version of katello? * Do we need to do this meeting regularly? ### Minutes: * quba42: * "What's katello's POV RE integrating w/pulpcore? When/how does katello decide to update pulpcore?" * katello-3.18/pulpcore-3.7 currently - can/would we ever move up from 3.7? * jsherrill: upstream katello integration-points stay close to product-levels * lots of QE work on integration already happened * stability highly desirable * quba42: * what if you have to backport something that needs a migration? * lots of sad engineering noises followed * katello-master: currently on 3.9 * just updated to 3.11 yesterday * what's katello upgrade/update strategy? * jsherrill: * packaging is the biggest time sync for updating reqs * packaging process discussion * AI: jsherrill to send process-doc-link to quba42 * quba42: * who should quba42 work with, to get pulp-deb packaging upgraded? Evgeni? * AI: jsherrill to walk quba42 thru process? - No * discussion about need for future/regular meetings * do we even need to? * maybe once a month? * AI: have one a month from now, re-evaluate at that time ### Action Items: * AI: jsherrill to send process-doc-link to quba42 (done) * AI: ggainey to sched 15-min mtg for May (done) * AI: ggainey to send minutes to pulp-dev@ (done) G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration meeting
April 7, 2021 Overview - Katello Schedule - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.12 tomorrow - Includes support for auto-publish/auto-distribute - Maybe Katello can use this feature? - Upgrade to django 3.2 - https://pulp.plan.io/issues/8488 - Katello 4.2 or 4.3 timeline - django-2.2-LTS updated this week - 3.7.5 planned for this week to include backports required for next downstream z-release - Discussion around “does it make sense to try and get pulpcore-3.8+ into downstream Y-release”? - FIPS - 3.11 has a lot of fixes, and MD5 patch only works against 3.11 - For next-downstream-Y, just 2to3-migration is needed - RPM - 3.11 release soon with auto-publish - Pulp3-depsolve-bug - under investigation - https://bugzilla.redhat.com/show_bug.cgi?id=1934545 - Slated for next downstream z-release - Migration - Discussion about 2to3, pulpcore-3.12, and plugin compatibility - ggainey investigating issue https://pulp.plan.io/issues/8453 - Dogfood fun! - Plan to work on the batch config option this week https://pulp.plan.io/issues/8470 - Ansible - No updates - Pulp Container - ‘Podman search’ - Katello provides this functionality - Pulp CLI - No updates Katello - Dogfood testing - Importing errata hrefs now, going to cleanup orphans and check missing/corrupted content - work on 3.11 upgrade awaiting review - Making download timeouts configurable - Auth’d podman pull support on container gateway - Pipeline for live vcr testing with latest bindings QE - Finishing up current migration testing matrix - next downstream-z-release snap 1 is out - Only 3 ON_QAs - 7 New, 1 Assigned. Expect those in snap 2 or later? - Helping James test 409 error issue. - Will start rerunning all migration test cases -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration meeting
March 31, 2021 Overview - Katello Schedule - 3.18 November 2020 - pulpcore 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - pulpcore 3.9 - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.12 still on track for 4/6 (or possibly 4/8) - go/no-go meeting tomorrow - Migration fixes - importer/exporter fixes - Katello to look at using 3.12 for 4.1 - Evgeni applied/tested FIPS downstream 3.11 and Things Worked - Jsherrill to coordinate releases that need it - RPM - 3.10 released w/ errata fixes - Mirroring-repodata discussion - can it be used in 4.1? - Ttereshc to investigate whether pulpcore-changes needed - Aiming for pulp_rpm mid-April might work - Pulp3-depsolve-bug - under investigation - Migration - No declared artifact error https://pulp.plan.io/issues/8377 - Issue in the pulpcore, should be solved - Might be causing pulp_container some pain :( - Worked on user reported issues - ggainey testing migration and FIPS - There is an existing issue raised https://pulp.plan.io/issues/8453 - Ansible - Token-auth work postponed - Pulp Container - Docker-push issues being investigated/resolved - Pulp CLI - Shell-mode added - 0.8.0 release ‘soon’ - PulpCon 2021 - Week of November 8-12 Katello - Dogfood testing - Reset, and rerun complete, tanya is going to look into missing rpms - Issue filed to make mongo-batch-size configurable - Maybe do a file-repo migration to reproduce rel-path issues that blocked us? - work on 3.11 upgrade looking good - Ansible collection enhancements ongoing - Auth’d podman pull support on container gateway - Pipeline for live vcr testing with latest bindings QE - Retesting all test scenarios for final snap - Continuing to verify ON_QAs - Reset - https://bugzilla.redhat.com/show_bug.cgi?id=1916759. Seems to work on small scale, but larger db such as customer seems to have issues. Created this bz: https://bugzilla.redhat.com/show_bug.cgi?id=1944375 - Waiting on dogfood once devs are done with their testing - Waiting on fix on customer db from dev so accurate testing on customer db can begin - “Fix” is “get actual RPMs on disk for imported customer DB” - Do we have enough docker testing? - There are docker test-cases, so codepaths are being tested - Running a migration after ‘switchover’ causes some grief - QE, please file katello BZ to handle this more gracefully G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] [Pulp-list] PulpCon 2021
On Wed, Mar 31, 2021 at 8:00 AM David Davis wrote: > After talking to some people this morning, it sounds like November 8-12 > would also work. I think people would rather meet in the fall when the > weather is more conducive to being indoors. > > Any objections/thoughts on us having PulpCon the week of Nov 8-12? > works4me! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration meeting
Overview - Katello Schedule - 3.18 November 2020 - pulpcore 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - pulpcore 3.9 - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.12 slated for 6-APR - Auto-distribution - Fixes for imp/exp opened by katello - RPM - No release yet, will (probably) happen this week - One outstanding PR remaining - Allow-unsafe-advisory-resolution - https://github.com/pulp/pulp_rpm/pull/1950 - ttereshc/jsherrill/dalley to discuss 3.10 fixes - Migration - 0.10.0 released - Jsherrill to retest fix - Pulp2-orphaned-data will now be cleaned up in pulp2content-table - Known issues, not fixed - Zck in pulp2 https://pulp.plan.io/issues/8400 - No declared artifact error https://pulp.plan.io/issues/8377 - Unable to reproduce so far but reported by a katello user - Stephen to get ggainey a 6.9-level machine to test on - Ansible - Update about question from last week: token authentication is for katello-4.2 - Pulp Container - 2.2.1 released - 2.4.0 released - Pulp CLI - FIPS discussion from bmbouters - GHA and public runners and security make us Sad - patch will be carried downstream-only Katello - Dogfood testing - ttereshc to talk w/KatelloKrew about some findings - work on 3.11 upgrade starting - Ansible collection work ongoing (token support for syncing) - Auth’d podman pull support on container gateway - Pipeline for live vcr testing with latest bindings QE - New test scenario added + continue testing - Remigration scenarios + switchover - Remigration after fixing corrupted/missing rpm + switchover - Customer & dogfood DB - Successful migration on both - Need to verify sample data set on dogfood to ensure migration is correct - Tanya created script to better list out missing/corrupted RPM - No new BZs filed - Still testing 4 ON_QAs - Reset ruby binding - https://bugzilla.redhat.com/show_bug.cgi?id=1929392 - Best to leave this until katello reset bz is in? - ForeignKeyViolation - https://bugzilla.redhat.com/show_bug.cgi?id=1929395 - Question regarding tfm-rubygem-pulp_2to3_migration_client-0.8.0-1 because snap 18 has 0.7.0 - The important package is python3-pulp_2to3_migration which should be 0.9.0 - Snap 19 - Will all pulp bz with QA whiteboard “Pulp3” be in final snap? - 13 BZs (New, assigned, post, on_dev, modified) - https://bugzilla.redhat.com/buglist.cgi?cmdtype=runnamed_id=11768633=Pulp3 - “Today’s build” is a GA Candidate for downstream - QE/Pulp3 discussion around automation of migration-related testing G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] pulp_deb CI Jobs are frequently getting stuck
Hey Quirin, On Thu, Mar 18, 2021 at 3:43 AM Quirin Pamp wrote: > Recently it has been happening with some regularity that the main "test" > CI workflow for pulp_deb gets stuck indefinitely during the "Install" step. > See here for an example: > https://github.com/pulp/pulp_deb/runs/2136544386?check_suite_focus=true > > I just wanted to ask if others have made similar observations? > I don't think I've seen that exact error - but I think we've all been having to restart CI jobs when GHA failed for transient problems. I think we're in a better place having moved to GHA than we were with Travis - but there's definitely been a lot of instability the last week or so there. I don't have any solutions, alas - only sympathy and solidarity with you for the problem :( . G > > regards, > Quirin (quba42) > ___ > Pulp-dev mailing list > Pulp-dev@redhat.com > https://listman.redhat.com/mailman/listinfo/pulp-dev > -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] pulp_rpm and The Pitfalls of urljoin()
On Thu, Mar 18, 2021 at 4:32 AM Quirin Pamp wrote:
> Hi,
>
> it looks like there is now a commit on pulp_rpm that uses something named
> "urlpath_sanitize" instead of urljoin.
> This is causing a major clash with this PR:
> https://github.com/pulp/pulp_rpm/pull/1896 which has replaced a lot of
> urljoin's with os.path.join.
>
> My problem is that since I am not the original author of this PR (I merely
> adopted it), I am not sure what the exact background of the switch to
> "os.path.join" is.
> I presume, it was necessary (and perhaps trying to solve the same issue as
> the "urlpath_sanitize").
>
> I can of course rebase the path and test it out in either version (using
> urlpath_sanitize vs os.path.join), but if you have already put some recent
> thought into the intricacies of urljoin, I would appreciate your input
> before I do.
>
Well, this raises a really good point.
Both of these PRs address the fact that the code was using 'urljoin' when
all it really wanted (in almost all cases) was to join a base-url (like,
say, "remote_url") and a relative-path fragment. Places where base-url
didn't end in '/' caused a variety of subtle problems.
os.path.join() does, in fact, address this problem - *on Linux systems*.
If you're running python on a system that uses something other than '/' for
the path-separator (like, say, Windows' '\') - then os.path.join() can do
Bad Things to your urls.
I built url_sanitize() to get what was wanted from the current uses of
urljoin(), while avoiding the pitfalls it was causing. Replacing it with
os.path.join() fixes that set of problems as well, but it feels like just
setting a different landmine to blow us up in the future, if/when we
support running Pulp3 on something non-Linux.
And in addition, the entities we're dealing with here, generally *aren't
filesystem paths*. It's misleading the future developer to imply that's
what they are.
Anyway, those were my thoughts when I went the url_sanitize path. More
and/or better ones, always welcome!
G
(My only other thoughts here are that the semantics of 'urljoin()' are WAY
too subtle, and we are not alone in writing code that is eventually
inadvertently broken by them. I honestly cannot recall ever seeing
urljoin() used in a place where those semantics were actually desired. But
that's just me being grumpy...)
> Regards,
> Quirin (quba42)
> --
> *From:* pulp-dev-boun...@redhat.com on
> behalf of Grant Gainey
> *Sent:* 15 March 2021 20:43
> *To:* Pulp-dev
> *Subject:* [Pulp-dev] pulp_rpm and The Pitfalls of urljoin()
>
> Hey folks,
>
> I was looking at https://pulp.plan.io/issues/7995 with an eye to fixing
> it for 3.12. The underlying problem can be summed up as "urljoin doesn't do
> what you expect if 'base' doesn't end in a '/'"[0][1]. Using urljoin() as
> if all it does is "concatenate these two strings with a '/' between them"
> is a pretty common misuse that works 'most of the time', alas.
>
> Looking around to see if we do this anywhere else, I find that pulp_rpm
> uses urljoin() in a number of places that I think might be subject to the
> same unintended problem. However, the semantics of sync'ing RPM
> repositories is even *more* nuanced than urljoin()'s is!
>
> I am tempted to replace urljoin() with just straightforward path-creation.
> Here's the list of places in pulp_rpm non-test python that uses it
> currently:
>
> (pulp3) (master) ~/github/Pulp3/pulp_rpm $ find . -name \*.py | grep -v
> tests | xargs grep -n "urljoin("
> ./pulp_rpm/app/kickstart/treeinfo.py:23:
> url=urljoin(remote_url, namespace),
> silence_errors_for_response_status_codes={403, 404}
> ./pulp_rpm/app/models/repository.py:355:gpgkey_path =
> urllib.parse.urljoin(
> ./pulp_rpm/app/models/repository.py:358:gpgkey_path =
> urllib.parse.urljoin(gpgkey_path, self.base_path, True)
> ./pulp_rpm/app/tasks/synchronizing.py:101:downloader =
> remote.get_downloader(url=urljoin(url, "repodata/repomd.xml"))
> ./pulp_rpm/app/tasks/synchronizing.py:138:downloader =
> remote.get_downloader(url=urljoin(remote_url, "repodata/repomd.xml"))
> ./pulp_rpm/app/tasks/synchronizing.py:243:new_url =
> urljoin(remote_url, path)
> ./pulp_rpm/app/tasks/synchronizing.py:430:
> url=urljoin(self.data.remote_url, "repodata/repomd.xml")
> ./pulp_rpm/app/tasks/synchronizing.py:460:
> url=urljoin(self.data.remote_url, self.treeinfo["filename"]),
> ./pulp_rpm/app/tasks/synchronizing.py:470:
> url=urljoin(self.data.remote_url, path),
> ./pulp_rpm/app/tasks/synchronizing.py:599:url =
> urljoin(self.data.remote_url, package.location_href)
> ./pulp_r
[Pulp-dev] Katello/Pulp3 Integration meeting
March 17, 2021 Overview - Katello Schedule - 3.18 November 2020 - pulpcore 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - pulpcore 3.9 - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.7.4 released - Dynaconf fix - 3.11 released! - 3.12 “soon” - Will *not* be breaking-change - Jsherrill to review proposed 3.12 list - https://pulp.plan.io/versions/182 - RPM - 3.10 release in the next few days, will resolve many issues with errata handling - Distribution-tree fix (backported to 3.9.1) - Migration - 0.9.0 and 0.9.1 released - Working with QE on issues - ttereshc is working on https://bugzilla.redhat.com/show_bug.cgi?id=1939090 - revalidate re-migration-post-changes-in-pulp2 paths? - Ansible - Working on sync collection dependencies - Scoping AH/Satellite token support - Mtg next week to make sure we know which katello version needs this - Pulp CLI - 0.7.0 released - Pulp-python and other fixes - https://github.com/pulp/pulp-cli/blob/develop/CHANGES.rst#070-2021-03-15 - pulp_container - Fixes backported to 3.7-compatible version (2.1) - How soon do we need 3.9-compatible backport release? (2.2) Katello - Trying to ensure all the upstream stuff makes it downstream - Ansible poc finished - Correlation id support in review - User & dogfood-clone db testing - What’s the best way to fix the tasking-issue? - ggainey/dalley/bmbouter to see if we can learn anything about the tasking-issues - Look into logs? - What if postgres failed commits? - Work ongoing on nightly bindings testing QE - Snap 17 > snap 16 - No self patching needed - Migration + pulp3 testing continue - Lai to talk to team about re-migration test cases - Dogfood migration is still at 25% since yesterday, not sure if hanging - https://dhcp-2-136.vms.sat.rdu2.redhat.com/foreman_tasks/tasks/9e0f7e77-5642-42c9-8b54-9139ff330438 - BZs - “Migrated_pulp3_href NULL value” - https://bugzilla.redhat.com/show_bug.cgi?id=1939090 - Tanya working on it - https://bugzilla.redhat.com/show_bug.cgi?id=1939558 - Couple of bz mistakenly put ON_QA - Verifying ON_QAs are still ongoing -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] pulp_rpm and The Pitfalls of urljoin()
On Mon, Mar 15, 2021 at 4:42 PM David Davis wrote: > I've been bitten by this a couple times. I also noticed that ansible > actually defines its own version of urljoin: > > > https://github.com/ansible/ansible/blob/00bd0b893d5d21de040b53032c466707bacb3b93/lib/ansible/galaxy/api.py#L166-L167 > "I have an arbitrary number of path-fragments, any of which may or may not begin and/or end with '/', that need to be joined, and end, with a single slash". Am I reading it correctly? G (who was of course Not At All thinking of writing something pretty much exactly that...) -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] pulp_rpm and The Pitfalls of urljoin()
Hey folks,
I was looking at https://pulp.plan.io/issues/7995 with an eye to fixing it
for 3.12. The underlying problem can be summed up as "urljoin doesn't do
what you expect if 'base' doesn't end in a '/'"[0][1]. Using urljoin() as
if all it does is "concatenate these two strings with a '/' between them"
is a pretty common misuse that works 'most of the time', alas.
Looking around to see if we do this anywhere else, I find that pulp_rpm
uses urljoin() in a number of places that I think might be subject to the
same unintended problem. However, the semantics of sync'ing RPM
repositories is even *more* nuanced than urljoin()'s is!
I am tempted to replace urljoin() with just straightforward path-creation.
Here's the list of places in pulp_rpm non-test python that uses it
currently:
(pulp3) (master) ~/github/Pulp3/pulp_rpm $ find . -name \*.py | grep -v
tests | xargs grep -n "urljoin("
./pulp_rpm/app/kickstart/treeinfo.py:23:url=urljoin(remote_url,
namespace), silence_errors_for_response_status_codes={403, 404}
./pulp_rpm/app/models/repository.py:355:gpgkey_path =
urllib.parse.urljoin(
./pulp_rpm/app/models/repository.py:358:gpgkey_path =
urllib.parse.urljoin(gpgkey_path, self.base_path, True)
./pulp_rpm/app/tasks/synchronizing.py:101:downloader =
remote.get_downloader(url=urljoin(url, "repodata/repomd.xml"))
./pulp_rpm/app/tasks/synchronizing.py:138:downloader =
remote.get_downloader(url=urljoin(remote_url, "repodata/repomd.xml"))
./pulp_rpm/app/tasks/synchronizing.py:243:new_url =
urljoin(remote_url, path)
./pulp_rpm/app/tasks/synchronizing.py:430:
url=urljoin(self.data.remote_url, "repodata/repomd.xml")
./pulp_rpm/app/tasks/synchronizing.py:460:
url=urljoin(self.data.remote_url, self.treeinfo["filename"]),
./pulp_rpm/app/tasks/synchronizing.py:470:
url=urljoin(self.data.remote_url, path),
./pulp_rpm/app/tasks/synchronizing.py:599:url =
urljoin(self.data.remote_url, package.location_href)
./pulp_rpm/app/tasks/synchronizing.py:722:repodata_url =
urljoin(self.data.remote_url, record.location_href)
./pulp_rpm/app/tasks/synchronizing.py:726:self.data.updateinfo_url
= urljoin(self.data.remote_url, record.location_href)
./pulp_rpm/app/tasks/synchronizing.py:731:comps_url =
urljoin(self.data.remote_url, record.location_href)
./pulp_rpm/app/tasks/synchronizing.py:735:self.data.modules_url =
urljoin(self.data.remote_url, record.location_href)
./pulp_rpm/app/tasks/synchronizing.py:743:
url=urljoin(self.data.remote_url, record.location_href),
./pulp_rpm/app/downloaders.py:84:url = urljoin(self.url,
auth_param)
(pulp3) (master) ~/github/Pulp3/pulp_rpm $
Any thoughts before I dive down this rabbithole? I'm afraid I don't even
have a pocketwatch...
G
[0] https://docs.python.org/2/library/urlparse.html#urlparse.urljoin
[1] https://stackoverflow.com/a/10893427
--
Grant Gainey
Principal Software Engineer, Red Hat System Management Engineering
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] FIPS checkin mtg minutes
Light attendance, quickly went over the status of the issues in https://pulp.plan.io/issues?query_id=171 * POST - 7988 release planning/review needed - 7986 work is still happening, finishing #soon - 7989 under review/revision, planned for 3.12 - 8323 under review/revision, planned for 3.12 * ASSIGNED - 3800 - still needs a github-runner - short-term, devs use fips-box to test manually - planning/review in progress - 8097 - shooting for end-of-week - PR fix-up/rebase, working w/mongodb * NEW - ppicka to pick up 8325 as available G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration meeting
March 10, 2021 Overview - Katello Schedule - 3.18 November 2020 - pulpcore 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - pulpcore 3.9 - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.11 go/no-go tomorrow - 3.12 still expected #soon (end of March-ish) - May introduce a breaking-chg RE auto-distribution? Discussion ongoing - RPM - No updates this week - Advisory Stuff continues - Metadata-mirroring starting #soon - Migration - 0.9.0 release yesterday, build-team notified - No known important issues - Waiting on any QE feedback or upstream users report - Ansible - 0.5.8 released on Monday (required for AH/Satellite POC) - Token-related issue fixed - Galaxy has released a compatible version - https://github.com/ansible/galaxy_ng/releases/tag/4.2.3b1 - Pulp CLI - CLI exists for migration plugin - Pulp Container - 2.1.1 released ( compatible with pulpcore 3.7) - Pulp Docker (pulp2) - 3.2.9 with one regression fix released - Satellite sync issue should be addressed https://bugzilla.redhat.com/show_bug.cgi?id=1934183 - Does there exist a canonical list of “things katello absolutely needs to have fixed ‘now’”? - Critical fixes, backports, etc Katello - Trying to ensure all the upstream stuff makes it downstream - Various fixes pulled in for next major build - Ansible poc ongoing - Updated Automation Hub box, able to sync from Satellite! - Correlation id support in review - user db testing - Work started on nightly bindings testing QE - Continuing to test migration/pulp3 - Several blockers/issues - Not on Pulp-Team to fix: - Migration ‘reset’: https://bugzilla.redhat.com/show_bug.cgi?id=1916759 (migration-renumbering problem) - Pulp 2.21.5 issue: https://bugzilla.redhat.com/show_bug.cgi?id=1937123 (foreman-maintain) - Candlepin_event: https://bugzilla.redhat.com/show_bug.cgi?id=1915705 - Missing errata: https://bugzilla.redhat.com/show_bug.cgi?id=1937072 - 3 ON_QAs - Need to run against a customer or dogfood db migration in order to verify b/c not easily reproducible - https://bugzilla.redhat.com/show_bug.cgi?id=1916361 - https://bugzilla.redhat.com/show_bug.cgi?id=1913477 - https://bugzilla.redhat.com/show_bug.cgi?id=1913470 G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration mtg
March 3, 2021 Overview - Katello Schedule - 3.18 November 2020 - pulpcore 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - pulpcore 3.9 - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.11 still slated for 9-MAR - Moving proxy username and passwords to their own fields - Migration will handle existing data - Workflow will need to be adjusted - Prevents this info from hitting the logs - 3.11 compatible plugins will likely declare as not compatible with 3.12 - Pulpcore-content performance investigation - Hoping to make more progress in the next week - RPM - Advisory-merge work continues - Pulling together doc to describe all use-cases - https://hackmd.io/ik50Rb6VTtWf8gB1YpIrUA - Migration - Dogfood-clone - With 0.8.0/2.21.5/patch for https://pulp.plan.io/issues/8245, migration worked (until publishing ran us out of disk) - Worker timeout issue https://pulp.plan.io/issues/7847 - RQ patch + worker ttl patch - Importer/distributor swap issue https://pulp.plan.io/issues/7889 - Mostly solved - More issues show up with testing - Orphan cleanup issue, started https://pulp.plan.io/issues/7887 - Pulp 2 inconsistency - Current katello - all iso repos are immediate - Ansible - Figured out root-cause of sync-issues, fixed in pulp_ansible - Fao89 to know which release it is in - https://pulp.plan.io/issues/8290 - 0.5.7 just released w/fix - 0.6.2 and 0.7.1 will get it next - Pulp CLI - 0.6.0 released - Prep for plugin-name-chg from pulpcore-3.11 - FIPS - Discuss up- vs down-stream - Upstream-pulp3 won’t claim fips-compatibility until django is FIPS-compliant - Katello still carries patched django - Django issue : https://code.djangoproject.com/ticket/28401 Katello - Trying to retest migration on walmart’s db - Remove puppet content from katello - Ansible poc ongoing - Correlation id support in review - /pulp/repos -> /pulp/content change for new clients QE - Lots of migration manual-testing - Some bugs opened, some patches applied - Pulpcore 3.7, pulp_rpm 3.9, migration 0.7.0 copy subset of packages from one tools.el7 repo to another - Copying all instead? (no modules in repo) - jsherrill/QE to work on trying to repro before opening issue - Discussion around FIPS support for downstream G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration meeting
February 24, 2021 Overview - Katello Schedule - 3.18 November 2020 - pulpcore 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - pulpcore 3.9 - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.11 delayed, now 9-MAR - Remove correlation id from tech preview? - Has Katello tested this out yet? - David to follow up with jjeffers - 3.11 breaking change in status API due to bugfix, the “component” names are changing - Does katello use the component-listing names? - yes. - Content-app/performance - nothing to report right now - RPM - No auto-publish/distribute in 3.11, moving to 3.12 - Errata merge issues highlighted by Katello are on the priority list, not started yet (other priorities higher) - Migration - 0.8.0 released - ttereshc checked bugzilla-status for related BZs - AI: jsherrill to check on brew-status w/build-team - Testing on dogfood-clone w/ 0.8.0 and pulp-2.21.5 - [in progress] Distributor swap issue https://pulp.plan.io/issues/7889 - [in progress] Worker timeout issue https://pulp.plan.io/issues/7847 - Slow-query-performance investigation ongoing - Pulp CLI - 0.5.0 released - Migration support - Worker command to view workers - Pulp Ansible - Needs to be part of this meeting regularly Katello - Pr merged for migration ‘reset’ (Ian) - Merged work on corrupted/missing content skipping (justin) - Will work on pulling migration plugin 0.8.0 into 3.18/downstream - Pulp-rpm 3.9 pulled into 4.0/nightly - Work started on pulling into 3.18/downstream - Ansible syncing/serving POC to/from ansible hub in progress (thanks for the help!) QE - Testing continue - Vlad encountered a couple of bugs and will get that written up - Customer DB migrated successfully - How to get list of corrupted/skipped data? - Needs full katello patch - AI: jsherrill will work to insure it’s in this week’s snap - Data Comparison? - Dogfod DB migration issue - Pulp latest not pulled from upstream and causing failure - When will QE expect all latest migration bits to be downstream? - 0.8.0 building #soon - Katello fixes may not make current snap G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration meeting
February 17, 2021 Overview - Katello Schedule - 3.18 November 2020 - pulpcore 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - pulpcore 3.9 - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - Import/export fixes - is 3.12 (April-ish) ok? - https://pulp.plan.io/issues/7438 - https://pulp.plan.io/issues/7904 - https://pulp.plan.io/issues/8116 - Jsherrill to discuss w/paji - FIPS (3.11) - ALLOWED_CONTENT_CHECKSUMS default will not include sha-1 and md5 - md5 patch - available for RPM builds, but will not be applied upstream - RPM - PR incoming for old sha-only repo syncing (8052 <https://pulp.plan.io/issues/8052>) - Auto-publish and distribution support is nearing merge - Will only be compatible with 3.11 as of its next release - Migration - Performance and bug-fixes based on dogfood-clone performance progress - https://pulp.plan.io/issues/7779 (not-changed performance improvements) - https://pulp.plan.io/issues/8245 (DistTree boolean prob) - https://pulp.plan.io/issues/8166 (errata pre-migration fix, merged) - Reached the last stage of migration (repo version/pub/dist creation) - Need to investigate if perf can be improved (2 min per publish) - No worker timeout after increasing it significantly (150s) - [in progress] Distributor swap issue https://pulp.plan.io/issues/7889 - [in progress] Worker timeout issue https://pulp.plan.io/issues/7847 - Working w/jsherrill to upgrade dogfood-clone to pulp2.21.5 and most-recent migration-plugin (0.7.0? Or wait to merge fixes into 0.8.0?) - CLI - Added support for migration plugin - https://github.com/pulp/pulp-cli/pull/142 - https://github.com/pulp/pulp-2to3-migration/pull/313 - Content-app performance woes - Dalley and bmbouters investigating - Gunicorn tuning is Important ; apache possibly less-so - Discussion around katello-install and pulp2/pulp3/katello selinux labels and NFS Fun Katello - Pr open migration ‘reset’ (Ian) - continued work on corrupted/missing rpms skipping (justin) (pr open) - merged work on command to remove pulp2 (james) - Work has started soon on pulling in pulp-rpm 3.9 into 3.18/4.0/downstream - Pulling in migration plugin 0.7.0 to katello 3.18 (including downstream) - Ansible syncing/serving POC to/from ansible hub upcoming - Docs updates for katello 4.0 QE - Customer DB templatized for testing (+1 for bherring + Radovan) - Satlab is working on templatizing dogfood. - Resolving some issues at the moment - Test cases + results. - Vlad found an issue with a test case: - https://bugzilla.redhat.com/show_bug.cgi?id=1925799 - https://bugzilla.redhat.com/show_bug.cgi?id=192793 <https://bugzilla.redhat.com/show_bug.cgi?id=1927930>0 -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration mtg
February 10, 2021 Overview - Katello Schedule - 3.18 November 2020 - pulpcore 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - pulpcore 3.9 - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.10 is out. <https://pulpproject.org/2021/02/04/pulpcore-3.10-is-now-generally-available/> - 3.11 <https://pulp.plan.io/versions/174> March 2 tentatively. - Lots of FIPS Fixes planned - https://pulp.plan.io/issues/8180 - Content requests are significantly slower than Pulp 2 - Investigation <https://hackmd.io/C7Sqm7akQMuvurnNiVEM-w> in progress - ehelms has a reproducible issue w/capsule-sync that may be related? - RPM - 3.9 is out - No current plan for 3.7.1 backports - Migration - 0.7.0 released - Pulp_deb migration finished, merged, will be in 0.8.0 - Any specific time Katello wants 0.8.0 to be released? - Addressed the recent migration issues reported by Katello upstream - https://pulp.plan.io/issues/8200 - https://pulp.plan.io/issues/8195 - https://pulp.plan.io/issues/8211 - Continued on dogfood clone migration issue - https://pulp.plan.io/issues/8166 - Work in progress on the speeding up re-runs - https://pulp.plan.io/issues/7779 - Dogfood clone testing showed that a re-run is slow when there are a lot of repos, 39K - Pulp CLI - Thanks for the testing and filing issues - 0.4.0 coming out this week - Includes cert authentication - Pulp Ansible - 0.7.0 release this week - Import/export support - Doesn’t transfer collection deprecation information - https://pulp.plan.io/issues/8205 - TBD: token based content protection - https://pulp.plan.io/issues/7118 - [katello feedback] Desired for katello-4.1 Katello - Pr open migration ‘reset’ (Ian) - continued work on corrupted/missing rpms skipping (justin) (pr open) - continued work on command to remove pulp2 (james) - Work will start soon on pulling in pulp-rpm 3.9 into 3.18/4.0/downstream - Ansible syncing/serving POC to/from ansible hub upcoming QE - Migration test cases assigned and teammates informed. - Discussion around dogfood db and customer db testing G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration mtg
February 3, 2021 Overview - Katello Schedule - 3.18 November 2020 - pulpcore 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - pulpcore 3.9 - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.10 release <https://pulp.plan.io/issues?c%5B%5D=project%5B%5D=tracker%5B%5D=status%5B%5D=priority%5B%5D=cf_5%5B%5D=subject%5B%5D=author%5B%5D=assigned_to%5B%5D=cf_3%5B%5D=cf_7%5B%5D=fixed_version_id%5B%5D=_by=%5Bfixed_version_id%5D=%3D_filter=1=status%2Cid%3Adesc%5B%5D==%E2%9C%93%5Bfixed_version_id%5D%5B%5D=166> still on track for 4-FEB - Includes import-check endpoint - FIPS kickoff next week. Aiming for 3.11. - Changing sensitive fields (e.g. Remote password) to write_only. - https://pulp.plan.io/issues/8202 - RPM - SUSE errata bug fixed, waiting on createrepo_c - Release this week - Migration - tasking issues fixed, CI is unblocked - 0.7.0 release today - Continue on fixes for Katello - https://pulp.plan.io/issues/8166 (ggainey) - If it’s in migration.plan.io and tagged ‘katello’, it has to be fixed in Feb Katello - Influx of migration issues from upstream (and some katello ones) - https://pulp.plan.io/issues/8200 - https://pulp.plan.io/issues/8195 - Enabling mirror mode for all content types (Samir) - starting migration ‘reset’ (Ian) - continued work on corrupted/missing rpms skipping (justin) - continued work on command to remove pulp2 (james) QE - Test cases are pretty much done. Just adding a couple more to the list - Reviewed by jsherrill - Justin Watts planning on talking to Devendra about upgrade and loading a db/snapshot onto a satellite - Technically testing can start today G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Katello/Pulp3 Integration mtg
When sent on the 27th, I managed to cut off the katello side of the report! Added below... On Wed, Jan 27, 2021 at 12:37 PM Grant Gainey wrote: > January 27, 2021 > > Overview > >- > >Katello Schedule >- > > Katello Schedule > - > > 3.18 November 2020 > - > > pulpcore 3.7 > - > > 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) > - > > pulpcore 3.9 > - > > 4.1 branching ~May 2021 > - > > pulpcore 3.10 (or newer) > - > > 4.2 branching ~August 2021 > - > > 4.3 branching ~Nov 2021 > > Pulp > >- > >Pulpcore >- > > 3.9.1 released 21-JAN > - > > 3.10 release delayed to 4-FEB > - > > Import-check API in review > - > > Demo <https://asciinema.org/a/387270> > - > > Adding rate_limiting (reqs/sec) parameter to Remotes in 3.10 > - > > https://pulp.plan.io/issues/7965 > - > > Proxy user/pass issue - heading for 3.10? > - > > https://pulp.plan.io/issues/8167 > - > >RPM >- > > 3.9 release happening this week > - > >Migration >- > > Continued on list of issues to complete before 6.9 GA - in-progress > - > > https://hackmd.io/SxXBwV1sSieBu2oQpWdahw?both#Priorities > - > > Still need to file BZs > - > > Since last week: > - > > sqlite generation is disabled > - > > distribution trees are properly skipped if needed > - > > Plan to release 0.7.0 this week after CI issues are resolved > - > > Pulp_deb PRs are ready for review > - > > Investigate the workers timing out on HDD systems > - > > What issue is tracking this? https://pulp.plan.io/issues/7847 > > > >- > >Pulp2 >- > > 2.21.5 released 25-JAN > - > > Next migration-plugin release *MUST* have 2.21.5 > - > >Ansible >- > > import/export in-review > - > > Waiting on “token auth needed” question > - > > Downstream does need to protect content > - > > Granularity question still open > - > > 0.7.0 to release, galaxy_ng is using this as their z-stream > - > > 0.8 to have “dependency syncing” feature added > - > >Pulp-cli >- > > 0.2.0 released > <https://www.redhat.com/archives/pulp-dev/2021-January/msg00045.html> > - > > pulp_ansible support, repository modify command, serverless help > screens > - > > Would love feedback from Katello/QE > - > > Jsherrill to add a card > > > Katello - New migration issue found on dogfood backup - https://pulp.plan.io/issues/8166 - Pulp 3.9 now in katello nightly - Katello 4.0 branching next week - Merged applicability w/ modularity fix ( https://projects.theforeman.org/issues/31270), will not be fixed in pulp2 - Working on commands to remove pulp2 packages/data - Filed: https://pulp.plan.io/issues/8162 strange errata updated date QE - Finishing up migration issue bz today while checking all pulp3 bits are in - https://bugzilla.redhat.com/show_bug.cgi?id=1914371 - Continuing organization and prioritization of test cases - pulpcore/filesystem changes landed Friday, testing in progress - upgrade testing in progress - Selinux when pulp2 and pulp3 exist at the same time? What might break? - Monday’s beta-blocker issue - https://bugzilla.redhat.com/show_bug.cgi?id=1918322 -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration mtg
January 27, 2021 Overview - Katello Schedule - Katello Schedule - 3.18 November 2020 - pulpcore 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - pulpcore 3.9 - 4.1 branching ~May 2021 - pulpcore 3.10 (or newer) - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.9.1 released 21-JAN - 3.10 release delayed to 4-FEB - Import-check API in review - Demo <https://asciinema.org/a/387270> - Adding rate_limiting (reqs/sec) parameter to Remotes in 3.10 - https://pulp.plan.io/issues/7965 - Proxy user/pass issue - heading for 3.10? - https://pulp.plan.io/issues/8167 - RPM - 3.9 release happening this week - Migration - Continued on list of issues to complete before 6.9 GA - in-progress - https://hackmd.io/SxXBwV1sSieBu2oQpWdahw?both#Priorities - Still need to file BZs - Since last week: - sqlite generation is disabled - distribution trees are properly skipped if needed - Plan to release 0.7.0 this week after CI issues are resolved - Pulp_deb PRs are ready for review - Investigate the workers timing out on HDD systems - What issue is tracking this? https://pulp.plan.io/issues/7847 - Pulp2 - 2.21.5 released 25-JAN - Next migration-plugin release *MUST* have 2.21.5 - Ansible - import/export in-review - Waiting on “token auth needed” question - Downstream does need to protect content - Granularity question still open - 0.7.0 to release, galaxy_ng is using this as their z-stream - 0.8 to have “dependency syncing” feature added - Pulp-cli - 0.2.0 released <https://www.redhat.com/archives/pulp-dev/2021-January/msg00045.html> - pulp_ansible support, repository modify command, serverless help screens - Would love feedback from Katello/QE - Jsherrill to add a card G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration mtg
January 20, 2021 Overview - Katello Schedule - 3.18 branching: Nov 2nd, Targeting Pulp 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - 4.1 branching ~May 2021 - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.10 <https://pulp.plan.io/issues/8087> scheduled GA for January 26 - Working on chunked upload dir fix: https://github.com/pulp/pulpcore/pull/1078 - RPM - Nothing major to report - 2 PRs merged recently - Dep-solving in Pulp3 - All-modules-bing-copied bug - Both have backports filed - Migration - List of issues to complete before 6.9 GA - in-progress - https://hackmd.io/SxXBwV1sSieBu2oQpWdahw?both#Priorities - Pulp 2.21.5 - Release in flight, GA planned for January 25 - https://pulp.plan.io/projects/pulp/wiki/2215_Release_Schedule - List of issues : https://pulp.plan.io/issues?query_id=167 - Beta RPMs available, under test - Pulp CLI - 0.1.0 released on PyPI - https://www.redhat.com/archives/pulp-list/2021-January/msg00011.html Katello - Testing migration on dogfood - ggainey cheers wildly - Blocking issues (all known): - https://pulp.plan.io/issues/8084 - distribution migration error (some migration testing blocked on) - https://pulp.plan.io/issues/8099 - chunked upload - Major issues - https://pulp.plan.io/issues/8114 - Wrong deps are copied to the repo when using "recursive" option to copy rpms (MERGED) - Replacing smart proxy/capsule “Content” page - Fixing applicability modularity with non-modular rpms QE - Waiting on 6.9 snap 10 to retest pulp3 bits - https://bugzilla.redhat.com/show_bug.cgi?id=1914371 - Prioritize test cases and begin fully testing migration Questions - Modular RPMs may no longer have the ‘is-module’ flag - Also - same-RPM may be both ‘bare’ and ‘in a module’ “#soon” in Modularity Land? - Will this break Satellite? Do we have timelines? G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration mtg
January 13, 2020 Overview - Katello Schedule - 3.18 branching: Nov 2nd, Targeting Pulp 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - 4.1 branching ~May 2021 - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.10 scheduled for January 26 - Will include https://pulp.plan.io/issues/7549 - RPM - No update - metadata-mirroring/relative-path is prob not going to happen in 3.10 - Migration - Customer testing and katello 4.0? - QE should be ready to test migration with the most recent snap - Pulp_deb - Some of migration’s assumptions may be clashing with debian data-models - Removal of migration plugin? - Make sure there’s a bug filed? - Current issue in pulpcore is to remove plugin generally - There is foreman-cmd-work for removing pulp2 info that could be used for this - Pulp 2.21.5 - Release in flight, GA planned for January 25 - https://pulp.plan.io/projects/pulp/wiki/2215_Release_Schedule - List of issues : https://pulp.plan.io/issues?query_id=167 Katello - Smart Proxy container gateway work - User migration testing continuing - Pulling in 3.9 for katello 4.0 - Dropping pulp2 for katello 4.0 QE - Progress, waiting on an image rebuild -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp2: 2.21.5 Release incoming
Hello folks, We have a number of Pulp2 bugs that have been fixed, that we would like to get into a formal release. To that end, I'm proposing the following dates: dev-freeze: *Thu 14-JAN* beta (rpms available): *Mon 18-JAN* GA: *Mon 25-JAN* Release page is here: https://pulp.plan.io/projects/pulp/wiki/2215_Release_Schedule You can see the list of proposed issues here: https://pulp.plan.io/issues?query_id=167 I will be the release-nanny for 2.21.5. If you have questions or counter-proposals, you can reach me at *ggai...@redhat.com * or via IRC, *ggainey *in* #pulp *or* #pulp-dev *on * Freenode*. I am in Raleigh's timezone (GMT-4) Thanks! Grant -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration mtg
January 6, 2021 Overview - Katello Schedule - 3.18 branching: Nov 2nd, Targeting Pulp 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - 4.1 branching ~May 2021 - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.10 slated for end of January - Discussion of backports and LTS releases - RPM - Make sqlite metadata generation optional (default: off) merged - relative-path/metadata mirroring work happening ‘soon’ - Needed for 4.0/4.1 - Migration - Work on tests wraps up to unblock distribution migration fix - 2.21.5/pulpcore restart-on-failure ordering issue - 2.21.5 - Need a list of issues - some slated for next-katello-release but not ‘done’ yet? - Working w/build to figure out if we can still make .0 - Katello needs BZs to track pulp-work slated for a specific release - BZ per issue? Or BZ per-pulp-release? - QE would prob prefer per-issue - Prob continue the existing pulp2 process, for pulp3 Katello - Smart Proxy container gateway work - Auth discussions happening on theforeman mailing-list - Might coincide w/ansible needs - Started setting up to migration test more user databases QE - Any info on the build? Pulp 3 installation with services disabled? - Waiting on issues hit by build-team - Nightly-test issues encountered, under investigation - Jsherrill will ping ltran when there’s more info available -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration mtg
December 17, 2020 Overview - Katello Schedule - 3.18 branching: Nov 2nd, Targeting Pulp 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - 4.1 branching ~May 2021 - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Last meeting for 2020 - Pulpcore - 3.9 changelog <https://docs.pulpproject.org/pulpcore/changes.html#features> - FIPS stories finished/opened - See Epic 7960 <https://pulp.plan.io/issues/7960> - FIPS is January priority - RPM - Implementing the feature to enable/disable sqlite metadata generation - https://pulp.plan.io/issues/7852 - Migration - Work on tests continues - 2.21.5/pulpcore restart-on-failure ordering issue - 3.9 compatibility releases - pulp-certguard 1.1.0 - pulp_file 1.5.0 - pulp-container 2.2.0 - NOT compatible with 3.8 - pulp-ansible 0.5.5 - bmbouters 2.0 :) Katello - Container registry work continuing - Will look to move to pulpcore-3.9 this sprint - Planning to test migration with another (large) customer db - Download timeouts question (per remote?) - Discussion/questions around correlation-id and where/how headers can be set - Maybe custom bindings-template? - Dkliban offered to help if this is the way we want to go - See also Gerrod when he starts in Jan - See pulp-openapigenerator <https://github.com/pulp/pulp-openapi-generator> QE - No updates -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration mtg
December 09, 2020 Overview - Katello Schedule - 3.18 branching: Nov 2nd, Targeting Pulp 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - 4.1 branching ~May 2021 - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.9 released 7-DEC - https://docs.pulpproject.org/pulpcore/changes.html#id1 - Correlation id https://pulp.plan.io/issues/7792 - Progress reporting for importing https://pulp.plan.io/issues/6559 - Handle-artifact-checksums command https://pulp.plan.io/issues/7561 - Pulp_container needs a compatibility release due to a bug fix - Object-labelling design <https://hackmd.io/N4WFhHGWQtib7D1JIKzCDQ?view> under review - FIPS requirements https://hackmd.io/KzcCx9ZZR26kvQqJYp46GQ?view - Katello ok with requirements - RPM - “No sqlite” and “relative path” (metadata mirroring) will be worked in Q1 - Migration - 0.6.0 is out - Tests - Correctness of re-runs and pulp 2.21.5 - Ansible - Import/export is outstanding https://pulp.plan.io/issues/6738 - Anything else? (e.g. https://pulp.plan.io/issues/5238 ?) - Aimed at katello-4.1 Katello - Migration Progress reporting was merged - Container registry work continuing - Will look to move to pulpcore-3.9 this sprint - https://pulp.plan.io/issues/7923 no longing QE - 3.18 ques: can we run pre-migration cmds yet? - Pulp3 not in latest snap, being worked on - Jsherrill will work w/ltran if/as needed G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration mtg
December 02, 2020 Overview - Katello Schedule - 3.18 branching: Nov 2nd, Targeting Pulp 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - 4.1 branching ~May 2021 - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - Pulpcore - 3.9 scheduled for 7-DEC - go/no-go Fri - Alternate Content Source Planning - Check-in meeting with jsherrill, bbuckingham, and bmbouter - RPM - Nothing new to report - Migration - CI, tests - Katello - please prioritize any issues w/ttereshc - Can we get a new release for 3.18-katello? - prob yes - Moved to GHA from Travis - expect some bumps yet Katello - Migration progress reporting - in 3.18 - Do you need unique codes? https://pulp.plan.io/issues/7732 - Capsule Container registry work - Ostree planning/estimates - Tagging container images with pulp3 - Just FYI - katello-agent/qpid/legacy-users discussion/design ongoing (Katello-agent continuity) QE - Nothing new to report G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] pulp_rpm : 2.21.4-2 [RELEASED]
Hey folks, We've released a new rev of pulp_rpm for Pulp2, 2.21.4-2. This is to address a regression discovered in the 2.21.4 release. Thanks to Konstantin M. Khankin for finding the issue! Details can be found at 7849 <https://pulp.plan.io/issues/7849> G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration mtg
November 18, 2020 Overview - Katello Schedule - 3.18 branching: Nov 2nd, Targeting Pulp 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - 4.1 branching ~May 2021 - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - 2.21.4 - 2.21.4-2 missed a spot in pulp_rpm - 7849 <https://pulp.plan.io/issues/7849> - Pulpcore - 3.9 release - https://pulp.plan.io/versions/152 - still on track for Nov 30 - go/no-go meeting on Nov 23 - 3.10 - tentatively “early January” - RPM - FIPS work in progress, issues to be opened today for some holes found - See https://hackmd.io/d5y1IaW_QaSJ-DsosMDkjg?view - Sqlite repodata issues opened by katello, under review - https://pulp.plan.io/issues/7851 & https://pulp.plan.io/issues/7852 - Migration - Added reset/ endpoint to cleanup data to run a migration from scratch - Will open a katello issue to use this [justin] - Fixed an importer/distributor re-migration issue - Distributor serialization issue, under review - https://github.com/pulp/pulp-2to3-migration/pull/264 - Skip corrupted content, in progress - https://pulp.plan.io/issues/7538 - Pulp_deb migration for packages, about to be merged Katello - Current work - Migration timing investigation - Migration Progress reporting - 3.18 migration/upgrade commands - Capsule Container gateway work continue (authenticated registry on capsules) - Questions: - 2 reports of 502 Proxy error on content fetches, claiming artifacts in /var/lib/pulp/media/artifact/ don’t exist, verify checksum operation fixes it - On_demand and immediate? Maybe? - Both cases hit errors during katello-indexing, both on HDDs - Jsherrill to open an issue so we have a single place to record PDI if/as it comes up QE - Pulp3 Jira cards exist, starting this sprint to implement - jsherrill/ltran working on migration - Ltran testing against 3.18, getting first snaps now - Still needs tooling, but #soon G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] [Pulp-list] Pulp2 2.21.4-2 [RELEASED]
Konstantin, Once more unto the breach, dear friends, once more Save the attached patch as /usr/lib/python2.7/site-packages/repo_file.patch. Then: $ cd /usr/lib/python2.7/site-packages/ $ sudo patch -*p2* < repo_file.patch Here's the run from my test machine: === [vagrant@centos7 ~]$ python Python 2.7.5 (default, Apr 2 2020, 13:16:51) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pulp_rpm.handlers.repo_file Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.7/site-packages/pulp_rpm/handlers/repo_file.py", line 6, in from pulp.plugins.util import misc *ImportError*: No module named plugins.util >>> [vagrant@centos7 ~]$ *cd /usr/lib/python2.7/site-packages/* [vagrant@centos7 site-packages]$ *sudo patch -p2 < repo_file.patch* *patching file pulp_rpm/handlers/repo_file.py*[vagrant@centos7 site-packages]$ python Python 2.7.5 (default, Apr 2 2020, 13:16:51) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pulp_rpm.handlers.repo_file >>> [vagrant@centos7 site-packages]$ === We'll work to get a 2.21.4-3 out to fix this in the repos, but this should get you past the problem in the meantime. G On Wed, Nov 18, 2020 at 7:47 AM Grant Gainey wrote: > Hey Konstantin, > > Ugh. Give me a bit this morning and I'll get you a patch, and then I'll > talk to the Build Gang about getting a 2.21.4-3 put out. Apologies for > missing these :( > > G > > On Wed, Nov 18, 2020 at 1:49 AM Konstantin M. Khankin < > khankin.konstan...@gmail.com> wrote: > >> Looks like there is one more place where client relies on libraries >> available only in the server package: >> >> goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - >> handler "bind", import failed >> goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - >> Traceback (most recent call last): >> goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - >> File "/usr/lib/python2.7/site-packages/pulp/agent/lib/container.py", line >> 284, in __load >> goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - >> mod = self.__import_module(path) >> goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - >> File "/usr/lib/python2.7/site-packages/pulp/agent/lib/container.py", line >> 313, in __import_module >> goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - >> mod = __import__(path[0], globals(), locals(), [path[-1]]) >> goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - >> File "/usr/lib/python2.7/site-packages/pulp_rpm/handlers/bind.py", line 9, >> in >> goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - >> from pulp_rpm.handlers import repolib >> goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - >> File "/usr/lib/python2.7/site-packages/pulp_rpm/handlers/repolib.py", line >> 16, in >> goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - >> from pulp_rpm.handlers.repo_file import Repo, RepoFile, MirrorListFile, >> RepoKeyFiles, CertFiles >> goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - >> File "/usr/lib/python2.7/site-packages/pulp_rpm/handlers/repo_file.py", >> line 6, in >> goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - >> from pulp.plugins.util import misc >> goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - >> ImportError: No module named plugins.util >> >> вт, 10 нояб. 2020 г. в 17:52, Grant Gainey : >> >>> 2.21.4-2 has been released. The only change is the fix for 7803 >>> <https://pulp.plan.io/issues/7803> >>> >>> G >>> >>> On Mon, Nov 9, 2020 at 10:26 AM Grant Gainey wrote: >>> >>>> Hey folks, >>>> >>>> Just a heads-up that there will be a 2.21.4-2 turn of the build crank >>>> this week in order to address issue 7803 >>>> <https://pulp.plan.io/issues/7803> . I will send out a notification >>>> once the fixed RPMs are available. >>>> >>>> G >>>> -- >>>> Grant Gainey >>>> Principal Software Engineer, Red Hat System Management Engineering >>>> >>> >>> >>> -- >>> Grant Gainey >>> Principal Software Engineer, Red Hat System Management Engineering >>> ___ >>> Pulp-l
Re: [Pulp-dev] [Pulp-list] Pulp2 2.21.4-2 [RELEASED]
Hey Konstantin, Ugh. Give me a bit this morning and I'll get you a patch, and then I'll talk to the Build Gang about getting a 2.21.4-3 put out. Apologies for missing these :( G On Wed, Nov 18, 2020 at 1:49 AM Konstantin M. Khankin < khankin.konstan...@gmail.com> wrote: > Looks like there is one more place where client relies on libraries > available only in the server package: > > goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - > handler "bind", import failed > goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - > Traceback (most recent call last): > goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - > File "/usr/lib/python2.7/site-packages/pulp/agent/lib/container.py", line > 284, in __load > goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - > mod = self.__import_module(path) > goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - > File "/usr/lib/python2.7/site-packages/pulp/agent/lib/container.py", line > 313, in __import_module > goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - > mod = __import__(path[0], globals(), locals(), [path[-1]]) > goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - > File "/usr/lib/python2.7/site-packages/pulp_rpm/handlers/bind.py", line 9, > in > goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - > from pulp_rpm.handlers import repolib > goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - > File "/usr/lib/python2.7/site-packages/pulp_rpm/handlers/repolib.py", line > 16, in > goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - > from pulp_rpm.handlers.repo_file import Repo, RepoFile, MirrorListFile, > RepoKeyFiles, CertFiles > goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - > File "/usr/lib/python2.7/site-packages/pulp_rpm/handlers/repo_file.py", > line 6, in > goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - > from pulp.plugins.util import misc > goferd: [ERROR][worker-0] pulp.agent.lib.container:290 - > ImportError: No module named plugins.util > > вт, 10 нояб. 2020 г. в 17:52, Grant Gainey : > >> 2.21.4-2 has been released. The only change is the fix for 7803 >> <https://pulp.plan.io/issues/7803> >> >> G >> >> On Mon, Nov 9, 2020 at 10:26 AM Grant Gainey wrote: >> >>> Hey folks, >>> >>> Just a heads-up that there will be a 2.21.4-2 turn of the build crank >>> this week in order to address issue 7803 >>> <https://pulp.plan.io/issues/7803> . I will send out a notification >>> once the fixed RPMs are available. >>> >>> G >>> -- >>> Grant Gainey >>> Principal Software Engineer, Red Hat System Management Engineering >>> >> >> >> -- >> Grant Gainey >> Principal Software Engineer, Red Hat System Management Engineering >> ___ >> Pulp-list mailing list >> pulp-l...@redhat.com >> https://www.redhat.com/mailman/listinfo/pulp-list > > > > -- > Ханкин Константин > -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration mtg
November 11, 2020 Overview - Katello Schedule - 3.18 branching: Nov 2nd, Targeting Pulp 3.7 - 4.0 branching ~February 2021 (dry-run needed by end-of-Dec) - 4.1 branching ~May 2021 - 4.2 branching ~August 2021 - 4.3 branching ~Nov 2021 Pulp - 2.21.4 - Needed a respin to fix a regression, use 2.21.4-2 - Pulpcore - Correlation ID feature going into 3.9, posted and waiting for review - https://pulp.plan.io/issues/4689 - https://asciinema.org/a/371941 - Q: can you set this using the bindings? - Yes - Alternate Content Source feature plan being written up this week and sent to pulp-dev for review - Scheduling a meeting with @jsherrill and @bbuckingham for next week to review, answer questions, so they can put together their katello estimate - Needed for katello-4.3ish - RPM - Not a lot to report - ‘Mirroring metadata’ support - work will be starting #Soon - FIPS work to continue - Migration - Priorities for tasks or painful bugs - https://www.redhat.com/archives/pulp-dev/2020-November/msg2.html - Let me know if katello opinion on priorities differs - Migration from scratch, finishing up, https://pulp.plan.io/issues/7714 - Distribution serialization problem, started, https://pulp.plan.io/issues/7809 - Work has started on this on the katello side - What is the priority for fixing the uniqueness of the `code` for a progress report https://pulp.plan.io/issues/7732 Justin: Perf issues take priority, but would like for 3.18 - Pulp_deb https://github.com/pulp/pulp-2to3-migration/pull/149 - Recently updated, seem to be a work in progress - Is there a known deadline to include it in Katello? - Prob not 3.18 GA, will therefore need backporting - Assumes 3.7 compatibility - Best way to share demo videos? - Release-notes blog - Email to pulp-dev - Bring to/record in this meeting please Katello - 3.18 migration tooling - close to done - Migration testing for timings, found https://pulp.plan.io/issues/7809 - Speedup for migration ‘import’ time wrt errata QE - Waiting for an environment to break w/migrations - Working on testcases - needs mtg w/jsherrill to review - Current focus is pytest-conversion G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Pulp2 2.21.4-2 [RELEASED]
2.21.4-2 has been released. The only change is the fix for 7803 <https://pulp.plan.io/issues/7803> G On Mon, Nov 9, 2020 at 10:26 AM Grant Gainey wrote: > Hey folks, > > Just a heads-up that there will be a 2.21.4-2 turn of the build crank this > week in order to address issue 7803 <https://pulp.plan.io/issues/7803> . > I will send out a notification once the fixed RPMs are available. > > G > -- > Grant Gainey > Principal Software Engineer, Red Hat System Management Engineering > -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp2 2.21.4-2
Hey folks, Just a heads-up that there will be a 2.21.4-2 turn of the build crank this week in order to address issue 7803 <https://pulp.plan.io/issues/7803> . I will send out a notification once the fixed RPMs are available. G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration mtg
November 4, 2020 Overview - Katello Schedule - 3.18 branching: Nov 2nd, Targeting Pulp 3.7 - PRs are open, process is starting Pulp - Pulp2 - 2.21.4 GA happened - Pulpcore - 3.9 still on track for Nov 30 - https://pulp.plan.io/issues/7789 - Alternate Content Sources feature planning - Met w/ jsherrill, mminar, and bbuckingham, bmbouter to write a plan, then katello and pulp to estimate level of efforts - Headed for katello 4.0 - RPM - Various fixes, including the correct handling of changed repo metadata without a revision bump. - Pulp folks are supportive to prioritise the mirror-rpm-metadata feature for Katello 4.0. - https://pulp.plan.io/issues/6353 - We’ll try, but it’s not clear whether we have enough capacity for it. - Migration - Memory issues seem to go away with memory leak fixes in creatrepo_c - https://github.com/rpm-software-management/createrepo_c/pull/231 - https://github.com/rpm-software-management/createrepo_c/pull/233 - Not merged or released yet, but createrepo_c folks are asked to prioritise this upstream - Customer-like repo testing - 1K+ repos, ~300K rpms, 600K+ errata relations and other smaller content types - Initial run - 7 hours for pulp, 9 hours (including pulp) for the rake task with reimport_all=true - Priorities for tasks or painful bugs - https://www.redhat.com/archives/pulp-dev/2020-November/msg2.html - Let me know if katello opinion on priorities differs - Migration from scratch, in progress - https://pulp.plan.io/issues/7714 - Potentially a need for another 2.21.z release to add indices to mongo to the content collections. - ggainey cries in release-nanny - Has katello started integration with progress reports? - What is the priority for fixing the uniqueness of the `code` for a progress report https://pulp.plan.io/issues/7732 - Pulp_deb https://github.com/pulp/pulp-2to3-migration/pull/149 - Recently updated, seem to be a work in progress - Is there a known deadline to include it in Katello? - Ansible - 0.5.0 was released. Has the url format change - Katello’s 3.7 repo is being upgraded due to it being shared with Galaxy NG also - Trying to sort out how to avoid these rush upgrades in the future - We’re working on supporting more fields in requirements.yml Katello - Capsule-sync optimization - Happening at katello end - Trying to get a test-box for metrics - results #soon QE - Waiting on test-system to start premigration work G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration mtg
October 28, 2020 Overview - Katello Schedule - 3.18 branching: ~Nov 2nd-Nov 9th, Targeting Pulp 3.7 Pulp - Pulpcore - Pulp 2.21.4 - RC candidate builds in progress - Shooting to release 2.21.4GA this week - Pulpcore 3.6.5 release - Will include https://pulp.plan.io/issues/7737 - Pulpcore 3.7 and 3.8 z-stream releases - Fixes serious bug that causes artifact files to be deleted https://pulp.plan.io/issues/7676 - Can also affect pulp 2 files during migration - Repository-version-repair can fix an affected repo - 3.9 will have a global-repair feature - Pulpcore 3.9 - Scheduled for November 30 - Daviddavis as Release Nanny - RPM - Migration - Important fix went into 0.5.1, released yesterday - https://pulp.plan.io/issues/7625 - Memory-consumption-spike work continues - Ansible - 0.5.0 will release soon - URL formats for collection endpoints are changing https://pulp.plan.io/issues/7686 - Eg https://galaxy.ansible.com/api/v2/collections/ will become https://galaxy.ansible.com or https://galaxy.ansible.com/api/ - Jsherrill and daviddavis to discuss Katello - Migration progress reporting in progress - Built 0.5.1 migration plugin for 3.7 repos - satellite-specific migration tooling - Capsule sync optimizations are nearing completion QE - Discussion about satellite and pulp-migration process G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp3 Integration scrum
October 21, 2020 Overview - Katello Schedule - 3.18 branching: Nov 2nd, Targeting Pulp 3.7 - Meeting lead role rotating to ggainey - Katello: please assign all your bugs to him as well (not!) Pulp - Pulpcore - 3.8 release is out today, compatible with 3.7 plugins - 3.7.2 (migration, imp/exp backports) is going out today or tomorrow - CLI - Check out our proof of concept - https://pulpproject.org/2020/09/28/pulp-3-cli-poc-call-for-feedback/ - Pulp 2.21.4 - Hit a blocker <https://pulp.plan.io/issues/7726> - Potential fix in hand, ipanova testing - RPM - Minor fixes - Migration - Customer db migrated in one go by pulp (not katello :)) - Memory issues Katello - Current work: - Capsule sync optimization - satellite migration tooling - Migration progress reporting - 3.7 in nightly - Import/Export and incrementals - PRs submitted G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp 2.21.4 release delay
Hey folks, Testing of the 2.21.4 branch has found a problem that we need to fix before releasing. You can find details here: https://pulp.plan.io/issues/7726 I've updated the release schedule to reflect this status: https://pulp.plan.io/projects/pulp/wiki/2214_Release_Schedule We'll announce a new proposed GA date once we have a fix in hand. Thanks! Grant -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Thoughts on static-doc-building
On Tue, Oct 13, 2020 at 11:25 AM Grant Gainey wrote: > After the discussion in yesterday's pulp-team-mtg about building docs, and > in honor of docs-day, I have been looking into what it would take to make > updating our docs not require a dev-environment installation. > > Building docs includes building the REST API docs, which wants to talk to > a running pulp-server. Sphinx also relies on django, which relies on > settings, which are Unhappy if you're not on an installed pulp-instance. > When all you want to do is edit the static pages, this is all overkill. > Kudos to Mel Corr for trying to make my approach work and flushing out all the remaining assumptions around "you have pulp installed"! If anyone else wants to experiment with this, and is willing to ignore a lot of scary-looking warnings, the recipe *that actually works* can be found here: https://github.com/ggainey/pulp_startup/blob/main/docs_build/directions.txt This is working for Mel, so I'm confident it should be 'usable' for all. Note: just to reiterate, this is a gross, blecherous workaround - next steps are to come up with changes to make this kind of build a supported part of the pulp experience. G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Katello/Pulp 3 integration scrum
October 14, 2020 Pulp - Pulp 2.21.4 release scheduled for 22-OCT - Release schedule <https://pulp.plan.io/projects/pulp/wiki/2214_Release_Schedule> - Issues included <https://pulp.plan.io/issues?query_id=59> - Zhunting working on release today - pulpcore==3.8.0 to be released Oct 20th - RPM - Multi-repo copy (7625 <https://pulp.plan.io/issues/7625>) - Daniel taking pulp_rpm lead role 拾 - Migration - Performance improvements - 0.5.0 with ^ - Testing customer db - Discussion around debian progress - 3-Month planning meeting - FIPS, migration discussions - Mtg next week w/PM re priorities - Ansible - Fabricio is taking over as the lead - Planning to align the CollectionRemote with the data the `ansible-galaxy` CLI uses - https://pulp.plan.io/issues/7686 Katello - 3.7 upgrade almost complete - Jsherril to backport migration-0.5.0 to 3.6 packaging - That puts it into katello-3.17 - Testing Pulp 3 Debian support (ATIX fixes/changes) - Ansible collection capsule syncing - Apache cfg issues - 7651 <https://pulp.plan.io/issues/7651> - closed as not-pulp -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Thoughts on static-doc-building
After the discussion in yesterday's pulp-team-mtg about building docs, and in honor of docs-day, I have been looking into what it would take to make updating our docs not require a dev-environment installation. Building docs includes building the REST API docs, which wants to talk to a running pulp-server. Sphinx also relies on django, which relies on settings, which are Unhappy if you're not on an installed pulp-instance. When all you want to do is edit the static pages, this is all overkill. As a (gross, brute-force) experiment, I did the following in my environment: 1. Create a virtual-env (you don't have to do this, although it is good practice. I'm doing it here to isolate this work from the rest of my dev-workstation) 2. Clone pulpcore 3. Install doc-requirements 4. Apply the following patch to the environment: diff --git a/docs/Makefile b/docs/Makefile index dba1b115c..9fe952dab 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -2,7 +2,7 @@ # # You can set these variables from the command line. -SPHINXOPTS = -W # turn warnings into errors +SPHINXOPTS = # -W # turn warnings into errors SPHINXBUILD = sphinx-build PAPER= BUILDDIR = _build diff --git a/docs/conf.py b/docs/conf.py index af6e0e79e..91828806e 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -29,7 +29,7 @@ import pulpcore os.environ["DJANGO_SETTINGS_MODULE"] = "pulpcore.app.settings" import django -django.setup() +#django.setup() # -- General configuration - 5. Build the docs using the '*dirhtml*' target, ignoring a flurry of warnings/errors (dirhtml skips trying to get the REST api.json) 6. Start a local webserver in the directory 7. Look at the built docs at http://localhost:8010 The exact commands on my F32 box were these: mkvirtualenv docs workon docs git clone https://github.com/pulp/pulpcore.git cd pulpcore/ pip install -r doc_requirements.txt git apply - < diff --git a/docs/Makefile b/docs/Makefile > index dba1b115c..9fe952dab 100644 > --- a/docs/Makefile > +++ b/docs/Makefile > @@ -2,7 +2,7 @@ > # > > # You can set these variables from the command line. > -SPHINXOPTS = -W # turn warnings into errors > +SPHINXOPTS = # -W # turn warnings into errors > SPHINXBUILD = sphinx-build > PAPER= > BUILDDIR = _build > diff --git a/docs/conf.py b/docs/conf.py > index af6e0e79e..91828806e 100644 > --- a/docs/conf.py > +++ b/docs/conf.py > @@ -29,7 +29,7 @@ import pulpcore > os.environ["DJANGO_SETTINGS_MODULE"] = "pulpcore.app.settings" > > import django > -django.setup() > +#django.setup() > > # -- General configuration - > > PATCH cd docs make dirhtml # ignore the angry errors cd _build/dirhtml/ python -m http.server 8010 And my newly-build docs are now being served at http://localhost:8010/ How do people feel about cleaning this up and making it an 'official' way to build docs? Would make it way easier for folk who only want/need to edit the static pages to make contributions. Thoughts/brickbats gratefully accepted! :) G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp2: dev-freeze for 2.21.4 is Now
Hey folks, The 2.21.4 build for Pulp2 is today, we are in dev-freeze for the release. - Release schedule: https://pulp.plan.io/projects/pulp/wiki/2214_Release_Schedule - Issues included: https://pulp.plan.io/issues?query_id=59 G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Pulp2: 2.21.4 Release Proposal
This is the final call for comments/objections - the dev-freeze for Pulp2 2.21.4 will be *TOMORROW*, 13-OCT-2020! G On Thu, Oct 8, 2020 at 1:39 PM Grant Gainey wrote: > On Thu, Oct 8, 2020 at 1:03 PM Grant Gainey wrote: > ... > >> GA: *Tues 22-OCT* >> > > *Thursday*. The 22nd is a Thursday. > > Sigh. > > G > -- > Grant Gainey > Principal Software Engineer, Red Hat System Management Engineering > -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp2: 2.21.4 Release Proposal
Hello folks, We have a number of Pulp2 bugs that have been fixed, that we would like to get into a formal release. To that end, I'm proposing the following dates: dev-freeze: *Tues 13-OCT* beta (rpms available): *Fri 16-OCT* GA: *Tues 22-OCT* Release page is here: https://pulp.plan.io/projects/pulp/wiki/2214_Release_Schedule You can see the list of proposed issues here: https://pulp.plan.io/issues?query_id=59 I will be the release-nanny for 2.21.4. If you have questions or counter-proposals, you can reach me at *ggai...@redhat.com * or via IRC, *ggainey *in* #pulp *or* #pulp-dev *on* Freenode*. I am in Raleigh's timezone (GMT-4) Thanks! Grant -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp2 and directory permissions
Hello pulp-devs,
Recently an issue was opened due to some Pulp2-behavior causing problems
when one is migrating from Pulp2 to Pulp3. You can find the issue here:
https://pulp.plan.io/issues/7445 <https://pulp.plan.io/issues/7445>
The crux is that python's os.makedirs(), its mode= parameter, and umask,
have results that aren't what the Pulp2 code actually expects. This
unexpected behavior then causes problems for pulp2-to-pulp3 migrations, if
the Pulp2 side of things is creating/syncing new things
post-pulp3-installation.
If you look at the writeup at https://pulp.plan.io/issues/7445#note-13 , I
am proposing a change to Pulp2 to standardize all the various makedirs() to
one existing utility-routine, and make sure the end result is what Pulp2
code expects. This will have the benefits of:
* making sure all Pulp2 code does the same thing,
* insures the results are what the Pulp2 code is actually expecting, and
* won't break the migration process
The downside is changes in a number of places in Pulp2, and in its plugins
(I know pulp_rpm is affected, I assume there will be other plugins)
The/another approach to this would be to extract the permission-cleanup
work that the pulp2topulp3migration installation does, into a script. The
admin would need to run the script prior to each migration-attempt. This
would address the immediate problem ("Migration fails if I sync new stuff
in Pulp2 post-migration-installation"), and be less intrusive. But it
requires more, manual, work from the migrating-user, increasing
migration-friction.
Can anyone see a better way to address this problem? I don't like making
changes to Pulp2 at this stage in its lifecycle, especially not general
ones like this; if anybody has a brilliant Better Way to approach this,
please chime in!
G
--
Grant Gainey
Principal Software Engineer, Red Hat System Management Engineering
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Documentation issues
GMTA, was thinking about exactly this over the weekend. Even we don't/can't declare a "Doc Day" (which I think is a fine idea, but I know how schedules are right now...), being intentional about "I will assign one doc-issue to myself and get a PR submitted by COB Friday" would knock a bunch of issues out that have been waiting around for quite some time. Thanks for writing this up, Ina! G On Mon, Aug 3, 2020 at 7:30 AM Ina Panova wrote: > Folks, > I noticed that currently on the sprint we have roughly ~10 issues with > Documentation tag. Some of them have been carried over multiple sprints and > probably realistically still will be carried over. > > What do you think about dedicating a day and knock them down? > It will roughly be an issue per person. I am not suggesting to dedicate > whole 8 hours but let's say setting a goal for every team member for that > day:"Today I am going to prioritize docs issues and knock down at least one > documentation issue and submit a PR" > > Please provide feedback, and if the team feels like this is a good idea > I'll go ahead and pick a day in our calendar and mark it as "Docs day". > > Regards, > > Ina Panova > Senior Software Engineer| Pulp| Red Hat Inc. > > "Do not go where the path may lead, > go instead where there is no path and leave a trail." > ___ > Pulp-dev mailing list > Pulp-dev@redhat.com > https://www.redhat.com/mailman/listinfo/pulp-dev > -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp3 Concurrency testing
Hey folks, Looking into issue 7212 <https://pulp.plan.io/issues/7212> , over the weekend I did some ad-hoc evaluations of sync-performance at various concurrency settings. I wrote up my observations here: https://hackmd.io/@ggainey/pulp3_sync_concurrency Just thought folk might be interested. G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Name uniqueness problem in Pulp 3 REST API
On Tue, Jul 21, 2020 at 11:38 AM Dennis Kliban wrote: > On Tue, Jul 21, 2020 at 11:22 AM Brian Bouterse > wrote: > >> I'm concerned if we don't make a change, here's the user experience I'm >> worried about. >> >> 1. User A creates repo 'rhel7' >> 2. user B can't see repo 'rhel7' because of queryset scoping >> 3. user B goes to create 'rhel7' >> 4. user B is told 'rhel7' already exists >> >> Users should be able to use simple names. I don't know what the answer is >> to the import/export implementation conflict, but let's brainstorm some. >> For the benefit of our users, I don't think that implementation should >> interfere with this basic use. >> >> > I agree that this is a usability problem for our users. > > With regard to import/export, the ideal solution would use the same UUID > in both the system that's exporting and the system that's importing. Is my > understanding correct? > For PIE to work, it needs to be able to identify whether something needs to be created otr updated in the 'downstream', and therefore needs to be able to identify instances as being "the same thing". pulp_id definitely does that. However, the use-case for PIE can't rely on pulp_id, because it's not a replica of upstream. Consider the migrated-use-case - starting from pulp2, I have the exact same content in upstream and downstream, but completely different pulp_ids. In any event - PIE isn't really the major issue as far as I am concerned, it's deciding if a single pulp instance is multi-user or multi-tenant and following the implications from there. G > > > >> Side note: from early on in Pulp3, pk's not names have been the primary >> identifier. I'm unclear on how we got away from that. >> >> >> >> On Tue, Jul 21, 2020 at 9:03 AM Matthias Dellweg >> wrote: >> >>> I always understood the "lifting the uniqueness" as allowing to have >>> the same name used for different resource types. So the new >>> natrual_key (aka unique_together) would be ["name", "type"]. >>> >>> On Tue, Jul 21, 2020 at 2:55 PM David Davis >>> wrote: >>> > >>> > Agreed. >>> > >>> > David >>> > >>> > >>> > On Tue, Jul 21, 2020 at 8:42 AM Grant Gainey >>> wrote: >>> >> >>> >> On Tue, Jul 21, 2020 at 8:14 AM Dennis Kliban >>> wrote: >>> >>> >>> >>> Does anyone else have an opinion? If not, I am going to start by >>> writing a task to remove this name uniqueness constraint for repositories. >>> >> >>> >> >>> >> Import/export relies on non-pulp_id-uniqueness to identify Things. I >>> was assuming we were talking about adding pulp_type to the Repository >>> uniqueness-constraint, so that a given name/type would be unique (which >>> would require a single change to RepositoryResource) >>> >> >>> >> If we're talking about just removing the uniqueness-constraint >>> altogether, then life gets a lot harder. >>> >> >>> >> G >>> >> -- >>> >> Grant Gainey >>> >> Principal Software Engineer, Red Hat System Management Engineering >>> >> ___ >>> >> Pulp-dev mailing list >>> >> Pulp-dev@redhat.com >>> >> https://www.redhat.com/mailman/listinfo/pulp-dev >>> > >>> > ___ >>> > Pulp-dev mailing list >>> > Pulp-dev@redhat.com >>> > https://www.redhat.com/mailman/listinfo/pulp-dev >>> >>> >>> ___ >>> Pulp-dev mailing list >>> Pulp-dev@redhat.com >>> https://www.redhat.com/mailman/listinfo/pulp-dev >>> >>> ___ >> Pulp-dev mailing list >> Pulp-dev@redhat.com >> https://www.redhat.com/mailman/listinfo/pulp-dev >> > ___ > Pulp-dev mailing list > Pulp-dev@redhat.com > https://www.redhat.com/mailman/listinfo/pulp-dev > -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Name uniqueness problem in Pulp 3 REST API
On Tue, Jul 21, 2020 at 11:22 AM Brian Bouterse wrote: > I'm concerned if we don't make a change, here's the user experience I'm > worried about. > > 1. User A creates repo 'rhel7' > 2. user B can't see repo 'rhel7' because of queryset scoping > 3. user B goes to create 'rhel7' > 4. user B is told 'rhel7' already exists > > Users should be able to use simple names. I don't know what the answer is > to the import/export implementation conflict, but let's brainstorm some. > For the benefit of our users, I don't think that implementation should > interfere with this basic use. > User B creates rhel7 because they're not allowed to see User A's rhel7. User B has a role added. Now User B gets to see two rhel7s. > Side note: from early on in Pulp3, pk's not names have been the primary > identifier. I'm unclear on how we got away from that. > I'm a little confused - unique=True and unique_together=() are in a fair number of models, in pulpcore and pulp_rpm. Artifacts are identified by hash, so we don't duplicate them. Do different users get to upload the same Artifact, just because their RBAC roles are different? One thing that maybe we need to think about - is RBAC in pulp about "mutiple logins for users controlled by one entity/organization/admin/company that 'owns' the pulp instance", or does it include "multiple organizations that should never know the others' users/content exists"? The first is multi-user, the second I've seen described as multi-*tenant*, and feels like more the usecase you're concerned about. in my head, a single pulp-instance might be multi-user, but *not* multi-tenant - mayhap that's wrong. G > > > On Tue, Jul 21, 2020 at 9:03 AM Matthias Dellweg > wrote: > >> I always understood the "lifting the uniqueness" as allowing to have >> the same name used for different resource types. So the new >> natrual_key (aka unique_together) would be ["name", "type"]. >> >> On Tue, Jul 21, 2020 at 2:55 PM David Davis >> wrote: >> > >> > Agreed. >> > >> > David >> > >> > >> > On Tue, Jul 21, 2020 at 8:42 AM Grant Gainey >> wrote: >> >> >> >> On Tue, Jul 21, 2020 at 8:14 AM Dennis Kliban >> wrote: >> >>> >> >>> Does anyone else have an opinion? If not, I am going to start by >> writing a task to remove this name uniqueness constraint for repositories. >> >> >> >> >> >> Import/export relies on non-pulp_id-uniqueness to identify Things. I >> was assuming we were talking about adding pulp_type to the Repository >> uniqueness-constraint, so that a given name/type would be unique (which >> would require a single change to RepositoryResource) >> >> >> >> If we're talking about just removing the uniqueness-constraint >> altogether, then life gets a lot harder. >> >> >> >> G >> >> -- >> >> Grant Gainey >> >> Principal Software Engineer, Red Hat System Management Engineering >> >> ___ >> >> Pulp-dev mailing list >> >> Pulp-dev@redhat.com >> >> https://www.redhat.com/mailman/listinfo/pulp-dev >> > >> > ___ >> > Pulp-dev mailing list >> > Pulp-dev@redhat.com >> > https://www.redhat.com/mailman/listinfo/pulp-dev >> >> >> ___ >> Pulp-dev mailing list >> Pulp-dev@redhat.com >> https://www.redhat.com/mailman/listinfo/pulp-dev >> >> ___ > Pulp-dev mailing list > Pulp-dev@redhat.com > https://www.redhat.com/mailman/listinfo/pulp-dev > -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Name uniqueness problem in Pulp 3 REST API
On Tue, Jul 21, 2020 at 8:14 AM Dennis Kliban wrote: > Does anyone else have an opinion? If not, I am going to start by writing a > task to remove this name uniqueness constraint for repositories. > Import/export relies on non-pulp_id-uniqueness to identify Things. I was assuming we were talking about adding pulp_type to the Repository uniqueness-constraint, so that a given name/type would be unique (which would require a single change to RepositoryResource <https://github.com/pulp/pulpcore/blob/master/pulpcore/app/modelresource.py#L35> ) If we're talking about just removing the uniqueness-constraint altogether, then life gets a lot harder. G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] RPM plugin meeting notes
AI review - AI: ppicka to discuss relative-path investigation with Many Interested Parties, This meeting. - AI: daviddavis to file an issue RE “repo can have more than one distribution tree” - https://pulp.plan.io/issues/7115 Pulp 3: - Distribution tree copy bug https://pulp.plan.io/issues/7046 <https://pulp.plan.io/issues/7046#note-3> - High prio - Needs some to pick it up - Relative_path (possibly) extend by https://pulp.plan.io/issues/5847 - We need to be able to sync RHEL8, by 3.6 release - Needs discussion at mtg-end - import/export natural-keys <https://hackmd.io/@ggainey/importexport_lowlevel_naturalkeys> - doc reviewed - Discussion around three-month-roadmap Open PRs: - https://github.com/pulp/pulp_rpm/pulls - Dist-trees PR/createrepo_c hate each other, test needs to be fixed a diff way Triage: - Un-triaged bugs https://pulp.plan.io/projects/pulp_rpm/issues?query_id=30 AI: - ddavis to update 7096 <https://pulp.plan.io/issues/7096> with more explanation - Bmbouters to add comment to 7095 <https://pulp.plan.io/issues/7095> - Ppicka/dkliban to investigate 5847 <https://pulp.plan.io/issues/5847> - Talk to RHEL/Platform/katello team to figure out what’s going on? G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Name uniqueness problem in Pulp 3 REST API
On Tue, Jun 30, 2020 at 4:56 PM Daniel Alley wrote: > Why would it cause pain (to relax the restriction)? > Oh, good point - I was thinking backwards. Carry on, carry on... :) G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Pulp Import/Export and Natural Keys review
We're reviewing a low-level design proposal for the Pulp ImportExport feature next week - extra eyes are always welcome. Implementation of Import/Export and natural-keys. <https://www.google.com/url?q=https://hackmd.io/@ggainey/importexport_lowlevel_naturalkeys=D=calendar=1593982464713000=AOvVaw2dWugVCzZjjGEzbeU4ft_H> Date: 2020-07-09 Time: 1000-1050 EDT (UTC-04:00) Google Meet link: meet.google.com/myn-snzg-cru POC PRs: pulpcore <https://www.google.com/url?q=https://github.com/pulp/pulpcore/pull/745=D=calendar=1593982464713000=AOvVaw1YeMsWD7C1AlJk7ffVT_RF> pulp_file <https://www.google.com/url?q=https://github.com/pulp/pulp_file/pull/406=D=calendar=1593982464713000=AOvVaw2qqPjZJbKD_CiSdakfoSmB> pulp_rpm <https://www.google.com/url?q=https://github.com/pulp/pulp_rpm/pull/1744=D=calendar=1593982464713000=AOvVaw3NYiHhzhKKGA6ecQsEPKBX> Overview of PulpImport/Export <https://www.google.com/url?q=https://hackmd.io/HLptudH9R6S4PCmm8nRmmg?view=D=calendar=1593982464713000=AOvVaw3VyDms63zoO5WWGI7oJ5z8> Please come prepared to throw bricks! G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Name uniqueness problem in Pulp 3 REST API
On Tue, Jun 30, 2020 at 3:54 PM Dennis Kliban wrote: > The REST APIs for repositories in Pulp 3 do not allow a name to be reused > between repository types. e.g.: A File Repository with name 'test' prevents > a user from creating an RPM Repository with name 'test', even though these > are two separate APIs. > > Do you agree that the namespace of one API should not interfere with the > namespace of another API? > Oh...ow. Yes, concur - but that's going to cause some pain, no? G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Delivery Test
Delivery test for pulp-dev@ G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] PulpExport performance anecdote
A bored period this weekend led me to some performance-numbers for the
PulpExport work we're doing right now.
- Imported CentOS7 and exported it as 2GB chunks.
- Memory use for the pulp-worker never exceeded *125MB*
- Disk usage went up only by the size of the resulting set of .tar.gz
- Time for full-export was *8m50s*
Found one bug around persisting the chunked-file-checksums (note to self:
don't call a routine that generates a string with "now to the minute" in
it, when the operation takes more than a minute), and how to get Content
and ContentArtifact exported without relying on pulp_ids...
G
Raw data:
(pulp) [vagrant@pulp3-source-fedora31 import_export]$ http
:/pulp/api/v3/tasks/fcea3cc6-f02d-4b2a-a552-3e407f0bb338/
{
"child_tasks": [],
"created_resources": [
"/pulp/api/v3/exporters/core/pulp/1cfb0006-5909-433c-b931-ef8dd1146794/exports/63960fdd-d01b-4af9-8527-0a5e788ae430/"
],
"error": null,
"finished_at": "2020-05-24T21:26:50.367799Z",
"name": "pulpcore.app.tasks.export.pulp_export",
"parent_task": null,
"progress_reports": [],
"pulp_created": "2020-05-24T21:18:00.663723Z",
"pulp_href": "/pulp/api/v3/tasks/fcea3cc6-f02d-4b2a-a552-3e407f0bb338/",
"reserved_resources_record": [
"/pulp/api/v3/exporters/core/pulp/1cfb0006-5909-433c-b931-ef8dd1146794/"
],
"started_at": "2020-05-24T21:18:00.795450Z",
"state": "completed",
"task_group": null,
"worker": "/pulp/api/v3/workers/648c157b-ebe6-4816-aa93-2b88dc2c19ea/"
}
(pulp) [vagrant@pulp3-source-fedora31 import_export]$ http
:/pulp/api/v3/exporters/core/pulp/1cfb0006-5909-433c-b931-ef8dd1146794/exports/63960fdd-d01b-4af9-8527-0a5e788ae430/
{
"exported_resources": [
"/pulp/api/v3/repositories/rpm/rpm/6c6e3e91-65e8-40d9-afce-18d720bdcaea/versions/1/"
],
"output_file_info": {
"/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.":
"6883aa4af0f264d6fc4904e92bd979c696528fd6820e83dd5d062c64b1551f2c",
"/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0001":
"15ef9fd397a9b6ecd08d140f81d6d75621e3eb38716320ba98db6789f121da10",
"/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0002":
"2e46f4bef01ef07443473c97577600b5f5269aa6b0319fa5f7021f40972af584",
"/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0003":
"bd0f7eb61be2b6cb852c12f6d726cd9a5d440446041da3352ec31d717bd74628",
"/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0004":
"64446d4d3530fbb37ce074969e1bc02027aa2957f7ce4443d6ac6d916fea1790"
},
"params": {
"chunk_size": "2GB"
},
"pulp_created": "2020-05-24T21:18:00.632656Z",
"pulp_href":
"/pulp/api/v3/exporters/core/pulp/1cfb0006-5909-433c-b931-ef8dd1146794/exports/63960fdd-d01b-4af9-8527-0a5e788ae430/",
"task": "/pulp/api/v3/tasks/fcea3cc6-f02d-4b2a-a552-3e407f0bb338/"
}
(pulp) [vagrant@pulp3-source-fedora31 import_export]$ ls -ltrh /tmp/exports/
total 9.7G
...2.0G May 24 21:19
export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.
...2.0G May 24 21:20
export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0001
...2.0G May 24 21:22
export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0002
...2.0G May 24 21:23
export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0003
...1.7G May 24 21:25
export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0004
(pulp) [vagrant@pulp3-source-fedora31 import_export]$
--
Grant Gainey
Principal Software Engineer, Red Hat System Management Engineering
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Building docs locally in dev-env?
On Mon, May 11, 2020 at 4:46 PM Brian Bouterse wrote: > On Mon, May 11, 2020 at 4:40 PM Grant Gainey wrote: > >> Hey folks, >> I am making some additions to the import-export docs and went to build >> them locally so I could review formatting. Since the last time I did this a >> month ago, I'm running into a problem. docs/Makefile now assumes that " >> http://pulp/...; will resolve to something, which it certainly does not >> for my vagrant-box. >> > I expected it to use localhost:24817 as the location to fetch the openapi > schema from at docs-build time. For example I expect each plugin's makefile > to do that with a line like this one: > https://github.com/pulp/pulp_file/blob/master/docs/Makefile#L45 > I'm in pulpcore, and the line is here: https://github.com/pulp/pulpcore/blob/master/docs/Makefile#L56 and was changed as part of the single-container work, it appears (eb1173a310) Is this a problem with my env? A problem with the Makefile? Or a problem >> with the 'how to build the docs", docs, that is perhaps missing a step? >> > Are you running `make html`? What docs are you working from? > 3.4, and yeah, I'm just running 'make html' > I'm also getting two docstring errors: >> >> /home/vagrant/devel/pulpcore/pulpcore/app/models/exporter.py:docstring of >> pulpcore.app.models.PulpExport::more than one target found for >> cross-reference 'models.RepositoryVersion': >> pulpcore.app.models.RepositoryVersion, >> pulpcore.plugin.models.RepositoryVersion >> >> and >> >> /home/vagrant/devel/pulpcore/pulpcore/tasking/tasks.py:docstring of >> pulpcore.tasking.tasks.enqueue_with_reservation::more than one target found >> for cross-reference 'models.TaskGroup': pulpcore.app.models.TaskGroup, >> pulpcore.plugin.models.TaskGroup >> >> The solution is easy - explicitly specify pulpcore.app. But This didn't >> bite me last time I generated the docs, and I'd like to make sure it's not >> just a symptom of something More Wrong in my env >> > > I don't think this is an error with your environment, but probably a > legitimate error. For anything we define in pulpcore and then import into > pulpcore.plugin we probably need to have the docstring use the full path. > Coolio, I will fix both spots - no sense in opening a new issue for TaskGroup when I'm looking right at the problem in my branch. The Makefile I'm not gonna touch in my PR tho - prob wants a few more eyes G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Building docs locally in dev-env?
Hey folks, I am making some additions to the import-export docs and went to build them locally so I could review formatting. Since the last time I did this a month ago, I'm running into a problem. docs/Makefile now assumes that " http://pulp/...; will resolve to something, which it certainly does not for my vagrant-box. Is this a problem with my env? A problem with the Makefile? Or a problem with the 'how to build the docs", docs, that is perhaps missing a step? I'm also getting two docstring errors: /home/vagrant/devel/pulpcore/pulpcore/app/models/exporter.py:docstring of pulpcore.app.models.PulpExport::more than one target found for cross-reference 'models.RepositoryVersion': pulpcore.app.models.RepositoryVersion, pulpcore.plugin.models.RepositoryVersion and /home/vagrant/devel/pulpcore/pulpcore/tasking/tasks.py:docstring of pulpcore.tasking.tasks.enqueue_with_reservation::more than one target found for cross-reference 'models.TaskGroup': pulpcore.app.models.TaskGroup, pulpcore.plugin.models.TaskGroup The solution is easy - explicitly specify pulpcore.app. But This didn't bite me last time I generated the docs, and I'd like to make sure it's not just a symptom of something More Wrong in my env Thoughts, anyone? G -- Grant Gainey Principal Software Engineer, Red Hat System Management Engineering ___ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
