Re: [Pulp-dev] How to list orphan content in pulp3 before remocing them ?

[Daniel Alley]

Oh, ignore me, I didn't realize you don't need this to be via public HTTP
API.

On Thu, Apr 14, 2022 at 11:52 AM Daniel Alley  wrote:

> There is not a way to do this, figuring out which content are orphans is
> pretty expensive so I'm not certain it would be suitable for an endpoint
> unless we changed how it works.
>
> On Thu, Apr 14, 2022 at 11:30 AM Sayan Das  wrote:
>
>> Hello Grant,
>>
>> I hope you are doing great.
>>
>> Few other support folks had reached out to me to understand if we can
>> list the Orphaned data anyway in Pulp3 (Sat 6.10\RHUI 4).
>>
>> By anychance, Were you able to find out a way to do so in the version of
>> pulpcore being shipped for Sat 6.10 or RHUI 4 (3.14 i believe)?
>>
>>
>> Thanks & Regards,
>>
>> Sayan das
>>
>> *S*enior* T*echnical *S*upport *E*ngineer, RHCE
>>
>> Red Hat India
>> <https://www.redhat.com/>
>>
>> Red Hat India Pvt. Ltd, Level-5, Tower-10, Cyber City
>>
>> Magarpatta City Hadapsar, Pune-411013, Maharashtra, India.
>>
>> say...@redhat.comM: +91-7890892756 IRC: Sayan
>> <https://red.ht/sig>
>>
>>
>> On Mon, Jan 31, 2022 at 9:31 PM Grant Gainey  wrote:
>>
>>> On Mon, Jan 31, 2022 at 9:09 AM Sayan Das  wrote:
>>>
>>>> Hi Grant,
>>>>
>>>> I was trying your Django method but it fails in Satellite 6.10.2 .
>>>>
>>>
>>> Oh...drat. This is what comes of answering emails w/out enough caffeine.
>>> orphan-cleanup being asynchronous, and therefore that call, isn't available
>>> for 6.10, it's in 3.15 and beyond (and so slated for Sat7). See
>>> https://docs.pulpproject.org/pulpcore/changes.html#id140
>>>
>>> Give me a little, I'll see what we can do in the 3.14 timeframe.
>>>
>>> Sorry about that,
>>> G
>>>
>>>
>>>>
>>>> # sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager
>>>> shell << EOF
>>>> from pulpcore.app.models.content import Content
>>>> minutes_since_touched = 1
>>>> print(Content.objects.orphaned(minutes_since_touched).all())
>>>> EOF
>>>>
>>>> Traceback (most recent call last):
>>>>   File "/bin/pulpcore-manager", line 11, in 
>>>> load_entry_point('pulpcore==3.14.9', 'console_scripts',
>>>> 'pulpcore-manager')()
>>>>   File "/usr/lib/python3.6/site-packages/pulpcore/app/manage.py", line
>>>> 11, in manage
>>>> execute_from_command_line(sys.argv)
>>>>   File
>>>> "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line
>>>> 381, in execute_from_command_line
>>>> utility.execute()
>>>>   File
>>>> "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line
>>>> 375, in execute
>>>> self.fetch_command(subcommand).run_from_argv(self.argv)
>>>>   File
>>>> "/usr/lib/python3.6/site-packages/django/core/management/base.py", line
>>>> 323, in run_from_argv
>>>> self.execute(*args, **cmd_options)
>>>>   File
>>>> "/usr/lib/python3.6/site-packages/django/core/management/base.py", line
>>>> 364, in execute
>>>> output = self.handle(*args, **options)
>>>>   File
>>>> "/usr/lib/python3.6/site-packages/django/core/management/commands/shell.py",
>>>> line 92, in handle
>>>> exec(sys.stdin.read())
>>>>   File "", line 3, in 
>>>> AttributeError: 'ContentManagerFromBulkTouchQuerySet' object has no
>>>> attribute 'orphaned'
>>>>
>>>> I definitely don't see *orphaned *inside *Content.objects*
>>>>
>>>> >>> print(Content.objects.
>>>> Content.objects.aggregate(Content.objects.datetimes(
>>>>  Content.objects.from_queryset(
>>>> Content.objects.prefetch_related(
>>>> Content.objects.all(  Content.objects.db
>>>>  Content.objects.get(  Content.objects.raw(
>>>> Content.objects.annotate( Content.objects.db_manager(
>>>> Content.objects.get_or_create(Content.objects.reverse(
>>>> Content.objects.auto_created  Content.objects.deconstruct(
>>>>  Content.objects.get_queryset(
>>>> C

Re: [Pulp-dev] How to list orphan content in pulp3 before remocing them ?

[Daniel Alley]

There is not a way to do this, figuring out which content are orphans is
pretty expensive so I'm not certain it would be suitable for an endpoint
unless we changed how it works.

On Thu, Apr 14, 2022 at 11:30 AM Sayan Das  wrote:

> Hello Grant,
>
> I hope you are doing great.
>
> Few other support folks had reached out to me to understand if we can list
> the Orphaned data anyway in Pulp3 (Sat 6.10\RHUI 4).
>
> By anychance, Were you able to find out a way to do so in the version of
> pulpcore being shipped for Sat 6.10 or RHUI 4 (3.14 i believe)?
>
>
> Thanks & Regards,
>
> Sayan das
>
> *S*enior* T*echnical *S*upport *E*ngineer, RHCE
>
> Red Hat India
> 
>
> Red Hat India Pvt. Ltd, Level-5, Tower-10, Cyber City
>
> Magarpatta City Hadapsar, Pune-411013, Maharashtra, India.
>
> say...@redhat.comM: +91-7890892756 IRC: Sayan
> 
>
>
> On Mon, Jan 31, 2022 at 9:31 PM Grant Gainey  wrote:
>
>> On Mon, Jan 31, 2022 at 9:09 AM Sayan Das  wrote:
>>
>>> Hi Grant,
>>>
>>> I was trying your Django method but it fails in Satellite 6.10.2 .
>>>
>>
>> Oh...drat. This is what comes of answering emails w/out enough caffeine.
>> orphan-cleanup being asynchronous, and therefore that call, isn't available
>> for 6.10, it's in 3.15 and beyond (and so slated for Sat7). See
>> https://docs.pulpproject.org/pulpcore/changes.html#id140
>>
>> Give me a little, I'll see what we can do in the 3.14 timeframe.
>>
>> Sorry about that,
>> G
>>
>>
>>>
>>> # sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager
>>> shell << EOF
>>> from pulpcore.app.models.content import Content
>>> minutes_since_touched = 1
>>> print(Content.objects.orphaned(minutes_since_touched).all())
>>> EOF
>>>
>>> Traceback (most recent call last):
>>>   File "/bin/pulpcore-manager", line 11, in 
>>> load_entry_point('pulpcore==3.14.9', 'console_scripts',
>>> 'pulpcore-manager')()
>>>   File "/usr/lib/python3.6/site-packages/pulpcore/app/manage.py", line
>>> 11, in manage
>>> execute_from_command_line(sys.argv)
>>>   File
>>> "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line
>>> 381, in execute_from_command_line
>>> utility.execute()
>>>   File
>>> "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line
>>> 375, in execute
>>> self.fetch_command(subcommand).run_from_argv(self.argv)
>>>   File
>>> "/usr/lib/python3.6/site-packages/django/core/management/base.py", line
>>> 323, in run_from_argv
>>> self.execute(*args, **cmd_options)
>>>   File
>>> "/usr/lib/python3.6/site-packages/django/core/management/base.py", line
>>> 364, in execute
>>> output = self.handle(*args, **options)
>>>   File
>>> "/usr/lib/python3.6/site-packages/django/core/management/commands/shell.py",
>>> line 92, in handle
>>> exec(sys.stdin.read())
>>>   File "", line 3, in 
>>> AttributeError: 'ContentManagerFromBulkTouchQuerySet' object has no
>>> attribute 'orphaned'
>>>
>>> I definitely don't see *orphaned *inside *Content.objects*
>>>
>>> >>> print(Content.objects.
>>> Content.objects.aggregate(Content.objects.datetimes(
>>>Content.objects.from_queryset(Content.objects.prefetch_related(
>>> Content.objects.all(  Content.objects.db
>>>Content.objects.get(  Content.objects.raw(
>>> Content.objects.annotate( Content.objects.db_manager(
>>> Content.objects.get_or_create(Content.objects.reverse(
>>> Content.objects.auto_created  Content.objects.deconstruct(
>>>Content.objects.get_queryset( Content.objects.select_for_update(
>>> Content.objects.bulk_create(  Content.objects.defer(
>>>Content.objects.in_bulk(  Content.objects.select_related(
>>> Content.objects.bulk_get_or_create(   Content.objects.difference(
>>> Content.objects.intersection( Content.objects.touch(
>>> Content.objects.bulk_update(  Content.objects.distinct(
>>> Content.objects.iterator( Content.objects.union(
>>> Content.objects.check(Content.objects.earliest(
>>> Content.objects.last( Content.objects.update(
>>> Content.objects.complex_filter(   Content.objects.exclude(
>>>Content.objects.latest(   Content.objects.update_or_create(
>>> Content.objects.contribute_to_class(  Content.objects.exists(
>>> Content.objects.model(Content.objects.use_in_migrations
>>> Content.objects.count(Content.objects.explain(
>>>Content.objects.name  Content.objects.using(
>>> Content.objects.create(   Content.objects.extra(
>>>Content.objects.none( Content.objects.values(
>>> Content.objects.creation_counter  Content.objects.filter(
>>> Content.objects.only( Content.objects.values_list(
>>> Content.objects.dates(Content.objects.first(
>>>Content.objects.order_by(

[Pulp-dev] pulp_rpm 3.15.0 is generally available

[Daniel Alley]

For more information, check out the post on Discourse:

https://discourse.pulpproject.org/t/pulp-rpm-3-15-0-release/102
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [Pulp-list] Github Discussions

[Daniel Alley]

And Matrix integration.
https://meta.discourse.org/t/chatroom-integration-plugin-discourse-chat-integration/66522


On Thu, Jun 17, 2021 at 1:11 PM David Davis  wrote:

> I was rather surprised but Discourse did approve us for a free plan. The
> instance has been set up at https://pulp.discourse.group/. Feel free to
> try it out and experiment with it.
>
> However, the main limitation I see is the 50k pageviews per month limit.
> For reference, community.theforeman.org gets about 160-220k pageviews per
> month. So I think this would end up being a problem. I also checked with
> OSCI though and they apparently have experience with setting up Discourse
> instances for other open source communities. So I think they might be
> willing to set up a Discourse instance for us. I'm going to follow up and
> confirm.
>
> If we go this route, then we can also add the calendar plugin which would
> give us a public calendar like foreman has:
>
> https://community.theforeman.org/c/events/13/l/calendar
>
> I'm imagining we can use this to list events that are open to the public
> like triage/open floor and perhaps plugin team meetings.
>
> The more I use Discourse, the more I feel like it ticks all the boxes from
> granular notifications (weekly digests, ability to follow/unfollow
> categories, etc) to a mailing list mode users can enable to SSO integration
> with services like Github.
>
> David
>
>
> On Wed, Jun 16, 2021 at 7:59 PM Neal Gompa  wrote:
>
>> On Wed, Jun 16, 2021 at 7:21 PM David Davis 
>> wrote:
>> >
>> > One of the main reasons we wanted to move off mailing lists is that
>> signing up is inconvenient for users that may just want to ask a single
>> question. But I agree that Github Discussions is not so great if users want
>> to periodically monitor news and discussions for a project like Pulp. I
>> wonder perhaps if Discourse[0] would be a better fit (see Foreman's use of
>> discourse as an example[1]). It provides a mailing list mode that users can
>> enable if they want to interact solely via email. And also, it also offers
>> more granular control of notifications and auth with services like Github.
>> >
>>
>> The Fedora mailing list system (HyperKitty) actually manages to strike
>> a good balance here. You can sign in with OIDC/Social Login with
>> Fedora, Google, etc. and post to a list and only be subscribed to that
>> thread. Or you can subscribe to the whole list and do things like we
>> do now.
>>
>> > One of the main concerns we've had around using Discourse is hosting it
>> ourselves but it does look like Discourse provides a free plan for open
>> source projects[2]. It has some limitations (not sure if they'd be an issue
>> for us?) but I've submitted an application to see if we qualify. I should
>> hear back in a couple days.
>> >
>>
>> I suspect not, but it'd be an interesting surprise if we qualified.
>>
>>
>>
>> --
>> 真実はいつも一つ！/ Always, there's only one truth!
>>
>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [Pulp-list] Github Discussions

[Daniel Alley]

>
> It's pretty hard to do filtering of GitHub mail...


I can't disagree with that.

On Wed, Jun 16, 2021 at 4:35 PM Neal Gompa  wrote:

> Well, let's see if that works. Another concern I have is about
> filtering stuff from discussions in Gmail. It's pretty hard to do
> filtering of GitHub mail...
>
> On Wed, Jun 16, 2021 at 4:29 PM David Davis  wrote:
> >
> > If you watch the repo, you should be able to get notifications when
> there's new activity?
> >
> > David
> >
> >
> > On Wed, Jun 16, 2021 at 3:13 PM Neal Gompa  wrote:
> >>
> >> On Wed, Jun 16, 2021 at 11:59 AM David Davis 
> wrote:
> >> >
> >> > Yesterday at open floor, we discussed decommissioning pulp-dev list
> in favor of of using Github Discussions[0] for developer discussions.
> >> >
> >> > If there are no objections, I plan to decommission the pulp-dev list
> next week.
> >> >
> >> > [0] https://github.com/pulp/community/discussions
> >> >
> >>
> >> I don't know how I'd be able to keep track of what's going on with
> >> GitHub Discussions. With mailing list discussions, it's relatively
> >> easy for me to see the topics as they're coming in and participate off
> >> the cuff. With GitHub Discussions, it doesn't seem possible for me to
> >> do that.
> >>
> >>
> >>
> >> --
> >> 真実はいつも一つ！/ Always, there's only one truth!
> >>
>
>
> --
> 真実はいつも一つ！/ Always, there's only one truth!
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Feedback Wanted, Upcoming Changes in Pulp Python

[Daniel Alley]

I mostly concur with Matthias, "pypi" is just as unlikely to lead to any
namespace conflicts as "pulp_python/pypi/".  It wouldn't be too significant
either way, though.

+1 to the general principle of the change.



On Wed, May 19, 2021 at 4:01 AM Matthias Dellweg 
wrote:

> Just one thought:
> I assume $PULP_BASE_ADDR is the part without "pulp/api/v3/". So why should
> you include the "pulp_python" segment? You could just host it as e.g. "
> http://localhost/pypi/foo/simple;. The interfaces provided there should
> not be in any way specific to pulp_anything.
>
> On Tue, May 18, 2021 at 11:44 PM Gerrod Ubben  wrote:
>
>> Hey everyone,
>>
>> I am currently working on the long-awaited Twine upload support [0] for
>> pulp_python. To add this functionality a new upload endpoint is needed.
>> Since this new endpoint will need to be able to accept POST requests I plan
>> to make the Python endpoints apart of the Pulp API.
>>
>> Proposed change: Have Python endpoints available at
>> "/pulp_python/pypi/{base_path}/"
>> API Current Location New Location Description
>> simple $PULP_CONTENT_ADDR/pulp/content/{base_path}/simple/
>> $PULP_BASE_ADDR/pulp_python/pypi/{base_path}/simple/ Used for installing
>> packages
>> pypi $PULP_CONTENT_ADDR/pulp/content/{base_path}/pypi/
>> $PULP_BASE_ADDR/pulp_python/pypi/{base_path}/pypi/ Used for syncing
>> repositories
>> upload
>> $PULP_BASE_ADDR/pulp_python/pypi/{base_path}/legacy/
>> **$PULP_BASE_ADDR/pulp_python/pypi/{base_path}/simple/* Used for
>> uploading packages
>>
>> The current Python endpoints will still be available at their present
>> locations (and will be used as the backbone for the new locations) and the
>> new upload endpoint will be available at '/legacy/' since the default PyPI
>> endpoint Twine uses is 'https://upload.pypi.org/legacy/'. Pip stores its
>> index URLs in a 'pip.conf' file [1]  and Twine stores its index URLs in the
>> '.pypirc' file [2], so to have an easy setup across all Python tools I am
>> thinking of also having the upload functionality available at the simple
>> URL. This way one URL can be used for the Pulp Python index across all the
>> Python tooling.
>>
>> An example workflow with these new changes:
>>
>>> pulp python repository create --name foo
>>> pulp python distribution create --name foo --base-path foo --repository
>>> foo
>>> export PULP_PYTHON_INDEX=$PULP_BASE_ADDR/pulp_python/pypi/foo/simple/
>>> twine upload --repository-url $PULP_PYTHON_INDEX dist/* --skip-existing
>>> pip install -i $PULP_PYTHON_INDEX pulpcore
>>>
>>
>> Do these changes concern anyone? Would a different name for the upload
>> endpoint be preferred? Should the simple endpoint also support uploading
>> packages? I plan on having this feature fully complete and ready for review
>> by the end of the week, May 21st. Any feedback before then would be
>> appreciated.
>>
>> [0] https://github.com/pulp/pulp_python/pull/367
>> [1] https://pip.pypa.io/en/stable/user_guide/#configuration
>> [2] https://packaging.python.org/specifications/pypirc/
>>
>> Thanks,
>> Gerrod
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://listman.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulpcore 3.13.0 release schedule & go/no-go irc meeting

[Daniel Alley]

3.13.0 has been pushed back one more week, with a new date of May 25th.

On Tue, May 11, 2021 at 10:58 PM Daniel Alley  wrote:

> Pulpcore 3.13.0 will be pushed back until next week, tentatively May 18th.
>
> Another go/no-go meeting will happen in #pulp-meeting at the time below:
>
> May 11th, 3:00 PM UTC/May 7, 10:00 AM ET
> https://everytimezone.com/s/26b08bc3
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulpcore 3.13.0 release schedule & go/no-go irc meeting

[Daniel Alley]

Pulpcore 3.13.0 will be pushed back until next week, tentatively May 18th.

Another go/no-go meeting will happen in #pulp-meeting at the time below:

May 11th, 3:00 PM UTC/May 7, 10:00 AM ET
https://everytimezone.com/s/26b08bc3
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulp_rpm and current backwards-compatibility problems

[Daniel Alley]

These are the commits that definitely need to be reverted (in-order).  We
also want to check up on the signing service, either to make sure the
migration plugin doesn't hit that codepath, or that we're still compatible
with 3.7 even though pulpcore 3.10 introduced some changes that we had to
be compatible with.  But just glancing through these are the obvious ones.

https://github.com/pulp/pulp_rpm/commit/d0c9badd21251e684376a6e0b0fca11c6faec208
- Matthias refactoring the auto-publish migration
https://github.com/pulp/pulp_rpm/commit/fbaadaca61b5795afcfab4fcfb6ddfb52438a2da
- Me adding auto-publish, and in the same commit, also the BaseDistribution
change
https://github.com/pulp/pulp_rpm/commit/3157ad10abf777009620ffa421b3eab45d85a5bb
- Oracle ULN repo, bumped the pulpcore floor version to 3.10 from 3.7, not
sure if that is 100% necessary or if it just happened

And then once 3.11 is out, we can re-apply those in reverse order, to get
back to where we were.  We'll need to fix up the migrations though.

@Tanya, I know you offered to help with this, if you have any time in the
morning to take a look at this, that would be excellent.  I think you're
the best person to figure out the answer to the signing service / migration
plugin question, as well.



On Tue, May 11, 2021 at 2:23 PM Grant Gainey  wrote:

> On Tue, May 11, 2021 at 1:36 PM Grant Gainey  wrote:
>
>>
>> I'm going to submit a PR to get pulp_rpm's CI unbroken, that has to
>> happen before we do anything else. THEN I will respond to final
>> review-comments on PR 1984. Once that passes, THEN we can talk about what
>> happens next. (That PR, for example, is affected by the proposed reverts,
>> because its migration is currently built on-top-of the one(s) being
>> reverted and will need to be renamed. So we/I probably don't want to merge
>> it, until post-revert, and an update to match the new State Of The World at
>> that point.)
>>
>> Anyway, first things first...
>>
>
> Update: no, I'm not. Marking pulp_rpm as "requires pulpcore/3.12" fails to
> build. Because core-3.12 depends on django~=2.2.20, which pulls in 2.2.21,
> which is the Broken Django, and Everything Is Sad. (Mostly me, to be honest)
>
> Right this second, we can't build against core-master, and we can't build
> against anything other than -master because of the Django Problem. PR#1985
>  is waiting while I think
> about how to make progress.
>
> G
> --
> Grant Gainey
> Principal Software Engineer, Red Hat System Management Engineering
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulpcore 3.11.0 release schedule & go/no-go irc meeting

[Daniel Alley]

Correction: Pulpcore 3.13.0

On Thu, May 6, 2021 at 5:11 PM Daniel Alley  wrote:

> Here's the tracker for the pulpcore 3.13.0 release
> https://pulp.plan.io/issues/8694
> The tentative GA date is May 12th, with a fallback date of May 18th.
>
> The first go/no-go meeting will happen in #pulp-meeting at the time below:
>
> May 7th, 3:00 PM UTC/May 7, 10:00 AM ET
> https://everytimezone.com/s/26b08bc3
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulpcore 3.11.0 release schedule & go/no-go irc meeting

[Daniel Alley]

Here's the tracker for the pulpcore 3.13.0 release
https://pulp.plan.io/issues/8694
The tentative GA date is May 12th, with a fallback date of May 18th.

The first go/no-go meeting will happen in #pulp-meeting at the time below:

May 7th, 3:00 PM UTC/May 7, 10:00 AM ET
https://everytimezone.com/s/26b08bc3
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Tracking issues for plugin_template

[Daniel Alley]

I doubt there's any "good" way to link external bugs from Github because
Mozilla doesn't do so.  They just add an extra tag and you just have to
find the link somewhere in the comments.

e.g.
https://github.com/servo/webrender/issues?q=is%3Aopen+is%3Aissue+label%3Abugzilled

On Fri, Apr 23, 2021 at 10:37 AM David Davis  wrote:

> We started working on a plan to move repos over to Github Issues after
> PulpCon last year but I think it kind of fell by the wayside over the past
> few months due to how busy we've been with stakeholder work. It would
> definitely be a requirement in my mind to sort out things like how to link
> BZs to issues before moving plugins and other repos over to Gihub Issues
> that might have issues that affect downstream.
>
> There is a way to view issues across the entire Pulp org in Github[0] but
> it's not great. Definitely not as good as the redmine issue search. We
> should figure out if it'll meet our needs and if not, find a  solution that
> will.
>
> [0]
> https://github.com/issues?q=is%3Aopen+is%3Aissue+archived%3Afalse+user%3Apulp+
>
> David
>
>
> On Thu, Apr 22, 2021 at 7:42 AM Grant Gainey  wrote:
>
>> On Thu, Apr 22, 2021 at 6:48 AM Ina Panova  wrote:
>>
>>>
>>> On Wed, Apr 21, 2021 at 7:55 PM David Davis 
>>> wrote:
>>>
 Wanted to bump this to hopefully get some feedback. Also, today during
 our CI/CD meeting we discussed also tracking issues for pulp-oci-images on
 github as well.

 If there are no objections by next week April 26, I'll assume I can
 proceed with moving these projects' issues over to Github.

 David


>>> I have no objections regarding moving plugin_template and
>>> pulp-oci-images to github.
>>>
>>> The question I have is - what is our long term goal? Do we aim to
>>> eventually move all of the projects from pulp.plan.io? I know there
>>> were discussions in the past but we have not found a solution on how to
>>> connect issues with downstream trackers. The concern I have, if we don't
>>> have such plan, then we might have plugins' issue tracking scattered
>>> between plan.io and github where users, as a result, will file issues
>>> in the redmine and then we will need to whether ask them to open a github
>>> issue or do it ourselves.
>>> We just had a python issue filed in the redmine last week.
>>>
>>
>> This is a good point. I just looked at bugzilla, and 'Github' is listed
>> as one of the Systems you can "Add Link" to. We'd (obviously) need to
>> modify the linking/state-change-scripts. The complicating issues I can see
>> there, are a) not having moved *all* the projects to github, so some BZs
>> would want links to redmine and some to github, and b) being able to get
>> "all" the issues - in redmine, I can go to pulp.plan.io/issues and see
>> all the issues for every project, not sure we can get something equivalent
>> from github.
>>
>> G
>>
>>
>>>
 On Fri, Apr 16, 2021 at 8:38 AM David Davis 
 wrote:

> I've always felt that tracking plugin_template issues under the main
> pulp project in plan.io was suboptimal and with other repos such as
> pulp-cli moving to github issues, I feel that it might make sense for the
> plugin_template to move to github issues as well.
>
> There's only 11 open issues right now in pulp.plan.io? for the
> plugin_template so I think it would be an easy move. I'd propose we also
> remove the plugin_template tag from the list of options (but keep it on 
> old
> issues for history).
>
> Thoughts?
>
> David
>
 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://listman.redhat.com/mailman/listinfo/pulp-dev

>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://listman.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
>>
>> --
>> Grant Gainey
>> Principal Software Engineer, Red Hat System Management Engineering
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Tasking System Changes and Feedback

[Daniel Alley]

>
> Are there any benefits to improving RQ vs the invented here method? I'm
> just curious about the cost of maintaining a tasking system versus being
> part of a community built one. This feels like the kind of problem that
> many other applications should have in the Python world -- or are there
> elements of Pulp's deployment architecture that make it unique here?
>

This shouldn't be viewed through an "invented here" lens, because most of
what the proposal does would actually reduce our dependence on "invented
here".

Basically there is a fundamental problem with having task state split
between the database, and some external service (RQ), which is incredibly
difficult to keep consistent.  There is a lot of existing complexity around
resource locking that would completely go away if we just switch to keeping
the "task queue" in the database, and using normal transactions and
row/table locks rather than separate "lock objects" in additional tables,
etc.

The idea is, we already have all of this information about the tasks in the
database (reporting what happened and so on), and if we just store 2 extra
pieces of information - the function to execute, and the parameters to
execute with - we will essentially have the "front half" of a task queue,
that we can much more easily keep in a consistent state with everything
else.

This actually is a fairly common problem - it's called the outbox pattern:
https://microservices.io/patterns/data/transactional-outbox.html

Regarding the "back half", which is dealing with the actual process of
spawning the process, I'm less certain.  Maybe Matthias can explain what
the plan is, there.  IMO even if we continued using RQ for that portion
(part 3 in the diagram in the link), the change to the "front half"
(everything up to and including the pulp resource manager") makes a lot of
sense and would be a significant net reduction in complexity.

This is sort of an aside to this general change. Are Pulp tasks cleaned up
> from the database today?
>

They aren't.  We don't clean up anything automatically, cleanup is
user-driven.

On Tue, Apr 13, 2021 at 11:18 AM Eric Helms  wrote:

>
>
> On Thu, Apr 8, 2021 at 5:24 PM Daniel Alley  wrote:
>
>> Eric,
>>
>> * The idea is to move away from RQ entirely.  RQ is fine (and vastly
>> better than Celery IMO), but managing task state across both 1) the
>> database and 2) a separate, external registry is still problematic.  If all
>> of the information can simply be kept in the database, then it will be much
>> easier to maintain consistent state.
>>
>
> Are there any benefits to improving RQ vs the invented here method? I'm
> just curious about the cost of maintaining a tasking system versus being
> part of a community built one. This feels like the kind of problem that
> many other applications should have in the Python world -- or are there
> elements of Pulp's deployment architecture that make it unique here?
>
>
>> * *Maybe*.  We're considering using Redis as a cache to improve content
>> serving performance (after all, caching is one of the primary uses of
>> Redis). If we do, then Redis would remain in the architecture, but it could
>> potentially be an optional component and would be easier to remove at some
>> point in the future.
>> * We'd just be adding a small amount of information to each task record,
>> and it wouldn't prevent cleanup later.
>>
>
> This is sort of an aside to this general change. Are Pulp tasks cleaned up
> from the database today?
>
>
>>
>>
>>
>> On Thu, Apr 8, 2021 at 4:42 PM Eric Helms  wrote:
>>
>>> A few initial questions that get a bit into the stack but will help the
>>> Foreman project think on the proposed changes:
>>>
>>>  * Does this move away from RQ entirely or just RQ workers?
>>>  * Do the new workers remove Pulp 3's use of Redis all together?
>>>  * Will using the database result in any additional build up of tasking
>>> information that can impact performance over time? (Or does all task data
>>> get cleaned up eventually?)
>>>
>>> Thanks for sending this along early.
>>>
>>> On Fri, Apr 2, 2021 at 4:43 PM Brian Bouterse 
>>> wrote:
>>>
>>>> FYI, @mdellweg and I have been collaborating on the tasking system
>>>> changes. This email is to share some info to transition the work to
>>>> @mdellweg while I'm out. With the new-style disabled by default I am hoping
>>>> it can go into 3.13.
>>>>
>>>> ## The PoC and ticket info
>>>>
>>>> The PoC is basically functional, but it's a PoC:
>>>>

Re: [Pulp-dev] Tasking System Changes and Feedback

[Daniel Alley]

Eric,

* The idea is to move away from RQ entirely.  RQ is fine (and vastly better
than Celery IMO), but managing task state across both 1) the database and
2) a separate, external registry is still problematic.  If all of the
information can simply be kept in the database, then it will be much easier
to maintain consistent state.
* *Maybe*.  We're considering using Redis as a cache to improve content
serving performance (after all, caching is one of the primary uses of
Redis). If we do, then Redis would remain in the architecture, but it could
potentially be an optional component and would be easier to remove at some
point in the future.
* We'd just be adding a small amount of information to each task record,
and it wouldn't prevent cleanup later.



On Thu, Apr 8, 2021 at 4:42 PM Eric Helms  wrote:

> A few initial questions that get a bit into the stack but will help the
> Foreman project think on the proposed changes:
>
>  * Does this move away from RQ entirely or just RQ workers?
>  * Do the new workers remove Pulp 3's use of Redis all together?
>  * Will using the database result in any additional build up of tasking
> information that can impact performance over time? (Or does all task data
> get cleaned up eventually?)
>
> Thanks for sending this along early.
>
> On Fri, Apr 2, 2021 at 4:43 PM Brian Bouterse  wrote:
>
>> FYI, @mdellweg and I have been collaborating on the tasking system
>> changes. This email is to share some info to transition the work to
>> @mdellweg while I'm out. With the new-style disabled by default I am hoping
>> it can go into 3.13.
>>
>> ## The PoC and ticket info
>>
>> The PoC is basically functional, but it's a PoC:
>> https://github.com/pulp/pulpcore/pull/1222/
>>
>> * The epic is being tracked here which recaps why we're doing this and
>> the high level approach. The sub-tasks capture the various detailed
>> changes. https://pulp.plan.io/issues/8495
>>
>> * This is totally separate from the RQ workers you use today, and those
>> will continue to be available for a while.
>>
>> ## Next Steps
>>
>> * @mdellweg will continue the work and hopefully merge the PoC while I'm
>> out
>>
>> * Once it's demo-able I've asked @mdellweg to give a 20 minute, public
>> (hopefully recorded) technical demo. While it is designed to be a drop-in
>> replacement from a user perspective, we think sharing the internals will be
>> helpful to get feedback and increase the list of those who understand the
>> work.
>>
>> All the best,
>> Brian
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://listman.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> --
> Eric Helms
> Principal Software Engineer
> Satellite
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Content management and disk space clean-up

[Daniel Alley]

These bugs are not related to content management, but they are related to
disk space (not being released)

4.  Temporary artifacts not always cleaned up
https://pulp.plan.io/issues/7316
5.  Disk usage during sink https://pulp.plan.io/issues/8295
6.  Worker directories not always cleaned up
https://pulp.plan.io/issues/8296

On Tue, Mar 16, 2021 at 11:29 AM Ina Panova  wrote:

> Folks,
> I am gathering a group of interest that would meet and design things
> around content management such as:
>
> 1. Content removal and content rejection https://pulp.plan.io/issues/8372
> 2. Keep N repo versions https://pulp.plan.io/issues/8368
> 3. Orphan clean up https://pulp.plan.io/issues/7659
>
> All these are whether directly or indirectly related to the problem of how
> to reclaim the disk space.
>
> If you have not received an invite and you want to participate please ping
> me so I add you.
>
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Breaking change (for plugin writers) announcement

[Daniel Alley]

We determined that there is a path for moving forwards which would be a
little more work (in the short term), but would preserve the deprecation
policy - so we are changing the plan described above and will instead be
maintaining the deprecation policy. It’s written into the epic (
https://pulp.plan.io/issues/8383 ) with most of the work happening in 3.12.
The timeline is being driven by auto-distribute being needed also in 3.12
and this work blocking that.

With the deprecation policy remaining intact, it is recommended for your
plugin releases compatible with 3.11 to also declare compatibility with
3.12.

On Thu, Mar 4, 2021 at 3:49 PM Daniel Alley  wrote:

> *tl;dr*
>
> 1. Plugins compatible with 3.11 should *not* declare compatibility with
> 3.12
> 2. Review the PR for your plugin below [0] and include it in your 3.12
> plugin compatibility release
>
> (PR review doesn't need to happen now, just in the next few weeks)
>
> *3.12 Plugin API Breaking Changes*
>
> In Pulpcore 3.12, which the release *after* the upcoming 3.11 release, we
> will be making a breaking change to Pulpcore which will require some
> intervention by plugin writers. This does not follow the deprecation
> policy; we tried very hard to look at other options, but after a lot of
> discussion this is the least disruptive approach.
>
> All plugins currently released should set their compatibility ceiling at
> <3.12, and all plugin releases after the change will need to set their
> compatibility floor to >= 3.12.
>
> *What you need to do:*
>
> Ensure your 3.11 compatibility releases don't declare compatibility with
> 3.12. Many plugins, including File, Container, Certguard, and Ansible
> already have a support ceiling set for 3.12 (or before). Other plugins such
> as RPM, Debian, Maven, don't have a support ceiling set. These should be
> added to pre-3.12-compatible releases.
>
> Then, for your 3.12 plugin compatibility releases, review the PR for your
> plugin below [0].
>
> Please feel free to reach out to me over email or IRC if you would like
> assistance with anything I've just discussed.
>
> [0] PRs:
>
> Pulpcore
> https://github.com/pulp/pulpcore/pull/1084
>
> Pulp File
> https://github.com/pulp/pulp_file/pull/467
>
> Pulp RPM
> https://github.com/pulp/pulp_rpm/pull/1938
>
> Pulp NPM
> https://github.com/pulp/pulp_npm/pull/71
>
> Pulp Container
> https://github.com/pulp/pulp_container/pull/249
>
> Pulp Ansible
> https://github.com/pulp/pulp_ansible/pull/527
>
> Pulp Python
> https://github.com/pulp/pulp_python/pull/329
>
> Pulp Debian
> https://github.com/pulp/pulp_deb/pull/248
>
> Pulp Maven and the 2to3 migration plugin
>
>- No patches necessary
>
> Pulp Gem and Pulp Cookbook, and any other plugins not listed:
>
>- I didn't make PRs, but I'd be happy to help anyone who does so
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Breaking change (for plugin writers) announcement

[Daniel Alley]

*tl;dr*

1. Plugins compatible with 3.11 should *not* declare compatibility with 3.12
2. Review the PR for your plugin below [0] and include it in your 3.12
plugin compatibility release

(PR review doesn't need to happen now, just in the next few weeks)

*3.12 Plugin API Breaking Changes*

In Pulpcore 3.12, which the release *after* the upcoming 3.11 release, we
will be making a breaking change to Pulpcore which will require some
intervention by plugin writers. This does not follow the deprecation
policy; we tried very hard to look at other options, but after a lot of
discussion this is the least disruptive approach.

All plugins currently released should set their compatibility ceiling at
<3.12, and all plugin releases after the change will need to set their
compatibility floor to >= 3.12.

*What you need to do:*

Ensure your 3.11 compatibility releases don't declare compatibility with
3.12. Many plugins, including File, Container, Certguard, and Ansible
already have a support ceiling set for 3.12 (or before). Other plugins such
as RPM, Debian, Maven, don't have a support ceiling set. These should be
added to pre-3.12-compatible releases.

Then, for your 3.12 plugin compatibility releases, review the PR for your
plugin below [0].

Please feel free to reach out to me over email or IRC if you would like
assistance with anything I've just discussed.

[0] PRs:

Pulpcore
https://github.com/pulp/pulpcore/pull/1084

Pulp File
https://github.com/pulp/pulp_file/pull/467

Pulp RPM
https://github.com/pulp/pulp_rpm/pull/1938

Pulp NPM
https://github.com/pulp/pulp_npm/pull/71

Pulp Container
https://github.com/pulp/pulp_container/pull/249

Pulp Ansible
https://github.com/pulp/pulp_ansible/pull/527

Pulp Python
https://github.com/pulp/pulp_python/pull/329

Pulp Debian
https://github.com/pulp/pulp_deb/pull/248

Pulp Maven and the 2to3 migration plugin

   - No patches necessary

Pulp Gem and Pulp Cookbook, and any other plugins not listed:

   - I didn't make PRs, but I'd be happy to help anyone who does so
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [Pulp-list] Removing MD5 and SHA-1 as default available checksums in 3.11

[Daniel Alley]

Didn't know that about SLE 11, thanks.

On Sun, Feb 14, 2021 at 9:28 PM Neal Gompa  wrote:

> On Sun, Feb 14, 2021 at 9:22 PM Daniel Alley  wrote:
> >>
> >> RPM and Migration plugin users will need to add this back in at 3.11
> upgrade time for your systems to continue working.
> >
> >
> > Just to clarify, this only applies if you are using RPM repositories
> that use MD5 or SHA1 checksums.  None of the Red Hat or CentOS repositories
> for RHEL / CentOS 8 do, and only a couple of the RHEL / CentOS 7
> repositories do, mosty the smaller and less used ones.
> >
> > MD5 can be left disabled unless you are managing extremely old or
> misconfigured repositories.
> >
>
> SHA1 is required for SLE 11 and RHEL 5, as neither can process
> SHA2-based checksums. While RHEL 5 is firmly EOL, SLE 11 is still
> supported for two more years for SUSE customers using LTSS for SLE 11
> SP4.
>
>
>
> --
> 真実はいつも一つ！/ Always, there's only one truth!
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Backports and LTS

[Daniel Alley]

>
> Matt, that is a good observation and meanwhile with pulp2 we had such a
> policy, we cannot adopt it with pulp3. Release fast and release often is
> something we are trying to stick to.
>

I don't think Matt was suggesting in any way that we not release fast and
often, it's just that we have to pick a combination of cadance and # of
supported releases that works for everyone.  This is basically Option 1
with a pre-determined number of supported releases (rather than "backport
to whatever version the stakeholder is using + all the others in between"
which is how I interpret Option 1).

I totally agree with David that it would be a pain to manage without
automation though, same as Option 1.

I think what would most likely happen under that plan is that we would see
each stakeholder will take a release, stick to it until it is about to be
dropped and then upgrade to the newest one, which is roughly the same as
the LTS plan except that each stakeholder can choose their own "LTS"
version and they could do an mid-cycle upgrade if there was ever a good
reason to.

Whatever option we choose, I'm kind of negative about actually using the
term "LTS" given that I don't think we'd be supporting any version for more
than 5-7 months.  The timeframe is a bit short for how the LTS moniker is
typically used.

On Wed, Jan 6, 2021 at 8:58 AM Ina Panova  wrote:

> Matt, that is a good observation and meanwhile with pulp2 we had such a
> policy, we cannot adopt it with pulp3. Release fast and release often is
> something we are trying to stick to.
>
> It would be best to agree on the LTS version that would make all the
> stakeholders happy but as it was pointed out not sure how we'd make this
> possible, everyone has a different pace and release cycle.
> Having that said, we should probably stick to option1 which puts more
> burden on us but in return users and stakeholders are happy and the
> project's upgrade path is stable.
> One thing we could possibly do about the backport requests is to really be
> thorough about which ones to accept by assessing the impact of the
> stakeholder who has created the request and their possibility to upgrade if
> any.
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Wed, Jan 6, 2021 at 2:18 PM Matt Pusateri  wrote:
>
>> Another option is Current Version minus X (N-1, N-2, etc.) If for
>> instance you say we support N-1, and current version is 3.9, then you'll
>> back port to latest 3.8.x.  N-2 at current version 3.9, would backport to
>> 3.8.x, 3.7.x, etc.  The advantages here are that you already set the
>> expectation for everyone, you limit the versions you support, you force
>> people to stay current to get fixes.  The downside is that people have to
>> upgrade and or it could impact downstream schedules.  The impact with going
>> this route is affected by the release velocity.  So if you're rapidly
>> releasing major versions, then it's more likely people won't keep up, or
>> that they'll have to upgrade and are not able to for some reason.
>>
>> Matt P.
>>
>>
>> On Wed, Jan 6, 2021 at 6:05 AM Matthias Dellweg 
>> wrote:
>>
>>> Thanks for putting all the options together.
>>> I would say that option 2 is a recipe for very bad user experience, and
>>> i'd vote strongly against it.
>>> I am ok with option 1, but I would add that the backport to the
>>> intermediate minor releases _must_ be performed (as in released) counting
>>> down, to always meet that upgrade expectation. Remember releasing can be
>>> delayed unexpectedly due to reasons.
>>> If we go for option 3, I think it's advisable to try to sync up
>>> stakeholders, because we don't want to maintain consecutive minor versions
>>> as LTS, and again, backporting a fix must traverse all maintained LTS in
>>> backward order. We should add expectations to how long we can keep an LTS
>>> version alive, and how often we bless one.
>>>
>>> On Tue, Jan 5, 2021 at 6:34 PM Tanya Tereshchenko 
>>> wrote:
>>>
 With more and further away backport requests coming in, there is more
 need for a clear policy/docs to set expectations for users and to define
 requirements for those performing a backport.

 The question which hasn't been answered yet (in documented way) is:

 *Should backports be backported to every (minor) version between the
 fix and the requested version?*

 E.g. A backport is requested for 3.7, the original fixed will be
 released in 3.10.
 Should the backport be added only to 3.7 or to 3.8 and 3.9 as well?
 Reminder: a backport can only be a bug fix and without a db migration.

 Option 1. Backport to all releases in between.
  + it's an expected upgrade path for users, no surprises, the fix is
 present all the way.
  - it's a lot of maintenance and release burden, and the further
 backport is the worse it is.

Re: [Pulp-dev] RBAC: Secure by default?

[Daniel Alley]

+1

What happens if a new account is created on an existing Pulp installation
(if that is possible)?  Would it then start following the deny-by-default
pattern?

On Wed, Jan 6, 2021 at 8:57 AM David Davis  wrote:

> +1 from me.
>
> David
>
>
> On Wed, Jan 6, 2021 at 8:28 AM Ina Panova  wrote:
>
>> +1 to the change.
>>
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Senior Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Wed, Dec 16, 2020 at 8:14 PM Tanya Tereshchenko 
>> wrote:
>>
>>> It sounds like a good idea,  and additional +1 that it doesn't break
>>> things.
>>>
>>> On Tue, Dec 15, 2020 at 5:57 PM Matthias Dellweg 
>>> wrote:
>>>
 In today's pulpcore meeting, we discussed that any endpoint that is not
 aware of RBAC yet will be open to every authenticated user.

 The suggestion that was given, is that we change that default. So all
 endpoints will raise permission errors unless RBAC opens them up.
 This would not affect any existing installation, where we only allowed
 the use of a single admin user. And by circumventing the permission
 framework this special user will remain to be able to talk to all available
 endpoints without restrictions.
 On the other hand it should smooth out the transition period until we
 have RBAC in all places. Since you could start giving permissions to users
 for viewsets that have an access_policy, while not risking to give them
 access to other sensitive parts that don't have it yet.

 What do you all think?
 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] RPM plugin meeting notes

[Daniel Alley]

### November 19, 2020

AI review: none

Pulp 3:

* Do we need to generate SQLiteDBs at all?
 * https://pulp.plan.io/issues/7851
 * https://pulp.plan.io/issues/7852
 * https://pulp.plan.io/issues/7842
 * introduce a switch, the default is not to generate sqlite metadata

* black PR: https://github.com/pulp/pulp_rpm/pull/1882

Pulp 2:
 * worked with Zhenech on releasing pulp_rpm-2.21.4-2 to fix [7849](
https://pulp.plan.io/issues/7849)
 * released today
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Travis pricing

[Daniel Alley]

It depends on how much the 80/20 rule applies.  If we had a breakdown of
which jobs are using how many minutes, I bet we could get that number way
down just by cleaning up a couple of jobs.

None of these platforms support nested virt (actually Travis doesn't
officially support it either, it just happens to work), so those jobs will
either need to stay behind on Travis, move to something like CentOS CI, or
else we need to bite the bullet and add some of our own hardware to the CI
pool (which is quite easy apparently with both Github and Gitlab).

On Mon, Nov 9, 2020 at 10:39 AM Brian Bouterse  wrote:

>
>
> On Mon, Nov 9, 2020 at 10:27 AM David Davis  wrote:
>
>> I looked a little bit more into Gitlab CI this morning. If my
>> understanding is correct, their cap is 50,000 minutes. Right now, we're
>> using about 44,000 minutes in Travis[0]. Some of our jobs (eg
>> pulp_installer, pulp-cli, etc) are on GHA already so in theory if we moved
>> everything over to Gitlab CI, we could potentially hit their cap?
>>
> This is my main concern. If we're going to spend the effort to switch, we
> need to not end up in the same situation.
>
>
>> [0] https://travis-ci.com/github/pulp?tab=insights
>>
>> David
>>
>>
>> On Wed, Nov 4, 2020 at 8:18 AM David Davis  wrote:
>>
>>> We looked at Fedora's Zuul instance before and decided against using it
>>> for two reasons: (A) it'd be a lot of work/maintaince (eg we'd have to
>>> write our own zuul jobs, bring our own compute resource, etc), and (B) we
>>> were worried about the support since it's not a paid (or freemium) option.
>>>
>>> That said, it might be worth considering again if nothing else for
>>> testing on specific environments such as selinux which is difficult to do
>>> on hosted CI providers.
>>>
>>> David
>>>
>>>
>>> On Tue, Nov 3, 2020 at 3:51 PM Neal Gompa  wrote:
>>>
 On Tue, Nov 3, 2020 at 3:35 PM David Davis 
 wrote:
 >
 > Travis recently announced changes to their plan pricing that will
 impact open-source projects such as Pulp[0]. It's likely that we'll exhaust
 the monthly budget that Travis is going to give OSS projects and we're not
 sure how generous Travis will be giving out extra build minutes.
 >
 > Given our concern, members of the CI team met today to discuss our
 options. We have some notes[1] from our meeting about some of the options
 that stood out to us. We'd like to have a plan in place when the new
 pricing gets rolled out to our organization.
 >
 > Any feedback is welcome.
 >
 > [0] https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing
 > [1] https://hackmd.io/n6kStnNiTPGekAWGdNvbhA

 Could we use the Fedora CI Zuul instance[2]? There's already a ton of
 other projects using one of the Zuul instances on
 softwarefactory-project.io, and leveraging the Fedora CI
 infrastructure could also help future efforts in doing auto-release to
 Fedora for Pulp releases, too.

 [2]: https://fedora.softwarefactory-project.io/zuul/projects

 --
 真実はいつも一つ！/ Always, there's only one truth!

 ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] RPM plugin meeting notes

[Daniel Alley]

### October 29, 2020

AI review: none

Pulp 3:

 * Need to be able to specify a custom 'gpgkey' string for a Publication.
 * Should do a new release - any blockers?
   * check for deprecations in pulpcore that affect the plugin
   * decide at next week's meeting

Pulp 2:

 * 2.21.4 release progressing w/pcreech

Open PRs:
 * https://github.com/pulp/pulp_rpm/pulls

Un-triaged bugs:
 * https://pulp.plan.io/projects/pulp_rpm/issues?query_id=159
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] RPM plugin meeting notes

[Daniel Alley]

AI review: none

Pulp 3:
 * Content upload API improvement: https://pulp.plan.io/issues/7708
 * Valid UX problem, could be difficult to fix "correctly"
 * Potentially changes necessary to the task response
 * Pulp 4 change?
 * Multi-repo copy issues: https://pulp.plan.io/issues/7625
 * Investigation in-progress, could either be an RPM plugin issue or a
Migration-from-Pulp2 issue

Pulp 2: none

Open PRs:
 * https://github.com/pulp/pulp_rpm/pulls

Un-triaged bugs:
 * https://pulp.plan.io/projects/pulp_rpm/issues?query_id=159
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] PR merging

[Daniel Alley]

I used it recently (last week).  Agreed that it is not commonly needed and
especially not in pulpcore though.

On Wed, Sep 30, 2020 at 8:52 AM Matthias Dellweg 
wrote:

>
>
> On Wed, Sep 30, 2020 at 2:37 PM David Davis  wrote:
>
>> I am also concerned about the lack of human involvement and the potential
>> for things to be merged accidentally. I definitely could see that happening.
>>
>> I like the idea of having the PR processor only merge if a label (eg
>> "Merge when Ready") is present. The question then is whether it should be
>> applied automatically or not when a PR is opened. Maybe to start out with,
>> it's not.
>>
>> One other thing I thought of is that the PR processor could also check PR
>> dependencies (ie "Required PRs") and see if they are all mergeable before
>> merging any PR.
>>
>
> I haven't used the required-something in a while, and with the new
> backwards compatibility policy it should not be needed anymore (definitely
> not on the pulpcore side). I'd say if there is such a thing on the PR it
> should never be merged automatically.
> Also we could say, that plugins should remove that tag, once the
> corresponding pulpcore change was merged.
>
>>
>> David
>>
>>
>> On Wed, Sep 30, 2020 at 4:10 AM Matthias Dellweg 
>> wrote:
>>
>>> This just reminds me that Gitlab has a very nice "merge when CI passes
>>> button" to decouple the merge question from the reviews.
>>> The only way i could see this happen in Github is via setting a label
>>> that instructs the PR processor to merge when (label is set) && (ci is
>>> green) && (other conditions).
>>> Does not sound too user friendly, but labels can only be set by people
>>> with merge permissions anyway (so not a security concern).
>>>
>>> On Tue, Sep 29, 2020 at 8:04 PM Daniel Alley  wrote:
>>>
>>>> I have some doubts about the cost/benefit ratio of coding automation to
>>>> merge PRs vs. simply changing the default option / being selective about
>>>> which options are available.
>>>>
>>>> I like the idea in general though.  A lot of projects do something like
>>>> this.  I occasionally contributed to the Servo project (RIP) and they used
>>>> automation to manage this.  The benefits were, their tests suites ran for
>>>> nearly 2 hours on 3 different operating systems, so the feedback loop was
>>>> quite slow, and automatically triggering the full test run every time a PR
>>>> was changed would be a horrendous waste of resources at the scale they were
>>>> running at.  So basically they would run a minimal CI within the PR, then
>>>> require a manual review, and once approved it would trigger the full CI,
>>>> and if that passed then it would automatically merge the PR hours later
>>>> whenever it finished.
>>>>
>>>> If our test suites keep getting longer and longer and take more than
>>>> the 20-25 minutes they currently take, I can definitely see how the commit
>>>> processor approach could add value.
>>>>
>>>> On the other hand, it has the downside that, sometimes even once a
>>>> commit is approved you might want to change something minor like rebasing
>>>> the commits manually, and automation might start getting in the way of
>>>> things like that.  Or, another example, sometimes there's only minor
>>>> changes requested and it doesn't warrant a re-review. Currently what we do
>>>> pretty often is just approve the PR, leave a note to change the wording
>>>> here and there, and the author can just merge it whenever they're done with
>>>> the minor changes.
>>>>
>>>> On Tue, Sep 29, 2020 at 12:49 PM David Davis 
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> Last week the discussion about how to merge PRs got me thinking about
>>>>> how we could potentially programmatically merge PRs. The openshift project
>>>>> on github does this[0] and I wondered if it would work for Pulp.
>>>>>
>>>>> I think the main benefits would be (1) we'd be able to code how to
>>>>> best merge PRs and (2) we'd no longer see PRs sitting around that are 
>>>>> ready
>>>>> to merge but not merged (granted this doesn't happen often).
>>>>>
>>>>> I created a PoC against the PR processor that's quite simple (here are
>>>>> the rele

Re: [Pulp-dev] PR merging

[Daniel Alley]

I have some doubts about the cost/benefit ratio of coding automation to
merge PRs vs. simply changing the default option / being selective about
which options are available.

I like the idea in general though.  A lot of projects do something like
this.  I occasionally contributed to the Servo project (RIP) and they used
automation to manage this.  The benefits were, their tests suites ran for
nearly 2 hours on 3 different operating systems, so the feedback loop was
quite slow, and automatically triggering the full test run every time a PR
was changed would be a horrendous waste of resources at the scale they were
running at.  So basically they would run a minimal CI within the PR, then
require a manual review, and once approved it would trigger the full CI,
and if that passed then it would automatically merge the PR hours later
whenever it finished.

If our test suites keep getting longer and longer and take more than the
20-25 minutes they currently take, I can definitely see how the commit
processor approach could add value.

On the other hand, it has the downside that, sometimes even once a commit
is approved you might want to change something minor like rebasing the
commits manually, and automation might start getting in the way of things
like that.  Or, another example, sometimes there's only minor changes
requested and it doesn't warrant a re-review. Currently what we do pretty
often is just approve the PR, leave a note to change the wording here and
there, and the author can just merge it whenever they're done with the
minor changes.

On Tue, Sep 29, 2020 at 12:49 PM David Davis  wrote:

> Hi all,
>
> Last week the discussion about how to merge PRs got me thinking about how
> we could potentially programmatically merge PRs. The openshift project on
> github does this[0] and I wondered if it would work for Pulp.
>
> I think the main benefits would be (1) we'd be able to code how to best
> merge PRs and (2) we'd no longer see PRs sitting around that are ready to
> merge but not merged (granted this doesn't happen often).
>
> I created a PoC against the PR processor that's quite simple (here are the
> relevant bits[1]). It's worth noting that we can still prevent PRs from
> being merged by setting them to drafts (I believe both the PR author or any
> repo owner do this) and the PR processor won't merge anything unless it's
> ready to be merged.
>
> I myself am a bit conflicted about this so feedback would be appreciated.
>
> [0] https://github.com/openshift
> [1]
> https://github.com/pulp/pulp-ci/pull/737/files#diff-cbdf61d38083f599940c37eeb49cb0a9R116-R141
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] github checklist as a part of the release process

[Daniel Alley]

I suggest that we re-use the pulp_packaging github repo since it has a
related purpose and already exists :)

Also, for what it's worth, HackMD supports markdown checklists as well.  I
lean more towards the github issue approach personally but it's worth
mentioning.

On Wed, Aug 19, 2020 at 11:03 AM David Davis  wrote:

> A separate github repo might make sense. Right now our release scripts
> live inside our .travis folders in repo. I don't know that they are project
> specific so perhaps we could move them to this new repo?
>
> David
>
>
> On Wed, Aug 19, 2020 at 5:57 AM Tatiana Tereshchenko 
> wrote:
>
>> Would a separate github repo with issues enabled make sense?
>> One place for all templates if we need many (I can think of at least Y
>> and Z releases).
>> One place for all release tracking, one can see what is released, and
>> what is not, without going from repo to repo (or from one redmine project
>> to another).
>> This repo can also have release compatibility information/table, or any
>> other release related data.
>>
>> I'm also not aware of any easy way of creating a template/checklist in
>> redmine.
>>
>> Tanya
>>
>> On Tue, Aug 18, 2020 at 4:22 PM David Davis 
>> wrote:
>>
>>> Big +1. I really like this idea and believe it could help us organize
>>> the work for releases.
>>>
>>> How we can apply this to Pulp though? We don't use github issues and
>>> there's no way to template checklists for redmine issues AFAICT.
>>>
>>> David
>>>
>>>
>>> On Tue, Aug 18, 2020 at 9:55 AM Fabricio Aguiar <
>>> fabricio.agu...@redhat.com> wrote:
>>>
 I like the idea,
 maybe it is possible to automate when closing the issue, triggering a
 github action

 Best regards,
 Fabricio Aguiar
 Software Engineer, Pulp Project
 Red Hat Brazil - Latam 
 +55 11 999652368


 On Tue, Aug 18, 2020 at 8:55 AM Tatiana Tereshchenko <
 ttere...@redhat.com> wrote:

> I learned recently how Fedora CoreOS folks do their releases and I
> really like their process.
> I think something similar can be useful for Pulp. We already have ~15
> steps in our release guide
>  and
> it's without some pre/post-release steps, like release announcement
> collaboration, writing blog posts, etc.
>
> The idea is simple.
> Have a checklist template (for each type of release if needed).
> Create a github issue with this checklist and mark it as you perform
> the steps.
> In addition post any relevant links as comments.
> Here is the example
> https://github.com/coreos/fedora-coreos-streams/issues/158
>
> Benefits:
>  - release progress is open and transparent to everyone, including our
> community
>  - it's easy to look at the history if needed
>  - release "guide" is always up to date
>  - if one started a release and can't finish for some reason (e.g. end
> of working day in their time zone), another one can take over
>  - keeps a release person more organized (those who released many
> times sometimes perform steps by memory and might forget some small steps;
> often people multitask and do something while waiting for the builds to be
> done. Our release guide serves the same purpose but one needs to
> consciously go back to it, here it requires you to click the checkbox.)
>
> Cons:
>   - a potential downside is that it's one more action to do and a new
> process to follow. Though it should be very close to the release guide, so
> I hope it does not add much to our processes, it should not feel like
> something new :)
>
> Thoughts?
>
> Tanya
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Meeting Minutes: Lockless pulp

[Daniel Alley]

I strongly support this, or at least strongly support having a discussion
about it.  I think we can't make this change for Pulp 3, but we should
absolutely consider moving in this direction for Pulp 4.

One (IMO large) "pro" which I didn't see mentioned here is auditability. It
is essentially impossible to know details like "what particular sync
options was a sync performed with" (because the remote settings can change,
you can't make assumptions about what it was at the time).  I can
definitely imagine that some users might strongly value that kind of
information and there's not really a good way to provide answers to those
questions under the current design.

So in addition to solving various difficult problems that can lead to
headaches for both ourselves and our users, there's also some potential for
creating new value here.

On Wed, Aug 12, 2020 at 11:25 AM Matthias Dellweg 
wrote:

> Today we met to brainstorm about the pulp design around locking and
> the resource manager.
> Here's the meeting transcript:
>
> # djnd (Lockless Pulp)
>
> # General Problem Statements
>
> - Resource manager is a problem.
> - It's a bottleneck. Every task goes through the resource manager.
> - When tasks die, inconsistent state rq vs Pulp
> - Pulp is inefficient. It's always FIFO when that may not be optimal.
> - Work has to wait if there's a reservation ahead of it waiting
> for another resource
> - Orphan cleanup blocks all work
> - Orphan cleanup might be handled separately by assuring that any
> resource the is to be clean up (currently content units and artifacts)
> is owned by at least one thing (RepositoryVersion, Task or User) at
> any time until not needed anymore
>
>
> # Why do we have mutable resources?
>
> Some things are immutable, e.g. content, but other things do, e.g. a
> Repository's data, e.g. it's name.
>
> * Mutable data creates write-write race conditions
> * Users have a first-come-first-serve (FCFS) expectation, e.g. a
> sync-then-publish
>
>
> # Why do we use locking today?
>
> * We use it to solve the base-path problem for Distributions
> * It provides the FCFS guarantee of work w.r.t a specific resource,
> e.g. Respotiory
> * Orphan cleanup is a singleton so it locks content units that are in
> use by other tasks when deleting
> * Deletion of resources is synchronized by locking
> * Updating of resources is synchronized by locking
> * Creation of RepositoryVersions is serialized by locking on the Repository
>
>
> # Solution
>
> ...getting out of the synchronization quadrant with immutability...
> Locks or bottlenecks are needed to prevent usage of resources by one
> process, while being changed by another.
> If all resources were immutable, they could be used by an unlimited
> number of processes simultaneously.
>
> Pros:
>   - no need for resource manager
>   - all (remaining) services are scalable
>
> Cons:
>   - Harder to design
>
>
> ## Exposed immutability
>
> *All* resources are immutable
>
> Pros:
>   - relatively easy to implement; no need to redesign the database
>   - blockless; no waiting on resources, ever (the user has to wait...)
>   - another user cannot modify ad resource you are about to use
>
> Cons:
>   - all burden pushed to the user
>   - changing a resource requires replacing it
>   - unable to change parameters behind a "name" (natural key)
>
> ## Futures (delayed immutability)
>
> Resources are created as futures and resolved by tasks.
> Once resolved, resources are immutable.
> Resources start in the unshared mutable corner and move to the shared
> immutable corner.
> Futures can readily be used to create new resources while not yet resolved.
>
> Pros:
>   - accounts for actions like sync, publish to take time
>   - futures form a DAG -> no possibility for dead locks
>   - still no waiting on finished resources
>
> Cons:
>   - waiting on resources; blocking
>   - futures can fail while waiting on failing dependent futures
>
> ## Copy-on-write with lookup table
>
> Resources are immutable, and referenced by a key lookup table.
> Changing a resource means creating a modified copy and changing the
> reference in the lookup table.
> Tasks reference the actual resources.
>
> Pros:
>   - users can "change" resources as they are used to
>   - blockless; no waiting on resources, ever (the user has to wait...)
>   - tasks can never fail on missing resources (being deleted in the
> meantime)
>
> Cons:
>   - extra table join on lookup
>   - impossible to retrieve the natural key
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] saftladen pulp's take on asciinema?

[Daniel Alley]

>
> Like a juice store whose shopman tells you no orange juice today, next
> delivery in two weeks.
>

Reminds me of this Monty Python sketch

https://www.youtube.com/watch?v=Hz1JWzyvv8A

On Thu, Aug 6, 2020 at 3:24 AM Matthias Dellweg  wrote:

>
>
> On Wed, Aug 5, 2020 at 8:02 PM Robin Chan  wrote:
>
>> I looked it up and according to the internets saftladen means juice store?
>>
> In the literal translation...  It has a second meaning, where you use it
> to paraphrase an organization in a slightly disrespectful manner that has
> disappointed you.
> Like a juice store whose shopman tells you no orange juice today, next
> delivery in two weeks.
> This very repository contains a bunch of tools hacked together in an
> afternoon. So it will either do exactly what you need, or it will frustrate
> you badly.
>
>> Could be a nice easter egg for the readme. :-)
>> +1 to Ina's suggestions those are great ideas.
>>
>> Robin Chan
>>
>> She/Her/Hers
>>
>> Satellite Software Engineering Manager - Pulp
>>
>> Red Hat 
>>
>> IRC: rchan
>>
>> Red Hat respects your work life balance. Therefore there is no need to
>> answer this email out of your office hours.
>> 
>>
>>
>>
>> On Wed, Aug 5, 2020 at 6:22 AM Ina Panova  wrote:
>>
>>> This is great, +1 to the automation business.
>>> Even though shell scripts are quite self explanatory, could you write
>>> some Readme?
>>> We could eventually add this to our Wiki
>>> https://pulp.plan.io/projects/pulp/wiki/Demo_Presenter_Notes#Recording-a-Video
>>>
>>>
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Senior Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>>
>>>
>>> On Thu, Jul 30, 2020 at 5:38 PM Matthias Dellweg 
>>> wrote:
>>>
 Hi list,
 as much as animations help our documentation to be appealing, as
 little appealing it sounded to me to record a demo over and over until
 i liked it. But what can i say: We are in the automation business. So
 this adventure ended in a set of tools to generate an asciinema
 demonstration from a rather simple shellscript, wrapped in a Makefile
 to be able to adjust and rerecord it over and over. Let's hope i can
 encourage you all to add similar demos to that repository:

 https://github.com/mdellweg/saftladen

 Cheers, Matthias

 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

 ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulp-owned pypi packages that pulp did not author

[Daniel Alley]

 The patch is fairly small, but you're definitely not wrong about that. I'm
just not sure if there's better options unfortunately.

On Wed, Jul 15, 2020 at 6:37 AM Evgeni Golov  wrote:

> Thanks for all the explanations!
>
> I think my main concerns are the fact that we (well, really, you) end up
> having to maintain this libsolv "fork" essentially forever (in contrast to
> the others, where you currently just need to upload the releases) and users
> get a slightly different result depending how they install libsolv --
> especially if used in other projects.
>
> On Tue, Jul 14, 2020 at 5:22 PM Daniel Alley  wrote:
>
>> Thanks Daniel,
>>>
>>> I've reposted to pulp-dev, so you might want to re-post your reply
>>> there too, but:
>>>
>>> Great, if createrepo and libcomps is just "intermediate" while we
>>> actively (try) to help upstream get there, I can totally take that.
>>>
>>> For solv: well, if they don't want it there, IMHO we should not
>>> publish it under their name either?
>>
>>
>> It's not so much that they didn't want it there; more that they don't
>> care about that use case at all and it's not something that they want to
>> maintain or think about.  Similar to the situation with createrepo_c except
>> that there was less hope that they would eventually take back control :)
>>
>> The primary reason is just that we wanted it to be possible to run the
>> RPM plugin on Debian-based distros, and relying on any RPMs would go
>> counter to that goal.  But there were some other motivations also - at the
>> time, the Python community was deciding on which technology to use for the
>> new dependency resolver in Pip, and libsolv was mentioned as a candidate
>> (but ultimately decided against for various reasons).  One of the reasons
>> was that nobody had ever made a Python package for libsolv and they weren't
>> even sure if it was possible - and I found this discussion at the same time
>> as I was already wanting to package it anyway for Pulp.
>>
>>>
>>>
>> On Tue, Jul 14, 2020 at 9:47 AM Daniel Alley  wrote:
>>
>>> My apologies!  s/Evengi/Evgeni, the letters got swapped in my brain : /
>>>
>>> On Tue, Jul 14, 2020 at 9:37 AM Daniel Alley  wrote:
>>>
>>>> Reposting my response from the other thread:
>>>>
>>>> Hi Evengi,
>>>>
>>>> In the case of createrepo_c and libsolv, the upstream merged all of the
>>>> build script changes that were necessary to enable producing Python
>>>> packages, so in that sense the packages we are producing are completely
>>>> unmodified.  However, the RPM team isn't particularly interested in
>>>> maintaining Python packages, and so they gave us permission to maintain the
>>>> packages ourselves.  I've forgotten in what medium that discussion took
>>>> place so I'm not sure where even to look for a record of it.  Nonetheless I
>>>> actually have a PR open against both projects which would automate the
>>>> entire packaging and release process for Python packages with Github
>>>> Actions, so that they could become the official owners/maintainers without
>>>> actually needing to do any work (hopefully).
>>>>
>>>> https://github.com/rpm-software-management/createrepo_c/pull/207
>>>> https://github.com/rpm-software-management/libcomps/pull/69
>>>>
>>>> However you may notice that those PRs have been sitting for a while :)
>>>> In any case we'd definitely love to transfer ownership back to them, and
>>>> I've been trying to facilitate that process a little bit (with the PRs) but
>>>> I don't really want to push on them too hard to do so.
>>>>
>>>> With respect to libsolv it's a little more complicated. As far as I can
>>>> tell the upstream is just not interested at all and would probably not
>>>> accept the changes into upstream regardless of whether we made the release
>>>> process automated.  I asked multiple times if they were interested and got
>>>> essentially no response.
>>>>
>>>> https://github.com/openSUSE/libsolv/issues/228
>>>>
>>>> + some discussion on IRC
>>>>
>>>> Feb 24 10:14:47  hello Igor, do you have any opinion on this?
>>>>> https://github.com/openSUSE/libsolv/issues/228#issuecomment-589915584
>>>>> Feb 24 10:18:24  Well, I don't see any reason to
>>>>> publish libsolv to pypi :)
>>>>&g

Re: [Pulp-dev] pulp-owned pypi packages that pulp did not author

[Daniel Alley]

>
> Thanks Daniel,
>
> I've reposted to pulp-dev, so you might want to re-post your reply
> there too, but:
>
> Great, if createrepo and libcomps is just "intermediate" while we
> actively (try) to help upstream get there, I can totally take that.
>
> For solv: well, if they don't want it there, IMHO we should not
> publish it under their name either?


It's not so much that they didn't want it there; more that they don't care
about that use case at all and it's not something that they want to
maintain or think about.  Similar to the situation with createrepo_c except
that there was less hope that they would eventually take back control :)

The primary reason is just that we wanted it to be possible to run the RPM
plugin on Debian-based distros, and relying on any RPMs would go counter to
that goal.  But there were some other motivations also - at the time, the
Python community was deciding on which technology to use for the new
dependency resolver in Pip, and libsolv was mentioned as a candidate (but
ultimately decided against for various reasons).  One of the reasons was
that nobody had ever made a Python package for libsolv and they weren't
even sure if it was possible - and I found this discussion at the same time
as I was already wanting to package it anyway for Pulp.

>
>
On Tue, Jul 14, 2020 at 9:47 AM Daniel Alley  wrote:

> My apologies!  s/Evengi/Evgeni, the letters got swapped in my brain : /
>
> On Tue, Jul 14, 2020 at 9:37 AM Daniel Alley  wrote:
>
>> Reposting my response from the other thread:
>>
>> Hi Evengi,
>>
>> In the case of createrepo_c and libsolv, the upstream merged all of the
>> build script changes that were necessary to enable producing Python
>> packages, so in that sense the packages we are producing are completely
>> unmodified.  However, the RPM team isn't particularly interested in
>> maintaining Python packages, and so they gave us permission to maintain the
>> packages ourselves.  I've forgotten in what medium that discussion took
>> place so I'm not sure where even to look for a record of it.  Nonetheless I
>> actually have a PR open against both projects which would automate the
>> entire packaging and release process for Python packages with Github
>> Actions, so that they could become the official owners/maintainers without
>> actually needing to do any work (hopefully).
>>
>> https://github.com/rpm-software-management/createrepo_c/pull/207
>> https://github.com/rpm-software-management/libcomps/pull/69
>>
>> However you may notice that those PRs have been sitting for a while :)
>> In any case we'd definitely love to transfer ownership back to them, and
>> I've been trying to facilitate that process a little bit (with the PRs) but
>> I don't really want to push on them too hard to do so.
>>
>> With respect to libsolv it's a little more complicated. As far as I can
>> tell the upstream is just not interested at all and would probably not
>> accept the changes into upstream regardless of whether we made the release
>> process automated.  I asked multiple times if they were interested and got
>> essentially no response.
>>
>> https://github.com/openSUSE/libsolv/issues/228
>>
>> + some discussion on IRC
>>
>> Feb 24 10:14:47  hello Igor, do you have any opinion on this?
>>> https://github.com/openSUSE/libsolv/issues/228#issuecomment-589915584
>>> Feb 24 10:18:24  Well, I don't see any reason to publish
>>> libsolv to pypi :)
>>>
>>
>> On Tue, Jul 14, 2020 at 9:31 AM Evgeni Golov  wrote:
>>
>>> Hi pulp-dev,
>>>
>>> While packaging pulp3 (more precisely pulp-rpm), I stumbled over the
>>> fact that the "pulp" pypi user has uploaded "solv", "libcomps" and
>>> "createrepo-c" without being the real author. To make matters worse,
>>> the uploads don't 100% represent the original artifacts released by
>>> the respective upstreams as they don't release python packages but
>>> classic tarballs. In the case of "solv" this lead to an interesting
>>> bug: solv upstream does not build a python egg, but your package did,
>>> and then as the pulp-rpm egg has "solv" as a dependency, it won't load on
>>> a system that uses the "real solv" without the egg. We patched that
>>> out in packaging, but it remains ugly.
>>>
>>> I kinda understand why Pulp did that, this way you can rely on "pip"
>>> to install everything for a working pulp-rpm environment, but I think
>>> we/you shouldn't do that and instead either persuade (and help!) the real
>>

Re: [Pulp-dev] pulp-owned pypi packages that pulp did not author

[Daniel Alley]

My apologies!  s/Evengi/Evgeni, the letters got swapped in my brain : /

On Tue, Jul 14, 2020 at 9:37 AM Daniel Alley  wrote:

> Reposting my response from the other thread:
>
> Hi Evengi,
>
> In the case of createrepo_c and libsolv, the upstream merged all of the
> build script changes that were necessary to enable producing Python
> packages, so in that sense the packages we are producing are completely
> unmodified.  However, the RPM team isn't particularly interested in
> maintaining Python packages, and so they gave us permission to maintain the
> packages ourselves.  I've forgotten in what medium that discussion took
> place so I'm not sure where even to look for a record of it.  Nonetheless I
> actually have a PR open against both projects which would automate the
> entire packaging and release process for Python packages with Github
> Actions, so that they could become the official owners/maintainers without
> actually needing to do any work (hopefully).
>
> https://github.com/rpm-software-management/createrepo_c/pull/207
> https://github.com/rpm-software-management/libcomps/pull/69
>
> However you may notice that those PRs have been sitting for a while :)  In
> any case we'd definitely love to transfer ownership back to them, and I've
> been trying to facilitate that process a little bit (with the PRs) but I
> don't really want to push on them too hard to do so.
>
> With respect to libsolv it's a little more complicated. As far as I can
> tell the upstream is just not interested at all and would probably not
> accept the changes into upstream regardless of whether we made the release
> process automated.  I asked multiple times if they were interested and got
> essentially no response.
>
> https://github.com/openSUSE/libsolv/issues/228
>
> + some discussion on IRC
>
> Feb 24 10:14:47  hello Igor, do you have any opinion on this?
>> https://github.com/openSUSE/libsolv/issues/228#issuecomment-589915584
>> Feb 24 10:18:24  Well, I don't see any reason to publish
>> libsolv to pypi :)
>>
>
> On Tue, Jul 14, 2020 at 9:31 AM Evgeni Golov  wrote:
>
>> Hi pulp-dev,
>>
>> While packaging pulp3 (more precisely pulp-rpm), I stumbled over the
>> fact that the "pulp" pypi user has uploaded "solv", "libcomps" and
>> "createrepo-c" without being the real author. To make matters worse,
>> the uploads don't 100% represent the original artifacts released by
>> the respective upstreams as they don't release python packages but
>> classic tarballs. In the case of "solv" this lead to an interesting
>> bug: solv upstream does not build a python egg, but your package did,
>> and then as the pulp-rpm egg has "solv" as a dependency, it won't load on
>> a system that uses the "real solv" without the egg. We patched that
>> out in packaging, but it remains ugly.
>>
>> I kinda understand why Pulp did that, this way you can rely on "pip"
>> to install everything for a working pulp-rpm environment, but I think
>> we/you shouldn't do that and instead either persuade (and help!) the real
>> upstreams to publish their stuff to PyPI or bite the bullet and accept
>> that pip is not able to install everything needed for a working
>> environment.
>>
>> Thanks!
>> Evgeni
>>
>> --
>> Beste Grüße/Kind regards,
>>
>> Evgeni Golov
>> Senior Software Engineer
>> 
>> Red Hat GmbH, http://www.de.redhat.com/, Sitz: Grasbrunn,
>> Handelsregister: Amtsgericht München, HRB 153243,
>> Geschäftsführer: Charles Cachera, Laurie Krebs, Michael O'Neill, Thomas
>> Savage
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulp-owned pypi packages that pulp did not author

[Daniel Alley]

Reposting my response from the other thread:

Hi Evengi,

In the case of createrepo_c and libsolv, the upstream merged all of the
build script changes that were necessary to enable producing Python
packages, so in that sense the packages we are producing are completely
unmodified.  However, the RPM team isn't particularly interested in
maintaining Python packages, and so they gave us permission to maintain the
packages ourselves.  I've forgotten in what medium that discussion took
place so I'm not sure where even to look for a record of it.  Nonetheless I
actually have a PR open against both projects which would automate the
entire packaging and release process for Python packages with Github
Actions, so that they could become the official owners/maintainers without
actually needing to do any work (hopefully).

https://github.com/rpm-software-management/createrepo_c/pull/207
https://github.com/rpm-software-management/libcomps/pull/69

However you may notice that those PRs have been sitting for a while :)  In
any case we'd definitely love to transfer ownership back to them, and I've
been trying to facilitate that process a little bit (with the PRs) but I
don't really want to push on them too hard to do so.

With respect to libsolv it's a little more complicated. As far as I can
tell the upstream is just not interested at all and would probably not
accept the changes into upstream regardless of whether we made the release
process automated.  I asked multiple times if they were interested and got
essentially no response.

https://github.com/openSUSE/libsolv/issues/228

+ some discussion on IRC

Feb 24 10:14:47  hello Igor, do you have any opinion on this?
> https://github.com/openSUSE/libsolv/issues/228#issuecomment-589915584
> Feb 24 10:18:24  Well, I don't see any reason to publish
> libsolv to pypi :)
>

On Tue, Jul 14, 2020 at 9:31 AM Evgeni Golov  wrote:

> Hi pulp-dev,
>
> While packaging pulp3 (more precisely pulp-rpm), I stumbled over the
> fact that the "pulp" pypi user has uploaded "solv", "libcomps" and
> "createrepo-c" without being the real author. To make matters worse,
> the uploads don't 100% represent the original artifacts released by
> the respective upstreams as they don't release python packages but
> classic tarballs. In the case of "solv" this lead to an interesting
> bug: solv upstream does not build a python egg, but your package did,
> and then as the pulp-rpm egg has "solv" as a dependency, it won't load on
> a system that uses the "real solv" without the egg. We patched that
> out in packaging, but it remains ugly.
>
> I kinda understand why Pulp did that, this way you can rely on "pip"
> to install everything for a working pulp-rpm environment, but I think
> we/you shouldn't do that and instead either persuade (and help!) the real
> upstreams to publish their stuff to PyPI or bite the bullet and accept
> that pip is not able to install everything needed for a working
> environment.
>
> Thanks!
> Evgeni
>
> --
> Beste Grüße/Kind regards,
>
> Evgeni Golov
> Senior Software Engineer
> 
> Red Hat GmbH, http://www.de.redhat.com/, Sitz: Grasbrunn,
> Handelsregister: Amtsgericht München, HRB 153243,
> Geschäftsführer: Charles Cachera, Laurie Krebs, Michael O'Neill, Thomas
> Savage
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Name uniqueness problem in Pulp 3 REST API

[Daniel Alley]

Why would it cause pain (to relax the restriction)?

On Tue, Jun 30, 2020 at 4:46 PM Grant Gainey  wrote:

> On Tue, Jun 30, 2020 at 3:54 PM Dennis Kliban  wrote:
>
>> The REST APIs for repositories in Pulp 3 do not allow a name to be reused
>> between repository types. e.g.: A File Repository with name 'test' prevents
>> a user from creating an RPM Repository with name 'test', even though these
>> are two separate APIs.
>>
>> Do you agree that the namespace of one API should not interfere with the
>> namespace of another API?
>>
>
> Oh...ow. Yes, concur - but that's going to cause some pain, no?
>
> G
> --
> Grant Gainey
> Principal Software Engineer, Red Hat System Management Engineering
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] the "relative path" problem

[Daniel Alley]

Cool, so the only difference is whether to try to store the relationship in
the DB, or leverage the fact that we already have the metadata and just
re-parse it.

I know the latter approach has yet to be written up, but my concern there
is that adding another layer of indirection between "repository version"
and "content" is going to have an adverse impact on performance, since it
is already the most complex and demanding query we issue to the DB and one
of the most common and important.

On Thu, Apr 30, 2020 at 12:50 PM David Davis  wrote:

> Yes but I was imagining the mapping would be stored not as Content but as
> a separate object. So we wouldn't use filename for the mapping (rather we'd
> use ContentArtifact pk) and  we wouldn't need to change ContentArtifact's
> relative_path at all. That said, I think your solution captures the idea
> though and is better in some ways.
>
> Changing the RepositoryContent model to point to ContentArtifacts and
> store relative_paths is probably the best and most correct solution in
> theory. However, it's going to be painful to implement for both core and
> plugins.
>
> David
>
>
> On Thu, Apr 30, 2020 at 12:33 PM Daniel Alley  wrote:
>
>> @David Davis   so this proposal would go
>> something like this, correct?:
>>
>> * For the signed metadata / exact mirror use-case we need to store the
>> repository metadata itself as a content unit inside the RepositoryVersion
>> anyway (because the hash must be equal)
>> * Because we have this metadata lying around, we can reference it at
>> publish time to discover the appropriate PublishedArtifact.relative_path
>>* Create a map of "filename" -> "location_href" and look up the
>> filename of each RPM package to find the appropriate path
>>* This should be pretty fast for the RPM plugin since createrepo_c is
>> doing all the hard work
>> * Data migration to ensure ContentArtifact.relative_path is only storing
>> the filename (and I would suggest we also change the name to "filename")
>> * If metadata isn't present in the RepositoryVersion, then just tweak the
>> PublishedArtifact.relative_path so that it uses whichever our default repo
>> layout is
>>
>> On Tue, Apr 28, 2020 at 11:41 AM David Davis 
>> wrote:
>>
>>> Yes, that's correct. During our meeting we discussed two options: the
>>> first was to extend RepositoryContent to store relative path per
>>> ContentArtifact as storing a relative_path per Content won't work for
>>> multi-Artifact Content units.
>>>
>>> An alternative that I pitched was to have plugins (or maybe even core
>>> someday) store this information outside RepositoryContent and then use this
>>> information during publishing to set relative_path on PublishedArtifacts.
>>> We'd have to modify the content app if we wanted to support pass through
>>> publications but I think asking plugins to use published artifacts in this
>>> case is warranted. That said, I don't think anyone else was keen on this
>>> idea though.
>>>
>>> David
>>>
>>>
>>> On Tue, Apr 28, 2020 at 10:30 AM Matthias Dellweg 
>>> wrote:
>>>
>>>> That is only used for passthrough publication afaik. If you publish
>>>> each content unit "by hand", you create a new relative path for each
>>>> published artifact. That is, why it can be empty and still the content can
>>>> be published.
>>>>
>>>> On Tue, Apr 28, 2020 at 4:09 PM Daniel Alley  wrote:
>>>>
>>>>> We realized in our discussion that the original proposal described in
>>>>> my email will not work, because "relative_path" ultimately describes the
>>>>> path of the published *artifacts* (not content), and for content
>>>>> types with multiple artifacts, storing this information in a field on
>>>>> RepositoryContent would not be possible.
>>>>>
>>>>> On Mon, Apr 27, 2020 at 6:08 PM Daniel Alley 
>>>>> wrote:
>>>>>
>>>>>> There is a video call scheduled to discuss this issue tomorrow
>>>>>> (Tuesday April 28th) at 13:30 UTC (please convert to your local time).
>>>>>> https://meet.google.com/scy-csbx-qiu
>>>>>>
>>>>>> On Sat, Apr 25, 2020 at 7:02 AM David Davis 
>>>>>> wrote:
>>>>>>
>>>>>>> I had a chance to think about this some more yesterday and wanted to
>>>>>>> em

Re: [Pulp-dev] the "relative path" problem

[Daniel Alley]

@David Davis   so this proposal would go something
like this, correct?:

* For the signed metadata / exact mirror use-case we need to store the
repository metadata itself as a content unit inside the RepositoryVersion
anyway (because the hash must be equal)
* Because we have this metadata lying around, we can reference it at
publish time to discover the appropriate PublishedArtifact.relative_path
   * Create a map of "filename" -> "location_href" and look up the filename
of each RPM package to find the appropriate path
   * This should be pretty fast for the RPM plugin since createrepo_c is
doing all the hard work
* Data migration to ensure ContentArtifact.relative_path is only storing
the filename (and I would suggest we also change the name to "filename")
* If metadata isn't present in the RepositoryVersion, then just tweak the
PublishedArtifact.relative_path so that it uses whichever our default repo
layout is

On Tue, Apr 28, 2020 at 11:41 AM David Davis  wrote:

> Yes, that's correct. During our meeting we discussed two options: the
> first was to extend RepositoryContent to store relative path per
> ContentArtifact as storing a relative_path per Content won't work for
> multi-Artifact Content units.
>
> An alternative that I pitched was to have plugins (or maybe even core
> someday) store this information outside RepositoryContent and then use this
> information during publishing to set relative_path on PublishedArtifacts.
> We'd have to modify the content app if we wanted to support pass through
> publications but I think asking plugins to use published artifacts in this
> case is warranted. That said, I don't think anyone else was keen on this
> idea though.
>
> David
>
>
> On Tue, Apr 28, 2020 at 10:30 AM Matthias Dellweg 
> wrote:
>
>> That is only used for passthrough publication afaik. If you publish each
>> content unit "by hand", you create a new relative path for each published
>> artifact. That is, why it can be empty and still the content can be
>> published.
>>
>> On Tue, Apr 28, 2020 at 4:09 PM Daniel Alley  wrote:
>>
>>> We realized in our discussion that the original proposal described in my
>>> email will not work, because "relative_path" ultimately describes the path
>>> of the published *artifacts* (not content), and for content types with
>>> multiple artifacts, storing this information in a field on
>>> RepositoryContent would not be possible.
>>>
>>> On Mon, Apr 27, 2020 at 6:08 PM Daniel Alley  wrote:
>>>
>>>> There is a video call scheduled to discuss this issue tomorrow (Tuesday
>>>> April 28th) at 13:30 UTC (please convert to your local time).
>>>> https://meet.google.com/scy-csbx-qiu
>>>>
>>>> On Sat, Apr 25, 2020 at 7:02 AM David Davis 
>>>> wrote:
>>>>
>>>>> I had a chance to think about this some more yesterday and wanted to
>>>>> email out my thoughts. I also think that this change sounds scary and will
>>>>> have a big impact on plugin writers so I thought of a couple alternatives:
>>>>>
>>>>> First, we could add a relative_path field to RepositoryContent instead
>>>>> of moving it there. This would be an optional field. It would be up to
>>>>> plugins to manage this field and they would still need to populate the
>>>>> relative_path field on ContentArtifact. But plugins could use this 
>>>>> optional
>>>>> field to store relative paths per repository and then use this field when
>>>>> generating publications.
>>>>>
>>>>> The second alternative is one that is already laid out in the original
>>>>> email but to call it out again: it would be to not solve this in pulpcore.
>>>>> RPM would create its own object that would map content in a repository to
>>>>> relative_paths.
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>> On Tue, Apr 21, 2020 at 9:22 AM Quirin Pamp  wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>> I am not currently very well versed in the classes involved, but
>>>>>> moving relative_path around sounds slightly scary with the potential to
>>>>>> break things.
>>>>>>
>>>>>>
>>>>>> As such, I would be interested to be kept in the loop as this moves
>>>>>> forward. (Mailing list once there is some movement is entirely sufficient
>>>>>> )
&g

Re: [Pulp-dev] the "relative path" problem

[Daniel Alley]

We realized in our discussion that the original proposal described in my
email will not work, because "relative_path" ultimately describes the path
of the published *artifacts* (not content), and for content types with
multiple artifacts, storing this information in a field on
RepositoryContent would not be possible.

On Mon, Apr 27, 2020 at 6:08 PM Daniel Alley  wrote:

> There is a video call scheduled to discuss this issue tomorrow (Tuesday
> April 28th) at 13:30 UTC (please convert to your local time).
> https://meet.google.com/scy-csbx-qiu
>
> On Sat, Apr 25, 2020 at 7:02 AM David Davis  wrote:
>
>> I had a chance to think about this some more yesterday and wanted to
>> email out my thoughts. I also think that this change sounds scary and will
>> have a big impact on plugin writers so I thought of a couple alternatives:
>>
>> First, we could add a relative_path field to RepositoryContent instead of
>> moving it there. This would be an optional field. It would be up to plugins
>> to manage this field and they would still need to populate the
>> relative_path field on ContentArtifact. But plugins could use this optional
>> field to store relative paths per repository and then use this field when
>> generating publications.
>>
>> The second alternative is one that is already laid out in the original
>> email but to call it out again: it would be to not solve this in pulpcore.
>> RPM would create its own object that would map content in a repository to
>> relative_paths.
>>
>> David
>>
>>
>> On Tue, Apr 21, 2020 at 9:22 AM Quirin Pamp  wrote:
>>
>>> Hi,
>>>
>>>
>>> I am not currently very well versed in the classes involved, but moving
>>> relative_path around sounds slightly scary with the potential to break
>>> things.
>>>
>>>
>>> As such, I would be interested to be kept in the loop as this moves
>>> forward. (Mailing list once there is some movement is entirely sufficient
>>> )
>>>
>>>
>>> Thanks,
>>>
>>> Quirin Pamp
>>> --
>>> *From:* pulp-dev-boun...@redhat.com  on
>>> behalf of Ina Panova 
>>> *Sent:* 21 April 2020 14:07:13
>>> *To:* Daniel Alley 
>>> *Cc:* Pulp-dev 
>>> *Subject:* Re: [Pulp-dev] the "relative path" problem
>>>
>>> Daniel,
>>>
>>> how about setting up a meeting and brainstorm the alternatives,
>>> pros/cons there?
>>>
>>>
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Senior Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>>
>>>
>>> On Fri, Apr 17, 2020 at 5:57 PM Daniel Alley  wrote:
>>>
>>> Bump, this item needs to move forwards soon.  Does anyone have any
>>> thoughts?
>>>
>>> On Wed, Apr 1, 2020 at 9:40 AM Pavel Picka  wrote:
>>>
>>> Hi,
>>> I'd like to add one more question to this topic. Do you think it is a
>>> blocker for PRs [0] & [1] as by testing [2] this features I haven't run
>>> into real world example where two really same name packages appears.
>>> I think this is a 'must have' feature but until we solve/decide it we
>>> can have two features working may with warning in docs for users that can
>>> happen in some 'special' repositories.
>>>
>>> To follow topic directly I like proposed move to 'RepositoryContent' and
>>> add it to its uniqueness constraint (if I understand well).
>>>
>>> [0] https://github.com/pulp/pulp_rpm/pull/1657
>>> [1] https://github.com/pulp/pulp_rpm/pull/1642
>>> [2] tested with centos 7, 8, opensuse and SLE repositories
>>>
>>> On Wed, Apr 1, 2020 at 3:22 PM Daniel Alley  wrote:
>>>
>>> We'd like to start a discussion on the "relative path problem"
>>> identified recently.
>>> Problem:
>>>
>>> Currently, a relative_path is tied to content in Pulp. This means that
>>> if a content unit exists in two places within a repository or across
>>> repositories, it has to be stored as two separate content units. This
>>> creates redundant data and potential confusion for users.
>>>
>>> As a specific example, we need to support mirroring content in pulp_rpm
>>> <https://pulp.plan.io/issues/6353>. Currently, for each location at
>>> which a single packa

Re: [Pulp-dev] the "relative path" problem

[Daniel Alley]

There is a video call scheduled to discuss this issue tomorrow (Tuesday
April 28th) at 13:30 UTC (please convert to your local time).
https://meet.google.com/scy-csbx-qiu

On Sat, Apr 25, 2020 at 7:02 AM David Davis  wrote:

> I had a chance to think about this some more yesterday and wanted to email
> out my thoughts. I also think that this change sounds scary and will have a
> big impact on plugin writers so I thought of a couple alternatives:
>
> First, we could add a relative_path field to RepositoryContent instead of
> moving it there. This would be an optional field. It would be up to plugins
> to manage this field and they would still need to populate the
> relative_path field on ContentArtifact. But plugins could use this optional
> field to store relative paths per repository and then use this field when
> generating publications.
>
> The second alternative is one that is already laid out in the original
> email but to call it out again: it would be to not solve this in pulpcore.
> RPM would create its own object that would map content in a repository to
> relative_paths.
>
> David
>
>
> On Tue, Apr 21, 2020 at 9:22 AM Quirin Pamp  wrote:
>
>> Hi,
>>
>>
>> I am not currently very well versed in the classes involved, but moving
>> relative_path around sounds slightly scary with the potential to break
>> things.
>>
>>
>> As such, I would be interested to be kept in the loop as this moves
>> forward. (Mailing list once there is some movement is entirely sufficient
>> )
>>
>>
>> Thanks,
>>
>> Quirin Pamp
>> --
>> *From:* pulp-dev-boun...@redhat.com  on
>> behalf of Ina Panova 
>> *Sent:* 21 April 2020 14:07:13
>> *To:* Daniel Alley 
>> *Cc:* Pulp-dev 
>> *Subject:* Re: [Pulp-dev] the "relative path" problem
>>
>> Daniel,
>>
>> how about setting up a meeting and brainstorm the alternatives, pros/cons
>> there?
>>
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Senior Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Fri, Apr 17, 2020 at 5:57 PM Daniel Alley  wrote:
>>
>> Bump, this item needs to move forwards soon.  Does anyone have any
>> thoughts?
>>
>> On Wed, Apr 1, 2020 at 9:40 AM Pavel Picka  wrote:
>>
>> Hi,
>> I'd like to add one more question to this topic. Do you think it is a
>> blocker for PRs [0] & [1] as by testing [2] this features I haven't run
>> into real world example where two really same name packages appears.
>> I think this is a 'must have' feature but until we solve/decide it we can
>> have two features working may with warning in docs for users that can
>> happen in some 'special' repositories.
>>
>> To follow topic directly I like proposed move to 'RepositoryContent' and
>> add it to its uniqueness constraint (if I understand well).
>>
>> [0] https://github.com/pulp/pulp_rpm/pull/1657
>> [1] https://github.com/pulp/pulp_rpm/pull/1642
>> [2] tested with centos 7, 8, opensuse and SLE repositories
>>
>> On Wed, Apr 1, 2020 at 3:22 PM Daniel Alley  wrote:
>>
>> We'd like to start a discussion on the "relative path problem" identified
>> recently.
>> Problem:
>>
>> Currently, a relative_path is tied to content in Pulp. This means that if
>> a content unit exists in two places within a repository or across
>> repositories, it has to be stored as two separate content units. This
>> creates redundant data and potential confusion for users.
>>
>> As a specific example, we need to support mirroring content in pulp_rpm
>> <https://pulp.plan.io/issues/6353>. Currently, for each location at
>> which a single package is stored, we’ll need to create a content unit. We
>> could end up with several records representing a single package. Users may
>> be confused about why they see multiple records for a package and they may
>> have trouble for example deciding which content unit to copy.
>> Proposed Solution:
>>
>> Move “relative_path” from its current location on ContentArtifact, to
>> RepositoryContent. This will require a sizable data migration. It is
>> possibly the case that in rare cases, repository versions may change
>> slightly due to deduplication.
>>
>> A repository-version-wide uniqueness constraint will be present on
>> “relative_path”, independently of any other repository uniquness
>> constraints (repo_key_fields) defined by 

Re: [Pulp-dev] pulp_file 1.0

[Daniel Alley]

+1 to 1.0

On Mon, Apr 27, 2020 at 5:57 AM Ina Panova  wrote:

> Based on the extended reply from David referring to semver, I am in favour
> or releasing pulp_file 1.0.
>
> Also, comments inline.
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Wed, Apr 22, 2020 at 7:02 PM Brian Bouterse 
> wrote:
>
>> tl;dr we follow semver.org and I agree with your reasoning, so I'm
>> convinced 1.0 would be fine. While I'm not in favor of the change, I'm
>> ready to disagree and commit.
>>
>> In the interests of sharing perspectives, here's mine. The issue with
>> semver.org is that it's exclusively focused on change management, and it
>> ignores what I perceive as a cultural association with > 1.0 software to
>> mean "broadly tested and low risk". Is pulp_file at a point where backwards
>> compatibility is a primary concern and prohibited yes. Do the developers of
>> pulp_file recommend it to be run in production, yes. As a user, is it a low
>> risk software due to many folks having already deployed it in production,
>> no. In fact pulp_file is maybe in the high to medium risk category based on
>> the number of folks who are actually running it in production.
>>
>
> Brian, this is a kind of chicken and the egg problem. Let's be fair and
> answer - how many folks will deploy something that is 0.y.z and not
> production ready?
> Not a lot of folks will deploy it in the production unless we release and
> say - this is stable enough for production use. Only after that we will
> have enough numbers to fairly say if it is low/high/medium risk software.
>
>
>> Having said all that, I'm ready to support your proposal on the semver
>> basis. Your reasoning is sound. Thank you for writing your thoughts here
>> and your effort to make it great.
>>
>> On Wed, Apr 22, 2020 at 11:32 AM David Davis 
>> wrote:
>>
>>> I want to expound on my own reasoning behind why pulp_file should be
>>> bumped to 1.0 because I realize my original email was probably too brief
>>> and I apologize for that.
>>>
>>> The thing that I would refer to is semver.org which we've used as a
>>> guide for versioning. semver.org defines a 0.Y release as:
>>>
>>>Major version zero (0.y.z) is for initial development. Anything MAY
>>> change at any time. The public API SHOULD NOT be considered stable.
>>>
>>> Moreover, semver.org has this question/answer:
>>>
>>> How do I know when to release 1.0.0?
>>>
>>> If your software is being used in production, it should probably
>>> already be 1.0.0. If you have a stable API on which users have come to
>>> depend, you should be 1.0.0. If you’re worrying a lot about backwards
>>> compatibility, you should probably already be 1.0.0.
>>>
>>> I think we meet both of these criteria. I expect that Pulp users are
>>> probably using pulp_file in production already. In terms of its API, we've
>>> had only two small features in the last couple releases of pulp_file since
>>> 0.1.0[0] and no major changes to the public API (there was the rename of
>>> one field). I don't foresee any major changes to the public api anytime
>>> soon. There's not a roadmap for new features either and certainly nothing I
>>> see that could cause major changes to pulp_file's API.
>>>
>>> I think that in this context it makes sense to bump it to 1.0 to
>>> communicate to our users that the pulp_file API is stable enough to use in
>>> production.
>>>
>>> Thoughts?
>>>
>>> [0] https://github.com/pulp/pulp_file/blob/master/CHANGES.rst
>>>
>>> David
>>>
>>>
>>> On Wed, Apr 22, 2020 at 10:59 AM David Davis 
>>> wrote:
>>>
 I feel differently especially when considering that most other Pulp
 plugins are at > 1.0. Can you explain why you think pulp_file shouldn't be
 at 1.0?

 David


 On Wed, Apr 22, 2020 at 10:57 AM Brian Bouterse 
 wrote:

> I've seen software live in the < 1.0 area for a long time and graduate
> when it's in broad, production use. That's a difficult thing to assess
> accurately, but to me, pulp_file hasn't reached that point.
>
>
> On Tue, Apr 21, 2020 at 2:20 PM David Davis 
> wrote:
>
>> With the next release of pulp_file, I'd propose we bump the version
>> to 1.0. The pulp_file plugin has reached a level of maturity and 
>> stability
>> that I think it could be considered production-ready. I've opened a PR to
>> bump the version to 1.0.0:
>>
>> https://github.com/pulp/pulp_file/pull/380
>>
>> Feedback welcome. I'll set a deadline of April 27, 2020.
>>
>> David
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> 

Re: [Pulp-dev] serializers in pulp 3 can't use write_only fields

[Daniel Alley]

I just want to point out that we have a somewhat similar kind of feature
implemented, "minimal serializers", which uses the 2-serializer approach.
It is a very tiny amount of extra code/work.

e.g.
https://github.com/pulp/pulpcore/blob/master/pulpcore/app/serializers/task.py#L119
https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/serializers.py#L290

the implementation
https://github.com/pulp/pulpcore/blob/master/pulpcore/app/viewsets/base.py#L114-L139

I lean towards option 2 as well for consistency.

On Mon, Apr 27, 2020 at 5:20 AM Ina Panova  wrote:

> I like option number 2 more, even if there is more work behind the scene
> it is more explicit and does not leave opened questions like ' why certain
> fields are null'
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Fri, Apr 24, 2020 at 5:24 PM Dennis Kliban  wrote:
>
>> Today during open floor we concluded that write_only fields cannot be
>> used in serializers because any automated handling of them during OpenAPI
>> schema generation will cause mutations to the schema which will result in
>> backwards incompatible changes to our bindings users.
>>
>> This change should only affect the Content serializers that have a 'file'
>> and 'repository' write_only fields that can be used for uploading content
>> into a repository in a single request. Two solutions were proposed:
>>
>> 1) Use a single serializer for GET and POST and leave 'file' and
>> 'repository' null for GET requests. This solution is the simplest for
>> plugin writers because they don't have to do anything extra. However, it
>> does cause users to wonder why those fields are present and null on GET
>> requests.
>>
>> 2) Require plugin writers to provide 2 serializers for Content - one for
>> GET and another for POST. This is more work for plugin writers, but will
>> make it clear to users what fields are expected in each case.
>>
>> Even though I initially proposed 1, I am in favor of adopting 2. What
>> does everyone else think?
>>
>> We will need to provide validation at startup that will check serializers
>> for presence of write_only fields. The application should fail to start if
>> any are present. This way plugin writers will know not to use the fields.
>> We will also need to provide plugin writer docs that clearly state the
>> guidelines around write_only fields and suggestions for getting by without
>> them.
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Moving to travis-ci.com

[Daniel Alley]

I kid, but it is a funny situation

On Thu, Apr 23, 2020 at 10:01 AM David Davis  wrote:

> LOL.
>
> To answer your question though, we're kind of in a holding pattern right
> now. There are some missing features like sharing jobs between workflows
> and restarting jobs within workflows that we're hoping Github Actions adds.
> Maybe we can revisit though at our next CI meeting.
>
> David
>
>
> On Thu, Apr 23, 2020 at 9:46 AM Daniel Alley  wrote:
>
>> travis-ci.org is broken
>>>
>> migrating away from travis-ci.org is also broken
>>>
>>
>> So, uh, when will github actions be ready xD
>>
>> On Thu, Apr 23, 2020 at 8:40 AM David Davis 
>> wrote:
>>
>>> I wrote to Travis support yesterday and they responded (below). I tried
>>> again this morning and it's still broken. I'll wait a few more days unless
>>> anyone objects.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Thank you for reaching out! We are sorry about the issue that you are
>>> having.We are currently aware of a bug related to migrating repositories
>>> from travis-ci.org <http://travis-ci.org> to travis-ci.com
>>> <http://travis-ci.com> and our Engineering Team is working on this issue to
>>> be solved as soon as possible.Until the fix is deployed, we would kindly
>>> suggest you build your projects on travis-ci.org
>>> <http://travis-ci.org>.Thank you for your patience and once again we are
>>> extremely sorry about the issue.*
>>>
>>> David
>>>
>>>
>>> On Thu, Apr 23, 2020 at 8:34 AM Brian Bouterse 
>>> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Apr 22, 2020 at 3:48 PM David Davis 
>>>> wrote:
>>>>
>>>>> Ok, I agree. I tried to migrate certguard to travis-ci.com and hit a
>>>>> 500 error. Looks like we're not alone both in status updates not being
>>>>> reported from travis-ci.org and having problems trying to migrate:
>>>>>
>>>>>
>>>>> https://travis-ci.community/t/github-status-not-posted-on-commits-on-repositories-using-legacy-service-integration/7798/16
>>>>>
>>>>> I'll wait a day for Travis support to respond but I think we could
>>>>> also just enable the repos and start from scratch in travis-ci.com. I
>>>>> think we'd lose past build information and settings though if we don't
>>>>> migrate.
>>>>>
>>>> This also sounds good to me. If you want to tag-team via irc or video
>>>> we can do that.
>>>>
>>>>
>>>>> David
>>>>>
>>>>>
>>>>> On Wed, Apr 22, 2020 at 3:21 PM Brian Bouterse 
>>>>> wrote:
>>>>>
>>>>>> OK now more than an annoyance, travis-ci.org not notifying github is
>>>>>> preventing certguard from merging without me disabling the CI check  :(
>>>>>>
>>>>>> I think we need to do this asap.
>>>>>>
>>>>>> On Wed, Apr 22, 2020 at 9:21 AM Brian Bouterse 
>>>>>> wrote:
>>>>>>
>>>>>>> I don't know the answer to your question about capacity, but I've
>>>>>>> been hitting this "not-updating-github" issue a lot recently. +1 to 
>>>>>>> moving
>>>>>>> everything to travis-ci.com.
>>>>>>>
>>>>>>> On Wed, Apr 15, 2020 at 1:55 PM David Davis 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> A week or two ago, build statuses for repos whose Ci is hosted on
>>>>>>>> travis-ci.org failed to be reported back to Github. This thread in
>>>>>>>> the Travis support forum about the issue[0] seems to recommend that we
>>>>>>>> migrate off travis-ci.org. My one question is whether we'd
>>>>>>>> experience more congestion if we had all our repos on travis-ci.com
>>>>>>>> I know this has been an issue in the past.
>>>>>>>>
>>>>>>>> [0]
>>>>>>>> https://travis-ci.community/t/github-status-not-posted-on-commits-on-repositories-using-legacy-service-integration/7798/14
>>>>>>>>
>>>>>>>> David
>>>>>>>> ___
>>>>>>>> Pulp-dev mailing list
>>>>>>>> Pulp-dev@redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>>
>>>>>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Moving to travis-ci.com

[Daniel Alley]

>
> travis-ci.org is broken
>
migrating away from travis-ci.org is also broken
>

So, uh, when will github actions be ready xD

On Thu, Apr 23, 2020 at 8:40 AM David Davis  wrote:

> I wrote to Travis support yesterday and they responded (below). I tried
> again this morning and it's still broken. I'll wait a few more days unless
> anyone objects.
>
>
>
>
>
>
>
>
>
> *Thank you for reaching out! We are sorry about the issue that you are
> having.We are currently aware of a bug related to migrating repositories
> from travis-ci.org  to travis-ci.com
>  and our Engineering Team is working on this issue to
> be solved as soon as possible.Until the fix is deployed, we would kindly
> suggest you build your projects on travis-ci.org
> .Thank you for your patience and once again we are
> extremely sorry about the issue.*
>
> David
>
>
> On Thu, Apr 23, 2020 at 8:34 AM Brian Bouterse 
> wrote:
>
>>
>>
>> On Wed, Apr 22, 2020 at 3:48 PM David Davis 
>> wrote:
>>
>>> Ok, I agree. I tried to migrate certguard to travis-ci.com and hit a
>>> 500 error. Looks like we're not alone both in status updates not being
>>> reported from travis-ci.org and having problems trying to migrate:
>>>
>>>
>>> https://travis-ci.community/t/github-status-not-posted-on-commits-on-repositories-using-legacy-service-integration/7798/16
>>>
>>> I'll wait a day for Travis support to respond but I think we could also
>>> just enable the repos and start from scratch in travis-ci.com. I think
>>> we'd lose past build information and settings though if we don't migrate.
>>>
>> This also sounds good to me. If you want to tag-team via irc or video we
>> can do that.
>>
>>
>>> David
>>>
>>>
>>> On Wed, Apr 22, 2020 at 3:21 PM Brian Bouterse 
>>> wrote:
>>>
 OK now more than an annoyance, travis-ci.org not notifying github is
 preventing certguard from merging without me disabling the CI check  :(

 I think we need to do this asap.

 On Wed, Apr 22, 2020 at 9:21 AM Brian Bouterse 
 wrote:

> I don't know the answer to your question about capacity, but I've been
> hitting this "not-updating-github" issue a lot recently. +1 to moving
> everything to travis-ci.com.
>
> On Wed, Apr 15, 2020 at 1:55 PM David Davis 
> wrote:
>
>> A week or two ago, build statuses for repos whose Ci is hosted on
>> travis-ci.org failed to be reported back to Github. This thread in
>> the Travis support forum about the issue[0] seems to recommend that we
>> migrate off travis-ci.org. My one question is whether we'd
>> experience more congestion if we had all our repos on travis-ci.com
>> I know this has been an issue in the past.
>>
>> [0]
>> https://travis-ci.community/t/github-status-not-posted-on-commits-on-repositories-using-legacy-service-integration/7798/14
>>
>> David
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] the "relative path" problem

[Daniel Alley]

Bump, this item needs to move forwards soon.  Does anyone have any thoughts?

On Wed, Apr 1, 2020 at 9:40 AM Pavel Picka  wrote:

> Hi,
> I'd like to add one more question to this topic. Do you think it is a
> blocker for PRs [0] & [1] as by testing [2] this features I haven't run
> into real world example where two really same name packages appears.
> I think this is a 'must have' feature but until we solve/decide it we can
> have two features working may with warning in docs for users that can
> happen in some 'special' repositories.
>
> To follow topic directly I like proposed move to 'RepositoryContent' and
> add it to its uniqueness constraint (if I understand well).
>
> [0] https://github.com/pulp/pulp_rpm/pull/1657
> [1] https://github.com/pulp/pulp_rpm/pull/1642
> [2] tested with centos 7, 8, opensuse and SLE repositories
>
> On Wed, Apr 1, 2020 at 3:22 PM Daniel Alley  wrote:
>
>> We'd like to start a discussion on the "relative path problem" identified
>> recently.
>> Problem:
>>
>> Currently, a relative_path is tied to content in Pulp. This means that if
>> a content unit exists in two places within a repository or across
>> repositories, it has to be stored as two separate content units. This
>> creates redundant data and potential confusion for users.
>>
>> As a specific example, we need to support mirroring content in pulp_rpm
>> <https://pulp.plan.io/issues/6353>. Currently, for each location at
>> which a single package is stored, we’ll need to create a content unit. We
>> could end up with several records representing a single package. Users may
>> be confused about why they see multiple records for a package and they may
>> have trouble for example deciding which content unit to copy.
>> Proposed Solution:
>>
>> Move “relative_path” from its current location on ContentArtifact, to
>> RepositoryContent. This will require a sizable data migration. It is
>> possibly the case that in rare cases, repository versions may change
>> slightly due to deduplication.
>>
>> A repository-version-wide uniqueness constraint will be present on
>> “relative_path”, independently of any other repository uniquness
>> constraints (repo_key_fields) defined by the plugin writer.
>>
>> Modify the Stages API so that the relative_path can be processed in the
>> correct location – instead of “DeclarativeArtifact” it will likely need to
>> go on “DeclarativeContent”
>>
>> Remove “location_href” from the RPM Package content model – it was never
>> a true part of the RPM (file) metadata, it is derived from the repository
>> metadata. So storing it as a part of the Content unit doesn’t entirely make
>> sense.
>> Alternatives
>>
>> In most cases, a content unit will have a single relative path for a
>> content unit. Creating a general solution to solve a one-off problem is
>> usually not a good idea. As an alternative, we could look at another
>> solution for mirroring content. One example might be to create a new object
>> (e.g. RpmRepoMirrorContentMapping) that maps content to specific paths
>> within a repo or repo version.
>> Questions
>>
>>- How do we handle this in pulp_file? How are content units
>>identified in pulp_file without relative_path?
>>   - Checksum?
>>   - How was this problem handled in Pulp 2?
>>
>>
>> Please weigh in if you have any input on potential problems with the
>> proposal, potential alternate solutions, or other insights or questions!
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> --
> Pavel Picka
> Red Hat
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] the "relative path" problem

[Daniel Alley]

We'd like to start a discussion on the "relative path problem" identified
recently.
Problem:

Currently, a relative_path is tied to content in Pulp. This means that if a
content unit exists in two places within a repository or across
repositories, it has to be stored as two separate content units. This
creates redundant data and potential confusion for users.

As a specific example, we need to support mirroring content in pulp_rpm
. Currently, for each location at which a
single package is stored, we’ll need to create a content unit. We could end
up with several records representing a single package. Users may be
confused about why they see multiple records for a package and they may
have trouble for example deciding which content unit to copy.
Proposed Solution:

Move “relative_path” from its current location on ContentArtifact, to
RepositoryContent. This will require a sizable data migration. It is
possibly the case that in rare cases, repository versions may change
slightly due to deduplication.

A repository-version-wide uniqueness constraint will be present on
“relative_path”, independently of any other repository uniquness
constraints (repo_key_fields) defined by the plugin writer.

Modify the Stages API so that the relative_path can be processed in the
correct location – instead of “DeclarativeArtifact” it will likely need to
go on “DeclarativeContent”

Remove “location_href” from the RPM Package content model – it was never a
true part of the RPM (file) metadata, it is derived from the repository
metadata. So storing it as a part of the Content unit doesn’t entirely make
sense.
Alternatives

In most cases, a content unit will have a single relative path for a
content unit. Creating a general solution to solve a one-off problem is
usually not a good idea. As an alternative, we could look at another
solution for mirroring content. One example might be to create a new object
(e.g. RpmRepoMirrorContentMapping) that maps content to specific paths
within a repo or repo version.
Questions

   - How do we handle this in pulp_file? How are content units identified
   in pulp_file without relative_path?
  - Checksum?
  - How was this problem handled in Pulp 2?


Please weigh in if you have any input on potential problems with the
proposal, potential alternate solutions, or other insights or questions!
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] SUSE repositories in Pulp

[Daniel Alley]

I think, as long as the metadata is correct, using just the location_href
would be OK.  It should contain all the other bits of information.

On Mon, Mar 23, 2020 at 3:57 PM David Davis  wrote:

> A couple questions below.
>
> On Mon, Mar 23, 2020 at 3:47 PM Tatiana Tereshchenko 
> wrote:
>
>> Clarification:
>> The proposal is to add  the 'location_href' attribute to
>> the  repo_key_fields, uniqueness constraint within a repository version, so
>> 2 packages with the same NEVRA but different location can be present in one
>> repo.
>>
>
> Why have nevra+relative_path instead of just relative_path? ie would it be
> possible for two packages in a repo version to have the same relative_paths
> but different nevras?
>
>
>> RPM package is still uniquely identified in Pulp by NEVRA +  checksum(aka
>> pkgId) + checksum type.
>>
>
> What if a user has the same package in a repo at two different locations
> or the same package in two different repos at the different locations.
> Since relative_path is attached to the content unit, I think this would
> prevent this from happening? I wonder if uniqueness in Pulp should also
> have location_href/relative_path?
>
>
>>
>> On Mon, Mar 23, 2020 at 7:33 PM Grant Gainey  wrote:
>>
>>> On Mon, Mar 23, 2020 at 2:01 PM Dennis Kliban 
>>> wrote:
>>>
 During last week's RPM team meeting, a concern was raised about using
 the same repository type for both Red Hat and SUSE repositories. Since that
 meeting I have only been able to identify a single difference between the
 two repositories. SUSE repos can contain the same package in two different
 locations in the same repository. Even though I just referred to this as a
 difference, I don't actually believe that to be true. All RPM repositories
 should be able to support this.

>>>
>>> If I'm reading the discussion w/the RPM folks correctly, this is 'odd
>>> but legal' for rpm-repositories. That means that, while SUSE may be the
>>> only current example, there's nothing to keep some other distro/thirdparty
>>> from doing the exact same thing, and we'd have to handle it.
>>>
>>>
 I propose that we not add a separate repository type for SUSE and
 simply add the 'location' attribute of an RPM to it's uniqueness
 constraint.  What do you all think?

>>>
>>> Yeah, concur. It feels messy - but only because the problem-domain
>>> itself is messy :(
>>>
>>> G
>>>
>>> ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>>
>>>
>>> --
>>> Grant Gainey
>>> Principal Software Engineer, Red Hat System Management Engineering
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] SUSE repositories in Pulp

[Daniel Alley]

>
> I propose that we not add a separate repository type for SUSE and simply
> add the 'location' attribute of an RPM to it's uniqueness constraint.  What
> do you all think?


+1

On Mon, Mar 23, 2020 at 2:01 PM Dennis Kliban  wrote:

> During last week's RPM team meeting, a concern was raised about using the
> same repository type for both Red Hat and SUSE repositories. Since that
> meeting I have only been able to identify a single difference between the
> two repositories. SUSE repos can contain the same package in two different
> locations in the same repository. Even though I just referred to this as a
> difference, I don't actually believe that to be true. All RPM repositories
> should be able to support this.
>
> I propose that we not add a separate repository type for SUSE and simply
> add the 'location' attribute of an RPM to it's uniqueness constraint.  What
> do you all think?
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] PR Processor

[Daniel Alley]

This is great!   Awesome work, David <3

On Sat, Mar 21, 2020 at 10:38 AM David Davis  wrote:

> I created a PR workflow in Github Actions so I could learn more about
> Github Actions.
>
> Here's the issue and my PR against plugin_template:
>
> https://pulp.plan.io/issues/4365
> https://github.com/pulp/plugin_template/pull/199
>
> The PR processor will listen for new PRs in Github and when it detects a
> newly opened PR it does two things:
>
> - Updates the pulp.plan.io issue
>   - Adds a comment with a link to the Github PR (regardless of the issue's
> status)
>   - Sets the issue status to POST if it's in NEW or ASSIGNED
> - Adds labels to the Github PR
>   - Right now, this is only "Needs Cherry Pick" which is added if tracker
> == "Issue"
>
> I have an example that demonstrates the PR processor. Here's a story and
> its PR:
>
> https://pulp.plan.io/issues/6381
> https://github.com/daviddavis/pulp_file/pull/11
>
> The PR processor will also comment with a warning if there is no issue
> attached:
>
> https://github.com/daviddavis/pulp_file/pull/12
>
> Questions and feedback welcome.
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Duplicate nevra but not pkgId (suse repos)

[Daniel Alley]

I discussed this a little bit on the #rpm.org channel.  Here is the gist of
that discussion

   - The metadata is "crazy, but technically valid"
   - "the entire SUSE ecosystem tends to do this a lot, anything using OBS,
   including nvidia and dell and friends"
   - "also, SUSE packages can have the same NEVRA with being completely
   different packages because of how their build system makes packages"

I'm not sure what the best means to fix it would be.  Perhaps the
uniqueness constraint should be on the location_href, instead of on the
NEVRA?  Or on NEVRA + location_href?

On Wed, Mar 18, 2020 at 9:47 AM Ina Panova  wrote:

> Pavel,
> I meant to say, that pulp3 does not have such limitation as pulp2 had (
> saving rpms on the filesystem with same nevra).
> The error is raised in pulp3 [0] when a repo version is created, because
> of the repo key[1], we cannot have 2 rpms with save NEVRA.
>
> We can enable that, if we decide to, by adding location_href to the
> repo_key, *but* this needs to be evaluated, it can have side effects and we
> should involve our stakeholders to weigh in.
>
> [0]
> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570
> [1]
> https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/package.py#L188
>
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Wed, Mar 18, 2020 at 2:24 PM Pavel Picka  wrote:
>
>> True in opensuse repository there are two possibilities 'src' and 'nosrc'
>> (this one should be legacy without source code), both are recognized by
>> createrepo_c as arch 'src'.
>>
>> To point the pulp2 code I mentioned I found here [0] (base rpm package
>> what I understood).
>>
>> The rise of error in pulp3 happening here [1] in pulpcore when adding
>> packages to repository version.
>> So as Ina mentioned it doesn't have to be an issue with packages itself
>> than the logic in sync.
>>
>> [0]
>> https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L779
>> [1]
>> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570
>>
>> On Wed, Mar 18, 2020 at 1:55 PM Ina Panova  wrote:
>>
>>> Tanya and Pavel,
>>> in this issue it is explained why we cannot keep 2 packages with same
>>> NEVRA but different checksums within a repo
>>> https://pulp.plan.io/issues/494
>>>
>>> Pulp2 had a limitation where it was not able to save on the filesystem 2
>>> rpms with same filename, it lead to the primary.xml that could have pointed
>>> to the rpm that did not actually get saved.
>>> I believe in Pulp3 we could allow having rpm with same NEVRA if they
>>> have different location_href within a repo.
>>>
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Senior Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>>
>>>
>>> On Wed, Mar 18, 2020 at 10:47 AM Tatiana Tereshchenko <
>>> ttere...@redhat.com> wrote:
>>>
 Hi Pavel,

 On Tue, Mar 17, 2020 at 7:31 PM Pavel Picka  wrote:

> Hello, would like to ask you how to proceed with issue with duplicate
> (but not really) packages.
>
> I am syncing suse repository (opensuse42 and SLE12) and get and
> duplicate error. But when checking the packages [0](from primary.xml) 
> glibc
> and glibc they got same nevra but different checksum (and a few more as
> size..) so doesn't look like real duplicates.
>
 Those are weird, the have the same nevra but see the location_href, one
 is src and the other one is nosrc! :/ :
 
 

 It looks like something OpenSUSE specific. I'm not sure if it's a valid
 way to create a repo with such metadata, we need to figure it out at some
 point.


> I've checked Pulp2 and there is used nevra+sum for repository
> uniqueness. In pulp3 we use only nevra.
>
 Why do you think that in pulp 2 we use NEVRA + checksum? have you
 tested it?  please point to the code.
 I believe in Pulp 2 as well as in Pulp 3 we allow to have packages with
 different checksums in Pulp storage.
 I don't think we allow having the same packages with different
 checksums in the same repo.
 FWIW, in pulp 2 the most recently added package is chosen to stay in a
 repo, no packages with duplicate NEVRA left after sync, see
 https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/purge.py#L285-L333


>
> My suggestion is to extend repo_key_fields for rpm package as is in
> pulp2 with pkgId (checksum). As I don't think they are really duplicates
> and other software can rely on specific version of package.
>

 Unfortunately, I don't remember the main reason to remove duplicates
 based on nevra. Was it because some tooling will complain, or 

Re: [Pulp-dev] Read access in Github

[Daniel Alley]

+1, I occasionally come across a repo where I can't even restart my own CI
jobs, which is frustrating.  Probably even moreso to any new devs with less
access.

On Thu, Feb 13, 2020 at 1:30 PM David Davis  wrote:

> At our CI/CD meeting, Fabricio pointed out that anyone with read access to
> a repository can restart a Travis job. I think we could create a team in
> Github with all Pulp organization members that has read access to all of
> our repos. Then anyone could restart Travis jobs.
>
> Thoughts?
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Moving to Github Actions

[Daniel Alley]

Thanks for your response Brian, I think all of those concerns are
reasonable!  I'll try to add to/help with some of them.

The approach Fabricio took with his PR to pulp_file is incredibly smart, I
think.  In his PR to pulp_file, all of the CI scripts remain unchanged.  He
just fakes being in a Travis environment by using the information that GHA
provides to set all of the $TRAVIS_* environment variables [0] that those
scripts use.  Not only was this much faster to do than doing a wholesale
conversion to idiomatic GHA (Fabricio got everything working with only
about 2 days of work!), it means that Travis can continue running for as
long or as short as we want it to, and once we do switch over the process
of converting the CI scripts to be more idiomatic with GHA can be done at
our leisure rather than frontloading a bunch of work.

Adding GHA support to the template and the other plugins should be as
simple as taking the GHA configs (analogous to .travis.yml) that are
already written for pulp_file, generalizing them for the template, and then
re-applying the template to all the other plugins.  I don't expect that it
would take longer than one engineer-day to complete the whole process!

[0]
https://github.com/pulp/pulp_file/pull/353/files#diff-d45cbc8d15de0f15cdce609ec195cf2eR34-R47

On Sat, Feb 8, 2020 at 10:22 AM Brian Bouterse  wrote:

> Thanks for replying @dalley and @daviddavis, both of your replies make
> good points that resonate with me. Rather than inline responses, I'll try
> to bring back some of your points and comment on them.
>
> @dalley, your articulation of how we would split up the CI to run each
> part on only one CI platform sounds good to me. +1 to the SELinux and FIPS
> testing running on Centos CI, and everything else running in another CI.
> This addresses my concern that we were going to duplicate features from one
> CI to another.
>
> @daviddavis +1 to merging PRs to give us more Github Actions data on repos
> that are not managed by the plugin_template. I'm concerned about merging
> Github Action PRs against plugin_template managed repos. For example with
> pulp_file, I work on that regularly and I'd like to continue using the
> existing CI capabilities it has as-is until the new system is ready. Let me
> know if you think we should do this aspect differently.
>
> @daviddavis to your point that we must move to Github Actions and off of
> Travis makes sense to me because Travis is a huge bottleneck and Github
> Actions can run a lot more in parallel. If we're going to do that though I
> think we need to see a plan on how and when Pulp would leave Travis for
> Github Actions. In terms of making such a plan I would think it would need
> a few aspects in it:
>
> * We need details on each piece of the Travis workflow, where it will be
> ported to, and a rough estimate of how long each piece would take. I think
> these things would make a great EPIC.
> * Who will work on it? It needs I think 2 fully dedicated people who
> already completely understand the Travis stuff in detail. It's too hard for
> one person and would take too long. Not being able to have these people
> fully-dedicated on this task would be a deal-breaker for me. This type of
> activity needs no distraction.
> * It's got to happen fully - If we're leaving Travis for Github Actions,
> we have to fully leave.
> * I think it would be good if when a plugin switches, they switch
> fully-and-at-once from Travis to Github Actions. I think this because
> otherwise, every few days, another plugin_template update will take away a
> Travis feature and move it to Github Actions, which across the 10+ plugins
> and 10+ features would be painful. This would be very confusing I think.
> * It needs to come with education somehow. Maybe a demo video, blog post
> recap, and certainly great docs replacing the Travis ones we have now.
>
> I'm suggesting a plan instead of a decision because without a plan. I
> don't know how long the work will take, and thus I can't know if we can
> afford it in terms of development capacity now. Given the whole convo, I'm
> more wondering if "now is the right time" and less about "if this is the
> right long-term idea". I think the best long-term situation for the Pulp
> development community is likely not with Travis. Now could be the right
> time, if we look at the development team and determine if we can meet all
> of our goals while fully dedicated 1-3 people to this other effort.
>
> Let me know how I can help. Thank you both and Fabricio for continuing to
> drive this improvement for the community.
>
> -Brian
>
>
>
>
>
>
>
>
>
>
>
> On Thu, Feb 6, 2020 at 12:29 PM Daniel Alley  wrote:
>
>> I agree that Centos CI should be a high priority, however I think it is
>>

Re: [Pulp-dev] Moving to Github Actions

[Daniel Alley]

I agree that Centos CI should be a high priority, however I think it is
still important to discuss what we want our end-state to look like, because
that will strongly influence our approach going forwards.  And FWIW, I
don't think Fabricio's work will do any harm in this respect, especially
given that the main focus has been on repos that don't use the template
(pulp-rpm-prerequisites, ansible-pulp), and are putting enough load on
Travis to cause us tangible problems (ansible-pulp, pulp_file performance
tests).

I don't believe Fabricio was suggesting that some plugins would use Travis
and other plugins would use Github Actions.  It was an idea thrown around
that maybe we would want to support a choice of CI for potential plugin
writers, but personally I think we should just ditch Travis entirely.  The
outages (such as the one on Monday) and resource restrictions are hindering
development, and I don't expect it to get better considering how many
senior engineers they laid off after being sold to a private equity firm
with a poor reputation. 

But I also don't think we should try to use Centos CI to replace all the
things Travis is currently doing.  I would rather use Github Actions for
everything except for the very few workflows that require Centos CI,
namely, running tests on a FIPS platform and with SELinux configured.  I
think that this proposal would be both the optimal outcome, and also the
easiest thing to do, and here is why.

Centos CI would not be involved with any of the following:

* Code formatting lints
* Commit message checks
* Changelog checks
* Everything involving a matrix of different combinations of Python /
PostgreSQL / Django versions
* Deploy to PyPI upon pushing a new tag
* Testing things against a specific PR or PRs (probably, if we were to run
the jobs nightly instead of on every PR, which doesn't strike me as
necessary)

The majority of CI complexity is due to these auxillary features and I
don't see any reason to try to port this to Jenkins/Centos CI, much less
try to maintain it across both CI systems.  Here we agree: that would be a
nightmare.  Almost all of the CI-service-specific code deals with these
auxillary checks.  But Fabricio has already proven that these things are
relatively easy to port to Github Actions, which, while different from
Travis, is much more similar to Travis than Jenkins is.  And this work is
already done, and will be really easy to port back into the plugin template
to use everywhere.

Of our various CI scripts, the only ones which would be remain in common
between GHA and CentosCI are install.sh and before_script.sh, which perform
the core setup tasks for our containers.  Every other script in our
.travis/ directory does something which can be the sole concern of Github
Actions.  So the maintenance burden of maintaining that small amount of
common code would not be very high, and certainly not double.




On Thu, Feb 6, 2020 at 10:03 AM David Davis  wrote:

> I think there is an immediate need to move to Github Actions. Yesterday,
> for example, I spent a good deal of time on failing pulp_file jobs, which
> are exceeding Travis' 50 minute threshold[0] (Github Actions has a 6 hour
> limit). We've also been working for weeks on alleviating the bottlenecks
> that we've been experiencing due to Travis' limit of 3 concurrent jobs.
> Paying the Travis tax is detracting from our stakeholder work.
>
> Regarding supporting two CIs, won't we have to support multiple CIs to run
> against selinux and FIPS? The only alternative would be to move everything
> to CentOS CI. Fabricio's pulp_file PR demonstrates though that our CI
> scripts can be made to run in multiple CIs. These scripts are the majority
> of our CI/CD code; the Travis/Github Actions configs are only a couple
> hundred lines. So most of our code will be shared across CIs, which should
> alleviate most of the burden of supporting more than one CI.
>
> I would suggest as a next step we merge the ansible-pulp PR[1] as it
> should provide some real world data about running on Github Actions which
> we can consider. Moreover, its CI is independent from the plugin_template
> and it should help to alleviate most of our bottlenecks in Travis. We can
> postpone the decision around plugins until we have more data and consensus.
>
> [0] https://pulp.plan.io/issues/6104
> [1] https://github.com/pulp/ansible-pulp/pull/217
>
> David
>
>
> On Thu, Feb 6, 2020 at 5:51 AM Brian Bouterse  wrote:
>
>> Inline replies to three convos would be too confusing, so I'm going to
>> try to bring it back to a single thread.
>>
>> The Pulp team can't afford to do two CI's. I estimate it's taken many
>> hundreds of hours cumulatively and probably >10 hours a week at least
>> maintaining the CI for Travis in the plugin template. The current
>> commitments and size of the pulp dev team can't sustain doubling that
>> additional level of investment. Think about a the changes 

Re: [Pulp-dev] Moving to Github Actions

[Daniel Alley]

I think we should start by discussing what we ultimately want our CI to
look like.  Do we want to replace all of our CI with Centos CI, universally
and exclusively?  Do we want to run some tests on Centos CI nightly but
retain something simpler for our standard PR test matrix?

I definitely agree that we will want to test FIPS and SELinux, and those
things will probably need to be tested on Centos CI, and as such supporting
Centos CI is a really good goal to have, and something we will likely need
to do.  I think it's probably a fair statement though that the different CI
platforms have different strengths, and that even if switching exclusively
to Centos CI would be good for us (which I'm not sure that I believe), it
might not be the best thing for community members.

If one of our goals is to give plugin writers the means to test their
plugins as easily as possible, as you mention, then I would argue that
adding Github Actions support to the plugin template is one of the best
possible things we can do, as would start working instantly with zero setup
required (assuming the user's repo is hosted on github).  Whereas, Centos
CI requires a manually-approved signup request

to have your project added and also, the UI is simply more difficult to
use, not just in terms of complexity but in terms of
clicks-necessary-to-reach-useful-information.  I think it's likely that
most plugin writers would care about these things moreso than being able to
test FIPS or SELinux configs, which is important to us but not necessarily
to them.

And if we want to support instant, simple CI for plugin writers, which it
doesn't look like Centos CI really satisfies... then we should probably
additionally support something like Travis or (preferably) Github Actions,
and if we want it to actually be maintained appropriately then we need to
be also using it ourselves to some degree.

There are other benefits to using GHA as our "standard" CI as well -- on
Jenkins/CentosCI, it would not be so simple to set up a matrix of different
versions of Python or Postgresql or Redis to test against, whereas in GHA
it is dead simple.

I'm not arguing against Centos CI, but, do you think it would make sense to
run the jobs to test FIPS and SELinux nightly, as opposed to with every PR?

On Wed, Feb 5, 2020 at 10:31 AM Brian Bouterse  wrote:

> I'm concerned about the move to GH actions and also the timing. The
> benefits of lowering the CI runtime are really great, but I'm worried it
> isn't helping us towards our goals and even takes us further from them.
>
> I'm worried about double the outage risk. There are outages, and
> structurally repo CI pipelines that require more services are at more risk
> for total outage. This raises the risk of "total CI pipelines halting" in a
> concerning way for me. Trading runtime for risk I don't think is an overall
> win; I'd like to find a way to lower the runtime and keep risk the same or
> lower.
>
> Whatever we do I want to make sure we're doing it fully through the plugin
> template. Is this through the plugin template? If it isn't, or it requires
> additional steps to configure it than they had before, then I'm concerned
> about it taking us further from our goals of having the plugin writer take
> as much burden from the plugin writer as possible. I use this thinking to
> answer the question posed from daviddavis. My take is that the plugin
> template's goal is to make writing a plugin with great CI as easy as
> possible. It's design to be a quality improver and a time saver.
>
> Having the lower runtime is nice, but if we're going to put effort in the
> CI, I'd like to bring up prioritizing getting the plugin_template
> integrated with https://ci.centos.org/ as a high-value goal. I'm
> concerned that we're about to ship the SELinux policy and we have no way to
> test it. Similar concerns with certguard's dependency and its dependencies
> not being packaged on Ubuntu (so it's hard to run on Travis). Also, I'm
> concerned we don't have an environment to evaluate FIPS compatibility with.
> Relatively speaking if we can only do one of these two initiatives at this
> time, I believe we should do the CentOS CI.
>
> Lowering the runtime I'm really in favor of, so I hope these concerns
> prompt discussion more than stop the initiative. What do you all think?
>
> On Wed, Feb 5, 2020 at 9:05 AM David Davis  wrote:
>
>> Great question. IMO the main benefit in continuing to support Travis is
>> that we could better separate our test/deployment code from the CI specific
>> bits so that most of the plugin_template code could be CI agnostic. That
>> said, this would be more work. I think it comes down to whether we want our
>> plugin_template to be more opinionated or more configurable.
>>
>> David
>>
>>
>> On Wed, Feb 5, 2020 at 8:18 AM Dana Walker  wrote:
>>
>>> +1 to moving to Github Actions.
>>>
>>> Can anyone think of reasons a 

Re: [Pulp-dev] Not equal filters

[Daniel Alley]

Going by the comments in this PR [0], if something like this would be used
with the API bindings, wouldn't the name__ne=value scheme be better?  It
also just generally seems a bit more consistent.

[0] https://github.com/pulp/pulpcore/pull/316/files

On Wed, Jan 29, 2020 at 10:51 AM David Davis  wrote:

> If we went with django-rest-framework-filters, I'm not sure we could go
> with other options later. Our REST API is semantically versioned so we
> couldn't switch from name!=value to name__ne=value. Perhaps we could label
> not equal filters as a tech preview to get around this?
>
> Also, I'm curious about how we'd upgrade to django 3.0. It looks like they
> haven't been merging PRs so I imagine we'd have to fork the project, add
> django 3.0 support ourselves, and then vendor it. Is my understanding
> correct?
>
> David
>
>
> On Wed, Jan 29, 2020 at 10:13 AM Brian Bouterse 
> wrote:
>
>>
>>
>> On Wed, Jan 29, 2020 at 10:03 AM David Davis 
>> wrote:
>>
>>> A few weeks ago, Katello opened an issue[0] requesting a set of "not
>>> equal" filters (ie filters where a field is not equal to a certain value).
>>> I created a pulpcore issue[0] to investigate whether pulpcore could provide
>>> this functionality and it seems there are a few different options. I wanted
>>> to ask for feedback though as this would affect the user experience and
>>> thus whatever option we choose would be permanent.
>>>
>>> There are three options:
>>>
>>> 1. First there is a package which adds functionality on top of
>>> django-rest-framework-filters[1] which dynamically provides filters for
>>> every field using ! (eg name!=value, state!=value). The problem is that the
>>> package doesn't look like it's well maintained[2] and we'd quickly run into
>>> problems when we try to upgrade to django 3 for example[3]. We'd probably
>>> have to fork the project or take over ownership somehow.
>>>
>> I default to this choice because even though it's not code that is in our
>> control and that can feel uncomfortable, they wrote it and it does what we
>> need. If it doesn't work out we can go with other options later.
>>
>>
>>> 2. The second solution is that we can try to import the code from
>>> django-rest-framework-filters that creates these dynamic filters. This
>>> would free us from having to support any other features
>>> from django-rest-framework-filters we don't want to support.
>>>
>>> 3. The third option is to create a django scope (ie "ne") that would
>>> allow plugin writers to manually define filters like "name__eq=value". This
>>> solution seems the most verbose/explicit/straightforward but also the most
>>> work for plugin writers. I have a couple PRs open to demonstrate this
>>> solution:
>>>
>>> https://github.com/pulp/pulp_rpm/pull/1559
>>> https://github.com/pulp/pulpcore/pull/452
>>>
>>> I'll try to move forward with a solution next week. Feedback before then
>>> would be much appreciated.
>>>
>>> [0] https://pulp.plan.io/issues/5854
>>> [1] https://github.com/philipn/django-rest-framework-filters
>>> [2] https://github.com/philipn/django-rest-framework-filters/issues/324
>>>  and https://github.com/philipn/django-rest-framework-filters/issues/287
>>> [3] https://github.com/philipn/django-rest-framework-filters/issues/326
>>>
>>> David
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Markdown conversion on pulp.plan.io

[Daniel Alley]

I vote "as soon as possible" :)  Definitely a welcome change!

On Wed, Jan 29, 2020 at 9:40 AM Brian Bouterse  wrote:

> plan.io (who hosts pulp.plan.io) is switching to markdown. See their
> blogpost here [0]. They are offering a "conversion" service to convert
> existing data to markdown.
>
> I propose we schedule a time to have pulp.plan.io switch to using
> markdown on pulp.plan.io. What do you think?
>
> [0]: https://support.plan.io/news/166
>
> Thanks!
> Brian
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Bindings - Installer and functional tests

[Daniel Alley]

Excellent work, Fabricio!  This is a big deal :)

On Tue, Jan 28, 2020 at 10:06 AM Fabricio Aguiar 
wrote:

>
> With #6020  *plugin_template*
> functional tests now use *bindings* instead of making REST calls to pulp.
> This brought the need for installing bindings for being able to run
> functional tests, due to this, pbindings
> 
> was introduced on pulp-devel role on ansible-pulp.
>
> pbindings is an alias for pulp-openapi-generator script
> ,
> it takes three positional arguments: module name, language, and version.
> When the optional version parameter is provided, it is used as the version
> string. When it is not provided, the version reported by Pulp's status API
> is used.
>
> *Examples:*
> - generating python bindings for pulp_file:
> $ pbindings pulp_file python
>
> - generating ruby bindings for pulp_file with '3.0.0rc1.dev.10' version
> $ pbindings pulp_file ruby 3.0.0rc1.dev.10
>
> *Pulp-devel, ansible-pulp, pulplift*
> pulp-devel role, during provisioning, automatically installs the python
> bindings for every plugin listed on pulp_install_plugins.
> If you want to make use of this, please update your pulplift
>
> Best regards,
> Fabricio Aguiar
> Software Engineer, Pulp Project
> Red Hat Brazil - Latam 
> +55 11 999652368
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] API for repairing bitrot -- input requested

[Daniel Alley]

Re: Option A, I'm not sure if my concern is warranted, but I'm a little
wary of adding features to the RepositoryVersion endpoint because it
already is special-cased in the bindings generator and it's kind of weird
due to the nesting and the way it's sort-of generic but also exists under
the plugins -- so I worry about making that situation more complex.

But both of those options sound generally fine to me from the UX
perspective.  I probably lean towards option B.

On Wed, Jan 22, 2020 at 10:26 AM Brian Bouterse  wrote:

> FYI, this is for issue: https://pulp.plan.io/issues/5613
>
> On Wed, Jan 22, 2020 at 10:24 AM Brian Bouterse 
> wrote:
>
>> A "repair" feature is being planned that will check-and-re-download all
>> Artifacts associated with a RepositoryVersion. The last thing for us to
>> work out is how the API will be offered to users.
>>
>> Initially, the feature will launch to fix Artifacts for the 'latest'
>> repository version for a given Repository, but over time it's likely we
>> would want to support non-latest RepositoryVersion, or even "the whole Pulp
>> system".
>>
>> # Option A: Compose the "repair" action endpoint under
>> RepositoryVersionViewset
>>
>> We could add an action endpoint to RepositoryVersionViewSet [0], which
>> would provide /pulp/api/v3/repositories/rpm/rpm//repair/. Plugins
>> could opt-out by overriding that endpoint to be a no-op which would also
>> remove it from the bindings, so I think having an opt-out policy is safe.
>>
>> Also since pulpcore analyzes+repairs Artifacts without plugin
>> involvement, this will work for all plugins. This API natively works on any
>> RepositoryVersion, which is nice. The one downside is that it if we add a
>> "fix all of Pulp" feature later this composition approach won't work and
>> we'd need to add a separate top-level API, like /pulp/api/v3/repair/.
>>
>> # Option B: Use a brand-new endpoint, e.g. /pulp/api/v3/repair/
>>
>> Have it take repository_version or repository (and assume latest) as an
>> argument and perform the repair on that. In the future the arguments could
>> become optional, and this would provide the "repair all of Pulp" feature.
>>
>> # Feedback:  What do you think is best?
>>
>> Please reply to the list with what you think is best.
>>
>> [bmbouter's take] I am ok with either, but I have a preference for Option
>> A because composition is a great user experience. Also we're not launching
>> the global pulp checking feature right now anyway.
>>
>> [0]:
>> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/viewsets/repository.py#L181
>>
>> Thanks!
>> Brian
>>
>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Hanging tasks (reloaded)

[Daniel Alley]

@Matthias Dellweg  Would you be willing to file an issue?

On Tue, Jan 21, 2020 at 4:51 PM Daniel Alley  wrote:

> I'm not sure why I can't reproduce the issue on either of my machines but
> I think it would be proper to re-pin redis until we get it figured out.  PR
> here:  https://github.com/pulp/pulpcore/pull/513
>
>
> On Tue, Jan 21, 2020 at 4:37 PM Simon Baatz  wrote:
>
>> I can reproduce the problem described by Matthias with pulp_cookbook's
>> tests almost instantly (I haven't seen a single complete run of the
>> functional tests).
>>
>> Box is pulp3-source-fedora31, redis-py 3.3.1, redis 5.0.7.
>> I don't see these hangs anymore after downgrading redis-py to 3.1.0.
>>
>> Vagrant's VM provider is libvirt with an additional
>> `domain.volume_cache = 'unsafe'` setting.  Write I/O is probably
>> extremely fast in this setup (VM can't force flush to disk)
>>
>>
>> On Tue, Jan 21, 2020 at 12:22:33PM -0500, Daniel Alley wrote:
>> >4 runs on Fedora 31 has not triggered it either. This is using the
>> >latest version of redis-py, 3.3.11, and redis 5.0.7.
>> >
>> >On Tue, Jan 21, 2020 at 11:16 AM Daniel Alley <[1]dal...@redhat.com>
>> >wrote:
>> >
>> >So far I've run it 4 times back to back on a CentOS 7 box and not had
>> >any lockups.  I'll try Fedora
>> >
>> >On Tue, Jan 21, 2020 at 9:36 AM Matthias Dellweg <[2]dell...@atix.de
>> >
>> >wrote:
>> >
>> >  @Daniel
>> >  It is happening with fresh installations (pulplift with libvirt) on
>> >  both boxes i have tried (pulp3-source-debian10 and
>> >  pulp3-source-fedora30). To test, i run `prestart; django-admin test
>> >  pulpcore` multiple times until it hangs (though i have seen it hang
>> >  on
>> >  the first try). There is no modification that i know of. Also, up
>> >  until
>> >  now, i seem to be able to reproduce reliably, while downgrading
>> >  (pinning) redis-py completely solves the issue.
>> >  On Tue, 21 Jan 2020 09:16:43 -0500
>> >  Daniel Alley <[3]dal...@redhat.com> wrote:
>> >  > @Matthias Dellweg <[4]dell...@atix.de>
>> >  > did you restart Pulp after upgrading redis-py?  Are you seeing
>> >  this on
>> >  > fresh boxes, or does it require some modification to reproduce?
>> I
>> >  > don't think any of us have experienced this thus far.
>> >  >
>> >  > The only thing I can think of is that maybe when the worker
>> >  process
>> >  > forks it ends up using a different version of redis-py than the
>> >  > parent worker.
>> >  >
>> >  > On Tue, Jan 21, 2020 at 4:49 AM Matthias Dellweg
>> >  <[5]dell...@atix.de>
>> >  > wrote:
>> >  >
>> >  > > [@ Brian:
>> >  > >
>> >  > >
>> >  [6]
>> https://github.com/pulp/pulpcore/commit/e36e7b5f0eccc176a6e6298df
>> >  29293b014f4710c
>> >  > > ]
>> >  > >
>> >  > > Hi Daniel,
>> >  > > thank you for looking into this.
>> >  > > What i am seeing is:
>> >  > >
>> >  > > Jan 21 09:23:10 [7]pulp3-source-fedora30.anubis.example.com
>> >  > > gunicorn[23274]: 127.0.0.1 - admin [21/Jan/2020:09:23:10 +]
>> >  > > "PATCH
>> >  > >
>> >
>> /pulp/api/v3/repositories/file/file/3a31ed13-585d-4a36-8398-7df40560
>> >  ffa4/
>> >  > > HTTP/1.1" 202 67 "-" "python-requests/2.22.0" Jan 21 09:23:10
>> >  > > [8]pulp3-source-fedora30.anubis.example.com rq[23269]: pulp:
>> >  > > rq.worker:INFO:
>> >  [9]23...@pulp3-source-fedora30.anubis.example.com:
>> >  > > 3a776d4d-ff0c-4b44-afae-08dc7c6cc415 Jan 21 09:23:10
>> >  > > [10]pulp3-source-fedora30.anubis.example.com rq[23270]: pulp:
>> >  > > rq.worker:INFO: resource-manager: Job OK
>> >  > > (aa22aed0-0363-45ae-8254-4ab14893a4a1) Jan 21 09:23:10
>> >  > > [11]pulp3-source-fedora30.anubis.example.com rq[23269]: pulp:
>> >  > > rq.worker:ERROR: Worker
>> >  > > [12]rq:worker:23...@pulp3-source-fedora30.anubis.example.com:
>> >  found an
>> >  > > unhand

Re: [Pulp-dev] Hanging tasks (reloaded)

[Daniel Alley]

I'm not sure why I can't reproduce the issue on either of my machines but I
think it would be proper to re-pin redis until we get it figured out.  PR
here:  https://github.com/pulp/pulpcore/pull/513


On Tue, Jan 21, 2020 at 4:37 PM Simon Baatz  wrote:

> I can reproduce the problem described by Matthias with pulp_cookbook's
> tests almost instantly (I haven't seen a single complete run of the
> functional tests).
>
> Box is pulp3-source-fedora31, redis-py 3.3.1, redis 5.0.7.
> I don't see these hangs anymore after downgrading redis-py to 3.1.0.
>
> Vagrant's VM provider is libvirt with an additional
> `domain.volume_cache = 'unsafe'` setting.  Write I/O is probably
> extremely fast in this setup (VM can't force flush to disk)
>
>
> On Tue, Jan 21, 2020 at 12:22:33PM -0500, Daniel Alley wrote:
> >4 runs on Fedora 31 has not triggered it either. This is using the
> >latest version of redis-py, 3.3.11, and redis 5.0.7.
> >
> >On Tue, Jan 21, 2020 at 11:16 AM Daniel Alley <[1]dal...@redhat.com>
> >wrote:
> >
> >So far I've run it 4 times back to back on a CentOS 7 box and not had
> >any lockups.  I'll try Fedora
> >
> >On Tue, Jan 21, 2020 at 9:36 AM Matthias Dellweg <[2]dell...@atix.de>
> >wrote:
> >
> >  @Daniel
> >  It is happening with fresh installations (pulplift with libvirt) on
> >  both boxes i have tried (pulp3-source-debian10 and
> >  pulp3-source-fedora30). To test, i run `prestart; django-admin test
> >  pulpcore` multiple times until it hangs (though i have seen it hang
> >  on
> >  the first try). There is no modification that i know of. Also, up
> >  until
> >      now, i seem to be able to reproduce reliably, while downgrading
> >  (pinning) redis-py completely solves the issue.
> >  On Tue, 21 Jan 2020 09:16:43 -0500
> >  Daniel Alley <[3]dal...@redhat.com> wrote:
> >  > @Matthias Dellweg <[4]dell...@atix.de>
> >  > did you restart Pulp after upgrading redis-py?  Are you seeing
> >  this on
> >  > fresh boxes, or does it require some modification to reproduce?  I
> >  > don't think any of us have experienced this thus far.
> >  >
> >  > The only thing I can think of is that maybe when the worker
> >  process
> >  > forks it ends up using a different version of redis-py than the
> >  > parent worker.
> >  >
> >  > On Tue, Jan 21, 2020 at 4:49 AM Matthias Dellweg
> >  <[5]dell...@atix.de>
> >  > wrote:
> >  >
> >  > > [@ Brian:
> >  > >
> >  > >
> >  [6]
> https://github.com/pulp/pulpcore/commit/e36e7b5f0eccc176a6e6298df
> >  29293b014f4710c
> >  > > ]
> >  > >
> >  > > Hi Daniel,
> >  > > thank you for looking into this.
> >  > > What i am seeing is:
> >  > >
> >  > > Jan 21 09:23:10 [7]pulp3-source-fedora30.anubis.example.com
> >  > > gunicorn[23274]: 127.0.0.1 - admin [21/Jan/2020:09:23:10 +]
> >  > > "PATCH
> >  > >
> >  /pulp/api/v3/repositories/file/file/3a31ed13-585d-4a36-8398-7df40560
> >  ffa4/
> >  > > HTTP/1.1" 202 67 "-" "python-requests/2.22.0" Jan 21 09:23:10
> >  > > [8]pulp3-source-fedora30.anubis.example.com rq[23269]: pulp:
> >  > > rq.worker:INFO:
> >  [9]23...@pulp3-source-fedora30.anubis.example.com:
> >  > > 3a776d4d-ff0c-4b44-afae-08dc7c6cc415 Jan 21 09:23:10
> >  > > [10]pulp3-source-fedora30.anubis.example.com rq[23270]: pulp:
> >  > > rq.worker:INFO: resource-manager: Job OK
> >  > > (aa22aed0-0363-45ae-8254-4ab14893a4a1) Jan 21 09:23:10
> >  > > [11]pulp3-source-fedora30.anubis.example.com rq[23269]: pulp:
> >  > > rq.worker:ERROR: Worker
> >  > > [12]rq:worker:23...@pulp3-source-fedora30.anubis.example.com:
> >  found an
> >  > > unhandled exception, quitting... Jan 21 09:23:10
> >  > > [13]pulp3-source-fedora30.anubis.example.com rq[23269]:
> >  Traceback (most
> >  > > recent call last): Jan 21 09:23:10
> >  > > [14]pulp3-source-fedora30.anubis.example.com rq[23269]:   File
> >  > >
> >  "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py",
> >  > > line 782, in prepare_job_execution Jan 21 0

Re: [Pulp-dev] Hanging tasks (reloaded)

[Daniel Alley]

4 runs on Fedora 31 has not triggered it either. This is using the latest
version of redis-py, 3.3.11, and redis 5.0.7.

On Tue, Jan 21, 2020 at 11:16 AM Daniel Alley  wrote:

> So far I've run it 4 times back to back on a CentOS 7 box and not had any
> lockups.  I'll try Fedora
>
> On Tue, Jan 21, 2020 at 9:36 AM Matthias Dellweg  wrote:
>
>> @Daniel
>> It is happening with fresh installations (pulplift with libvirt) on
>> both boxes i have tried (pulp3-source-debian10 and
>> pulp3-source-fedora30). To test, i run `prestart; django-admin test
>> pulpcore` multiple times until it hangs (though i have seen it hang on
>> the first try). There is no modification that i know of. Also, up until
>> now, i seem to be able to reproduce reliably, while downgrading
>> (pinning) redis-py completely solves the issue.
>>
>> On Tue, 21 Jan 2020 09:16:43 -0500
>> Daniel Alley  wrote:
>>
>> > @Matthias Dellweg 
>> > did you restart Pulp after upgrading redis-py?  Are you seeing this on
>> > fresh boxes, or does it require some modification to reproduce?  I
>> > don't think any of us have experienced this thus far.
>> >
>> > The only thing I can think of is that maybe when the worker process
>> > forks it ends up using a different version of redis-py than the
>> > parent worker.
>> >
>> > On Tue, Jan 21, 2020 at 4:49 AM Matthias Dellweg 
>> > wrote:
>> >
>> > > [@ Brian:
>> > >
>> > >
>> https://github.com/pulp/pulpcore/commit/e36e7b5f0eccc176a6e6298df29293b014f4710c
>> > > ]
>> > >
>> > > Hi Daniel,
>> > > thank you for looking into this.
>> > > What i am seeing is:
>> > >
>> > > Jan 21 09:23:10 pulp3-source-fedora30.anubis.example.com
>> > > gunicorn[23274]: 127.0.0.1 - admin [21/Jan/2020:09:23:10 +]
>> > > "PATCH
>> > >
>> /pulp/api/v3/repositories/file/file/3a31ed13-585d-4a36-8398-7df40560ffa4/
>> > > HTTP/1.1" 202 67 "-" "python-requests/2.22.0" Jan 21 09:23:10
>> > > pulp3-source-fedora30.anubis.example.com rq[23269]: pulp:
>> > > rq.worker:INFO: 23...@pulp3-source-fedora30.anubis.example.com:
>> > > 3a776d4d-ff0c-4b44-afae-08dc7c6cc415 Jan 21 09:23:10
>> > > pulp3-source-fedora30.anubis.example.com rq[23270]: pulp:
>> > > rq.worker:INFO: resource-manager: Job OK
>> > > (aa22aed0-0363-45ae-8254-4ab14893a4a1) Jan 21 09:23:10
>> > > pulp3-source-fedora30.anubis.example.com rq[23269]: pulp:
>> > > rq.worker:ERROR: Worker
>> > > rq:worker:23...@pulp3-source-fedora30.anubis.example.com: found an
>> > > unhandled exception, quitting... Jan 21 09:23:10
>> > > pulp3-source-fedora30.anubis.example.com rq[23269]: Traceback (most
>> > > recent call last): Jan 21 09:23:10
>> > > pulp3-source-fedora30.anubis.example.com rq[23269]:   File
>> > > "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py",
>> > > line 782, in prepare_job_execution Jan 21 09:23:10
>> > > pulp3-source-fedora30.anubis.example.com rq[23269]:
>> > > pipeline.execute() Jan 21 09:23:10
>> > > pulp3-source-fedora30.anubis.example.com rq[23269]:   File
>> > > "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/client.py",
>> > > line 3707, in execute Jan 21 09:23:10
>> > > pulp3-source-fedora30.anubis.example.com rq[23269]:
>> > > self.reset() Jan 21 09:23:10
>> > > pulp3-source-fedora30.anubis.example.com rq[23269]: File
>> > > "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/client.py",
>> > > line 3476, in reset Jan 21 09:23:10
>> > > pulp3-source-fedora30.anubis.example.com rq[23269]:
>> > > self.connection_pool.release(self.connection) Jan 21 09:23:10
>> > > pulp3-source-fedora30.anubis.example.com rq[23269]:   File
>> > >
>> "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/connection.py",
>> > > line 1114, in release Jan 21 09:23:10
>> > > pulp3-source-fedora30.anubis.example.com rq[23269]:
>> > > self._in_use_connections.remove(connection) Jan 21 09:23:10
>> > > pulp3-source-fedora30.anubis.example.com rq[23269]: KeyError:
>> > > Connection Jan 21 09:23:10
>> > > pulp3-source-fedora30.anubis.example.com rq[23269]: During handling
>> > > of the above exception, another exception occurred: Jan 21 09:23:10
>> > > pulp3-source-fedora

Re: [Pulp-dev] Hanging tasks (reloaded)

[Daniel Alley]

So far I've run it 4 times back to back on a CentOS 7 box and not had any
lockups.  I'll try Fedora

On Tue, Jan 21, 2020 at 9:36 AM Matthias Dellweg  wrote:

> @Daniel
> It is happening with fresh installations (pulplift with libvirt) on
> both boxes i have tried (pulp3-source-debian10 and
> pulp3-source-fedora30). To test, i run `prestart; django-admin test
> pulpcore` multiple times until it hangs (though i have seen it hang on
> the first try). There is no modification that i know of. Also, up until
> now, i seem to be able to reproduce reliably, while downgrading
> (pinning) redis-py completely solves the issue.
>
> On Tue, 21 Jan 2020 09:16:43 -0500
> Daniel Alley  wrote:
>
> > @Matthias Dellweg 
> > did you restart Pulp after upgrading redis-py?  Are you seeing this on
> > fresh boxes, or does it require some modification to reproduce?  I
> > don't think any of us have experienced this thus far.
> >
> > The only thing I can think of is that maybe when the worker process
> > forks it ends up using a different version of redis-py than the
> > parent worker.
> >
> > On Tue, Jan 21, 2020 at 4:49 AM Matthias Dellweg 
> > wrote:
> >
> > > [@ Brian:
> > >
> > >
> https://github.com/pulp/pulpcore/commit/e36e7b5f0eccc176a6e6298df29293b014f4710c
> > > ]
> > >
> > > Hi Daniel,
> > > thank you for looking into this.
> > > What i am seeing is:
> > >
> > > Jan 21 09:23:10 pulp3-source-fedora30.anubis.example.com
> > > gunicorn[23274]: 127.0.0.1 - admin [21/Jan/2020:09:23:10 +]
> > > "PATCH
> > >
> /pulp/api/v3/repositories/file/file/3a31ed13-585d-4a36-8398-7df40560ffa4/
> > > HTTP/1.1" 202 67 "-" "python-requests/2.22.0" Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]: pulp:
> > > rq.worker:INFO: 23...@pulp3-source-fedora30.anubis.example.com:
> > > 3a776d4d-ff0c-4b44-afae-08dc7c6cc415 Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23270]: pulp:
> > > rq.worker:INFO: resource-manager: Job OK
> > > (aa22aed0-0363-45ae-8254-4ab14893a4a1) Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]: pulp:
> > > rq.worker:ERROR: Worker
> > > rq:worker:23...@pulp3-source-fedora30.anubis.example.com: found an
> > > unhandled exception, quitting... Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]: Traceback (most
> > > recent call last): Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]:   File
> > > "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py",
> > > line 782, in prepare_job_execution Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]:
> > > pipeline.execute() Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]:   File
> > > "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/client.py",
> > > line 3707, in execute Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]:
> > > self.reset() Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]: File
> > > "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/client.py",
> > > line 3476, in reset Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]:
> > > self.connection_pool.release(self.connection) Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]:   File
> > >
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/connection.py",
> > > line 1114, in release Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]:
> > > self._in_use_connections.remove(connection) Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]: KeyError:
> > > Connection Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]: During handling
> > > of the above exception, another exception occurred: Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]: Traceback (most
> > > recent call last): Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]:   File
> > > "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py",
> > > line 515, in work Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]:
> > > self.execute_job(job, queue) Jan 21 09:23:10
> > > pulp3-source-fedora30.anubis.example.com rq[23269]:   File
> 

Re: [Pulp-dev] Hanging tasks (reloaded)

[Daniel Alley]

fedora30.anubis.example.com rq[23269]:   File
> "/home/vagrant/devel/pulpcore/pulpcore/tasking/worker.py", line 103, in
> perform_job Jan 21 09:23:10 pulp3-source-fedora30.anubis.example.com
> rq[23269]: return super().perform_job(job, queue) Jan 21 09:23:10
> pulp3-source-fedora30.anubis.example.com rq[23269]:   File
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py", line
> 866, in perform_job Jan 21 09:23:10
> pulp3-source-fedora30.anubis.example.com rq[23269]:
> self.prepare_job_execution(job, heartbeat_ttl) Jan 21 09:23:10
> pulp3-source-fedora30.anubis.example.com rq[23269]:   File
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py", line
> 782, in prepare_job_execution Jan 21 09:23:10
> pulp3-source-fedora30.anubis.example.com rq[23269]:
> pipeline.execute() Jan 21 09:23:10
> pulp3-source-fedora30.anubis.example.com rq[23269]:   File
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/client.py",
> line 3445, in __exit__ Jan 21 09:23:10
> pulp3-source-fedora30.anubis.example.com rq[23269]: self.reset()
> Jan 21 09:23:10 pulp3-source-fedora30.anubis.example.com rq[23269]:
> File
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/client.py",
> line 3476, in reset Jan 21 09:23:10
> pulp3-source-fedora30.anubis.example.com rq[23269]:
> self.connection_pool.release(self.connection) Jan 21 09:23:10
> pulp3-source-fedora30.anubis.example.com rq[23269]:   File
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/connection.py",
> line 1114, in release Jan 21 09:23:10
> pulp3-source-fedora30.anubis.example.com rq[23269]:
> self._in_use_connections.remove(connection) Jan 21 09:23:10
> pulp3-source-fedora30.anubis.example.com rq[23269]: KeyError:
> Connection Jan 21 09:23:10
> pulp3-source-fedora30.anubis.example.com gunicorn[23274]: 127.0.0.1 -
> admin [21/Jan/2020:09:23:10 +] "GET
> /pulp/api/v3/tasks/3a776d4d-ff0c-4b44-afae-08dc7c6cc415/ HTTP/1.1" 200
> 477 "-" "python-requests/2.22.0"
>
> and that last line repeats forever.
>
> On Fri, 17 Jan 2020 09:09:01 -0500
> Daniel Alley  wrote:
>
> > Different issue perhaps?  Are you seeing anything in the logs that
> > looks like this? https://github.com/rq/rq/issues/1044
> >
> > On Fri, Jan 17, 2020 at 9:06 AM Daniel Alley 
> > wrote:
> >
> > > Strange, I'm pretty sure an issue like this is why we pinned
> > > originally, but upstream said that that particular issue was
> > > (supposedly) fixed.
> > >
> > >
> https://github.com/andymccurdy/redis-py/issues/1136#issuecomment-571168161
> > >
> > > On Fri, Jan 17, 2020 at 4:15 AM Matthias Dellweg 
> > > wrote:
> > >> Hello all,
> > >> I believe i have  found a new incarnation of hanging tasks (tm).
> > >> This time it is pulp3 and as hard to nail down as ever.
> > >> I think, it is introduced by
> > >> e36e7b5f0eccc176a6e6298df29293b014f4710c. Where the dependency on
> > >> redis was dropped with the result that 3.3.smth instead of
> > >> 3.1.smth was installed.
> > >>
> > >> Before filing an issue, is there anyone out there to share that
> > >> experience?
> > >>
> > >> Also as a thought protocol of how to reproduce:
> > >> I have seen tasks hanging in both "waiting" and "running" state
> > >> when using the command `prestart; django-admin test pulp_deb` or
> > >> `<...> test pulpcore`. All the tasks I have seen were `sync`,
> > >> `general_create` or `general_delete` and looked like they never
> > >> started to do anything for real. To have consistent results, i had
> > >> the impression that i needed to rebuild the vagrant boxes for
> > >> every bisecting step. Also updating the python-redis package on a
> > >> box that worked, produced a hanging task in the next run.
> > >>
> > >> Have a good day,
> > >>   Matthias
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Hanging tasks (reloaded)

[Daniel Alley]

Different issue perhaps?  Are you seeing anything in the logs that looks
like this? https://github.com/rq/rq/issues/1044

On Fri, Jan 17, 2020 at 9:06 AM Daniel Alley  wrote:

> Strange, I'm pretty sure an issue like this is why we pinned originally,
> but upstream said that that particular issue was (supposedly) fixed.
>
> https://github.com/andymccurdy/redis-py/issues/1136#issuecomment-571168161
>
> On Fri, Jan 17, 2020 at 4:15 AM Matthias Dellweg  wrote:
>
>> Hello all,
>> I believe i have  found a new incarnation of hanging tasks (tm). This
>> time it is pulp3 and as hard to nail down as ever.
>> I think, it is introduced by e36e7b5f0eccc176a6e6298df29293b014f4710c.
>> Where the dependency on redis was dropped with the result that 3.3.smth
>> instead of 3.1.smth was installed.
>>
>> Before filing an issue, is there anyone out there to share that
>> experience?
>>
>> Also as a thought protocol of how to reproduce:
>> I have seen tasks hanging in both "waiting" and "running" state when
>> using the command `prestart; django-admin test pulp_deb` or `<...> test
>> pulpcore`. All the tasks I have seen were `sync`, `general_create` or
>> `general_delete` and looked like they never started to do anything for
>> real. To have consistent results, i had the impression that i needed to
>> rebuild the vagrant boxes for every bisecting step.
>> Also updating the python-redis package on a box that worked, produced a
>> hanging task in the next run.
>>
>> Have a good day,
>>   Matthias
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Hanging tasks (reloaded)

[Daniel Alley]

Strange, I'm pretty sure an issue like this is why we pinned originally,
but upstream said that that particular issue was (supposedly) fixed.

https://github.com/andymccurdy/redis-py/issues/1136#issuecomment-571168161

On Fri, Jan 17, 2020 at 4:15 AM Matthias Dellweg  wrote:

> Hello all,
> I believe i have  found a new incarnation of hanging tasks (tm). This
> time it is pulp3 and as hard to nail down as ever.
> I think, it is introduced by e36e7b5f0eccc176a6e6298df29293b014f4710c.
> Where the dependency on redis was dropped with the result that 3.3.smth
> instead of 3.1.smth was installed.
>
> Before filing an issue, is there anyone out there to share that
> experience?
>
> Also as a thought protocol of how to reproduce:
> I have seen tasks hanging in both "waiting" and "running" state when
> using the command `prestart; django-admin test pulp_deb` or `<...> test
> pulpcore`. All the tasks I have seen were `sync`, `general_create` or
> `general_delete` and looked like they never started to do anything for
> real. To have consistent results, i had the impression that i needed to
> rebuild the vagrant boxes for every bisecting step.
> Also updating the python-redis package on a box that worked, produced a
> hanging task in the next run.
>
> Have a good day,
>   Matthias
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] RPM plugin Copy API discussion

[Daniel Alley]

Hi Bruno,

Could you specify in what context?  Do you mean with respect to the units
that the user is telling Pulp to copy, or the dependencies of those units?


On Mon, Jan 13, 2020 at 7:48 AM Bruno Rocha  wrote:

> HI Daniel,,
>
> I have one question: What happens when the destination repository already
> exists and already has some contents in it? is there some control that can
> be set such as "skip", "replace", "keep the higher version" etc ?
>
> Thanks.
>
> Bruno Rocha
>
> Senior Quality Engineer - Management QE - Pulp Project
> *irc*: rochacbruno
>
> *social*: http://about.me/rochacbruno
>
> “Progress is the realization of utopia.”
> <https://red.ht/sig>
>
>
> On Thu, Dec 12, 2019 at 5:52 PM Daniel Alley  wrote:
>
>> In the coming weeks we will need to settle on a strategy for the Pulp 3
>> advanced copy APIs for the RPM plugin. This is one of, if not the most
>> complicated plugins, so there are a lot of factors to consider.  We'd like
>> to invite the community to participate in the discussion and get an idea of
>> what patterns and workflows you would like to have, and help elaborate on
>> the pros and cons of each approach, and possibly suggest new approaches we
>> haven't thought of.
>>
>> Here are the basic use cases that we have come up with, and some points
>> of interest/concern that we noted during the RPM meeting today:
>>
>> Use cases?
>>
>>-
>>
>>As a user, I can copy all content from one repository to another
>>repository
>>-
>>
>>As a user, I can copy content matching certain "search criteria" from
>>one repository to another repository
>>-
>>
>>   Search criteria with copying more multiple content types is a
>>   difficult problem -- what content types do the criteria correspond to?
>>   - Possible solutions:
>>  - Allow criteria to be specified as a JSON data structure so
>>  that it can be kept organized by type
>>  - Only allow copying one content type at a time -- but couple
>>  this with a feature to allow "incomplete" repository versions to be 
>> built
>>  up over the course of many different operations
>> - This was suggested as a possible necessary use case, but
>> one that we need more feedback on
>> -
>>
>>As a user, when copying content that directly references other
>>content, the referenced content is *always* copied (if present)
>>-
>>
>>   e.g. Modules referencing Modules, Modules referencing RPMs,
>>   Erratum referencing RPMs, and {{other types}}
>>   - Can't think of a reason to allow bypassing this -- the manual UX
>>   if you wanted to bypass this would be excruciatingly painful and 
>> require
>>   looking at the details of the individual content units
>>   - Modules declare debug packages as artifacts, debug packages are
>>   not always present in the same repo, so we can only copy referenced 
>> units
>>   "if present"
>>
>>
>>-
>>
>>As a user, I can optionally choose to copy all indirect dependencies
>>of content that is being copied (recursive copy)
>>-
>>
>>   Should this be the default?
>>   -
>>
>>   Arguments for both 'yes' and 'no'
>>   -
>>
>>   Let’s see what perf looks like in real-world scenarios
>>   -
>>
>>Some content types create a new content unit in the destination repo
>>instead of just copying
>>-
>>
>>   e.g. yum_repo_metadata_file
>>   -
>>
>>Multi-repo copy needed for modularity
>>- Multiple source repositories used for depsolving, search criteria
>>   only applies to the primary source repo, multiple target repositories
>>   correspond with matching source repos
>>
>>
>> What should the API look like?
>>
>>-
>>
>>If we want to support all of the use cases with API endpoint and one
>>task, then we might need to use ggainey’s proposal for a complex filter
>>provided by a JSON blob (proposal #3 on the issue)
>>- {
>>   'package': ["name=firefox, arch=x86_64, version>=70",
>>   "name=chrome, version==72.0.1"],
>>   'modulemd': ["name=ripgrep, stream=master"]
>>   }
>>   -
>>
>>   The Python plugin does something like this, but th

[Pulp-dev] RPM plugin Copy API discussion

[Daniel Alley]

In the coming weeks we will need to settle on a strategy for the Pulp 3
advanced copy APIs for the RPM plugin. This is one of, if not the most
complicated plugins, so there are a lot of factors to consider.  We'd like
to invite the community to participate in the discussion and get an idea of
what patterns and workflows you would like to have, and help elaborate on
the pros and cons of each approach, and possibly suggest new approaches we
haven't thought of.

Here are the basic use cases that we have come up with, and some points of
interest/concern that we noted during the RPM meeting today:

Use cases?

   -

   As a user, I can copy all content from one repository to another
   repository
   -

   As a user, I can copy content matching certain "search criteria" from
   one repository to another repository
   -

  Search criteria with copying more multiple content types is a
  difficult problem -- what content types do the criteria correspond to?
  - Possible solutions:
 - Allow criteria to be specified as a JSON data structure so that
 it can be kept organized by type
 - Only allow copying one content type at a time -- but couple this
 with a feature to allow "incomplete" repository versions to
be built up
 over the course of many different operations
- This was suggested as a possible necessary use case, but one
that we need more feedback on
-

   As a user, when copying content that directly references other content,
   the referenced content is *always* copied (if present)
   -

  e.g. Modules referencing Modules, Modules referencing RPMs, Erratum
  referencing RPMs, and {{other types}}
  - Can't think of a reason to allow bypassing this -- the manual UX if
  you wanted to bypass this would be excruciatingly painful and require
  looking at the details of the individual content units
  - Modules declare debug packages as artifacts, debug packages are not
  always present in the same repo, so we can only copy referenced units "if
  present"


   -

   As a user, I can optionally choose to copy all indirect dependencies of
   content that is being copied (recursive copy)
   -

  Should this be the default?
  -

  Arguments for both 'yes' and 'no'
  -

  Let’s see what perf looks like in real-world scenarios
  -

   Some content types create a new content unit in the destination repo
   instead of just copying
   -

  e.g. yum_repo_metadata_file
  -

   Multi-repo copy needed for modularity
   - Multiple source repositories used for depsolving, search criteria only
  applies to the primary source repo, multiple target repositories
correspond
  with matching source repos


What should the API look like?

   -

   If we want to support all of the use cases with API endpoint and one
   task, then we might need to use ggainey’s proposal for a complex filter
   provided by a JSON blob (proposal #3 on the issue)
   - {
  'package': ["name=firefox, arch=x86_64, version>=70",
  "name=chrome, version==72.0.1"],
  'modulemd': ["name=ripgrep, stream=master"]
  }
  -

  The Python plugin does something like this, but the criteria matching
  library is provided for us by the Python packaging utils. We
would have to
  implement this ourselves for RPM
  -

  Brian proposed a modification of this idea where the search criteria
  can be saved and re-used between tasks rather than provided each
and every
  time
  -

  Ina’s concern: How we would process these queries?
  -

 We will perform search on each content type separately and return
 a result only if both of the queries would give back a result?
 -

 It can happen that repo will have package foo but not modulemd
 ripgrep
 - Do we fail if any are missing, or succeed regardless of matching
 content?
  -

   If we had a feature where the user can progressively build up a complete
   repository version by modifying an incomplete repository version, then one
   single very complex search criteria layout is unnecessary. You could run
   several copy tasks, one for each content type you want to copy, with
   criteria corresponding only to that type.
   -

  BUT, with recursive copy, that might lead to a lot of overhead since
  it has to be set up for each individual task
  -

 BUT, there are ways to mitigate that overhead, albeit it would be
 very challenging


Hopefully it is clear from this summary that the topic is complicated and
that it could be accomplished in several very different ways. We would love
your feedback!
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] [Breaking] Minor change to typed repositories

[Daniel Alley]

There is a minor change for Pulp 3 plugin writers Repository models merging
today.  It's a very simple one-line code change.

On your Repository subclass, define the CONTENT_TYPES constant as a list
with all of the Content types (the classes) supported by this type of
repository, like this:

https://github.com/pulp/pulp_container/pull/25/files#diff-90a314d7b2d5ddb20d360be02a4ec146R189

This change will allow for better performance and validation inside of
Pulpcore, as it knows what types it's dealing with ahead of time.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Memory consumption on RPM sync

[Daniel Alley]

Fabricio, this is great work!

One thing that stands out is that a very large amount of time is being
spent in remove_duplicates(), 65% of the total runtime of the sync.

   - 13% of the total runtime spent on this inside cast()
   
https://github.com/pulp/pulpcore/blob/master/pulpcore/plugin/repo_version_utils.py#L31
   - 31% of the total runtime spent in these lines
   
https://github.com/pulp/pulpcore/blob/master/pulpcore/plugin/repo_version_utils.py#L37-L38
   - 20% of the total runtime spent on this line
   
https://github.com/pulp/pulpcore/blob/master/pulpcore/plugin/repo_version_utils.py#L42

There's a couple of suggestions on how to improve this code written up here
https://pulp.plan.io/issues/5701

The memory usage seems to be coming from the Django ORM backend.  My guess
is that lines 37 & 38 (linked above) are generating some extremely long
SQL, because it's basically doing OR with a vast number of individual
queries. And I think that is also not great for performance.
https://www.cybertec-postgresql.com/en/avoid-or-for-better-performance/

Another hint that that might be the case is that the memory usage grows and
grows and then collapses all at once.  That would make sense if the SQL
compiler's data structure is growing and growing and then being destroyed
after the query executes.

I would imagine that if we fix our queries, we'll both fix the memory
consumption and improve performance at the same time.

On Tue, Nov 26, 2019 at 11:57 AM Fabricio Aguiar 
wrote:

> Hi everyone,
>
> I've been investigating memory consumption for syncing on RPM plugin - issue
> 5688 , and I want to share what I've
> found so far.
>
> First, it seems that a new process is created when we started to sync, and
> this new process is responsible for the large amount of memory consumption
> observed.
>
> As I shared on some notes here: https://pulp.plan.io/issues/5688
> The problem seems to affect only RHEL, I did not see a large consumption
> of memory for the following repos:
> - CentOS 6: http://mirror.centos.org/centos-6/6.10/os/x86_64/
> - CentOS 7: http://mirror.linux.duke.edu/pub/centos/7/os/x86_64/
> - CentOS 8 AppStrem:
> http://mirror.linux.duke.edu/pub/centos/8/AppStream/x86_64/os/
> - CentOS 8 BaseOS:
> http://mirror.linux.duke.edu/pub/centos/8/BaseOS/x86_64/os/
> - EPEL 7: https://dl.fedoraproject.org/pub/epel/7/x86_64/
>
> As stated on the issue, with RHEL 7 we can observe the problem, for my
> tests, I used the following repo:
> http://cdn.stage.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/
>
> For my tests, I used pulp3-source-fedora30 vagrant box, initially set with
> 4GB, then I increased to 16GB, and it was not enough. I only was able to
> sync RHEL 7 when I increased the memory up to 23GB.
>
> Utilizing py-spy I got those SVG profiles attached, you can find them
> hosted here:
> Sync - https://sendeyo.com/up/d/90a8ae4c8f
> Re-sync - https://sendeyo.com/up/d/4c855bcce3
>
> With the following branch, I was able to take some tracemalloc
>  snapshots (every 30
> seconds):
>
> https://github.com/pulp/pulp_rpm/commit/f3f079010cfe81b7f5cf3ef94c2402b1ccf7d90c
>
> *Tracemalloc for Sync at peak of memory consumption:*
>
> #1: sql/compiler.py:1512: 7825340.4 KiB
> #2: tasks/synchronizing.py:154: 4062651.9 KiB
> #3: stages/declarative_version.py:149: 4062601.9 KiB
> #4: models/repository.py:631: 4062130.0 KiB
> #5: models/repository.py:96: 4062127.7 KiB
> #6: rq/job.py:611: 4060459.2 KiB
> #7: rq/job.py:605: 4058273.8 KiB
> #8: rq/worker.py:822: 4053883.3 KiB
> #9: tasking/worker.py:100: 4053875.9 KiB
> #10: rq/worker.py:684: 4053849.9 KiB
> #11: rq/worker.py:610: 4051645.7 KiB
> #12: plugin/repo_version_utils.py:31: 4028192.9 KiB
> #13: models/base.py:124: 4028191.2 KiB
> #14: fields/related_descriptors.py:401: 4028165.2 KiB
> #15: models/query.py:1242: 4021212.9 KiB
> #16: models/query.py:402: 3995284.5 KiB
> #17: models/query.py:256: 3995284.5 KiB
> #18: models/query.py:55: 3952718.2 KiB
> #19: sql/compiler.py:1133: 3919417.7 KiB
> #20: db/utils.py:96: 3912674.4 KiB
> #21: psycopg2/_json.py:166: 3837534.7 KiB
> #22: json/decoder.py:337: 3837533.2 KiB
> obj, end = self.raw_decode(s, idx=_w(s, 0).end())
> #23: json/__init__.py:348: 3837533.2 KiB
> return _default_decoder.decode(s)
> #24: json/decoder.py:353: 3837532.7 KiB
> obj, end = self.scan_once(s, idx)
> #25: rq/worker.py:670: 211891.5 KiB
> #26: tasking/worker.py:69: 198267.0 KiB
> #27: rq/worker.py:477: 191261.5 KiB
> #28: cli/cli.py:252: 186110.9 KiB
> #29: click/core.py:555: 185957.8 KiB
> #30: cli/cli.py:78: 104822.7 KiB
> #31: models/query.py:73: 48350.9 KiB
> #32: models/base.py:513: 48350.9 KiB
> #33: sql/compiler.py:405: 38126.1 KiB
> #34: sql/compiler.py:1087: 33355.6 KiB
> #35: click/core.py:956: 29580.3 KiB
> #36: click/core.py:1137: 29498.3 KiB
> #37: models/base.py:430: 29340.9 KiB
> #38: models/query.py:274: 25946.8 KiB
> #39: 

Re: [Pulp-dev] [Breaking Change] Typed Repositories

[Daniel Alley]

>
> Another reply to this thread will be sent once this is fully merged.
> Please share any concerns, corrections, questions, or ideas!


As of today, this is now merged.  Thank you to all of the people who
contributed to this change, especially @gmbnomis who "accepted the
challenge" and got this work unblocked.

- Daniel

On Sat, Nov 9, 2019 at 11:54 AM Brian Bouterse  wrote:

> I'm excited to share that typed repositories is going forward, and will
> likely merge on Monday; see PRs here:
> https://pulp.plan.io/issues/5625#note-7. A huge thanks to @gmbnomis for
> unblocking the work with code and addressing the written concerns with his
> post here: https://pulp.plan.io/issues/5625#note-12 I'd also like to call
> out the great work from @dalley, @dkliban, @ttereshc, @ipanova, and
> @davidavis.
>
>
> # What does this mean for integrators/users?
>
> 1) Instead of all repositories living at /pulp/api/v3/repositories/ each
> plugin will have a repositories endpoint. For example, pulp_file would be
> /pulp/api/v3/repositories/file/file/.
> 2) RepositoryVersions will continue to be nested under the new repository
> endpoints, e.g. /pulp/api/v3/repositories/file/file//versions/
> 3) Operations that create a new RepositoryVersion are moved to the
> repository URL itself or an action URL underneath it. For example, 'sync'
> is no longer on a Remote, but should be an action endpoint under repository
> if plugin support it. For example, here's how to trigger a sync for
> pulp_file.
>
> POST /pulp/api/v3/repositories/file/file/sync/ remote={remote_href}.
>
>
> # What does this mean for plugin writers?
>
> You'll need to define several new objects:
>
> * A detail Repository, e.g. pulp_file's
> https://github.com/pulp/pulp_file/pull/299/files#diff-7b38205db1c4e6004b1a30ea0a2db477R41
> * A RepositorySerializer, e.g. pulp_file's
> https://github.com/pulp/pulp_file/pull/299/files#diff-0f3aaa9013e997a4c15352be12eb1881R60
> * A RepositoryViewSet, e.g. pulp_file's
> https://github.com/pulp/pulp_file/pull/299/files#diff-4ea13541b01c775a55f25a8c4fab0aebR66
> * A RepositoryVersionViewset, e.g. pulp_file's
> https://github.com/pulp/pulp_file/pull/299/files#diff-4ea13541b01c775a55f25a8c4fab0aebR103
>
>
> # Next Steps
>
> Another reply to this thread will be sent once this is fully merged.
> Please share any concerns, corrections, questions, or ideas!
>
> Thank you to everyone for the excellent collaboration and contribution.
> Pulp's users and its API is significantly better for this.
>
> -Brian
>
>
> On Wed, Nov 6, 2019 at 2:29 PM Brian Bouterse  wrote:
>
>> After much, great work from @dalley on the typed repository prototype,
>> unfortunately, we believe it cannot be adopted at this time. Here's a
>> writeup of why:  https://pulp.plan.io/issues/5625#note-8 and another
>> writeup of alternatives considered:
>> https://pulp.plan.io/issues/5625#note-9
>>
>> This writeup came from a discussion with other core devs giving input on
>> various options as well. If we've misread the opportunity here please let
>> us know!
>>
>> The primary motivation for typed repositories was to resolve the design
>> challenges encountered on https://pulp.plan.io/issues/3541 I'm going to
>> do the following ASAP:
>>
>> 1) update 3541's design with the input from the thread:
>> https://www.redhat.com/archives/pulp-dev/2019-November/msg00012.html
>> 2) create proof of concept PRs for the changes described so others can
>> see asap.
>>
>>
>>
>>
>> On Sun, Nov 3, 2019 at 11:54 AM Daniel Alley  wrote:
>>
>>> In accordance with the rationale laid out in issue #5625 [0], we will be
>>> merging a change that will make repositories a typed object in Pulp.  This
>>> will require some work by plugin writers to become compatible post-merge.
>>>
>>> We're aiming to merge these changes on Wednesday if possible.  There is
>>> still a bit of work to be done to fix the client REST API bindings, and to
>>> get all of the plugins migrated.  The PRs are being tracked in the comments
>>> of the issue [1].
>>>
>>> [0] https://pulp.plan.io/issues/5625
>>> [1] https://pulp.plan.io/issues/5625#note-7
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [noissue] considered harmful

[Daniel Alley]

Agreed, I would also rather keep [noissue].  That is a good example of when
it is useful.  We can and should pay more attention to when it's being used
in the PR review process, though.

On Wed, Nov 6, 2019 at 12:26 PM Tatiana Tereshchenko 
wrote:

> I'm leaning towards keeping [noissue] and have it a responsibility of PR
> reviewer/merger to evaluate if it's a proper use or an issue is needed.
>
> Alternatively, can we block a merge of PR if there is no issue? at github
> level, not in travis? so tests run but you can only merge it "using your
> admin privileges" by pressing a very red button :)
> One good case for noissue is all the release PRs I think. So if we let
> tests run regardless of the issue specified in the commit and only block
> merge, it's probably fine to press the red button for the release PRs only.
>
> On Wed, Nov 6, 2019 at 3:55 PM Matthias Dellweg  wrote:
>
>> Being a regular user of noissue, i think it is ok to keep it, if the
>> reviewer can tell the contributer at some time that the change is
>> effecting something worth noting in the changelog. Sometimes, what
>> starts as a very small change evolves into something bigger.
>> Sometimes you might even want to show some code without knowing whether
>> there is an issue at all. Removing this loophole would prevent those
>> code paths from ever running in the ci.
>>
>> I'd vote for not removing the [noissue] functionality, but instead keep
>> an open eye on it's usage, and in case request that a ticket be opened.
>>
>> On Wed, 6 Nov 2019 08:52:14 -0500
>> David Davis  wrote:
>>
>> > When we added commit validation to our CI, we created a loophole that
>> > would allow small changes like typo fixes to not have a redmine issue
>> > or changelog entry. By having '[noissue]' in the commit message,
>> > users could bypass our commit requirements. However, this loophole is
>> > not being used as intended--it's being used for all sorts of changes
>> > from new features to bug fixes.
>> >
>> > This is going to be a major problem for users and plugin writers when
>> > we release 3.0 and they start to upgrade from release to release.
>> > These [noissue] changes won't being reflected in our changelog which
>> > invalidates the motivation behind having a changelog.
>> >
>> > My first inkling is to remove the '[noissue]' functionality. Other
>> > projects like Katello and foreman already do this. If you don't want
>> > your change reflected in the changelog, you should be creating .misc
>> > entries for these changes in the CHANGES folder. The downside to this
>> > is every change will require a changelog and issue, and this creates
>> > extra steps for devs and contributors.
>> >
>> > Another option is to make our check smarter by incorporating some
>> > other condition(s) or metric(s). For example, perhaps we could allow
>> > [noissue] changes only to certain directories like our tests
>> > directory. The problem here is getting the parameters right. What
>> > sorts of changes changes should we allow to be [noissue] ones? I've
>> > thought about this a bit and couldn't think of a good set of criteria.
>> >
>> > Thoughts?
>> >
>> > David
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] [Breaking Change] Typed Repositories

[Daniel Alley]

In accordance with the rationale laid out in issue #5625 [0], we will be
merging a change that will make repositories a typed object in Pulp.  This
will require some work by plugin writers to become compatible post-merge.

We're aiming to merge these changes on Wednesday if possible.  There is
still a bit of work to be done to fix the client REST API bindings, and to
get all of the plugins migrated.  The PRs are being tracked in the comments
of the issue [1].

[0] https://pulp.plan.io/issues/5625
[1] https://pulp.plan.io/issues/5625#note-7
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Merging pulpcore.plugin into pulp/pulpcore repo?

[Daniel Alley]

This will hopefully be merged today across all plugins:
https://pulp.plan.io/issues/5580#note-8

On Wed, Oct 23, 2019 at 5:19 AM Brian Bouterse  wrote:

> Thanks @dalley!
>
> Yes the Pulp developers talked this over; here's what we came up with
> (bulleted takeaways). Overall we want to make this a lot simpler, so we do
> want to go forward this week with a few things:
>
> # Actions
> 1. Move the relevant bits from pulp/pulpcore-plugin to pulpcore.plugin in
> the pulp/pulpcore repo
> 2. Delete pulp/pulpcore-plugin repository
> 3. pulpcore will be the only versioned package, and will start with 3.0
> 4. A plugin API changelog will and plugin API docs will be available in
> the existing site in pulpcore
> 5. Breaking changes for the pulpcore.plugin will be released only with 3.y
> releases so any 3.0.z changes should stay plugin compatible
>
> # Implications for Plugin Writers
> 1. Plugins should probably pin to pulpcore<=3.y+1 to guard against
> pulpcore.plugin breaking changes in the 3.y
> 2. Must inspect the changelog API with each 2.y release to determine if
> your plugin is compatible
> 3. Plugin writers must only import from pulpcore.plugin (still)
>
> On Wed, Oct 23, 2019 at 10:13 AM Daniel Alley  wrote:
>
>> Since we decided to move forwards with this, here are the PRs:
>> https://pulp.plan.io/issues/5580#note-8
>>
>> On Fri, Oct 18, 2019 at 5:29 AM Ina Panova  wrote:
>>
>>> +1 let's talk about this more next week.
>>>
>>>
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Senior Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>>
>>>
>>> On Thu, Oct 17, 2019 at 7:25 PM David Davis 
>>> wrote:
>>>
>>>> I agree that it makes sense to talk about it next week. I have a few
>>>> concerns both in favor and against merging the repos that are not expressed
>>>> on the issue. Let's remove it off the sprint for now.
>>>>
>>>> David
>>>>
>>>>
>>>> On Thu, Oct 17, 2019 at 1:13 PM Brian Bouterse 
>>>> wrote:
>>>>
>>>>> Thank you for sharing your concerns. Let's think it over more.
>>>>>
>>>>>
>>>>> On Thu, Oct 17, 2019 at 12:39 PM Tatiana Tereshchenko <
>>>>> ttere...@redhat.com> wrote:
>>>>>
>>>>>> Do I understand correctly, that the suggestion is not to have
>>>>>> versioned api at all but document some conventions/expectations between
>>>>>> pulpcore releases and plugin API?
>>>>>>
>>>>> We could version pulpcore.plugin as 0.1 as we were. It's just that
>>>>> plugins can't use setuptools tooling to declare their compatibility. The
>>>>> pulp plugin loader could check plugin minimum/maximum versions declared at
>>>>> runtime though and disinclude plugins that aren't compatible with the
>>>>> running pulpcore version. That would be a neat upgrade experience actually
>>>>> whenever the pulpcore.plugin version goes to 0.2 which would be backwards
>>>>> incompatible.
>>>>>
>>>>>
>>>>>> I wonder if there is still a benefit to keep it as a separate package
>>>>>> and as-is.
>>>>>>
>>>>> There is a benefit to keeping it as-is. What we have now is slightly
>>>>> more awesome. The amount of problems it creates though is a lot.
>>>>>
>>>>> I understand that now we need to release both at the same time,
>>>>>> however we are in active development now and when things are more stable 
>>>>>> we
>>>>>> might benefit from having it separate.
>>>>>>
>>>>> The code imports heavily across the package lines, so I think we'll
>>>>> always have to co-release, which is the pain because you have to wait a
>>>>> long time for Travis to fully retest + release.
>>>>>
>>>>>>
>>>>>> It sounds like we are back to pulp2 case, where we would need to
>>>>>> release the core if any plugin api changes are needed, just this time we
>>>>>> organised code better.
>>>>>>
>>>>> We could still have a versioned pulpcore.plugin which would be much
>>>>> stronger.
>>>>>
>>>>>
>>>>>&g

Re: [Pulp-dev] PulpCon Community Day videos

[Daniel Alley]

Link correction: https://bluejeans.com/s/4zL_F

On Wed, Oct 23, 2019 at 6:59 AM Daniel Alley  wrote:

> Big thanks to all of our speakers yesterday!
>
>
> https://redhat.bluejeans.com/playback/guid/OTQ4NDEwNzA1NjozOTU2MjgtNTJkMThlZjQtNjMxNC00OTExLTkwZWEtNjIxNjIwMzVkMTRl
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] PulpCon Community Day videos

[Daniel Alley]

Big thanks to all of our speakers yesterday!

https://redhat.bluejeans.com/playback/guid/OTQ4NDEwNzA1NjozOTU2MjgtNTJkMThlZjQtNjMxNC00OTExLTkwZWEtNjIxNjIwMzVkMTRl
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Merging pulpcore.plugin into pulp/pulpcore repo?

[Daniel Alley]

49 AM Mike DePaulo 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Thu, Oct 17, 2019 at 10:25 AM Brian Bouterse <
>>>>>>>> bmbou...@redhat.com> wrote:
>>>>>>>>
>>>>>>>>> I put some responses inline. I'm interested in what you think.
>>>>>>>>>
>>>>>>>>> On Thu, Oct 17, 2019 at 10:10 AM Mike DePaulo <
>>>>>>>>> mikedep...@redhat.com> wrote:
>>>>>>>>>
>>>>>>>>>> Q: Will both pulpcore & pulpcore-plugin be published on PyPI as 1
>>>>>>>>>> package, or as 2?
>>>>>>>>>>
>>>>>>>>> I had imagined it would be 1. Users wouldn't be able to receive
>>>>>>>>> pulpcore.plugin except through an install of the pulpcore==3.y.z 
>>>>>>>>> itself.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Great, that makes things easier for users' manual installs, and
>>>>>>>> easier for containers/packaging.
>>>>>>>>
>>>>>>>>
>>>>>>>>>> If so, how will you pip install from a pip VCS URL
>>>>>>>>>> <https://pip.pypa.io/en/stable/reference/pip_install/#id32>?
>>>>>>>>>> With #egg=pulpcore , and then #egg=pulpcore-plugin ?
>>>>>>>>>>
>>>>>>>>> You wouldn't perform the second one anymore.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Great, that makes things easier for users' manual installs, and
>>>>>>>> easier for containers/packaging.
>>>>>>>>
>>>>>>>> -Mike
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> -Mike
>>>>>>>>>>
>>>>>>>>>> On Thu, Oct 17, 2019 at 5:17 AM Simon Baatz 
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> +1
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Oct 16, 2019 at 03:50:54PM -0400, Mike DePaulo wrote:
>>>>>>>>>>> >+1.
>>>>>>>>>>> >
>>>>>>>>>>> >On Wed, Oct 16, 2019 at 3:26 PM Pavel Picka <[1]
>>>>>>>>>>> ppi...@redhat.com>
>>>>>>>>>>> >wrote:
>>>>>>>>>>> >
>>>>>>>>>>> >+1
>>>>>>>>>>> >
>>>>>>>>>>> >On Wed, Oct 16, 2019 at 9:23 PM Dana Walker <[2]
>>>>>>>>>>> dawal...@redhat.com>
>>>>>>>>>>> >wrote:
>>>>>>>>>>> >
>>>>>>>>>>> >+1, more straightforward
>>>>>>>>>>> >
>>>>>>>>>>> >Dana Walker
>>>>>>>>>>> >
>>>>>>>>>>> >She / Her / Hers
>>>>>>>>>>> >
>>>>>>>>>>> >Software Engineer, Pulp Project
>>>>>>>>>>> >
>>>>>>>>>>> >[3]Red Hat
>>>>>>>>>>> >
>>>>>>>>>>> >[4]dawal...@redhat.com
>>>>>>>>>>> >[5][Logo-RedHat-Email.png]
>>>>>>>>>>> >
>>>>>>>>>>> >On Wed, Oct 16, 2019 at 2:55 PM David Davis <[6]
>>>>>>>>>>> davidda...@redhat.com>
>>>>>>>>>>> >wrote:
>>>>>>>>>>> >
>>>>>>>>>>> >+1
>>>>>>>>>>> >David
>>>>>>>>>>> >
>>>>>>>>>>> >On Wed, Oct 16, 2019 at 2:38 PM Daniel Alley <[7]
>>>>>>>>>>> dal...@redhat.com>
>>>>>>>>>>> >wrote:
>>>>>>>>>>> >
>>

Re: [Pulp-dev] Merging pulpcore.plugin into pulp/pulpcore repo?

[Daniel Alley]

Very Large +1

On Wed, Oct 16, 2019 at 2:10 PM Brian Bouterse  wrote:

> Having just released RC7, there are a variety of problems we are dealing
> with as a result of having pulpcore and pulpcore-plugin being in separate
> repos. @daviddavis and I were talking, and we want to ask for feedback on
> merging the code from pulpcore-plugin into pulpcore.
>
> I wrote this up as an issue here:  https://pulp.plan.io/issues/5580
>
> Thanks!
> Brian
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] [BREAKING] Custom JSONField

[Daniel Alley]

Pulp 3 has previously exposed a custom JSONField implementation (backed by
a normal Django TextField) for use by plugins.  With the decision to
specialize on Postgres, this is no longer necessary as Postgres has it's
own native and well-supported JSONField implementation.  Thus,
"pulpcore.plugin.fields.JSONField" will be removed in the next release of
pulpcore and pulpcore-plugin.

PRs:

https://github.com/pulp/pulpcore/pull/301
https://github.com/pulp/pulpcore-plugin/pull/132
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulp 2.21.0 Dev Freeze

[Daniel Alley]

The code for 2.21.0 is frozen. The list of features and fixes can be found
here [0].
The RC build is expected on September 18, 2019 (one week from today), see
the release schedule [1].

[0] https://pulp.plan.io/issues?query_id=61
[1] https://pulp.plan.io/projects/pulp/wiki/2210_Release_Schedule

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulp 2.21.0 schedule

[Daniel Alley]

A 2.21.0 release is being planned with some features and recent fixes. Here
[0] is a release schedule page which outlines some tentative dates,
starting with a dev freeze on September 11th, 2019 @ 22:00 UTC.

The full list of features and bug fixes to be released can be found here
[1].

[0] https://pulp.plan.io/projects/pulp/wiki/2210_Release_Schedule
[1] https://pulp.plan.io/issues?query_id=61
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Namespacing one shot upload and copy endpoints

[Daniel Alley]

Ina and I discussed this earlier today and she made good points -- she
pointed out that what I suggested wouldn't work for content types that have
relations on non-content models, as several content types in the Docker and
Python plugins do. So, I no longer think it's a good idea to have an
endpoint with "types=[]" in core.

However, I still think that the "types=[]" pattern doesn't compose very
well with other features like filtering so I'm not sure if we want to adopt
that yet.

On Thu, Aug 1, 2019 at 8:54 AM Ina Panova  wrote:

> For the one shot upload usecase I tend to lean towards
> /v3///upload/ and for content_types that require
> special treatment we can define separate endpoints.
> If talking about modulemd or modulemd_defaults, it could be
> /v3/rpm/modules/upload/.
>
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Wed, Jul 31, 2019 at 1:04 PM Tatiana Tereshchenko 
> wrote:
>
>> If the goal is to make endpoints unified across all actions, then I think
>> we can only do
>> POST /pulp/api/v3//plugin/action/ types=[]
>>
>> Having plugin/content_type/upload would be nice, however I'm not sure if
>> it covers enough use cases.
>> E.g. For pulp_rpm,  it makes sense for packages or advisories to have a
>> dedicated endpoint each, however it doesn't make much sense for modulemd or
>> modulemd_defaults, because usually they are in the same file and uploaded
>> in bulk (maybe a separate endpoint is needed for this case).
>>
>> For the copy case, it's common to copy more than one type, I think, so
>> probably 'plugin/copy/ types=[]' makes more sense.
>>
>> It would be great to here from more people and other plugins.
>>
>>
>>
>> On Mon, Jul 29, 2019 at 5:46 PM Pavel Picka  wrote:
>>
>>> +1 for discuss this to keep some standard as I have already opened PRs
>>> for rpm modulemd[-defaults].
>>> I like idea of /upload in the end.
>>> But also think it can work without as it will be differ by POST/GET
>>> methods.
>>>
>>> On Mon, Jul 29, 2019 at 4:49 PM Dana Walker  wrote:
>>>
 Just to provide an added data point, I'll be merging the one-shot PR
 for pulp_python soon and it currently uses /api/v3/python/upload/

 I wanted to keep it simple as well, and so would be happy to change it
 for consistency based on whatever we decide.

 --Dana

 Dana Walker

 She / Her / Hers

 Software Engineer, Pulp Project

 Red Hat 

 dawal...@redhat.com
 



 On Mon, Jul 29, 2019 at 10:42 AM Ina Panova  wrote:

> Hi all,
> As of today, plugins have the freedom to define whichever endpoints
> they want ( to some extent).
> This leads to the question - shall we namespace one-shot upload and
> copy endpoints for some consistency?
>
> POST /api/v3/content/rpm/packages/upload/
> POST /api/v3/content/rpm/packages/copy/
>
> or
>
> POST /api/v3/content/rpm/upload/ type =package
> POST /api/v3/content/rpm/copy/ type = [package, modulemd]
>
> I wanted to bring this up, before it diverges a lot. For the record, I
> have checked only RPM plugin, I am not aware of the state of the other
> plugins.
> Right now we have an active endpoint for one-shot upload of rpm
> package:
> POST /api/v3/content/rpm/upload/
>
> And there is PR for one-shot upload of modulemd-defaults:
> POST /api/v3/content/rpm/modulemd-defaults/
>
> For rpm copy we have POST /api/v3/content/rpm/copy/ types=[]
>
> We are starting some work on docker recursive copy, so it would be
> helpful to reach some agreement before going further that path.
>
> Thank you!
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>>
>>>
>>> --
>>> Pavel Picka
>>> Red Hat
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com

Re: [Pulp-dev] Namespacing one shot upload and copy endpoints

[Daniel Alley]

I suppose I also worry that doing a switch on "type=" means that adding any
new content type to a plugin means our endpoints would be full of

if type == X:
>
...
>
elif type == Y:
>
...
>

and about the complexity of that vs the complexity of separate endpoints
once we start gradually more and more advanced features.

Also, if we want a feature for some content types in a plugin but not for
others, if we only have a few heavily overloaded endpoint for copy and
upload, then how do we keep it documented what is supported for type X and
what is not supported?  And if something is not supported then how do we
tell the user this?  The error checking would be pushed down into the
business logic, instead of whether the endpoint accepts a parameter or not.

So, I suppose I prefer this scheme

POST /api/v3/content/rpm/packages/upload/
> POST /api/v3/content/rpm/packages/copy/
>

And pushing as many of the truly generic features to core as possible, such
as copying all units of certain type(s).

On Wed, Jul 31, 2019 at 12:43 PM Daniel Alley  wrote:

> revision to use case C)
>
> copying all units of a type (e.g. RPM) --> copying all units of one or
> many types (e.g. RPM)
>
> On Wed, Jul 31, 2019 at 12:40 PM Daniel Alley  wrote:
>
>> For the copy case, it's common to copy more than one type, I think, so
>>> probably 'plugin/copy/ types=[]' makes more sense.
>>>
>>
>> Thanks for this thread. I'm +1 on documenting these general conventions
>>> in the pulpcore-plugin docs for plugin writers. Maybe we could include a
>>> section on URLs so plugin writers could self-assess if they are doing this
>>> or not.
>>>
>>
>> Regarding POST /pulp/api/v3/plugin/action/ types=[]
>>
>> Since this doesn't need to do anything plugin specific (as the type is
>> part of the base content model), my thought was that we would just make it
>> part of core.  Individual plugins would then implement endpoints for more
>> specific behaviors.  I would hope that each plugin wouldn't need to
>> implement the exact same code to copy units based on their type.
>>
>> My thoughts are based on these use cases (I hope I'm not missing
>> anything):
>>
>> A) copying specific content units by content ID
>> B) copying the contents of entire repositories into other repositories
>> C) copying all units of a type (e.g. RPM)  from one repo to another, with
>> no filtering
>> D) copying units of a type (e.g. RPM) from one repo to another if they
>> match some user-provided filters (e.g. name, version)
>> E) copying units of a type (e.g. RPM) from one repo to another if they
>> match some user-provided filters (e.g. name, version), and also copying all
>> their dependencies (using various schemes)
>>
>> Providing a list of types is semi-incompatible with cases D and E
>> (because how would it know how to apply filters to multiple different types
>> at once?  it could probably be done, but it would probably be very messy
>> and very plugin-specific code) so I don't know if that is a good basis for
>> an API that all plugins would build on top of.  On the other hand, A, B,
>> and C can all be accomplished without any plugin involvement -- core could
>> handle those use cases fine, and plugins would be free to handle D and E
>> however they see fit (although we probably do want some common patterns
>> there).
>>
>> But the common patterns for solving D and E are probably going to be
>> different from what we're discussing at the moment.  So I'm not sure if we
>> want to codify POST /pulp/api/v3/plugin/action/ types=[] as our common
>> pattern just yet, at least not without more discussion.
>>
>> On Wed, Jul 31, 2019 at 7:04 AM Tatiana Tereshchenko 
>> wrote:
>>
>>> If the goal is to make endpoints unified across all actions, then I
>>> think we can only do
>>> POST /pulp/api/v3//plugin/action/ types=[]
>>>
>>> Having plugin/content_type/upload would be nice, however I'm not sure if
>>> it covers enough use cases.
>>> E.g. For pulp_rpm,  it makes sense for packages or advisories to have a
>>> dedicated endpoint each, however it doesn't make much sense for modulemd or
>>> modulemd_defaults, because usually they are in the same file and uploaded
>>> in bulk (maybe a separate endpoint is needed for this case).
>>>
>>> For the copy case, it's common to copy more than one type, I think, so
>>> probably 'plugin/copy/ types=[]' makes more sense.
>>>
>>> It would be great to here from more people and other plugins.
>>>
>>>
>>>
>>> On Mo

Re: [Pulp-dev] Namespacing one shot upload and copy endpoints

[Daniel Alley]

revision to use case C)

copying all units of a type (e.g. RPM) --> copying all units of one or many
types (e.g. RPM)

On Wed, Jul 31, 2019 at 12:40 PM Daniel Alley  wrote:

> For the copy case, it's common to copy more than one type, I think, so
>> probably 'plugin/copy/ types=[]' makes more sense.
>>
>
> Thanks for this thread. I'm +1 on documenting these general conventions in
>> the pulpcore-plugin docs for plugin writers. Maybe we could include a
>> section on URLs so plugin writers could self-assess if they are doing this
>> or not.
>>
>
> Regarding POST /pulp/api/v3/plugin/action/ types=[]
>
> Since this doesn't need to do anything plugin specific (as the type is
> part of the base content model), my thought was that we would just make it
> part of core.  Individual plugins would then implement endpoints for more
> specific behaviors.  I would hope that each plugin wouldn't need to
> implement the exact same code to copy units based on their type.
>
> My thoughts are based on these use cases (I hope I'm not missing anything):
>
> A) copying specific content units by content ID
> B) copying the contents of entire repositories into other repositories
> C) copying all units of a type (e.g. RPM)  from one repo to another, with
> no filtering
> D) copying units of a type (e.g. RPM) from one repo to another if they
> match some user-provided filters (e.g. name, version)
> E) copying units of a type (e.g. RPM) from one repo to another if they
> match some user-provided filters (e.g. name, version), and also copying all
> their dependencies (using various schemes)
>
> Providing a list of types is semi-incompatible with cases D and E (because
> how would it know how to apply filters to multiple different types at
> once?  it could probably be done, but it would probably be very messy and
> very plugin-specific code) so I don't know if that is a good basis for an
> API that all plugins would build on top of.  On the other hand, A, B, and C
> can all be accomplished without any plugin involvement -- core could handle
> those use cases fine, and plugins would be free to handle D and E however
> they see fit (although we probably do want some common patterns there).
>
> But the common patterns for solving D and E are probably going to be
> different from what we're discussing at the moment.  So I'm not sure if we
> want to codify POST /pulp/api/v3/plugin/action/ types=[] as our common
> pattern just yet, at least not without more discussion.
>
> On Wed, Jul 31, 2019 at 7:04 AM Tatiana Tereshchenko 
> wrote:
>
>> If the goal is to make endpoints unified across all actions, then I think
>> we can only do
>> POST /pulp/api/v3//plugin/action/ types=[]
>>
>> Having plugin/content_type/upload would be nice, however I'm not sure if
>> it covers enough use cases.
>> E.g. For pulp_rpm,  it makes sense for packages or advisories to have a
>> dedicated endpoint each, however it doesn't make much sense for modulemd or
>> modulemd_defaults, because usually they are in the same file and uploaded
>> in bulk (maybe a separate endpoint is needed for this case).
>>
>> For the copy case, it's common to copy more than one type, I think, so
>> probably 'plugin/copy/ types=[]' makes more sense.
>>
>> It would be great to here from more people and other plugins.
>>
>>
>>
>> On Mon, Jul 29, 2019 at 5:46 PM Pavel Picka  wrote:
>>
>>> +1 for discuss this to keep some standard as I have already opened PRs
>>> for rpm modulemd[-defaults].
>>> I like idea of /upload in the end.
>>> But also think it can work without as it will be differ by POST/GET
>>> methods.
>>>
>>> On Mon, Jul 29, 2019 at 4:49 PM Dana Walker  wrote:
>>>
>>>> Just to provide an added data point, I'll be merging the one-shot PR
>>>> for pulp_python soon and it currently uses /api/v3/python/upload/
>>>>
>>>> I wanted to keep it simple as well, and so would be happy to change it
>>>> for consistency based on whatever we decide.
>>>>
>>>> --Dana
>>>>
>>>> Dana Walker
>>>>
>>>> She / Her / Hers
>>>>
>>>> Software Engineer, Pulp Project
>>>>
>>>> Red Hat <https://www.redhat.com>
>>>>
>>>> dawal...@redhat.com
>>>> <https://www.redhat.com>
>>>>
>>>>
>>>>
>>>> On Mon, Jul 29, 2019 at 10:42 AM Ina Panova  wrote:
>>>>
>>>>> Hi all,
>>>>> As of today, plugins have the freedom to define whichever endpoints
>>>>

Re: [Pulp-dev] Namespacing one shot upload and copy endpoints

[Daniel Alley]

>
> For the copy case, it's common to copy more than one type, I think, so
> probably 'plugin/copy/ types=[]' makes more sense.
>

Thanks for this thread. I'm +1 on documenting these general conventions in
> the pulpcore-plugin docs for plugin writers. Maybe we could include a
> section on URLs so plugin writers could self-assess if they are doing this
> or not.
>

Regarding POST /pulp/api/v3/plugin/action/ types=[]

Since this doesn't need to do anything plugin specific (as the type is part
of the base content model), my thought was that we would just make it part
of core.  Individual plugins would then implement endpoints for more
specific behaviors.  I would hope that each plugin wouldn't need to
implement the exact same code to copy units based on their type.

My thoughts are based on these use cases (I hope I'm not missing anything):

A) copying specific content units by content ID
B) copying the contents of entire repositories into other repositories
C) copying all units of a type (e.g. RPM)  from one repo to another, with
no filtering
D) copying units of a type (e.g. RPM) from one repo to another if they
match some user-provided filters (e.g. name, version)
E) copying units of a type (e.g. RPM) from one repo to another if they
match some user-provided filters (e.g. name, version), and also copying all
their dependencies (using various schemes)

Providing a list of types is semi-incompatible with cases D and E (because
how would it know how to apply filters to multiple different types at
once?  it could probably be done, but it would probably be very messy and
very plugin-specific code) so I don't know if that is a good basis for an
API that all plugins would build on top of.  On the other hand, A, B, and C
can all be accomplished without any plugin involvement -- core could handle
those use cases fine, and plugins would be free to handle D and E however
they see fit (although we probably do want some common patterns there).

But the common patterns for solving D and E are probably going to be
different from what we're discussing at the moment.  So I'm not sure if we
want to codify POST /pulp/api/v3/plugin/action/ types=[] as our common
pattern just yet, at least not without more discussion.

On Wed, Jul 31, 2019 at 7:04 AM Tatiana Tereshchenko 
wrote:

> If the goal is to make endpoints unified across all actions, then I think
> we can only do
> POST /pulp/api/v3//plugin/action/ types=[]
>
> Having plugin/content_type/upload would be nice, however I'm not sure if
> it covers enough use cases.
> E.g. For pulp_rpm,  it makes sense for packages or advisories to have a
> dedicated endpoint each, however it doesn't make much sense for modulemd or
> modulemd_defaults, because usually they are in the same file and uploaded
> in bulk (maybe a separate endpoint is needed for this case).
>
> For the copy case, it's common to copy more than one type, I think, so
> probably 'plugin/copy/ types=[]' makes more sense.
>
> It would be great to here from more people and other plugins.
>
>
>
> On Mon, Jul 29, 2019 at 5:46 PM Pavel Picka  wrote:
>
>> +1 for discuss this to keep some standard as I have already opened PRs
>> for rpm modulemd[-defaults].
>> I like idea of /upload in the end.
>> But also think it can work without as it will be differ by POST/GET
>> methods.
>>
>> On Mon, Jul 29, 2019 at 4:49 PM Dana Walker  wrote:
>>
>>> Just to provide an added data point, I'll be merging the one-shot PR for
>>> pulp_python soon and it currently uses /api/v3/python/upload/
>>>
>>> I wanted to keep it simple as well, and so would be happy to change it
>>> for consistency based on whatever we decide.
>>>
>>> --Dana
>>>
>>> Dana Walker
>>>
>>> She / Her / Hers
>>>
>>> Software Engineer, Pulp Project
>>>
>>> Red Hat 
>>>
>>> dawal...@redhat.com
>>> 
>>>
>>>
>>>
>>> On Mon, Jul 29, 2019 at 10:42 AM Ina Panova  wrote:
>>>
 Hi all,
 As of today, plugins have the freedom to define whichever endpoints
 they want ( to some extent).
 This leads to the question - shall we namespace one-shot upload and
 copy endpoints for some consistency?

 POST /api/v3/content/rpm/packages/upload/
 POST /api/v3/content/rpm/packages/copy/

 or

 POST /api/v3/content/rpm/upload/ type =package
 POST /api/v3/content/rpm/copy/ type = [package, modulemd]

 I wanted to bring this up, before it diverges a lot. For the record, I
 have checked only RPM plugin, I am not aware of the state of the other
 plugins.
 Right now we have an active endpoint for one-shot upload of rpm package:
 POST /api/v3/content/rpm/upload/

 And there is PR for one-shot upload of modulemd-defaults:
 POST /api/v3/content/rpm/modulemd-defaults/

 For rpm copy we have POST /api/v3/content/rpm/copy/ types=[]

 We are starting some work on docker recursive copy, so it would be
 helpful to reach some agreement before going further 

Re: [Pulp-dev] Pinning dependencies in Pulp 3

[Daniel Alley]

+1 Y releases

On Tue, Jul 30, 2019 at 12:01 PM Brian Bouterse  wrote:

> +1 to pin Y releases
>
> On Tue, Jul 30, 2019 at 8:41 AM Tatiana Tereshchenko 
> wrote:
>
>> +1 to pin dependencies and use dependabot
>>
>> If we were to pin to Z releases, then we'd need to release pulp 3 package
>> with any Z release of any dependency we pin.
>> And in case of any [security] fix in any dependency, users would need to
>> wait for us to release pulp with updated dependency version.
>>
>> If my logic above is correct, I'm +1 to pin to Y releases. I think most
>> (if not all) breaking changes we observed were in the Y releases.
>>
>> Tanya
>>
>>
>>
>> On Fri, Jul 26, 2019 at 7:40 PM Brian Bouterse 
>> wrote:
>>
>>> +1. This brings increased stability to Pulp users, and keeps Pulp
>>> forward compatible with all dependency releases. It's the best of both
>>> worlds and automated!
>>>
>>> On Fri, Jul 26, 2019 at 12:33 PM Dennis Kliban 
>>> wrote:
>>>
 +1

 I really like that there is automation to help us update the deps. If
 the PR from dependabot passes CI, we can just merge. Otherwise we will file
 an issue.

 On Fri, Jul 26, 2019 at 11:38 AM David Davis 
 wrote:

> Recently, Pulp 3 package installs were broken by a new version of DRF
> which necessitated a new release of pulpcore (RC4)[0]. Our releases are
> fragile and unstable because they don't pin versions of dependencies.
>
> I was thinking of a new strategy whereby we pin pulpcore's
> dependencies to specific versions (either y or z releases) and we use
> something like dependabot[1] to notify us of new updates for pulpcore
> dependencies. It looks like it'll open new PRs when it detects a 
> dependency
> is out of date.
>
> The one downside I do see is that dependabot PRs could be ignored.
> However, I think the stability of our releases outweighs this potential
> risk especially as we get closer to GA.
>
> Thoughts?
>
> [0] https://www.redhat.com/archives/pulp-dev/2019-July/msg00076.html
> [1] https://dependabot.com/
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Install Pulp3 rpm plugin (createrepo_c install failure)

[Daniel Alley]

There's a couple of suggestions how to fix this in these issues

https://github.com/pypa/setuptools/issues/1694#issuecomment-466010982
https://github.com/pypa/setuptools/issues/1685#issuecomment-462942577

On Mon, Jul 29, 2019 at 7:37 AM Pavel Picka  wrote:

> Hey, anyone tried install pulp3 rpm plugin lately? after new bindings
> release I getting issue when trying install rpm plugin.
>
> Trying on our fedora29-source dev env with allowed system-wide packages
> installed pre-req like
>
> *sudo dnf install -y gcc make cmake bzip2-devel expat-devel file-devel
>> glib2-devel libcurl-devel libxml2-devel python3-devel rpm-devel
>> openssl-devel sqlite-devel xz-devel zchunk-devel zlib-devel libmodulemd
>> ninja-build doxygen python3-nose-testconfig graphviz python3-libmodulemd*
>>
>
> but getting
>
>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *$ pip install -e .Obtaining file:///home/vagrant/devel/pulp_rpm
>> Installing build dependencies ... done  Getting requirements to build wheel
>> ... error  ERROR: Command errored out with exit status 1:   command:
>> /usr/local/lib/pulp/bin/python3
>> /usr/local/lib/pulp/lib64/python3.7/site-packages/pip/_vendor/pep517/_in_process.py
>> get_requires_for_build_wheel /tmp/tmpm1syv8g5   cwd:
>> /home/vagrant/devel/pulp_rpm  Complete output (10 lines):  Traceback (most
>> recent call last):File
>> "/usr/local/lib/pulp/lib64/python3.7/site-packages/pip/_vendor/pep517/_in_process.py",
>> line 207, in   main()File
>> "/usr/local/lib/pulp/lib64/python3.7/site-packages/pip/_vendor/pep517/_in_process.py",
>> line 197, in main  json_out['return_val'] =
>> hook(**hook_input['kwargs'])File
>> "/usr/local/lib/pulp/lib64/python3.7/site-packages/pip/_vendor/pep517/_in_process.py",
>> line 48, in get_requires_for_build_wheel  backend = _build_backend()
>> File
>> "/usr/local/lib/pulp/lib64/python3.7/site-packages/pip/_vendor/pep517/_in_process.py",
>> line 39, in _build_backend  obj = getattr(obj, path_part)
>> AttributeError: module 'setuptools.build_meta' has no attribute
>> '__legacy__'  ERROR: Command
>> errored out with exit status 1: /usr/local/lib/pulp/bin/python3
>> /usr/local/lib/pulp/lib64/python3.7/site-packages/pip/_vendor/pep517/_in_process.py
>> get_requires_for_build_wheel /tmp/tmpm1syv8g5 Check the logs for full
>> command output.*
>>
>
> Same output for direct pip install createrepo_c.
> Any ideas if something missing or where to look?
>
> --
> Pavel Picka
> Red Hat
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposal to move Pulp 2 dev environment to CentOS

[Daniel Alley]

PR here: https://github.com/pulp/devel/pull/222

On Fri, Jul 26, 2019 at 4:49 AM Ina Panova  wrote:

> +1
>
>
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Wed, Jul 24, 2019 at 9:34 PM Matthias Dellweg  wrote:
>
>> +1
>>
>> > >>>>> Our Pulp 3 development environment already gives you the option
>> > >>>>> to pick between different Fedora versions, CentOS, CentOS +
>> > >>>>> FIPS, Debian, etc.
>>
>> And at least for developing pulp3 in the debian box, i can state that
>> it works like charm. Thank you for making this distro agnostic.
>>
>>
>> On Wed, 24 Jul 2019 13:53:52 -0400
>> Dana Walker  wrote:
>>
>> > +1
>> >
>> > Dana Walker
>> >
>> > She / Her / Hers
>> >
>> > Software Engineer, Pulp Project
>> >
>> > Red Hat <https://www.redhat.com>
>> >
>> > dawal...@redhat.com
>> > <https://www.redhat.com>
>> >
>> >
>> >
>> > On Wed, Jul 24, 2019 at 1:18 PM Grant Gainey 
>> > wrote:
>> >
>> > > +1
>> > >
>> > > On Wed, Jul 24, 2019 at 12:52 PM Kersom  wrote:
>> > >
>> > >> +1
>> > >>
>> > >> On Wed, Jul 24, 2019 at 12:33 PM Brian Bouterse
>> > >>  wrote:
>> > >>
>> > >>> +1
>> > >>>
>> > >>> On Wed, Jul 24, 2019 at 12:13 PM David Davis
>> > >>>  wrote:
>> > >>>
>> > >>>> +1
>> > >>>>
>> > >>>> David
>> > >>>>
>> > >>>>
>> > >>>> On Wed, Jul 24, 2019 at 11:50 AM Daniel Alley 
>> > >>>> wrote:
>> > >>>>
>> > >>>>> Absolutely!  This applies to Pulp 2 only.
>> > >>>>>
>> > >>>>> Our Pulp 3 development environment already gives you the option
>> > >>>>> to pick between different Fedora versions, CentOS, CentOS +
>> > >>>>> FIPS, Debian, etc.
>> > >>>>>
>> > >>>>>
>> > >>>>> On Wed, Jul 24, 2019 at 11:44 AM Neal Gompa 
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>>> On Wed, Jul 24, 2019, 11:40 AM Daniel Alley 
>> > >>>>>> wrote:
>> > >>>>>>
>> > >>>>>>> Proposal:  The pulp/devel repository will install a CentOS 7
>> > >>>>>>> base box instead of a Fedora 28 one
>> > >>>>>>>
>> > >>>>>>> Rationale:
>> > >>>>>>>
>> > >>>>>>> * CentOS / RHEL is what our users are using. Using a different
>> > >>>>>>> platform can result in different behavior in the development
>> > >>>>>>> environment vs what users see -- this happened to me recently
>> > >>>>>>> and wasted a couple of hours on both the Dev and QE sides
>> > >>>>>>> * Fedora 28 is EOL and no longer receiving updates
>> > >>>>>>> * Pulp 2 the custom dev installation script (pulp-dev.py) is
>> > >>>>>>> broken on Fedora 29+ and I deemed it probably not worth the
>> > >>>>>>> time or effort to fix
>> > >>>>>>> * The RPM plugin is also broken on Fedora 29+ -- legacy
>> > >>>>>>> "createrepo" was obsoleted entirely and as far as I can tell
>> > >>>>>>> you can't import some of the libraries we are using from that
>> > >>>>>>> package anymore
>> > >>>>>>> * There's little point in tracking fast-moving Fedora changes
>> > >>>>>>> at this point in Pulp 2's lifecycle anyways
>> > >>>>>>>
>> > >>>>>>> I have a branch that already works on CentOS.  It's a few
>> > >>>>>>> patches behind but would take about 5 minutes to get cleaned
>> > >>>>>>> up.
>> > >>>>>>>
>> > >>>>>>> Any objections?
>> > >>>>>>>
>> > >>>>>>
>> > >>>>>> This sounds good to me. I hope we are still developing and
>> > >>>>>> testing on Fedora for Pulp 3, though?
>> > >>>>>>
>> > >>>>> ___
>> > >>>>> Pulp-dev mailing list
>> > >>>>> Pulp-dev@redhat.com
>> > >>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>> > >>>>>
>> > >>>> ___
>> > >>>> Pulp-dev mailing list
>> > >>>> Pulp-dev@redhat.com
>> > >>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>> > >>>>
>> > >>> ___
>> > >>> Pulp-dev mailing list
>> > >>> Pulp-dev@redhat.com
>> > >>> https://www.redhat.com/mailman/listinfo/pulp-dev
>> > >>>
>> > >> ___
>> > >> Pulp-dev mailing list
>> > >> Pulp-dev@redhat.com
>> > >> https://www.redhat.com/mailman/listinfo/pulp-dev
>> > >>
>> > >
>> > >
>> > > --
>> > > Grant Gainey
>> > > Principal Software Engineer, Red Hat System Management Engineering
>> > > ___
>> > > Pulp-dev mailing list
>> > > Pulp-dev@redhat.com
>> > > https://www.redhat.com/mailman/listinfo/pulp-dev
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposal to move Pulp 2 dev environment to CentOS

[Daniel Alley]

Absolutely!  This applies to Pulp 2 only.

Our Pulp 3 development environment already gives you the option to pick
between different Fedora versions, CentOS, CentOS + FIPS, Debian, etc.


On Wed, Jul 24, 2019 at 11:44 AM Neal Gompa  wrote:

> On Wed, Jul 24, 2019, 11:40 AM Daniel Alley  wrote:
>
>> Proposal:  The pulp/devel repository will install a CentOS 7 base box
>> instead of a Fedora 28 one
>>
>> Rationale:
>>
>> * CentOS / RHEL is what our users are using. Using a different platform
>> can result in different behavior in the development environment vs what
>> users see -- this happened to me recently and wasted a couple of hours on
>> both the Dev and QE sides
>> * Fedora 28 is EOL and no longer receiving updates
>> * Pulp 2 the custom dev installation script (pulp-dev.py) is broken on
>> Fedora 29+ and I deemed it probably not worth the time or effort to fix
>> * The RPM plugin is also broken on Fedora 29+ -- legacy "createrepo" was
>> obsoleted entirely and as far as I can tell you can't import some of the
>> libraries we are using from that package anymore
>> * There's little point in tracking fast-moving Fedora changes at this
>> point in Pulp 2's lifecycle anyways
>>
>> I have a branch that already works on CentOS.  It's a few patches behind
>> but would take about 5 minutes to get cleaned up.
>>
>> Any objections?
>>
>
> This sounds good to me. I hope we are still developing and testing on
> Fedora for Pulp 3, though?
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Proposal to move Pulp 2 dev environment to CentOS

[Daniel Alley]

Proposal:  The pulp/devel repository will install a CentOS 7 base box
instead of a Fedora 28 one

Rationale:

* CentOS / RHEL is what our users are using. Using a different platform can
result in different behavior in the development environment vs what users
see -- this happened to me recently and wasted a couple of hours on both
the Dev and QE sides
* Fedora 28 is EOL and no longer receiving updates
* Pulp 2 the custom dev installation script (pulp-dev.py) is broken on
Fedora 29+ and I deemed it probably not worth the time or effort to fix
* The RPM plugin is also broken on Fedora 29+ -- legacy "createrepo" was
obsoleted entirely and as far as I can tell you can't import some of the
libraries we are using from that package anymore
* There's little point in tracking fast-moving Fedora changes at this point
in Pulp 2's lifecycle anyways

I have a branch that already works on CentOS.  It's a few patches behind
but would take about 5 minutes to get cleaned up.

Any objections?
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Require Django 2.2+ and PostgreSQL 9.6

[Daniel Alley]

We've already adopted Django 2.2 several months ago (in setup.py), so I'm
not sure if any action item is needed there

On Tue, Jul 23, 2019 at 10:42 AM Brian Bouterse  wrote:

> +1 to adopting the Django 2.2 since it's an LTS. We should get a ticket
> written for it.
>
> In terms of the installer, I think only CentOS current doesn't have 9.6+.
> I'm working on getting the installer to installer 9.6 via scl on CentOS 7
> now and I think I'm almost done https://pulp.plan.io/issues/5147
>
> We also need to document this requirement change (and the mariaDB
> dropping) in the docs. At some point later we could consider a runtime
> version check where Pulp refuses to migrate/start if PostgreSQL version <
> 9.6.
>
>
> On Tue, Jul 23, 2019 at 10:35 AM David Davis 
> wrote:
>
>> +1. I wonder how we require a minimum version of PostgreSQL though? Will
>> that be a documentation change or is there some way to require it
>> programmatically (e.g. in the pulp 3 installer)?
>>
>> David
>>
>>
>> On Tue, Jul 23, 2019 at 10:28 AM Tatiana Tereshchenko <
>> ttere...@redhat.com> wrote:
>>
>>> Any objections to require Django 2.2+? And PostgreSQL 9.6+?
>>>
>>> It's not clear yet which Django will be in Centos8 or EPEL 8, if any.
>>> But I was told by package maintainers that if it's added, then it will
>>> likely be an LTS version, so Django 2.2
>>> https://www.djangoproject.com/download/#supported-versions
>>>
>>> One of the nice features 2.2 offers is ignore_conflicts option for
>>> bulk_create which will be helpful for the migration tool
>>> https://docs.djangoproject.com/en/2.2/ref/models/querysets/#django.db.models.query.QuerySet.bulk_create
>>>
>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Database support in Pulp 3

[Daniel Alley]

+1 to remove MariaDB/MySQL support




On Mon, Jul 15, 2019 at 10:09 AM Dennis Kliban  wrote:

> +1 to drop support for mariadb
>
> On Mon, Jul 15, 2019 at 10:04 AM Austin Macdonald 
> wrote:
>
>> +1 to remove.
>>
>> On Sun, Jul 14, 2019 at 4:10 PM Brian Bouterse 
>> wrote:
>>
>>> I believe we have reached a point where Pulp (core and its plugins) can
>>> no longer support MariaDB due to technical problems. I've been an advocate
>>> for Pulp to support MariaDB because it's what our users want. The community
>>> survey has 16 respondents (IIRC) and 30% of them said they wanted to use
>>> MariaDB. Also, anecdotally at conference booths, users want choice in their
>>> database. To not give them that, we need good, technical reasons. I didn't
>>> feel we had enough of them before, but now I feel we're at a point where we
>>> aren't able to solve these issues; it's becoming a choice of giving users
>>> features they want versus db portability.
>>>
>>> Here are the main reasons I think about (mostly what others have also
>>> stated):
>>>
>>> * utf8 issues - 3-byte utf-8 issues   < is this fixable with just a
>>> schema change?
>>> * full text search - pulp_ansible recently implemented full text search.
>>> To my knowledge it's not possible with MySQL. full text search is amazing,
>>> and it's driving pulp_ansible to drop postgreSQL. I bet all plugins will
>>> want this. With MariaDB, there is no search.
>>> * mixed plugin support concerns - With some plugins dropping support for
>>> MariaDB, the pulp community will fracture based on what DB you choose to
>>> use because not all plugins can run in all places.
>>> * specific field support - plugin writers have expressed the desire to
>>> store json data when their plugins natively contain json data and perhaps
>>> you want to filter on it for example. Having mariaDB support prevents us
>>> from using the JSONField.
>>> * performance concerns - The performance testing showed that Pulp does
>>> run significantly slower
>>>
>>> If we drop MariaDB we should publish a blog post and drop it with RC4.
>>> To remove MariaDB testing from Travis I propose we remove it from the
>>> plugin_template and use the Travis CI tool from @dkliban to push that
>>> config out to all repositories.
>>>
>>> I'll be offline this week. I wanted to get this reply out there in the
>>> hope that you all can make and enact the final decision.
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>> On Thu, Jul 11, 2019 at 4:02 PM Daniel Alley  wrote:
>>>
>>>> One more note:  Not all MySQL / MariaDB installations support
>>>> transactions, which we use heavily (and rely on?)
>>>>
>>>>
>>>> https://docs.djangoproject.com/en/2.2/topics/db/transactions/#transactions-in-mysql
>>>>
>>>> On Thu, Jul 11, 2019 at 3:55 PM David Davis 
>>>> wrote:
>>>>
>>>>> Two plugins are currently running into issues trying to support
>>>>> mariadb/mysql. The pulp_ansible plugin is interested in adding full text
>>>>> search and JSONFields. Meanwhile, the pulp_python plugin is trying to 
>>>>> store
>>>>> emojis in text which mariadb/mysql doesn't handle well since it uses 
>>>>> 3-byte
>>>>> utf-8 by default[0]. Given such cases, I wonder if we'd be better served 
>>>>> by
>>>>> dropping mariadb/mysql support and going with Postgresql only. Gitlab
>>>>> recently came to a similar conclusion[1].
>>>>>
>>>>> I personally am hesitant to give up being database agnostic but we
>>>>> already don't support sqlite. Also, I see some advantages like utilizing
>>>>> several Postgresql-only features like extra field types, full text search,
>>>>> etc. Also, supporting multiple databases means we'll likely have to write
>>>>> db specific code in some places or have plugins that only work with 
>>>>> certain
>>>>> database types.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> [0]
>>>>> https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-utf8mb4-11761243e434
>>>>>  https://code.djangoproject.com/ticket/18392
>>>>> [1] https://about.gitlab.com/2019/06/27/removing-mysql-support/
>>>>>
>>>>> David
>>>>> ___
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev@redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>> ___
>>>> Pulp-dev mailing list
>>>> Pulp-dev@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Database support in Pulp 3

[Daniel Alley]

One more note:  Not all MySQL / MariaDB installations support transactions,
which we use heavily (and rely on?)

https://docs.djangoproject.com/en/2.2/topics/db/transactions/#transactions-in-mysql

On Thu, Jul 11, 2019 at 3:55 PM David Davis  wrote:

> Two plugins are currently running into issues trying to support
> mariadb/mysql. The pulp_ansible plugin is interested in adding full text
> search and JSONFields. Meanwhile, the pulp_python plugin is trying to store
> emojis in text which mariadb/mysql doesn't handle well since it uses 3-byte
> utf-8 by default[0]. Given such cases, I wonder if we'd be better served by
> dropping mariadb/mysql support and going with Postgresql only. Gitlab
> recently came to a similar conclusion[1].
>
> I personally am hesitant to give up being database agnostic but we already
> don't support sqlite. Also, I see some advantages like utilizing several
> Postgresql-only features like extra field types, full text search, etc.
> Also, supporting multiple databases means we'll likely have to write db
> specific code in some places or have plugins that only work with certain
> database types.
>
> Thoughts?
>
> [0]
> https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-utf8mb4-11761243e434
>  https://code.djangoproject.com/ticket/18392
> [1] https://about.gitlab.com/2019/06/27/removing-mysql-support/
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] black

[Daniel Alley]

+0

On Mon, Jun 17, 2019 at 1:15 PM Brian Bouterse  wrote:

> +1 to adopting this. Thank you @daviddavis for writing
>
> On Mon, Jun 10, 2019 at 1:58 PM David Davis  wrote:
>
>> I opened PUP-8 that proposes adopting black and pydocstyle[0] along with
>> a PR against pulpcore to demonstrate how it would change pulpcore's code.
>> Please review and respond with votes[2]. The deadline will be June 22, 2019.
>>
>> [0] https://github.com/pulp/pups/pull/17
>> [1] https://github.com/pulp/pulpcore/pull/170
>> [2] https://github.com/pulp/pups/blob/master/pup-0001.md#voting
>>
>> David
>>
>>
>> On Tue, Jun 4, 2019 at 2:53 PM Simon Baatz  wrote:
>>
>>> On Tue, Jun 04, 2019 at 08:25:47AM -0400, David Davis wrote:
>>> >I wanted to get feedback from the Pulp community on using black[0]
>>> to
>>> >auto-format our Pulp 3 code. I have some mixed feelings about it as
>>> I
>>> >see some potential benefits of using it but also some downsides as
>>> >well.
>>>
>>> +1 for black (pulp_cookbook uses black for a couple of months now).
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Docstring linting

[Daniel Alley]

+1 to create a PUP

(-0 to actually adopting it. No particularly strong feelings though.)

On Thu, Jun 6, 2019 at 4:21 PM Dana Walker  wrote:

> +1
>
> Dana Walker
>
> She / Her / Hers
>
> Software Engineer, Pulp Project
>
> Red Hat 
>
> dawal...@redhat.com
> 
>
>
>
> On Thu, Jun 6, 2019 at 1:34 PM Brian Bouterse  wrote:
>
>>
>>
>> On Wed, Jun 5, 2019 at 4:39 PM David Davis  wrote:
>>
>>> Given the generally favorable response so far to using black, I was
>>> thinking of writing up a PUP to add black into pulpcore, pulpcore-plugin,
>>> pulp_file, and pulp_template. And to make it the recommended format for
>>> plugins. I can include docstring linting in that PUP as well.
>>>
>> +1 this sounds good to me.
>>
>>
>>> David
>>>
>>>
>>> On Wed, Jun 5, 2019 at 9:25 AM Brian Bouterse 
>>> wrote:
>>>
 I'm +1 on merging the proposals; it just seems easier. If not, I'd
 bring it as a followup proposal because I see value in this docstring
 linting.

 On Tue, Jun 4, 2019 at 11:00 AM Matthias Dellweg 
 wrote:

> The core problem this proposal tried to counteract is, just like the
> one with black, inconsistency across different repositories in the pulp
> namespace. Some lint docstrings and others don't even adhere to the
> linted style. Given the architecture of flake8 this leads to strange
> effects when you try to lint your code in the pulplift boxes.
> So what i really am aiming for here is consistency wrt to docstrings
> and docstring linting. This sounds like beeing almost the same goal as
> the black proposal. It would be fine for me to even merge those
> proposals.
>
> Matthias
>
> On Tue, 4 Jun 2019 10:29:58 -0400
> David Davis  wrote:
>
> > Black doesn't format docstrings[0] so it won't really help us. Flake8
> > is a wrapper for a collection of tools and the one that lints
> > docstrings (pydocstyle[1]) can be run independently without flake8.
> > So I think this questions around how/if to lint docstrings and
> > whether or not we want to use black are independent.
> >
> > [0] https://github.com/python/black/issues/144
> > [1] https://github.com/PyCQA/pydocstyle
> >
> > David
> >
> >
> > On Tue, Jun 4, 2019 at 10:05 AM Brian Bouterse 
> > wrote:
> >
> > > @mdellweg if we adopt Black broadly, how does that affect your
> > > proposal here?
> > >
> > > On Tue, May 21, 2019 at 11:50 AM Austin Macdonald
> > >  wrote:
> > >
> > >> Something else to consider: some docstrings are rendered as
> > >> user-facing documentation in the autogenerated REST docs. This
> > >> means that docstring linting needs to be ignored for ViewSets. For
> > >> example, I have a PR open that alters pulp_file viewset docstrings
> > >> to contain html, to pass the linter, we have add docstring
> > >> exceptions to the flake8 config.
> > >>
> > >> My initial reaction is that we might be better off keeping the
> > >> flake8-docstring package out of pulplift, and we should also
> > >> remove it from travis.
> > >>
> > >>
> > >> On Tue, May 21, 2019 at 11:08 AM Matthias Dellweg <
> dell...@atix.de>
> > >> wrote:
> > >>
> > >>> tl;dr: Docstring linting is inconsistent across pulp
> repositories.
> > >>> To make it consistent, do we want to enforce it everywhere, and
> > >>> repair more than 700 findings?
> > >>>
> > >>> What started out as a oneliner [0] surfaced as a bigger problem:
> > >>>
> > >>> Whether flake8 performs linting on docstrings is solely dependent
> > >>> (afaik) on the existence of a specific python package
> > >>> (flake8-docstring) in the system.
> > >>> At the same time, there are repositories (pulpcore,
> > >>> pulpcore-plugin, ???) that do not install this package in their
> ci
> > >>> pipeline, as well as repos that do (pulp_deb, pulp_ansible, ???).
> > >>> So it is hard to select whether it should be installed in a
> > >>> pulplift source box.
> > >>> Not installing it means, there are linting errors showing up in
> > >>> travis only, however installing it will prevent linting pulpcore
> > >>> locally.
> > >>> That said, i think we should follow the same linting rules in all
> > >>> repositories, and more specific i tend to include docstring
> > >>> linting. However there are over 700 findings in pulpcore alone.
> > >>>
> > >>> [0] https://github.com/pulp/pulpcore/pull/138
> > >>> ___
> > >>> Pulp-dev mailing list
> > >>> Pulp-dev@redhat.com
> > >>> https://www.redhat.com/mailman/listinfo/pulp-dev
> > >>>
> > >> ___
> > >> Pulp-dev mailing list
> > >> Pulp-dev@redhat.com
> > >> 

Re: [Pulp-dev] Release Note Process Improvements

[Daniel Alley]

+1

On Tue, May 28, 2019 at 2:23 PM Dennis Kliban  wrote:

> +1
>
> I updated the task[0] slightly and marked it as groomed.
>
>
> [0] https://pulp.plan.io/issues/4875
>
> On Tue, May 28, 2019 at 12:14 PM Austin Macdonald 
> wrote:
>
>> The proposed changes look awesome! I'm +1 for moving forward with it for
>> pulpcore and pulpcore-plugin.
>>
>> If there is consensus (looks like we are close), lets go ahead. If anyone
>> has concerns, we also have the option to implement this change for one
>> plugin before we go all in.
>>
>> On Mon, May 27, 2019 at 5:26 AM Ina Panova  wrote:
>>
>>> +1
>>>
>>>
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Senior Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>>
>>>
>>> On Sat, May 25, 2019 at 10:18 PM Tatiana Tereshchenko <
>>> ttere...@redhat.com> wrote:
>>>
 +1 to improve release notes process

 If we decide to use PR numbers and not redmine issues in the release
 notes, then there will be no limitation/requirement to have a redmine issue
 to add something to the release notes.

 Tanya

 On Fri, May 24, 2019 at 3:46 PM David Davis 
 wrote:

> +1 to bmbouter's proposal and not including '[noissue]' items in
> release notes.
>
> David
>
>
> On Fri, May 24, 2019 at 3:52 AM Matthias Dellweg 
> wrote:
>
>> I am fine with stating "[noissue] means 'not worth mentioning in
>> release notes'".
>> This would require the reviewer to decide to tell the contributor: "We
>> want that to be part of the release notes. Please open up a ticket."
>> And that process scales better than handpicking the notes in the end.
>>
>> On Thu, 23 May 2019 16:22:36 -0400
>> Dana Walker  wrote:
>>
>> > My initial thought is this looks useful to the user and very clean.
>> > I've also found it to be a burden trying to write good release
>> notes,
>> > having to dig through commits and try to decide what's important
>> > enough and what's not, so +1 to trying to improve this process for
>> > both the releaser and user.
>> >
>> > However:
>> > "towncrier works best in a development system where all merges
>> involve
>> > closing a ticket."
>> > We frequently make use of "[noissue]" in our PRs, in part to lower
>> the
>> > burden on contributors making small fixes.  Would we want to move
>> to a
>> > model where we *must* have an issue?  Are we instead assuming those
>> > items are small enough that the user doesn't need to see it in the
>> > release notes?
>> >
>> > Thoughts?
>> >
>> > --Dana
>> >
>> > Dana Walker
>> >
>> > She / Her / Hers
>> >
>> > Software Engineer, Pulp Project
>> >
>> > Red Hat 
>> >
>> > dawal...@redhat.com
>> > 
>> >
>> >
>> >
>> > On Thu, May 23, 2019 at 3:49 PM Brian Bouterse > >
>> > wrote:
>> >
>> > > In discussion with some other devs, I've realized that pulpcore
>> and
>> > > pulpcore-plugin would benefit from better release notes. Here are
>> > > some of the reasons that have come up:
>> > >
>> > > * The release notes are incomplete. One person tries to go through
>> > > and write release notes just before the release happens, and by
>> > > that point, the number of changes are too many for this approach
>> to
>> > > produce complete and robust notes.
>> > > * They are hard to produce. Producing "all the release notes" is a
>> > > mentally difficult task.
>> > > * We try to substitute with Redmine, but this approach limits us
>> > > (a) it's now difficult and time consuming to see what changed, (b)
>> > > there is way more detail than you actually want, and they aren't
>> > > self-contained (can't be browsed off-line).
>> > > * overall all ^ leads to both users and plugin writers feeling
>> > > uncertain about what has changed in the last release, week, or
>> even
>> > > day.
>> > >
>> > > So what can we do? Recently I contributed to aiohttp and I found
>> > > their release note process light and easy. It produces
>> high-quality
>> > > release notes like these:
>> > > https://aiohttp.readthedocs.io/en/stable/changes.html
>> > >
>> > > You can read about their process here:
>> > >
>> https://aiohttp.readthedocs.io/en/stable/contributing.html#changelog-update
>> > > You can see some examples of these release note files in their
>> repo
>> > > here: https://github.com/aio-libs/aiohttp/tree/master/CHANGES
>> > > Overall it makes use of the towncrier project
>> > > https://github.com/hawkowl/towncrier
>> > >
>> > > What do you all think about trying something like this for
>> pulpcore
>> > > and 

[Pulp-dev] ergonomics of providing Pulp with lists of items

[Daniel Alley]

Providing Pulp with lists of values from the command line is rather
unweildy.  There's a lot of unnecessary escaping going on.

http POST :24817${REPO_HREF}versions/
add_content_units:="[\"$CONTENT_HREF\",\"$CONTENT_2_HREF\"]"

http POST http://localhost:24817/pulp/api/v3/rpm/copy/
source_repo=${SRC_REPO_HREF} dest_repo=${DEST_REPO_HREF}
types:="[\"errata\"]"

Tanya, Ina and myself thought it would be worth discussing the idea of
using something more ergonomic, like a comma-separated string.  This would
make the endpoints much easier to use manually.

http POST :24817${REPO_HREF}versions/
add_content_units="$CONTENT_HREF,$CONTENT_2_HREF"

http POST http://localhost:24817/pulp/api/v3/rpm/copy/
source_repo=${SRC_REPO_HREF} dest_repo=${DEST_REPO_HREF} types="errata"

On the other hand, we're planning to have an actual CLI, then this probably
isn't really an issue.  The way we're doing things now isn't wrong, it's
just frustrating to do from a shell.  But I don't know exactly what our CLI
plans are.

What are your thoughts?
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Content Copy (between repos)

[Daniel Alley]

The one-shot approach basically means that every plugin will need to
implement this separately from boilerplate code.  Therefore, I suppose the
best course of action is to write a story to implement this for Pulp RPM,
implement it there, and copy said boilerplate to the plugin template.

Any objections?

On Mon, Apr 8, 2019 at 10:35 AM Brian Bouterse  wrote:

>
>
> On Sat, Apr 6, 2019 at 11:48 AM Justin Sherrill 
> wrote:
>
>>
>> On 4/4/19 2:35 PM, Daniel Alley wrote:
>>
>> Content copy between repositories is critically important to Katello
>> integration and is something that we have not really addressed yet.  It
>> also needs to be completed before the RPM plugin can begin work on
>> depsolving.  The story that results from this discussion should probably be
>> put on one of the next 1-3 sprints and not wait any longer than that.
>>
>> Repositories are generic to all types of content, but copy operations
>> between repositories will need type-specific options defined by the plugin
>> for e.g. advanced copy w/ depsolving.  Therefore we need to find a design
>> for this functionality where it will make sense that makes sense from a
>> usage perspective and an implementation perspective.
>>
>> To get this discussion started, here's some suggestions about how this
>> could work from the user perspective (presented without much thought put
>> into how it would be implemented).
>>
>> Create a "Copier" object, like a Remote, that stores it's own settings
>> and has one or many endpoints (like "/sync/") that can be POSTed to and
>> return a task and create a new repository version.
>>
>> POST /pulp/v3/api/copiers/rpm/$endpoint_name/ content_units=[...]  [more
>> params...]
>>
>> The copier would store settings such as "recursive" and the "source" and
>> "destination" repositories.  And let's say they can be overriden.
>>
>> Or, create new endpoints without any associated DB models, like one-shot
>> upload does:
>>
>> variant 1:  POST /pulp/v3/api/content/rpm/packages/copy/
>> content_units=[...] source="..." destination="..."  [more params...]
>>
>> variant 2: POST  /pulp/v3/api/copy/rpm/package/ content_units=[...]
>> source="..." destination="..."  [more params...]
>>
>> Since basic copy support (just copying the units, no depsolving, etc.)
>> could in theory be implemented completely generically, it would be nice if
>> we could provide that for free somehow.  But I don't immediately see a way
>> of doing so.
>>
>> I welcome any suggestions or input.
>>
>> A 'copier' object that has to be CRUD'd seems like unnecessary
>> complication from a user perspective.  I would go with the 'one shot
>> approach' at first, i think most users are going to desire that over an
>> actual object.
>>
> The one shot also makes more sense to me. It has fewer steps for the user
> to take. I also don't see a generic version that core could offer to make
> this easier for plugin writers than it is today.
>
>> Justin
>>
>>
>>
>>
>> ___
>> Pulp-dev mailing 
>> listPulp-dev@redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 tag

[Daniel Alley]

>
> I feel like using the Pulp 3 tag is suboptimal since we are closing out
> most of our Pulp 2 issues and a majority of the new bugs/tasks/stories are
> for Pulp 3. Wouldn’t it make more sense to create a Pulp 2 tag and use it
> on the rare Pulp 2 issues that get filed?


Agreed, +1 to having a "Pulp 2" tag to replace the Pulp 3 tag.

On Thu, Apr 11, 2019 at 9:19 AM David Davis  wrote:

> I feel like using the Pulp 3 tag is suboptimal since we are closing out
> most of our Pulp 2 issues and a majority of the new bugs/tasks/stories are
> for Pulp 3. Wouldn’t it make more sense to create a Pulp 2 tag and use it
> on the rare Pulp 2 issues that get filed?
>
> David
>
>
> On Thu, Apr 11, 2019 at 8:27 AM Ina Panova  wrote:
>
>> +1 to handle tag during the triage process
>>
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Tue, Apr 2, 2019 at 3:53 PM Brian Bouterse 
>> wrote:
>>
>>> +1 to Tanya's suggestion.
>>>
>>> Also I have an action item to start a separate thread on how we can
>>> plan-and-then-close all pulp2 bugs that we are not actively working on. I
>>> wanted to mention it here to identify it as a possible resolution to this
>>> issue since it was related. I'll send this today.
>>>
>>> On Tue, Apr 2, 2019 at 7:54 AM Tatiana Tereshchenko 
>>> wrote:
>>>
 In my opinion, it's a part of triage or grooming process.
 Triage leader (for issues) and a person who grooms the task should pay
 attention to the tags.

 Tanya

 On Tue, Apr 2, 2019 at 1:47 PM David Davis 
 wrote:

> Looking in redmine at our current sprint, I see several issues that
> don’t have Pulp 3 tag that should. Here are some:
>
> https://pulp.plan.io/issues/4587
> https://pulp.plan.io/issues/4595
> https://pulp.plan.io/issues/4577
> https://pulp.plan.io/issues/4576
> https://pulp.plan.io/issues/4545
>
> Talking with the QE team, it’s
> difficult for them to get a feel for how much Pulp 2 vs Pulp 3 work there
> is. I guess we could try to get better about tagging issues when we create
> them?
>
> However, at least one issue is by someone outside the core Pulp team
> and I am not sure community members will know to tag their issues w/ Pulp 
> 3
> tag. Are there any other alternatives to ensuring Pulp 3 issues get
> properly tagged?
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] YouTube Custom URL

[Daniel Alley]

"Only way to be sure is to try it, but if no one has any other suggestions,
then the order we will try things would be Pulp, PulpProject, and PulpProj."

+1

On Thu, Apr 11, 2019 at 8:47 AM Dana Walker  wrote:

> So it turns out youtube doesn't reserve channels to match users.  The
> *user* Pulp and the *user* PulpProject is taken, but the *channel* is at 
> *youtube.com/c/yourcustomname
>  *and it appears that when you add
> in the /c/ those are not taken.  Only way to be sure is to try it, but if
> no one has any other suggestions, then the order we will try things would
> be Pulp, PulpProject, and PulpProj.
>
> *--*Dana
>
> Dana Walker
>
> Software Engineer
>
> Red Hat
>
> 
>
> 
>
>
> On Thu, Apr 11, 2019 at 8:26 AM Ina Panova  wrote:
>
>> turns out this is occupied as well
>> https://www.youtube.com/user/PulpProject
>>  so +1 to the one as our twitter handle
>>
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Thu, Apr 11, 2019 at 2:22 PM Ina Panova  wrote:
>>
>>> +1 PulpProject
>>>
>>>
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>>
>>>
>>> On Tue, Mar 26, 2019 at 6:50 PM Brian Bouterse 
>>> wrote:
>>>
 +1 to PulpProject (like the site)

 On Mon, Mar 25, 2019 at 2:05 PM Mike DePaulo 
 wrote:

> B) PulpProject (like the site)
>
> On Fri, Mar 22, 2019 at 4:22 PM Bruno Rocha  wrote:
>
>> My vote is for B) PulpProject like the site
>>
>> Em sex, 22 de mar de 2019 17:09, Dennis Kliban 
>> escreveu:
>>
>>> PulpProj
>>>
>>> On Fri, Mar 22, 2019 at 4:07 PM Dana Walker 
>>> wrote:
>>>
 Hey!

 We've grown and now qualify for a custom URL on our YouTube
 Channel--but which one?  youtube.com/Pulp is already taken, but I
 suggest either:

 A) youtube.com/PulpProj (like our Twitter handle)
 or
 B) youtube.com/PulpProject (like our website)

 You can change capitalization as much as you want with YouTube urls
 and they will all be automagically converted to all lowercase and 
 directed
 to the same place.

 Survey says?

 --Dana

 Dana Walker

 Associate Software Engineer

 Red Hat

 
 
 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Content Copy (between repos)

[Daniel Alley]

Content copy between repositories is critically important to Katello
integration and is something that we have not really addressed yet.  It
also needs to be completed before the RPM plugin can begin work on
depsolving.  The story that results from this discussion should probably be
put on one of the next 1-3 sprints and not wait any longer than that.

Repositories are generic to all types of content, but copy operations
between repositories will need type-specific options defined by the plugin
for e.g. advanced copy w/ depsolving.  Therefore we need to find a design
for this functionality where it will make sense that makes sense from a
usage perspective and an implementation perspective.

To get this discussion started, here's some suggestions about how this
could work from the user perspective (presented without much thought put
into how it would be implemented).

Create a "Copier" object, like a Remote, that stores it's own settings and
has one or many endpoints (like "/sync/") that can be POSTed to and return
a task and create a new repository version.

POST /pulp/v3/api/copiers/rpm/$endpoint_name/ content_units=[...]  [more
params...]

The copier would store settings such as "recursive" and the "source" and
"destination" repositories.  And let's say they can be overriden.

Or, create new endpoints without any associated DB models, like one-shot
upload does:

variant 1:  POST /pulp/v3/api/content/rpm/packages/copy/
content_units=[...] source="..." destination="..."  [more params...]

variant 2: POST  /pulp/v3/api/copy/rpm/package/ content_units=[...]
source="..." destination="..."  [more params...]

Since basic copy support (just copying the units, no depsolving, etc.)
could in theory be implemented completely generically, it would be nice if
we could provide that for free somehow.  But I don't immediately see a way
of doing so.

I welcome any suggestions or input.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Mirroring Anaconda Repositories

[Daniel Alley]

Hello Nathaniel,


> I currently don’t see a Anaconda plugin available for Pulp. Do you know if
> any of the other plugins would help facilitate that, or if there is an
> Anaconda plugin being developed (didn’t see one on Git)?


There isn't one yet (either developed or in development), but it's
something we've been asked about before, and it's something the scientific
community is particularly interested in.  I don't think it would be overly
difficult to produce a Pulp *3* plugin for Anaconda support, although I'm
not with how Anaconda works.  The closest analogue is probably the Python
plugin [0] if you would like to take a look.

The caveat there is that Pulp 3 is pre-release and isn't quite production
ready yet, and Pulp 2 is what is currently used in Satellite.  I would not
recommend trying to write an Anaconda plugin for Pulp *2* due to the
complexity that would be involved, but  Pulp 3 makes writing a functional
plugin much much simpler, and we'd definitely be willing to offer guidance
if you would be willing to to try to do so.

Both Pulp 2 and Pulp 3 have existing Debian and Python plugins, if you
would like to experiment with those.

Thanks,
Daniel

[0] https://github.com/pulp/pulp_python/

On Wed, Mar 27, 2019 at 1:23 PM Priddy, Nathaniel G CIV WXQ, 1041 <
nathaniel.pri...@navy.mil> wrote:

> Hello,
>
>
>
> I, and a colleague, are currently tackling the issue of mirroring multiple
> repos, RPM, DEB, and Python, and then exporting all of these as incremental
> exports to a disconnected environment. I’ve used Satellite extensively, and
> am looking at, at least for the package management, moving straight to Pulp
> to help support these other requirements. In addition to our existing
> mirror needs, we’re also hoping to be able to mirror Anaconda repos. I
> currently don’t see a Anaconda plugin available for Pulp. Do you know if
> any of the other plugins would help facilitate that, or if there is an
> Anaconda plugin being developed (didn’t see one on Git)? If not, would this
> be terribly difficult to start working in? My colleague already has a
> custom Anaconda repo mirroring tool that he built, was wondering how hard
> it would be to port to a plugin so we didn’t have multiple tools doing the
> same thing? I apologize if this isn’t the correct email to present this
> question, and thank you in advance for any guidance presented!
>
>
>
> Thanks,
>
>
>
> Nathaniel G. Priddy
>
> IT Specialist
>
> Naval Surface Warfare Center, Crane Division, (NSWC Crane)
>
> Code WXQW/1041, Bldg. 3173
>
> 300 Highway 361
>
> Crane, IN 47522-5001
>
> Ph:  812.854.2367
>
> DSN: 482-2367
>
> Email: nathaniel.pri...@navy.mil
>
>
>
> FOR OFFICIAL USE ONLY - Privacy Sensitive - any misuse or unauthorized
> disclosure can result in both civil and criminal penalties.  This
> transmission should be viewed only by personnel having an official
> "need-to-know".  If you are not the intended recipient, be aware that any
> disclosure, copying, distribution or use of the content of this information
> is prohibited.  If you have received this communication in error, please
> notify me immediately by email and delete the original message.
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] SUSE has an unrelated library named "libpulp"

[Daniel Alley]

A user on IRC pointed out that there is a new-ish C library by SUSE named
libpulp.

https://github.com/SUSE/libpulp

Since it's a C library and contains no Python, there is no PyPI name
conflict like there we had with PuLP, and I don't particularly see a need
to ask them to rename their project.  But I'm wondering what others think.
Should we contact them just to let them know?
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev