Re: [Pulp-list] [Feedback needed] Pulp 2 to Pulp 3 migration, export_distributor, RPM/ISO plugin

[Rohan McGovern]

Tatiana Tereshchenko  writes:

> As you probably know, we are in the process of designing a tool which will
> provide a way to migrate from Pulp 2 to Pulp 3, so we will ask you for
> feedback every now and then.
>
> Export distributor [0] is used for generating ISO images which can contain
> one or more RPM or File (ISO plugin in Pulp 2 terms) repositories in it.
> This distributor is also used to export a repository into a specific
> directory on a filesystem.
>
> Please, let us know if you use export_distributor:

We use export_distributor, indirectly via group_export_distributor.

> 0. Do you use Pulp directly or through Katello? This thread is more
> relevant for those who use Pulp directly.

Direct usage of Pulp.

> 1. What is the scale? E.g. "I use it for 10% of all my repositories, ~50
> repositories."

Looks like we've got ~60 repo groups with a total of ~250 repositories.
Some of those repos have tens of thousands of packages.

> 2. Do you export into an ISO or to a directory? Or both?

We export to ISOs.

> 3. Is it important that export_distributors and their configuration are
> migrated to Pulp3? E.g. "It would be nice-to-have, not critical, I export
> very few repositories."
>

I think it's not important for us. Our approach is that the tool which
triggers the distributors understands what the configuration should be,
and sets it into the desired state before triggering tasks if needed.

--
Rohan

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] RHEL 6 Server ISOs CDN channel problem?

[Rohan McGovern]

Kodiak Firesmith  writes:

> Hi Tyrone,
> Unfortunately the problem is ongoing.  I've just tried once more to be
> sure, it's still related to the rhel-server-6.10-x86_64-dvd.iso image
> sync.  I'm wondering if the CDN team might be able to scrutinize that ISO
> compared to other ISOs that do work, and see what makes this particular ISO
> uniquely problematic.
>

Could you please try that once more?

What happened here is that Red Hat published a new version of this ISO
(and several others) with the same filename.  This is an unusual event
and some of the publishing tools didn't cope with it correctly,
particularly with respect to CDN caching.  It meant that some clients
were still seeing the old version of the ISOs, leading to a checksum
mismatch on sync.

The issue should be resolved now, with all clients only seeing the
up-to-date content.

I've also filed https://pulp.plan.io/issues/3845 regarding the unhelpful
error messages from Pulp.

>
> On Fri, Jul 6, 2018 at 12:58 AM Tyrone Abdy  wrote:
>
>> Is that fixed for you now?
>>
>> On 3 July 2018 at 01:21, Kodiak Firesmith  wrote:
>>
>>> I've created Red Hat support case #02132814 to track this Red Hat CDN
>>> problem.
>>>
>>> On Mon, Jul 2, 2018 at 9:49 AM Kodiak Firesmith 
>>> wrote:
>>>
>>>> Yay!  I love it when a problem isn't specific to my site!   I'll raise a
>>>> support ticket and see if I can't convince Red Hat GSS that there is a
>>>> problem with that CDN channel.
>>>> @redhat.com folks on this list, please feel free to reach out if you
>>>> have contacts on the Red Hat CDN side of the house and you'd like a copy of
>>>> the ticket number.
>>>>
>>>> Huge thanks to Mr. Gersting for confirming!
>>>>
>>>>  - Kodiak
>>>>
>>>> On Mon, Jul 2, 2018 at 9:42 AM David Gersting 
>>>> wrote:
>>>>
>>>>> Looks like I'm seeing the same thing with ours. It's interesting that
>>>>> the error message is blank in the exception, and that you only get the
>>>>> ISO's name.
>>>>>
>>>>> https://paste.fedoraproject.org/paste/~GoCoCFIyfH0UovVDl5zQA
>>>>>
>>>>>
>>>>> */* David Gersting
>>>>>
>>>>> UNIX/Linux System Administrator
>>>>>
>>>>> Information Technology Services
>>>>>
>>>>> West Virginia University
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jul 2, 2018 at 8:44 AM Kodiak Firesmith 
>>>>> wrote:
>>>>>
>>>>>> This is a classic case of:  "Is his broken for everyone or just me?"
>>>>>>
>>>>>> I'm seeing the following problem with the RHEL 6 Server ISOs Red Hat
>>>>>> CDN channel since some time ~10 days ago.  It seems like the channel is
>>>>>> broken somehow or at least something is coming down that is confusing
>>>>>> Pulp.  All other ISO channels on Red Hat CDN for RHEL 5,6,7
>>>>>> Server,Workstation all seem to work fine.
>>>>>>
>>>>>> Here's what Pulp sees:
>>>>>>
>>>>>> https://paste.fedoraproject.org/paste/EzAQRp64UkoFaXdZOkjRLg
>>>>>>
>>>>>> If anyone else could please try out this CDN channel and report back
>>>>>> with results, that would be much appreciated!
>>>>>>
>>>>>> Thanks!
>>>>>>  - Kodiak Firesmith, Pittsburgh

-- 
Rohan McGovern

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Pulp fails to sync rhel 7 server repo with repomd.xml not found error, but the optional packages repo syncs okay

[Rohan McGovern]

Donald Wolfe  writes:

> I am having an issue with Pulp syncing the rhel 7 server repository, but a 
> similarly setup optional packages repository sync works.  Does anyone have 
> any suggestions about what to do, or where to look?  Any help would be 
> greatly appreciated.
>
> They were created like this and I got the information out of the 
> /etc/yum.repos.d/redhat.repo file on one of our rhel 7 systems, and the cert 
> files were copied from it as well:
>
> certnum=x
> pulp-admin rpm repo create --repo-id=rhel-7-server-rpms \
> 
> --feed=https://cdn.redhat.com/content/dist/rhel/server/7/7server/x86_64/os
> \

The URL has a mistake, it contains "7server" where it should contain
"7Server".  The optional URL doesn't have the mistake, so that would be
why that one works.

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Space needed by yum repo ISO exports

[Rohan McGovern]

Rohan McGovern <rmcgo...@redhat.com> writes:

> We use the export distributor in pulp_rpm[1] to export yum repositories
> as ISOs.
>
> Sometimes, if many of these exports happen concurrently on the same
> host, we find that the filesystem containing /var/lib/pulp/working runs
> out of space.
>
> Is anyone able to provide some estimate on the amount of space required
> for these export tasks?  For example, is it expected to be the sum of
> the size of all RPMs in the exported repos, multipled by some constant
> factor?
>
> [1] 
> https://docs.pulpproject.org/plugins/pulp_rpm/tech-reference/export-distributor.html
>

It seems to need approximately three times the amount of space of the
data being exported:

- it will export the repo to a directory (copy 1)
- copy it to another directory with different layout (copy 2)
- include all the exported data into an ISO via mkisofs (copy 3)
- it doesn't clean up the copies as it goes

Let me know if this sounds wrong.

-- 
Rohan

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] Space needed by yum repo ISO exports

[Rohan McGovern]

We use the export distributor in pulp_rpm[1] to export yum repositories
as ISOs.

Sometimes, if many of these exports happen concurrently on the same
host, we find that the filesystem containing /var/lib/pulp/working runs
out of space.

Is anyone able to provide some estimate on the amount of space required
for these export tasks?  For example, is it expected to be the sum of
the size of all RPMs in the exported repos, multipled by some constant
factor?

[1] 
https://docs.pulpproject.org/plugins/pulp_rpm/tech-reference/export-distributor.html

-- 
Rohan

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] Pulp migration order

[Rohan McGovern]

We're working on some custom Pulp migrations to fix up some data issues
in our Pulp installation.
https://docs.pulpproject.org/dev-guide/newtypesupport/plugin/migrations.html

The migration mechanism has an annoying restriction: there aren't
allowed to be any gaps in the migration sequence.  For example, if I
have 0001_my_first_migration.py and 0003_add_email_addresses_to_users.py
then pulp-manage-db won't run migration 3 due to the absence of a
migration numbered 2.

There is a good reason for it not to work this way: this means that the
next number in the sequence is in contention between all separately
developed patches which might need to add migrations.  For example, if
there are currently migrations 1-3, multiple developers working on new
migrations concurrently are all forced to use number 4 (otherwise their
code won't run) but in the end, only one of them can "win" and all the
others will have to renumber theirs later.  If the migrations are
independent from each other, that's wasted time.

Compare with e.g. ActiveRecord where migrations are "numbered" by
timestamp at time of generation, and there's no problem merging branches
where unrelated migrations were authored separately.
http://guides.rubyonrails.org/active_record_migrations.html

Is there a particular reason for the current pulp migration system to
work this way?

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Clean uploaded but not imported content

[Rohan McGovern]

Brian Bouterse  writes:

> I'm familiar with this limitation. You can manually delete them [0], but 
> there is nothing in Pulp which cleans these up automatically as far as I 
> know.
>
> Pulp does have a monthly maintenance [1] codepath which could also 
> delete unimported content. It could look for files older than some 
> reasonable threshold like 2 weeks or something like that. Feel free to 
> file the feature request if you want.
>
> There is a plan for a new Upload API [2]. That plan will provide 
> auto-cleanup so if [2] is implemented soon then we would no longer need 
> the monthly cleanup story you might file.
>

Thanks.  This hasn't caused us difficulty so far, so I think we'd be
fine to wait for the new API.

Is there any way that I can vote for an issue or express on the tracker
that I'd find it useful?

> [2]: https://pulp.plan.io/issues/892



___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] Clean uploaded but not imported content

[Rohan McGovern]

The pulp upload API docs [1] explain that uploading a unit to a repo is
a 4 step process:

1. Make upload request
2. Upload the content (file)
3. Trigger importer(s), wait for their tasks to run
4. Delete upload request

The process interacting with pulp's API might be prone to errors in
steps 2 or 3, leaving unimported content behind on the server
(e.g. files under /var/lib/pulp/uploads/ ).

Is there any mechanism to periodically clean these up?

If there's nothing already, would it be fair to include this in the
scope of pulp or is it considered this should instead be handled externally?

[1] 
https://docs.pulpproject.org/dev-guide/integration/rest-api/content/upload.html

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list