Re: [Pulp-dev] the "relative path" problem

2020-10-21 Thread Tatiana Tereshchenko 
TL;DR: An attempt to propose the least invasive option to solve the case
when remote repository metadata needs to be mirrored. Please provide
feedback if you are interested in the outcome.

There have been multiple attempts and discussions to solve the
relative_path problem in a general way which covers multiple use cases.
They all look very invasive and only possible to be done in Pulp 4+ due to
the amount and significance of changes that needs to be made, to the data
models and/or to the API.

The following proposal solves only this use case: As a user, I can mirror
remote repository metadata as is.
An additional goal is to avoid backward incompatible changes and ideally
leave a way for further improvement to solve the problem in a more general
way.
(The following proposal does NOT solve a use case: As a user, I can have
the same content under different relative paths in any repository.)

Proposal:
- Have a way to distinguish between repositories with managed content and
with the exact mirror (e.g. create a repository with exact_mirror=True or a
new dedicated repository type)
 - For such repos, create a publication at sync time (includes published
artifacts and metadata).
 - For such repos, publish is no-op and always returns the existing
publication for the requested repo version.
 - For such repos, no modifications are allowed except the sync in mirror
mode. (At least for RPM plugin, I believe we can't allow discrepancies
between metadata and content in a repo, especially if some content is
removed.)

Pros:
 - non-invasive, only additive model changes
 - can be implemented in a plugin which needs it or it can be moved to the
pulpcore if it allows plugin input at certain points.
 - leaves a way for further improvement to handle a more general case, see
the full proposal here
https://hackmd.io/02KBjCD3Q0WP7p4ALwzhJw#relative_path-in-PublishedArtifact-only

Cons:
 - doesn't solve the problem of various relative paths for the same content
in general way
 - a separate code path (at times) to handle this type of repositories.

For reference:
 - hackmd doc with all the considered proposals and the summary of the
potentially valid ones https://hackmd.io/02KBjCD3Q0WP7p4ALwzhJw
 - pulpcon video with the discussion of the proposals
https://www.youtube.com/watch?v=7IzxAQYr5-I

Thanks,
Tanya

On Thu, May 7, 2020 at 2:07 PM Brian Bouterse  wrote:

> I agree with that problem statement. pulp_file may want to have the same
> Content at two different paths in different RepositoryVersions (or even the
> same RepositoryVersion). Without this capability a user could never "move"
> where content lives in a RepositoryVersion if its already been placed in
> any other RepositoryVersion.
>
> Additionally pulp_maven may need to sync two repositories in the wild that
> already contain the same content in two locations. I offer this as example
> not to pile-on, but because it's a multi-content artifact which I believe
> we will need to consider also as we work towards a solution.
>
> I've been spending time on developing a solution, but it needs more work
> so it's not ready yet. Also other katello and galaxy_ng work continues to
> pre-empt this, so it could take a while.
>
> On Thu, May 7, 2020 at 3:39 AM Matthias Dellweg 
> wrote:
>
>> > Users need to be able to store the same content unit at different
>> relative paths in different repository versions. This problem is not unique
>> to the RPM plugin. Do we agree about that?
>> Yes, we agree. In pulp_deb relative_path is part of the contents
>> natural_key to circumvent this problem. So this creates two content units
>> that only differ in relativ_path. At least they share the artifact.
>>
>> On Thu, May 7, 2020 at 2:06 AM Dennis Kliban  wrote:
>>
>>> I'd like to provide a little bit more context for my previous email by
>>> going back to the original problem statement:
>>>
>>> On Wed, Apr 1, 2020 at 9:23 AM Daniel Alley  wrote:
>>>
 Problem:

 Currently, a relative_path is tied to content in Pulp. This means that
 if a content unit exists in two places within a repository or across
 repositories, it has to be stored as two separate content units. This
 creates redundant data and potential confusion for users.

 As a specific example, we need to support mirroring content in pulp_rpm
 . Currently, for each location at
 which a single package is stored, we’ll need to create a content unit. We
 could end up with several records representing a single package. Users may
 be confused about why they see multiple records for a package and they may
 have trouble for example deciding which content unit to copy.

>>> Users need to be able to store the same content unit at different
>>> relative paths in different repository versions. This problem is not unique
>>> to the RPM plugin. Do we agree about that?
>>>
>>> I've been working on a potential solution that solves this problem in a
>>>

[Pulp-dev] Katello/Pulp3 Integration scrum

2020-10-21 Thread Grant Gainey 
October 21, 2020

Overview

   -

   Katello Schedule
   -

  3.18 branching:  Nov 2nd, Targeting Pulp 3.7



   -

   Meeting lead role rotating to ggainey


   -

   Katello: please assign all your bugs to him as well (not!)


Pulp

   -

   Pulpcore
   -

  3.8 release is out today, compatible with 3.7 plugins
  -

  3.7.2 (migration, imp/exp backports) is going out today or tomorrow
  -

   CLI
   -

  Check out our proof of concept
  -

  https://pulpproject.org/2020/09/28/pulp-3-cli-poc-call-for-feedback/
  -

   Pulp 2.21.4
   -

  Hit a blocker 
  -

  Potential fix in hand, ipanova testing
  -

   RPM
   -

  Minor fixes
  -

   Migration
   -

  Customer db migrated in one go by pulp (not katello :))
  -

  Memory issues


Katello

   -

   Current work:
   -

  Capsule sync optimization
  -

  satellite migration tooling
  -

  Migration progress reporting
  -

   3.7 in nightly
   -

   Import/Export and incrementals - PRs submitted


G
-- 
Grant Gainey
Principal Software Engineer, Red Hat System Management Engineering
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulp Installers Team Meeting 2020-10-21

2020-10-21 Thread Mike DePaulo 
## October 21 Agenda

* Meeting keeps needing to be rescheduled due to company meeting
  * Agreed: Reschedule for 9:00 / 15:00 if schedules are free.
  * Backup: Reschedule for 8:30 / 14:30 or so. Americans will wake up
earlier :)
* [bmbouter] [pulpcore 3.7.0 retro] installer requires pulp-file and
pulp-rpm to be released, which prevents pulpcore from fully releasing
* installer needs to drop pulp_rpm tests for y-releases
* pulp-rpm needs to be tested due to
https://pulp-installer.readthedocs.io/en/latest/prereq_roles/pulp_rpm_prerequisites/
* Agreed: this was addressed by the deprecation policy implemented in
the 3.8.0 development cycle.
* [mdellweg] pulplift merge has been ready for some while now and i want to
get it merged
* https://github.com/pulp/pulp_installer/pull/440
* Mike to review it, now that he is back at work.
* redis SCL switch from 10/14 Agenda?
* Agreed: write a ticket, then discussion with AH. Mike shouldn't do
lots of investigation 1st due to workload.

-- 

Mike DePaulo

He / Him / His

Service Reliability Engineer, Pulp

Red Hat 

IM: mikedep333

GPG: 51745404

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] CLI meeting

2020-10-21 Thread David Davis 
## Oct 22, 2020

* Recorded talk?
* How to write a CLI command
* AI: ggainey to work on exporter commands and create presentation
based on this work
* Timeframe: before end of November
* Adding type annotations
* https://github.com/pulp/pulp-cli/pull/15
* [david] will review and merge
* Settings feature has been merged
* https://github.com/pulp/pulp-cli#configuration
* Planning for next quarter?
* Create a 0.1.0 release milestone in redmine?
* for next quarter:
* pulpcore
* happy/main path through pulp_rpm
* packaging
* ddavis to write up user-stories. meet to discuss planning in mid/late
november
* can we log the actual API calls?
* https://pulp.plan.io/issues/7729

David
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulpcore team meeting notes

2020-10-21 Thread David Davis 
## October 20, 2020

### Previous action items
* [david] to update https://github.com/pulp/pulp-ci/pull/737
* ~~When PR is ready, email about the PR and ask for feedback~~
* ~~Done. Date set to Oct 23, 2020~~
* Will send an announcement and plugin information once merged
* ~~[ina] to file redmine issue about demo video requirement~~
* ~~done https://pulp.plan.io/issues/7703 and
https://pulp.plan.io/issues/7704~~
* ~~[david] to make meeting one hour~~
* ~~Done!~~
* [ttereshc] follow up on relative_path problem on mailing list, in progress

### Topics
* SigningService changes (on sprint, fyi really)
* https://pulp.plan.io/issues/7700
* https://pulp.plan.io/issues/7701
* Improvements to FIPS checks
* https://pulp.plan.io/issues/7696
* need one on checking for "future checksum use" e.g. during publish
time
* We no longer use PUPS, do we? we should update docs
*
https://docs.pulpproject.org/pulpcore/contributing/index.html#suggesting-changes-to-the-pulp-development-process
* maybe also mark repo a deprecated?
* how and where should we propose process improvement suggestions?
* [bmbouter] I can't attend so leaving comment here, +1 to
archiving this repo and announcing the process as no longer used. We should
try to move the relavent parts to the docs somehow (maybe file tickets). I
don't expect to replace this process with a formal process.
* Enforce funtional tests in pulp_file
* Add a check in Travis
* docs style guide question: newlines after each sentence or no?
* https://pulp.plan.io/issues/7570
* [bmbouter] my concern with having newlines instead of paragraphs was
that the docs unrendered become difficult to read (even though the blame
log does get easier).
* Backport to 3.7
* https://pulp.plan.io/issues/7702
* Should we regularly review redmine issues filed in between pulpcore
meetings? All except the "issue" type, since it's covered at triage.
* https://bit.ly/3m1e3nr
* running functional tests for all FIPS compliant plugins
* re-prioritize https://pulp.plan.io/issues/4832
* +1

### Action items
* [david] To send out last call for feedback before merging
https://github.com/pulp/pulp-ci/pull/737
* [ttereshc] follow up on relative_path problem on mailing list
* [dkliban] to follow up with bmbouters about fips checks
* [ipanova] send an email to archive PUPS repository and file docs task
* [dkliban] file a task for running tests for multiple plugins in one fips
environment in the installer nightly

David
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev