On Tue, Sep 29, 2020 at 1:37 PM Maarten Beeckmans <maart...@inuits.eu> wrote:
> On 2020-09-29 18:22, Dennis Kliban wrote: > > On Tue, Sep 29, 2020 at 10:30 AM Maarten Beeckmans <maart...@inuits.eu> > wrote: > >> We are trying to automate our Pulp3 setup the same way as our Pulp2 setup. >> >> We have the following use cases in our Pulp2 setup: >> >> a) We create mirrors and repositories automatically for every environment >> (using puppet and the API). The repositories are defined in a yaml >> configuration file (Puppet Hiera). >> - We have several problems for doing this in Pulp3. It's very complex >> to automatically create mirrors (ex. centos-8-appstream) for the >> development environment (that are in sync with the upstream repositories). >> A possible solution for this problem would be to add a 'latest' flag to >> publications and distributions so that they are always pointing to the >> latest version of the repository. >> - It is possible to create the repositories with scripts, but that's >> specific and quite hacky. It's also not possible to automatically update >> the publications and distributions this way. We would need to keep a local >> copy of what distribution maps to what repository, but that's not a good >> idea. Why keep a local copy of the mapping if that's present in pulp. >> > > We are already planning to improve this workflow. A single API call will > create a task that will sync a repo, create a new publication, and update > the distribution. We are first going to introduce the change in pulp_file > and then other plugins will follow. https://pulp.plan.io/issues/7469 > > This issue seems to be solving my issue. > > > >> b) We upload packages from one repository (that is not served to clients) >> to another repository based on lists in yaml (cherry picking), or straight >> from a (jenkins) Pipeline. This is done by the pulp-admin commands. >> - cherry picking could be done by copying content from one repository >> to another using the copy API call, but we want to automatically expose >> those changes to the development environment (promotion can be done as in >> c). >> - Uploading content isn't an issue because this is possible by >> converting the pulp-admin commands to api calls. >> > > You want to be able to upload a package and have that operation create a > new publication and update the distribution. This would be similar to the > story that I linked above for syncing - except for the upload use case. Am > I understanding correctly? > > Yes, that's what we want. If we create an rpm itself, we want that's it > directly available in the (development) repository so we don't have to > create a new publication and update the distribution. For adding it to > production, we would promote the repository as described in c. Same when > copying packages between repositories. > I definitely see the utility in making the content available to the users right away. Please file a story at https://pulp.plan.io/issues/new/ so we have this use captured. This use case should be considered also when implementing #7469. > > >> c) We want to be able to 'promote' a set of repositories from one >> environment to another. We are using a script that is generated by puppet >> when creating the repositories in a). >> For 'promoting' a set of repositories to one environment to another >> environment, this is explained >> https://docs.pulpproject.org/pulpcore/workflows/promotion.html. >> >> > I don't understand what challenge you are describing in C. > > Now we have different repositories for development and production. We want > to be able to easily promote repositories from development to production > (with archiving, the production repository keeps X older versions of > packages), this may be done with one command in the cli interface or > different commands that can be scripted. This is somewhat explained in the > docs, but it's very high level. Maybe it's a good idea to add an example in > the rpm docs (with the promoting of an CentOS repository)? > I agree that we need more docs around this. Can you please file an issue about this also? There are two ways to accomplish this. 1) Create a distribution for each environment. In this case whenever you are ready to promote a publication from the Dev distribution to the Production distribution you just simply look at which publication is currently assigned to Dev and assign it to Production. When you want to roll back to a previous publication you need to query for a publication that is associated with a specific repository version (or publication date of creation) that you are interested in. Then you need to update the distribution with that publication. This workflow gets tricky when you need to release a hotfix to production and skip all the other environments. Since the repository version history is probably much further along than Production is right now, you need to find the repository version that is associated with the current publication that is associated with the Production distribution. Then you need to use the 'modify' API[0] with the Production repository version as the base_version and add/remove any packages that you need for Production. Then you need to create a new publication and associate it with the Production distribution. Then you will need to create another repository version using the previous repository version as the base_version. This will get you back to where you were before in your Development environment. 2) Create a repository and distribution for each environment. Each time you want to promote a repository to the Production environment you need to use the repository 'modify' API[0] to create a new Production repository version. The base_version would be the Dev repository version which you want to promote. You would then create a publication for that new Production repository version and associate that publication with the Production distribution. The rollback process is similar to the above, but the advantage of this approach is that you have separate repository version history for each environment. When you need to skip environments and go straight to Production, you can simply add/remove packages to the latest Production repository version without having to figure out which version your Production repository is at. [0] https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/repositories_rpm_rpm_modify > > >> Thanks for all the work >> >> Maarten >> _______________________________________________ >> Pulp-list mailing list >> Pulp-list@redhat.com >> https://www.redhat.com/mailman/listinfo/pulp-list > > -- > (o- Maarten Beeckmans > //\ Open Source Consultant > V_/_ Inuits - https://www.inuits.eu > > _______________________________________________ > Pulp-list mailing list > Pulp-list@redhat.com > https://www.redhat.com/mailman/listinfo/pulp-list
_______________________________________________ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list
