Re: [Pulp-dev] Tasking System Changes and Feedback

[Eric Helms]

On Thu, Apr 8, 2021 at 5:24 PM Daniel Alley  wrote:

> Eric,
>
> * The idea is to move away from RQ entirely.  RQ is fine (and vastly
> better than Celery IMO), but managing task state across both 1) the
> database and 2) a separate, external registry is still problematic.  If all
> of the information can simply be kept in the database, then it will be much
> easier to maintain consistent state.
>

Are there any benefits to improving RQ vs the invented here method? I'm
just curious about the cost of maintaining a tasking system versus being
part of a community built one. This feels like the kind of problem that
many other applications should have in the Python world -- or are there
elements of Pulp's deployment architecture that make it unique here?


> * *Maybe*.  We're considering using Redis as a cache to improve content
> serving performance (after all, caching is one of the primary uses of
> Redis). If we do, then Redis would remain in the architecture, but it could
> potentially be an optional component and would be easier to remove at some
> point in the future.
> * We'd just be adding a small amount of information to each task record,
> and it wouldn't prevent cleanup later.
>

This is sort of an aside to this general change. Are Pulp tasks cleaned up
from the database today?


>
>
>
> On Thu, Apr 8, 2021 at 4:42 PM Eric Helms  wrote:
>
>> A few initial questions that get a bit into the stack but will help the
>> Foreman project think on the proposed changes:
>>
>>  * Does this move away from RQ entirely or just RQ workers?
>>  * Do the new workers remove Pulp 3's use of Redis all together?
>>  * Will using the database result in any additional build up of tasking
>> information that can impact performance over time? (Or does all task data
>> get cleaned up eventually?)
>>
>> Thanks for sending this along early.
>>
>> On Fri, Apr 2, 2021 at 4:43 PM Brian Bouterse 
>> wrote:
>>
>>> FYI, @mdellweg and I have been collaborating on the tasking system
>>> changes. This email is to share some info to transition the work to
>>> @mdellweg while I'm out. With the new-style disabled by default I am hoping
>>> it can go into 3.13.
>>>
>>> ## The PoC and ticket info
>>>
>>> The PoC is basically functional, but it's a PoC:
>>> https://github.com/pulp/pulpcore/pull/1222/
>>>
>>> * The epic is being tracked here which recaps why we're doing this and
>>> the high level approach. The sub-tasks capture the various detailed
>>> changes. https://pulp.plan.io/issues/8495
>>>
>>> * This is totally separate from the RQ workers you use today, and those
>>> will continue to be available for a while.
>>>
>>> ## Next Steps
>>>
>>> * @mdellweg will continue the work and hopefully merge the PoC while I'm
>>> out
>>>
>>> * Once it's demo-able I've asked @mdellweg to give a 20 minute, public
>>> (hopefully recorded) technical demo. While it is designed to be a drop-in
>>> replacement from a user perspective, we think sharing the internals will be
>>> helpful to get feedback and increase the list of those who understand the
>>> work.
>>>
>>> All the best,
>>> Brian
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://listman.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
>>
>> --
>> Eric Helms
>> Principal Software Engineer
>> Satellite
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://listman.redhat.com/mailman/listinfo/pulp-dev
>>
>

-- 
Eric Helms
Principal Software Engineer
Satellite
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Tasking System Changes and Feedback

[Eric Helms]

A few initial questions that get a bit into the stack but will help the
Foreman project think on the proposed changes:

 * Does this move away from RQ entirely or just RQ workers?
 * Do the new workers remove Pulp 3's use of Redis all together?
 * Will using the database result in any additional build up of tasking
information that can impact performance over time? (Or does all task data
get cleaned up eventually?)

Thanks for sending this along early.

On Fri, Apr 2, 2021 at 4:43 PM Brian Bouterse  wrote:

> FYI, @mdellweg and I have been collaborating on the tasking system
> changes. This email is to share some info to transition the work to
> @mdellweg while I'm out. With the new-style disabled by default I am hoping
> it can go into 3.13.
>
> ## The PoC and ticket info
>
> The PoC is basically functional, but it's a PoC:
> https://github.com/pulp/pulpcore/pull/1222/
>
> * The epic is being tracked here which recaps why we're doing this and the
> high level approach. The sub-tasks capture the various detailed changes.
> https://pulp.plan.io/issues/8495
>
> * This is totally separate from the RQ workers you use today, and those
> will continue to be available for a while.
>
> ## Next Steps
>
> * @mdellweg will continue the work and hopefully merge the PoC while I'm
> out
>
> * Once it's demo-able I've asked @mdellweg to give a 20 minute, public
> (hopefully recorded) technical demo. While it is designed to be a drop-in
> replacement from a user perspective, we think sharing the internals will be
> helpful to get feedback and increase the list of those who understand the
> work.
>
> All the best,
> Brian
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>


-- 
Eric Helms
Principal Software Engineer
Satellite
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Moving Content Guard Authorization to Webserver and out of pulp-content

[Eric Helms]

On Wed, Mar 11, 2020 at 2:12 PM Brian Bouterse  wrote:

> tl;dr: What we have today cannot work with rhsm certificates which Katello
> uses. To resolve, we need to have content guard checking moved to the
> webserver configs for apache and nginx and not done in pulp-content as it
> is today.  https://pulp.plan.io/issues/6323
>
> We need to bring the auth to where TLS is terminated because we can't
> being the client certs to pulp-content due to invalid header characters. As
> is, pulp-certguard cannot work with Katello's cert types (rhsm certs) so
> that is driving my changes.
>
> If anyone has major concerns or other ideas please let me know. In the
> meantime I'm proceeding moving the authorization to the webserver and then
> updating pulp-certguard to work with that. This will make pulp-certguard's
> GA tied to pulpcore 3.3.0. Feedback is welcome.
>

What will this mean from a runtime perspective? Or rather, what within the
webserver layer will be handling this auth?


>
> [0]: https://pulp.plan.io/issues/6323
>
> Thanks,
> Brian
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>


-- 
Eric Helms
Principal Software Engineer
Satellite and Cloud Services
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [Pulp-list] Pulp 2 to 3 Migration meeting on Wednesday June 19th

[Eric Helms]

Thanks for these updates.

On Wed, Jun 19, 2019, 1:59 PM Tatiana Tereshchenko 
wrote:

> The logs from the meeting can be viewed here[0].
>
> We covered:
>   - hard links approach for "migrating" files, good or bad; alternatives
> * current plan: Pulp 3's migration task will migrate content by either
> creating a hard link in the Pulp 3 artifact storage location or by copying
> it there if hard links are not supported. There will be a way to see how
> much content has been migrated and how much is left.
>

In the copy case, will Pulp 3 migrations prevent a user from potentially
filling their disk up and ending up in a broken state?


>   - protected repos migration
> * current plan: CertGuards witll be created and associated with
> Distributions in Pulp3. The Migration Plan will provide the ability to
> configure which repositories content protection is migrated for to Pulp 3.
>
> The etherpad [1] is updated with the suggested changes to the Migration
> Plan structure, you can see it in the examples for use cases on lines L316,
> L325.
>
> [0]
> https://pulpadmin.fedorapeople.org/triage/pulp-2to3-sig/2019/pulp-2to3-sig.2019-06-19-16.00.log.html
> [1] https://etherpad.net/p/pulp-2to3-migration
>
> On Tue, Jun 18, 2019 at 6:34 PM Tatiana Tereshchenko 
> wrote:
>
>> An IRC meeting will be held on Wednesday June 19th at 12pm Eastern[0].
>> The meeting will take place in #pulp-2to3-sig on Freenode IRC network.
>>
>> Tentative agenda:
>>   - continue on use cases, see etherpad L18+
>> * especially distributors migration part
>> * verify that everything is covered by the current MP structure
>> * discuss any questions around use cases that can affect the MP
>> structure
>>  - hardlinks approach for "migrating" files, good or bad; alternatives
>>
>> Please put any questions or ideas into the etherpad [1] before the
>> meeting.
>>
>> [0]
>> https://www.worldtimebuddy.com/?qm=1=100,4487042,3078610=100=2019-6-19=16-17
>> [1] https://etherpad.net/p/pulp-2to3-migration
>>
> ___
> Pulp-list mailing list
> pulp-l...@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [Pulp-list] Pulp 2 to 3 Migration meeting on Thursday June 14th

[Eric Helms]

Thanks for sending along and sharing. Will migrations be additive only? I
am not seeing any language to handle removal of content from Pulp 3. I am
thinking of the cases around my Pulp 2 content has changed since I last
attempted a Pulp 3 migration and I now have cruft in my Pulp 3 that does
not mirror my Pulp 2 before I have switched to Pulp 3.

On Thu, Jun 13, 2019 at 4:16 PM Tatiana Tereshchenko 
wrote:

> The logs from the meeting can be viewed here[0].
> The agenda was quite ambitious, the first item was partially covered and
> discussion will continue next week.
> We covered:
>  - content or no content migration
>  - how to keep the MigrationPlan declarative and migrate various plugins
> at different times without unintentionally removing pulp3 content which has
> been fully migrated
>
> The etherpad [1] is updated with the suggested changes to the Migration
> Plan structure, you can see it in the examples for use cases on lines 188+.
>
> [0]
> https://repos.fedorapeople.org/pulp/pulp/meetings/pulp-2to3-sig-meeting-June-13-2019.txt
> [1] https://etherpad.net/p/pulp-2to3-migration
>
> On Wed, Jun 12, 2019 at 8:56 PM Tatiana Tereshchenko 
> wrote:
>
>> An IRC meeting will be held on Thursday June 14rg at 10am Eastern[0]. The
>> meeting will take place in #pulp-2to3-sig on Freenode IRC network.
>>
>> Tentative agenda:
>>   - go through use cases and see if we forgot anything, see etherpad
>> L184+ [1]
>>   - verify that everything is covered by the current MP structure
>>   - discuss questions around use cases that can affect the MP structure
>>   - hard link approach for "migrating" files, good or bad; alternatives
>>
>> Please put any questions or ideas into the etherpad [1] before the
>> meeting.
>>
>> [0]
>> https://www.worldtimebuddy.com/?qm=1=100,4487042,3078610=100=2019-6-14=14-15
>> [1] https://etherpad.net/p/pulp-2to3-migration
>>
> ___
> Pulp-list mailing list
> pulp-l...@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 2 to 3 Migration meeting on Friday June 7th

[Eric Helms]

The subject and body are time incongruent. June 7th vs Monday June 3rd.

On Thu, Jun 6, 2019, 1:34 PM Dennis Kliban  wrote:

> An IRC meeting will be held on Monday June 3rd at 11am Eastern[0]. The
> meeting will take place in #pulp-2to3-sig on Freenode IRC network. We will
> be discussing the schema for the Migration Plan described here[1]. Please
> put any questions or ideas into that etherpad before the meeting.
>
>
> [0]
> https://www.worldtimebuddy.com/?qm=1=100,4487042,3078610=100=2019-6-7=15-16
> [1] https://etherpad.net/p/pulp-2to3-migration
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulp 3 Dependencies and Services

[Eric Helms]

Howdy,

I had a couple of questions around how each of the Pulp 3 services loads
code and how that relates back to core code.

 * Do all services (workers, resource-manager, pulp-api and pulp-content)
load the entire stack of pulpcore and plugin code?
 * Is there a single python package that contains all code needed to run
pulp-api, workers and resource manager? is the same true for content?


Thanks,
Eric
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 Default Ports

[Eric Helms]

Thanks for following up with an issue Mike!

On Wed, Mar 20, 2019, 1:45 PM Mike DePaulo  wrote:

> On Thu, Mar 7, 2019 at 4:36 PM Mike DePaulo  wrote:
> >
> > On Thu, Mar 7, 2019 at 3:46 PM Eric Helms  wrote:
> > >
> > > I'm fine if someone wants to take up the effort to find and suggest
> two ports that match all of those as the defaults.
> >
> > I suggest:
> > 24816 (powers of 2: 2, 4, 8, 16)
> > 24817
> >
> > They're not officially reserved, and only used by Apple "med-ltp" as
> > part of a block of 1000 ports. [1] [2]
> >
> > I am new to Pulp, but I think other devs indicated that users may
> > browse available content. If so, I suggest we use 24816 for content,
> > and 24817 for API.
> >
> > > I've already opened a PR to make port customization a reality. In most
> environments, these ports won't see the light of day as they will be
> running services on localhost with a webserver proxying to them. I was
> aiming for sane defaults, that users and developers could easily rely on
> and expect across basic environments. And allow customization in
> environments that need it.
> >
> > Understood, I was not aware of this. But we need to prevent any
> > conflicts by default. Even if we only listen on localhost, we can
> > conflict with services listening on all interfaces. Users are likely
> > to give up on Pulp if they run into a port conflict; either because
> > they cannot figure it out ("why isn't this service starting?") or
> > because they spend too much time trying to set it up and other
> > priorities come up.
>
> Since users can interact with the API via web browser, I submitted a
> task for 24816 (API) & 24817 (content):
> https://pulp.plan.io/issues/4556
>
> -Mike
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Nomenclature Clarification

[Eric Helms]

Thanks for the feedback. I should know this, but if I wanted to propose an
official change to the nomenclature what is the correct process?

On Wed, Mar 13, 2019 at 10:47 AM Dennis Kliban  wrote:

> On Fri, Mar 8, 2019 at 10:34 AM Eric Helms  wrote:
>
>> Howdy,
>>
>> Within Pulp 3, I've found myself struggling to refer to the pieces of the
>> runtime deployment and wanted to raise a few questions I keep coming back
>> to.
>>
>>  1) Per [1], pulpcore seems to reference the entirety of the server,
>> workers and resource manager that live within the repository at [2]. Is
>> that correct?
>>
>
> Yes, that is what pulpcore means.
>
>
>>  2) What is the 'server' called? I keep wanting to call it the
>> 'pulp-api-server' or 'pulp-api-app' or 'pulp-api'.
>>
>>
> I like the name pulp-rest-api or pulp-api.
>
>
>>
>> - Eric
>>
>> [1]
>> https://docs.pulpproject.org/en/3.0/nightly/glossary.html#term-pulpcore
>> [2] https://github.com/pulp/pulpcore
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 Default Ports

[Eric Helms]

I'm fine if someone wants to take up the effort to find and suggest two
ports that match all of those as the defaults. I've already opened a PR to
make port customization a reality. In most environments, these ports won't
see the light of day as they will be running services on localhost with a
webserver proxying to them. I was aiming for sane defaults, that users and
developers could easily rely on and expect across basic environments. And
allow customization in environments that need it.

On Thu, Mar 7, 2019 at 3:13 PM Mike DePaulo  wrote:

> On Wed, Mar 6, 2019 at 10:08 AM Neal Gompa  wrote:
> >
> > On Wed, Mar 6, 2019 at 10:05 AM Neal Gompa  wrote:
> > >
> > > On Wed, Mar 6, 2019 at 10:01 AM Eric Helms  wrote:
> > > >
> > > > For most Pulp 3 installations, it seems there are two default
> applications that will be running: API and content. Those applications are
> set to run on 8000 and 8080 respectively. I was thinking that it might be
> more obvious for operators and developers to have the defaults next to each
> other in order to make it more predictable and easier to remember.
> Ultimately, these ports should be configurable for different environments,
> but sane easy to remember defaults have their value.
> > > >
> > > > My suggestion is: 8080 (API) and 8081 (content).
> > > >
> > >
> > > I would suggest not using any standard HTTP auxiliary ports by
> > > default. Is there a compelling reason to do so?
> > >
> >
> > Welp, this isn't clear enough. I mean that the ports should be unique
> > to Pulp rather than something that could be construed as something
> > that would unknowingly conflict.
>
> I agree with Neal,
>
> I cannot find a definitive list of the standard HTTP auxiliary ports,
> but lots of websites and open source/commercial web apps/web GUIs use
> 8080 or 8008 because 80 is already in use or is expected to be in use.
> I think the same applies to 8000 as well. And lots of small-scale
> sysadmins run multiple applications on the same server.
>
> Cockpit uses 9090 partially for this reason.
>
> What I suggest we do is find 2 ports (sequential like Eric suggests)
> over 1024 that are not officially IANA assigned, and just Google to
> make sure they are not commonly & unofficially used by any
> application.
> https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers
>
> There is one downside, which is that some organizations' firewalls
> allow standard HTTP auxiliary ports like 8080 but not arbitrary ports.
> This may be less common nowadays, and I feel it is outweighed.
>
> -Mike
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 2 and 3 Service Name Clashes

[Eric Helms]

I thought of a third option I'll throw out there aimed it reducing the
confusion through complete name changes of the services in Pulp 3. The
problem today is Pulp 2 and Pulp 3 have a 'resource-manager' and thus only
differentiate them via underscores and hyphens. Same with workers. Option 3
would be to change the naming of pulp-resource-manager and pulp-worker to
entirely new nomenclature within Pulp 3+. For example:

 * pulp-resource-manager becomes pulp-task-manager, pulp-tasking-manager,
pulp-queue-manager, pulp-arborist, arborist
 * pulp-worker becomes pulp-task-runner, pulp-task-worker,
pulp-async-worker, pulp-tasking-worker, pulp-seedling

This option still requires developers and operates with both to remember
which corresponds to which version of Pulp but makes it more obvious given
the complete naming difference than remembering which is the hyphen and
which is the underscore release.

Let the bike shedding begin!

On Wed, Mar 6, 2019 at 9:50 AM Eric Helms  wrote:

> My key with proposal with Option 2 is to set Pulp 3+ up to be the future
> without carrying any baggage. Let's put the baggage on the older bits and
> keep it there and leave the future as wide open as possible for Pulp 3+.
>
> As I am spending time looking at deploying Pulp 3 alongside Pulp 2 in a
> Katello environment, I'd like to get this change implemented as soon as
> possible. This is mostly an operational change and should have a minimal
> impact.
>
> is my next step to file a Redmine issue against Pulp 2?
>
> On Tue, Mar 5, 2019 at 11:15 AM Tatiana Tereshchenko 
> wrote:
>
>> +1 to option 2, rename of Pulp2 services.
>> It's a low risk change for Pulp2, in my opinion, and clear distinction of
>> legacy version.
>> I also agree with all the mentioned reasons to keep Pulp3 ones unchanged
>> and more importantly without version in the name.
>> -0 to make names configurable.
>>
>> Tanya
>>
>> On Tue, Mar 5, 2019 at 5:01 PM Ina Panova  wrote:
>>
>>> +1 to rename Pulp2 services. This way we would ensure that the users
>>> have  upgraded to  a minimal version of Pulp 2 before upgrading to Pulp 3.
>>> As a suggestion i would not make this change with the next Pulp2 release
>>> but whenever we'd be able to tell for sure that this Pulp2.Y version is the
>>> version we are supporting the upgrade from.
>>> +1 on Eric's reasoning about being more strict and allow less variation
>>> in naming conventions.
>>> +1 on Eric's point about if renaming Pulp3 services then this will lock
>>> services names to Pulp version.
>>>
>>> @dana eventually in the discussion on the issue we decided to make only
>>> the hyphens change.
>>> @asmacdo  https://pulp.plan.io/issues/4497 i think
>>> this is a dupe of https://pulp.plan.io/issues/4429
>>>
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>>
>>>
>>> On Tue, Mar 5, 2019 at 3:44 PM Matt Pusateri 
>>> wrote:
>>>
>>>> I like Option2, as long as we do it with and upgrade and we put Doc
>>>> notes in, I don't see it as a problem.
>>>>
>>>> Matt P.
>>>>
>>>> On Tue, Mar 5, 2019 at 8:48 AM Robin Chan  wrote:
>>>>
>>>>> To clarify, regarding @dana's comment - I wasn't necessarily voting
>>>>> for Option 1. Just pointing out the downside to option 2 wasn't a concern
>>>>> to my knowledge.
>>>>>
>>>>> @bherring - we have made changes to pulp 3 service names as @david
>>>>> pointed out. I do agree that making changes to pulp3 names seems to be the
>>>>> least invasive in the short term at first glance. Eric has given us
>>>>> feedback that the previous name change was not distinct enough. However I
>>>>> agree with his observation that specifying "3" won't be a great future
>>>>> proofed solution. I would argue that Option 2 is the  "least invasive" in
>>>>> the short term because the lasting impacts would be the most short lived
>>>>> (ironically for the same reasons you noted.)
>>>>>
>>>>> @kersom & @bherring - given your concerns about Option 2, can you
>>>>> suggest any variations/names for Option 1 that addresses the concern about
>>>>> longevity of the solution? Do you share Eric's concern regarding Austin's
>>>>> proposal to allow a user to s

[Pulp-dev] Pulp 3 Default Ports

[Eric Helms]

For most Pulp 3 installations, it seems there are two default applications
that will be running: API and content. Those applications are set to run on
8000 and 8080 respectively. I was thinking that it might be more obvious
for operators and developers to have the defaults next to each other in
order to make it more predictable and easier to remember. Ultimately, these
ports should be configurable for different environments, but sane easy to
remember defaults have their value.

My suggestion is: 8080 (API) and 8081 (content).



- Eric
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 2 and 3 Service Name Clashes

[Eric Helms]

t;>>>>
>>>>> QUALITY ENGINEER - PULP QE
>>>>>
>>>>> Red Hat
>>>>>
>>>>> <https://www.redhat.com/>
>>>>>
>>>>> 100 East Davie Street
>>>>>
>>>>> Raleigh, NC, 27601
>>>>>
>>>>> bherr...@redhat.comM: +19193238427 IM: bherring
>>>>> <https://red.ht/sig>
>>>>>
>>>>>
>>>>> On Mon, Mar 4, 2019 at 1:44 PM Kersom  wrote:
>>>>>
>>>>>> I do not think we should names in Pulp 2. Since this can cause
>>>>>> impacts that we do not know. This will increase the amount of time that 
>>>>>> we
>>>>>> will spend working on Pulp 2, changing, fixing, testing. At this point 
>>>>>> less
>>>>>> changes in Pulp 2 is what I think we should do.
>>>>>>
>>>>>> On Mon, Mar 4, 2019 at 1:10 PM Dana Walker 
>>>>>> wrote:
>>>>>>
>>>>>>> As I understand the discussion on 4497, it was to be hyphens *in
>>>>>>> addition to* a name change, but you're right @ehelms that I only see the
>>>>>>> hyphen change.
>>>>>>>
>>>>>>> I'm +1 on @rchan's suggestion that the change take place in pulp2.
>>>>>>>
>>>>>>> Also given the migration and complexities with support, I agree with
>>>>>>> @ehelms that custom configuration of these names would be problematic, 
>>>>>>> so
>>>>>>> I'm -0 on this unless we have a compelling user story for needing the
>>>>>>> customizability (assuming we are making the change to the service names 
>>>>>>> in
>>>>>>> pulp2 ourselves).
>>>>>>>
>>>>>>> --Dana
>>>>>>>
>>>>>>> Dana Walker
>>>>>>>
>>>>>>> Associate Software Engineer
>>>>>>>
>>>>>>> Red Hat
>>>>>>>
>>>>>>> <https://www.redhat.com>
>>>>>>> <https://red.ht/sig>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 4, 2019 at 1:06 PM Dennis Kliban 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I agree with @rchan that we will require users to upgrade to  a
>>>>>>>> minimal version of Pulp 2 before they can upgrade to Pulp 3.
>>>>>>>>
>>>>>>>> We should just rename Pulp 2 services in a future release of Pulp
>>>>>>>> 2.
>>>>>>>>
>>>>>>>> On Mon, Mar 4, 2019 at 11:31 AM Eric Helms 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Howdy,
>>>>>>>>>
>>>>>>>>> In some migration of Pulp 2 to Pulp 3 cases, both will need to be
>>>>>>>>> ran side-by-side on the same box. Given that pulp workers and pulp 
>>>>>>>>> resource
>>>>>>>>> manager are the same concept in both, this leads to their systemd 
>>>>>>>>> resources
>>>>>>>>> being named the same (or in today's case so slightly different enough 
>>>>>>>>> you
>>>>>>>>> can't tell them apart).
>>>>>>>>>
>>>>>>>>> I'd like to propose a change to the service names to facilitate
>>>>>>>>> this situation.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Option 1: Include Pulp version in Pulp 3 services
>>>>>>>>>
>>>>>>>>> Example: pulp3-resource-manager
>>>>>>>>>
>>>>>>>>> Pro: Explicit naming and understanding of new services.
>>>>>>>>>
>>>>>>>>> Con: This locks services names to Pulp version, which will be odd
>>>>>>>>> with semantic versioning if 4 or 5 comes along.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Option 2: Re-name Pulp 2 services to pulp2-
>>>>>>>>>
>>>>>>>>> Example: pulp2-resource-manager
>>>>>>>>>
>>>>>>>>> Pro: Explicitly identifies pulp2 services, easy to retro-fit by
>>>>>>>>> users onto their setups or through RPM releases.
>>>>>>>>>
>>>>>>>>> Con: Requires users to have upgraded to at least a particular
>>>>>>>>> Pulp2 version to migrate to Pulp 3 (this may be required anyway).
>>>>>>>>> ___
>>>>>>>>> Pulp-dev mailing list
>>>>>>>>> Pulp-dev@redhat.com
>>>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>>>
>>>>>>>> ___
>>>>>>>> Pulp-dev mailing list
>>>>>>>> Pulp-dev@redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>>
>>>>>>> ___
>>>>>>> Pulp-dev mailing list
>>>>>>> Pulp-dev@redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>
>>>>>> ___
>>>>>> Pulp-dev mailing list
>>>>>> Pulp-dev@redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>
>>>>> ___
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev@redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>> ___
>>>> Pulp-dev mailing list
>>>> Pulp-dev@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 2 and 3 Service Name Clashes

[Eric Helms]

On Mon, Mar 4, 2019 at 12:50 PM Austin Macdonald 
wrote:

> There is some additional work to be done with the installer
> https://pulp.plan.io/issues/4187#note-3
>
> I've created a new story for the installer to allow a user to override the
> default and specify whatever name they choose for each component.
> https://pulp.plan.io/issues/4497
>

Sorry to be a bit of nacker here, but I don't want to set my own naming
conventions. And, as a project, I wouldn't want users overriding that
either. The less opinionated you are the harder the supportability, as you
are introducing variations that any user or developer has to parse to
understand the issue.

The changing of Pulp 2's service names seems the least invasive as they
would be changed in the RPM spec files and automatically updated when that
version was installed. This gives Pulp 3+ the cleanest flexibility going
forward and more clearly identifies legacy components.


>
>
> On Mon, Mar 4, 2019 at 12:32 PM Eric Helms  wrote:
>
>> If I read the solution as hyphens vs underscores as implemented in
>> ansible-pulp3 today then yes, it's still very confusing which is which.
>>
>> On Mon, Mar 4, 2019, 12:25 PM David Davis  wrote:
>>
>>> I agree with rchan and am thus leaning towards option 2.
>>>
>>> Just to be clear though, we renamed pulp 3’s services recently to avoid
>>> conflict[0] with pulp 2. However, it sounds like this solution isn’t good
>>> enough as it’s hard for users to identify which set of services go with
>>> which version of pulp?
>>>
>>> [0] https://pulp.plan.io/issues/4187
>>>
>>> David
>>>
>>>
>>> On Mon, Mar 4, 2019 at 11:55 AM Robin Chan  wrote:
>>>
>>>> See comment below on option 2.
>>>>
>>>> On Mon, Mar 4, 2019 at 11:31 AM Eric Helms  wrote:
>>>>
>>>>> Howdy,
>>>>>
>>>>> In some migration of Pulp 2 to Pulp 3 cases, both will need to be ran
>>>>> side-by-side on the same box. Given that pulp workers and pulp resource
>>>>> manager are the same concept in both, this leads to their systemd 
>>>>> resources
>>>>> being named the same (or in today's case so slightly different enough you
>>>>> can't tell them apart).
>>>>>
>>>>> I'd like to propose a change to the service names to facilitate this
>>>>> situation.
>>>>>
>>>>>
>>>>> Option 1: Include Pulp version in Pulp 3 services
>>>>>
>>>>> Example: pulp3-resource-manager
>>>>>
>>>>> Pro: Explicit naming and understanding of new services.
>>>>>
>>>>> Con: This locks services names to Pulp version, which will be odd with
>>>>> semantic versioning if 4 or 5 comes along.
>>>>>
>>>>>
>>>>> Option 2: Re-name Pulp 2 services to pulp2-
>>>>>
>>>>> Example: pulp2-resource-manager
>>>>>
>>>>> Pro: Explicitly identifies pulp2 services, easy to retro-fit by users
>>>>> onto their setups or through RPM releases.
>>>>>
>>>>> Con: Requires users to have upgraded to at least a particular Pulp2
>>>>> version to migrate to Pulp 3 (this may be required anyway).
>>>>>
>>>> [rchan] My expectation is that we will levy this requirement on
>>>> upgrades/migrations anyway, so I don't think this con applies for this
>>>> suggestion.
>>>>
>>>> ___
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev@redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>> ___
>>>> Pulp-dev mailing list
>>>> Pulp-dev@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 2 and 3 Service Name Clashes

[Eric Helms]

If I read the solution as hyphens vs underscores as implemented in
ansible-pulp3 today then yes, it's still very confusing which is which.

On Mon, Mar 4, 2019, 12:25 PM David Davis  wrote:

> I agree with rchan and am thus leaning towards option 2.
>
> Just to be clear though, we renamed pulp 3’s services recently to avoid
> conflict[0] with pulp 2. However, it sounds like this solution isn’t good
> enough as it’s hard for users to identify which set of services go with
> which version of pulp?
>
> [0] https://pulp.plan.io/issues/4187
>
> David
>
>
> On Mon, Mar 4, 2019 at 11:55 AM Robin Chan  wrote:
>
>> See comment below on option 2.
>>
>> On Mon, Mar 4, 2019 at 11:31 AM Eric Helms  wrote:
>>
>>> Howdy,
>>>
>>> In some migration of Pulp 2 to Pulp 3 cases, both will need to be ran
>>> side-by-side on the same box. Given that pulp workers and pulp resource
>>> manager are the same concept in both, this leads to their systemd resources
>>> being named the same (or in today's case so slightly different enough you
>>> can't tell them apart).
>>>
>>> I'd like to propose a change to the service names to facilitate this
>>> situation.
>>>
>>>
>>> Option 1: Include Pulp version in Pulp 3 services
>>>
>>> Example: pulp3-resource-manager
>>>
>>> Pro: Explicit naming and understanding of new services.
>>>
>>> Con: This locks services names to Pulp version, which will be odd with
>>> semantic versioning if 4 or 5 comes along.
>>>
>>>
>>> Option 2: Re-name Pulp 2 services to pulp2-
>>>
>>> Example: pulp2-resource-manager
>>>
>>> Pro: Explicitly identifies pulp2 services, easy to retro-fit by users
>>> onto their setups or through RPM releases.
>>>
>>> Con: Requires users to have upgraded to at least a particular Pulp2
>>> version to migrate to Pulp 3 (this may be required anyway).
>>>
>> [rchan] My expectation is that we will levy this requirement on
>> upgrades/migrations anyway, so I don't think this con applies for this
>> suggestion.
>>
>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulp 2 and 3 Service Name Clashes

[Eric Helms]

Howdy,

In some migration of Pulp 2 to Pulp 3 cases, both will need to be ran
side-by-side on the same box. Given that pulp workers and pulp resource
manager are the same concept in both, this leads to their systemd resources
being named the same (or in today's case so slightly different enough you
can't tell them apart).

I'd like to propose a change to the service names to facilitate this
situation.


Option 1: Include Pulp version in Pulp 3 services

Example: pulp3-resource-manager

Pro: Explicit naming and understanding of new services.

Con: This locks services names to Pulp version, which will be odd with
semantic versioning if 4 or 5 comes along.


Option 2: Re-name Pulp 2 services to pulp2-

Example: pulp2-resource-manager

Pro: Explicitly identifies pulp2 services, easy to retro-fit by users onto
their setups or through RPM releases.

Con: Requires users to have upgraded to at least a particular Pulp2 version
to migrate to Pulp 3 (this may be required anyway).
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Providing Pulp 3 Container Images and Installation Support

[Eric Helms]

On Tue, Feb 12, 2019 at 1:00 PM Daniel Alley  wrote:

> There's another thing to think about which is that not all dependencies
> plugins will need need will be available as Python packages without
> substantial effort.
>
> Take the RPM plugin as an example.  It's likely to depend on at least 3
> native libraries once we reach feature parity with Pulp 2.
>
>- createrepo_c
>- libsolv
>- libmodulemd
>
> Of these:
>
>- createrepo_c can be installed via Python package, but you have to
>install the build dependencies on your local machine because there's a
>compilation step that happens behind the scenes.  We could make pre-built
>wheels available which would eliminate that issue, but we would need to set
>up (and maintain) a pipeline to do this because upstream isn't likely to be
>willing to do so themselves.
>- Libsolv is not available as a Python package, you have to install
>the RPM packages "libsolv" and "python{2|3}-solv".  "libsolv" is available
>on Debian/Ubuntu also but I don't think that the Python bindings for it
>are.  I'm not 100% sure about this.
>- Libmodulemd is not available as a Python package, nor is it
>available on Debian/Ubuntu at all.  It's only available in the RPM
>ecosystem.
>
> If we're going for primarily container-based installation anyways using a
> Fedora base image, it would probably be substantially easier to provide
> these things via RPM than to make every single dependency installable from
> PyPI.  We'd would probably be the only users of those packaging methods,
> and hence, likely need to maintain it ourselves...
>

If these packages are available already as RPMs, from the distributions
repositories what would require us to do more work than simply install the
RPM packages inside the container?


>
>
> On Mon, Jan 21, 2019 at 2:47 PM Brian Bouterse 
> wrote:
>
>> Thank you for this. I put some responses inline. Does it make any sense
>> to put this well written proposal (and its updates over time) into a wiki
>> page?  https://pulp.plan.io/projects/pulp/wiki/
>>
>>
>> On Fri, Jan 11, 2019 at 1:27 PM Eric Helms  wrote:
>>
>>> Howdy,
>>>
>>> A few months back, the team that handles RPM packaging for Pulp 2 (my
>>> team) and the Pulp team got together to discuss production 'packaging'
>>> support for Pulp 3. Where we landed was to take a container native approach
>>> from the outset and provide installation support via the Ansible installer
>>> for Pulp 3. This RFC aims to outline the general strategy, link to existing
>>> work and aim towards next steps.
>>>
>>> Note: We do not plan to provide RPMs at this time in favor of a
>>> container native solution for deploying Pulp 3. If the user demand becomes
>>> high enough, we'll look to discuss and collaborate on how to move forward.
>>>
>>> Proposal
>>>
>>> The proposal is to package Pulp 3 via containers using pypi assets as
>>> the de-facto input source for building the container image(s). The image(s)
>>> would be stored on https://quay.io and initially provided with support
>>> through the ansible-pulp3 installer. The installer would pull and run the
>>> images via systemd using podman as the default with support for Docker.
>>> This will provide users with a singular interface for deploying and
>>> managing Pulp 3 no matter the choice of installation (e.g. source, pypi,
>>> containers). In the future, existing work to deploy on to Kubernetes would
>>> be brought into the installer to support deploying the containers on top of
>>> Kubeneretes or Openshift.
>>>
>> This sounds consistent with the discussions at PulpCon. +1
>>
>>>
>>> Given Pulp is a plugin based ecosystem, and images are immutable plugins
>>> need to be handled in a user configurable way. In this space, the original
>>> proposal was to provide enough build tooling and documentation to allow
>>> users to re-build Pulp images with any combination of plugins they require.
>>> The Pulp published image(s) would include a base image and at least one
>>> image with a specific combination of plugins provided. This is an area that
>>> has had much discussion, and ideas. Ideas such as images that deploy plugin
>>> code to a shared volume, sidecar containers that provide plugins at
>>> runtime, a single image with all plugins and runtime configuration and the
>>> Discourse model of rebuilding the image locally when a plugin is added via
>>> a commandline.
>>>
>&g

Re: [Pulp-dev] pulplift with source installs for developers

[Eric Helms]

Ready to transfer! Dennis I sent you a collaborator invite to make you an
admin to do the transfer.

On Mon, Jan 14, 2019 at 5:08 PM Dennis Kliban  wrote:

> I just merged the changes to ansible-pulp3 and the Eric already merged the
> changes to pulplift.
>
> On Mon, Jan 14, 2019 at 4:00 PM Brian Bouterse 
> wrote:
>
>> I switched to it today, and it's been working well for me. +1 to adopting
>> it as the official dev environment since it consolidates the ansible
>> playbooks into one asset so we can stop maintaining two.
>>
>> When is the right time to move pulplift from @ehelms to the Pulp
>> organization on github?
>>
>
> I'd like this to happen ASAP.
>
>
>> When we do move it can we just land it where we land all the general
>> repos which is the "Pulp Team" group on github?
>>
>
> Yes.
>
>
>> On Fri, Jan 11, 2019 at 11:16 AM Dennis Kliban 
>> wrote:
>>
>>> Thanks for putting this together. After checking out your PRs I was able
>>> to provision a development environment. Everything worked as expected. I'd
>>> like to see us switch to these developer environments next week.
>>>
>>> Has anyone else had a chance to try it out?
>>>
>>> On Wed, Jan 9, 2019 at 6:10 PM Dana Walker  wrote:
>>>
>>>> I have some initial PRs [0][1][2] up for using pulplift to create a
>>>> development environment for pulp3 as described in 4243. [3]  Please take a
>>>> look and provide feedback on any changes we might want to make.  Feel free
>>>> to discuss what we'd like to see here or comment directly on the PRs.
>>>>
>>>> Note, I haven't added the requirements for building the docs yet, so
>>>> that update will come tomorrow, but I wanted to get this out here and start
>>>> hearing from all of you.
>>>>
>>>> [0] https://github.com/pulp/ansible-pulp3/pull/61
>>>> [1] https://github.com/ehelms/pulplift/pull/3
>>>> [2] https://github.com/pulp/pulp/pull/3824
>>>> [3] https://pulp.plan.io/issues/4234
>>>>
>>>> Thanks!
>>>>
>>>> --Dana
>>>>
>>>> Dana Walker
>>>>
>>>> Associate Software Engineer
>>>>
>>>> Red Hat
>>>>
>>>> <https://www.redhat.com>
>>>> <https://red.ht/sig>
>>>>
>>>>
>>>> On Mon, Jan 7, 2019 at 8:34 PM Eric Helms  wrote:
>>>>
>>>>> The current setup would have each individual configure custom boxes to
>>>>> suit their needs. If there was a fairly common setup for location of 
>>>>> source
>>>>> code and mount options we could update the source boxes or create a set of
>>>>> boxes designed for mounting by default to make spinning up easier.
>>>>>
>>>>> On Fri, Jan 4, 2019 at 4:58 PM Dennis Kliban 
>>>>> wrote:
>>>>>
>>>>>> Thanks Brian! This looks like exactly what we need.
>>>>>>
>>>>>> On Fri, Jan 4, 2019 at 4:54 PM Brian Bouterse 
>>>>>> wrote:
>>>>>>
>>>>>>> pulplift itself relies heavily on "forklift" which itself has some
>>>>>>> good docs. Take a look at these docs and see if it generates new 
>>>>>>> questions.
>>>>>>>
>>>>>>> https://github.com/theforeman/forklift#using-sshfs-to-share-folders
>>>>>>> https://github.com/theforeman/forklift#adding-custom-boxes
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 4, 2019 at 4:49 PM Dennis Kliban 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> We currently use Vagrant directly to create our development
>>>>>>>> environment[0]. I would like to be able to use pulplift to create a
>>>>>>>> development environment for Pulp 3. Our current Vagrantfile uses SSHFS 
>>>>>>>> to
>>>>>>>> mount source code on the host onto the vagrant box[1]. This allows
>>>>>>>> developers to make changes locally on their laptop and the same code 
>>>>>>>> to run
>>>>>>>> inside the vagrant box.
>>>>>>>>
>>>>>>>> pulplift already provides boxes named 'pulp3-source-*'. These boxes
>>>>>>>> use a playbook[2] from ansible-pulp3 repo to perform

[Pulp-dev] Providing Pulp 3 Container Images and Installation Support

[Eric Helms]

Howdy,

A few months back, the team that handles RPM packaging for Pulp 2 (my team)
and the Pulp team got together to discuss production 'packaging' support
for Pulp 3. Where we landed was to take a container native approach from
the outset and provide installation support via the Ansible installer for
Pulp 3. This RFC aims to outline the general strategy, link to existing
work and aim towards next steps.

Note: We do not plan to provide RPMs at this time in favor of a container
native solution for deploying Pulp 3. If the user demand becomes high
enough, we'll look to discuss and collaborate on how to move forward.

Proposal

The proposal is to package Pulp 3 via containers using pypi assets as the
de-facto input source for building the container image(s). The image(s)
would be stored on https://quay.io and initially provided with support
through the ansible-pulp3 installer. The installer would pull and run the
images via systemd using podman as the default with support for Docker.
This will provide users with a singular interface for deploying and
managing Pulp 3 no matter the choice of installation (e.g. source, pypi,
containers). In the future, existing work to deploy on to Kubernetes would
be brought into the installer to support deploying the containers on top of
Kubeneretes or Openshift.

Given Pulp is a plugin based ecosystem, and images are immutable plugins
need to be handled in a user configurable way. In this space, the original
proposal was to provide enough build tooling and documentation to allow
users to re-build Pulp images with any combination of plugins they require.
The Pulp published image(s) would include a base image and at least one
image with a specific combination of plugins provided. This is an area that
has had much discussion, and ideas. Ideas such as images that deploy plugin
code to a shared volume, sidecar containers that provide plugins at
runtime, a single image with all plugins and runtime configuration and the
Discourse model of rebuilding the image locally when a plugin is added via
a commandline.

Existing Work

The existing work in this area is around two pull requests that have
provided a working proof of concept. The first is the image itself [1] and
the second is to the installer [2] that makes use of this image.

The current works' biggest design assumption (up for debate) is around a
single image vs. an image per service. The strategy difference is around a
single image that can, with the right start command, take on the role of
any Pulp 3 service. The multi-image strategy would aim to ensure there is
an image per service that knows how to properly run itself. A single image
provides a simpler build strategy to start with potentially more confusion
to users wanting to use them within their own context.

Open Questions

 * What are the communities thoughts on this approach?
 * What are the gaps to get both [1] and [2] into the respective code bases?
 * Should we stick with a single image strategy or opt to pivot to
multi-image?
 * What changes need to be made to selinux policies?
 * How is plugin support handled?

Next Steps

To move forward, I'll be looking for a Pulp maintainer to help sponsor and
shepherd this work into the Pulp ecosystem. Along with that a rough list:

 * Resolve open questions
 * Create Pulp account on quay.io
 * Add deployment support to Travis configuration
 * Get initial installer work merged

[1] https://github.com/pulp/pulp/pull/3740
[2] https://github.com/pulp/ansible-pulp3/pull/45
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulplift with source installs for developers

[Eric Helms]

The current setup would have each individual configure custom boxes to suit
their needs. If there was a fairly common setup for location of source code
and mount options we could update the source boxes or create a set of boxes
designed for mounting by default to make spinning up easier.

On Fri, Jan 4, 2019 at 4:58 PM Dennis Kliban  wrote:

> Thanks Brian! This looks like exactly what we need.
>
> On Fri, Jan 4, 2019 at 4:54 PM Brian Bouterse  wrote:
>
>> pulplift itself relies heavily on "forklift" which itself has some good
>> docs. Take a look at these docs and see if it generates new questions.
>>
>> https://github.com/theforeman/forklift#using-sshfs-to-share-folders
>> https://github.com/theforeman/forklift#adding-custom-boxes
>>
>>
>> On Fri, Jan 4, 2019 at 4:49 PM Dennis Kliban  wrote:
>>
>>> We currently use Vagrant directly to create our development
>>> environment[0]. I would like to be able to use pulplift to create a
>>> development environment for Pulp 3. Our current Vagrantfile uses SSHFS to
>>> mount source code on the host onto the vagrant box[1]. This allows
>>> developers to make changes locally on their laptop and the same code to run
>>> inside the vagrant box.
>>>
>>> pulplift already provides boxes named 'pulp3-source-*'. These boxes use
>>> a playbook[2] from ansible-pulp3 repo to perform the installation. By
>>> default, this playbook assumes that the vagrant box has pulp source code at
>>> "/home/vagrant/devel/pulp". However, without the synced folders, the
>>> source code is not there and the install fails.
>>>
>>> I would like to extend pulplift to use synced folders for the source
>>> install boxes. However, I am not sure where to add such directives for
>>> vagrant.
>>>
>>> Eric, can you point me in the right direction?
>>>
>>>
>>> [0] https://github.com/pulp/devel/blob/master/Vagrantfile.example
>>> [1] https://github.com/pulp/devel/blob/master/Vagrantfile.example#L8
>>> [2] https://github.com/pulp/ansible-pulp3/blob/master/source-install.yml
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Using commits instead of PRs in Pulp 3 release notes

[Eric Helms]

On Katello and Foreman, we use a template and some code from a tool we use
to do release activities to generate a formatted changelog that provides
both sets of information (Redmine issue links and commit links) broken down
by categories, see example at [1].

I'd be happy to give more information on where and how we generate this if
desired.

[1] https://github.com/Katello/katello/blob/KATELLO-3.9/CHANGELOG.md

On Thu, Dec 13, 2018 at 2:29 PM Dana Walker  wrote:

> I feel like this is more accurate and comprehensive, but the initial
> summary is less readable.  I could go either way.  Others?
>
> --Dana
>
> Dana Walker
>
> Associate Software Engineer
>
> Red Hat
>
> 
> 
>
>
> On Thu, Dec 13, 2018 at 2:24 PM David Davis  wrote:
>
>> Currently in our Pulp 3 release notes[0], we link to a query of PRs with
>> the 3.0 tag merged in a certain timeframe. I feel like this is prone to
>> user errors as in cases where we might forget to add the 3.0 label to a PR
>> or get the timeframe slightly wrong because of timezones. I was thinking
>> instead we could maybe link to the list of commits between tags. Here’s an
>> example:
>>
>> https://github.com/pulp/pulp/compare/pulpcore-3.0.0b15...pulpcore-3.0.0b16
>>
>> If you click on any of these commits, there is a link to the PR by the
>> branch name so I feel like there’s no loss of information.
>>
>> Thoughts?
>>
>> [0]
>> https://docs.pulpproject.org/en/3.0/nightly/release-notes/pulpcore/3.0.x.html
>>
>> David
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Splitting ansible-pulp3 responsibilities

[Eric Helms]

If this is better off as an issue in Redmine, please let me know but I was
honestly not sure where to raise this kind of issue so mailing list it is
to start.

Background

While playing around with the ansible-pulp3 repository I found myself
wanting to run the scenarios to ensure that I had a basic grasp of the
install flow and to ensure that any changes I made did not break existing
scenarios. I found myself having to copy Vagrantfile's around for each
scenario, including adding new ones based on other OSes (e.g. CentOS) until
things got messy.

General Proposal

Based on my experience, I am proposing to split the "installer" portion
(e.g. all of the core Ansible) from the provisioning pieces (e.g. Vagrant)
across two separate git repositories.

Potential Solution for Proposal

The above is a general proposal. What follows is a potential solution that
I built out and used locally to get myself up to speed running and testing
for multiple scenarios and OSes. In the Foreman community we built
ourselves a simple little tool called Forklift that allows configuring
Vagrant through yaml, and building custom boxes built based on previous box
definitions. This has worked quite well for both creating production, test
and development environments for many years. Given my experience I popped
out a "Pulplift" using Forklift as a library to provide boxes for Fedora
27, 28 and CentOS 7 for source and sandbox installations. You can find the
repo at [1] with some starting notes about what the repository can do in
the README.

Whether Pulplift is adopted or not, I think there is a benefit to splitting
per the general proposal. Further, this split will allow the installer bits
to remain Ansible focused and the Vagrant bits to be seen as testing, and
evaluation code.

Thanks for consideration,
Eric

[1] https://github.com/ehelms/pulplift
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Using Gunicorn and Static Files

[Eric Helms]

I tested this method out and it did work with gunicorn. The only ugly part
is having to re-define the MIDDLEWARE entirely. I couldn't find a way to
simply just add the middleware I wanted in the spot I needed it.

Brian any thoughts on supporting this out of the box? I'm happy to do the
work but I'd like to know it's desired.

On Tue, Oct 9, 2018 at 9:11 AM Patrick Creech  wrote:

> On Wed, 2018-10-03 at 16:28 -0400, Eric Helms wrote:
> > Howdy,
> >
> > When switching a deployment over to use gunicorn, DEBUG = TRUE for
> serving static files stopped working. I endeavored to follow the production
> install method using collectstatic. This required
> > setting STATIC_ROOT which appeared to not be set by default.
> >
> >  1) Is there a default value for STATIC_ROOT I can set for collectstatic?
> >  2) Can gunicorn serve these static files for me?
> >
> > I realize the documentation calls for serving static files via a
> webserver like nginx or Apache. However, that becomes a bit overkill in
> something like a container deployment. If I run a separate
> > webserver, then I would run it as a separate container and have to mount
> the static files volume. This felt like overkill to me to have a persistent
> volume to store and mount static files instead of
> > the application server providing them for the application.
> >
> > Any thoughts and help are appreciated.
> >
> > Eric
>
> Eric,
>
> I got curious and took a look into your situation.
>
> I came across this project, suggested to be used by heroku, to achieve the
> goal you are working towards: http://whitenoise.evans.io/en/stable/
>
> It comes with django middleware support out of the box, so all you would
> need to do is to 'pip install whietnoise' and edit the MIDDLEWARE
> collection in the settings.py to include it,  You want it to
> load AFTER SecurityMiddleware, but BEFORE everything else, so make it the
> second line, here:
> https://github.com/pulp/pulp/blob/master/pulpcore/pulpcore/app/settings.py#L91
>
> For your deployment model, it shouldn't be hard to apply a patch to the
> code at a stage in your container process to inject this middleware into
> the settings.py
>
> Otherwise, it's the standard 'staticfiles' process for django,  They have
> it simplified in step 1 here:
> http://whitenoise.evans.io/en/stable/django.html
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Using Gunicorn and Static Files

[Eric Helms]

On Fri, Oct 5, 2018 at 4:45 PM Brian Bouterse  wrote:

> Hi Eric, Thanks for writing, see inline.
>
> On Wed, Oct 3, 2018 at 4:28 PM Eric Helms  wrote:
>
>> Howdy,
>>
>> When switching a deployment over to use gunicorn, DEBUG = TRUE for
>> serving static files stopped working. I endeavored to follow the production
>> install method using collectstatic. This required setting STATIC_ROOT which
>> appeared to not be set by default.
>>
>>  1) Is there a default value for STATIC_ROOT I can set for collectstatic?
>>
> I think we should set one, and more and more we're having everything live
> in /var/lib/pulp// so it could have a default of
> /var/lib/pulp/static_media/. Do you have any interest in sending us a PR
> and opening an issue?
>

Sure.


>
>  2) Can gunicorn serve these static files for me?
>>
> I looked into doing this a while back and I reached the conclusion that we
> could but we shouldn't. It is convenient to not have to do anything extra
> to deploy it, but the Django docs say in several places that we should
> expressly not do this. Convenience is more than a nicety, the longer Pulp
> takes to deploy the fewer users we will have. I think the Ansible Installer
> and templates on openshift should let us do it the recommended way (with
> apache or nginx, etc) and have it be just as easy for the user.
> Unfortunately no one has done that ... yet!
>
> Reach out to me on irc if you want any help with any of those things.
>

Can you clarify why the deployment would take longer if the application
itself is serving up the static files?

In a containerized setup, this current model would require a persistent
volume store dedicated just to static files (seems like overkill) and
running a separate webserver just to serve those files. I will agree the
likelyhood of running Apache or Nginx is high to handle SSL and routing for
the user in a deployment, however, I still find the static files part a bit
odd to be required to run the application.

>
>
>> I realize the documentation calls for serving static files via a
>> webserver like nginx or Apache. However, that becomes a bit overkill in
>> something like a container deployment. If I run a separate webserver, then
>> I would run it as a separate container and have to mount the static files
>> volume. This felt like overkill to me to have a persistent volume to store
>> and mount static files instead of the application server providing them for
>> the application.
>>
>> Any thoughts and help are appreciated.
>>
>> Eric
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Using Gunicorn and Static Files

[Eric Helms]

Howdy,

When switching a deployment over to use gunicorn, DEBUG = TRUE for serving
static files stopped working. I endeavored to follow the production install
method using collectstatic. This required setting STATIC_ROOT which
appeared to not be set by default.

 1) Is there a default value for STATIC_ROOT I can set for collectstatic?
 2) Can gunicorn serve these static files for me?

I realize the documentation calls for serving static files via a webserver
like nginx or Apache. However, that becomes a bit overkill in something
like a container deployment. If I run a separate webserver, then I would
run it as a separate container and have to mount the static files volume.
This felt like overkill to me to have a persistent volume to store and
mount static files instead of the application server providing them for the
application.

Any thoughts and help are appreciated.

Eric
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposal to change default logging to console in Pulp 3

[Eric Helms]

Sorry, meant to include the issue:

https://pulp.plan.io/issues/3883

On Wed, Jul 25, 2018 at 11:31 AM, Eric Helms  wrote:

> Thanks for the quick discussion. I went ahead and opened an issue and
> accompanying PR for the work to make this change for y'all to consider.
>
> On Wed, Jul 25, 2018 at 10:59 AM, Daniel Alley  wrote:
>
>> Not to say syslog is dead, it's especially useful for clustered installs
>>> which need fancier logging like centralization or off-site replication, etc.
>>
>>
>> And a lot of people who do use syslog are just telling journald to
>> forward logs there, in which case, console logging is still a good default.
>>
>> On Wed, Jul 25, 2018 at 10:42 AM, Brian Bouterse 
>> wrote:
>>
>>> I think console logging is also probably the right logging default in
>>> non containerized environments too. The majority of OSes Pulp will run on
>>> will run systemd which has it's logging facilities reading from stdout and
>>> stderr [0]. So for the majority of non-containerized userbase systemd is
>>> their logging system not syslog. Not to say syslog is dead, it's especially
>>> useful for clustered installs which need fancier logging like
>>> centralization or off-site replication, etc. Those users can take the extra
>>> step to configure logging to syslog if they apply a bit of extra config
>>> after https://pulp.plan.io/issues/3879 is fixed.
>>>
>>> It also being the expected logging for containerized environments makes
>>> me think we should switch Pulp3 to console logging as the default. What do
>>> you think?
>>>
>>> [0]: https://www.freedesktop.org/software/systemd/man/systemd-jou
>>> rnald.service.html#Description
>>>
>>>
>>> On Wed, Jul 25, 2018 at 10:31 AM, Eric Helms  wrote:
>>>
>>>> You actually can run with console logging today. There is a bug where
>>>> if you are in an environment without syslog and enable console logging you
>>>> will get errors, see https://pulp.plan.io/issues/3879
>>>>
>>>> Since background running typically translates to systemd based, perhaps
>>>> considering making logging type configurable as an environment variable or
>>>> command line option to set it per service. For example, perhaps you want to
>>>> run workers in the background but the main application server in your
>>>> console?
>>>>
>>>>
>>>> On Wed, Jul 25, 2018 at 9:17 AM, David Davis 
>>>> wrote:
>>>>
>>>>> I definitely think that we should support console logging (and it
>>>>> looks like we currently don’t) but I’m a bit hesitant to make it the
>>>>> default. I think right now a majority of non-dev users will be running
>>>>> these services in the background. But I could see perhaps one day where
>>>>> Pulp is mostly run in containers where console logging would be the better
>>>>> default.
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>> On Wed, Jul 25, 2018 at 8:17 AM Eric Helms  wrote:
>>>>>
>>>>>> Howdy,
>>>>>>
>>>>>> As I understand things, the default logging in Pulp 3 is set to
>>>>>> syslog. I am proposing changing this to be console by default and
>>>>>> optionally letting users configure syslog. I think these days console as 
>>>>>> a
>>>>>> default makes general sense for running things in a simple terminal, or 
>>>>>> as
>>>>>> a developer. Further, the natural way to log in a container based
>>>>>> environment is to log to stdout. The use of syslog makes sense in a
>>>>>> traditional systemd based deployment which can be configured by users as
>>>>>> needed.
>>>>>>
>>>>>>
>>>>>> - Eric
>>>>>> ___
>>>>>> Pulp-dev mailing list
>>>>>> Pulp-dev@redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>
>>>>>
>>>>
>>>> ___
>>>> Pulp-dev mailing list
>>>> Pulp-dev@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 Release Process Questions

[Eric Helms]

On Fri, Apr 13, 2018 at 4:40 PM, Dennis Kliban  wrote:

> On Fri, Apr 13, 2018 at 1:25 PM, Patrick Creech 
> wrote:
>
>> Pulp,
>>
>> So, while working on the packaging work, I figured it be nice to start
>> talking about release process expectations around nightlies, beta, and GA.
>>
>> Generally, what is pulp's release plan?  What are the expectations here?
>>
>>
> The release process for Pulp 3 will be different from what we do for Pulp
> 2. Our plan for publishing Pulp 3 with quality to PyPI is outlined on our
> wiki[0]. We are hoping to be able to release to PyPI once a week during the
> beta cycle. After the packages are published to PyPI,  any of the
> derivative packaging (RPM, Debian, etc) can be performed. The build team
> can decide how often the derivative packages need to be produced.
>

This implies that, for the Pulp developer team, Pypi is considered the
release vector and that derivative release vectors (e.g. RPM, Deb, etc.)
are considered community contributions that are not part of the core
release process. Is that a fair summary of the position? Consumers of
non-pypi release vectors will need to assume a delay between announced
release and RPM release. Which then, unlike Pulp 2, means the team handling
RPM for example would manage build and release announcement on our own
schedule. I want to clarify so that we set expectations for developers and
users and so that we can set our expectations for how we shift compared to
Pulp 2.

If the above is the agreed workflow (and change for Pulp 3) I think the
rest of the questions I'd ask related to the points below are answered and
we can talk a bit further on these points above.

- Eric


>
> [0] https://pulp.plan.io/projects/pulp/wiki/Continuous_Delivery_of_Pulp_3
>
>
>> And also, more specifically,
>>
>> Based on what we do for pulp 2, when will pulp 'code freeze'? What is the
>> expected turnaround from 'code freeze to 'packages shipped'.  We should
>> probably agree on some expectations of turnaround
>> time.
>>
>>
> The code will be frozen when it is published to PyPI.
>
>
>> Is there a staging process in place yet for packages (pypi or rpm)? Is
>> there testing expectations of these pre-release bits to ensure quality?
>> With pypi being a valid install location, are these
>> releases to be coordinated?
>>
>>
> As outlined on the wiki, we plan to ensure quality at merge time of every
> commit.
>
>
>> Where are pulp 3 bits expected to be hosted?  How are we going to handle
>> signing packages?
>>
>
> Pulp 3 will always be published to PyPI. Any derivative packages can be
> hosted on fedorapeople.org. I'd like to defer to someone else to speak
> about the signing.
>
>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Seeking Nomenclature

[Eric Helms]

Thanks for the replies so far. Sounds like parent -> child might be the
best nomenclature that has been used previously and fits our style. In our
case there is only one Pulp that is master of the universe and every other
Pulp is simply a copy of pieces of the master errr parent but never
contains independent data.

On Mon, Feb 19, 2018 at 3:30 PM, Austin Macdonald <amacd...@redhat.com>
wrote:

> We also have called it "Pulp to Pulp sync" informally.
>
> On Mon, Feb 19, 2018 at 3:23 PM, David Davis <davidda...@redhat.com>
> wrote:
>
>> So syncing from one Pulp server to another Pulp server is usually called
>> 'natural syncing'. See [0][1].
>>
>> AFAIK, there is no official concept of master/nodes/etc in Pulp anymore
>> since in the case of natural syncing, there’s nothing special about either
>> instance of Pulp. We have used the terms ‘parent’ and ‘child’ though (as in
>> the blog post I linked to).
>>
>> [0] https://pulp.plan.io/issues/1488
>> [1] https://pulpproject.org/2016/12/07/deprecating-nodes/
>>
>>
>>
>> David
>>
>> On Mon, Feb 19, 2018 at 2:59 PM, Eric Helms <ehe...@redhat.com> wrote:
>>
>>> Howdy,
>>>
>>> The simple question is there any nomenclature used when referring to a
>>> Pulp server and another Pulp server that syncs from the previous one?
>>>
>>> The background behind this question is as follows. Pulp at a time had
>>> the concept of masters and nodes. With the removal of the official
>>> node-concept and code, we still have references to Pulp "nodes" in our code
>>> base. If there is newer nomenclature for the case of having a Pulp master
>>> that other Pulp instances sync from I'd like to adopt that to help with
>>> parity across communities.
>>>
>>> I should also ask if "Pulp master" is correct as well.
>>>
>>>
>>> Thanks,
>>> Eric
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Seeking Nomenclature

[Eric Helms]

Howdy,

The simple question is there any nomenclature used when referring to a Pulp
server and another Pulp server that syncs from the previous one?

The background behind this question is as follows. Pulp at a time had the
concept of masters and nodes. With the removal of the official node-concept
and code, we still have references to Pulp "nodes" in our code base. If
there is newer nomenclature for the case of having a Pulp master that other
Pulp instances sync from I'd like to adopt that to help with parity across
communities.

I should also ask if "Pulp master" is correct as well.


Thanks,
Eric
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulp 3 distributor use cases

[Eric Helms]

Thanks for the clarity. That explanation definitely helped me wrap my head
around it.

On Mon, Feb 5, 2018 at 5:47 AM, Dennis Kliban <dkli...@redhat.com> wrote:

> I update issue 2894[0] with the new name, Exporter. Can someone please
> groom this ticket?
>
> [0]https://pulp.plan.io/issues/2894
>
>
> Thanks,
> Dennis
>
> On Fri, Feb 2, 2018 at 5:28 PM, Daniel Alley <dal...@redhat.com> wrote:
>
>> +1 rename
>>
>> On Fri, Feb 2, 2018 at 10:51 AM, Austin Macdonald <amacd...@redhat.com>
>> wrote:
>>
>>> +1 rename to Exporter.
>>>
>>> On Fri, Feb 2, 2018 at 10:13 AM, Brian Bouterse <bbout...@redhat.com>
>>> wrote:
>>>
>>>> +1 to renaming a Pulp3 'Distributor' to be 'Exporter'. The name of a
>>>> 'Distribution' (when Pulp serves bits using its webserver) would stay the
>>>> same. Currently the names Distributor and Distribution sound so similar
>>>> that its very confusing. A Distribution and an Exporter would be clearer
>>>> names I think.
>>>>
>>>> On Thu, Feb 1, 2018 at 10:30 AM, Dennis Kliban <dkli...@redhat.com>
>>>> wrote:
>>>>
>>>>> Distributors are only used for exporting Publications out of Pulp.
>>>>>
>>>>> Distributions make Publications available to consume via http(s) at
>>>>> /content/relative/path/of/distribution/. Each Publication can be
>>>>> associated with any number of Distributions.
>>>>>
>>>>> The names are very similar, but perform different function. Perhaps
>>>>> Distributors should be called Exporters instead.
>>>>>
>>>>> On Thu, Feb 1, 2018 at 2:35 PM, Eric Helms <ehe...@redhat.com> wrote:
>>>>>
>>>>>> Will Pulp 3 support multiple distributors on a single repository?
>>>>>> Could I, for example, attach the yum distributor multiple times but with 
>>>>>> a
>>>>>> different output path to publish the same repository at different 
>>>>>> locations?
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Eric
>>>>>>
>>>>>> On Thu, Feb 1, 2018 at 6:54 AM, Dennis Kliban <dkli...@redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I've updated the MVP use cases for the Distributors. The diff is
>>>>>>> here[0]. I removed the use case of a distributor exporting a Repository
>>>>>>> Version. The idea is to give users a single way to export repository
>>>>>>> versions. Users will first create a publication and then use a 
>>>>>>> distributor
>>>>>>> to export that publication.
>>>>>>>
>>>>>>> Please reply to this thread with any suggestions for the use cases
>>>>>>> related to Distributors.
>>>>>>>
>>>>>>> I'd like to write up stories for these use cases early next week, so
>>>>>>> the work can be merged by Feb 15th.
>>>>>>>
>>>>>>> [0] https://pulp.plan.io/projects/pulp/wiki/Pulp_3_Minimum_Viabl
>>>>>>> e_Product/diff?utf8=%E2%9C%93=143_from=142
>>>>>>> ommit=View+differences
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Dennis
>>>>>>>
>>>>>>> ___
>>>>>>> Pulp-dev mailing list
>>>>>>> Pulp-dev@redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> ___
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev@redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>>>
>>>>
>>>> ___
>>>> Pulp-dev mailing list
>>>> Pulp-dev@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulp 3 distributor use cases

[Eric Helms]

On Thu, Feb 1, 2018 at 10:30 AM, Dennis Kliban <dkli...@redhat.com> wrote:

> Distributors are only used for exporting Publications out of Pulp.
>
> Distributions make Publications available to consume via http(s) at
> /content/relative/path/of/distribution/. Each Publication can be
> associated with any number of Distributions.
>

You lost me a bit here getting into distributions. I think I want to be
able to export the same repository to two separate published locations.

Say I have a repository A. I want to make repository A available at the
following paths:

https://myserver/pulp/repos/Acme/Dev/Version_2/
https://myserver/pulp/repos/Acme/Test/Version_2/

Is that possible in the design?


>
> The names are very similar, but perform different function. Perhaps
> Distributors should be called Exporters instead.
>

For me, either naming scheme works though perhaps for most exporters might
click easier.


>
> On Thu, Feb 1, 2018 at 2:35 PM, Eric Helms <ehe...@redhat.com> wrote:
>
>> Will Pulp 3 support multiple distributors on a single repository? Could
>> I, for example, attach the yum distributor multiple times but with a
>> different output path to publish the same repository at different locations?
>>
>>
>> Thanks,
>> Eric
>>
>> On Thu, Feb 1, 2018 at 6:54 AM, Dennis Kliban <dkli...@redhat.com> wrote:
>>
>>> I've updated the MVP use cases for the Distributors. The diff is
>>> here[0]. I removed the use case of a distributor exporting a Repository
>>> Version. The idea is to give users a single way to export repository
>>> versions. Users will first create a publication and then use a distributor
>>> to export that publication.
>>>
>>> Please reply to this thread with any suggestions for the use cases
>>> related to Distributors.
>>>
>>> I'd like to write up stories for these use cases early next week, so the
>>> work can be merged by Feb 15th.
>>>
>>> [0] https://pulp.plan.io/projects/pulp/wiki/Pulp_3_Minimum_Viabl
>>> e_Product/diff?utf8=%E2%9C%93=143_from=142
>>> ommit=View+differences
>>>
>>> Thanks,
>>> Dennis
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Webserver owning the entire url namespace?

[Eric Helms]

First, I appreciate this being a proposal and discussion before going this
route given it's implications for applications used to consuming Pulp
heavily. Secondly, I believe some of my questions and concerns have been
asked and addressed throughout the thread, but I do feel like it's reached
a point where a summary would be useful for those just now entering the
conversation to parse.

I know some of these concerns start to quickly broach into more advanced
architecture discussion than the original proposal raised Brian. I am happy
to break those out into other threads, but for now since someone else
mentioned it I am including them.

My generalized concerns are:

 * How do I deploy Pulp alongside another Apache based application (aka the
current Katello use case) ?
 * Can I deploy Pulp and the Content separately? From two perspectives,
splitting load across multiple hosts or separating concerns into
independent pieces that can be scaled per differing demands? (e.g. the Pulp
API itself may get a little traffic, whereas my 50k hosts might hammer the
content API)
 * If the Content is separate from the main Pulp API, does that mean that I
can scale my content delivery more easily horizontally and across
geographies? This kinda goes to in the new world, how does the current
setup of Pulp talking to Pulp to create replicated data endpoints for
geography and scaling look like given this is affected by how the URL
namespace is consumed.


Eric

On Fri, Nov 10, 2017 at 11:03 AM, Patrick Creech  wrote:

> On Fri, 2017-11-10 at 10:49 -0500, Brian Bouterse wrote:
>
>
>
>
>
>
> From a deployment perspective, it's been a key use case to deploy crane at
> the perimeter, rsync published image files out to a file or CDN service,
> and run the rest of Pulp on a well-protected internal network.
>
>
> Pulp can also be installed at the perimeter. Core should support a setting
> that enables/disables the REST API. Each plugin could support a setting
> that enables/disables its content API.
>
>
> I think we're envisioning a similar goal, but with a different mechanism.
> I like the idea of a user selecting which components should be active.
> Making each component a WSGI app is very easy for us and very convenient
> for users. You can see Pulp 2's WSGI apps defined here:
>
> https://github.com/pulp/pulp/tree/master/server/usr/share/pulp/wsgi
>
> Depending on whether a user wants to run each component embedded in normal
> httpd processes, or in separate daemon processes, it's just a matter of
> enabling or not a small httpd config file like this one:
>
> https://github.com/pulp/pulp/blob/master/server/etc/httpd/co
> nf.d/pulp_content.conf
>
> This gives the most flexibility. A user won't need to deploy the entire
> stack of Pulp dependencies with all of their plugins at the perimeter if
> they don't want to; we can choose to deliver each WSGI app separately, or
> not, depending on what is convenient.
>
>
> How can one process group serve multiple WSGI applications? I don't think
> it can, so it requires the user to deploy it with multiple process groups
> (one for each WSGI application). This prevents a use case that goes like:
> "As a user, I can deploy Pulp to serve content, REST API, and live APIs all
> from one WSGI process". This use case is valuable because it's both a
> simple deployment model (fewer WSGI apps) and it uses less memory because
> there are fewer process groups. This is why I'm suggesting we ship all urls
> to be handled by one WSGI application, which also allows for the deployment
> that you outline also. So shipping one WSGI app makes Pulp the most
> flexible.
>
>
> This separation has worked very well in Pulp 2, and as far as I know there
> have been no complaints about it.
>
>
> There are complaints that Pulp is hard to deploy (multiple WSGI apps), and
> that it uses too much memory.
>
>
> There are some important isolation concerns from the security and
> reliability points here that should be weight as well. You are talking
> about having general 'user facing/unauthenticated' services like content
> serving share the same process and memory space as your management
> interface (rest api). There should probably be some thought given to what
> your acceptible level of risk and exposure are when someone finds a flaw in
> your content serving code and can now see your management interface's
> memory footprint. Or say you have some unstable code that can crash in your
> management interface, which will end up bringing down your entire
> application, instead of just the management interface. Just some food for
> thought while thinking about all the ins and outs of this tradeoff.
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] file repository type

[Eric Helms]

A few suggestions from my own heavy use of the current "plugin" (and I may
repeat some previously said ideas):

 1) consider supporting syncing from a few well known types: ftp, apache
static file server
 2) Support "uploading" a file from a url
 3) Look at what wget does and recursively sync down whatever is over at
location X
 4) Use name and path as the "ID" for a file to allow re-uploads of the
same file name

I agree that syncing Git would be cool, but should be it's own entity since
Git is a known "protocol". The wget way is interesting in that I used that
to sync debian to a file repo:

 1) I synced via recursive wget a debian repository
 2) I used the script at
https://gist.github.com/ehelms/3fd956ee887db3d7bac20b29efa3dd51 to generate
a recursive (with full file path) PULP_MANIFEST
 3) I then synced that to fully mirror the repository

Steps 1-3 seem common enough that I shouldn't have to do it and/or waste
storage space just to replicate a potentially large file location.

Eric

On Tue, Jun 6, 2017 at 6:02 PM, Tom McKay  wrote:

> As a user, the standard is "wget". In foreman I entered a url to a raw
> github file... and it failed (obscurely). While one could argue for better
> error messaging, I would argue that I was pretty explicit in indicating I
> wanted the results returned from the url. When I sync or resync the
> content, just go wget it again and stuff it into the repo. Yes, it's a repo
> of one but in foreman I make "products" which are collections of repos.
> (Agreed that my next ask would be to "sync all the files listed at a url. :)
>
>
>
>
> On Tue, Jun 6, 2017 at 5:03 PM, Michael Hrivnak 
> wrote:
>
>> The key is in discovery of files. There's no standard we know of that
>> would be helpful, which is why we depend on a PULP_MANIFEST to tell us what
>> files are available. Maybe there are some cases when a particular service
>> has a well-defined way of expressing what files are available, in which
>> case we could instead support that as the discovery mechanism. It's a good
>> question about whether that would be an addition to the pulp_file plugin,
>> or a separate plugin. It probably needs to be evaluated case-by-case.
>>
>> As one related example, the Pulp 2 puppet importer supports both the
>> forge API and a PULP_MANIFEST for discovery of available puppet modules on
>> a remote server.
>>
>> On Tue, Jun 6, 2017 at 4:46 PM, Brian Bouterse 
>> wrote:
>>
>>> What if the pulp_file plugin (the pulp3 one) would support syncing files
>>> directly from remotes without a manifest? Like syncing from a remote
>>> webserver or version control system. Maybe that would be a VCS plugin, not
>>> the file_plugin, but regardless, other software similar to Pulp supports
>>> [0] that use case.
>>>
>>> [0]: http://bit.ly/2s1pCmF
>>>
>>> On Mon, Jun 5, 2017 at 1:10 PM, Michael Hrivnak 
>>> wrote:
>>>
 You do not need to create a manifest in order to upload. You only need
 one in order to sync. Here is documentation about upload:

 http://docs.pulpproject.org/plugins/pulp_rpm/user-guide/isos
 .html#uploading-isos-to-a-repository

 You can also create the manifest file in 3-4 lines of bash. Maybe it
 would be useful for Pulp to include a small script that can generate the
 manifest.

 That said, you can always upload your files to a Pulp repo, and publish
 it, which will cause Pulp to create a new manifest for you.

 On Mon, Jun 5, 2017 at 6:41 AM, Tom McKay 
 wrote:

> I'd like to sync individual files and folders (recursively?) into pulp
> via foreman. Apparently, though, I need to create a pulp manifest in order
> to upload. Is this necessary or can pulp do it for me? Or should I build
> this into foreman? @ehelms provided a gist of the steps[1] and it seems
> pretty basic and something pulp could be doing itself.
>
> My use case is that I wanted to import a build file for openshift from
> github, then promote that file (and other artifacts) through foreman's 
> life
> cycle environments. A lot of demos and tutorials for openshift start from
> git and other non-local components.
>
> [1] https://gist.github.com/ehelms/3fd956ee887db3d7bac20b29efa3dd51
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>


 --

 Michael Hrivnak

 Principal Software Engineer, RHCE

 Red Hat

 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev


>>>
>>
>>
>> --
>>
>> Michael Hrivnak
>>
>> Principal Software Engineer, RHCE
>>
>> Red Hat
>>
>
>
> ___
> Pulp-dev mailing list
> 

Re: [Pulp-dev] PUP-3: Proposal to change our git workflow

[Eric Helms]

As an outside contributor, I'd +1 switching to cherry picking to avoid
having this conversation everytime a contributor wants to clone the repo,
make a change, push the change [1]. I think you increase the likelihood of
first time and repeat contributions if users do not have to be aware of the
release process and can simply clone the repository, get the default branch
which most developers these days perceive as the head of development and
where changes should be made.

As someone who has managed project release that heavily use cherry picking,
I've found this process to be painless especially after a bit of tooling
that uses Redmine to tell me what set of issues are required, what are
their changesets and what git hashes are associated with them. Two commands
and cherry picking done.


[1] https://github.com/pulp/pulp/pull/3037

On Tue, May 30, 2017 at 1:30 PM, Bihan Zhang  wrote:

> +1 I think it's worth giving this new process a try.
> An additional benefit that I have not seen listed is that github push
> protection can be turned on to all the branches. Which would have someone
> else sanity check changes before merging.
>
> It also means each release only generates one additional commit instead of
> the 6+ commits per release generated now.
>
>
> On Tue, May 30, 2017 at 10:11 AM, Jeff Ortel  wrote:
>
>> +1
>>
>> On 05/24/2017 03:00 PM, David Davis wrote:
>> > I’d like to kick off the voting on PUP-3 which is the proposal to
>> change our git workflow by using
>> > cherry-picks instead of merging changes forward. The proposal can be
>> viewed here:
>> >
>> > https://github.com/daviddavis/pups/blob/pup3/pup-0003.md
>> >
>> > Feel free to submit any comments/nitpicks/etc on the PR[0].
>> >
>> > PUP-1 outlines our voting system:
>> >
>> > https://github.com/pulp/pups/blob/master/pup-0001.md
>> >
>> > But to sum it up:
>> >
>> > +1: "Will benefit the project and should definitely be adopted."
>> > +0: "Might benefit the project and is acceptable."
>> > -0: "Might not be the right choice but is acceptable."
>> > -1: "Not the right choice and should definitely not be adopted."
>> >
>> > I’ll set the initial deadline for the voting to be June 5th 9pm UTC.
>> >
>> > [0] https://github.com/pulp/pups/pull/3
>> >
>> > David
>> >
>> >
>> > ___
>> > Pulp-dev mailing list
>> > Pulp-dev@redhat.com
>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>> >
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev