On Thu, Apr 8, 2021 at 5:24 PM Daniel Alley <dal...@redhat.com> wrote:
> Eric, > > * The idea is to move away from RQ entirely. RQ is fine (and vastly > better than Celery IMO), but managing task state across both 1) the > database and 2) a separate, external registry is still problematic. If all > of the information can simply be kept in the database, then it will be much > easier to maintain consistent state. > Are there any benefits to improving RQ vs the invented here method? I'm just curious about the cost of maintaining a tasking system versus being part of a community built one. This feels like the kind of problem that many other applications should have in the Python world -- or are there elements of Pulp's deployment architecture that make it unique here? > * *Maybe*. We're considering using Redis as a cache to improve content > serving performance (after all, caching is one of the primary uses of > Redis). If we do, then Redis would remain in the architecture, but it could > potentially be an optional component and would be easier to remove at some > point in the future. > * We'd just be adding a small amount of information to each task record, > and it wouldn't prevent cleanup later. > This is sort of an aside to this general change. Are Pulp tasks cleaned up from the database today? > > > > On Thu, Apr 8, 2021 at 4:42 PM Eric Helms <ehe...@redhat.com> wrote: > >> A few initial questions that get a bit into the stack but will help the >> Foreman project think on the proposed changes: >> >> * Does this move away from RQ entirely or just RQ workers? >> * Do the new workers remove Pulp 3's use of Redis all together? >> * Will using the database result in any additional build up of tasking >> information that can impact performance over time? (Or does all task data >> get cleaned up eventually?) >> >> Thanks for sending this along early. >> >> On Fri, Apr 2, 2021 at 4:43 PM Brian Bouterse <bmbou...@redhat.com> >> wrote: >> >>> FYI, @mdellweg and I have been collaborating on the tasking system >>> changes. This email is to share some info to transition the work to >>> @mdellweg while I'm out. With the new-style disabled by default I am hoping >>> it can go into 3.13. >>> >>> ## The PoC and ticket info >>> >>> The PoC is basically functional, but it's a PoC: >>> https://github.com/pulp/pulpcore/pull/1222/ >>> >>> * The epic is being tracked here which recaps why we're doing this and >>> the high level approach. The sub-tasks capture the various detailed >>> changes. https://pulp.plan.io/issues/8495 >>> >>> * This is totally separate from the RQ workers you use today, and those >>> will continue to be available for a while. >>> >>> ## Next Steps >>> >>> * @mdellweg will continue the work and hopefully merge the PoC while I'm >>> out >>> >>> * Once it's demo-able I've asked @mdellweg to give a 20 minute, public >>> (hopefully recorded) technical demo. While it is designed to be a drop-in >>> replacement from a user perspective, we think sharing the internals will be >>> helpful to get feedback and increase the list of those who understand the >>> work. >>> >>> All the best, >>> Brian >>> >>> _______________________________________________ >>> Pulp-dev mailing list >>> Pulp-dev@redhat.com >>> https://listman.redhat.com/mailman/listinfo/pulp-dev >>> >> >> >> -- >> Eric Helms >> Principal Software Engineer >> Satellite >> _______________________________________________ >> Pulp-dev mailing list >> Pulp-dev@redhat.com >> https://listman.redhat.com/mailman/listinfo/pulp-dev >> > -- Eric Helms Principal Software Engineer Satellite
_______________________________________________ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
