I like the idea of having plain ActiveJob for simple background tasks and providing tasks/dynflow for complex orchestration processes (also available via ActiveJob API). While I understand whole motivation why dynflow/task was created in the first place (providing a way to do one-way orchestrations that can be fixed by users), this could open doors to rewriting some non-critical Katello operations via simple ActiveJob/Ruby (if there are any - I think calculate errata is repeatable action that goes only to Pulp to name a candidate).
I think this gives us the flexibility of both worlds - simple background task that either succeeds or fails (and its deleted from queue) or complex orchestration process written in dynflow for things like register a system in katello (multiple backends involved). Ivan, what do you think about providing non-foreman task interface via ActiveJob API? LZ On Tue, Jan 24, 2017 at 5:51 PM, Ivan Necas <ine...@redhat.com> wrote: > Ohad Levy <ohadl...@gmail.com> writes: > >> On Tue, Jan 24, 2017 at 9:51 AM, Ivan Necas <ine...@redhat.com> wrote: >> >>> Ewoud Kohl van Wijngaarden <ew...@kohlvanwijngaarden.nl> writes: >>> >>> > On Tue, Jan 24, 2017 at 12:10:34AM +0100, Ivan Necas wrote: >>> >> Ohad Levy <ohadl...@gmail.com> writes: >>> >> >>> >> > On Fri, Jan 20, 2017 at 7:14 PM, Lukas Zapletal <l...@redhat.com> >>> wrote: >>> >> > >>> >> >> Why we haven't considered ActiveJob API in the first place? This >>> looks >>> >> >> like ideal solution, easy development and documented API. >>> >> >> >>> >> >> Can't ActiveJob help us with memory issues we have with external task >>> >> >> executor in Katello? I mean what if we run background tasks inside >>> >> >> passenger process by default for easier deployment and maintenance. >>> Of >>> >> >> course, restart would mean lost queue. >>> >> >> >>> >> > >>> >> > I don't know if that's what you had in mind or not, but if it end up >>> being >>> >> > blocking the request, its not useful at all, as that would lead to >>> more >>> >> > confusion, and a developer will need to care for more usage cases (is >>> it >>> >> > blocking now, is it running async) and especially when there is a >>> need for >>> >> > ui, imho it just makes it harder. >>> >> > >>> >> > if we do move to active job api in core(+1), I could see us starting >>> >> > without tasks first for pure async operations, over time converting >>> dynflow >>> >> > to be orchestration engine vs async and orchestration. >>> >> > Futher, I would challenge a bit the orchestration implementation in >>> Katello >>> >> > (not in a negative way) but if a task fails due to unavailability of >>> one of >>> >> > the backends, the application should know how to deal with it (micro >>> >> > service / soa) rather then fail / lock the task for the future, I >>> would ask >>> >> > if this is the correct long term approach for complex operations. >>> >> >>> >> While I like challenging the things, let's not make this thread to >>> >> broad. >>> >> >>> >> I've made quite a progress with ActiveJob, and while it's not perfect, >>> >> for basic introduction of async operations it should work well enough. >>> >> >>> >> However, I'm still few days from having it ready, and I want keep the >>> >> promise to propose the solution to falling nightlies by Monday evening. >>> >> Therefore, I've opened a PR for reverting the original change >>> >> >>> >> https://github.com/theforeman/foreman/pull/4217 >>> >> >>> >> This is not a resignation of trying to get the tasks functionality into >>> >> core, but rather lowering a pressure a bit so that we can focus on the >>> >> right things to do. I hope the PR adding the functionality back, using >>> >> the approach we talked about, will come sooner rather than later, >>> followed >>> >> by adding the tasking plumbing into core, to support async operations, >>> >> as well as dynflow orchestration and current host orchestration. >>> > >>> > I greatly appreciate your effort to fix the situation. >>> > >>> > Given nightly has already been broken for a bit, I'd find it acceptable >>> > if we pushed the reversal to next week. That should give you a little >>> > bit extra time to work on ActiveJob which is most likely where we want >>> > to end up anyway. >>> > >>> > My biggest question is: in a few days (I'm guessing that's 3 or 4), will >>> > it be in mergeable shape? Would that mean we can expect it to be merged >>> > on Monday? With FOSDEM and cfgmgtcamp coming up I'd like to have nightly >>> > back into shape before it. >>> >>> I think it's realistic. If it's acceptable for people to wait for few >>> days, I'm ok with the updated plan. >>> >> >> TBH I dont fully understand the solution. >> >> we will not ship foreman-tasks by default, but rather allow a developer to >> write in a foreman-task compatible way (e.g. can work with and without >> foreman tasks), and merge foreman tasks into core at some time in the >> future. >> >> What I don't get is: >> 1. if there is no foreman tasks, then regardless of how i execute (e.g. via >> active job) it would still be blocking, UI would need to account for that - >> meaning that we still going to have to modify the code that uses it. >> 2. we are going to remove foreman-tasks from core for the time being >> anyway, so why wait on the revert? tbh, report importing in the background >> is really low prio, and puppet class importing while useful, we can live >> without just as we did for 6 years or so before. >> >> so unless I miss understand something, I would vote to remove it now to get >> nightly back. > > As I said, I don't really mind going one way or another. I think > if we revert or not doesn't have much influence on the result of > the next PR. If there are doubts, perhaps it's better to revert now (to > unblock stabilization), so that we are not pressed by the blocked build > thought the additional discussions. > > -- Ivan > >> >> Ohad >> >>> >>> -- Ivan >>> >>> > >>> > I'd propose to aim for working nightlies built on Wednesday 1 February. >>> > We most likely want a day to fix any regressions that might have slipped >>> > in so we need to merge something to Foreman develop on Monday 30 >>> > January. Since reviewers need time to review that means the ActiveJob PR >>> > needs to be there some time in advance. I can't speak for the reviewers, >>> > but I'm guessing before Monday morning would be nice. >>> > >>> > If no consensus on the ActiveJob PR can be reached by Monday evening, >>> > then the reversal is merged instead. This still allows to verify/fix >>> > other regressions on Tuesday and have working nightlies on Wednesday. >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "foreman-dev" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to foreman-dev+unsubscr...@googlegroups.com. >>> > For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "foreman-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to foreman-dev+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
