I'm not fully following the different options, hence why I will ask this question here because I probably missing something or I don't fully understand it, but I understand the problem Jens trying to solve which is in some sort avoiding scheduler waiting times when using deferred operators once the trigger has finished but in the meantime lot's of new deferred operators keep getting scheduled.
So for Option A, it would mean that we would just set the state to QUEUED directly instead of SCHEDULED, which means the task wouldn't be picked up by the scheduler anymore and the next_method specified in the trigger would get directly executed on the worker without any supervision of the scheduler, correct? For Option B, I would be against it because this probably needs more refactoring on trigger side first so it's loosely coupled, like the work that was done with Task SDK (meaning no direct db access for example), which requires more work, but if that would be the pre-requisite and still doable in short notice, then I would be in favor. For Option C I have no strong opinion, all depends how quickly this can be implemented and tested as 3.2.0 is coming close. Kind regards, David ________________________________ From: Vikram Koka via dev <[email protected]> Sent: Thursday, March 26, 2026 14:25 To: [email protected] <[email protected]> Cc: Vikram Koka <[email protected]> Subject: Re: [VOTE] Improve problems of additional latency for tasks returning from Triggerer to Worker EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar [email protected]<mailto:[email protected]>. Very similar to Jarek and Kaxil's analysis: Option A: -1 This is architecturally inconsistent with the direction we have been moving towards over the last couple of years. Option B: -0.1 Architecturally better, but the core change has significant ripple effects, requiring significant test coverage expansion. I just don't think we can achieve this quickly and maintain stability at this late stage. Option C: +1 An AIP even a mini-AIP would give us some time to work through the details. I understand the strong desire for urgency, but I believe it can be addressed by having Airflow 3.3.0 follow quickly. It is currently targeted for June and I believe that we should make every effort to hit that and have it be early in June, before people starting vanishing for summer holidays. Best regards, Vikram On Thu, Mar 26, 2026 at 4:59 AM Jarek Potiuk <[email protected]> wrote: > Very similar with Kaxil's assessment: > > - Option A: -1: I think it is against the direction of Airflow > architectural changes we work for about 1.5 years. While those aren't > completed, it would basically invalidate that work. > - Option B: -0.1 (for the reasons Kaxil mentioned > - Option C: +1 - and I think this should be addressed quickly with high > priority. We already discussed that 3.3 should be released **way faster** > than 3.2 and I think this one is yet another reason for itV > > J. > > On Thu, Mar 26, 2026 at 4:28 AM Kaxil Naik <[email protected]> wrote: > > > (details cut in last email) > > > > Hi Jens, > > > > My vote: > > > > - Option A: -0.5 > > - Option B: 0 > > - Option C: +1 > > > > Reasoning: > > > > Option A has architectural issues. The triggerer today is a client > > component running user code; it runs user triggers and reports events > back. > > PR #63489 turns it into a mini-scheduler by having it instantiate > executor > > instances > > (init_executors()), generate JWT tokens, and dispatch workloads directly > to > > Celery brokers. That's scheduler territory. > > > > This moves in the opposite direction from AIP-92, which aims to isolate > the > > triggerer further from core services (removing even direct DB access in > > favor of API-only communication). If we're planning to restrict > > what the triggerer can talk to, giving it executor ownership and broker > > dispatch now creates tech debt we'd need to undo. > > > > Option B is a better fit architecturally, but it still adds new surface > > area (XCom push from triggerer, changes to TriggerEvent, KPO behavior > > changes). For a 3.2 release where we should be focused on stability, > > introducing new triggerer capabilities increases what we need to test and > > validate. I'd rather not risk 3.2 stability for this. > > > > Option C is my preference. This is a real problem and worth solving > > properly, but 3.2 should be a stability-focused release. An AIP would > give > > us time to design something that aligns with AIP-92's direction rather > > than working against it. Target 3.3. > > > > While I understand the frustration for waiting for several months, but > any > > new thing added is a new surface to test, validate and potential for > > introducing new bugs. > > > > Thanks, > > Kaxil > > > > On Thu, 26 Mar 2026 at 03:22, Kaxil Naik <[email protected]> wrote: > > > > > Option C: 0 > > > > > > On Wed, 25 Mar 2026 at 23:04, Jens Scheffler <[email protected]> > > wrote: > > > > > >> Hi Airflow Devs, > > >> > > >> Following-up on the discussions in > > >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2F6znvd5rtqnxt5r4hys7qn64j5mflr9g1&data=05%7C02%7Cdavid.blain%40infrabel.be%7C6fa353dde3a7468e57bf08de8b3b69b1%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639101284456829672%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kXiLrT39Hi8fFKeUrQWQDgQ6mvh6EMe6UxuYXaSGbHE%3D&reserved=0<https://lists.apache.org/thread/6znvd5rtqnxt5r4hys7qn64j5mflr9g1> > > >> I'd > > >> like to cast a VOTE - even if the discussion is fresh.. but my > aim/call > > >> would be to improve the problem in the core with Airflow 3.2.0 not to > > >> have users stuck in this situation for additional multiple months (as > it > > >> would be a new feature, not a fix most probably not applicable to a > fix > > >> release) > > >> > > >> As we discussed two options we can not have a simple +/- Vote, > therefore > > >> I call for a VOTE with options, please rate from 0 to +1 for the > option > > >> and only -1 on an option to call for veto. > > >> > > >> * OPTION A) Integrate feature to directly queue tasks from Triggerer > > >> as of PR > > >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F63489&data=05%7C02%7Cdavid.blain%40infrabel.be%7C6fa353dde3a7468e57bf08de8b3b69b1%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639101284456865852%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=lGZturEQPtN9Lpn0qlRJlb4JBTwyiveydnxQ06E%2Fno8%3D&reserved=0<https://github.com/apache/airflow/pull/63489> > > >> in Airflow > > >> 3.2.0 > > >> * OPTION B) Extend Triggerer to support XCom, change KPO to end from > > >> Triggerer as of PR > > >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F64068&data=05%7C02%7Cdavid.blain%40infrabel.be%7C6fa353dde3a7468e57bf08de8b3b69b1%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639101284456889125%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ODnVOUcPxzfeLSB7o80%2B7knZCWvXduBJqZww%2FFVrfJA%3D&reserved=0<https://github.com/apache/airflow/pull/64068> > > >> (Core) with Target Airflow 3.2.0 and > > >> > > >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F64069&data=05%7C02%7Cdavid.blain%40infrabel.be%7C6fa353dde3a7468e57bf08de8b3b69b1%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639101284456916153%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=fyYrUduDAw2LYT0FM4tQugLfedLthAmGps0BD%2BDuyeg%3D&reserved=0<https://github.com/apache/airflow/pull/64069> > > >> (KPO Adjustment) > > >> * OPTION C) Do not immediately change this for Airflow 3.2.0, > further > > >> discuss alternatives, e.g. raise an AIP > > >> > > >> For details of the soluitions please check the discussion thread > (might > > >> need a bit of reading!) and the linked PRs > > >> > > >> The VOTE is open for the next 72 hours to be done before RC1, until > > >> 2026-03-29 23:59 CET - I will sum-up all vores except any potential > veto > > >> and will report the summary. > > >> > > >> My Vote is: > > >> > > >> * Option A: +0.8 > > >> * Option B: +1 > > >> * Option C: 0 > > >> > > >> Thanks, > > >> > > >> Jens > > >> > > > > > >
