Also another good case that just happened - https://github.com/apache/airflow/discussions/44211 a user added a very valid question on overloading triggerers (i.e. trigger scalability). If we are able to answer the question of the user now - this would be great (I can't).
But I think the answer won't be pretty (i.e. we can't help with that now) - so my question is how confident is that we can support such user's questions and how confident current scalability promises for example can be met ? J. On Wed, Nov 20, 2024 at 11:11 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Actually that's an extremely good point and I agree with Elad here. > > If we commit to something as first priority, we should treat issues we > have with it with priority. > There is another - very similar - issue with on_kill handling for > deferrables that I just recalled. > > The issue is here: https://github.com/apache/airflow/issues/36090 that > Matthew Wicks has been following on multiple times, and proposed a way to > approach it in general way - but the only solution we were able to come up > with was to implement workaround in a few deferrable operators. > > I am here with Elad that until we solve this and prioritise other issues > there, making deferrable a default is a bad idea. > > J. > > > > > On Wed, Nov 20, 2024 at 6:56 PM Elad Kalif <elad...@apache.org> wrote: > >> I don't think deferrable can act as a replacement at this point. >> Removing/changing defaults of poke and reschedule in favor of deferrable >> means that deferrable becomes a critical feature. This means that any bug >> related to it must be triaged and fixed ASAP. >> We are not there yet. I don't think many maintainers know this area of the >> code. >> >> In my perspective as a user I have a functionality that is working fine. I >> learned to trust it. I know how it behaves and am able to write code >> efficiently with it. The suggested replacement is a feature that has 30ish >> bug reports with little community focus. That is a lot of risk with small >> profit to gain. For me this is an indication of -1 to this proposal. >> >> For example issue https://github.com/apache/airflow/issues/36734 with >> suggested PR: https://github.com/apache/airflow/pull/33718 >> I admit I am not into the details but if we accept this proposal this bug >> becomes top priority problem and this is open since 2023. >> >> So my position is -1 for any change about poke and reschedule (even for >> changing defaults). >> >> >> >> On Thu, Nov 14, 2024 at 10:28 PM Jarek Potiuk <ja...@potiuk.com> wrote: >> >> > > I think that if we have implemented async for an operator, we should >> > remove >> > the non-async version. >> > >> > I think we all agree here. The question is only "when". I see two >> options: >> > >> > 1) When Airflow 3 is released >> > 2) When Airflow 2 stops being supported >> > >> > I'd be for 2) . >> > >> > Why? Because it has the least impact on our users while it does not make >> > our life much harder, while we might commit to clean-up eventually. >> > >> > I personally (with my ADHD personality twist) often have an urge to fix >> > things "now". It's very rewarding and has great "feedback" of feeling >> > better. But I think the next best thing is to have a clear timeline to >> > remove things like that and relentlessly follow it through. And it has >> the >> > added benefits of avoiding surprises for users who follow our decisions >> and >> > policies. >> > >> > I think the biggest problem I have with things that are deprecated are >> not >> > that they are lying around, but with the feeling that they will stay >> there >> > forever (aka - we will not clean our technical debt). I think having a >> > policy when we remove things and following that, avoids that problem (as >> > long as we do follow). >> > >> > See for example this PR for Google Provider that Max opened today: >> > https://github.com/apache/airflow/pull/43953.= ~ -8000 lines of code we >> > will not have to maintain. and whoever followed it, they had 5 months >> or so >> > to adjust as Google team set some rules for that and in most cases set >> > the deprecation date. This is cool. Most of those warnings have a >> > "planned_removal_date" set. >> > >> > And I think it also shows our maturity - that we do not "jump" on >> changing >> > something, but we plan it, we have a reason behind it, whe know why we >> are >> > doing it. what are consequences to our users, and follow through, giving >> > enough warning to the users (those who care to listen). >> > >> > And the more we do it, the more we plan and announce in advance and >> > relentlessly follow it through, the more our users will listen to those >> > things. >> > >> > J. >> > >> > >> > On Thu, Nov 14, 2024 at 4:12 PM Daniel Standish >> > <daniel.stand...@astronomer.io.invalid> wrote: >> > >> > > To me, I don't really mind reschedule mode so much. But I *really >> *don't >> > > like that we have *both* styles implemented in operators (talking >> about >> > > providers here). >> > > >> > > I think that if we have implemented async for an operator, we should >> > remove >> > > the non-async version. >> > > >> > > And we should remove the `deferrable` param for 3.0 -- this is >> something >> > > that to me makes no sense, since it implies that sensors should have >> both >> > > ways implemented. >> > > >> > > But users should still be allowed to write rescheduling sensors. >> > > >> > >> >
