I am all +1 on this one. This thing gave me headaches when working on AIP-44 
and I could not understand the difference between the private "_try_number" and 
the public "try_number". Thanks for simplifying it!

This is obviously assuming it does not break anything I am not aware of :)

On 2024/05/02 19:37:32 Daniel Standish wrote:
> TLDR
> * changing handling of try_number in
> https://github.com/apache/airflow/pull/39336
> * no more private attr
> * no more getter that changes value based on state of task
> * no more decrementing
> * try number now only handled by scheduler
> * hope that sounds good to all of you
> 
> For more detail read on...
> 
> In https://github.com/apache/airflow/pull/39336 I am doing some work to
> resolve some longstanding pain and frustration caused by try_number.
> 
> The way we handle try_number has for quite some time been messy and
> problematic.
> 
> For example, if you access `ti.try_number` and then change the state to or
> from RUNNING, you will get a different value if you access it again!
> 
> And the responsibility for managing this number has been distributed
> throughout the codebase.  For example the task itself always increments
> when it starts running.  But then if it defers or reschedules itself, it
> decrements it back down so that when it runs again and naively increments,
> then it will be right again.
> 
> Recently more issues have become visible as I have worked with AIP-44
> because for example pydantic does not like private attrs and it's just
> awkward to know *what value to use* when serializing it when the TI will
> give you a different answer depending on the state of the task!
> 
> And there's yet another edge case being solved in this community PR
> <https://github.com/apache/airflow/pull/38984#issuecomment-2090944403>.
>  And then when we start looking at try history and AIP-64, it also forces a
> look at this.
> 
> So it all sounds bad and indeed it is bad but I think I have a solution.
> 
> What I do is, just have the scheduler increment try_number at the moment
> when it schedules the task.  It alone will have the responsibility for
> incrementing try_number.  And no where will it ever be decremented.  It
> will not be incremented when resuming after deferral or reschedule.  And
> that's about all there is to it.
> 
> I've tested it out and it works.  But I'm working through many test
> failures that need to be resolved (there's lots of asserts re try_number).
> 
> One small thing I just want to point out is that if a user were previously
> to be doing `task.run()` sort of manually without the task having been
> scheduled by the scheduler, well now their try_number won't be
> automatically incremented.  Same if they just do `airflow tasks run` --
> because now the responsibility is going to be solely with the scheduler.
> But airflow was never designed to assume that tasks will be run without
> having been scheduled, so I do not think that counts as a breaking change.
> So I don't think that's a blocker for this.
> 
> Thanks for the consideration.  Let me know if you have any concerns.
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Reply via email to