potiuk commented on code in PR #27067:
URL: https://github.com/apache/airflow/pull/27067#discussion_r1020614511
##########
airflow/dag_processing/manager.py:
##########
@@ -1129,7 +1129,6 @@ def _kill_timed_out_processors(self):
)
Stats.decr('dag_processing.processes')
Stats.incr('dag_processing.processor_timeouts')
Review Comment:
I simply try to think and analyse and adapt my understanding of the world. I
used to think "backwards compatibillity is easy" and SemVer provides answers.
But this was "past". Now we are in the "current".
I've changed my mind seeing some real-world effect and Hyrum's Law is simply
reflecting my current understanding. Maybe that's why @o-nikolas some past
discussions and push-back of mine migh have mislead you. The push back was
based on my (then) strict interpretation of what "backwards incompatibility"
is. But since then - things happened.
That includes a user who complained tha their deployment started to throw
OOM. Since they had not changed anything between the upgrades, the user
questioned backwards compatibility of Airlfow. True story.
For me that was a bit of revelation and true embodiment of Hyrum's Law and
the XKCD comic I quoted above. And while I am pretty much still in the campl of
using SemVer as communication tool, I am also more on adding more freedom in
interpreting of what "backwards compatibility" is.
This has been mostly summarized in my "lazy consensus" post from today about
providers: https://lists.apache.org/thread/nvzdgshwm7s0c2sr4sbokp8kbfjnc039.
Not later than 6 months ago when we released 2.2+ providers I was an advocate
of "breaking change" and "bump major versions" of all providers. Now - with 6
more months of experience, listening to users and looking at other projects
(and thinking hard about it) I changed my opinion: the changes in
min-airflow-version are unlikely to break someone's workflow, so let's not
treat them as breaking. Minor version upgrade is enough.
The most important learning is that there is no "objective" criterium for
breaking/non-breaking. Something that is not breaking for me, might be breaking
for others. And there is very litttle we can do about it. Realisation of that
is pretty mind-boggling and ot made me change my mind when it comes to
"breaking" interpretation.
This issue here is siimilar:
* Is SemVer still valid for us ? I think yes.
* But - can we remove this metrics without bumping major version? I think
yes.
While those two seem contradicting at first, they are not (in my head).
While I think breaking changes should bump major versions, such removal is
unlikely to break someone workflow, so it is not breaking.
Bottomline:
Yes I think we should remove this metrics now. That would be my vote now.
Proposal:
Now, how we can do some actionable steps from that statement ?
I think there is a way. We could raise a thread on devlist to see if that
interpretation of "breaking change" is something that the community is ok with.
I think we've never agreed on "what breaking change is". And while it seems
obvious - it is not. What's breaking for me might not be necessarily breaking
for you.
We could ask on the devlist whether the community is ok with "relaxed"
interpretation of "breaking change" - not every removal is breaking, not every
parameter change is breaking. Even if technically there are some workflows that
will be broken. If we assess that a change is not likely to break many user's
workflow - we shoudl be able to treat removals like that as non-breaking.
While this seems like a nuance and unnecessary discussion, I think it is an
important decision if we can make it. This will mean that some of the old stuff
that we are pretty sure will not wreak havoc with our customers, can be removed
without going to Airlfow 3 - while still keeping SemVer promises (interpreted
SemVer, yes, but still showing that we promise "easy migration" for all Airflow
2 versions). Eventually it might mean that we will never have Airflow 3 (which
I think is not going to happen anyway). And that we will gradually get rid of
the deprecations in Airflow 2 rather than cumulating them.
Another option - if we will not be able to get to consensus - is to get rid
of the strict SemVer compliance and propose our own non-fully-semver compatible
versioning with some more relaxed meaning of "breaking".
I am willing to start such a thread in the next few days. I see in a way
what happens with the 2.3+ providers as a test for it. This is an
"interpretation" of the SemVer for providers - very similar to the above line
of thoughts. If people will be ok with the "non-breaking" 2.3+ provider change,
it's a good sign that we can likely apply more relaxed approach for Airflow
core as well.
@o-nikolas, @dstandish - is that clearer now where I am coming from ? Is
that something that we can agree on as a direction :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
