potiuk commented on code in PR #27067:
URL: https://github.com/apache/airflow/pull/27067#discussion_r1016595803
##########
airflow/dag_processing/manager.py:
##########
@@ -1129,7 +1129,6 @@ def _kill_timed_out_processors(self):
)
Stats.decr('dag_processing.processes')
Stats.incr('dag_processing.processor_timeouts')
Review Comment:
Yes. Very much agree with the last proposal from @dstandish . There is no
particular need to define in which exact it will be removed. We have no
"crystal ball" to tell us the future. Whatever we write there might be a lie. I
prefer to be vague rather than potentialy lie. It might well be that at some
point in time we relax a bit our approach to backwards compatibiity and we
decide that we will allow some, rarely used features to be removed.
Context:
Backwards compatibility and deprecation removal and SemVer might seem like
0/1 game but it isn't. Not even close. Whether something is classified as
backwards incomatibility / major version bump (in terms of SemVer versioning)
or not highly depends on many factors but most of all it should be a decision
made by release manager (based on consensus of the community/PMC members). The
code does not decide for us. For example we might decide to disable some
feature (we did in the past) when there is a security vulnerabiliyt. Does it
break someone's workkflow? Probably yes. SHOULD it break someone workflow ?
Probably YES. Is it backwards compatible? Technically yes. Should we bump major
version of Airfow to 3 when we do it ? Definitely NOT.
For me backwards compatibility is more close to quantum phusics than binary
choices. It's all about probabilty that we will break majority of our user's
workflows. Every single release of ours (whether major, minor or patchlevel)
breaks someone workflow and this is absolutely inevitable.
This is very well captured in Hyrum's Law: https://www.hyrumslaw.com/ and in
the obligatory XKCD:
> With a sufficient number of users of an API, it does not matter what you
promise in the contract: all observable behaviors of your system will be
depended on by somebody.
IMHO - what we treat as "backwards incompatible" change should be solely
based on our assesment on how big impact it has on our user's migration and
whether it is easy to recover for them. It should not be solely based on the
actual 'removal" of some feature or changing the defaults. I think this is a
very narrow-scoped approach that is not even correct assesment in the first
place. You WILL break other's workflows with every release. It's inevitable.
Pretending that it's different is just not coping well with reality (which is
very far from 0/1 binary choices).
After quite some time of thinking on how we can approach deprecations, I
start to think more and more that we should be slightly more liberal when it
comes to some deprecations and remove them earlier. The goal of adopting SemVer
was not the "versioning" strictness. The main reason we adopted it was because
we wanted to make our users comfortable in knowing that their migration will be
- very likely - a painless or very easy to handle - experience. We cannot
guarantee painless (because of Hyrum's Law). But we can make it super-easy to
handle. Goal achieved.
We just need to understand the consequences and deliberately decide, rather
than blindly rely on "removed/changed" feature. Blind 0/1 decisions based on
code added/removed is goting an easy (and wrong IMHO) path. Nothing frees us
from thinking and making decisions. Community over the code is the ASF motto.
If we put everything as fully automated rule, we do not need PMC and community
any more. Those are guidelines for us, but we should have final say as humans
how we interpret those.
INHO - if we do such removal infrequently and we have an easy (possibly even
automated or semi-automated or even super-easy to follow by our users) ways how
to deal with those, I think we should be able to remove some of the deprecated
code without actually breaking our SamVer promises. Something that we will
likely be discussing in the future. And I will advocate for that in the future
rather than going to Airflow 3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
