Yes. I quite agree. We discussed it that "dependencies" are not "bound" by semver. And I think technically we **could** make `dask` non-preinstalled without breaking Semver. Similarly we could - and maybe even will (after AIP-56) have FAB as an optional provider.
For me this is really the way how we could generally handle separating out things and moving them "away" from Airflow's core without actually having to bump Airflow to 3 (which I think would be a major marketing issue if we start announcing "breaking" Airflow version. We already have some precedent for that - I moved out Hive Macros that were in "airflow" but now they are part of the Hive provider (and we have PEP-562 dynamic attribute access with deprecation warnings in the core - which is super small and 0-maintenance). So we have already done that. And we could apply it to other things. For example if we would like to remove SubDag support from core without bumping Airflow to 3, we could do it. We just would need to find an easy way to extract it and make an API that will allow an external provider to add a "subdag" feature and move it out of the core. Not sure if it is feasible because SubDag is quite "deeply" integrated in Airflow, but I think we could work it out and it could be a great way to keep airflow core "lean" and move faster in general. I guess the question is not "can we do it" but more "should we do it" :D J. On Thu, Jul 20, 2023 at 9:15 PM Daniel Standish <daniel.stand...@astronomer.io.invalid> wrote: > Just on the question of semver, I am not convinced that semver prohibits > this. > > As a user, your concerns and expectations regarding semver are about > essentially how the code works, e.g. are you going to have to refactor all > of your 500 dags. In other words the API.. But to me this is a lower > level thing that doesn't seem should strictly be forbidden. It's like... > when we change the dependencies in airflow, inevitably that will break > someone's cluster if they blindly deploy to prod. But this is not a > prudent thing to do. So I think it's not an unreasonable expectation that a > user verify that their cluster will spin up. It's something you can verify > without running any dags at all (i.e. if you set dask executor and you > start airflow, i think airflow will not start). And then users can rely on > semver about changing behavior, which is not something you necessarily can > tell without running dags. >
