Thanks for starting this discussion Jarek! I think it's very important for us to get on the same page as a community about this.
I'd love to go with a more flexible/common sense approach for considering breaking changes, and in a perfect world I think this would be best. However, I also think it will be very hard to document a policy that is clear enough to make the decision making for each case straightforward (we're also missing data to make those customer impact assessments too). Suggestion 1 from Bolke is actually quite an appealing approach to me. The advantage I love is that it allows us to remain quite strict on SemVer which is a huge benefit. Since being very strict means that the decision-making is easy and keeps the overhead low (gets you out of the "discussions, discussions, discussions" loop). Since, again, it will be harder to otherwise document a common-sense based policy that covers all the edge cases clearly enough (like ones which have already been brought up in this thread). Cheers, Niko ________________________________ From: Bolke de Bruin <bdbr...@gmail.com> Sent: Tuesday, November 22, 2022 5:51:32 AM To: dev@airflow.apache.org Subject: RE: [EXTERNAL][DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. I beg to differ too and I do think that Alexander is right in what he wants to accomplish. Large installations do want to do rolling upgrades and not bring a cluster down for an upgrade. It should be possible to run Airflow Core 2.4 with for example 2.3 workers. It should however not happen through reliance on the DB schema but through a (non-public) API specification. This requires significant architectural improvements. It means decoupling workers from the DB, but also Tasks. It requires backoff periods and strict versioning on the API. It would really improve security (availability, integrity). It is partially addressed in one of the AIPs but imho not sufficiently. Now Airflow is just a monolith packaged in PODs or microservices. Something for the future (hopefully :-) ). B. On 22 November 2022 at 12:40:07, Abhishek Bhakat (abhishek.bha...@astronomer.io.invalid<mailto:abhishek.bha...@astronomer.io.invalid>) wrote: Hi, I Beg to differ with Alexander and agree with Jarek. There are multiple ways to deploy Airflow. Mostly commonly used is docker images, in that case using one image for all components is standard practice. If using native pip installations, airflow components are launched by a single pip module. So, to have different versions of components (as you mentioned) is adding extra work just to keep them out of sync. A basic common sense would be not to take extra steps to self sabotage. Thanks, Abhishek On 22-Nov-2022 at 4:35:09 PM, Alexander Shorin <kxe...@gmail.com<mailto:kxe...@gmail.com>> wrote: On Tue, Nov 22, 2022 at 1:37 PM Jarek Potiuk <ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote: BTW. "Workers from 2.2" used with "Airflow 2.4" is not even a thing. This is something that you should never, ever, try to do. This is even more common sense, and there are of course limits of what you can describe in the docs (whatever you come up with, someone might have a super crazy idea that you have not thought about and - for example - run Airflow 1.10 worker With Airflow 2 (why not? We have not written it should not happen). At scale, you cannot upgrade all the versions and keep them in sync all the time. For minor versions compatibility is expected. Obviously, it doesn't for major one. It is common sense and practice in the real world, sorry. -- ,,,^..^,,,
