My idea toward managing migration is mostly drawn from how Python (eventually) managed to pull most people over, so let me use that as a parallel to describe what I want to do.
Initially Python tried using an automated script (2to3) to magically convert code into Python 3 compatible, which people absolutely hated, so we’re not going down that same route. (I think we all agree on that.) Eventually two things happened. First, Python back-ported some Python 3 syntax back to Python 2. Specifically, some things that you can already do in Python 2 received a new syntax, so you can first rewrite those things you are already doing to use the same syntax as Python 3, and then upgrade without changing code. Examples include the b'' byte literal, the bytes() built-in, and the // operator. The community also came up with is six, basically a compatibility layer between both Python 2 and 3 so people can write code both syntactically compatible and behavioural identical to both—although sometimes not in either 2 or 3 syntax, but a new one. This is especially nice for library authors, who generally need to support both Python 2 and 3 users at the time, but with one single code base. That’s how Python went through the migration. I imagine the same for Airflow. First, we’ll support StringTemplate and FileTemplate in Airflow 2 (likely with a 2.11 minor version), so people can write code in the new way without needing to migrate (and taking the potential that things break). Once they make all the needed changes, Airflow can be upgraded to 3 without any code change from the user. Second, the compat package (either as a provider, or an entirely separate project) would be our six, containing tools libraries can use the implement cross-compatible types for users. The providers are at the same position as third-party libraries in the Python 2-3 transition—non-official Airflow packages also are, but they are out of our control—and should change the code to use the compat package where needed, so users using them on Airflow 2 do not need to change how they write DAGs, while users using them on Airflow 3 can also write new code that works as expected. With Airflow back porting the new constructs to Airflow 2, a user that wants to migrate from Airflow 3 can simply start using the back ported constructs to interact with the provider. After all interactions are rewritten to use the new constructs, an upgrade to Airflow 3 should not require more code changes. Does this make sense? I feel the process is quite well trotted at this point. There are of course details to figure out—how should the compat layer implement a cross-compatible syntax, for example—but honestly the design work seems to be already done by others, from I can tell, and we shouldn’t need to do the exploration and design again. TP > On Jul 25, 2024, at 19:15, Jarek Potiuk <ja...@potiuk.com> wrote: > > +0.5 (binding). (See https://www.apache.org/foundation/voting - for > fractional votes). > > A bit more comment here: > > See my last comment in the AIP. I am not sure if this is the right way but > it's conditional +1. I love the idea, and proposal. Mostly because it will > make DAG authoring more "modern" looking and remove a lot of confusion on > where templating can be used. > > But we need to very carefully design the migration process for our users > and make it easy and painless. While we have other breaking changes - those > are mostly "environmental/deployment related" - but this one requires 9X% > of DAGs of pretty much anyone who migrates to Airflow 3 to change their > dags - possibly before in a transitional Airflow 2.11 version to be > modified. Albeit the modifications can be automated). Initially when we > discussed Airflow 3 there were a few voices telling "if a change will > require changing DAGs in bulk - it's a non-go". This one changes the > assumption - it will basically REQUIRE to change pretty much all the dags > of Anyone who migrates to Airflow 3. > > I do not think any other change we propose has the same property so far - I > think for now this is the only one that has such a big implication and > requires a deliberate migration of all the DAGs. IF that change is > accepted, it also means that the migration process involving DAG > modifications has to be designed (because no other change requires it - > with the exception of Task Isolation but this one has much less impact IMHO > - and AIP-44 in Airflow 2 might actually help in the migration process). > > So IMHO - if we figure out exactly how incremental migrations should be > done by our users in this case, I am all in. If we leave it to the users > and do not provide suitable process/tooling/incremental approach they could > take, this one might be a huge blocker (because of the risk that the users > will have to take during migration). > > J. > > On Thu, Jul 25, 2024 at 10:00 AM Aritra Basu <aritrabasu1...@gmail.com> > wrote: > >> +1 (non-binding) >> -- >> Regards, >> Aritra Basu >> >> On Thu, 25 Jul 2024, 1:14 pm Tzu-ping Chung, <t...@astronomer.io.invalid> >> wrote: >> >>> Hi all, >>> >>> I’m calling for a vote on AIP-80: Explicit Template Fields in Operator >>> Arguments. >>> https://cwiki.apache.org/confluence/x/2grOEg >>> >>> This proposal aims to improve how Airflow defines template fields, and >>> help users avoid annoying pitfalls currently exist. >>> >>> Discussion thread: >>> https://lists.apache.org/thread/yjcgb6fhn365n3307blq4y4v50gjynsy >>> >>> Please vote accordingly: >>> >>> [ ] +1 approve >>> [ ] +0 no opinion >>> [ ] -1 disapprove with the reason >>> >>> Votes from PMC members and committers are binding, but everyone in the >>> community is also encouraged to vote. >>> >>> The vote will run for 5 days and last until 2024-07-30 8:00 UTC. >>> >>> Consider this as my vote as +1. >>> >>> TP >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
