Hey everyone, I think we had a bit of clash in https://github.com/apache/airflow/pull/32319 where both "ideas" (serialization and common.sql) had not been sufficiently discussed/explained and I hope we can address it by adding (a bit) more "whys" to our (developer) documentation.
I think a number of our past decisions and reasoning behind them are often staying in the heads of the people who were discussing them, and even if it is captured in past discussions, PRs, it's difficult to do "archeology" on them and re-process them and understand what we wanted to achieve and why. Some of those are big enough to have impact on future PRs etc. while not big enough to get to Airflow Improvement Proposals and I think we miss a bit of persistent "decision records" for them. Two cases in question: Serialization and Common.sql API - both of which have not been understood well by people involved in one, but not the other in the past. With the "common.sql" PR (https://github.com/apache/airflow/pull/36015) - my proposal is to add it in the form of ADR ("Architecture Decision Records' - which is a very simple and lightweight way of storing the decisions we made - and evolving them. ADRs are pretty popular and adopted in mature organisations/projects and I've used them in `breeze` https://github.com/apache/airflow/tree/main/dev/breeze/doc/adr and I think they are perfect for capturing, context, decisions and putting down the consequences of some decisions. They are usually kept close to the code the decision is about, they are usually short and describe a single aspect of architectural decision, and they are aimed to be read whenever in the future, people who were not involved in those decisions can easily recover why the decisions are made and what are the consequences of it. I am not saying - of course - we should do it for all or even most changes - I am talking about decisions that have potential impact on others - in the future. I.e. when we tell (this is how our approach should look in the future for "general" behaviour. Both - serialization and common.sql are good examples of such decisions - that I believe deserve to be captured "why" we are doing them and what we wanted to achieve. WDYT? J.
