Hi all, I am calling a LAZY CONSENSUS to officially anoint Ruff as the tool to implement lint rules that detect potential incompatibilities in existing user code (including DAG files and plugins) that may break in Airflow 3. The rationale is
* Ruff provides the best set of features for our needs according to my research, most importantly import resolution, and also a nice CLI “for free”. * While a rule in Ruff must be implemented in Rust, which can be a hurdle to most Airflow contributors, it does not actually require a very deep understanding to Rust, but more toward the Python AST parser that Ruff provides (which is very similar to Python’s built-in ast). * We intend to only implement lint rules, not “fixes” (which would allow Ruff to rewrite offending files automatically. This reduces the learning curve even further. * Most Airflow contributors likely don’t know Python’s ast library very well to begin with anyway, so the learning curve would not be much flatter even if we choose a Python-based tool. * We would not be the first Python library to use Ruff for this—Numpy also uses Ruff to detect potential incompatibilities for Numpy 2.0. We are also not the first to create Airflow rules in Ruff either—it already contains one Airflow-specific rule. If we are to implement an auto fixer, it can still be rewritten in Python. With Ruff handling most of the user interaction (including generating proper error messages and a clear error code), the fixer can have a relatively simplistic CLI that can be more easily implemented with just argparse and print() (or a little drop of Rich if we feel fancy). I have created a POC on generating lint errors against creating an Airflow DAG without using the 'schedule' argument correctly for Airflow 3: https://github.com/uranusjr/ruff/pull/1 If no one objects, the vote will pass three days from now, at 12:00 (noon) UTC, 24 November 2024. I intend to formally submit the rule implemented above to Ruff, and start contributing other rules to Ruff. A list of rule candidates can be found in Wei’s thread “[Discuss] Airflow 2 to 3 migration rules” sent to this list earlier. Thanks all, TP --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
