That's a nice one, looking forward to it :) On Thu, Nov 21, 2024 at 12:55 PM Tzu-ping Chung <t...@astronomer.io.invalid> wrote:
> Hi all, > > I am calling a LAZY CONSENSUS to officially anoint Ruff as the tool to > implement lint rules that detect potential incompatibilities in existing > user code (including DAG files and plugins) that may break in Airflow 3. > The rationale is > > * Ruff provides the best set of features for our needs according to my > research, most importantly import resolution, and also a nice CLI “for > free”. > > * While a rule in Ruff must be implemented in Rust, which can be a hurdle > to most Airflow contributors, it does not actually require a very deep > understanding to Rust, but more toward the Python AST parser that Ruff > provides (which is very similar to Python’s built-in ast). > > * We intend to only implement lint rules, not “fixes” (which would allow > Ruff to rewrite offending files automatically. This reduces the learning > curve even further. > > * Most Airflow contributors likely don’t know Python’s ast library very > well to begin with anyway, so the learning curve would not be much flatter > even if we choose a Python-based tool. > > * We would not be the first Python library to use Ruff for this—Numpy also > uses Ruff to detect potential incompatibilities for Numpy 2.0. We are also > not the first to create Airflow rules in Ruff either—it already contains > one Airflow-specific rule. > > If we are to implement an auto fixer, it can still be rewritten in Python. > With Ruff handling most of the user interaction (including generating > proper error messages and a clear error code), the fixer can have a > relatively simplistic CLI that can be more easily implemented with just > argparse and print() (or a little drop of Rich if we feel fancy). > > I have created a POC on generating lint errors against creating an Airflow > DAG without using the 'schedule' argument correctly for Airflow 3: > https://github.com/uranusjr/ruff/pull/1 > > If no one objects, the vote will pass three days from now, at 12:00 (noon) > UTC, 24 November 2024. I intend to formally submit the rule implemented > above to Ruff, and start contributing other rules to Ruff. A list of rule > candidates can be found in Wei’s thread “[Discuss] Airflow 2 to 3 migration > rules” sent to this list earlier. > > Thanks all, > TP > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
