Agree. Very good direction! On Thu, Nov 21, 2024 at 5:00 PM Julian LaNeve <jul...@astronomer.io.invalid> wrote:
> This is awesome to see - I wrote the first rule in Ruff for Airflow, and > it was super easy to do despite not having written any Rust prior. > > The Ruff maintainers are also incredibly responsive and helpful. I think > Airflow’s one of the largest projects they work with, and they have us > listed on the Ruff README. > > > On Nov 21, 2024, at 10:44 AM, Pierre Jeambrun <pierrejb...@gmail.com> > wrote: > > > > That's a nice one, looking forward to it :) > > > > On Thu, Nov 21, 2024 at 12:55 PM Tzu-ping Chung <t...@astronomer.io.invalid > > > > wrote: > > > >> Hi all, > >> > >> I am calling a LAZY CONSENSUS to officially anoint Ruff as the tool to > >> implement lint rules that detect potential incompatibilities in existing > >> user code (including DAG files and plugins) that may break in Airflow 3. > >> The rationale is > >> > >> * Ruff provides the best set of features for our needs according to my > >> research, most importantly import resolution, and also a nice CLI “for > >> free”. > >> > >> * While a rule in Ruff must be implemented in Rust, which can be a > hurdle > >> to most Airflow contributors, it does not actually require a very deep > >> understanding to Rust, but more toward the Python AST parser that Ruff > >> provides (which is very similar to Python’s built-in ast). > >> > >> * We intend to only implement lint rules, not “fixes” (which would allow > >> Ruff to rewrite offending files automatically. This reduces the learning > >> curve even further. > >> > >> * Most Airflow contributors likely don’t know Python’s ast library very > >> well to begin with anyway, so the learning curve would not be much > flatter > >> even if we choose a Python-based tool. > >> > >> * We would not be the first Python library to use Ruff for this—Numpy > also > >> uses Ruff to detect potential incompatibilities for Numpy 2.0. We are > also > >> not the first to create Airflow rules in Ruff either—it already contains > >> one Airflow-specific rule. > >> > >> If we are to implement an auto fixer, it can still be rewritten in > Python. > >> With Ruff handling most of the user interaction (including generating > >> proper error messages and a clear error code), the fixer can have a > >> relatively simplistic CLI that can be more easily implemented with just > >> argparse and print() (or a little drop of Rich if we feel fancy). > >> > >> I have created a POC on generating lint errors against creating an > Airflow > >> DAG without using the 'schedule' argument correctly for Airflow 3: > >> https://github.com/uranusjr/ruff/pull/1 > >> > >> If no one objects, the vote will pass three days from now, at 12:00 > (noon) > >> UTC, 24 November 2024. I intend to formally submit the rule implemented > >> above to Ruff, and start contributing other rules to Ruff. A list of > rule > >> candidates can be found in Wei’s thread “[Discuss] Airflow 2 to 3 > migration > >> rules” sent to this list earlier. > >> > >> Thanks all, > >> TP > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> For additional commands, e-mail: dev-h...@airflow.apache.org > >> > >> > >
