In the meantime, the PR is ready for review. Feel free to review and provide any feedback.
Regards, Pavan On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <[email protected]> wrote: > Sorry, I forgot to add: the draft PR is here > https://github.com/apache/airflow/pull/69413; it's still a WIP. > > some screenshots > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468 :) > > Pavan > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu < > [email protected]> wrote: > >> Hi Airflow community, >> >> I would like to start a discussion regarding a new provider: >> apache-airflow-providers-dq. >> >> While Airflow already includes SQL check operators that many users rely >> on for data quality, this new provider builds on that foundation by >> introducing DQRule and RuleSet objects, stable rule identity, persisted >> history, and direct connections to Airflow assets. This approach makes >> quality results easier to inspect over time, allows downstream consumers to >> gate tasks based on recent quality results, and provides a unified schema >> for LLM-assisted workflows. Execution will continue to utilize existing >> DbApiHook connections. >> >> The initial version of the provider is intentionally focused: >> >> - Declarative DQRule and RuleSet objects. >> - DQCheckOperator and @task.dq_check. >> - DbApiHook-based SQL checks, including built-in checks and custom_sql. >> - Persisted results for tasks, runs, and rules. >> - A minimal Airflow UI plugin for viewing results and rule history. >> - Experimental asset helpers such as asset_quality() and >> require_quality(). >> >> Regarding scope, this first iteration uses object storage only to persist >> DQ results and history; checks are executed via database connections. >> Future iterations may include file or object-store based checks (e.g., S3, >> GCS) where Airflow runs quality rules against data directly. >> >> This proposal does not require changes to Airflow core. Asset support is >> currently provider-owned metadata, with static configuration stored on the >> asset and runtime summaries stored on asset events. If the provider gains >> traction, we can discuss making Data Quality a first-class component of >> Airflow assets. >> >> This work also serves as a practical follow-up to the data quality >> direction mentioned in AIP-99. Persisted history is valuable for users and >> future LLM-assisted workflows, such as those from Anthropic or common.ai, >> to understand rule performance and generate candidate rules based on schema >> context. >> >> A rough pseudo-flow is provided below: >> >> seed_rules = RuleSet( >> name="orders_quality", >> rules=[ >> DQRule(name="order_id_not_null", check="null_count", >> column="order_id", condition={"equal_to": 0}), >> DQRule(name="amount_valid", check="min", column="amount", >> condition={"geq_to": 0}), >> ], >> ) >> >> orders_asset = asset_quality( >> Asset("orders"), >> conn_id="warehouse", >> table="orders", >> ruleset=seed_rules, >> ) >> >> # Optional: common.ai / Anthropic provider can generate a RuleSet from >> schema context. >> generated_rules = generate_rules_from_schema(...) >> >> @task.dq_check(asset=orders_asset) >> def check_orders(ruleset): >> return ruleset >> >> checked_orders = check_orders(generated_rules) >> >> with DAG("orders_consumer", schedule=orders_asset): >> require_quality(orders_asset, min_score=0.95) >> consume_orders() >> >> The UI remains deliberately minimal for this initial release, focusing on >> result and history inspection. >> >> You can view examples [1] of how it's integrated with assets/llms. >> >> currently i named it providers `apache-airflow-providers-dq`. if any >> other preference likely with `dataquality`. Please let me know if you have >> a preference. naming is hard :) >> >> [1]: >> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/ >> >> Thanks, >> Pavan >> >>
