Hi Pavan, First, I would like to say that this new provider definitely adds value and I liked the brief walkthrough you did today during the call. But after going through the PR (and hearing Vikram's thoughts), there are a few concerns that popped up:
1) I'm struggling to reconcile this with the existing philosophy of Airflow providers. Most providers integrate with an external service or expose a common abstraction over multiple services. This proposal instead introduces a fairly comprehensive data quality framework that is implemented entirely within Airflow. I am just wondering whether this should be a standalone provider versus additional functionality that could be added to the common provider. 2) I'm not yet sure who the primary target user is. If the intended audience is Analytics Engineers, I'd expect the framework to expose a simple declarative API for defining and extending reusable checks without requiring changes to provider internals (you can correct me here if you intend on exposing a simplified API eventually). In many organisations, Airflow providers are owned by platform teams and are treated as third-party dependencies, so modifying provider code to introduce new reusable checks is often not a practical workflow. Arguably, the case for this new framework is stronger if you are not going too deep into domain-rich business rules but just want to limit it to operational checks data platform owners might be interested in. To ground my critique with a very straightforward question: if you intend for this new provider to support domain-rich data validation, what benefit would it have over tools like DBT which are more accessible to practitioners of this specific area i.e Analytics Engineers. Vikram did share the perspective of a Data Engineer who is building data pipelines but I just wanted to expand on that a bit more by differentiating between the needs of platform-leaning and analytics-leaning Data Engineers. These 2 roles are increasingly bifurcated and I think we should really think about which group we are targeting here. 3) This proposal introduces a significant amount of new framework surface area that Airflow will own and maintain going forward. Beyond the operator itself, we're introducing rule definitions, persistence, history, scoring, asset integration, decorators, configuration, etc. That represents a substantial long-term maintenance commitment, so I think it's worth discussing whether the value provided justifies that scope and whether this is the right architectural boundary for Airflow. Overall, I would say I am +0.5. It's a good proposal but it needs more refinement because we do not want to introduce a provider that our users are just not going to want to use. On Wed, 15 Jul 2026 at 21:27, Pavankumar Gopidesu <[email protected]> wrote: > Hi Vikram, > > Thanks for the feedback. I agree that helpers would be useful for users who > want to run data quality checks from existing tasks or operators, without > always needing to use a dedicated DQ operator. > > I updated the PR to include > > - importable helpers for running and persisting checks inside tasks or > any existing operators. > > For asset integration, this PR does not require Airflow core changes. The > provider adds two asset-oriented helpers: > > - asset_quality() to attach quality configuration to an asset. This lets > the asset carry the ruleset that is expected to be executed for it. In > the > current implementation this is stored as provider-owned metadata in the > asset extra field. > - require_quality() to let downstream tasks gate on the latest quality > result from the triggering asset event. > > Below is an example view of how assets can use data quality with > partition-aware producer and consumer Dags. > > ``` > > *Example ruleset:* > > orders_quality_rules = RuleSet( > name="orders_quality", > rules=[ > DQRule( > id="orders.amount_non_negative", > name="amount_non_negative", > check="min", > column="amount", > condition={"geq_to": 0}, > ), > ], > ) > > raw_orders = asset_quality( > Asset("daily_raw_orders"), > ruleset=orders_quality_rules, > conn_id="warehouse", > table="raw_orders", > ) > > *Producer Dag:* > > with DAG( > dag_id="daily_raw_orders", > schedule=CronPartitionTimetable("0 2 * * *", timezone="UTC"), > ): > @task(outlets=[raw_orders]) > def load_and_check(): > context = get_current_context() > partition_key = context["partition_key"] > > load_raw_orders(partition_key) > > result = run_quality_checks( > ruleset=orders_quality_rules, > table="raw_orders", > partition_clause=f"ds = '{partition_key}'", > ) > > persist_quality_results( > result, > context=context, > outlets=[raw_orders], > ) > > load_and_check() > > > *Consumer Dag:* > > with DAG( > dag_id="curate_daily_orders", > schedule=PartitionedAssetTimetable(assets=raw_orders), > ): > quality_gate = require_quality(raw_orders, min_score=1.0) > > @task > def curate(): > context = get_current_context() > partition_key = context["partition_key"] > > curate_orders(partition_key) > > quality_gate >> curate() > > ``` > > The flow is: > > 1. A partitioned producer Dag runs and loads data for the current > partition. > 2. The producer runs run_quality_checks() with a partition-scoped > filter. > 3. persist_quality_results() records the result and adds a DQ summary to > the produced asset event. > 4. PartitionedAssetTimetable triggers the consumer Dag for that > partition. > 5. require_quality() reads the DQ summary from the triggering asset > event. > 6. The consumer processes the partition only if quality passes. > > Hope this makes sense? > > Regards, > > Pavan > > > On Wed, Jul 15, 2026 at 6:47 PM Vikram Koka via dev < > [email protected]> > wrote: > > > Hey Pavan, > > > > Thanks for bringing this up and working on it. I strongly support the > > overall concept of the Data Quality provider. > > > > However, I have two concerns / questions: > > 1. I am unsure of the Operator approach shown in the PR. > > I am not certain that these SHOULD be separate Operators which result in > > separate tasks. > > If these are made available as Helper functions, they could be included > in > > existing Operators. > > I am not saying they shouldn't be separate Tasks in Dags, but the Helper > > function approach would provide more optionality. > > > > 2. I am wondering how to integrate these with Asset Partitions > > I do believe this can be integrated with the Asset Partitions concept, > but > > I am personally unclear on exactly how. > > I think adding an example illustrating this would be very helpful. > > > > Best regards, > > Vikram > > > > > > On Mon, Jul 13, 2026 at 3:26 PM Pavankumar Gopidesu < > > [email protected]> > > wrote: > > > > > Thanks Kaxil.. > > > > > > Thanks Everyone, consensus looks positive; I will merge the skeleton PR > > > first. > > > > > > Regards, > > > Pavan > > > > > > On Mon, Jul 13, 2026 at 10:05 PM Kaxil Naik <[email protected]> > wrote: > > > > > > > +1 for the high-level need for such data quality checks. Thanks > Pavan, > > > will > > > > review the PRs shortly. > > > > > > > > On Fri, 10 Jul 2026 at 01:58, Pavankumar Gopidesu < > > > [email protected] > > > > > > > > > wrote: > > > > > > > > > Thanks Bugra, > > > > > > > > > > I have updated the naming convention now. > > > > > > > > > > Pavan > > > > > > > > > > On Wed, Jul 8, 2026 at 7:07 PM Buğra Öztürk < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > Thanks Pavan for bringing this together and starting the > > discussion! > > > > > > Sounds good! +1 on the idea. > > > > > > > > > > > > Harder than solving problems. Not a strong suggestion, but > > > > > > `common-dataquality` sounds more reasonable to me. It also adds > the > > > > value > > > > > > of the `common` part, which provides the separation pattern Jarek > > > > > > mentioned. It gives a better understanding that it is a common > > > > offering. > > > > > > > > > > > > Best regards, > > > > > > Bugra Ozturk > > > > > > > > > > > > On Wed, Jul 8, 2026 at 7:15 PM Pavankumar Gopidesu < > > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Thanks Jarek, I agree that the separate provider approach > offers > > > much > > > > > > more > > > > > > > flexibility for iterating on features and fixes. > > > > > > > > > > > > > > Naming is always hard :) > > > > > > > > > > > > > > Option 1: apache-airflow-providers-dataquality > > > > > > > Option 2: apache-airflow-providers-common-dataquality (This > goes > > > > inside > > > > > > the > > > > > > > common providers folder we already have) > > > > > > > > > > > > > > So, I am up for either option :) > > > > > > > > > > > > > > have removed first short name `apache-airflow-providers-dq`. > > > > > > > > > > > > > > Thanks, > > > > > > > Pavan > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk <[email protected] > > > > > > wrote: > > > > > > > > > > > > > > > +1 Good design/ idea. No objections - dataquality is a good > > name > > > - > > > > > but > > > > > > I > > > > > > > > would also consider `common-dataquality" - even if it's > longer, > > > it > > > > > > builds > > > > > > > > on the pattern we have already with common-ai. But not a > > blocker. > > > > > > > > > > > > > > > > I also think it's good to have it as a separate provider, > even > > if > > > > it > > > > > > > gains > > > > > > > > traction for two reasons: > > > > > > > > > > > > > > > > a) ability to add features or fix issues independently from > the > > > > core > > > > > > > > b) an explicit "optional" feature that is easy to promote > > > > > > > > > > > > > > > > I think what we saw with common is that people see airflow > > > already > > > > as > > > > > > too > > > > > > > > heavy - and "too many releases" sometimes, so quite > > > > > > counter-intuitively - > > > > > > > > by having separate providers adding features that "hook in" > > > > existing > > > > > > > > functionalities of core - we do not make airflow "heavier" > and > > we > > > > do > > > > > > not > > > > > > > > force people to migrating to future newer versions to use new > > > > > features. > > > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu < > > > > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Amogh, > > > > > > > > > > > > > > > > > > Thanks for the feedback. > > > > > > > > > > > > > > > > > > I am happy to change the provider name to dataquality. > > > > > > > > > > > > > > > > > > Regarding the LLM-assisted features, the current PR does > not > > > > > include > > > > > > > any > > > > > > > > > implementation. It only adds the SKILLS [1 ]and the > reference > > > > > schema > > > > > > > for > > > > > > > > > the DQ Rule structure. Are you suggesting that I move this > > > SKILL > > > > > > > > > documentation to a separate PR? > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Pavan > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Pavan, > > > > > > > > > > > > > > > > > > > > First of all, +1 to this. > > > > > > > > > > > > > > > > > > > > Now, few things: > > > > > > > > > > > > > > > > > > > > * On naming: dataquality over dq for me honestly. Our > > > existing > > > > > > > provider > > > > > > > > > > names spell things out > > > > > > > > > > (common.sql, openlineage, not abbreviated forms) and dq > is > > > > > > genuinely > > > > > > > > > > ambiguous outside context. > > > > > > > > > > > > > > > > > > > > * On scope: I also agree with Niko that #69413 is too > large > > > for > > > > > one > > > > > > > > pass > > > > > > > > > & > > > > > > > > > > I am glad to see the > > > > > > > > > > backend/UI split already happening in #69575. Would also > > > > suggest > > > > > > > > keeping > > > > > > > > > > the LLM assisted rule > > > > > > > > > > generation pieces (*schema-based > > generate_rules_from_schema*) > > > > out > > > > > > of > > > > > > > > the > > > > > > > > > > initial provider PR entirely > > > > > > > > > > cos as I see it, its a separable capability and bundling > it > > > > will > > > > > > slow > > > > > > > > > > review of the core DQRule or > > > > > > > > > > RuleSet or operator surface, which is the part that > > actually > > > > > needs > > > > > > > the > > > > > > > > > most > > > > > > > > > > detailed review. > > > > > > > > > > > > > > > > > > > > In short: go for it! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks & Regards, > > > > > > > > > > Amogh Desai > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu < > > > > > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > In the meantime, the PR is ready for review. Feel free > to > > > > > review > > > > > > > and > > > > > > > > > > > provide any feedback. > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > Pavan > > > > > > > > > > > > > > > > > > > > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu < > > > > > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Sorry, I forgot to add: the draft PR is here > > > > > > > > > > > > https://github.com/apache/airflow/pull/69413; it's > > > still a > > > > > > WIP. > > > > > > > > > > > > > > > > > > > > > > > > some screenshots > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468 > > > > > > > > > > :) > > > > > > > > > > > > > > > > > > > > > > > > Pavan > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu < > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > >> Hi Airflow community, > > > > > > > > > > > >> > > > > > > > > > > > >> I would like to start a discussion regarding a new > > > > provider: > > > > > > > > > > > >> apache-airflow-providers-dq. > > > > > > > > > > > >> > > > > > > > > > > > >> While Airflow already includes SQL check operators > > that > > > > many > > > > > > > users > > > > > > > > > > rely > > > > > > > > > > > >> on for data quality, this new provider builds on > that > > > > > > foundation > > > > > > > > by > > > > > > > > > > > >> introducing DQRule and RuleSet objects, stable rule > > > > > identity, > > > > > > > > > > persisted > > > > > > > > > > > >> history, and direct connections to Airflow assets. > > This > > > > > > approach > > > > > > > > > makes > > > > > > > > > > > >> quality results easier to inspect over time, allows > > > > > downstream > > > > > > > > > > > consumers to > > > > > > > > > > > >> gate tasks based on recent quality results, and > > > provides a > > > > > > > unified > > > > > > > > > > > schema > > > > > > > > > > > >> for LLM-assisted workflows. Execution will continue > to > > > > > utilize > > > > > > > > > > existing > > > > > > > > > > > >> DbApiHook connections. > > > > > > > > > > > >> > > > > > > > > > > > >> The initial version of the provider is intentionally > > > > > focused: > > > > > > > > > > > >> > > > > > > > > > > > >> - Declarative DQRule and RuleSet objects. > > > > > > > > > > > >> - DQCheckOperator and @task.dq_check. > > > > > > > > > > > >> - DbApiHook-based SQL checks, including built-in > > > checks > > > > > and > > > > > > > > > > > custom_sql. > > > > > > > > > > > >> - Persisted results for tasks, runs, and rules. > > > > > > > > > > > >> - A minimal Airflow UI plugin for viewing results > > and > > > > rule > > > > > > > > > history. > > > > > > > > > > > >> - Experimental asset helpers such as > asset_quality() > > > and > > > > > > > > > > > >> require_quality(). > > > > > > > > > > > >> > > > > > > > > > > > >> Regarding scope, this first iteration uses object > > > storage > > > > > only > > > > > > > to > > > > > > > > > > > persist > > > > > > > > > > > >> DQ results and history; checks are executed via > > database > > > > > > > > > connections. > > > > > > > > > > > >> Future iterations may include file or object-store > > based > > > > > > checks > > > > > > > > > (e.g., > > > > > > > > > > > S3, > > > > > > > > > > > >> GCS) where Airflow runs quality rules against data > > > > directly. > > > > > > > > > > > >> > > > > > > > > > > > >> This proposal does not require changes to Airflow > > core. > > > > > Asset > > > > > > > > > support > > > > > > > > > > is > > > > > > > > > > > >> currently provider-owned metadata, with static > > > > configuration > > > > > > > > stored > > > > > > > > > on > > > > > > > > > > > the > > > > > > > > > > > >> asset and runtime summaries stored on asset events. > If > > > the > > > > > > > > provider > > > > > > > > > > > gains > > > > > > > > > > > >> traction, we can discuss making Data Quality a > > > first-class > > > > > > > > component > > > > > > > > > > of > > > > > > > > > > > >> Airflow assets. > > > > > > > > > > > >> > > > > > > > > > > > >> This work also serves as a practical follow-up to > the > > > data > > > > > > > quality > > > > > > > > > > > >> direction mentioned in AIP-99. Persisted history is > > > > valuable > > > > > > for > > > > > > > > > users > > > > > > > > > > > and > > > > > > > > > > > >> future LLM-assisted workflows, such as those from > > > > Anthropic > > > > > or > > > > > > > > > > > common.ai, > > > > > > > > > > > >> to understand rule performance and generate > candidate > > > > rules > > > > > > > based > > > > > > > > on > > > > > > > > > > > schema > > > > > > > > > > > >> context. > > > > > > > > > > > >> > > > > > > > > > > > >> A rough pseudo-flow is provided below: > > > > > > > > > > > >> > > > > > > > > > > > >> seed_rules = RuleSet( > > > > > > > > > > > >> name="orders_quality", > > > > > > > > > > > >> rules=[ > > > > > > > > > > > >> DQRule(name="order_id_not_null", > > > > check="null_count", > > > > > > > > > > > >> column="order_id", condition={"equal_to": 0}), > > > > > > > > > > > >> DQRule(name="amount_valid", check="min", > > > > > > > column="amount", > > > > > > > > > > > >> condition={"geq_to": 0}), > > > > > > > > > > > >> ], > > > > > > > > > > > >> ) > > > > > > > > > > > >> > > > > > > > > > > > >> orders_asset = asset_quality( > > > > > > > > > > > >> Asset("orders"), > > > > > > > > > > > >> conn_id="warehouse", > > > > > > > > > > > >> table="orders", > > > > > > > > > > > >> ruleset=seed_rules, > > > > > > > > > > > >> ) > > > > > > > > > > > >> > > > > > > > > > > > >> # Optional: common.ai / Anthropic provider can > > > generate a > > > > > > > RuleSet > > > > > > > > > > from > > > > > > > > > > > >> schema context. > > > > > > > > > > > >> generated_rules = generate_rules_from_schema(...) > > > > > > > > > > > >> > > > > > > > > > > > >> @task.dq_check(asset=orders_asset) > > > > > > > > > > > >> def check_orders(ruleset): > > > > > > > > > > > >> return ruleset > > > > > > > > > > > >> > > > > > > > > > > > >> checked_orders = check_orders(generated_rules) > > > > > > > > > > > >> > > > > > > > > > > > >> with DAG("orders_consumer", schedule=orders_asset): > > > > > > > > > > > >> require_quality(orders_asset, min_score=0.95) >> > > > > > > > > > consume_orders() > > > > > > > > > > > >> > > > > > > > > > > > >> The UI remains deliberately minimal for this initial > > > > > release, > > > > > > > > > focusing > > > > > > > > > > > on > > > > > > > > > > > >> result and history inspection. > > > > > > > > > > > >> > > > > > > > > > > > >> You can view examples [1] of how it's integrated > with > > > > > > > assets/llms. > > > > > > > > > > > >> > > > > > > > > > > > >> currently i named it providers > > > > > `apache-airflow-providers-dq`. > > > > > > if > > > > > > > > any > > > > > > > > > > > >> other preference likely with `dataquality`. Please > let > > > me > > > > > know > > > > > > > if > > > > > > > > > you > > > > > > > > > > > have > > > > > > > > > > > >> a preference. naming is hard :) > > > > > > > > > > > >> > > > > > > > > > > > >> [1]: > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/ > > > > > > > > > > > >> > > > > > > > > > > > >> Thanks, > > > > > > > > > > > >> Pavan > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
