Yes, the primary target is platform-leaning data practitioners: operational checks (nulls, uniqueness, volume, freshness) at pipeline boundaries, with the built-in catalog and defaults designed around that use case. Analytics-leaning folks can still consume it through YAML rulesets and custom_sql, but domain-rich business validation and complex derivations stay out of scope — we won't build a catalog for them. That said, if someone can express those derivations purely as custom_sql rules, I wouldn't mind at all, the provider just provides the helpers, and with agents these days, folding queries that join multiple tables and carry their business logic is getting pretty easy :)
Regards, Pavan. On Fri, Jul 17, 2026 at 1:15 AM Sameer Mesiah <[email protected]> wrote: > Okay. I think you addressed points 1) and 3) very well. > > I believe for 2), it seems like we were already aligned. So this is > fundamentally a provider intended for platform-leaning data practitioners? > Is that correct? In that case, I would flip my vote to +1. > > Thanks for taking out the time to address my questions. It was interesting > hearing about your future plans for this provider (especially DataFusion). > > On Fri, 17 Jul 2026 at 01:00, Pavankumar Gopidesu <[email protected] > > > wrote: > > > Hi Sameer, > > > > Thanks for taking the time to go through the PR, glad you have questions, > > let's discuss. My responses are below, sorry for the long response. > > > > Firstly, a side note: this is not an AI response; I typed it completely > but > > used AI for rephrasing. I am not a native English speaker. :) > > > > 1) Provider philosophy — framework within Airflow vs. > > integration/abstraction > > > > I agree providers come with various levels of integration and > abstraction, > > but I wouldn't call this a framework at all. All we are doing is defining > > rules as declarative data, providing a few helper functions, and > persisting > > the results. The core execution sits entirely on the common.sql DbApiHook > > interface, so any provider that implements that interface can execute the > > checks, nothing is reimplemented inside this provider. In that sense, > this > > provider is a consumer of common.sql, exactly the way many providers > today > > already are. > > > > On why a separate provider rather than folding it into an existing common > > one, a few concrete reasons: > > > > - It will grow beyond SQL execution. The next engine I'm working on is > > DataFusion, for running checks directly against ingested data in > object > > stores (a skeleton PR is already in progress). DataFusion is an > > excellent > > fit for exactly this workload — I use it day to day, and it routinely > > returns results over millions of records in milliseconds — so quality > > checks on raw ingested data need no heavy machinery: no Spark cluster, > > no > > separate compute job, just a fast scan right after ingestion, before > the > > data ever reaches a warehouse. That engine would be an odd thing to > > graft > > onto common.sql, whose scope is the DB-API interface. > > - It composes with the AI providers. Because rules are declarative > > specs, the anthropic and common.ai providers can generate rule specs > > against a schema — an integration that belongs with the quality > domain, > > not > > inside any of the providers it draws on. > > - The boundary is the domain, not a service. Everything here is > > quality-related work — rules, results, scoring, gating — hence the > name. > > Placing it in its own package keeps that domain in one place, evolving > > on > > its own cadence, without disrupting existing common providers that > > nearly > > everything depends on. > > > > So the shape is: one small quality-domain provider that consumes existing > > abstractions (common.sql today, DataFusion next, object storage for > > results, AI providers for rule generation) rather than a framework that > > reimplements any of them. > > > > 2) Target user ? > > > > Fair point on being explicit about the audience. The provider ships with > a > > set of built-in operational checks (nulls, uniqueness, volume, min/max) > > plus custom_sql on top — and I'd note that the declarative API you're > > describing already exists: rules are data, not code. A ruleset can be a > > plain YAML file loaded at parse time, and extending checks means adding a > > custom_sql rule in your Dag repo — never modifying provider internals. > > > > On platform-leaning vs. analytics-leaning: I take the distinction, though > > in the teams I've worked with, the line keeps blurring — engineers end up > > owning more of their stack end to end. The way I'd frame the question is > > fit for purpose: does this help me do the task without investing in a > > larger job or integration? If I have simple queries and declarative > checks, > > this provider is that fit. > > > > On domain-rich validation and dbt: everyone has their own way of > > implementing business logic, and where it runs is their choice — if dbt, > > GX, or Soda is the right fit there, use them; we are not promising > > domain-rich validation and we are not competing with those tools. But > > consider the case where I'm a consumer and you're a producer who promised > > to send certain fields with certain values. To verify that contract with > a > > few simple checks, I wouldn't stand up dbt or GX. Airflow already ships > > everything needed to run those checks — what's been missing is a unified > > entry point to define them. That's what this provider builds. > > > > And once rules have a declarative definition and durable per-rule > execution > > history, something else opens up: agents are very good at generating > rules > > against a schema, and at answering questions like "how has my producer's > > feed quality looked over the last month?" — which is not straightforward > to > > answer today. With the stored execution history per asset, an agent can > > answer that and even suggest additional rules. The provider ships that > > durable view. > > > > So overall: it's about fit for purpose. We're promising simple, > declarative > > operational checks over Airflow's existing provider interfaces — execute > > rules, get a score, gate your business flow on it. And if someone finds > it > > useful for checks after their transformations too, there's no harm in > that. > > > > 3) Maintenance surface: > > > > This follows from (1): what we'd own is a small declarative layer, not a > > framework. Rules are plain models with no behavior, results go through > the > > existing object storage abstraction, scoring is simple arithmetic — no > new > > tables, no migrations, no scheduler, API server, or UI changes. And "does > > the value justify the scope" is the right test: everything here reuses > > machinery Airflow already ships, so if users genuinely benefit, this thin > > layer is a cost worth carrying. If they don't, we'll learn that quickly > and > > cheaply — the provider ships in the incubation state, and our provider > > governance model lets us suspend or remove it entirely if it doesn't > prove > > useful. > > > > Happy to discuss any of this further — and thanks again for the detailed > > review, it genuinely helps sharpen the proposal. > > > > Regards, > > Pavan > > > > On Thu, Jul 16, 2026 at 10:21 PM Sameer Mesiah <[email protected]> > > wrote: > > > > > Hi Pavan, > > > > > > First, I would like to say that this new provider definitely adds value > > and > > > I liked the brief walkthrough you did today during the call. But after > > > going through the PR (and hearing Vikram's thoughts), there are a few > > > concerns that popped up: > > > > > > 1) I'm struggling to reconcile this with the existing philosophy of > > Airflow > > > providers. Most providers integrate with an external service or expose > a > > > common abstraction over multiple services. This proposal instead > > introduces > > > a fairly comprehensive data quality framework that is implemented > > entirely > > > within Airflow. I am just wondering whether this should be a > > > standalone provider versus additional functionality that could be added > > to > > > the common provider. > > > > > > 2) I'm not yet sure who the primary target user is. If the intended > > > audience is Analytics Engineers, I'd expect the framework to expose a > > > simple declarative API for defining and extending reusable checks > without > > > requiring changes to provider internals (you can correct me here if you > > > intend on exposing a simplified API eventually). In many organisations, > > > Airflow providers are owned by platform teams and are treated as > > > third-party dependencies, so modifying provider code to introduce new > > > reusable checks is often not a practical workflow. Arguably, the case > for > > > this new framework is stronger if you are not going too deep into > > > domain-rich business rules but just want to limit it to operational > > checks > > > data platform owners might be interested in. To ground my critique > with a > > > very straightforward question: if you intend for this new provider to > > > support domain-rich data validation, what benefit would it have over > > tools > > > like DBT which are more accessible to practitioners of this specific > area > > > i.e Analytics Engineers. Vikram did share the perspective of a Data > > > Engineer who is building data pipelines but I just wanted to expand on > > that > > > a bit more by differentiating between the needs of platform-leaning and > > > analytics-leaning Data Engineers. These 2 roles are increasingly > > bifurcated > > > and I think we should really think about which group we are targeting > > here. > > > > > > 3) This proposal introduces a significant amount of new framework > surface > > > area that Airflow will own and maintain going forward. Beyond the > > operator > > > itself, we're introducing rule definitions, persistence, history, > > scoring, > > > asset integration, decorators, configuration, etc. That represents a > > > substantial long-term maintenance commitment, so I think it's worth > > > discussing whether the value provided justifies that scope and whether > > this > > > is the right architectural boundary for Airflow. > > > > > > Overall, I would say I am +0.5. It's a good proposal but it needs more > > > refinement because we do not want to introduce a provider that our > users > > > are just not going to want to use. > > > > > > On Wed, 15 Jul 2026 at 21:27, Pavankumar Gopidesu < > > [email protected] > > > > > > > wrote: > > > > > > > Hi Vikram, > > > > > > > > Thanks for the feedback. I agree that helpers would be useful for > users > > > who > > > > want to run data quality checks from existing tasks or operators, > > without > > > > always needing to use a dedicated DQ operator. > > > > > > > > I updated the PR to include > > > > > > > > - importable helpers for running and persisting checks inside > tasks > > or > > > > any existing operators. > > > > > > > > For asset integration, this PR does not require Airflow core changes. > > The > > > > provider adds two asset-oriented helpers: > > > > > > > > - asset_quality() to attach quality configuration to an asset. > This > > > lets > > > > the asset carry the ruleset that is expected to be executed for > it. > > In > > > > the > > > > current implementation this is stored as provider-owned metadata > in > > > the > > > > asset extra field. > > > > - require_quality() to let downstream tasks gate on the latest > > quality > > > > result from the triggering asset event. > > > > > > > > Below is an example view of how assets can use data quality with > > > > partition-aware producer and consumer Dags. > > > > > > > > ``` > > > > > > > > *Example ruleset:* > > > > > > > > orders_quality_rules = RuleSet( > > > > name="orders_quality", > > > > rules=[ > > > > DQRule( > > > > id="orders.amount_non_negative", > > > > name="amount_non_negative", > > > > check="min", > > > > column="amount", > > > > condition={"geq_to": 0}, > > > > ), > > > > ], > > > > ) > > > > > > > > raw_orders = asset_quality( > > > > Asset("daily_raw_orders"), > > > > ruleset=orders_quality_rules, > > > > conn_id="warehouse", > > > > table="raw_orders", > > > > ) > > > > > > > > *Producer Dag:* > > > > > > > > with DAG( > > > > dag_id="daily_raw_orders", > > > > schedule=CronPartitionTimetable("0 2 * * *", timezone="UTC"), > > > > ): > > > > @task(outlets=[raw_orders]) > > > > def load_and_check(): > > > > context = get_current_context() > > > > partition_key = context["partition_key"] > > > > > > > > load_raw_orders(partition_key) > > > > > > > > result = run_quality_checks( > > > > ruleset=orders_quality_rules, > > > > table="raw_orders", > > > > partition_clause=f"ds = '{partition_key}'", > > > > ) > > > > > > > > persist_quality_results( > > > > result, > > > > context=context, > > > > outlets=[raw_orders], > > > > ) > > > > > > > > load_and_check() > > > > > > > > > > > > *Consumer Dag:* > > > > > > > > with DAG( > > > > dag_id="curate_daily_orders", > > > > schedule=PartitionedAssetTimetable(assets=raw_orders), > > > > ): > > > > quality_gate = require_quality(raw_orders, min_score=1.0) > > > > > > > > @task > > > > def curate(): > > > > context = get_current_context() > > > > partition_key = context["partition_key"] > > > > > > > > curate_orders(partition_key) > > > > > > > > quality_gate >> curate() > > > > > > > > ``` > > > > > > > > The flow is: > > > > > > > > 1. A partitioned producer Dag runs and loads data for the current > > > > partition. > > > > 2. The producer runs run_quality_checks() with a partition-scoped > > > > filter. > > > > 3. persist_quality_results() records the result and adds a DQ > > summary > > > to > > > > the produced asset event. > > > > 4. PartitionedAssetTimetable triggers the consumer Dag for that > > > > partition. > > > > 5. require_quality() reads the DQ summary from the triggering > asset > > > > event. > > > > 6. The consumer processes the partition only if quality passes. > > > > > > > > Hope this makes sense? > > > > > > > > Regards, > > > > > > > > Pavan > > > > > > > > > > > > On Wed, Jul 15, 2026 at 6:47 PM Vikram Koka via dev < > > > > [email protected]> > > > > wrote: > > > > > > > > > Hey Pavan, > > > > > > > > > > Thanks for bringing this up and working on it. I strongly support > the > > > > > overall concept of the Data Quality provider. > > > > > > > > > > However, I have two concerns / questions: > > > > > 1. I am unsure of the Operator approach shown in the PR. > > > > > I am not certain that these SHOULD be separate Operators which > result > > > in > > > > > separate tasks. > > > > > If these are made available as Helper functions, they could be > > included > > > > in > > > > > existing Operators. > > > > > I am not saying they shouldn't be separate Tasks in Dags, but the > > > Helper > > > > > function approach would provide more optionality. > > > > > > > > > > 2. I am wondering how to integrate these with Asset Partitions > > > > > I do believe this can be integrated with the Asset Partitions > > concept, > > > > but > > > > > I am personally unclear on exactly how. > > > > > I think adding an example illustrating this would be very helpful. > > > > > > > > > > Best regards, > > > > > Vikram > > > > > > > > > > > > > > > On Mon, Jul 13, 2026 at 3:26 PM Pavankumar Gopidesu < > > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > Thanks Kaxil.. > > > > > > > > > > > > Thanks Everyone, consensus looks positive; I will merge the > > skeleton > > > PR > > > > > > first. > > > > > > > > > > > > Regards, > > > > > > Pavan > > > > > > > > > > > > On Mon, Jul 13, 2026 at 10:05 PM Kaxil Naik <[email protected] > > > > > > wrote: > > > > > > > > > > > > > +1 for the high-level need for such data quality checks. Thanks > > > > Pavan, > > > > > > will > > > > > > > review the PRs shortly. > > > > > > > > > > > > > > On Fri, 10 Jul 2026 at 01:58, Pavankumar Gopidesu < > > > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Thanks Bugra, > > > > > > > > > > > > > > > > I have updated the naming convention now. > > > > > > > > > > > > > > > > Pavan > > > > > > > > > > > > > > > > On Wed, Jul 8, 2026 at 7:07 PM Buğra Öztürk < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks Pavan for bringing this together and starting the > > > > > discussion! > > > > > > > > > Sounds good! +1 on the idea. > > > > > > > > > > > > > > > > > > Harder than solving problems. Not a strong suggestion, but > > > > > > > > > `common-dataquality` sounds more reasonable to me. It also > > adds > > > > the > > > > > > > value > > > > > > > > > of the `common` part, which provides the separation pattern > > > Jarek > > > > > > > > > mentioned. It gives a better understanding that it is a > > common > > > > > > > offering. > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > Bugra Ozturk > > > > > > > > > > > > > > > > > > On Wed, Jul 8, 2026 at 7:15 PM Pavankumar Gopidesu < > > > > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Thanks Jarek, I agree that the separate provider approach > > > > offers > > > > > > much > > > > > > > > > more > > > > > > > > > > flexibility for iterating on features and fixes. > > > > > > > > > > > > > > > > > > > > Naming is always hard :) > > > > > > > > > > > > > > > > > > > > Option 1: apache-airflow-providers-dataquality > > > > > > > > > > Option 2: apache-airflow-providers-common-dataquality > (This > > > > goes > > > > > > > inside > > > > > > > > > the > > > > > > > > > > common providers folder we already have) > > > > > > > > > > > > > > > > > > > > So, I am up for either option :) > > > > > > > > > > > > > > > > > > > > have removed first short name > > `apache-airflow-providers-dq`. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Pavan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > +1 Good design/ idea. No objections - dataquality is a > > good > > > > > name > > > > > > - > > > > > > > > but > > > > > > > > > I > > > > > > > > > > > would also consider `common-dataquality" - even if it's > > > > longer, > > > > > > it > > > > > > > > > builds > > > > > > > > > > > on the pattern we have already with common-ai. But not > a > > > > > blocker. > > > > > > > > > > > > > > > > > > > > > > I also think it's good to have it as a separate > provider, > > > > even > > > > > if > > > > > > > it > > > > > > > > > > gains > > > > > > > > > > > traction for two reasons: > > > > > > > > > > > > > > > > > > > > > > a) ability to add features or fix issues independently > > from > > > > the > > > > > > > core > > > > > > > > > > > b) an explicit "optional" feature that is easy to > promote > > > > > > > > > > > > > > > > > > > > > > I think what we saw with common is that people see > > airflow > > > > > > already > > > > > > > as > > > > > > > > > too > > > > > > > > > > > heavy - and "too many releases" sometimes, so quite > > > > > > > > > counter-intuitively - > > > > > > > > > > > by having separate providers adding features that "hook > > in" > > > > > > > existing > > > > > > > > > > > functionalities of core - we do not make airflow > > "heavier" > > > > and > > > > > we > > > > > > > do > > > > > > > > > not > > > > > > > > > > > force people to migrating to future newer versions to > use > > > new > > > > > > > > features. > > > > > > > > > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu < > > > > > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Amogh, > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the feedback. > > > > > > > > > > > > > > > > > > > > > > > > I am happy to change the provider name to > dataquality. > > > > > > > > > > > > > > > > > > > > > > > > Regarding the LLM-assisted features, the current PR > > does > > > > not > > > > > > > > include > > > > > > > > > > any > > > > > > > > > > > > implementation. It only adds the SKILLS [1 ]and the > > > > reference > > > > > > > > schema > > > > > > > > > > for > > > > > > > > > > > > the DQ Rule structure. Are you suggesting that I move > > > this > > > > > > SKILL > > > > > > > > > > > > documentation to a separate PR? > > > > > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > Pavan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai < > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Pavan, > > > > > > > > > > > > > > > > > > > > > > > > > > First of all, +1 to this. > > > > > > > > > > > > > > > > > > > > > > > > > > Now, few things: > > > > > > > > > > > > > > > > > > > > > > > > > > * On naming: dataquality over dq for me honestly. > Our > > > > > > existing > > > > > > > > > > provider > > > > > > > > > > > > > names spell things out > > > > > > > > > > > > > (common.sql, openlineage, not abbreviated forms) > and > > dq > > > > is > > > > > > > > > genuinely > > > > > > > > > > > > > ambiguous outside context. > > > > > > > > > > > > > > > > > > > > > > > > > > * On scope: I also agree with Niko that #69413 is > too > > > > large > > > > > > for > > > > > > > > one > > > > > > > > > > > pass > > > > > > > > > > > > & > > > > > > > > > > > > > I am glad to see the > > > > > > > > > > > > > backend/UI split already happening in #69575. Would > > > also > > > > > > > suggest > > > > > > > > > > > keeping > > > > > > > > > > > > > the LLM assisted rule > > > > > > > > > > > > > generation pieces (*schema-based > > > > > generate_rules_from_schema*) > > > > > > > out > > > > > > > > > of > > > > > > > > > > > the > > > > > > > > > > > > > initial provider PR entirely > > > > > > > > > > > > > cos as I see it, its a separable capability and > > > bundling > > > > it > > > > > > > will > > > > > > > > > slow > > > > > > > > > > > > > review of the core DQRule or > > > > > > > > > > > > > RuleSet or operator surface, which is the part that > > > > > actually > > > > > > > > needs > > > > > > > > > > the > > > > > > > > > > > > most > > > > > > > > > > > > > detailed review. > > > > > > > > > > > > > > > > > > > > > > > > > > In short: go for it! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks & Regards, > > > > > > > > > > > > > Amogh Desai > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu > < > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > In the meantime, the PR is ready for review. Feel > > > free > > > > to > > > > > > > > review > > > > > > > > > > and > > > > > > > > > > > > > > provide any feedback. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Pavan > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar > Gopidesu > > < > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sorry, I forgot to add: the draft PR is here > > > > > > > > > > > > > > > https://github.com/apache/airflow/pull/69413; > > it's > > > > > > still a > > > > > > > > > WIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some screenshots > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468 > > > > > > > > > > > > > :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Pavan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar > > Gopidesu > > > < > > > > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Airflow community, > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> I would like to start a discussion regarding a > > new > > > > > > > provider: > > > > > > > > > > > > > > >> apache-airflow-providers-dq. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> While Airflow already includes SQL check > > operators > > > > > that > > > > > > > many > > > > > > > > > > users > > > > > > > > > > > > > rely > > > > > > > > > > > > > > >> on for data quality, this new provider builds > on > > > > that > > > > > > > > > foundation > > > > > > > > > > > by > > > > > > > > > > > > > > >> introducing DQRule and RuleSet objects, stable > > > rule > > > > > > > > identity, > > > > > > > > > > > > > persisted > > > > > > > > > > > > > > >> history, and direct connections to Airflow > > assets. > > > > > This > > > > > > > > > approach > > > > > > > > > > > > makes > > > > > > > > > > > > > > >> quality results easier to inspect over time, > > > allows > > > > > > > > downstream > > > > > > > > > > > > > > consumers to > > > > > > > > > > > > > > >> gate tasks based on recent quality results, > and > > > > > > provides a > > > > > > > > > > unified > > > > > > > > > > > > > > schema > > > > > > > > > > > > > > >> for LLM-assisted workflows. Execution will > > > continue > > > > to > > > > > > > > utilize > > > > > > > > > > > > > existing > > > > > > > > > > > > > > >> DbApiHook connections. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> The initial version of the provider is > > > intentionally > > > > > > > > focused: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> - Declarative DQRule and RuleSet objects. > > > > > > > > > > > > > > >> - DQCheckOperator and @task.dq_check. > > > > > > > > > > > > > > >> - DbApiHook-based SQL checks, including > > built-in > > > > > > checks > > > > > > > > and > > > > > > > > > > > > > > custom_sql. > > > > > > > > > > > > > > >> - Persisted results for tasks, runs, and > > rules. > > > > > > > > > > > > > > >> - A minimal Airflow UI plugin for viewing > > > results > > > > > and > > > > > > > rule > > > > > > > > > > > > history. > > > > > > > > > > > > > > >> - Experimental asset helpers such as > > > > asset_quality() > > > > > > and > > > > > > > > > > > > > > >> require_quality(). > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Regarding scope, this first iteration uses > > object > > > > > > storage > > > > > > > > only > > > > > > > > > > to > > > > > > > > > > > > > > persist > > > > > > > > > > > > > > >> DQ results and history; checks are executed > via > > > > > database > > > > > > > > > > > > connections. > > > > > > > > > > > > > > >> Future iterations may include file or > > object-store > > > > > based > > > > > > > > > checks > > > > > > > > > > > > (e.g., > > > > > > > > > > > > > > S3, > > > > > > > > > > > > > > >> GCS) where Airflow runs quality rules against > > data > > > > > > > directly. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> This proposal does not require changes to > > Airflow > > > > > core. > > > > > > > > Asset > > > > > > > > > > > > support > > > > > > > > > > > > > is > > > > > > > > > > > > > > >> currently provider-owned metadata, with static > > > > > > > configuration > > > > > > > > > > > stored > > > > > > > > > > > > on > > > > > > > > > > > > > > the > > > > > > > > > > > > > > >> asset and runtime summaries stored on asset > > > events. > > > > If > > > > > > the > > > > > > > > > > > provider > > > > > > > > > > > > > > gains > > > > > > > > > > > > > > >> traction, we can discuss making Data Quality a > > > > > > first-class > > > > > > > > > > > component > > > > > > > > > > > > > of > > > > > > > > > > > > > > >> Airflow assets. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> This work also serves as a practical follow-up > > to > > > > the > > > > > > data > > > > > > > > > > quality > > > > > > > > > > > > > > >> direction mentioned in AIP-99. Persisted > history > > > is > > > > > > > valuable > > > > > > > > > for > > > > > > > > > > > > users > > > > > > > > > > > > > > and > > > > > > > > > > > > > > >> future LLM-assisted workflows, such as those > > from > > > > > > > Anthropic > > > > > > > > or > > > > > > > > > > > > > > common.ai, > > > > > > > > > > > > > > >> to understand rule performance and generate > > > > candidate > > > > > > > rules > > > > > > > > > > based > > > > > > > > > > > on > > > > > > > > > > > > > > schema > > > > > > > > > > > > > > >> context. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> A rough pseudo-flow is provided below: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> seed_rules = RuleSet( > > > > > > > > > > > > > > >> name="orders_quality", > > > > > > > > > > > > > > >> rules=[ > > > > > > > > > > > > > > >> DQRule(name="order_id_not_null", > > > > > > > check="null_count", > > > > > > > > > > > > > > >> column="order_id", condition={"equal_to": 0}), > > > > > > > > > > > > > > >> DQRule(name="amount_valid", > check="min", > > > > > > > > > > column="amount", > > > > > > > > > > > > > > >> condition={"geq_to": 0}), > > > > > > > > > > > > > > >> ], > > > > > > > > > > > > > > >> ) > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> orders_asset = asset_quality( > > > > > > > > > > > > > > >> Asset("orders"), > > > > > > > > > > > > > > >> conn_id="warehouse", > > > > > > > > > > > > > > >> table="orders", > > > > > > > > > > > > > > >> ruleset=seed_rules, > > > > > > > > > > > > > > >> ) > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> # Optional: common.ai / Anthropic provider > can > > > > > > generate a > > > > > > > > > > RuleSet > > > > > > > > > > > > > from > > > > > > > > > > > > > > >> schema context. > > > > > > > > > > > > > > >> generated_rules = > > generate_rules_from_schema(...) > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> @task.dq_check(asset=orders_asset) > > > > > > > > > > > > > > >> def check_orders(ruleset): > > > > > > > > > > > > > > >> return ruleset > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> checked_orders = check_orders(generated_rules) > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> with DAG("orders_consumer", > > > schedule=orders_asset): > > > > > > > > > > > > > > >> require_quality(orders_asset, > > min_score=0.95) > > > >> > > > > > > > > > > > > consume_orders() > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> The UI remains deliberately minimal for this > > > initial > > > > > > > > release, > > > > > > > > > > > > focusing > > > > > > > > > > > > > > on > > > > > > > > > > > > > > >> result and history inspection. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> You can view examples [1] of how it's > integrated > > > > with > > > > > > > > > > assets/llms. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> currently i named it providers > > > > > > > > `apache-airflow-providers-dq`. > > > > > > > > > if > > > > > > > > > > > any > > > > > > > > > > > > > > >> other preference likely with `dataquality`. > > Please > > > > let > > > > > > me > > > > > > > > know > > > > > > > > > > if > > > > > > > > > > > > you > > > > > > > > > > > > > > have > > > > > > > > > > > > > > >> a preference. naming is hard :) > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> [1]: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/ > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Thanks, > > > > > > > > > > > > > > >> Pavan > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
