This is an automated email from the ASF dual-hosted git repository.
shahar1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new f01e432bedb Databricks: add Delta table version producer for
externally written tables (#76156)
f01e432bedb is described below
commit f01e432bedb43f04556824a5893d6aff38231ea2
Author: Yossi Eliaz <[email protected]>
AuthorDate: Wed Nov 18 21:32:30 2026 +0200
Databricks: add Delta table version producer for externally written tables
(#76156)
* Add DatabricksDeltaTableVersionProducerOperator for externally written
Delta tables
Publish an asset event for each Delta table version committed outside of
Airflow. The latest event the producer published is its checkpoint, so the
event and the checkpoint advance in one metadata-database write when the
task
succeeds. Delivery is at least once and each event carries a dedup_id.
* Fix mypy errors in Delta table producer tests
* Bound the Delta producer checkpoint lookup and require the table asset in
outlets
Airflow 3.2 and 3.3 can order and limit inlet events but cannot filter them
by extra, so the checkpoint lookup read every event of the asset on each
run.
It now reads the newest events 100 at a time and stops at the first event
with the producer's checkpoint key. The extra filter is still used where the
accessor supports it.
Explicit outlets that leave out unity_table.to_asset() never record a
checkpoint, so every run would publish a new enrollment event. That is now a
ValueError. A single inlet or outlet is accepted as well as a list.
Signed-off-by: zozo123 <[email protected]>
* Document the checkpoint lookup per Airflow version and use the compat DAG
import
Signed-off-by: zozo123 <[email protected]>
* Fix Delta table producer tests on Airflow 3.0 and 3.1
Asset is not a usable plain dict key on those versions (unhashable Asset
on 3.0; hashing fails on mutable extra on 3.1). Key the test event
accessors by URI the same way runtime accessors resolve assets.
Co-authored-by: Yossi Eliaz <[email protected]>
* Report Delta producer operation from the newest data commit
When maintenance commits follow a data write, consumers reading the event
operation should see the write rather than OPTIMIZE or VACUUM, while still
using the head version for VERSION AS OF.
Co-authored-by: Yossi Eliaz <[email protected]>
* Drop databricks_ prefix from new Delta table producer modules
Newer provider modules omit the provider name from the filename; keep that
convention for the files this PR adds.
Signed-off-by: Yossi Eliaz <[email protected]>
Co-authored-by: Yossi Eliaz <[email protected]>
* Address review feedback on Delta table producer tests
Tighten the maintenance-window operation assertion so newest-first
scanning is required, harden the hook mock with autospec, and warn when
asset event paging cannot advance past a shared timestamp.
Signed-off-by: Yossi Eliaz <[email protected]>
Co-authored-by: Yossi Eliaz <[email protected]>
* Keep Delta producer checkpoint key on the table asset only
Additional outlets still need version and change metadata for consumers,
but the
checkpoint marker must stay on the observed table asset so other events do
not
look like producer checkpoints.
Signed-off-by: Yossi Eliaz <[email protected]>
Co-authored-by: Yossi Eliaz <[email protected]>
* Collect Delta producer tests on Airflow 2.11
Import AssetEvent from the tests_common compat shim so pytest can
collect test_delta_table.py against Airflow 2.11.1.
Signed-off-by: Yossi Eliaz <[email protected]>
---------
Signed-off-by: zozo123 <[email protected]>
Signed-off-by: Yossi Eliaz <[email protected]>
Co-authored-by: Yossi Eliaz <[email protected]>
---
.../docs/operators/delta_table_producer.rst | 138 +++++
providers/databricks/provider.yaml | 2 +
.../src/airflow/providers/databricks/exceptions.py | 4 +
.../providers/databricks/get_provider_info.py | 2 +
.../providers/databricks/operators/delta_table.py | 452 +++++++++++++++
.../example_databricks_delta_producer.py | 81 +++
.../unit/databricks/operators/test_delta_table.py | 635 +++++++++++++++++++++
7 files changed, 1314 insertions(+)
diff --git a/providers/databricks/docs/operators/delta_table_producer.rst
b/providers/databricks/docs/operators/delta_table_producer.rst
new file mode 100644
index 00000000000..8da36533cc8
--- /dev/null
+++ b/providers/databricks/docs/operators/delta_table_producer.rst
@@ -0,0 +1,138 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/operator:DatabricksDeltaTableVersionProducerOperator:
+
+
+DatabricksDeltaTableVersionProducerOperator
+===========================================
+
+Use
:class:`~airflow.providers.databricks.operators.delta_table.DatabricksDeltaTableVersionProducerOperator`
+when a Unity Catalog Delta table is written by jobs or systems outside
Airflow, and Dags should run
+when it changes. Put the operator in a Dag that runs on a schedule. Each run
reads the newest
+Delta version and publishes an asset event for ``unity_table.to_asset()`` when
that version is newer
+than the last one it published. Otherwise the task is skipped and no event is
published.
+
+Downstream Dags schedule on the same asset that
+:class:`~airflow.providers.databricks.operators.databricks_sql.DatabricksCopyIntoAssetOperator`
uses
+for the table, so there is a single ``databricks://`` URI per table.
+
+To wait inside an already scheduled Dag run instead, use a sensor. This
operator is for producing
+events continuously.
+
+Example
+-------
+
+.. exampleinclude::
/../../databricks/tests/system/databricks/example_databricks_delta_producer.py
+ :language: python
+ :start-after: [START
howto_operator_databricks_delta_table_version_producer]
+ :end-before: [END howto_operator_databricks_delta_table_version_producer]
+
+What an event means
+-------------------
+
+An event means that the producer observed a newer committed Delta version. Its
``extra`` contains:
+
+* ``version`` and ``observed_version``: the newest Delta version at the time
of the check.
+* ``previous_version``: the version of the previous event, or
``baseline_version``. ``None`` for enrollment.
+* ``table_identity``: the ``host``, ``catalog``, ``schema`` and ``table`` of
the table.
+* ``operation``: the newest non-ignored Delta history operation in
+ ``(previous_version, version]``. When the producer publishes without a full
check of that
+ window (incomplete history or more than ``max_history_rows`` commits), this
can be an ignored
+ operation such as ``OPTIMIZE``.
+* ``timestamp``: the Delta history timestamp of ``version``.
+* ``change``: ``enrollment``, ``new_version`` or ``recreated``.
+* ``delta_table_id``: the Delta table id from ``DESCRIBE DETAIL``.
+* ``dedup_id``: the asset URI, table id and version. Use it to drop duplicate
events.
+* ``databricks_delta_checkpoint_key``: the ``checkpoint_key`` of the producer.
Present only on
+ the table asset's event, not on other outlets.
+
+Commits are coalesced. If the table moved from version 5 to version 9 between
two runs, the producer
+publishes one event for version 9. It does not publish events for versions 6,
7 and 8.
+
+An event for version N does not make downstream queries read version N. The
table can change again
+before the consumer runs. Consumers that need a reproducible read query
``VERSION AS OF`` the
+``version`` in the event, as the example does. Other tasks can publish events
for the same asset,
+for example ``COPY INTO`` with outlets, and those events may not have a
``version``. Consumers
+that read ``version`` check ``change`` or ``databricks_delta_checkpoint_key``
first.
+
+Checkpoints
+-----------
+
+The producer does not store state of its own. Its checkpoint is the newest
event it published for the
+asset with the same ``checkpoint_key``. The operator adds the asset to its
``inlets`` to read these
+events. Events from other tasks for the same asset, such as ``COPY INTO`` or
sensor outlet events, do
+not change the checkpoint.
+
+``checkpoint_key`` defaults to ``"<dag_id>.<task_id>"``. Renaming the Dag or
task, or changing the
+key, starts again with enrollment.
+
+How the checkpoint is found depends on the Airflow version. Versions that can
filter asset events by
+``extra`` ask the API server for the newest event with the producer's
``checkpoint_key``. Airflow 3.2
+and 3.3 read the asset's events newest first, 100 per request, and stop at the
first match. Airflow 3.0
+and 3.1 cannot order or limit asset events, so each run reads all events on
the asset.
+
+On the first run without a checkpoint, the producer publishes an
``enrollment`` event for the current
+version, so that later runs have a checkpoint. To avoid triggering consumers
on enrollment, pass
+``baseline_version``. It is used as the checkpoint until the first event is
published.
+
+Delivery guarantees
+-------------------
+
+Airflow records the outlet event when the task succeeds, in the same database
transaction that marks
+the task successful. Publishing the event and advancing the checkpoint are
therefore one write:
+
+* If the task fails or the worker dies before the task succeeds, no event is
published and the
+ checkpoint does not move. The next run or retry publishes the newest version
at that time.
+* After the task succeeds, the event and the checkpoint exist together.
+
+Delivery is at least once, not exactly once. The same version can be published
twice when two runs of
+the producer read the checkpoint at the same time, or when a successful run is
cleared and runs again.
+The operator sets ``max_active_tis_per_dag=1`` by default, and the producer
Dag should set
+``max_active_runs=1``. Consumers that must not process a version twice
deduplicate on ``dedup_id``.
+
+Maintenance commits
+-------------------
+
+``OPTIMIZE``, ``VACUUM START`` and ``VACUUM END`` commits do not change the
data, so they do not
+produce events by default. Pass ``ignored_operations`` to change the list, or
an empty list to publish
+for every commit.
+
+If the newest commit is an ignored operation, the producer reads the history
back to the checkpoint
+to check that every commit after it is ignored. When a data commit is followed
by ignored
+operations, the event still carries the head ``version`` (for ``VERSION AS
OF``), while
+``operation`` reports the newest non-ignored commit in that window. It
publishes when it cannot
+check the window fully, because the history was removed by Delta log retention
or because more
+than ``max_history_rows`` commits were made since the checkpoint. The task is
not skipped when it
+is not sure that the data did not change.
+
+Recreated and renamed tables
+----------------------------
+
+The checkpoint stores the Delta table id. If the table is dropped and created
again, or another table
+is renamed to the observed name, the id changes and the version can go back.
With
+``on_recreation="publish"`` (the default), the producer publishes a
``recreated`` event and uses the new
+table from then on. With ``on_recreation="fail"``, the task fails and the
checkpoint is kept.
+
+If the table is missing, cannot be accessed, or is not a Delta table, the task
fails and publishes
+nothing. Task retries apply as usual.
+
+Cost
+----
+
+Each run starts or uses a SQL warehouse to run ``DESCRIBE DETAIL`` and
``DESCRIBE HISTORY``. Choose the
+schedule of the producer Dag with this cost in mind.
diff --git a/providers/databricks/provider.yaml
b/providers/databricks/provider.yaml
index 894c986dd59..93260bb5d38 100644
--- a/providers/databricks/provider.yaml
+++ b/providers/databricks/provider.yaml
@@ -134,6 +134,7 @@ integrations:
- /docs/apache-airflow-providers-databricks/operators/sql_statements.rst
- /docs/apache-airflow-providers-databricks/operators/warehouse.rst
- /docs/apache-airflow-providers-databricks/operators/copy_into.rst
+ -
/docs/apache-airflow-providers-databricks/operators/delta_table_producer.rst
tags: [service]
- integration-name: Databricks Repos
external-doc-url: https://docs.databricks.com/repos/index.html
@@ -159,6 +160,7 @@ operators:
- integration-name: Databricks SQL
python-modules:
- airflow.providers.databricks.operators.databricks_sql
+ - airflow.providers.databricks.operators.delta_table
- airflow.providers.databricks.operators.warehouse
- integration-name: Databricks Repos
python-modules:
diff --git
a/providers/databricks/src/airflow/providers/databricks/exceptions.py
b/providers/databricks/src/airflow/providers/databricks/exceptions.py
index 88e44011a0b..1a63de52605 100644
--- a/providers/databricks/src/airflow/providers/databricks/exceptions.py
+++ b/providers/databricks/src/airflow/providers/databricks/exceptions.py
@@ -59,6 +59,10 @@ class DatabricksApiError(AirflowException):
self.error_code = error_code
+class DatabricksDeltaTableError(AirflowException):
+ """Raised when a Delta table cannot be read or its version cannot be
determined."""
+
+
class DatabricksUnityMCPError(DatabricksApiError):
"""Raised when a call to a Unity Gateway MCP Service fails."""
diff --git
a/providers/databricks/src/airflow/providers/databricks/get_provider_info.py
b/providers/databricks/src/airflow/providers/databricks/get_provider_info.py
index 76af2eebfc1..ce56c2cdcfb 100644
--- a/providers/databricks/src/airflow/providers/databricks/get_provider_info.py
+++ b/providers/databricks/src/airflow/providers/databricks/get_provider_info.py
@@ -49,6 +49,7 @@ def get_provider_info():
"/docs/apache-airflow-providers-databricks/operators/sql_statements.rst",
"/docs/apache-airflow-providers-databricks/operators/warehouse.rst",
"/docs/apache-airflow-providers-databricks/operators/copy_into.rst",
+
"/docs/apache-airflow-providers-databricks/operators/delta_table_producer.rst",
],
"tags": ["service"],
},
@@ -86,6 +87,7 @@ def get_provider_info():
"integration-name": "Databricks SQL",
"python-modules": [
"airflow.providers.databricks.operators.databricks_sql",
+ "airflow.providers.databricks.operators.delta_table",
"airflow.providers.databricks.operators.warehouse",
],
},
diff --git
a/providers/databricks/src/airflow/providers/databricks/operators/delta_table.py
b/providers/databricks/src/airflow/providers/databricks/operators/delta_table.py
new file mode 100644
index 00000000000..db36ea88ea6
--- /dev/null
+++
b/providers/databricks/src/airflow/providers/databricks/operators/delta_table.py
@@ -0,0 +1,452 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Produce asset events for Delta table versions committed outside of
Airflow."""
+
+from __future__ import annotations
+
+from collections.abc import Collection, Iterable, Iterator, Sequence
+from dataclasses import dataclass
+from datetime import date
+from functools import cached_property
+from typing import TYPE_CHECKING, Any, Literal
+
+from airflow.providers.common.compat.sdk import AirflowSkipException,
BaseOperator
+from airflow.providers.common.sql.hooks.handlers import fetch_all_handler
+from airflow.providers.databricks.exceptions import DatabricksDeltaTableError
+from airflow.providers.databricks.hooks.databricks_sql import DatabricksSqlHook
+from airflow.providers.databricks.utils.query_tags import build_query_tags
+
+if TYPE_CHECKING:
+ from airflow.providers.common.compat.assets import Asset
+ from airflow.providers.common.compat.sdk import Context
+ from airflow.providers.databricks.assets.databricks import
UnityTableIdentity
+
+CHECKPOINT_KEY_EXTRA = "databricks_delta_checkpoint_key"
+"""Asset event ``extra`` key that marks an event as a checkpoint written by
this producer."""
+
+DEFAULT_IGNORED_OPERATIONS = ("OPTIMIZE", "VACUUM START", "VACUUM END")
+
+_EVENT_PAGE_SIZE = 100
+
+ChangeKind = Literal["enrollment", "new_version", "recreated"]
+
+
+@dataclass(frozen=True)
+class DeltaCheckpoint:
+ """Last Delta version this producer published for a table, read back from
an asset event."""
+
+ version: int
+ table_id: str | None
+
+
+@dataclass(frozen=True)
+class _HistoryEntry:
+ version: int
+ operation: str | None
+ timestamp: str | None
+
+
+def _quote_identifier(part: str) -> str:
+ return "`" + part.replace("`", "``") + "`"
+
+
+def _as_list(value: Any) -> list[Any]:
+ if value is None:
+ return []
+ if isinstance(value, (list, tuple)):
+ return list(value)
+ return [value]
+
+
+def _row_to_dict(row: Any) -> dict[str, Any]:
+ if hasattr(row, "_asdict"):
+ return row._asdict()
+ if hasattr(row, "asDict"):
+ return row.asDict()
+ return dict(row)
+
+
+class DatabricksDeltaTableVersionProducerOperator(BaseOperator):
+ """
+ Publish an asset event when a Unity Catalog Delta table gets a newer
committed version.
+
+ Use this operator in a recurring producer Dag to turn Delta commits made
by other jobs or
+ systems into Airflow asset events. Downstream Dags schedule on
``unity_table.to_asset()``.
+
+ Each run reads the newest Delta version, compares it with the checkpoint,
and either publishes
+ one asset event for the newest version or skips. Several commits between
two runs are coalesced
+ into one event for the newest version. Intermediate commits are not
delivered one by one.
+
+ The checkpoint is the newest asset event that this producer published for
the asset, found by
+ ``checkpoint_key`` in the event ``extra``. Airflow records the event when
the task succeeds, so
+ publishing the event and advancing the checkpoint are the same write.
Delivery is at least
+ once: concurrent runs or a cleared run can publish the same version again.
Consumers that must
+ not process a version twice deduplicate on ``extra["dedup_id"]``.
+
+ An event for version N does not pin downstream reads to N. A consumer that
needs a
+ reproducible read queries ``VERSION AS OF`` ``extra["version"]``.
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the
guide:
+ :ref:`howto/operator:DatabricksDeltaTableVersionProducerOperator`
+
+ :param unity_table: Static identity of the observed table. It is added to
``inlets`` (to read
+ the checkpoint) and, when ``outlets`` is omitted, ``outlets`` is
``[unity_table.to_asset()]``.
+ Explicit ``outlets`` must include ``unity_table.to_asset()``, because
the checkpoint is the
+ event published on that asset.
+ :param checkpoint_key: Scope of the checkpoint. Producers with different
keys keep separate
+ checkpoints for the same asset. Defaults to ``"<dag_id>.<task_id>"``.
Changing it starts a
+ new enrollment.
+ :param baseline_version: Version treated as already delivered when no
checkpoint exists. When
+ ``None`` (the default), the first run publishes an ``enrollment``
event for the current version.
+ :param ignored_operations: Delta history operations that do not count as a
change. Defaults to
+ ``OPTIMIZE``, ``VACUUM START`` and ``VACUUM END``. Pass an empty
collection to publish for
+ every commit.
+ :param max_history_rows: Largest ``DESCRIBE HISTORY`` window read to prove
that all commits after
+ the checkpoint are ignored operations. When the window or the retained
history does not
+ reach the checkpoint, the operator publishes rather than risk missing
a data change.
+ :param on_recreation: What to do when the table id differs from the
checkpoint, or the version
+ moved backwards, which means the table was dropped and recreated or
replaced by a rename.
+ ``"publish"`` publishes a ``recreated`` event and checkpoints the new
table. ``"fail"``
+ raises and leaves the checkpoint untouched.
+ :param databricks_conn_id: Reference to :ref:`Databricks connection
id<howto/connection:databricks>`.
+ :param http_path: HTTP path of a Databricks SQL warehouse or all-purpose
cluster.
+ :param sql_warehouse_name: Name of a Databricks SQL warehouse, used when
``http_path`` is not set.
+ :param session_configuration: Spark session parameters.
+ :param http_headers: (k, v) pairs set as HTTP headers on every request.
+ :param client_parameters: Additional Databricks SQL connector parameters.
+ :param query_tags: Query tags attached to the Databricks SQL queries.
(templated)
+ :param include_airflow_query_tags: If True, add Airflow Dag/task/run
metadata as query tags.
+ """
+
+ template_fields: Sequence[str] = ("databricks_conn_id", "http_headers",
"query_tags")
+
+ def __init__(
+ self,
+ *,
+ unity_table: UnityTableIdentity,
+ checkpoint_key: str | None = None,
+ baseline_version: int | None = None,
+ ignored_operations: Collection[str] | None = None,
+ max_history_rows: int = 1000,
+ on_recreation: Literal["publish", "fail"] = "publish",
+ databricks_conn_id: str = DatabricksSqlHook.default_conn_name,
+ http_path: str | None = None,
+ sql_warehouse_name: str | None = None,
+ session_configuration: dict[str, str] | None = None,
+ http_headers: list[tuple[str, str]] | None = None,
+ client_parameters: dict[str, Any] | None = None,
+ query_tags: dict[str, str | None] | None = None,
+ include_airflow_query_tags: bool = True,
+ **kwargs,
+ ) -> None:
+ if baseline_version is not None and (
+ isinstance(baseline_version, bool)
+ or not isinstance(baseline_version, int)
+ or baseline_version < 0
+ ):
+ raise ValueError(f"baseline_version must be a non-negative int,
got {baseline_version!r}.")
+ if max_history_rows < 1:
+ raise ValueError(f"max_history_rows must be at least 1, got
{max_history_rows!r}.")
+ if on_recreation not in ("publish", "fail"):
+ raise ValueError(f"on_recreation must be 'publish' or 'fail', got
{on_recreation!r}.")
+ if checkpoint_key is not None and not checkpoint_key:
+ raise ValueError("checkpoint_key must not be empty.")
+
+ asset = unity_table.to_asset()
+ outlets = _as_list(kwargs.get("outlets", [asset]))
+ if not any(getattr(outlet, "uri", None) == asset.uri for outlet in
outlets):
+ # The checkpoint is the event published on this asset. Without it
every run re-enrolls.
+ raise ValueError(f"outlets must include unity_table.to_asset()
({asset.uri!r}).")
+ kwargs["outlets"] = outlets
+ inlets = _as_list(kwargs.pop("inlets", None))
+ if not any(getattr(inlet, "uri", None) == asset.uri for inlet in
inlets):
+ inlets.append(asset)
+ kwargs["inlets"] = inlets
+ # Concurrent runs of one producer read the same checkpoint and publish
duplicates.
+ kwargs.setdefault("max_active_tis_per_dag", 1)
+ super().__init__(**kwargs)
+
+ self.unity_table = unity_table
+ self._checkpoint_key = checkpoint_key
+ self.baseline_version = baseline_version
+ self.ignored_operations = tuple(
+ DEFAULT_IGNORED_OPERATIONS if ignored_operations is None else
ignored_operations
+ )
+ self._ignored = frozenset(op.casefold() for op in
self.ignored_operations)
+ self.max_history_rows = max_history_rows
+ self.on_recreation = on_recreation
+ self.databricks_conn_id = databricks_conn_id
+ self.http_path = http_path
+ self.sql_warehouse_name = sql_warehouse_name
+ self.session_configuration = session_configuration
+ self.http_headers = http_headers
+ self.client_parameters = client_parameters or {}
+ self.query_tags = query_tags or {}
+ self.include_airflow_query_tags = include_airflow_query_tags
+
+ @property
+ def checkpoint_key(self) -> str:
+ return self._checkpoint_key or f"{self.dag_id}.{self.task_id}"
+
+ @cached_property
+ def asset(self) -> Asset:
+ return self.unity_table.to_asset()
+
+ @cached_property
+ def table_name(self) -> str:
+ t = self.unity_table
+ return ".".join(_quote_identifier(p) for p in (t.catalog, t.schema,
t.table))
+
+ @cached_property
+ def hook(self) -> DatabricksSqlHook:
+ return DatabricksSqlHook(
+ self.databricks_conn_id,
+ http_path=self.http_path,
+ sql_endpoint_name=self.sql_warehouse_name,
+ session_configuration=self.session_configuration,
+ http_headers=self.http_headers,
+ caller="DatabricksDeltaTableVersionProducerOperator",
+ **self.client_parameters,
+ )
+
+ def execute(self, context: Context) -> dict[str, Any]:
+ hook = self.hook
+ if (hook.host or "").lower() != self.unity_table.host:
+ raise ValueError(
+ f"Databricks connection host {hook.host!r} does not match "
+ f"unity_table host {self.unity_table.host!r}."
+ )
+ hook.query_tags = build_query_tags(context, self.query_tags,
self.include_airflow_query_tags)
+
+ checkpoint = self.read_checkpoint(context)
+ table_id = self._describe_table_id()
+ latest = self._read_history(limit=1)
+ if not latest:
+ raise DatabricksDeltaTableError(f"DESCRIBE HISTORY returned no
rows for {self.table_name}.")
+ head = latest[0]
+
+ change: ChangeKind
+ previous: int | None
+ if checkpoint is None:
+ if self.baseline_version is None:
+ change, previous = "enrollment", None
+ elif head.version < self.baseline_version:
+ raise DatabricksDeltaTableError(
+ f"Delta table {self.table_name} is at version
{head.version}, "
+ f"older than baseline_version {self.baseline_version}."
+ )
+ else:
+ previous = self.baseline_version
+ change = "new_version"
+ elif (
+ checkpoint.table_id is not None and table_id is not None and
checkpoint.table_id != table_id
+ ) or head.version < checkpoint.version:
+ if self.on_recreation == "fail":
+ raise DatabricksDeltaTableError(
+ f"Delta table {self.table_name} changed identity since the
checkpoint "
+ f"(table id {checkpoint.table_id!r} version
{checkpoint.version} -> "
+ f"table id {table_id!r} version {head.version}). Was it
dropped and recreated?"
+ )
+ change, previous = "recreated", checkpoint.version
+ else:
+ change, previous = "new_version", checkpoint.version
+
+ operation_entry = head
+ if change == "new_version" and previous is not None:
+ if head.version == previous:
+ raise AirflowSkipException(
+ f"Delta table {self.table_name} is still at version
{previous}; no event published."
+ )
+ relevant = self._find_newest_relevant_commit(head, previous)
+ if relevant is None:
+ raise AirflowSkipException(
+ f"Delta table {self.table_name} only has
{sorted(self.ignored_operations)} commits "
+ f"after version {previous}; no event published."
+ )
+ # ``version`` stays at head for VERSION AS OF; ``operation`` names
the data change.
+ operation_entry = relevant
+
+ extra = self._build_extra(
+ head=head, operation_entry=operation_entry, table_id=table_id,
previous=previous, change=change
+ )
+ for outlet in self.outlets:
+ if getattr(outlet, "uri", None) == self.asset.uri:
+ context["outlet_events"][outlet].extra = extra
+ else:
+ # Checkpoint markers are only for the table asset event used
as the producer checkpoint.
+ context["outlet_events"][outlet].extra = {
+ key: value for key, value in extra.items() if key !=
CHECKPOINT_KEY_EXTRA
+ }
+ self.log.info(
+ "Publishing %s event for %s at Delta version %s (previous %s).",
+ change,
+ self.asset.uri,
+ head.version,
+ previous,
+ )
+ return extra
+
+ def read_checkpoint(self, context: Context) -> DeltaCheckpoint | None:
+ """Return the newest checkpoint for ``checkpoint_key`` from the
asset's events, if any."""
+ accessor = context["inlet_events"][self.asset]
+ events: Iterable[Any]
+ if all(hasattr(accessor, name) for name in ("extra", "ascending",
"limit")):
+ # Airflow versions that filter asset events by ``extra`` on the
API server.
+ events = accessor.extra(CHECKPOINT_KEY_EXTRA,
self.checkpoint_key).ascending(False).limit(1)
+ elif all(hasattr(accessor, name) for name in ("ascending", "limit",
"before")):
+ # Airflow 3.2 and 3.3 can order and limit events but not filter
them by ``extra``.
+ events = self._newest_events_paged(accessor)
+ else:
+ events = reversed(accessor)
+ for event in events:
+ extra = event.extra or {}
+ if extra.get(CHECKPOINT_KEY_EXTRA) != self.checkpoint_key:
+ continue
+ version = extra.get("version")
+ if isinstance(version, bool) or not isinstance(version, int):
+ raise DatabricksDeltaTableError(
+ f"Checkpoint event for {self.asset.uri} with key
{self.checkpoint_key!r} "
+ f"has no integer version: {extra!r}."
+ )
+ table_id = extra.get("delta_table_id")
+ return DeltaCheckpoint(version=version, table_id=str(table_id) if
table_id is not None else None)
+ return None
+
+ def _newest_events_paged(self, accessor: Any) -> Iterator[Any]:
+ """Yield the asset's events newest first, fetching
``_EVENT_PAGE_SIZE`` events per request."""
+ seen: set[Any] = set()
+ page = accessor.ascending(False).limit(_EVENT_PAGE_SIZE)
+ while True:
+ events = list(page)
+ for event in events:
+ event_id = getattr(event, "id", None)
+ if event_id is not None and event_id in seen:
+ continue
+ seen.add(event_id)
+ yield event
+ if len(events) < _EVENT_PAGE_SIZE:
+ return
+ oldest = events[-1].timestamp
+ if oldest == events[0].timestamp:
+ # A full page with one timestamp. ``before`` is inclusive, so
stop instead of looping.
+ self.log.warning(
+ "Cannot advance asset event paging for %s: a full page of
%s events shares "
+ "timestamp %s. Older events are not scanned; this run may
emit an enrollment event.",
+ self.asset.uri,
+ _EVENT_PAGE_SIZE,
+ oldest,
+ )
+ return
+ page = accessor.before(oldest)
+
+ def _describe_table_id(self) -> str | None:
+ rows = self.hook.run(f"DESCRIBE DETAIL {self.table_name}",
handler=fetch_all_handler)
+ if not rows:
+ raise DatabricksDeltaTableError(f"DESCRIBE DETAIL returned no rows
for {self.table_name}.")
+ detail = _row_to_dict(rows[0])
+ table_format = detail.get("format")
+ if table_format is not None and str(table_format).lower() != "delta":
+ raise DatabricksDeltaTableError(
+ f"{self.table_name} is a {table_format!r} table; only Delta
tables have versions."
+ )
+ table_id = detail.get("id")
+ return str(table_id) if table_id is not None else None
+
+ def _read_history(self, *, limit: int) -> list[_HistoryEntry]:
+ rows = self.hook.run(
+ f"DESCRIBE HISTORY {self.table_name} LIMIT {int(limit)}",
handler=fetch_all_handler
+ )
+ entries = []
+ for row in rows or []:
+ data = _row_to_dict(row)
+ timestamp = data.get("timestamp")
+ entries.append(
+ _HistoryEntry(
+ version=int(data["version"]),
+ operation=data.get("operation"),
+ timestamp=timestamp.isoformat()
+ if isinstance(timestamp, date)
+ else (str(timestamp) if timestamp is not None else None),
+ )
+ )
+ return sorted(entries, key=lambda e: e.version, reverse=True)
+
+ def _find_newest_relevant_commit(self, head: _HistoryEntry, previous: int)
-> _HistoryEntry | None:
+ """
+ Newest non-ignored commit in ``(previous, head.version]``.
+
+ Returns ``None`` when every retained commit in the window is an
ignored operation. When the
+ window cannot be fully checked (too large or incomplete history),
returns ``head`` so the
+ caller publishes without proof — ``operation`` may then be an ignored
operation.
+ """
+ if not self._ignored:
+ return head
+ if not self._is_ignored(head):
+ return head
+ needed = head.version - previous
+ if needed > self.max_history_rows:
+ self.log.warning(
+ "%s commits since version %s exceed max_history_rows=%s;
publishing without proof "
+ "that they are all ignored operations.",
+ needed,
+ previous,
+ self.max_history_rows,
+ )
+ return head
+ # History is newest-first, so the first non-ignored entry is the
newest relevant commit.
+ window = [e for e in self._read_history(limit=needed) if e.version >
previous]
+ if len(window) < needed:
+ self.log.warning(
+ "Delta history after version %s is incomplete (%s of %s
commits retained); "
+ "publishing without proof that they are all ignored
operations.",
+ previous,
+ len(window),
+ needed,
+ )
+ return head
+ for entry in window:
+ if not self._is_ignored(entry):
+ return entry
+ return None
+
+ def _is_ignored(self, entry: _HistoryEntry) -> bool:
+ return entry.operation is not None and entry.operation.casefold() in
self._ignored
+
+ def _build_extra(
+ self,
+ *,
+ head: _HistoryEntry,
+ operation_entry: _HistoryEntry,
+ table_id: str | None,
+ previous: int | None,
+ change: ChangeKind,
+ ) -> dict[str, Any]:
+ t = self.unity_table
+ return {
+ "version": head.version,
+ "observed_version": head.version,
+ "previous_version": previous,
+ "table_identity": {"host": t.host, "catalog": t.catalog, "schema":
t.schema, "table": t.table},
+ "operation": operation_entry.operation,
+ "timestamp": head.timestamp,
+ "change": change,
+ "delta_table_id": table_id,
+ "dedup_id": f"{self.asset.uri}@{table_id or '-'}:{head.version}",
+ CHECKPOINT_KEY_EXTRA: self.checkpoint_key,
+ }
diff --git
a/providers/databricks/tests/system/databricks/example_databricks_delta_producer.py
b/providers/databricks/tests/system/databricks/example_databricks_delta_producer.py
new file mode 100644
index 00000000000..0c3d7efce2d
--- /dev/null
+++
b/providers/databricks/tests/system/databricks/example_databricks_delta_producer.py
@@ -0,0 +1,81 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example Dags that turn Delta table commits made outside Airflow into asset
events.
+
+The producer Dag polls a Unity Catalog table every 10 minutes and publishes an
asset event when the
+table has a newer Delta version. The consumer Dag is scheduled on that asset
and reads the exact
+version that the event reports.
+"""
+
+from __future__ import annotations
+
+import os
+from datetime import datetime
+
+from airflow.providers.common.compat.sdk import DAG
+from airflow.providers.databricks.assets.databricks import UnityTableIdentity
+from airflow.providers.databricks.operators.databricks_sql import
DatabricksSqlOperator
+from airflow.providers.databricks.operators.delta_table import (
+ DatabricksDeltaTableVersionProducerOperator,
+)
+
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = "example_databricks_delta_producer"
+DATABRICKS_HOST = os.environ.get("DATABRICKS_HOST",
"my-workspace.cloud.databricks.com")
+SQL_WAREHOUSE_NAME = os.environ.get("DATABRICKS_SQL_WAREHOUSE_NAME", "My
Warehouse")
+
+# [START howto_operator_databricks_delta_table_version_producer]
+orders = UnityTableIdentity(host=DATABRICKS_HOST, catalog="main",
schema="sales", table="orders")
+
+with DAG(
+ dag_id=DAG_ID,
+ schedule="*/10 * * * *",
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ max_active_runs=1,
+ tags=["example"],
+) as dag:
+ DatabricksDeltaTableVersionProducerOperator(
+ task_id="publish_orders_version",
+ unity_table=orders,
+ sql_warehouse_name=SQL_WAREHOUSE_NAME,
+ )
+
+with DAG(
+ dag_id=f"{DAG_ID}_consumer",
+ schedule=[orders.to_asset()],
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=["example"],
+) as consumer_dag:
+ DatabricksSqlOperator(
+ task_id="read_published_version",
+ sql_endpoint_name=SQL_WAREHOUSE_NAME,
+ sql=(
+ "SELECT count(*) FROM main.sales.orders VERSION AS OF "
+ "{{ (triggering_asset_events.values() | first |
last).extra['version'] }}"
+ ),
+ )
+# [END howto_operator_databricks_delta_table_version_producer]
+
+
+from tests_common.test_utils.system_tests import get_test_run # noqa: E402
+
+# Needed to run the example DAG with pytest (see:
contributing-docs/testing/system_tests.rst)
+test_run = get_test_run(dag)
diff --git
a/providers/databricks/tests/unit/databricks/operators/test_delta_table.py
b/providers/databricks/tests/unit/databricks/operators/test_delta_table.py
new file mode 100644
index 00000000000..21813fcedc3
--- /dev/null
+++ b/providers/databricks/tests/unit/databricks/operators/test_delta_table.py
@@ -0,0 +1,635 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import re
+from collections.abc import Sequence
+from datetime import UTC, datetime, timedelta
+from types import SimpleNamespace
+from typing import Any
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+
+from airflow.providers.common.compat.assets import Asset
+from airflow.providers.common.compat.sdk import DAG, AirflowSkipException
+from airflow.providers.databricks.assets.databricks import UnityTableIdentity
+from airflow.providers.databricks.exceptions import DatabricksDeltaTableError
+from airflow.providers.databricks.hooks.databricks_sql import
DatabricksSqlExecutionError
+from airflow.providers.databricks.operators.delta_table import (
+ CHECKPOINT_KEY_EXTRA,
+ DatabricksDeltaTableVersionProducerOperator,
+ DeltaCheckpoint,
+)
+from airflow.utils.session import create_session
+
+from tests_common.test_utils.compat import AssetEvent
+from tests_common.test_utils.dag import sync_dag_to_db
+from tests_common.test_utils.db import clear_db_assets, clear_db_dags,
clear_db_runs
+from tests_common.test_utils.version_compat import AIRFLOW_V_3_2_PLUS
+
+HOST = "xx.cloud.databricks.com"
+TABLE = UnityTableIdentity(host=HOST, catalog="Main", schema="Sales",
table="Orders")
+URI = f"databricks://{HOST}/main/sales/orders"
+QUOTED = "`main`.`sales`.`orders`"
+TABLE_ID = "11111111-1111-1111-1111-111111111111"
+OTHER_TABLE_ID = "22222222-2222-2222-2222-222222222222"
+KEY = "producer.watch_orders"
+HOOK_PATH =
"airflow.providers.databricks.operators.delta_table.DatabricksSqlHook"
+
+
+class FakeDeltaTable:
+ """In-memory stand-in for ``DESCRIBE DETAIL`` / ``DESCRIBE HISTORY`` on
one table."""
+
+ def __init__(
+ self,
+ commits: list[tuple[int, str]],
+ *,
+ table_id: str = TABLE_ID,
+ table_format: str = "delta",
+ retained_from: int = 0,
+ ):
+ self.commits = commits
+ self.table_id = table_id
+ self.table_format = table_format
+ self.retained_from = retained_from
+ self.statements: list[str] = []
+
+ def run(self, sql: str, handler=None):
+ self.statements.append(sql)
+ if sql.startswith("DESCRIBE DETAIL"):
+ return [SimpleNamespace(_asdict=lambda: {"format":
self.table_format, "id": self.table_id})]
+ match = re.fullmatch(r"DESCRIBE HISTORY (.+) LIMIT (\d+)", sql)
+ assert match, sql
+ limit = int(match.group(2))
+ retained = [c for c in self.commits if c[0] >= self.retained_from]
+ newest_first = sorted(retained, reverse=True)[:limit]
+ return [
+ SimpleNamespace(
+ _asdict=lambda v=v, op=op: {
+ "version": v,
+ "operation": op,
+ "timestamp": datetime(2026, 10, 1, 12, v % 60, tzinfo=UTC),
+ }
+ )
+ for v, op in newest_first
+ ]
+
+
+def make_event(extra: dict[str, Any]) -> SimpleNamespace:
+ return SimpleNamespace(extra=extra)
+
+
+def checkpoint_event(version: int, *, key: str = KEY, table_id: str =
TABLE_ID) -> SimpleNamespace:
+ return make_event({"version": version, "delta_table_id": table_id,
CHECKPOINT_KEY_EXTRA: key})
+
+
+class FilteringAccessor(Sequence):
+ """Accessor shaped like the Airflow 3 one that filters events on the API
server."""
+
+ def __init__(self, events):
+ self._events = list(events)
+ self.calls: list[tuple] = []
+
+ def extra(self, key, value):
+ self.calls.append(("extra", key, value))
+ self._events = [e for e in self._events if (e.extra or {}).get(key) ==
value]
+ return self
+
+ def ascending(self, ascending=True):
+ self.calls.append(("ascending", ascending))
+ if not ascending:
+ self._events = list(reversed(self._events))
+ return self
+
+ def limit(self, limit):
+ self.calls.append(("limit", limit))
+ self._events = self._events[:limit]
+ return self
+
+ def __getitem__(self, i):
+ return self._events[i]
+
+ def __len__(self):
+ return len(self._events)
+
+
+class PagingAccessor(Sequence):
+ """Accessor shaped like the Airflow 3.2/3.3 one: ordering, limit and
``before``, no ``extra`` filter."""
+
+ def __init__(self, events):
+ self._all = list(events)
+ self._ascending = True
+ self._limit = None
+ self._before = None
+ self.requests = 0
+
+ self._cache = None
+
+ def ascending(self, ascending=True):
+ self._ascending, self._cache = ascending, None
+ return self
+
+ def limit(self, limit):
+ self._limit, self._cache = limit, None
+ return self
+
+ def before(self, before):
+ self._before, self._cache = before, None
+ return self
+
+ @property
+ def _events(self):
+ # Like the real accessor, one request per filter change, cached until
the filter changes again.
+ if self._cache is None:
+ self.requests += 1
+ events = [e for e in self._all if self._before is None or
e.timestamp <= self._before]
+ events.sort(key=lambda e: (e.timestamp, e.id), reverse=not
self._ascending)
+ self._cache = events[: self._limit] if self._limit is not None
else events
+ return self._cache
+
+ def __iter__(self):
+ return iter(self._events)
+
+ def __getitem__(self, i):
+ return self._events[i]
+
+ def __len__(self):
+ return len(self._events)
+
+
+def timed_event(i: int, extra: dict[str, Any]) -> SimpleNamespace:
+ return SimpleNamespace(
+ id=i, timestamp=datetime(2026, 10, 1, tzinfo=UTC) +
timedelta(minutes=i), extra=extra
+ )
+
+
+class _EventsByAssetUri:
+ """
+ Map Asset keys by URI.
+
+ Real ``inlet_events`` / ``outlet_events`` accessors resolve Assets to a
hashable unique
+ key. Older Airflow versions ship an Asset that is not usable as a plain
``dict`` key
+ (unhashable Asset on 3.0; hashing fails on the mutable ``extra`` dict on
3.1), so the
+ test stand-in must not rely on ``Asset.__hash__``.
+ """
+
+ def __init__(self, initial: dict[str, Any] | None = None, *,
default_factory=None):
+ self._by_uri: dict[str, Any] = dict(initial or {})
+ self._default_factory = default_factory
+
+ @staticmethod
+ def _uri(key: Any) -> str:
+ return key if isinstance(key, str) else key.uri
+
+ def __getitem__(self, key: Any) -> Any:
+ uri = self._uri(key)
+ if uri not in self._by_uri:
+ if self._default_factory is None:
+ raise KeyError(key)
+ self._by_uri[uri] = self._default_factory()
+ return self._by_uri[uri]
+
+ def items(self):
+ return ((SimpleNamespace(uri=uri), value) for uri, value in
self._by_uri.items())
+
+
+def make_context(events=None, accessor=None) -> Any:
+ """Return a minimal stand-in for the task ``Context`` with only the asset
event accessors."""
+ inlet = accessor if accessor is not None else list(events or [])
+ return {
+ "inlet_events": _EventsByAssetUri({URI: inlet}),
+ "outlet_events": _EventsByAssetUri(default_factory=lambda:
SimpleNamespace(extra={})),
+ }
+
+
+def published(context) -> dict[str, Any] | None:
+ for asset, event in context["outlet_events"].items():
+ if asset.uri == URI:
+ return event.extra
+ return None
+
+
[email protected]
+def table():
+ return FakeDeltaTable([(0, "CREATE TABLE"), (1, "WRITE"), (2, "WRITE")])
+
+
[email protected]
+def hook(table):
+ with mock.patch(HOOK_PATH, autospec=True) as hook_cls:
+ instance = hook_cls.return_value
+ type(instance).host = mock.PropertyMock(return_value=HOST)
+ instance.run.side_effect = table.run
+ yield instance
+
+
+def make_operator(**kwargs) -> DatabricksDeltaTableVersionProducerOperator:
+ kwargs.setdefault("checkpoint_key", KEY)
+ return DatabricksDeltaTableVersionProducerOperator(
+ task_id="watch_orders", unity_table=TABLE,
http_path="/sql/1.0/warehouses/abc", **kwargs
+ )
+
+
+class TestConstruction:
+ def test_defaults(self):
+ op = make_operator(checkpoint_key=None)
+ assert [a.uri for a in op.outlets] == [URI]
+ assert [a.uri for a in op.inlets] == [URI]
+ assert op.max_active_tis_per_dag == 1
+ assert op.ignored_operations == ("OPTIMIZE", "VACUUM START", "VACUUM
END")
+ assert op.table_name == QUOTED
+
+ def test_keeps_user_inlets_and_outlets(self):
+ other = Asset(uri="s3://bucket/other")
+ op = make_operator(inlets=[other], outlets=[other, TABLE.to_asset()],
max_active_tis_per_dag=4)
+ assert [a.uri for a in op.inlets] == ["s3://bucket/other", URI]
+ assert [a.uri for a in op.outlets] == ["s3://bucket/other", URI]
+ assert op.max_active_tis_per_dag == 4
+
+ def test_accepts_single_inlet_and_outlet(self):
+ other = Asset(uri="s3://bucket/other")
+ op = make_operator(inlets=other, outlets=TABLE.to_asset())
+ assert [a.uri for a in op.inlets] == ["s3://bucket/other", URI]
+ assert [a.uri for a in op.outlets] == [URI]
+
+ @pytest.mark.parametrize("outlets", [[], ["s3://bucket/other"]])
+ def test_rejects_outlets_without_the_table_asset(self, outlets):
+ with pytest.raises(ValueError, match="outlets must include"):
+ make_operator(outlets=[Asset(uri=uri) for uri in outlets])
+
+ def test_does_not_duplicate_inlet(self):
+ op = make_operator(inlets=[TABLE.to_asset()])
+ assert [a.uri for a in op.inlets] == [URI]
+
+ @pytest.mark.parametrize(
+ ("kwargs", "message"),
+ [
+ ({"baseline_version": -1}, "baseline_version"),
+ ({"baseline_version": "3"}, "baseline_version"),
+ ({"baseline_version": True}, "baseline_version"),
+ ({"max_history_rows": 0}, "max_history_rows"),
+ ({"on_recreation": "ignore"}, "on_recreation"),
+ ({"checkpoint_key": ""}, "checkpoint_key"),
+ ],
+ )
+ def test_rejects_invalid_arguments(self, kwargs, message):
+ with pytest.raises(ValueError, match=message):
+ make_operator(**kwargs)
+
+ def test_quotes_backticks_in_identifiers(self):
+ odd = UnityTableIdentity(host=HOST, catalog="main", schema="sales",
table="we`ird")
+ op = DatabricksDeltaTableVersionProducerOperator(task_id="t",
unity_table=odd)
+ assert op.table_name == "`main`.`sales`.`we``ird`"
+
+
+class TestEnrollment:
+ def test_first_run_publishes_enrollment_event(self, hook, table):
+ context = make_context()
+ result = make_operator().execute(context)
+
+ assert published(context) == result
+ assert result == {
+ "version": 2,
+ "observed_version": 2,
+ "previous_version": None,
+ "table_identity": {"host": HOST, "catalog": "main", "schema":
"sales", "table": "orders"},
+ "operation": "WRITE",
+ "timestamp": "2026-10-01T12:02:00+00:00",
+ "change": "enrollment",
+ "delta_table_id": TABLE_ID,
+ "dedup_id": f"{URI}@{TABLE_ID}:2",
+ CHECKPOINT_KEY_EXTRA: KEY,
+ }
+ assert table.statements == [f"DESCRIBE DETAIL {QUOTED}", f"DESCRIBE
HISTORY {QUOTED} LIMIT 1"]
+
+ def test_baseline_at_current_version_skips(self, hook):
+ context = make_context()
+ with pytest.raises(AirflowSkipException, match="still at version 2"):
+ make_operator(baseline_version=2).execute(context)
+ assert published(context) is None
+
+ def test_baseline_older_than_current_publishes_new_version(self, hook):
+ context = make_context()
+ result = make_operator(baseline_version=1).execute(context)
+ assert result["change"] == "new_version"
+ assert result["previous_version"] == 1
+ assert result["version"] == 2
+
+ def test_baseline_newer_than_table_fails(self, hook):
+ with pytest.raises(DatabricksDeltaTableError, match="older than
baseline_version 5"):
+ make_operator(baseline_version=5).execute(make_context())
+
+ def test_checkpoint_wins_over_baseline(self, hook):
+ context = make_context([checkpoint_event(2)])
+ with pytest.raises(AirflowSkipException):
+ make_operator(baseline_version=0).execute(context)
+
+ def test_default_checkpoint_key_is_dag_and_task(self, hook):
+ with DAG("producer", schedule=None):
+ op = make_operator(checkpoint_key=None)
+ result = op.execute(make_context())
+ assert result[CHECKPOINT_KEY_EXTRA] == "producer.watch_orders"
+
+ def test_checkpoint_key_only_on_table_asset_outlet(self, hook, table):
+ other = Asset(uri="s3://bucket/other")
+ op = make_operator(outlets=[other, TABLE.to_asset()])
+ context = make_context()
+ result = op.execute(context)
+
+ table_extra = context["outlet_events"][TABLE.to_asset()].extra
+ other_extra = context["outlet_events"][other].extra
+
+ assert table_extra == result
+ assert table_extra["version"] == other_extra["version"] == 2
+ assert table_extra[CHECKPOINT_KEY_EXTRA] == KEY
+ assert CHECKPOINT_KEY_EXTRA not in other_extra
+ assert other_extra["delta_table_id"] == TABLE_ID
+ assert other_extra["change"] == "enrollment"
+
+
+class TestVersionChanges:
+ def test_unchanged_version_skips_without_event(self, hook):
+ context = make_context([checkpoint_event(2)])
+ with pytest.raises(AirflowSkipException, match="still at version 2"):
+ make_operator().execute(context)
+ assert published(context) is None
+
+ def test_new_versions_are_coalesced_into_one_event_for_the_latest(self,
hook, table):
+ table.commits += [(3, "WRITE"), (4, "MERGE"), (5, "UPDATE")]
+ context = make_context([checkpoint_event(2)])
+ result = make_operator().execute(context)
+ assert (result["change"], result["previous_version"],
result["version"]) == ("new_version", 2, 5)
+ assert result["operation"] == "UPDATE"
+ assert published(context) == result
+ # The head commit is a data change, so the operator does not scan
further history.
+ assert table.statements[-1] == f"DESCRIBE HISTORY {QUOTED} LIMIT 1"
+
+ def test_only_maintenance_commits_skip(self, hook, table):
+ table.commits += [(3, "OPTIMIZE"), (4, "VACUUM START"), (5, "VACUUM
END")]
+ context = make_context([checkpoint_event(2)])
+ with pytest.raises(AirflowSkipException, match="only has"):
+ make_operator().execute(context)
+ assert published(context) is None
+ assert table.statements[-1] == f"DESCRIBE HISTORY {QUOTED} LIMIT 3"
+
+ @pytest.mark.parametrize(
+ ("commits", "operation"),
+ [
+ ([(3, "WRITE"), (4, "OPTIMIZE"), (5, "VACUUM END")], "WRITE"),
+ # WRITE then MERGE then OPTIMIZE: fails if the window is walked
oldest-first.
+ ([(3, "WRITE"), (4, "MERGE"), (5, "OPTIMIZE")], "MERGE"),
+ ],
+ ids=["write-behind-optimize-vacuum", "write-merge-optimize"],
+ )
+ def test_data_commit_hidden_behind_maintenance_publishes(self, hook,
table, commits, operation):
+ table.commits += commits
+ result = make_operator().execute(make_context([checkpoint_event(2)]))
+ assert result["version"] == 5
+ assert result["operation"] == operation
+
+ def test_ignored_operations_are_case_insensitive(self, hook, table):
+ table.commits += [(3, "optimize")]
+ with pytest.raises(AirflowSkipException):
+ make_operator().execute(make_context([checkpoint_event(2)]))
+
+ def test_empty_ignored_operations_publish_for_every_commit(self, hook,
table):
+ table.commits += [(3, "OPTIMIZE")]
+ result =
make_operator(ignored_operations=()).execute(make_context([checkpoint_event(2)]))
+ assert result["version"] == 3
+ assert table.statements[-1] == f"DESCRIBE HISTORY {QUOTED} LIMIT 1"
+
+ def test_expired_history_publishes_conservatively(self, hook, table):
+ table.commits += [(3, "WRITE"), (4, "OPTIMIZE"), (5, "OPTIMIZE")]
+ table.retained_from = 4 # Versions 3 and earlier were removed from
the Delta log.
+ result = make_operator().execute(make_context([checkpoint_event(2)]))
+ assert result["version"] == 5
+ # Incomplete window: operation stays at the head commit.
+ assert result["operation"] == "OPTIMIZE"
+
+ def test_history_beyond_scan_window_publishes_conservatively(self, hook,
table):
+ table.commits += [(v, "OPTIMIZE") for v in range(3, 13)]
+ result =
make_operator(max_history_rows=5).execute(make_context([checkpoint_event(2)]))
+ assert result["version"] == 12
+ assert result["operation"] == "OPTIMIZE"
+ # It did not run the large scan at all.
+ assert all("LIMIT 10" not in s for s in table.statements)
+
+
+class TestRecreation:
+ def test_new_table_id_publishes_recreated(self, hook, table):
+ table.table_id = OTHER_TABLE_ID
+ context = make_context([checkpoint_event(2)])
+ result = make_operator().execute(context)
+ assert result["change"] == "recreated"
+ assert result["delta_table_id"] == OTHER_TABLE_ID
+ assert result["dedup_id"] == f"{URI}@{OTHER_TABLE_ID}:2"
+
+ def test_version_moved_backwards_publishes_recreated(self, hook, table):
+ table.commits = [(0, "CREATE OR REPLACE TABLE")]
+ result = make_operator().execute(make_context([checkpoint_event(7,
table_id=None)]))
+ assert (result["change"], result["previous_version"],
result["version"]) == ("recreated", 7, 0)
+
+ def test_recreated_with_only_maintenance_still_publishes(self, hook,
table):
+ table.table_id = OTHER_TABLE_ID
+ table.commits = [(0, "CREATE TABLE"), (1, "OPTIMIZE")]
+ result = make_operator().execute(make_context([checkpoint_event(1)]))
+ assert result["change"] == "recreated"
+
+ def test_on_recreation_fail_raises_and_publishes_nothing(self, hook,
table):
+ table.table_id = OTHER_TABLE_ID
+ context = make_context([checkpoint_event(2)])
+ with pytest.raises(DatabricksDeltaTableError, match="dropped and
recreated"):
+ make_operator(on_recreation="fail").execute(context)
+ assert published(context) is None
+
+
+class TestTableErrors:
+ def test_non_delta_table_fails(self, hook, table):
+ table.table_format = "parquet"
+ context = make_context()
+ with pytest.raises(DatabricksDeltaTableError, match="only Delta tables
have versions"):
+ make_operator().execute(context)
+ assert published(context) is None
+
+ def
test_missing_or_inaccessible_table_propagates_and_publishes_nothing(self, hook):
+ hook.run.side_effect =
DatabricksSqlExecutionError("[TABLE_OR_VIEW_NOT_FOUND]")
+ context = make_context([checkpoint_event(2)])
+ with pytest.raises(DatabricksSqlExecutionError):
+ make_operator().execute(context)
+ assert published(context) is None
+
+ def test_empty_history_fails(self, hook, table):
+ table.commits = []
+ with pytest.raises(DatabricksDeltaTableError, match="DESCRIBE HISTORY
returned no rows"):
+ make_operator().execute(make_context())
+
+ def test_host_mismatch_fails_before_any_query(self, hook):
+ type(hook).host =
mock.PropertyMock(return_value="other.cloud.databricks.com")
+ with pytest.raises(ValueError, match="does not match unity_table
host"):
+ make_operator().execute(make_context())
+ hook.run.assert_not_called()
+
+ def test_host_match_is_case_insensitive(self, hook):
+ type(hook).host = mock.PropertyMock(return_value=HOST.upper())
+ assert make_operator().execute(make_context())["version"] == 2
+
+
+class TestCheckpoint:
+ def test_ignores_events_from_other_producers_and_scopes(self, hook):
+ events = [
+ checkpoint_event(1),
+ checkpoint_event(2, key="another.producer"),
+ make_event({"version": 2, "observed_version": 2}), # Sensor
outlet event.
+ make_event({}), # COPY INTO outlet event.
+ make_event(None),
+ ]
+ op = make_operator()
+ assert op.read_checkpoint(make_context(events)) ==
DeltaCheckpoint(version=1, table_id=TABLE_ID)
+
+ def test_uses_newest_matching_event(self, hook):
+ events = [checkpoint_event(1), checkpoint_event(2), make_event({})]
+ assert make_operator().read_checkpoint(make_context(events)).version
== 2
+
+ def test_no_matching_event_means_no_checkpoint(self, hook):
+ assert make_operator().read_checkpoint(make_context([make_event({})]))
is None
+
+ def test_uses_server_side_filter_when_available(self, hook):
+ accessor = FilteringAccessor([checkpoint_event(1), checkpoint_event(5,
key="x"), make_event({})])
+ assert
make_operator().read_checkpoint(make_context(accessor=accessor)).version == 1
+ assert accessor.calls == [("extra", CHECKPOINT_KEY_EXTRA, KEY),
("ascending", False), ("limit", 1)]
+
+ def test_pages_newest_first_when_extra_filter_is_unavailable(self, hook):
+ events = [timed_event(0, {"version": 1, "delta_table_id": TABLE_ID,
CHECKPOINT_KEY_EXTRA: KEY})]
+ events += [timed_event(i, {}) for i in range(1, 250)]
+ accessor = PagingAccessor(events)
+ assert
make_operator().read_checkpoint(make_context(accessor=accessor)).version == 1
+ assert accessor.requests == 3
+
+ def test_paging_stops_at_the_newest_checkpoint(self, hook):
+ events = [timed_event(i, {}) for i in range(250)]
+ events.append(timed_event(250, {"version": 7, "delta_table_id":
TABLE_ID, CHECKPOINT_KEY_EXTRA: KEY}))
+ accessor = PagingAccessor(events)
+ assert
make_operator().read_checkpoint(make_context(accessor=accessor)).version == 7
+ assert accessor.requests == 1
+
+ def test_paging_without_checkpoint_reads_all_pages_once(self, hook):
+ accessor = PagingAccessor([timed_event(i, {}) for i in range(150)])
+ assert
make_operator().read_checkpoint(make_context(accessor=accessor)) is None
+ assert accessor.requests == 2
+
+ def test_corrupt_checkpoint_fails(self, hook):
+ bad = make_event({CHECKPOINT_KEY_EXTRA: KEY, "version": "2"})
+ with pytest.raises(DatabricksDeltaTableError, match="no integer
version"):
+ make_operator().read_checkpoint(make_context([bad]))
+
+
+class TestDelivery:
+ """
+ The checkpoint is the published event, so these tests replay the producer
across runs.
+
+ A run that is killed before the task succeeds leaves no event; Airflow
records the outlet
+ event only on success. These tests model that by appending the event to
the asset's
+ history only for runs that complete.
+ """
+
+ def run_once(self, history: list, *, succeed: bool = True) -> dict[str,
Any] | None:
+ context = make_context(list(history))
+ try:
+ make_operator().execute(context)
+ except AirflowSkipException:
+ return None
+ extra = published(context)
+ if succeed:
+ assert extra is not None
+ history.append(make_event(extra))
+ return extra
+
+ def test_crash_before_publication_redelivers_same_version(self, hook,
table):
+ history: list = [checkpoint_event(2)]
+ table.commits.append((3, "WRITE"))
+
+ lost = self.run_once(history, succeed=False)
+ retried = self.run_once(history)
+
+ assert lost is not None
+ assert retried == lost
+ assert retried["dedup_id"] == f"{URI}@{TABLE_ID}:3"
+
+ def test_crash_before_publication_coalesces_commits_made_meanwhile(self,
hook, table):
+ history: list = [checkpoint_event(2)]
+ table.commits.append((3, "WRITE"))
+ self.run_once(history, succeed=False)
+ table.commits.append((4, "WRITE"))
+
+ retried = self.run_once(history)
+ assert (retried["previous_version"], retried["version"]) == (2, 4)
+
+ def test_after_publication_checkpoint_has_advanced(self, hook, table):
+ history: list = [checkpoint_event(2)]
+ table.commits.append((3, "WRITE"))
+ assert self.run_once(history)["version"] == 3
+ assert self.run_once(history) is None
+ table.commits.append((4, "WRITE"))
+ assert self.run_once(history)["previous_version"] == 3
+
+ def test_concurrent_runs_publish_same_dedup_id(self, hook, table):
+ history: list = [checkpoint_event(2)]
+ table.commits.append((3, "WRITE"))
+ # Both runs read the checkpoint before either publishes.
+ first_ctx, second_ctx = make_context(list(history)),
make_context(list(history))
+ first = make_operator().execute(first_ctx)
+ second = make_operator().execute(second_ctx)
+ assert first["dedup_id"] == second["dedup_id"]
+ history += [make_event(first), make_event(second)]
+ assert self.run_once(history) is None
+
+
[email protected]_test
[email protected](not AIRFLOW_V_3_2_PLUS, reason="Runs the Dag through the
Airflow 3 execution API")
+class TestProducerDagRuns:
+ """Run a producer Dag against the metadata database, so checkpoints are
real asset events."""
+
+ @pytest.fixture(autouse=True)
+ def _clean_db(self):
+ clear_db_runs()
+ clear_db_assets()
+ clear_db_dags()
+ yield
+ clear_db_runs()
+ clear_db_assets()
+ clear_db_dags()
+
+ def test_events_and_skips_across_runs(self, hook, table):
+ with DAG("delta_producer", schedule=None) as dag:
+
DatabricksDeltaTableVersionProducerOperator(task_id="watch_orders",
unity_table=TABLE)
+ sync_dag_to_db(dag)
+
+ states = []
+ for commit in (None, None, (3, "WRITE"), (4, "OPTIMIZE"), None):
+ if commit:
+ table.commits.append(commit)
+ states.append(dag.test().get_task_instance("watch_orders").state)
+
+ with create_session() as session:
+ events =
session.scalars(select(AssetEvent).order_by(AssetEvent.id)).all()
+ published = [(e.asset.uri, e.extra["change"], e.extra["version"])
for e in events]
+
+ assert states == ["success", "skipped", "success", "skipped",
"skipped"]
+ assert published == [(URI, "enrollment", 2), (URI, "new_version", 3)]