zozo123 commented on PR #74225: URL: https://github.com/apache/airflow/pull/74225#issuecomment-5997865817
Thanks @goyaladitay11 — really appreciate you jumping on Phase 1 of #74195. The shape is right: a deferrable `DatabricksDeltaTableVersionSensor` + trigger on `DESCRIBE HISTORY … LIMIT 1`, with enrollment / `target_version` / `allow_recreation`, docs, and tests is exactly the first slice we wanted for “someone else wrote the table, wake my Dag.” A few notes before this is mergeable, in the spirit of keeping Unity assets trustworthy for Airflow + Databricks SQL / jobs users: 1. **Please wait on / rebase onto #74191.** That PR (already approved) introduces `UnityTableIdentity` and the `databricks://…/catalog/schema/table` URI conventions with the case-folding fixes potiuk asked for. #74195 explicitly says to reuse that outcome and not introduce a second identity/URI scheme. This branch re-adds `UnityTableIdentity` in `assets/databricks.py` — that will collide and risk two slightly different “same table” assets for downstream Dags. Drop the local copy and import from whatever #74191 lands. 2. **Be precise about what the outlet event means.** Same line we just settled on #74196: if the sensor declares outlets, the task-success asset event means “a newer Delta version was observed,” with version + table identity in `extra` — not a vague “table refreshed,” and not a promise that every intermediate commit was delivered. Authors who need a pinned read still have to `VERSION AS OF` that observed version themselves (your docs already nod at this; please keep that loud). 3. **Label scope as Phase 1 only.** Sensor-inside-a-scheduled-task is good. The continuous external producer / coalescing path in #74195 can stay a follow-up so review stays focused. Happy to re-review once this sits on top of #74191’s identity types. Thanks again for taking this on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
