Vamsi-klu commented on PR #66681:
URL: https://github.com/apache/airflow/pull/66681#issuecomment-4484829197
@moomindani thanks for the review. Two things addressed:
(A) CI — the failing `provider distributions tests / Compat 3.0.6:P3.10`
shard was artifact infra: `gh run download` couldn't find the CI-image stash
`ci-image-save-v3-linux_amd64-3.10-66681_merge`. All other compat jobs (2.11.1
/ 3.1.8 / 3.2.1), DB tests, mypy, static checks, and integration tests were
green. I've rebased onto current `main` (the branch was 129 commits behind) and
force-with-lease pushed, which re-runs the full matrix from scratch.
(B) Wire-payload validation — added a new `TestWorkflowDependsOnWirePayload`
class in
`providers/databricks/tests/unit/databricks/operators/test_databricks_workflow.py`
with two tests that drive
`_CreateDatabricksWorkflowOperator._create_or_reset_job` end to end and assert
on what the hook receives:
- `test_create_job_payload_carries_parent_depends_on` — captures
`launch_task._hook.create_job.call_args.args[0]` (the new-job path) and asserts
`tasks[task_b]["depends_on"] == [{"task_key": <md5(task_a)>}]`.
- `test_reset_job_payload_carries_parent_depends_on` — captures
`launch_task._hook.reset_job.call_args.args[1]` (the existing-job path,
`job_id=42`) and asserts the same shape.
Both tests build a real DAG + `DatabricksWorkflowTaskGroup` populated with
real `DatabricksNotebookOperator`s (no operator mocks), so the captured
`job_spec` is exactly what would be POSTed to `/api/2.1/jobs/create` and
`/api/2.1/jobs/reset`. The existing `TestWorkflowDependsOn` class still covers
the in-process `create_workflow_json` payload; this new class closes the loop
at the wire boundary and lives in CI forever.
I don't have a Databricks workspace handy for the live `airflow dags test` +
job-graph screenshot, but the call-args assertion verifies the same JSON the
Jobs API would receive. The runtime fix has also been live since #48492 (April
2025) without regressions reported on the `depends_on` path. Happy to add the
screenshot too if you'd still like it.
---
Drafted-by: Claude Code (Opus 4.7); reviewed by @Vamsi-klu before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
