nyxrv0id opened a new pull request, #75429:
URL: https://github.com/apache/airflow/pull/75429
### Description
When tasks are placed inside a `DatabricksWorkflowTaskGroup`, the group
creates a `_CreateDatabricksWorkflowOperator` ("launch task") that constructs a
Databricks Workflow job definition via REST API by calling
`_convert_to_databricks_workflow_task()` on each member task.
Previously, member task template fields (e.g.
`DatabricksTaskOperator.task_config` or custom operator template attributes)
were not rendered against the launch task's execution context before building
the payload. In addition, `DatabricksNotebookOperator` omitted `notebook_path`
and `source` from its declared `template_fields`.
This PR:
1. Adds `notebook_path` and `source` to
`DatabricksNotebookOperator.template_fields`.
2. Updates `DatabricksTaskBaseOperator._convert_to_databricks_workflow_task`
to render declared `template_fields` against the launch task's execution
context using an isolated shallow proxy (`copy.copy(self)`), safely excluding
`workflow_run_metadata` to avoid resolving `XComArg` prematurely and preventing
in-place attribute mutation across repeated DAG runs.
3. Updates `DatabricksNotebookOperator._convert_to_databricks_workflow_task`
to merge parent TaskGroup packages and notebook parameters on a shallow proxy,
preserving the original operator instance's parameters and packages across runs.
4. Adds documentation in `workflow.rst` and the
`DatabricksWorkflowTaskGroup` docstring clarifying that run-scoped Jinja
expressions (`{{ ds }}`, `{{ dag_run }}`, `{{ params }}`) are rendered during
workflow definition, whereas member task-scoped variables (e.g. `{{ ti }}` or
task-specific XCom pulls) are not supported because member task instances do
not exist when the launch task executes.
5. Adds unit and regression tests verifying template rendering for built-in
and custom operators, backward compatibility for `context=None`, immutability
of operator attributes across repeated runs, and preservation of unresolved
`workflow_run_metadata` `XComArg` wiring.
Closes: #46563
### Use Case / Motivation
Users submitting Spark jobs, JAR applications, or notebooks via
`DatabricksWorkflowTaskGroup` often parameterize script paths, arguments, or
job configurations with execution date or run metadata (such as `{{ ds }}`).
Rendering member task template fields during workflow JSON creation ensures the
Databricks REST API receives the evaluated parameters.
### Related Issues
Closes: #46563
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Antigravity following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
