moomindani commented on issue #46563:
URL: https://github.com/apache/airflow/issues/46563#issuecomment-4212011090
Thanks for the cc @eladkal. I took a look at this.
The reported behavior looks plausible based on the code. The root cause
would be that child task operators inside `DatabricksWorkflowTaskGroup` never
have their template fields rendered before being converted to the Databricks
workflow JSON.
Here's what happens:
1. `_CreateDatabricksWorkflowOperator` (the "launch" task) stores references
to child task operator instances in `tasks_to_convert`.
2. When the launch task executes, Airflow's task runner renders **only the
launch operator's own `template_fields`** before calling `execute()`.
3. Inside `execute()`, `create_workflow_json()` calls
`_convert_to_databricks_workflow_task()` → `_get_task_base_json()` on each
child task — but **no one has called `render_template_fields()` on those child
operators**, so any Jinja expressions (`{{ ds }}`, `{{ params.x }}`, custom
macros, etc.) would be sent to Databricks as raw strings.
Additionally, even if rendering were triggered, some key fields are missing
from `template_fields`:
- `DatabricksTaskOperator.template_fields` does not include `task_config`
- `DatabricksNotebookOperator.template_fields` does not include
`notebook_path`, `source`, or `notebook_packages`
The workarounds shared by @cshaddox and @aldouslpzchr are consistent with
this analysis — they work because `notebook_params` and `extra_job_params` are
rendered as part of the **launch operator's own** `template_fields`, bypassing
the unrendered child tasks.
**To fix this**, `create_workflow_json()` (or
`_convert_to_databricks_workflow_task()`) should call
`render_template_fields(context)` on each child task before extracting the
JSON, and the missing fields above should be added to `template_fields`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
