boraberke commented on issue #40788: URL: https://github.com/apache/airflow/issues/40788#issuecomment-2230228094
> Previously, before 6.7.0, even though the named parameters were not templated, they were [placed in a templated field named json](https://github.com/boraberke/airflow/blob/68bd42a7ffc3d2828b1ac4102e65944429d34012/airflow/providers/databricks/operators/databricks.py#L810-L829) in the init function. When execute is called, the template field json is resolved. > > In 6.7.0, the change made it so that the named parameters are saved to a non-templated field [overridden_json_params](https://github.com/apache/airflow/blob/providers-databricks/6.7.0/airflow/providers/databricks/operators/databricks.py#L814-L823) to be later used in the execute function in via calling [_setup_and_validate_json](https://github.com/apache/airflow/blob/providers-databricks/6.7.0/airflow/providers/databricks/operators/databricks.py#L841-L847). This means that named parameters that would have been templated are no longer resolved. > > @boraberke Was this an intended change? #40471 intended to solve the issue #35433, which is to fix `json` parameter to be templated as it should be. However, as you stated, named parameters that were implicitly templated (i.e. not in the `template_fields` but merged with `json`) are no longer resolved correctly. This affected all of the below operators: - DatabricksCreateJobsOperator - DatabricksSubmitRunOperator - DatabricksRunNowOperator > I tried using templating json param as expected, however, existing example DAG using named param with templating should not break by this new change. As mentioned [here](https://github.com/apache/airflow/blob/main/airflow/providers/databricks/operators/databricks.py#L643-L656) using only named params instead of json is very common in use. I agree @vatsrahul1001, apparently docs mentioned some of the params, including `job_id` as templated [here](https://github.com/apache/airflow/blob/main/airflow/providers/databricks/operators/databricks.py#L683-L684) but I did not see them before. Adding necessary named params into `template_fields` may be a way to fix it. WDYT @wolfier @vatsrahul1001? Additionally, @potiuk should we revert #40471 or add a new commit that fixes this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
