moomindani commented on code in PR #74120:
URL: https://github.com/apache/airflow/pull/74120#discussion_r4581715446
##########
providers/databricks/src/airflow/providers/databricks/operators/databricks.py:
##########
@@ -307,6 +307,13 @@ def
_handle_deferrable_databricks_operator_completion(event: dict, log: Logger)
"run_job_task": "job_parameters",
}
+# Per-task parameter slots that the Databricks API rejects when combined with
the dict-shaped field
+# above. ``python_wheel_task.parameters`` (positional CLI arguments) is
mutually exclusive with
+# ``python_wheel_task.named_parameters``, so a task that already carries one
must not receive the other.
+_TASK_PARAM_SLOTS_CONFLICTING_WITH_DICT_PARAM_FIELD = {
+ "python_wheel_task": ("parameters",),
Review Comment:
`run_job_task` has the same conflict: `job_parameters` cannot be combined
with the legacy slots that
`_RUN_NOW_PARAM_SLOTS_CONFLICTING_WITH_JOB_PARAMETERS` already lists. Unlike
the wheel case, `runs/submit` accepts it. I submitted `run_job_task` with
`job_parameters` plus each of `python_params`, `notebook_params`, `jar_params`
and `python_named_params`. All four runs started and then failed with
`INTERNAL_ERROR`: "Failed to run job <id>: INVALID_PARAMETER_VALUE:
job_parameters can not be used in combination with python_params" (and likewise
for the others). So a `run_job_task` with `python_params` breaks as soon as the
Dag gains a param.
Since this map exists for exactly this, could you add
"run_job_task": _RUN_NOW_PARAM_SLOTS_CONFLICTING_WITH_JOB_PARAMETERS,
(moving that tuple above this dict) and a `run_job_task` case to the new
test? There is no `run_job_task` injection test today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]