moomindani commented on code in PR #66613:
URL: https://github.com/apache/airflow/pull/66613#discussion_r3258111772
##########
providers/databricks/docs/operators/jobs_create.rst:
##########
@@ -58,6 +58,28 @@ Currently the named parameters that
``DatabricksCreateJobsOperator`` supports ar
- ``access_control_list``
+Forwarding Airflow Dag params as Databricks job parameters
+----------------------------------------------------------
+
+If ``parameters`` is not set in ``json`` and the operator's ``params`` dict is
non-empty,
+the operator's ``params`` are automatically converted to job-level
``parameters`` (a list
+of ``{"name": k, "default": v}`` entries, the shape required by the
+``api/2.2/jobs/create`` endpoint) so that Airflow Dag params can be forwarded
as
+Databricks job parameters without hardcoding them in ``json``. If ``json``
already
+contains ``parameters``, it is left untouched.
+
+.. code-block:: python
+
+ create_job = DatabricksCreateJobsOperator(
+ task_id="create_job",
+ json={"name": "my-job", "tasks": [...]},
+ params={"env": "prod", "batch_size": "100"},
+ )
+ # The created/reset job will have:
+ # parameters=[{"name": "env", "default": "prod"},
+ # {"name": "batch_size", "default": "100"}]
Review Comment:
Thanks @Lee-W — pushed e4c73f3c3a to make the mapping more explicit.
The example tripped people up because `params` is an arbitrary key/value
dict in Airflow, but the Databricks `api/2.2/jobs/create` endpoint expects [a
list of `{"name", "default"}`
objects](https://docs.databricks.com/api/workspace/jobs/create#parameters) — so
the operator has to flip each key into the `name` field and each value into the
`default` field. The updated section now:
- opens with a one-paragraph description of the Databricks API shape and
links to its schema,
- states explicitly that each `<key>: <value>` pair in `params` becomes one
`{"name": <key>, "default": <value>}` entry,
- splits `params` into a named variable in the code example so the key→name
/ value→default mapping reads top-to-bottom.
Also applied the same treatment to `run_now.rst` and `submit_run.rst` for
consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
