diederikperdok commented on PR #74120: URL: https://github.com/apache/airflow/pull/74120#issuecomment-6576268358
I'd like to kindly ask the maintainers to please also consider the alternative of completely removing the injection of DAG parameters into Airflow tasks. Then the special case added in this PR won't be needed either. I suspect silently injecting DAG-level parameters into tasks is surprising to users and it makes DAGs brittle: every time a new DAG-level parameter is added it will likely break all Databricks jobs with python wheel tasks. The [issue that prompted this implementation](https://github.com/apache/airflow/issues/39002) provides as justification that it allows to "Dynamically update job parameter in spark job", but that was already possible using templating, for example like so: ```python # Inside a DAG which has at least the parameters "environment" and "batch_size" submit_wheel_task = DatabricksSubmitRunOperator( task_id="submit_wheel_task", databricks_conn_id="databricks_default", run_name="my_wheel_etl_run", existing_cluster_id="1234-567890-abcde123", libraries=[{"whl": "dbfs:/FileStore/wheels/my_package-0.1.0-py3-none-any.whl"}], python_wheel_task={ "package_name": "my_package", "entry_point": "main", "parameters": [ "--env", "{{ params.environment }}", "--batch-size", "{{ params.batch_size }}", ], }, ) ``` In the above example the DAG author has full control over which DAG parameters are dynamically passed on to which Databricks parameters, and no tasks would break if a DAG-level parameter were to be added which should not be passed on to Databricks tasks. I can't find a discussion on the consequence and pro's and cons of silently injecting all DAG-level parameters into tasks; perhaps this bug is a good cause for reconsideration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
