nivangio opened a new issue, #29931: URL: https://github.com/apache/airflow/issues/29931
### Apache Airflow Provider(s) databricks ### Versions of Apache Airflow Providers apache-airflow-providers-databricks==4.0.0 ### Apache Airflow version 2.4.3 ### Operating System MAC OS Monterrey ### Deployment Virtualenv installation ### Deployment details The issue is consistent across multiple Airflow deployments (locally on Docker Compose, remotely on MWAA in AWS, locally using virualenv) ### What happened This Issue is the continuation of the discussion started here: https://github.com/apache/airflow/issues/29746 `base_parameters` parameter in `DatabricksSubmitRunOperator` is typed as `dict[str,str]` when it actually accepts a broader variety of types in the value section, also supported by `normalise_json_content`. This mistyping causes a contraidctory behaviour where `DatabricksSubmitRunOperator` runs succesfully and as expected but mypy checks fail. ### What you think should happen instead `DatabricksSubmitRunOperator` should definitely accept a broader type variety. Given that `normalise_json_content` is a recursive function, all parameters in `DatabricksSubmitRunOperator` should be of recursable type and probably the type hinting of the function itself needs a change (actually its `content` parameter is untyped ATM) something in the direction of: ```python3 DatabricksSubmitRunOperatorParameter: TypeAlias = Union[bool, int, float, str, Iterable['DatabricksSubmitRunOperatorParameter']] ``` Additionally, if https://github.com/apache/airflow/issues/29746 is accepted it should accept `XComArg` and `XComPlainArg` too ### How to reproduce Mypy checks over this: ```python3 from airflow import DAG from airflow.providers.databricks.operators.databricks import DatabricksSubmitRunOperator with DAG( "dag_erroring", ) as dag: DatabricksSubmitRunOperator( task_id="my_notebook_task", new_cluster={ "cluster_name": "single-node-cluster", "spark_version": "7.6.x-scala2.12", "node_type_id": "i3.xlarge", "num_workers": 0, "spark_conf": { "spark.databricks.cluster.profile": "singleNode", "spark.master": "[*, 4]", }, "custom_tags": {"ResourceClass": "SingleNode"}, }, notebook_task={ "notebook_path": "some/path/to/a/notebook", "base_parameters": {"some_param": "abc1", "some_other_param": "abc2"}, }, libraries=[], databricks_retry_limit=3, timeout_seconds=86400, polling_period_seconds=20, ) ``` returns `dags/my_dag_to_test.py:24: error: Dict entry 1 has incompatible type "str": "Dict[str, str]"; expected "str": "str" [dict-item]` ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
