1fanwang commented on issue #70953: URL: https://github.com/apache/airflow/issues/70953#issuecomment-5186696602
Verified [Validate DatabricksCopyIntoOperator template fields after rendering](https://github.com/apache/airflow/pull/70339) on `databricks 7.18.1rc1`. Works as intended. Airflow 3.3.0, two identical virtualenvs: one with the RC providers, one with the previous stable release. Same Dag, same command, run through the real task runner. RPC is stubbed, and the stub asserts the value that actually reaches it. A `DatabricksCopyIntoOperator` with `pattern="*.json"` and a templated `files` that renders to empty, which is the "use the pattern unless an explicit file list is passed" case. The stub prints the SQL that reaches the warehouse. ```bash # databricks 7.18.1rc1 AIRFLOW_HOME=$PWD/airflow_home \ AIRFLOW__CORE__DAGS_FOLDER=$PWD/dags_e2e/rc_70339_databricks_copy_into \ .venv/bin/airflow dags test rc_70339_databricks_copy_into # databricks 7.18.0, identical Dag AIRFLOW_HOME=$PWD/airflow_home_prev \ AIRFLOW__CORE__DAGS_FOLDER=$PWD/dags_e2e/rc_70339_databricks_copy_into \ .venv-prev/bin/airflow dags test rc_70339_databricks_copy_into ``` ``` # databricks 7.18.1rc1 RC_CHECK_OK sql reached the warehouse: COPY INTO my_catalog.my_schema.my_table FROM 's3://bucket/data' FILEFORMAT = JSON PATTERN = '*.json' state=success # databricks 7.18.0, no task ever runs Error loading DAG from rc_70339_databricks_copy_into.py: AirflowException: Only one of 'pattern' or 'files' should be specified ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
