meel-ayush opened a new pull request, #76828: URL: https://github.com/apache/airflow/pull/76828
### Description In `SQLExecuteQueryOperator`, queries are dispatched directly to remote database drivers without client-side syntactic verification or partition-filter auditing. When workflows contain dialect syntax errors or accidental unbounded scans on large partitioned tables, errors only surface after provisioning expensive database sessions or cloud warehouses. Following Discussion [#76373](https://github.com/apache/airflow/discussions/76373) and the pattern established in `common.ai`, this PR adds opt-in pre-flight SQL validation and partition-filter enforcement powered by `sqlglot`: - Adds an optional `sqlglot` dependency extra (`sqlglot>=30.0.0`) to `providers/common/sql`. - Adds `validate_sql`, `check_partition_filters`, `partition_columns`, and `allow_unbounded` parameters to `SQLExecuteQueryOperator`. - Normalizes SQLAlchemy and hook connection dialects to supported SQLGlot dialects (Postgres, MySQL, Snowflake, BigQuery, T-SQL, Trino, Redshift, etc.). - Normalizes DBAPI parameter formats (`%s`, `%(param)s`) during syntax parsing. - Verifies required partition columns in `WHERE` and `JOIN ... ON` expressions across `SELECT`, `UPDATE`, `DELETE`, and `INSERT INTO ... SELECT` statements while ignoring in-scope CTE aliases. - Lazily imports validation utilities in `SQLExecuteQueryOperator` so non-validating operators and DAG parsing incur zero import overhead. Related Discussion: [#76373](https://github.com/apache/airflow/discussions/76373) ### Tests - Unit and operator tests added in `providers/common/sql/tests/unit/common/sql/utils/test_sql_validation.py` and `providers/common/sql/tests/unit/common/sql/operators/test_sql_execute_query.py` covering multi-dialect syntax validation, partition column enforcement, CTE scoping, DBAPI placeholder handling, and operator hook integration. --- ##### Was generative AI tooling used to co-author this PR? - [ ] Yes (please specify the tool below) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
