arno-fukuda opened a new pull request, #73646: URL: https://github.com/apache/airflow/pull/73646
`CloudSqlProxyRunner` only knows how to run Cloud SQL Auth Proxy v1: the download URLs point at the v1 buckets and the command line uses v1-only flags (`-dir`, `-instances`, `-credential_file`, `-enable_iam_login`). As reported in #72681, v1 fails `caching_sha2_password` full authentication on Cloud SQL for MySQL 8.4, so the first connection after a restart/failover fails with `1045 Access denied`. v1 is also no longer developed. This PR adds opt-in v2 support through a new `sql_proxy_major_version` runner argument and `CloudSQLDatabaseHook` connection extra. The default stays `1`, so existing behaviour does not change. With `sql_proxy_major_version=2`: - The binary is downloaded from `https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy/<version>/cloud-sql-proxy.<os>.<arch>`, which is where Google publishes v2. v2 has no "latest" download, so `sql_proxy_version` (e.g. `v2.14.0`) is required unless `sql_proxy_binary_path` points to an existing binary. - The command line uses v2 flags: `--unix-socket <dir> <instance>`, `--credentials-file`, `--auto-iam-authn`. In TCP mode the hook builds `<instance>?port=<port>` instead of `<instance>=tcp:<port>` and leaves out `--unix-socket`, because v2 rejects the two together. - v2 logs its ready line (`The proxy has started successfully and is ready for new connections!`) to stdout, not stderr. So in v2 mode `start_proxy` merges stderr into stdout and reads from there. The ready-line check is now case-insensitive so it matches both versions. - `get_proxy_version` parses v2's `--version` output (`cloud-sql-proxy version 2.14.0+linux.amd64`). - If `instance_specification` is empty, v2 mode raises an error, because v2 can't forward every instance in a project (no `-projects` equivalent). The connection docs cover the new extra. ### Relation to #72785 This supersedes #72785 by @kokhlo, which had the same idea. Compared with that PR, this one: - downloads from the GCS bucket above. The `cloud-sql-proxy` GitHub releases have no binary assets, so the GitHub URL used in #72785 returns 404. - reads the v2 ready line from stdout. #72785 only read stderr and matched `ready to accept new connections`, which v2 never prints, so `start_proxy` would block. - passes `--credentials-file` (v2 rejects `-credential_file`) when `key_path`/`keyfile_dict` is set. - wires the option through `CloudSQLDatabaseHook`/`CloudSQLExecuteQueryOperator`, including TCP mode. - raises `ValueError` instead of adding new `AirflowException` usages (`check-no-new-airflow-exceptions`). - drops the `airflow-core/newsfragments` entry, since this is a provider-only change. ### Testing - New unit tests cover v2 command lines (unix socket, TCP, IAM), download URL and version validation, the credentials flag, the empty-instance error, `start_proxy` reading the ready line from stdout, `get_proxy_version` parsing, and hook wiring in TCP mode. `test_cloud_sql.py` for both hooks and operators: 188 passed. - `prek run --stage pre-commit` on the changed files passes. Only `check-provider-yaml-valid` was skipped, because it needs breeze and `provider.yaml` is unchanged. `mypy` on the hook module is clean. - Manual check with the real v2.14.0 binary through `CloudSqlProxyRunner` (darwin/arm64, ADC): download, `get_proxy_version()` → `2.14.0+darwin.arm64`, `start_proxy()` in TCP mode returns on the ready line, and in unix-socket mode an invalid instance shows up as the existing `googleapi: Error` failure. - Not tested against a real Cloud SQL instance, so the MySQL 8.4 `caching_sha2_password` behaviour relies on the reproduction in #72681. closes: #72681 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code (Claude Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) I reviewed the generated changes, checked the v2 behaviour described above against the `cloud-sql-proxy` v2 source and the real binary, and ran the tests and static checks listed above. --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
