dabla commented on code in PR #71941: URL: https://github.com/apache/airflow/pull/71941#discussion_r3830888960
########## providers/google/docs/connections/gcp.rst: ########## @@ -393,6 +393,79 @@ Using a quota project affects where API usage is billed, which quotas are applie usage is reported for monitoring and auditing. +.. _howto/connection:google_cloud_platform:corporate_proxy: + +Using the Google Cloud Connection Behind a Corporate Proxy +---------------------------------------------------------- + +If Airflow workers are deployed behind a corporate HTTP proxy, two things are required for +Google API calls to reach the internet. + +**1. Set the standard proxy environment variables on each worker:** + +.. code-block:: bash + + export HTTPS_PROXY=http://<proxy-host>:<port> + export HTTP_PROXY=http://<proxy-host>:<port> + export NO_PROXY=localhost,127.0.0.1,.cluster.local + +In a Kubernetes / Helm deployment add them to ``values.yaml``: + +.. code-block:: yaml + + env: + - name: HTTPS_PROXY + value: "http://<proxy-host>:<port>" + - name: HTTP_PROXY + value: "http://<proxy-host>:<port>" + - name: NO_PROXY + value: "localhost,127.0.0.1,.cluster.local" + +**2. Install the** ``PySocks`` **package in the worker image — this is mandatory.** + +.. code-block:: bash + + pip install pysocks + +Why PySocks is required even for an HTTP proxy +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All Google API endpoints (``oauth2.googleapis.com``, ``bigquery.googleapis.com``, etc.) are +HTTPS. To route an HTTPS request through an HTTP proxy, the client must first open an +``HTTP CONNECT`` tunnel and then do TLS end-to-end inside it. + +The ``httplib2`` library — used by ``google-api-python-client`` and the +``_authorize()`` path of ``GoogleBaseHook`` for services such as BigQuery Jobs API, +Dataflow, Compute, Cloud SQL, Datastore, Cloud Functions, and Marketing Platform — does +**not** implement ``CONNECT`` tunneling itself. It delegates *all* proxying to +`PySocks <https://github.com/Anorov/PySocks>`_ via a ``socks.socksocket``. If PySocks is Review Comment: [nit] The PySocks link targets the unmaintained original repository. https://github.com/Anorov/PySocks is the original repo by the now-inactive upstream author; it is effectively archived. Prefer the PyPI canonical page: https://pypi.org/project/PySocks/ --- Drafted-by: Claude Sonnet 4.6 (claude-sonnet-4.6); no human review before posting ########## providers/google/docs/connections/gcp.rst: ########## @@ -393,6 +393,79 @@ Using a quota project affects where API usage is billed, which quotas are applie usage is reported for monitoring and auditing. +.. _howto/connection:google_cloud_platform:corporate_proxy: + +Using the Google Cloud Connection Behind a Corporate Proxy +---------------------------------------------------------- + +If Airflow workers are deployed behind a corporate HTTP proxy, two things are required for +Google API calls to reach the internet. + +**1. Set the standard proxy environment variables on each worker:** + +.. code-block:: bash + + export HTTPS_PROXY=http://<proxy-host>:<port> + export HTTP_PROXY=http://<proxy-host>:<port> + export NO_PROXY=localhost,127.0.0.1,.cluster.local + +In a Kubernetes / Helm deployment add them to ``values.yaml``: + +.. code-block:: yaml + + env: + - name: HTTPS_PROXY + value: "http://<proxy-host>:<port>" + - name: HTTP_PROXY + value: "http://<proxy-host>:<port>" + - name: NO_PROXY + value: "localhost,127.0.0.1,.cluster.local" + +**2. Install the** ``PySocks`` **package in the worker image — this is mandatory.** + +.. code-block:: bash Review Comment: [warning] The Kubernetes/Helm section shows how to set env vars but the PySocks install instruction (pip install pysocks) is ephemeral in containers. The PR provides a Kubernetes/Helm example for proxy env vars via values.yaml, creating an implicit expectation that PySocks guidance applies equally in that context. However, pip install pysocks inside a running worker container is lost on the next pod restart. For Kubernetes/Helm deployments, users need to either: 1. Build a custom worker image with PySocks pre-installed (RUN pip install pysocks in the Dockerfile). 2. Add it via _PIP_ADDITIONAL_REQUIREMENTS in Helm values. Without this guidance, users in Kubernetes environments may be confused when the proxy works after a manual pip install but silently breaks after a pod restart. --- Drafted-by: Claude Sonnet 4.6 (claude-sonnet-4.6); no human review before posting ########## providers/google/docs/connections/gcp.rst: ########## @@ -393,6 +393,79 @@ Using a quota project affects where API usage is billed, which quotas are applie usage is reported for monitoring and auditing. +.. _howto/connection:google_cloud_platform:corporate_proxy: + +Using the Google Cloud Connection Behind a Corporate Proxy +---------------------------------------------------------- + +If Airflow workers are deployed behind a corporate HTTP proxy, two things are required for +Google API calls to reach the internet. + +**1. Set the standard proxy environment variables on each worker:** + +.. code-block:: bash + + export HTTPS_PROXY=http://<proxy-host>:<port> + export HTTP_PROXY=http://<proxy-host>:<port> + export NO_PROXY=localhost,127.0.0.1,.cluster.local + +In a Kubernetes / Helm deployment add them to ``values.yaml``: + +.. code-block:: yaml + + env: + - name: HTTPS_PROXY + value: "http://<proxy-host>:<port>" + - name: HTTP_PROXY + value: "http://<proxy-host>:<port>" + - name: NO_PROXY + value: "localhost,127.0.0.1,.cluster.local" + +**2. Install the** ``PySocks`` **package in the worker image — this is mandatory.** + +.. code-block:: bash + + pip install pysocks + +Why PySocks is required even for an HTTP proxy +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All Google API endpoints (``oauth2.googleapis.com``, ``bigquery.googleapis.com``, etc.) are +HTTPS. To route an HTTPS request through an HTTP proxy, the client must first open an +``HTTP CONNECT`` tunnel and then do TLS end-to-end inside it. + +The ``httplib2`` library — used by ``google-api-python-client`` and the +``_authorize()`` path of ``GoogleBaseHook`` for services such as BigQuery Jobs API, +Dataflow, Compute, Cloud SQL, Datastore, Cloud Functions, and Marketing Platform — does +**not** implement ``CONNECT`` tunneling itself. It delegates *all* proxying to Review Comment: [nit] The claim that httplib2 does not implement CONNECT tunneling and delegates all proxying to PySocks is slightly imprecise. httplib2 ships with a bundled socks.py but conditionally imports the external socks package at runtime. Its proxy detection path (proxy_info_from_environment()) only activates the CONNECT tunnel when that import succeeds. Without the external PySocks package, httplib2 falls back to a direct socket. More accurate phrasing: httplib2 bundles a minimal socks adapter but relies on the external PySocks package being importable at runtime to activate its HTTP CONNECT tunnel path. Without it, httplib2 silently falls back to a direct connection. --- Drafted-by: Claude Sonnet 4.6 (claude-sonnet-4.6); no human review before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
