ColtenOuO commented on code in PR #71941: URL: https://github.com/apache/airflow/pull/71941#discussion_r3846479621
########## providers/google/docs/connections/gcp.rst: ########## @@ -393,6 +393,95 @@ Using a quota project affects where API usage is billed, which quotas are applie usage is reported for monitoring and auditing. +.. _howto/connection:google_cloud_platform:corporate_proxy: + +Using the Google Cloud Connection Behind a Corporate Proxy +---------------------------------------------------------- + +If Airflow workers are deployed behind a corporate HTTP proxy, two things are required for +Google API calls to reach the internet. + +**1. Set the standard proxy environment variables on each worker:** + +.. code-block:: bash + + export HTTPS_PROXY=http://<proxy-host>:<port> + export HTTP_PROXY=http://<proxy-host>:<port> + export NO_PROXY=localhost,127.0.0.1,.cluster.local + +In a Kubernetes / Helm deployment add them to ``values.yaml``: + +.. code-block:: yaml + + env: + - name: HTTPS_PROXY + value: "http://<proxy-host>:<port>" + - name: HTTP_PROXY + value: "http://<proxy-host>:<port>" + - name: NO_PROXY + value: "localhost,127.0.0.1,.cluster.local" + +**2. Install the** ``PySocks`` **package in the worker image — this is mandatory.** + +For a bare-metal or custom Docker image, add it at build time: + +.. code-block:: dockerfile + + RUN pip install pysocks + +In a Kubernetes / Helm deployment, add it to ``values.yaml`` so it survives pod restarts: + +.. code-block:: yaml + + workers: + extraEnv: + - name: _PIP_ADDITIONAL_REQUIREMENTS + value: "pysocks" + +.. warning:: + Running ``pip install pysocks`` inside a running container is **not** sufficient for + Kubernetes deployments — the package will be lost on the next pod restart. Review Comment: Introducing `_PIP_ADDITIONAL_REQUIREMENTS` in the docs here doesn't seem like a good idea? -- per [Installing additional requirements](https://airflow.apache.org/docs/docker-stack/entrypoint.html#installing-additional-requirements), there are security risks when using this in a production environment. ```suggestion In a Kubernetes / Helm deployment, bake ``pysocks`` into your custom worker image the same way as above, then point the chart at that image in ``values.yaml``: .. code-block:: yaml images: airflow: repository: your-registry/airflow-with-pysocks tag: "3.x.x" ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
