gaviiin opened a new pull request, #73250: URL: https://github.com/apache/airflow/pull/73250
The Papermill provider supports remote Jupyter kernels only over raw ZMQ/TCP, which requires the kernel's five ZMQ ports to be directly reachable. Deployments based on JupyterHub, Jupyter Kernel Gateway, or Enterprise Gateway instead expose kernels through an HTTPS REST + WebSocket API secured with a token, so their users cannot run notebooks on remote kernels at all. This PR adds a second, parallel "gateway" mode to the Jupyter Kernel connection: - When the connection host is an `http(s)://` URL (or the `use_gateway` extra is set), the hook returns a `GatewayKernelConnection` and the notebook is executed through a new papermill engine built on `jupyter_server`'s gateway machinery (`GatewayKernelManager` / `GatewayKernelClient`), which already implements the kernel REST + WebSocket protocol. - The auth token comes from the connection password (or a `token` extra). Optional extras cover `verify_ssl` (with proper `CERT_NONE` handling for self-signed certificates), `ca_certs`/`client_cert`/`client_key`, `request_timeout`/`connect_timeout`, custom `headers`, and `kernel_id` to attach to a pre-existing kernel instead of starting a new one. - Two gaps in `jupyter_server`'s `GatewayKernelClient.start_channels` (present on `main` and v2.21.0) are worked around in a small subclass: the kernel-channels WebSocket handshake is opened without the authorization/cookie headers that `load_connection_args` applies to REST calls (any token-secured server rejects it with 403), and `validate_cert=False` is not honored for the WebSocket. The subclass routes the WebSocket headers through `load_connection_args` so token renewal and session cookies behave exactly as they do for REST requests. - Kernel lifecycle is managed defensively: kernels started for a run are shut down even when execution or client setup fails (no kernel leaks on the gateway), attached kernels are never shut down by any cleanup path, attaching to an unknown `kernel_id` fails fast instead of silently starting a leaked default-named kernel, and the gateway client singleton is rebuilt per run so one connection's credentials/timeouts cannot leak into another run in the same process. - The existing raw-ZMQ path is unchanged and remains the default for plain hostname/IP connections. Includes unit tests, a system-test example Dag, and documentation for the new connection mode. The one new dependency is `jupyter_server` (BSD-3-Clause, ASF Category A). Verified end-to-end against a local token-secured Jupyter Kernel Gateway over plain HTTP and over self-signed HTTPS, including attach mode and kernel-leak checks. closes: #52893 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (AI-assisted development tooling) Generated with AI assistance following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions); all code was reviewed, statically checked, and tested locally (unit tests plus live end-to-end runs against a token-secured Jupyter Kernel Gateway). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
