This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7932-bf61a206e5bdbaf9f68b7f2041814e4a617df5c3 in repository https://gitbox.apache.org/repos/asf/texera.git
commit c4f5389ebd9182f55674eb329f6e41f3b2a0dc1c Author: Ryan Zhang <[email protected]> AuthorDate: Tue Aug 25 22:25:33 2026 +0000 feat(notebook-migration, single-node): deploy the notebook migration service and JupyterLab (#7932) ### What changes were proposed in this PR? The notebook migration tool shipped its backend service and workspace UI, but nothing deployed them. This PR makes the tool work out of the box in the single-node Docker Compose stack: it publishes images for the service and for the customized JupyterLab server, runs both in the stack, and routes the tool's API traffic through the existing nginx proxy. **`StorageConfig`, `NotebookMigrationResource` (one Jupyter URL becomes two)** - `storage.jupyter.url` splits into `internal-url` and `public-url`. The service calls Jupyter at the internal address (health probes, notebook upload and delete); the iframe URL handed to the browser is built from the public one. A containerized deployment needs both at once, since the in-network service name does not resolve from the browser. - `public-url` defaults to `internal-url`, so native local development and every existing deployment are unchanged. The old key had a single consumer and appeared in no deployment configuration, so no compatibility alias is kept. - The existing note about this service still targeting one Jupyter per process (#7665) is updated to name which URL is which. **`bin/dockerfiles/notebook-migration-service.dockerfile` (new)** - Follows `config-service.dockerfile`, which copies the shared config resources the service reads at startup. Omits the `.git` copy that exists elsewhere for jgit calls from `OPVersion`: that lives in `workflow-core` and `workflow-operator`, which this service does not depend on. **`bin/dockerfiles/jupyter.dockerfile` (new)** - Builds the customized JupyterLab image from the files that already live with the service, so the image can be published rather than built by hand. - Replaces `notebook-migration-service/src/main/resources/Dockerfile` and the standalone `docker-compose.yml` beside it, both now deleted. Every compose file in the repo consumes published images, so the per-service compose file had no remaining role once the image is built by CI. **`build-and-push-images.yml`** - Adds image name mappings for both new dockerfiles. Without them the discovery step falls through to its default and publishes under bare names, missing the `texera-` prefix every other image uses. **`bin/single-node/`** - `docker-compose.yml` runs `notebook-migration-service` and `jupyter`, the latter publishing its port because the browser loads it directly rather than through nginx. - The Jupyter container deliberately does not receive `env_file`. Users execute arbitrary code there, so it gets only the two values it needs instead of the whole environment, which carries the database, S3, lakeFS and LiteLLM credentials. This also keeps `JUPYTER_PORT` out of that container, which the base image would otherwise read and use to move the server off its expected port. - `nginx.conf` routes `/api/notebook-migration/` to the service. Longest prefix matching gives it priority over the `/api/` catch all. - `.env` gains the published port, the Jupyter token, the internal Jupyter address, and the GUI toggle for the tool. Values needing interpolation are set in the compose file instead, matching how lakeFS composes its browser facing presigned endpoint. **Docs** - `bin/single-node/README.md` covers the new port in all three places ports are listed, how to use and disable the tool, and a warning that all users share one JupyterLab server and one token, which is acceptable on a single machine but must not be carried into a multi-user deployment. ### Any related issues, documentation, discussions? Closes #7931 Parent issue #4301 ### How was this PR tested? Added a test in `StorageConfigSpec.scala` pinning that the public Jupyter URL defaults to the internal one, so splitting them stays a no-op outside containerized deployments. The rest of this PR is deployment configuration, which the existing suites do not cover. Existing suites pass: `NotebookMigrationService/test` (42), `Config/testOnly StorageConfigSpec`, and `bin/single-node/tests/test_single_node_sh.sh`. Manually verified by running the full stack with `bin/single-node.sh up`: - All containers reach a healthy state, including both new ones. - Authenticated through nginx, then called `/api/notebook-migration/get-jupyter-url`, which returned the browser reachable address rather than the in-network one. That request also confirms the internal direction, since the endpoint only succeeds after its reachability probe to Jupyter passes. - `POST /api/notebook-migration/set-notebook` through nginx succeeded and the notebook landed inside the Jupyter container. The iframe URL then returned 200 with the expected `frame-ancestors` header naming the Texera origin. - Confirmed no credentials are present in the Jupyter container's environment. - The existing routes (`/api/dataset`, `/api/computing-unit`, `/api/models`, `/api/compile`, `/`) are unaffected. - With an API key configured, a live model call through the tool's LLM path returned successfully. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) --------- Co-authored-by: Meng Wang <[email protected]> --- .github/workflows/build-and-push-images.yml | 6 ++ bin/dockerfiles/jupyter.dockerfile | 45 ++++++++++++ .../notebook-migration-service.dockerfile | 79 ++++++++++++++++++++++ bin/single-node/.env | 10 +++ bin/single-node/README.md | 18 ++++- bin/single-node/docker-compose.yml | 48 +++++++++++++ bin/single-node/nginx.conf | 6 ++ common/config/src/main/resources/storage.conf | 9 ++- .../texera/common/config/StorageConfig.scala | 6 +- .../texera/common/config/StorageConfigSpec.scala | 11 +++ .../src/main/resources/Dockerfile | 32 --------- .../src/main/resources/docker-compose.yml | 44 ------------ .../resource/NotebookMigrationResource.scala | 53 +++++++++------ .../resource/NotebookMigrationResourceSpec.scala | 70 +++++++++++++++++++ 14 files changed, 333 insertions(+), 104 deletions(-) diff --git a/.github/workflows/build-and-push-images.yml b/.github/workflows/build-and-push-images.yml index a4513469dd..cc0d86d8f4 100644 --- a/.github/workflows/build-and-push-images.yml +++ b/.github/workflows/build-and-push-images.yml @@ -279,6 +279,12 @@ jobs: "agent-service") image_name="texera-agent-service" ;; + "notebook-migration-service") + image_name="texera-notebook-migration-service" + ;; + "jupyter") + image_name="texera-jupyter" + ;; *) # Default: use service name as-is image_name="$service" diff --git a/bin/dockerfiles/jupyter.dockerfile b/bin/dockerfiles/jupyter.dockerfile new file mode 100644 index 0000000000..1f911c5de4 --- /dev/null +++ b/bin/dockerfiles/jupyter.dockerfile @@ -0,0 +1,45 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Apache Texera is an effort undergoing incubation at The Apache Software +# Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is +# required of all newly accepted projects until a further review indicates +# that the infrastructure, communications, and decision-making process have +# stabilized in a manner consistent with other successful ASF projects. +# While incubation status is not necessarily a reflection of the +# completeness or stability of the code, it does indicate that the project +# has yet to be fully endorsed by the ASF. + +FROM jupyter/base-notebook:notebook-6.5.4 + +# The customizations live with notebook-migration-service, which owns the Jupyter +# integration. Paths are repo-root relative: the build context is the repo root. +COPY notebook-migration-service/src/main/resources/custom.js /home/jovyan/.jupyter/custom/custom.js +COPY notebook-migration-service/src/main/resources/custom.css /home/jovyan/.jupyter/custom/custom.css +COPY notebook-migration-service/src/main/resources/start-texera-jupyter.sh /usr/local/bin/start-texera-jupyter.sh + +# custom.js must stay writable by jovyan: the startup script substitutes the origin +# placeholder into it at runtime. +USER root +RUN chown -R jovyan:users /home/jovyan/.jupyter && \ + chmod +x /usr/local/bin/start-texera-jupyter.sh + +USER jovyan + +CMD ["start-texera-jupyter.sh"] + +EXPOSE 8888 diff --git a/bin/dockerfiles/notebook-migration-service.dockerfile b/bin/dockerfiles/notebook-migration-service.dockerfile new file mode 100644 index 0000000000..1f45d24ff2 --- /dev/null +++ b/bin/dockerfiles/notebook-migration-service.dockerfile @@ -0,0 +1,79 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Apache Texera is an effort undergoing incubation at The Apache Software +# Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is +# required of all newly accepted projects until a further review indicates +# that the infrastructure, communications, and decision-making process have +# stabilized in a manner consistent with other successful ASF projects. +# While incubation status is not necessarily a reflection of the +# completeness or stability of the code, it does indicate that the project +# has yet to be fully endorsed by the ASF. + +FROM sbtscala/scala-sbt:eclipse-temurin-jammy-17.0.5_8_1.9.3_2.13.11 AS build + +# Set working directory +WORKDIR /texera + +# Copy modules for building the service +COPY common/ common/ +COPY notebook-migration-service/ notebook-migration-service/ +COPY project/ project/ +COPY build.sbt build.sbt +COPY .jvmopts .jvmopts + +# Update system and install dependencies +RUN apt-get update && apt-get install -y \ + netcat \ + unzip \ + libpq-dev \ + && apt-get clean + +COPY LICENSE NOTICE DISCLAIMER ./ +COPY licenses/ licenses/ + +RUN sbt clean NotebookMigrationService/dist + +# Unzip the texera binary +RUN unzip notebook-migration-service/target/universal/notebook-migration-service-*.zip -d target/ + +FROM eclipse-temurin:17-jre-jammy AS runtime + +WORKDIR /texera + +# Copy the built texera binary from the build phase +COPY --from=build /texera/target/notebook-migration-service-* /texera/ +# Copy resources directories from build phase +COPY --from=build /texera/common/config/src/main/resources /texera/common/config/src/main/resources +COPY --from=build /texera/notebook-migration-service/src/main/resources /texera/notebook-migration-service/src/main/resources +# Copy ASF licensing files. LICENSE-binary and NOTICE-binary describe the +# bundled third-party contents of this image and ship as /texera/LICENSE +# and /texera/NOTICE; licenses/ holds the per-license full texts referenced +# by LICENSE-binary. +COPY --from=build /texera/notebook-migration-service/LICENSE-binary /texera/LICENSE +COPY --from=build /texera/notebook-migration-service/NOTICE-binary /texera/NOTICE +COPY --from=build /texera/licenses /texera/licenses +COPY --from=build /texera/DISCLAIMER /texera/ + +RUN groupadd --system --gid 1001 texera \ + && useradd --system --uid 1001 --gid texera --home-dir /texera --no-create-home texera \ + && chown -R texera:texera /texera +USER texera + +CMD ["bin/notebook-migration-service"] + +EXPOSE 9098 diff --git a/bin/single-node/.env b/bin/single-node/.env index a6f631ae56..1f38de1002 100644 --- a/bin/single-node/.env +++ b/bin/single-node/.env @@ -19,6 +19,7 @@ TEXERA_HOST=http://localhost TEXERA_PORT=8080 MINIO_PORT=9000 +JUPYTER_PORT=9100 # Log level for all Texera services (valid values: ERROR, WARN, INFO, DEBUG) TEXERA_SERVICE_LOG_LEVEL=INFO @@ -99,3 +100,12 @@ LLM_ENDPOINT=http://nginx:8080 TEXERA_DASHBOARD_SERVICE_ENDPOINT=http://dashboard-service:8080 WORKFLOW_COMPILING_SERVICE_ENDPOINT=http://workflow-compiling-service:9090 WORKFLOW_EXECUTION_SERVICE_ENDPOINT=http://workflow-runtime-coordinator-service:8085 + +# Notebook migration tool +# Toggles the tool in the GUI; the migration itself needs an LLM key above. +GUI_WORKFLOW_WORKSPACE_PYTHON_NOTEBOOK_MIGRATION_ENABLED=true +# Weak default token so Jupyter is not fully open on its published port. +JUPYTER_TOKEN=texera +# Where notebook-migration-service reaches Jupyter. The browser-facing URL is +# STORAGE_JUPYTER_PUBLIC_URL, set in docker-compose.yml. +STORAGE_JUPYTER_INTERNAL_URL=http://texera-jupyter:8888 diff --git a/bin/single-node/README.md b/bin/single-node/README.md index d9fda11617..17ab8262ca 100644 --- a/bin/single-node/README.md +++ b/bin/single-node/README.md @@ -51,16 +51,17 @@ Docker Compose version v2.23.0-desktop.1 ``` -By default, Texera services require ports **8080** and **9000** to be free. If either port is already in use, the services will fail to start. +By default, Texera services require ports **8080**, **9000**, and **9100** to be free. If any of these ports is already in use, the services will fail to start. On macOS or Linux, run the following commands to check: ``` lsof -i :8080 lsof -i :9000 +lsof -i :9100 ``` -If either command produces output, that port is occupied by another process. You will need to either stop that process or change Texera's port configuration. See [Advanced Settings > Run Texera on other ports](#run-texera-on-other-ports) for instructions. +If any command produces output, that port is occupied by another process. You will need to either stop that process or change Texera's port configuration. See [Advanced Settings > Run Texera on other ports](#run-texera-on-other-ports) for instructions. --- @@ -140,6 +141,14 @@ Once Texera is up, create a new workflow and open the Texera agent panel at the To switch providers or add more LLMs, see [Add more LLMs or providers](#add-more-llms-or-providers). +## Use the notebook migration tool + +The notebook migration tool converts a Jupyter notebook into a Texera workflow. It runs a JupyterLab server alongside Texera (published on port 9100) and embeds it in the workspace. The conversion itself is powered by an LLM, so it needs an API key exactly like [the Texera agent](#enable-the-texera-agent). Without one the tool still appears but the conversion fails with a provider auth error. + +The tool is enabled by default. To turn it off, set `GUI_WORKFLOW_WORKSPACE_PYTHON_NOTEBOOK_MIGRATION_ENABLED=false` in the `.env` file. + +Once Texera is up, go to your workflow list and click the robot button ("AI generate a workflow from a Python notebook"). Upload a `.ipynb` file and pick a model; Texera generates the workflow and opens it. In the workspace, a Jupyter button then appears in the menu bar to expand the notebook alongside the generated workflow. + ## Advanced Settings @@ -151,10 +160,12 @@ All changes below are to the `.env` file in the installation folder, unless othe By default, Texera uses: - Port 8080 for its web service - Port 9000 for its MinIO storage service +- Port 9100 for the JupyterLab service used by the notebook migration tool To change these ports, open the `.env` file and update the corresponding variables: - For the web service port (8080): change `TEXERA_PORT=8080` to your desired port, e.g., `TEXERA_PORT=8081`. - For the MinIO port (9000): change `MINIO_PORT=9000` to your desired port, e.g., `MINIO_PORT=9001`. +- For the JupyterLab port (9100): change `JUPYTER_PORT=9100` to your desired port, e.g., `JUPYTER_PORT=9101`. ### Change the locations of Texera data By default, Docker manages Texera's data locations. To change them to your own locations: @@ -227,11 +238,12 @@ For the full list of supported providers and model IDs, see the [LiteLLM proxy c ### Port conflicts -If Texera fails to start, a common cause is that ports 8080 or 9000 are already in use by another application. Check which ports are occupied: +If Texera fails to start, a common cause is that ports 8080, 9000, or 9100 are already in use by another application. Check which ports are occupied: ``` lsof -i :8080 lsof -i :9000 +lsof -i :9100 ``` Stop the conflicting process, or change Texera's ports following the instructions in [Advanced Settings > Run Texera on other ports](#run-texera-on-other-ports). diff --git a/bin/single-node/docker-compose.yml b/bin/single-node/docker-compose.yml index 2f35752392..de6b45420b 100644 --- a/bin/single-node/docker-compose.yml +++ b/bin/single-node/docker-compose.yml @@ -470,6 +470,53 @@ services: timeout: 3s retries: 10 + # NotebookMigrationService provides endpoints for the Jupyter notebook migration tool + notebook-migration-service: + image: ${IMAGE_REGISTRY:-ghcr.io/apache}/texera-notebook-migration-service:${IMAGE_TAG:-latest} + container_name: notebook-migration-service + restart: unless-stopped + depends_on: + postgres: + condition: service_healthy + env_file: + - .env + environment: + # Interpolation only works in the compose file, not inside .env, so the + # browser-facing URL is composed here (same as lakefs's pre-signed endpoint). + - STORAGE_JUPYTER_PUBLIC_URL=${TEXERA_HOST}:${JUPYTER_PORT:-9100} + healthcheck: + test: ["CMD", "curl", "-sf", "http://localhost:9098/api/healthcheck";] + interval: 5s + timeout: 3s + retries: 10 + + # JupyterLab instance that the notebook migration tool embeds in an iframe. + # Its port is published because the browser loads it directly, not through nginx + # (STORAGE_JUPYTER_PUBLIC_URL); notebook-migration-service reaches it in-network + # via STORAGE_JUPYTER_INTERNAL_URL. + jupyter: + image: ${IMAGE_REGISTRY:-ghcr.io/apache}/texera-jupyter:${IMAGE_TAG:-latest} + container_name: texera-jupyter + restart: unless-stopped + ports: + - "${JUPYTER_PORT:-9100}:8888" + # No env_file here on purpose. Users run arbitrary code in Jupyter, so this + # container gets only the two values it needs, not the whole .env (which holds + # the DB, S3, LakeFS, and LiteLLM credentials). It also keeps JUPYTER_PORT out: + # the base image reads it and would move the server off 8888. + environment: + # Texera app origin for the iframe CSP frame-ancestors and custom.js postMessage checks. + - TEXERA_ORIGIN=${TEXERA_HOST}:${TEXERA_PORT:-8080} + - JUPYTER_TOKEN=${JUPYTER_TOKEN:-texera} + healthcheck: + # /api returns the server version without requiring the token, so it is a + # reliable liveness probe even with auth enabled. + test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8888/api')"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 15s + # Part 3: reverse proxy service for Texera's micro services nginx: image: nginx:alpine @@ -483,6 +530,7 @@ services: - access-control-service - workflow-computing-unit-managing-service - agent-service + - notebook-migration-service volumes: - ./nginx.conf:/etc/nginx/nginx.conf:ro ports: diff --git a/bin/single-node/nginx.conf b/bin/single-node/nginx.conf index c5e500e667..0f091a5cb4 100644 --- a/bin/single-node/nginx.conf +++ b/bin/single-node/nginx.conf @@ -81,6 +81,12 @@ http { proxy_send_timeout 1d; } + location /api/notebook-migration/ { + proxy_pass http://notebook-migration-service:9098; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + } + location /api/pve/ { proxy_pass http://workflow-runtime-coordinator-service:8085; proxy_set_header Host $host; diff --git a/common/config/src/main/resources/storage.conf b/common/config/src/main/resources/storage.conf index b290480843..9af2924901 100644 --- a/common/config/src/main/resources/storage.conf +++ b/common/config/src/main/resources/storage.conf @@ -179,10 +179,13 @@ storage { # Configurations of the JupyterLab service jupyter { - url = "http://localhost:9100"; - url = ${?STORAGE_JUPYTER_URL} + internal-url = "http://localhost:9100"; + internal-url = ${?STORAGE_JUPYTER_INTERNAL_URL} + + public-url = ${storage.jupyter.internal-url} + public-url = ${?STORAGE_JUPYTER_PUBLIC_URL} + # Read from the same JUPYTER_TOKEN env var as the Jupyter container - # (notebook-migration-service/src/main/resources/docker-compose.yml) token = "texera" token = ${?JUPYTER_TOKEN} } diff --git a/common/config/src/main/scala/org/apache/texera/common/config/StorageConfig.scala b/common/config/src/main/scala/org/apache/texera/common/config/StorageConfig.scala index 9a98108e19..e48fe4f84e 100644 --- a/common/config/src/main/scala/org/apache/texera/common/config/StorageConfig.scala +++ b/common/config/src/main/scala/org/apache/texera/common/config/StorageConfig.scala @@ -155,7 +155,9 @@ object StorageConfig { val ENV_S3_AUTH_USERNAME = "STORAGE_S3_AUTH_USERNAME" val ENV_S3_AUTH_PASSWORD = "STORAGE_S3_AUTH_PASSWORD" - // Jupyter - val jupyterURL: String = conf.getString("storage.jupyter.url") + // Jupyter. Internal for server-side calls, public for the browser iframe. Equal + // unless a deployment overrides one. + val jupyterInternalURL: String = conf.getString("storage.jupyter.internal-url") + val jupyterPublicURL: String = conf.getString("storage.jupyter.public-url") val jupyterToken: String = conf.getString("storage.jupyter.token") } diff --git a/common/config/src/test/scala/org/apache/texera/common/config/StorageConfigSpec.scala b/common/config/src/test/scala/org/apache/texera/common/config/StorageConfigSpec.scala index cfc2461ea6..ac34c46764 100644 --- a/common/config/src/test/scala/org/apache/texera/common/config/StorageConfigSpec.scala +++ b/common/config/src/test/scala/org/apache/texera/common/config/StorageConfigSpec.scala @@ -62,4 +62,15 @@ class StorageConfigSpec extends AnyFlatSpec with Matchers { it should "expose the warehouse environment-variable override name" in { StorageConfig.ENV_WAREHOUSE_ENABLED shouldBe "STORAGE_WAREHOUSE_ENABLED" } + + "StorageConfig jupyter settings" should "default the public URL to the internal one" in { + // Keeps the split a no-op outside containerized deployments. + // Only assert when neither env override is set, since either would win otherwise. + if ( + sys.env.get("STORAGE_JUPYTER_INTERNAL_URL").isEmpty && + sys.env.get("STORAGE_JUPYTER_PUBLIC_URL").isEmpty + ) { + StorageConfig.jupyterPublicURL shouldBe StorageConfig.jupyterInternalURL + } + } } diff --git a/notebook-migration-service/src/main/resources/Dockerfile b/notebook-migration-service/src/main/resources/Dockerfile deleted file mode 100644 index bea79793f2..0000000000 --- a/notebook-migration-service/src/main/resources/Dockerfile +++ /dev/null @@ -1,32 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -FROM jupyter/base-notebook:notebook-6.5.4 - -# Copy custom JavaScript/CSS for Jupyter and the startup script -COPY custom.js /home/jovyan/.jupyter/custom/custom.js -COPY custom.css /home/jovyan/.jupyter/custom/custom.css -COPY start-texera-jupyter.sh /usr/local/bin/start-texera-jupyter.sh - -# Ensure correct permissions. custom.js must stay writable by jovyan so the -# startup script can substitute the origin placeholder at runtime. -USER root -RUN mkdir -p /home/jovyan/.jupyter/custom && \ - chown -R jovyan:users /home/jovyan/.jupyter && \ - chmod +x /usr/local/bin/start-texera-jupyter.sh - -USER jovyan diff --git a/notebook-migration-service/src/main/resources/docker-compose.yml b/notebook-migration-service/src/main/resources/docker-compose.yml deleted file mode 100644 index d442a57386..0000000000 --- a/notebook-migration-service/src/main/resources/docker-compose.yml +++ /dev/null @@ -1,44 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -name: texera-jupyter -services: - - jupyter: - build: - context: . - dockerfile: Dockerfile - container_name: texera-jupyter - restart: unless-stopped - ports: - - "9100:8888" - environment: - # Texera app origin, used for the iframe CSP frame-ancestors and the - # postMessage origin checks in custom.js. Override for non-local deployments. - - TEXERA_ORIGIN=http://localhost:4200 - # Weak default token so the server is not fully open. The Texera-side iframe - # URL must pass this through ?token=<value>. - - JUPYTER_TOKEN=texera - command: ["start-texera-jupyter.sh"] - healthcheck: - # /api returns the server version without requiring the token, so it is a - # reliable liveness probe even with auth enabled. - test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8888/api')"] - interval: 10s - timeout: 5s - retries: 5 - start_period: 15s diff --git a/notebook-migration-service/src/main/scala/org/apache/texera/service/resource/NotebookMigrationResource.scala b/notebook-migration-service/src/main/scala/org/apache/texera/service/resource/NotebookMigrationResource.scala index 3aa0c357c9..f048eda3f3 100644 --- a/notebook-migration-service/src/main/scala/org/apache/texera/service/resource/NotebookMigrationResource.scala +++ b/notebook-migration-service/src/main/scala/org/apache/texera/service/resource/NotebookMigrationResource.scala @@ -106,12 +106,20 @@ object NotebookMigrationResource extends LazyLogging { } } - // jupyterUrl and jupyterToken are single process-wide values, so this service still - // targets one Jupyter per process (the per-user-pod model) and must not be deployed as a - // shared global instance yet: every user would get the same Jupyter and the same token. - // Resolving these per user is a later stage of the migration (#7665). - private val jupyterUrl = StorageConfig.jupyterURL - private val jupyterToken = StorageConfig.jupyterToken + // The Jupyter server a request targets. internalUrl is what this service calls, publicUrl + // is what the browser loads; they differ once Jupyter is containerized, since the + // in-network name does not resolve from the browser. Passed per call so the two can be + // made distinct, and so per-user resolution (#7665) can build one of these per uid. + final case class JupyterEndpoints(internalUrl: String, publicUrl: String, token: String) + + // Configured default. Process-wide, so this service still targets one Jupyter per process + // (the per-user-pod model) and must not be deployed as a shared global instance yet: every + // user would get the same Jupyter and token. Per-user resolution is #7665. + private val configuredEndpoints = JupyterEndpoints( + StorageConfig.jupyterInternalURL, + StorageConfig.jupyterPublicURL, + StorageConfig.jupyterToken + ) // Default notebook name used when a request does not specify one, so a param-less // getJupyterIframeURL call reproduces the URL from before this service became stateless. @@ -139,7 +147,10 @@ object NotebookMigrationResource extends LazyLogging { } // Returns the Jupyter iframe reference URL for the given notebook. - def getJupyterIframeURL(notebookName: String): Response = { + def getJupyterIframeURL( + notebookName: String, + jupyter: JupyterEndpoints = configuredEndpoints + ): Response = { // notebookName flows into the returned URL, so validate it the same way setNotebook does: // block path traversal and keep it to a plain .ipynb filename. if (!notebookName.matches("[A-Za-z0-9._-]+\\.ipynb")) { @@ -149,26 +160,28 @@ object NotebookMigrationResource extends LazyLogging { .build() } - if (!isJupyterAvailable(jupyterUrl)) { + if (!isJupyterAvailable(jupyter.internalUrl)) { return jupyterUnavailableResponse } Response - .ok(successUrlJson(s"$jupyterUrl/notebooks/work/$notebookName?token=$jupyterToken")) + .ok( + successUrlJson(s"${jupyter.publicUrl}/notebooks/work/$notebookName?token=${jupyter.token}") + ) .build() } // Returns the URL of Jupyter - def getJupyterURL(): Response = { - if (!isJupyterAvailable(jupyterUrl)) { + def getJupyterURL(jupyter: JupyterEndpoints = configuredEndpoints): Response = { + if (!isJupyterAvailable(jupyter.internalUrl)) { return jupyterUnavailableResponse } - Response.ok(successUrlJson(jupyterUrl)).build() + Response.ok(successUrlJson(jupyter.publicUrl)).build() } // Set the notebook in Jupyter - def setNotebook(body: String): Response = { + def setNotebook(body: String, jupyter: JupyterEndpoints = configuredEndpoints): Response = { var conn: HttpURLConnection = null try { val json = parseBody(body) match { @@ -189,12 +202,12 @@ object NotebookMigrationResource extends LazyLogging { .build() } - if (!isJupyterAvailable(jupyterUrl)) { + if (!isJupyterAvailable(jupyter.internalUrl)) { return jupyterUnavailableResponse } // Construct Jupyter API URL - val apiUrl = s"$jupyterUrl/api/contents/work/$notebookName" + val apiUrl = s"${jupyter.internalUrl}/api/contents/work/$notebookName" val url = new URL(apiUrl) conn = url.openConnection().asInstanceOf[HttpURLConnection] @@ -203,7 +216,7 @@ object NotebookMigrationResource extends LazyLogging { conn.setDoOutput(true) conn.setRequestProperty("Content-Type", "application/json") // The Jupyter Contents API requires authentication; send the configured token. - conn.setRequestProperty("Authorization", s"token $jupyterToken") + conn.setRequestProperty("Authorization", s"token ${jupyter.token}") val requestBody = s""" @@ -258,7 +271,7 @@ object NotebookMigrationResource extends LazyLogging { } // Delete the notebook file from Jupyter's work/ directory: - def deleteNotebook(body: String): Response = { + def deleteNotebook(body: String, jupyter: JupyterEndpoints = configuredEndpoints): Response = { var conn: HttpURLConnection = null try { val json = parseBody(body) match { @@ -277,17 +290,17 @@ object NotebookMigrationResource extends LazyLogging { .build() } - if (!isJupyterAvailable(jupyterUrl)) { + if (!isJupyterAvailable(jupyter.internalUrl)) { return jupyterUnavailableResponse } - val url = new URL(s"$jupyterUrl/api/contents/work/$notebookName") + val url = new URL(s"${jupyter.internalUrl}/api/contents/work/$notebookName") conn = url.openConnection().asInstanceOf[HttpURLConnection] conn.setRequestMethod("DELETE") conn.setConnectTimeout(2000) conn.setReadTimeout(2000) - conn.setRequestProperty("Authorization", s"token $jupyterToken") + conn.setRequestProperty("Authorization", s"token ${jupyter.token}") val status = conn.getResponseCode diff --git a/notebook-migration-service/src/test/scala/org/apache/texera/service/resource/NotebookMigrationResourceSpec.scala b/notebook-migration-service/src/test/scala/org/apache/texera/service/resource/NotebookMigrationResourceSpec.scala index 09a978efcb..10d5829b87 100644 --- a/notebook-migration-service/src/test/scala/org/apache/texera/service/resource/NotebookMigrationResourceSpec.scala +++ b/notebook-migration-service/src/test/scala/org/apache/texera/service/resource/NotebookMigrationResourceSpec.scala @@ -535,6 +535,76 @@ class NotebookMigrationResourceSpec } } + // -- internal vs public Jupyter URL ---------------------------------------- + // The rest of the suite runs with the configured default, where both URLs are + // localhost:9100, so it cannot tell the two apart. These pin the split itself. + + // Reachable stub for what the service dials; an unroutable address for what the browser + // gets. 192.0.2.0/24 is TEST-NET-1 (RFC 5737) and routes nowhere, so a call that wrongly + // dials the public URL fails rather than silently passing. Numeric on purpose: a hostname + // would go through the resolver, which setConnectTimeout does not bound. + private val splitEndpoints = NotebookMigrationResource.JupyterEndpoints( + internalUrl = "http://localhost:9100";, + publicUrl = "http://192.0.2.1:1234";, + token = "texera" + ) + + "the internal/public URL split" should "dial the internal URL and return only the public one" in { + withFakeJupyter(contentsStatus = 201) { + val urlResp = NotebookMigrationResource.getJupyterURL(splitEndpoints) + urlResp.getStatus shouldBe Response.Status.OK.getStatusCode + urlResp.getEntity.toString should include("192.0.2.1:1234") + urlResp.getEntity.toString should not include "localhost:9100" + + val iframe = NotebookMigrationResource.getJupyterIframeURL("notebook.ipynb", splitEndpoints) + iframe.getStatus shouldBe Response.Status.OK.getStatusCode + iframe.getEntity.toString should include("192.0.2.1:1234") + iframe.getEntity.toString should not include "localhost:9100" + } + } + + it should "send the notebook to the internal URL, not the public one" in { + withFakeJupyter(contentsStatus = 201) { + val resp = NotebookMigrationResource.setNotebook( + """{"notebookName": "notebook.ipynb", "notebookData": {"cells": []}}""", + splitEndpoints + ) + // Reaching the stub at all proves the upload used internalUrl: the public one is + // unroutable, so a swap would surface here as a 500. + resp.getStatus shouldBe Response.Status.OK.getStatusCode + lastContentsRequest shouldBe Some(("PUT", "/api/contents/work/notebook.ipynb")) + } + } + + it should "delete against the internal URL, not the public one" in { + withFakeJupyter(contentsStatus = 204) { + val resp = NotebookMigrationResource.deleteNotebook( + """{"notebookName": "notebook.ipynb"}""", + splitEndpoints + ) + resp.getStatus shouldBe Response.Status.OK.getStatusCode + lastContentsRequest shouldBe Some(("DELETE", "/api/contents/work/notebook.ipynb")) + } + } + + it should "report Jupyter unavailable when only the public URL is reachable" in { + // The inverse of the tests above, and the one that catches the fields being swapped: + // the reachability probe must follow internalUrl, so a reachable public URL must not + // rescue an unreachable internal one. + withFakeJupyter(contentsStatus = 201) { + // Port 9 on loopback: refused immediately, so this fails fast and without DNS. + val swapped = NotebookMigrationResource.JupyterEndpoints( + internalUrl = "http://127.0.0.1:9";, + publicUrl = "http://localhost:9100";, + token = "texera" + ) + NotebookMigrationResource.getJupyterURL(swapped).getStatus shouldBe 500 + NotebookMigrationResource + .getJupyterIframeURL("notebook.ipynb", swapped) + .getStatus shouldBe 500 + } + } + it should "build the iframe URL from an explicit notebook name" in { withFakeJupyter(contentsStatus = 201) { val resp = resource.getJupyterIframeURL("other.ipynb", sessionUser(writerUid))
