This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/texera.git
The following commit(s) were added to refs/heads/main by this push:
new bda3400261 feat(notebook-migration, k8s): deploy the notebook
migration tool on Kubernetes (#8073)
bda3400261 is described below
commit bda3400261d38e674086f3b44cfcae169ccc55e5
Author: Ryan Zhang <[email protected]>
AuthorDate: Wed Sep 9 00:05:53 2026 +0000
feat(notebook-migration, k8s): deploy the notebook migration tool on
Kubernetes (#8073)
### What changes were proposed in this PR?
The notebook migration tool runs in the single-node Docker stack (#7931)
and in local development (#7933), but the Helm chart has no
`notebook-migration-service` and no JupyterLab. A Kubernetes deployment
shows the tool in the workspace with nothing serving its endpoints. This
adds both, following the chart's existing convention that orchestrator
services are global and stateful resources are per user.
`notebook-migration-service` becomes a global Deployment alongside the
other services, with a ServiceAccount whose Role is scoped to the
JupyterLab namespace only. Per-user JupyterLab pods live in their own
namespace behind a headless Service, addressed as
`jupyter-<uid>.<service>.<namespace>.svc.cluster.local`, with a
ResourceQuota bounding the pool and an optional prepull DaemonSet. A
NetworkPolicy stops one user's pod reaching another's. One `values.yaml`
switch, `notebookMigrationService.enabled`, gates the Deployment, the
routes, the pool, and the button in the workspace, and it defaults to
off.
**Enabling the tool now requires a token secret**
`notebookMigrationService.jupyterTokenSecret` has no default. Every
user's JupyterLab token is `HMAC(secret, uid)` and `/jupyter/<uid>/` is
deliberately unauthenticated at the gateway, so a secret shipped in this
repo would be public and would let anyone derive any user's token. The
Secret template wraps the value in `required`, so enabling the tool
without one fails at install with a message naming the key rather than
coming up quietly insecure. Generate one with `openssl rand -hex 32`.
`values-development.yaml` carries an explicitly dev scoped value so the
Minikube flow keeps working.
**How browser traffic reaches the right pod**
This is the design decision worth reviewing closely. JupyterLab is
loaded in an iframe and then issues its own requests for assets,
contents and kernel websockets. None of those can carry a Texera token,
and Texera has no session cookie, so the caller cannot be authenticated
per request. Each user's JupyterLab is therefore served under
`/jupyter/<uid>/`, and the gateway's existing ExtAuthz hook resolves
that uid to the pod's recorded address and rewrites `Host`. It reuses
`dynamic-routes`, its `DynamicResolver` backend, and the
`SecurityPolicy` that already forwards `Host`, so no new gateway
machinery is introduced.
That mechanism routes; it does not authorize. What keeps users apart is
the per-user JupyterLab token from #7665, derived from a server held
secret and unguessable. Anyone who can reach the gateway can route to
any user's pod and will get a 403 from JupyterLab without that user's
token. The NetworkPolicy is defence in depth on top, closing the one
case where a genuinely hostile neighbour runs: another user's pod. This
is a weaker posture than a per-request authorization check, and it is
stated plainly in the code rather than left to be inferred.
Resolution fails closed. A uid too large for an `Int`, a missing
registry row, and a recorded address that will not parse all take one
path to 403, never a 500.
**This PR is not Helm only, unlike what #8006 anticipated**
Four backend changes are inseparable from a working deployment:
* JupyterLab serves under a configurable base path, since routing by
path requires it, and the recorded internal address carries that path
because JupyterLab serves every endpoint under it, `/api` included.
* `access-control-service` gains the Jupyter case in `authorize()`. The
uid regex is built from `kubernetes.jupyter-base-url` and the prefix is
quoted, so a base path is matched as a path and not as a pattern.
* JupyterLab is told which origin may embed it. The setting and the
pod's `TEXERA_ORIGIN` arrived with #8032; this PR passes it to
`--NotebookApp.allow_origin`, without which a proxy that rewrites `Host`
makes JupyterLab's own cross origin check fail on cookie authenticated
requests and kernel startup is blocked entirely.
* The browser facing origin is settable directly, not only derived from
a gateway hostname, because a deployment reached by port forward or
NodePort has no hostname to derive from. When it is derived, the scheme
comes from whether a certificate is configured, `gatewayConfig.issuer`
or `tlsSecretName`, because the chart declares both listeners
unconditionally and a hostname alone does not say which one serves.
`jupyterPool.basePath` is honoured by every layer that reads it: the
pod's `base_url`, the recorded internal URL, the public URL template,
the gateway route match, and the regex `access-control-service` matches
on. All five derive from the one value, so the prefix cannot be obeyed
in one place and ignored in another.
**Rebuilt pods and pooled connections**
A Jupyter pod is named after its uid, so a rebuild reuses the hostname
with a new IP. The gateway keeps pooled upstream connections to the old
address for its default idle hour, and a request handed one of those
hangs until the route timeout, because a departed pod IP is unrouted
rather than refused. A BackendTrafficPolicy sets `connectionIdleTimeout`
to 30s on the dynamic route so those connections are retired promptly.
Measured on the cluster below, a rebuild went from 14 failures in 40
requests scattered over minutes to 4, confined to the moment of the
switch. Computing units share this route and are unaffected either way:
their pods are keyed on a never reused cuid, so no hostname of theirs
ever moves.
**Two routes needed explicit timeouts**
Envoy's default request timeout is 15 seconds, which is shorter than two
things this tool does.
`/api/notebook-migration` now carries a 3m timeout. Provisioning waits
for a pod to terminate and then to answer, up to 60s each, so the
default was truncating work the service would have completed: rebuilds
measured 30 seconds and returned 504 to the browser while the pod came
up fine underneath, with an immediate retry succeeding.
The route serving `/api/chat` and `/api/models` takes its timeout from
`gatewayConfig.llmRequestTimeout`, defaulting to 10m to match
`python-notebook-migration-timeout-minutes`. LLM completions routinely
run past 15 seconds, and the upstream call succeeds and is then
discarded, so every conversion failed while burning the API call. This
also fixes the agent chat, which shares the route.
Known limitations, both worth their own issues:
* JupyterLab pods have no persistent volume, so a pod restart empties
`work/`. Notebooks survive in the database but are not re-uploaded
automatically. The same is true of a single-node container restart, so
this is not specific to Kubernetes.
* A dead pod self-heals on the next request, but an idle one is never
reclaimed.
### Any related issues, documentation, discussions?
Closes #8006
Parent issue #4301
Builds on #7665 (PR #8032, merged), which resolves the Jupyter URL and
token per user. Roadmap context is the architectural note on #5258.
### How was this PR tested?
Unit tests pass across the touched modules: `Config/test` (71),
`AccessControlService/test` (68), `NotebookMigrationService/test` (122),
with `scalafmtCheckAll` clean. `helm template` renders with the feature
on and off; with it disabled the chart emits no Jupyter or
notebook-migration objects at all, and with it enabled but no token
secret the render fails as intended.
Deployed and exercised end to end on a local Minikube cluster running
Calico, chosen because Minikube's default CNI creates NetworkPolicy
objects without enforcing them, which would make the isolation commit
look correct while enforcing nothing.
All seven endpoints were driven through the gateway: `get-jupyter-url`,
`get-jupyter-iframe-url`, `set-notebook`, `delete-notebook`, and the
three mapping endpoints. `set-notebook` was confirmed by reading the
file back through Jupyter's own contents API, and `delete-notebook` by
that read then returning 404. Requests with no token, a garbage token,
and an `INACTIVE` role are refused; `../../etc/x.ipynb`, a non `.ipynb`
name, and malformed JSON bodies are all 400.
Per-user separation was tested with two real users. Each got its own
pod, the quota moved from one to two, and the registry held two rows
with uid scoped addresses. Each served token matched an independent HMAC
derivation of that uid, and each token returned 200 on its own pod and
403 on the other. Routing to a uid with no pod, and to a uid that does
not exist, both return 403.
Isolation was tested with an ablation rather than a single observation.
`jupyter-1` cannot reach `jupyter-2` on 8888 by pod IP or by DNS, in
both directions, while the same pod reaches the `texera-dev` namespace
in the same probe, which rules out broken networking. Removing the
policy makes the connection succeed and restoring it blocks again.
Kernel startup was tested the way a browser does it, with a cookie and
an `Origin` header rather than a token query parameter, because a token
authenticated request skips JupyterLab's origin check and would pass
even when the browser path is broken. `POST /api/sessions` returns 201
and the pod logs `Kernel started`. Cell click sync was confirmed in the
browser.
The failure paths were provoked rather than assumed. Writing an
unparseable address into a registry row returns 403 and logs the
`URISyntaxException`, and 200 returns once restored. Deleting a pod
without touching its row causes the next request to log that the pod no
longer accepts its current token, discard the row and provision a
replacement. Setting `jupyterPool.basePath` to `/lab/notebooks` moves
the pod's `base_url`, the recorded URL, the gateway route and the regex
together, and removing `KUBERNETES_JUPYTER_BASE_URL` from
`access-control-service` restores the break, which is what proves the
wiring rather than the default doing the work.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
---------
Co-authored-by: Meng Wang <[email protected]>
---
.../service/resource/AccessControlResource.scala | 54 ++++++++-
.../apache/texera/AccessControlResourceSpec.scala | 130 +++++++++++++++++++++
bin/k8s/templates/base/_helpers.tpl | 6 +
.../access-control-service-deployment.yaml | 5 +
.../config-service/config-service-deployment.yaml | 5 +
.../gateway/gateway-jupyter-traffic-policy.yaml | 40 +++++++
bin/k8s/templates/base/gateway/gateway-routes.yaml | 24 ++++
.../base/jupyter-pool/jupyter-namespace.yaml | 22 ++++
.../base/jupyter-pool/jupyter-network-policy.yaml | 49 ++++++++
.../jupyter-pool/jupyter-prepull-daemonset.yaml | 57 +++++++++
.../base/jupyter-pool/jupyter-resource-quota.yaml | 29 +++++
.../base/jupyter-pool/jupyter-service.yaml | 36 ++++++
.../notebook-migration-service-deployment.yaml | 105 +++++++++++++++++
.../notebook-migration-service-secret.yaml | 30 +++++
...notebook-migration-service-service-account.yaml | 49 ++++++++
.../notebook-migration-service-service.yaml | 31 +++++
bin/k8s/values-development.yaml | 24 ++++
bin/k8s/values.yaml | 62 ++++++++++
common/config/src/main/resources/kubernetes.conf | 6 +
.../texera/common/config/KubernetesConfig.scala | 3 +-
.../common/config/KubernetesConfigSpec.scala | 2 +
.../src/main/resources/start-texera-jupyter.sh | 14 ++-
.../service/util/JupyterKubernetesClient.scala | 12 ++
.../texera/service/util/JupyterProvisioner.scala | 4 +-
.../resource/NotebookMigrationResourceSpec.scala | 30 ++++-
.../service/util/JupyterKubernetesClientSpec.scala | 17 +++
26 files changed, 833 insertions(+), 13 deletions(-)
diff --git
a/access-control-service/src/main/scala/org/apache/texera/service/resource/AccessControlResource.scala
b/access-control-service/src/main/scala/org/apache/texera/service/resource/AccessControlResource.scala
index 792a0dfd8a..2d209be59f 100644
---
a/access-control-service/src/main/scala/org/apache/texera/service/resource/AccessControlResource.scala
+++
b/access-control-service/src/main/scala/org/apache/texera/service/resource/AccessControlResource.scala
@@ -27,12 +27,12 @@ import jakarta.ws.rs.{Consumes, DELETE, GET, POST, PUT,
Path, Produces}
import org.apache.texera.auth.JwtParser.parseToken
import org.apache.texera.auth.SessionUser
import org.apache.texera.auth.util.{ComputingUnitAccess, HeaderField}
-import org.apache.texera.common.config.{GuiConfig, LLMConfig}
+import org.apache.texera.common.config.{GuiConfig, KubernetesConfig, LLMConfig}
import org.apache.texera.dao.SqlServer
import org.apache.texera.dao.jooq.generated.enums.PrivilegeEnum
-import
org.apache.texera.dao.jooq.generated.tables.daos.WorkflowComputingUnitDao
+import org.apache.texera.dao.jooq.generated.tables.daos.{UserJupyterDao,
WorkflowComputingUnitDao}
-import java.net.URLDecoder
+import java.net.{URI, URLDecoder}
import java.nio.charset.StandardCharsets
import java.util.Optional
import scala.jdk.CollectionConverters.{CollectionHasAsScala, MapHasAsScala}
@@ -51,6 +51,18 @@ object AccessControlResource extends LazyLogging {
private val pvePvesCuidPath: Regex =
"""^/?(?:auth/)?(?:api/|wsapi/)?pve/pves/([0-9]+)$""".r
private val pvePackagesCuidPath: Regex =
"""^/?(?:auth/)?(?:api/|wsapi/)?pve/([0-9]+)/[^/]+/packages/.+$""".r
+ // Per-user JupyterLab. The uid is in the path because a browser cannot
attach Texera
+ // credentials to the requests Jupyter's own scripts make, so it is the only
place the
+ // owner can be read from.
+ private val jupyterPath: Regex =
jupyterPathRegex(KubernetesConfig.jupyterBaseUrl)
+
+ // Built from the same setting the gateway route and the pods are rendered
from, so the
+ // prefix cannot be honoured in one place and not another. Quoted: it is a
path, not a
+ // pattern.
+ private[texera] def jupyterPathRegex(basePath: String): Regex = {
+ val prefix = basePath.stripPrefix("/").stripSuffix("/")
+ ("^/?(?:auth/)?" + Regex.quote(prefix) + "/([0-9]+)(?:/.*)?$").r
+ }
/**
* Authorize the request based on the path and headers.
@@ -68,6 +80,7 @@ object AccessControlResource extends LazyLogging {
logger.info(s"Authorizing request for path: $path")
path match {
+ case jupyterPath(uid) => routeToJupyter(uid)
case wsapiWorkflowWebsocket() | apiExecutionsStats() |
apiExecutionsResultExport() |
pveRoute() =>
checkComputingUnitAccess(uriInfo, headers, bodyOpt)
@@ -77,6 +90,41 @@ object AccessControlResource extends LazyLogging {
}
}
+ /**
+ * Resolve which JupyterLab pod a request belongs to. This routes; it does
not authorize.
+ *
+ * Jupyter is loaded in an iframe and then issues its own requests for
assets, contents and
+ * kernel websockets. None of those can carry a Texera token, and there is
no session cookie
+ * to fall back on, so the caller cannot be authenticated per request. What
protects one
+ * user's notebooks from another is the per-user Jupyter token, which is
derived from a
+ * server-held secret and is unguessable; reaching the right pod without it
yields a 403 from
+ * Jupyter itself. Cross-pod traffic is blocked separately by a
NetworkPolicy.
+ */
+ private def routeToJupyter(uid: String): Response = {
+ // Envoy routes on an authority, so the scheme and base path are stripped
off. The parse
+ // stays inside the guard so a malformed row is denied rather than raised
as a 500.
+ val authority =
+ try {
+ val dao = new
UserJupyterDao(SqlServer.getInstance().createDSLContext().configuration())
+ Option(dao.fetchOneByUid(uid.toInt))
+ .map(row => new URI(row.getInternalUrl).getAuthority)
+ .filter(a => a != null && a.nonEmpty)
+ } catch {
+ case e: Exception =>
+ logger.error(s"Failed to resolve the Jupyter registered for user
$uid", e)
+ return Response.status(Response.Status.FORBIDDEN).build()
+ }
+
+ authority match {
+ case Some(host) =>
+ logger.info(s"Routing Jupyter for user $uid to recorded host: $host")
+ Response.ok().header("Host", host).build()
+ case None =>
+ logger.warn(s"Refusing Jupyter for user $uid: no usable Jupyter
address is registered")
+ Response.status(Response.Status.FORBIDDEN).build()
+ }
+ }
+
private def checkComputingUnitAccess(
uriInfo: UriInfo,
headers: HttpHeaders,
diff --git
a/access-control-service/src/test/scala/org/apache/texera/AccessControlResourceSpec.scala
b/access-control-service/src/test/scala/org/apache/texera/AccessControlResourceSpec.scala
index 10ff44db7f..2892a6026a 100644
---
a/access-control-service/src/test/scala/org/apache/texera/AccessControlResourceSpec.scala
+++
b/access-control-service/src/test/scala/org/apache/texera/AccessControlResourceSpec.scala
@@ -27,11 +27,13 @@ import org.apache.texera.dao.jooq.generated.enums.{
WorkflowComputingUnitTypeEnum
}
import org.apache.texera.dao.jooq.generated.tables.daos.{
+ UserJupyterDao,
ComputingUnitUserAccessDao,
UserDao,
WorkflowComputingUnitDao
}
import org.apache.texera.dao.jooq.generated.tables.pojos.{
+ UserJupyter,
ComputingUnitUserAccess,
User,
WorkflowComputingUnit
@@ -73,6 +75,11 @@ class AccessControlResourceSpec
private val testNoAccessRecordedUri: String =
"computing-unit-6.compute-unit-svc.default.svc.cluster.local:7777"
+ // What the provisioner records for a user's Jupyter: scheme, authority and
the base path
+ // the pod serves under. Only the authority may reach Envoy as a Host header.
+ private val testJupyterInternalUrl: String =
+
"http://jupyter-1.jupyter-svc.texera-jupyter-pool.svc.cluster.local:8888/jupyter/1";
+
private val testUser1: User = {
val user = new User()
user.setUid(1)
@@ -190,6 +197,14 @@ class AccessControlResourceSpec
readOnlyAccess.setPrivilege(PrivilegeEnum.READ)
computingUnitOfUserDao.insert(readOnlyAccess)
+ // Per-user Jupyter: user 1 has one registered, user 2 deliberately does
not.
+ val jupyterDao = new UserJupyterDao(getDSLContext.configuration())
+ val jupyter = new UserJupyter()
+ jupyter.setUid(testUser1.getUid)
+ jupyter.setInternalUrl(testJupyterInternalUrl)
+ jupyter.setPublicUrl("https://texera.example.com/jupyter/1";)
+ jupyterDao.insert(jupyter)
+
token = JwtAuth.jwtToken(JwtAuth.jwtClaims(testUser1))
token2 = JwtAuth.jwtToken(JwtAuth.jwtClaims(testUser2))
}
@@ -722,4 +737,119 @@ class AccessControlResourceSpec
response.getStatus shouldBe Response.Status.OK.getStatusCode
response.getHeaderString("Host") shouldBe testRecordedUri
}
+
+ // -- per-user JupyterLab routing
--------------------------------------------
+
+ it should "route a Jupyter request to the pod recorded for the uid in the
path" in {
+ val (uri, headers) =
mockRequest("/jupyter/1/notebooks/work/notebook.ipynb", None)
+ val response = new AccessControlResource().authorizeGet(uri, headers)
+
+ response.getStatus shouldBe Response.Status.OK.getStatusCode
+ // The scheme and base path are stripped: Envoy routes on an authority
alone.
+ response.getHeaderString("Host") shouldBe
+ "jupyter-1.jupyter-svc.texera-jupyter-pool.svc.cluster.local:8888"
+ }
+
+ it should "route Jupyter's own subrequests, which carry no token" in {
+ // The iframe's asset and API calls cannot present Texera credentials, so
routing has to
+ // work without one. The per-user Jupyter token is what authorizes them.
+ val (uri, headers) =
+ mockRequest("/jupyter/1/api/contents", None, authorizationHeader = None)
+ val response = new AccessControlResource().authorizeGet(uri, headers)
+
+ response.getStatus shouldBe Response.Status.OK.getStatusCode
+ response.getHeaderString("Host") should startWith("jupyter-1.")
+ }
+
+ it should "refuse a Jupyter request for a user with none registered" in {
+ val (uri, headers) = mockRequest("/jupyter/2/tree", None)
+ new AccessControlResource()
+ .authorizeGet(uri, headers)
+ .getStatus shouldBe Response.Status.FORBIDDEN.getStatusCode
+ }
+
+ it should "refuse a Jupyter request for a uid that does not exist" in {
+ val (uri, headers) = mockRequest("/jupyter/999999/tree", None)
+ new AccessControlResource()
+ .authorizeGet(uri, headers)
+ .getStatus shouldBe Response.Status.FORBIDDEN.getStatusCode
+ }
+
+ it should "not treat a Jupyter path without a uid as routable" in {
+ // Falls through to the catch-all, which denies.
+ val (uri, headers) = mockRequest("/jupyter/tree", None)
+ new AccessControlResource()
+ .authorizeGet(uri, headers)
+ .getStatus shouldBe Response.Status.FORBIDDEN.getStatusCode
+ }
+
+ it should "route the gateway-relative form of a Jupyter path" in {
+ val (uri, headers) = mockRequest("auth/jupyter/1/tree", None)
+ new AccessControlResource()
+ .authorizeGet(uri, headers)
+ .getStatus shouldBe Response.Status.OK.getStatusCode
+ }
+
+ it should "refuse a Jupyter uid too large to be a user id" in {
+ // The path regex accepts any run of digits, so the conversion to Int has
to be caught
+ // rather than allowed out as a 500.
+ val (uri, headers) = mockRequest("/jupyter/9999999999/tree", None)
+ new AccessControlResource()
+ .authorizeGet(uri, headers)
+ .getStatus shouldBe Response.Status.FORBIDDEN.getStatusCode
+ }
+
+ // Registers an address for testUser2, who otherwise has none, for the rows
that should be
+ // refused rather than routed.
+ private def withRecordedJupyter(internalUrl: String)(check: => Unit): Unit =
{
+ val jupyterDao = new UserJupyterDao(getDSLContext.configuration())
+ val row = new UserJupyter()
+ row.setUid(testUser2.getUid)
+ row.setInternalUrl(internalUrl)
+ row.setPublicUrl(internalUrl)
+ jupyterDao.insert(row)
+ try check
+ finally jupyterDao.deleteById(testUser2.getUid)
+ }
+
+ it should "refuse a Jupyter request whose recorded address has no authority"
in {
+ // Parses cleanly but yields a null authority, which must not become a
Host header of
+ // "null" for Envoy to route on.
+ withRecordedJupyter("jupyter-2-with-no-scheme:8888") {
+ val (uri, headers) = mockRequest(s"/jupyter/${testUser2.getUid}/tree",
None)
+ new AccessControlResource()
+ .authorizeGet(uri, headers)
+ .getStatus shouldBe Response.Status.FORBIDDEN.getStatusCode
+ }
+ }
+
+ it should "build the uid regex from the configured base path" in {
+ val custom = AccessControlResource.jupyterPathRegex("/lab/notebooks")
+ "/lab/notebooks/7/tree" should fullyMatch regex custom
+ "auth/lab/notebooks/7" should fullyMatch regex custom
+ "/jupyter/7/tree" should not(fullyMatch regex custom)
+ }
+
+ it should "tolerate a base path written with surrounding slashes" in {
+ val slashed = AccessControlResource.jupyterPathRegex("/jupyter/")
+ "/jupyter/7/tree" should fullyMatch regex slashed
+ }
+
+ it should "treat the base path as a path, not a pattern" in {
+ // Unquoted, the dot would match any character and route /axb to the /a.b
pool.
+ val dotted = AccessControlResource.jupyterPathRegex("/a.b")
+ "/a.b/7" should fullyMatch regex dotted
+ "/axb/7" should not(fullyMatch regex dotted)
+ }
+
+ it should "refuse a Jupyter request whose recorded address will not parse"
in {
+ // A stray escape throws from the URI constructor. That parse sits inside
the same guard
+ // as the lookup, so this is denied like any other unusable row instead of
raising a 500.
+ withRecordedJupyter("http://jupyter-2:8888/%zz";) {
+ val (uri, headers) = mockRequest(s"/jupyter/${testUser2.getUid}/tree",
None)
+ new AccessControlResource()
+ .authorizeGet(uri, headers)
+ .getStatus shouldBe Response.Status.FORBIDDEN.getStatusCode
+ }
+ }
}
diff --git a/bin/k8s/templates/base/_helpers.tpl
b/bin/k8s/templates/base/_helpers.tpl
index bcb8e33a2d..a4e2b9e130 100644
--- a/bin/k8s/templates/base/_helpers.tpl
+++ b/bin/k8s/templates/base/_helpers.tpl
@@ -77,3 +77,9 @@ values override would let an install quietly hand the
privileged mounter to anot
{{- printf "system:serviceaccount:%s:%s" .Release.Namespace
.Values.accessControlService.serviceAccountName -}}
{{- end -}}
+{{/* Jupyter base path, as exactly one leading slash and no trailing one.
Several places
+append the uid to it, so a bare value would render "http://<origin>jupyter/7".
*/}}
+{{- define "texera.jupyter.basePath" -}}
+{{- printf "/%s" (trimAll "/" .Values.jupyterPool.basePath) -}}
+{{- end -}}
+
diff --git
a/bin/k8s/templates/base/access-control-service/access-control-service-deployment.yaml
b/bin/k8s/templates/base/access-control-service/access-control-service-deployment.yaml
index 85fbbdd5f7..62bb8f3b21 100644
---
a/bin/k8s/templates/base/access-control-service/access-control-service-deployment.yaml
+++
b/bin/k8s/templates/base/access-control-service/access-control-service-deployment.yaml
@@ -55,6 +55,11 @@ spec:
value: {{ .Values.workflowComputingUnitPool.name }}
- name: KUBERNETES_COMPUTE_UNIT_POOL_NAMESPACE
value: {{ .Values.workflowComputingUnitPool.namespace }}
+ {{- if .Values.notebookMigrationService.enabled }}
+ # The uid regex this service routes Jupyter on is built from the
same prefix.
+ - name: KUBERNETES_JUPYTER_BASE_URL
+ value: {{ include "texera.jupyter.basePath" . }}
+ {{- end }}
{{- if .Values.litellm.enabled }}
# LLM gateway used to serve /api/chat and /api/models to the agent
service.
- name: LITELLM_BASE_URL
diff --git
a/bin/k8s/templates/base/config-service/config-service-deployment.yaml
b/bin/k8s/templates/base/config-service/config-service-deployment.yaml
index f0748785c3..d6775483eb 100644
--- a/bin/k8s/templates/base/config-service/config-service-deployment.yaml
+++ b/bin/k8s/templates/base/config-service/config-service-deployment.yaml
@@ -47,6 +47,11 @@ spec:
secretKeyRef:
name: {{ .Release.Name }}-postgresql
key: postgres-password
+ # Shows or hides the notebook migration tool in the workspace.
Derived from the
+ # service's own toggle rather than listed in texeraEnvVars, so
enabling the tool
+ # is one switch instead of two that can disagree.
+ - name: GUI_WORKFLOW_WORKSPACE_PYTHON_NOTEBOOK_MIGRATION_ENABLED
+ value: "{{ .Values.notebookMigrationService.enabled }}"
{{- range .Values.texeraEnvVars }}
- name: {{ .name }}
value: "{{ .value }}"
diff --git a/bin/k8s/templates/base/gateway/gateway-jupyter-traffic-policy.yaml
b/bin/k8s/templates/base/gateway/gateway-jupyter-traffic-policy.yaml
new file mode 100644
index 0000000000..4dd4e809ce
--- /dev/null
+++ b/bin/k8s/templates/base/gateway/gateway-jupyter-traffic-policy.yaml
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Jupyter pods are named after the uid, so a rebuilt pod reuses its
predecessor's hostname
+# with a new IP. The gateway keeps pooled upstream connections to the old
address for the
+# default idle hour, and requests handed one of those hang until the route
timeout, since a
+# departed pod IP is unrouted rather than refused. Retiring idle connections
quickly bounds
+# that window. Computing units on this route are unaffected either way: their
pods are keyed
+# on a never-reused cuid, so no hostname of theirs ever moves.
+#
+# Active requests hold the connection out of idle, so a running kernel's
websocket is not cut.
+{{- if .Values.notebookMigrationService.enabled }}
+apiVersion: gateway.envoyproxy.io/v1alpha1
+kind: BackendTrafficPolicy
+metadata:
+ name: {{ .Release.Name }}-jupyter-traffic-policy
+ namespace: {{ .Release.Namespace }}
+spec:
+ targetRefs:
+ - group: gateway.networking.k8s.io
+ kind: HTTPRoute
+ name: {{ .Release.Name }}-dynamic-routes
+ timeout:
+ http:
+ connectionIdleTimeout: 30s
+{{- end }}
diff --git a/bin/k8s/templates/base/gateway/gateway-routes.yaml
b/bin/k8s/templates/base/gateway/gateway-routes.yaml
index f07a3157a4..eb88a2a09d 100644
--- a/bin/k8s/templates/base/gateway/gateway-routes.yaml
+++ b/bin/k8s/templates/base/gateway/gateway-routes.yaml
@@ -74,6 +74,19 @@ spec:
backendRefs:
- name: config-service-svc
port: 9094
+ {{- if .Values.notebookMigrationService.enabled }}
+ - matches:
+ - path:
+ type: PathPrefix
+ value: /api/notebook-migration
+ # Provisioning waits for a pod to terminate and then to answer, 60s
each, so Envoy's
+ # 15s default cuts off requests the service would have completed.
+ timeouts:
+ request: "3m"
+ backendRefs:
+ - name: {{ .Values.notebookMigrationService.name }}-svc
+ port: {{ .Values.notebookMigrationService.service.port }}
+ {{- end }}
- matches:
- path:
type: PathPrefix
@@ -81,6 +94,10 @@ spec:
- path:
type: PathPrefix
value: /api/chat
+ {{- if and .Values.gatewayConfig .Values.gatewayConfig.llmRequestTimeout
}}
+ timeouts:
+ request: {{ .Values.gatewayConfig.llmRequestTimeout | quote }}
+ {{- end }}
backendRefs:
- name: access-control-service-svc
port: 9096
@@ -135,6 +152,13 @@ spec:
- path:
type: PathPrefix
value: /api/pve
+ {{- if .Values.notebookMigrationService.enabled }}
+ # Per-user JupyterLab. ExtAuthz reads the uid from the path and
rewrites Host to
+ # that user's pod; the per-user Jupyter token is what authorizes the
request.
+ - path:
+ type: PathPrefix
+ value: {{ include "texera.jupyter.basePath" . }}
+ {{- end }}
backendRefs:
- group: gateway.envoyproxy.io
kind: Backend
diff --git a/bin/k8s/templates/base/jupyter-pool/jupyter-namespace.yaml
b/bin/k8s/templates/base/jupyter-pool/jupyter-namespace.yaml
new file mode 100644
index 0000000000..9e228d919e
--- /dev/null
+++ b/bin/k8s/templates/base/jupyter-pool/jupyter-namespace.yaml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+{{- if and .Values.notebookMigrationService.enabled
.Values.jupyterPool.createNamespaces }}
+apiVersion: v1
+kind: Namespace
+metadata:
+ name: {{ .Values.jupyterPool.namespace }}
+{{- end }}
diff --git a/bin/k8s/templates/base/jupyter-pool/jupyter-network-policy.yaml
b/bin/k8s/templates/base/jupyter-pool/jupyter-network-policy.yaml
new file mode 100644
index 0000000000..37d9184957
--- /dev/null
+++ b/bin/k8s/templates/base/jupyter-pool/jupyter-network-policy.yaml
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+{{- if and .Values.notebookMigrationService.enabled
.Values.jupyterPool.networkPolicy.enabled }}
+# Stops one user's JupyterLab from reaching another's, and stops computing
unit pods
+# reaching any of them. Both pools run user code by design and neither has any
reason to
+# call the other, so both are denied. Allowing every other namespace leaves
the real callers
+# working: the notebook migration service, and the Envoy proxy wherever the
gateway
+# installation runs it.
+#
+# Defence in depth, not the authorisation boundary: the per-user Jupyter token
is what stops
+# one user reading another's notebooks. Egress is left alone, since notebooks
legitimately
+# install packages and call out.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+ name: {{ .Values.jupyterPool.name }}-deny-cross-user
+ namespace: {{ .Values.jupyterPool.namespace }}
+spec:
+ podSelector:
+ matchLabels:
+ type: jupyter
+ policyTypes:
+ - Ingress
+ ingress:
+ - from:
+ # kubernetes.io/metadata.name is set automatically on every namespace,
so the two
+ # pools can be named here without labelling anything by hand.
+ - namespaceSelector:
+ matchExpressions:
+ - key: kubernetes.io/metadata.name
+ operator: NotIn
+ values:
+ - {{ .Values.jupyterPool.namespace }}
+ - {{ .Values.workflowComputingUnitPool.namespace }}
+{{- end }}
diff --git a/bin/k8s/templates/base/jupyter-pool/jupyter-prepull-daemonset.yaml
b/bin/k8s/templates/base/jupyter-pool/jupyter-prepull-daemonset.yaml
new file mode 100644
index 0000000000..e6fca48a27
--- /dev/null
+++ b/bin/k8s/templates/base/jupyter-pool/jupyter-prepull-daemonset.yaml
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+{{- if and .Values.notebookMigrationService.enabled
.Values.jupyterPool.prepullImage }}
+# Pulls the JupyterLab image onto every node ahead of time. Pods are created
on demand and
+# the service waits a bounded time for one to answer, so a first-time pull on
a cold node can
+# outlast that wait and the provisioning attempt is discarded. Mirrors the
computing unit
+# pool's prepuller. Set jupyterPool.prepullImage to false to trade cold starts
for one fewer
+# pod per node.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+ name: {{ .Release.Name }}-jupyter-prepuller
+ namespace: {{ .Release.Namespace }}
+ labels:
+ app: {{ .Release.Name }}-jupyter-prepuller
+spec:
+ selector:
+ matchLabels:
+ app: {{ .Release.Name }}-jupyter-prepuller
+ template:
+ metadata:
+ labels:
+ app: {{ .Release.Name }}-jupyter-prepuller
+ spec:
+ restartPolicy: Always
+ tolerations:
+ - operator: "Exists"
+ initContainers:
+ - name: prepuller
+ image: {{ .Values.texera.imageRegistry }}/{{
.Values.jupyterPool.imageName }}:{{ .Values.texera.imageTag }}
+ imagePullPolicy: {{ .Values.texeraImages.pullPolicy }}
+ command: ["sh", "-c", "true"]
+ containers:
+ - name: pause
+ image: gcr.io/google_containers/pause:3.2
+ resources:
+ limits:
+ cpu: 1m
+ memory: 8Mi
+ requests:
+ cpu: 1m
+ memory: 8Mi
+{{- end }}
diff --git a/bin/k8s/templates/base/jupyter-pool/jupyter-resource-quota.yaml
b/bin/k8s/templates/base/jupyter-pool/jupyter-resource-quota.yaml
new file mode 100644
index 0000000000..7e186b18eb
--- /dev/null
+++ b/bin/k8s/templates/base/jupyter-pool/jupyter-resource-quota.yaml
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+{{- if and .Values.notebookMigrationService.enabled
.Values.jupyterPool.createNamespaces .Values.jupyterPool.maxRequestedResources
}}
+# Ceiling for the pool as a whole. Pods are created on demand, one per user,
so without this
+# a busy deployment has no upper bound on what the tool can consume.
+apiVersion: v1
+kind: ResourceQuota
+metadata:
+ name: {{ .Values.jupyterPool.name }}-resource-quota
+ namespace: {{ .Values.jupyterPool.namespace }}
+spec:
+ hard:
+ requests.cpu: "{{ .Values.jupyterPool.maxRequestedResources.cpu }}"
+ requests.memory: {{ .Values.jupyterPool.maxRequestedResources.memory }}
+{{- end }}
diff --git a/bin/k8s/templates/base/jupyter-pool/jupyter-service.yaml
b/bin/k8s/templates/base/jupyter-pool/jupyter-service.yaml
new file mode 100644
index 0000000000..2f0fb1f5e9
--- /dev/null
+++ b/bin/k8s/templates/base/jupyter-pool/jupyter-service.yaml
@@ -0,0 +1,36 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+{{- if .Values.notebookMigrationService.enabled }}
+# Headless, so each user's pod is addressable individually rather than load
balanced across
+# the pool: <pod>.<this service>.<namespace>.svc.cluster.local. The notebook
migration
+# service creates pods whose hostname is the pod name and whose subdomain is
this service's
+# name, which is what makes that address resolve. The selector matches the
"type" label the
+# service stamps on every JupyterLab pod it creates.
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ .Values.jupyterPool.name }}-svc
+ namespace: {{ .Values.jupyterPool.namespace }}
+spec:
+ clusterIP: None
+ selector:
+ type: jupyter
+ ports:
+ - protocol: TCP
+ port: {{ .Values.jupyterPool.service.port }}
+ targetPort: {{ .Values.jupyterPool.service.targetPort }}
+{{- end }}
diff --git
a/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-deployment.yaml
b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-deployment.yaml
new file mode 100644
index 0000000000..2d18f7dc6f
--- /dev/null
+++
b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-deployment.yaml
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+{{- if .Values.notebookMigrationService.enabled }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: {{ .Release.Name }}-{{ .Values.notebookMigrationService.name }}
+ namespace: {{ .Release.Namespace }}
+ labels:
+ app: {{ .Release.Name }}-{{ .Values.notebookMigrationService.name }}
+spec:
+ replicas: {{ .Values.notebookMigrationService.numOfPods | default 1 }}
+ selector:
+ matchLabels:
+ app: {{ .Release.Name }}-{{ .Values.notebookMigrationService.name }}
+ template:
+ metadata:
+ labels:
+ app: {{ .Release.Name }}-{{ .Values.notebookMigrationService.name }}
+ spec:
+ # Needed to create and delete each user's JupyterLab pod in the pool
namespace.
+ serviceAccountName: {{
.Values.notebookMigrationService.serviceAccountName }}
+ containers:
+ - name: {{ .Values.notebookMigrationService.name }}
+ image: {{ .Values.texera.imageRegistry }}/{{
.Values.notebookMigrationService.imageName }}:{{ .Values.texera.imageTag }}
+ imagePullPolicy: {{ .Values.texeraImages.pullPolicy }}
+ ports:
+ - containerPort: {{ .Values.notebookMigrationService.service.port
}}
+ env:
+ - name: STORAGE_JDBC_URL
+ value: jdbc:postgresql://{{ .Release.Name
}}-postgresql:5432/texera_db?currentSchema=texera_db,public
+ - name: STORAGE_JDBC_PASSWORD
+ valueFrom:
+ secretKeyRef:
+ name: {{ .Release.Name }}-postgresql
+ key: postgres-password
+ # Resolve each user's JupyterLab rather than one shared server.
+ - name: KUBERNETES_JUPYTER_ENABLED
+ value: "true"
+ - name: KUBERNETES_JUPYTER_NAMESPACE
+ value: {{ .Values.jupyterPool.namespace }}
+ - name: KUBERNETES_JUPYTER_SERVICE_NAME
+ value: {{ .Values.jupyterPool.name }}-svc
+ - name: KUBERNETES_JUPYTER_IMAGE_NAME
+ value: {{ .Values.texera.imageRegistry }}/{{
.Values.jupyterPool.imageName }}:{{ .Values.texera.imageTag }}
+ - name: KUBERNETES_JUPYTER_CPU_LIMIT
+ value: "{{ .Values.jupyterPool.resources.cpuLimit }}"
+ - name: KUBERNETES_JUPYTER_MEMORY_LIMIT
+ value: {{ .Values.jupyterPool.resources.memoryLimit }}
+ # The pod's own prefix and the browser-facing address are rendered
from one
+ # basePath, so they cannot drift apart.
+ - name: KUBERNETES_JUPYTER_BASE_URL
+ value: {{ include "texera.jupyter.basePath" . }}
+ {{- $origin := .Values.notebookMigrationService.publicOrigin }}
+ {{- if and (not $origin) .Values.gatewayConfig
.Values.gatewayConfig.hostname }}
+ {{- /* Both listeners always render, so only a configured
certificate says
+ which one actually serves. */}}
+ {{- if or .Values.gatewayConfig.issuer
.Values.gatewayConfig.tlsSecretName }}
+ {{- $origin = printf "https://%s"; .Values.gatewayConfig.hostname }}
+ {{- else }}
+ {{- $origin = printf "http://%s"; .Values.gatewayConfig.hostname }}
+ {{- end }}
+ {{- end }}
+ {{- if $origin }}
+ - name: KUBERNETES_JUPYTER_PUBLIC_URL_TEMPLATE
+ value: {{ $origin }}{{ include "texera.jupyter.basePath" .
}}/{uid}
+ - name: KUBERNETES_JUPYTER_TEXERA_ORIGIN
+ value: {{ $origin }}
+ {{- end }}
+ - name: JUPYTER_TOKEN_SECRET
+ valueFrom:
+ secretKeyRef:
+ name: {{ .Release.Name }}-notebook-migration-service-secret
+ key: jupyter-token-secret
+ {{- range .Values.texeraEnvVars }}
+ - name: {{ .name }}
+ value: "{{ .value }}"
+ {{- end }}
+ livenessProbe:
+ httpGet:
+ path: /api/healthcheck
+ port: {{ .Values.notebookMigrationService.service.port }}
+ initialDelaySeconds: 30
+ periodSeconds: 10
+ readinessProbe:
+ httpGet:
+ path: /api/healthcheck
+ port: {{ .Values.notebookMigrationService.service.port }}
+ initialDelaySeconds: 5
+ periodSeconds: 5
+{{- end }}
diff --git
a/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-secret.yaml
b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-secret.yaml
new file mode 100644
index 0000000000..d9c7f8746a
--- /dev/null
+++
b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-secret.yaml
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+{{- if .Values.notebookMigrationService.enabled }}
+# Key the per-user JupyterLab tokens are derived from. Kept in a Secret rather
than the
+# deployment's env list because it is a credential: anyone holding it can
derive any user's
+# token.
+apiVersion: v1
+kind: Secret
+metadata:
+ name: {{ .Release.Name }}-notebook-migration-service-secret
+ namespace: {{ .Release.Namespace }}
+type: Opaque
+{{- $help := "notebookMigrationService.jupyterTokenSecret is required when the
notebook migration tool is enabled: it derives every user's JupyterLab token,
so generate one per deployment, e.g. openssl rand -hex 32" }}
+stringData:
+ jupyter-token-secret: "{{ required $help
.Values.notebookMigrationService.jupyterTokenSecret }}"
+{{- end }}
diff --git
a/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-service-account.yaml
b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-service-account.yaml
new file mode 100644
index 0000000000..1b5d37ec3a
--- /dev/null
+++
b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-service-account.yaml
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+{{- if .Values.notebookMigrationService.enabled }}
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+ name: {{ .Values.notebookMigrationService.serviceAccountName }}
+ namespace: {{ .Release.Namespace }}
+---
+# Scoped to the JupyterLab pool namespace only: the service starts and stops a
user's own
+# notebook server and needs nothing in the release namespace.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+ name: {{ .Values.notebookMigrationService.name }}
+ namespace: {{ .Values.jupyterPool.namespace }}
+rules:
+ - apiGroups: [""]
+ resources: ["pods"]
+ verbs: ["get", "list", "watch", "create", "delete"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+ name: {{ .Values.notebookMigrationService.name }}-binding
+ namespace: {{ .Values.jupyterPool.namespace }}
+subjects:
+ - kind: ServiceAccount
+ name: {{ .Values.notebookMigrationService.serviceAccountName }}
+ namespace: {{ .Release.Namespace }}
+roleRef:
+ kind: Role
+ name: {{ .Values.notebookMigrationService.name }}
+ apiGroup: rbac.authorization.k8s.io
+{{- end }}
diff --git
a/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-service.yaml
b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-service.yaml
new file mode 100644
index 0000000000..d13a2d4799
--- /dev/null
+++
b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-service.yaml
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+{{- if .Values.notebookMigrationService.enabled }}
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ .Values.notebookMigrationService.name }}-svc
+ namespace: {{ .Release.Namespace }}
+spec:
+ type: {{ .Values.notebookMigrationService.service.type }}
+ selector:
+ app: {{ .Release.Name }}-{{ .Values.notebookMigrationService.name }}
+ ports:
+ - protocol: TCP
+ port: {{ .Values.notebookMigrationService.service.port }}
+ targetPort: {{ .Values.notebookMigrationService.service.port }}
+{{- end }}
diff --git a/bin/k8s/values-development.yaml b/bin/k8s/values-development.yaml
index 01fdc2c957..62a3117650 100644
--- a/bin/k8s/values-development.yaml
+++ b/bin/k8s/values-development.yaml
@@ -295,6 +295,30 @@ workflowComputingUnitPool:
port: 8085
targetPort: 8085
+# Notebook migration tool. The base chart ships it off; development turns it
on.
+notebookMigrationService:
+ enabled: true
+ # Reached through a port-forward, so there is no DNS name to derive an
origin from.
+ publicOrigin: "http://localhost:30080";
+ # Development only, and public by virtue of living here. A real deployment
must generate
+ # its own, since this one derives every user's JupyterLab token.
+ jupyterTokenSecret: "development-only-not-a-secret"
+
+# Per-user JupyterLab pods, sized for a single-node development cluster. Only
the values that
+# differ from the base chart are listed, since Helm merges the rest.
+jupyterPool:
+ # The base default of a full CPU and 2Gi per user exhausts a laptop cluster
after two of
+ # them. Limits become requests here, because the pods declare no requests of
their own.
+ resources:
+ cpuLimit: "0.5"
+ memoryLimit: 1Gi
+ # Low enough to actually bind, so a provisioning loop cannot fill the node.
+ maxRequestedResources:
+ cpu: 4
+ memory: 8Gi
+ # One node, so prepulling saves nothing and costs a large image pull at
install time.
+ prepullImage: false
+
texeraEnvVars:
- name: USER_SYS_ADMIN_USERNAME
value: "texera"
diff --git a/bin/k8s/values.yaml b/bin/k8s/values.yaml
index 94f30a5b32..46bb349fca 100644
--- a/bin/k8s/values.yaml
+++ b/bin/k8s/values.yaml
@@ -195,6 +195,61 @@ webserver:
type: ClusterIP
port: 8080
+notebookMigrationService:
+ # Turns the whole notebook migration tool on or off: the service, its route,
the per-user
+ # JupyterLab pool, and the button in the workspace.
+ enabled: false
+ name: notebook-migration-service
+ numOfPods: 1
+ serviceAccountName: notebook-migration-service-service-account
+ imageName: texera-notebook-migration-service
+ service:
+ type: ClusterIP
+ port: 9098
+ # Origin the browser reaches Texera on, used for the JupyterLab iframe URL
and for the
+ # CSP that lets Texera embed it. Required wherever there is no DNS name,
such as a
+ # port-forward or a NodePort. Falls back to the gateway hostname when left
empty.
+ publicOrigin: ""
+ # HMAC key each user's JupyterLab token is derived from. Must stay stable
across restarts
+ # or issued tokens stop matching. Empty on purpose: a default here would be
public, and
+ # the token is all that guards /jupyter/<uid>/. Generate with: openssl rand
-hex 32
+ jupyterTokenSecret: ""
+
+# Per-user JupyterLab pods, the stateful half of the notebook migration tool.
One pod per
+# user, addressed through the headless service below.
+jupyterPool:
+ createNamespaces: true
+ name: texera-jupyter
+ # Note: like the computing unit pool, this namespace can collide when
several Texera
+ # deployments share a cluster.
+ namespace: texera-jupyter-pool
+ imageName: texera-jupyter
+ # Pull the image onto every node ahead of time, so a first-time pull cannot
outlast the
+ # bounded wait for a new pod to answer. Costs one small pod per node.
+ prepullImage: true
+ # Path prefix each user's JupyterLab is served under; the uid is appended,
so user 7 is
+ # served at /jupyter/7/. Drives the pod's base_url, the gateway route and
the uid regex
+ # access-control matches on, so all three move together.
+ basePath: /jupyter
+ # Must match kubernetes.jupyter-port-num, which the service reads from its
own config
+ # and no environment variable overrides.
+ service:
+ port: 8888
+ targetPort: 8888
+ # Per-pod limits.
+ resources:
+ cpuLimit: "1"
+ memoryLimit: 2Gi
+ networkPolicy:
+ # Denies pod-to-pod traffic inside the pool, so one user's JupyterLab
cannot reach
+ # another's. Requires a cluster with a NetworkPolicy controller; without
one the object
+ # is created but not enforced.
+ enabled: true
+ # Ceiling for the pool as a whole.
+ maxRequestedResources:
+ cpu: 50
+ memory: 50Gi
+
workflowComputingUnitManager:
name: workflow-computing-unit-manager
numOfPods: 1
@@ -450,6 +505,13 @@ metrics-server:
gatewayConfig:
# Routes are available at bin/k8s/templates/gateway-routes.yaml
+ # Ceiling for LLM requests through /api/chat and /api/models. Completions
routinely run
+ # for tens of seconds, and Envoy's default route timeout is 15s, which
severs them while
+ # the upstream call is still in flight and then succeeds unseen. Keep this
at or above
+ # GUI_WORKFLOW_WORKSPACE_PYTHON_NOTEBOOK_MIGRATION_TIMEOUT_MINUTES, which is
what the
+ # frontend is prepared to wait.
+ llmRequestTimeout: 10m
+
# The hostname for the Gateway listener (HTTP/HTTPS).
# e.g., "texera.example.com"
hostname: ""
diff --git a/common/config/src/main/resources/kubernetes.conf
b/common/config/src/main/resources/kubernetes.conf
index f423784334..da72b930d3 100644
--- a/common/config/src/main/resources/kubernetes.conf
+++ b/common/config/src/main/resources/kubernetes.conf
@@ -82,6 +82,12 @@ kubernetes {
jupyter-texera-origin = ""
jupyter-texera-origin = ${?KUBERNETES_JUPYTER_TEXERA_ORIGIN}
+ # Prefix each user's Jupyter serves under; the uid is appended, so user 7 is
served at
+ # <prefix>/7/. The gateway reads that uid to pick the pod, so this has to
match the path
+ # component of jupyter-public-url-template.
+ jupyter-base-url = "/jupyter"
+ jupyter-base-url = ${?KUBERNETES_JUPYTER_BASE_URL}
+
jupyter-cpu-limit = "1"
jupyter-cpu-limit = ${?KUBERNETES_JUPYTER_CPU_LIMIT}
diff --git
a/common/config/src/main/scala/org/apache/texera/common/config/KubernetesConfig.scala
b/common/config/src/main/scala/org/apache/texera/common/config/KubernetesConfig.scala
index f52a14fa4e..7cb177c6b6 100644
---
a/common/config/src/main/scala/org/apache/texera/common/config/KubernetesConfig.scala
+++
b/common/config/src/main/scala/org/apache/texera/common/config/KubernetesConfig.scala
@@ -72,9 +72,10 @@ object KubernetesConfig {
val jupyterServiceName: String =
conf.getString("kubernetes.jupyter-service-name")
val jupyterImageName: String =
conf.getString("kubernetes.jupyter-image-name")
val jupyterPortNumber: Int = conf.getInt("kubernetes.jupyter-port-num")
+ val jupyterBaseUrl: String = conf.getString("kubernetes.jupyter-base-url")
+ val jupyterTexeraOrigin: String =
conf.getString("kubernetes.jupyter-texera-origin")
val jupyterCpuLimit: String = conf.getString("kubernetes.jupyter-cpu-limit")
val jupyterMemoryLimit: String =
conf.getString("kubernetes.jupyter-memory-limit")
- val jupyterTexeraOrigin: String =
conf.getString("kubernetes.jupyter-texera-origin")
// Browser-facing address with {uid} substituted; empty means use the
in-network one.
val jupyterPublicUrlTemplate: String =
diff --git
a/common/config/src/test/scala/org/apache/texera/common/config/KubernetesConfigSpec.scala
b/common/config/src/test/scala/org/apache/texera/common/config/KubernetesConfigSpec.scala
index b52568594c..3b3c194b63 100644
---
a/common/config/src/test/scala/org/apache/texera/common/config/KubernetesConfigSpec.scala
+++
b/common/config/src/test/scala/org/apache/texera/common/config/KubernetesConfigSpec.scala
@@ -86,6 +86,8 @@ class KubernetesConfigSpec extends AnyFlatSpec with Matchers {
)
// Empty by default: only a real deployment knows its own origin.
ifUnset("KUBERNETES_JUPYTER_TEXERA_ORIGIN")(KubernetesConfig.jupyterTexeraOrigin
shouldBe "")
+ // A prefix, not a full path: the provisioner appends the uid.
+ ifUnset("KUBERNETES_JUPYTER_BASE_URL")(KubernetesConfig.jupyterBaseUrl
shouldBe "/jupyter")
ifUnset("KUBERNETES_JUPYTER_CPU_LIMIT")(KubernetesConfig.jupyterCpuLimit
shouldBe "1")
ifUnset("KUBERNETES_JUPYTER_MEMORY_LIMIT")(
KubernetesConfig.jupyterMemoryLimit shouldBe "2Gi"
diff --git
a/notebook-migration-service/src/main/resources/start-texera-jupyter.sh
b/notebook-migration-service/src/main/resources/start-texera-jupyter.sh
index 2bfb5a3baf..b38be02604 100644
--- a/notebook-migration-service/src/main/resources/start-texera-jupyter.sh
+++ b/notebook-migration-service/src/main/resources/start-texera-jupyter.sh
@@ -19,14 +19,22 @@
set -euo pipefail
# Texera app origin used by custom.js (postMessage targetOrigin + inbound
origin
-# check) and by the iframe CSP frame-ancestors. Override TEXERA_ORIGIN for
-# deployments under a real hostname; defaults to the local dev origin.
+# check), by the iframe CSP frame-ancestors, and by Jupyter's own cross-origin
check.
+# That last one matters wherever a proxy rewrites the Host header: Jupyter
compares
+# Origin against Host and rejects cookie-authenticated API calls when they
differ, which
+# leaves the notebook without a kernel. Override TEXERA_ORIGIN for deployments
under a
+# real hostname; defaults to the local dev origin.
TEXERA_ORIGIN="${TEXERA_ORIGIN:-http://localhost:4200}";
# Weak default token so the server is not fully open to anyone reachable on the
# published port. The Texera-side iframe URL must pass this through
?token=<value>.
JUPYTER_TOKEN="${JUPYTER_TOKEN:-texera}"
+# Path Jupyter serves under. A deployment that puts every user's Jupyter on
one hostname
+# routes by path, so the server has to know its own prefix. Defaults to "/",
which is what
+# single-node and local dev use.
+JUPYTER_BASE_URL="${JUPYTER_BASE_URL:-/}"
+
# Substitute the origin placeholder in custom.js before the server starts
serving it.
sed -i "s|__TEXERA_ORIGIN__|${TEXERA_ORIGIN}|g"
/home/jovyan/.jupyter/custom/custom.js
@@ -34,5 +42,7 @@ exec start-notebook.sh \
--NotebookApp.token="${JUPYTER_TOKEN}" \
--NotebookApp.password='' \
--NotebookApp.disable_check_xsrf=True \
+ --NotebookApp.allow_origin="${TEXERA_ORIGIN}" \
--NotebookApp.tornado_settings="{'headers': {'Content-Security-Policy':
'frame-ancestors ${TEXERA_ORIGIN}'}}" \
+ --NotebookApp.base_url="${JUPYTER_BASE_URL}" \
--NotebookApp.default_url=/tree
diff --git
a/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterKubernetesClient.scala
b/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterKubernetesClient.scala
index 426912516c..1179de65f5 100644
---
a/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterKubernetesClient.scala
+++
b/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterKubernetesClient.scala
@@ -43,6 +43,14 @@ class JupyterKubernetesClient(client:
io.fabric8.kubernetes.client.KubernetesCli
def generatePodURI(uid: Int): String =
s"${generatePodName(uid)}.${KubernetesConfig.jupyterServiceName}.$namespace.svc.cluster.local:${KubernetesConfig.jupyterPortNumber}"
+ /**
+ * Path a user's Jupyter serves under. The uid is in the path because the
browser cannot
+ * present Texera credentials on the requests Jupyter's own scripts make,
so the gateway
+ * has to read the owner out of the URL instead.
+ */
+ def basePathFor(uid: Int): String =
+
s"/${KubernetesConfig.jupyterBaseUrl.stripPrefix("/").stripSuffix("/")}/$uid"
+
def podExists(uid: Int): Boolean =
getPodByName(generatePodName(uid)).isDefined
def getPodByName(podName: String): Option[Pod] =
@@ -84,6 +92,10 @@ class JupyterKubernetesClient(client:
io.fabric8.kubernetes.client.KubernetesCli
new EnvVarBuilder()
.withName("TEXERA_ORIGIN")
.withValue(KubernetesConfig.jupyterTexeraOrigin)
+ .build(),
+ new EnvVarBuilder()
+ .withName("JUPYTER_BASE_URL")
+ .withValue(s"${basePathFor(uid)}/")
.build()
)
.withResources(resources)
diff --git
a/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala
b/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala
index 2ba3ee0a18..f0b607e5b5 100644
---
a/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala
+++
b/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala
@@ -104,7 +104,9 @@ class JupyterProvisioner(
}
private def provision(uid: Int, token: String): Option[JupyterEndpoints] = {
- val internalUrl = s"http://${kubernetes.generatePodURI(uid)}"
+ // Jupyter serves every endpoint under its base path, /api included, so
the recorded
+ // address has to carry it or each later call lands on a 404.
+ val internalUrl =
s"http://${kubernetes.generatePodURI(uid)}${kubernetes.basePathFor(uid)}"
val endpoints = JupyterEndpoints(internalUrl, publicUrlFor(uid,
internalUrl), token)
try {
createIfAbsent(uid, token)
diff --git
a/notebook-migration-service/src/test/scala/org/apache/texera/service/resource/NotebookMigrationResourceSpec.scala
b/notebook-migration-service/src/test/scala/org/apache/texera/service/resource/NotebookMigrationResourceSpec.scala
index 66cbb4406a..e1c1f7ab8e 100644
---
a/notebook-migration-service/src/test/scala/org/apache/texera/service/resource/NotebookMigrationResourceSpec.scala
+++
b/notebook-migration-service/src/test/scala/org/apache/texera/service/resource/NotebookMigrationResourceSpec.scala
@@ -722,7 +722,9 @@ class NotebookMigrationResourceSpec
kubernetes.created.map(_._1) shouldBe List(writerUid.intValue())
registeredUids() shouldBe List(writerUid)
- result.map(_.internalUrl) shouldBe
Some(s"http://${kubernetes.generatePodURI(writerUid)}")
+ result.map(_.internalUrl) shouldBe Some(
+
s"http://${kubernetes.generatePodURI(writerUid)}${kubernetes.basePathFor(writerUid)}"
+ )
}
it should "give the pod the user's own derived token" in {
@@ -752,7 +754,9 @@ class NotebookMigrationResourceSpec
kubernetes.deleted shouldBe List(writerUid.intValue())
kubernetes.created.map(_._1) shouldBe List(writerUid.intValue())
- result.map(_.internalUrl) shouldBe
Some(s"http://${kubernetes.generatePodURI(writerUid)}")
+ result.map(_.internalUrl) shouldBe Some(
+
s"http://${kubernetes.generatePodURI(writerUid)}${kubernetes.basePathFor(writerUid)}"
+ )
}
it should "register nothing and clean up when the pod never becomes ready"
in {
@@ -765,6 +769,18 @@ class NotebookMigrationResourceSpec
registeredUids() shouldBe empty
}
+ it should "record the user's own base path in the internal address" in {
+ // Jupyter serves /api under its base path too, so an address without the
prefix would
+ // make every later probe and contents call 404.
+ val kubernetes = new StubKubernetes
+ val result = provisionerFor(kubernetes, (_, _) => true)
+ .ensure(writerUid, jupyterEnabled = true, tokenSecret = specSecret)
+
+ result.map(_.internalUrl) shouldBe Some(
+
s"http://${kubernetes.generatePodURI(writerUid)}${kubernetes.basePathFor(writerUid)}"
+ )
+ }
+
it should "build the public URL from the configured template" in {
val kubernetes = new StubKubernetes
val result =
@@ -856,7 +872,7 @@ class NotebookMigrationResourceSpec
kubernetes.deleted shouldBe List(writerUid.intValue())
kubernetes.created.map(_._1) shouldBe List(writerUid.intValue())
result.map(_.internalUrl) shouldBe Some(
- s"http://${kubernetes.generatePodURI(writerUid)}"
+
s"http://${kubernetes.generatePodURI(writerUid)}${kubernetes.basePathFor(writerUid)}"
)
}
@@ -871,7 +887,7 @@ class NotebookMigrationResourceSpec
kubernetes.created.map(_._1) shouldBe List(writerUid.intValue())
result.map(_.internalUrl) shouldBe Some(
- s"http://${kubernetes.generatePodURI(writerUid)}"
+
s"http://${kubernetes.generatePodURI(writerUid)}${kubernetes.basePathFor(writerUid)}"
)
}
@@ -908,7 +924,7 @@ class NotebookMigrationResourceSpec
.ensure(writerUid, jupyterEnabled = true, tokenSecret = specSecret)
result.map(_.internalUrl) shouldBe Some(
- s"http://${kubernetes.generatePodURI(writerUid)}"
+
s"http://${kubernetes.generatePodURI(writerUid)}${kubernetes.basePathFor(writerUid)}"
)
kubernetes.deleted shouldBe empty
registeredUids() shouldBe List(writerUid)
@@ -944,7 +960,9 @@ class NotebookMigrationResourceSpec
.ensure(writerUid, jupyterEnabled = true, tokenSecret = specSecret)
kubernetes.created.map(_._1) shouldBe List(writerUid.intValue())
- result.map(_.internalUrl) shouldBe
Some(s"http://${kubernetes.generatePodURI(writerUid)}")
+ result.map(_.internalUrl) shouldBe Some(
+
s"http://${kubernetes.generatePodURI(writerUid)}${kubernetes.basePathFor(writerUid)}"
+ )
}
it should "report unavailable when registration fails for a reason other
than a race" in {
diff --git
a/notebook-migration-service/src/test/scala/org/apache/texera/service/util/JupyterKubernetesClientSpec.scala
b/notebook-migration-service/src/test/scala/org/apache/texera/service/util/JupyterKubernetesClientSpec.scala
index 624ff3ebc3..e86d8c5668 100644
---
a/notebook-migration-service/src/test/scala/org/apache/texera/service/util/JupyterKubernetesClientSpec.scala
+++
b/notebook-migration-service/src/test/scala/org/apache/texera/service/util/JupyterKubernetesClientSpec.scala
@@ -167,6 +167,23 @@ class JupyterKubernetesClientSpec extends AnyFlatSpec with
Matchers {
Some(KubernetesConfig.jupyterTexeraOrigin)
}
+ it should "tell the pod which base path it serves under" in {
+ // The image passes this to --NotebookApp.base_url. Jupyter wants a
trailing slash, and
+ // without the prefix a path-routed deployment serves every endpoint from
the wrong place.
+ val env = createdPod(7,
"tok").getSpec.getContainers.asScala.head.getEnv.asScala
+ env.find(_.getName == "JUPYTER_BASE_URL").map(_.getValue) shouldBe
Some("/jupyter/7/")
+ }
+
+ "basePathFor" should "put the uid in the path so the gateway can read it" in
{
+ // The browser cannot present Texera credentials on the requests Jupyter's
own scripts
+ // make, so the owner has to be recoverable from the URL alone.
+ bare.basePathFor(7) shouldBe "/jupyter/7"
+ }
+
+ it should "give every user a distinct base path" in {
+ (1 to 50).map(bare.basePathFor).distinct.size shouldBe 50
+ }
+
it should "carry the configured image, pull policy and port" in {
val container = createdPod(7, "tok").getSpec.getContainers.asScala.head
container.getImage shouldBe KubernetesConfig.jupyterImageName
