This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7078-f7fb3c1715470ddeb4428c131dd5ba60cd71a5c3 in repository https://gitbox.apache.org/repos/asf/texera.git
commit 7a6ee7cf9198de7b103bacf685b4237b804a6ef4 Author: Yicong Huang <[email protected]> AuthorDate: Thu Aug 13 06:14:43 2026 +0000 feat(local-dev): pick a usable Python and offer to install a missing toolchain (#7078) > **Depends on the Linux-support PR** (`feat(local-dev): support Linux`), which this needs for the platform detection. That branch is this one's parent commit, so it shows up in this diff too — review it there, and read only the second commit here. I'll rebase once it lands. ### What changes were proposed in this PR? Two halves of the same fresh-machine problem. **The interpreter that runs Python UDFs was picked blindly.** `UDF_PYTHON_PATH` defaulted to `command -v python3` at source time — the system interpreter, which almost never has `amber/requirements.txt` installed. Python UDFs then failed at worker launch on a stack of import errors that pointed nowhere near the interpreter choice. The venv AGENTS.md tells contributors to create (`<workspace>/venv312`) was ignored, as was an already-activated `$VIRTUAL_ENV`. Resolution is now lazy, and takes the first candidate that is Python 3.12 **and** can import what a worker needs: ``` $UDF_PYTHON_PATH -> $VIRTUAL_ENV -> <workspace>/venv312 -> pyenv 3.12 -> python3.12 on PATH -> python3 on PATH ``` The choice is reported the way the JDK probe reports `JAVA_HOME`. An explicit `UDF_PYTHON_PATH` still wins — it is the documented override — but now says so when it can't import the deps. "No 3.12 anywhere" and "3.12 without amber's deps" are reported differently, because they need different fixes and the old default silently produced the second one. Lazy so `status` / `logs` / `--help` don't spawn an interpreter per candidate, and never fatal: the JVM services run fine without a Python toolchain, only UDFs don't. **A missing tool was described, never offered.** `_install_hint` printed a suggestion and gave up, so a fresh machine meant hand-installing five or six things, re-running `up` after each to discover the next gap. A missing JDK 17, Node or Python 3.12 is now offered for install, after showing the exact command: | Tool | Installer | | --- | --- | | JDK 17 | distro package manager (`apt-get`/`dnf`/`yum`/`pacman`/`zypper`), `brew` on macOS, SDKMAN when there is none | | Node 24 | `nvm` (bootstraps nvm first if absent) | | Python 3.12 | `pyenv` (bootstraps pyenv first if absent) | `jenv` is not used: it switches between JDKs, it cannot install one. The decision helpers (`_pkg_manager`, `_install_cmd_for`) only ever *print* what would run; a separate step executes it. That keeps the choice unit-testable without installing anything, and it means the user sees the command — `sudo` included — before being asked. Behaviour deliberately preserved for non-interactive callers: **without a TTY nothing is ever prompted** and the old print-a-hint-and-fail path stands, so scripted and CI runs cannot hang on a `read`. `--install-missing` answers yes without asking, `--no-install` only ever prints the hint, and the contradiction is refused rather than silently resolved. docker and sbt stay hint-only — docker needs a daemon, group membership and a re-login, which is not something to do behind a y/n prompt. ### Any related issues, documentation, discussions? Closes #7066 Depends on the Linux-support PR (#7065). ### How was this PR tested? New coverage in the existing `infra`-job suite. Everything goes through the pure decision helpers, so CI asserts on *what would be installed* without installing anything; the Python probes are driven by fake interpreters that echo a version or set an exit code: ``` $ bash bin/local-dev/tests/test_local_dev_sh.sh ... ✓ pkg manager: apt-get → apt-get ✓ pkg manager: dnf only → dnf ✓ pkg manager: pacman only → pacman ✓ pkg manager: apt-get preferred over dnf → apt-get ✓ pkg manager: nothing installed (refuses) ✓ _pkg_manager keeps the Darwin/brew branch ✓ install cmd: java via apt names openjdk-17-jdk ✓ install cmd: java via apt asks for sudo explicitly ✓ install cmd: java via dnf names java-17-openjdk-devel ✓ install cmd: java with no package manager falls back to SDKMAN ✓ install cmd: node uses nvm ✓ install cmd: python uses pyenv ✓ install cmd: python pins 3.12 ✓ install cmd: refuses 'docker' ✓ install cmd: refuses 'sbt' ✓ install cmd: refuses 'definitely-not-a-tool' ✓ consent: refuses without a TTY (never prompts) ✓ consent: TEXERA_INSTALL_MISSING=1 assumes yes ✓ consent: TEXERA_INSTALL_MISSING=0 refuses ✓ python probe: 3.12 accepted ✓ python probe: 3.11 rejected ✓ python probe: missing interpreter rejected ✓ python probe: importable deps accepted ✓ python probe: missing deps rejected ✓ python probe: empty path rejected ✓ python candidates: $VIRTUAL_ENV before sibling venv312 ✓ python candidates: venv312 before bare python3 on PATH ✓ _require_udf_python helper is defined ✓ cmd_up resolves the UDF interpreter before launching ✓ cmd_auto resolves the UDF interpreter before launching ✓ cmd_up_one resolves the UDF interpreter before launching ✓ _require_udf_python points at amber's requirements, not the TUI hint ✓ UDF_PYTHON_PATH no longer defaults to bare python3 at source time ✓ --help documents --install-missing / --no-install ✓ up rejects --install-missing together with --no-install (rc=2) 103 passed, 0 failed ``` ``` $ python -m pytest bin/local-dev/tests/ -q 1 failed, 42 passed ``` `test_is_dirty_after_seed_then_edit` is **pre-existing and unrelated** — it reproduces identically on a pristine `upstream/main` checkout on this machine, and this PR touches neither `tui.py` nor that test. Filed as #7075 and fixed there. Manually, on Ubuntu 24.04.4 with the amber deps installed into a sibling `venv312` and nothing in `UDF_PYTHON_PATH` — it finds the venv rather than the system interpreter the old default would have taken: ``` $ bin/local-dev.sh auto ✓ python: /home/…/Repos/venv312/bin/python (runs Python UDFs) $ command -v python3 # what the old default resolved to /usr/bin/python3 $ /usr/bin/python3 -c "import pyarrow" ModuleNotFoundError: No module named 'pyarrow' ``` The consent path, forced by pinning a version that cannot exist (`TEXERA_PYTHON_VERSION=3.99`). With a TTY it shows the command and asks; the answer here was `n`, and the run continued: ``` ⚠ python: no Python 3.99 found — Python UDFs will not run looked in: $VIRTUAL_ENV, <workspace>/venv312, pyenv, PATH or point at one yourself: export UDF_PYTHON_PATH=/path/to/python3.99 python is missing. I can run: curl -fsSL https://pyenv.run | bash && export PYENV_ROOT="$HOME/.pyenv" && … && pyenv install -s 3.99 Run it now? [y/N] ``` Without a TTY (`</dev/null`) the same run prints the warning, never prompts, does not hang, and exits 0. Both flags are accepted on their own (`auto --no-install`, `auto --install-missing` → rc 0) and rejected together (rc 2). Not exercised end-to-end: actually running the JDK / Node / pyenv installers, since this machine already has all three. The commands themselves are asserted by the tests above; the execution step is a `bash -lc` of exactly the string shown to the user. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) --- bin/local-dev/main.sh | 280 ++++++++++++++++++++++++++++- bin/local-dev/tests/test_local_dev_sh.sh | 292 +++++++++++++++++++++++++++++++ 2 files changed, 569 insertions(+), 3 deletions(-) diff --git a/bin/local-dev/main.sh b/bin/local-dev/main.sh index c342ad8bd0..7ef7a20cb6 100755 --- a/bin/local-dev/main.sh +++ b/bin/local-dev/main.sh @@ -40,6 +40,7 @@ # bin/local-dev.sh up [service] # [--fresh|--build|--skip-build] [--skip=svc1,svc2] [--json] # [--worktree=PATH | --branch=NAME] +# [--install-missing | --no-install] # Default: skip build if no source/lock # changes since last build. --build forces # incremental sbt dist + yarn/bun install. @@ -72,11 +73,22 @@ # branch and run its own local-dev.sh # instead (a warning is printed when such # drift is detected). +# TOOLCHAIN: a missing JDK 17 / Node / +# Python 3.12 is offered for install +# (distro package manager for the JDK, +# nvm for Node, pyenv for Python) after +# asking. --install-missing answers yes +# without asking, --no-install only ever +# prints the hint. Without a TTY the +# prompt is skipped entirely, so scripted +# and CI runs behave as before. Same flags +# on `auto`. # bin/local-dev.sh down [service] [--skip=svc1,svc2] [--json] # stop every non-skipped service, or just # <service> (--json: summary JSON on # stdout). # bin/local-dev.sh auto [--skip=svc1,svc2] [--json] +# [--install-missing | --no-install] # rebuild + bounce only the services whose # source changed since the last build. # bin/local-dev.sh logs <service> tail this service's log. @@ -288,6 +300,115 @@ amap_append() { eval "$_var=\"\${$_var:-}\$_suffix\"" } +# --------- toolchain versions + consented install --------- +# The versions this repo needs (AGENTS.md's table, and the CI matrices). +# Overridable so a contributor can try a newer runtime without editing this. +TEXERA_PYTHON_VERSION="${TEXERA_PYTHON_VERSION:-3.12}" +TEXERA_NODE_VERSION="${TEXERA_NODE_VERSION:-24}" + +# This block sits above the JDK probe on purpose: the probe is the first thing +# that can fail on a fresh machine, and it should be able to offer a fix. That +# also means no colour variables here — those are defined further down, with +# the rest of the TUI helpers. + +# The package manager we'd install with, or non-zero if we don't recognise one. +_pkg_manager() { + case "$(uname -s 2>/dev/null)" in + Darwin) + command -v brew >/dev/null 2>&1 && { printf 'brew\n'; return 0; } + ;; + Linux) + local m="" + for m in apt-get dnf yum pacman zypper; do + command -v "$m" >/dev/null 2>&1 && { printf '%s\n' "$m"; return 0; } + done + ;; + esac + return 1 +} + +# What we would run to install $1. Prints the command and never executes it — +# that split is what lets us show the user the exact command before asking, and +# lets CI assert on the decision without installing anything. +# +# Java prefers the distro package manager: it is system-wide and the distro +# patches it. SDKMAN is the fallback when there's no package manager (or no +# root). Node goes through nvm and Python through pyenv, because the versions +# this repo needs are newer than most distros ship — and both bootstrap +# themselves first if absent. +# +# docker and sbt are deliberately absent. Docker needs a daemon, group +# membership and a re-login; that is not something to do behind a y/n prompt. +_install_cmd_for() { + local tool="${1:-}" pm="" + pm=$(_pkg_manager) || pm="" + case "$tool" in + java) + case "$pm" in + apt-get) printf 'sudo apt-get install -y openjdk-17-jdk\n' ;; + dnf) printf 'sudo dnf install -y java-17-openjdk-devel\n' ;; + yum) printf 'sudo yum install -y java-17-openjdk-devel\n' ;; + pacman) printf 'sudo pacman -S --noconfirm jdk17-openjdk\n' ;; + zypper) printf 'sudo zypper install -y java-17-openjdk-devel\n' ;; + brew) printf 'brew install openjdk@17\n' ;; + *) printf 'curl -s https://get.sdkman.io | bash && sdk install java 17.0.13-tem\n' ;; + esac + ;; + node) + if command -v nvm >/dev/null 2>&1 || [[ -s "${NVM_DIR:-$HOME/.nvm}/nvm.sh" ]]; then + printf 'nvm install %s && nvm alias default %s\n' \ + "$TEXERA_NODE_VERSION" "$TEXERA_NODE_VERSION" + else + printf 'curl -fsSL https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash && . "$HOME/.nvm/nvm.sh" && nvm install %s && nvm alias default %s\n' \ + "$TEXERA_NODE_VERSION" "$TEXERA_NODE_VERSION" + fi + ;; + python) + if command -v pyenv >/dev/null 2>&1; then + printf 'pyenv install -s %s\n' "$TEXERA_PYTHON_VERSION" + else + printf 'curl -fsSL https://pyenv.run | bash && export PYENV_ROOT="$HOME/.pyenv" && export PATH="$PYENV_ROOT/bin:$PATH" && pyenv install -s %s\n' \ + "$TEXERA_PYTHON_VERSION" + fi + ;; + *) return 1 ;; + esac +} + +# Ask before installing anything. Never prompts without a TTY: `up` has to stay +# scriptable, and AGENTS.md points agents at the non-interactive subcommands, so +# a non-interactive run keeps the old behaviour of printing a hint and failing. +# --install-missing / TEXERA_INSTALL_MISSING=1 assume yes +# --no-install / TEXERA_INSTALL_MISSING=0 never install +_consent_to_install() { + local tool="${1:-}" cmd="${2:-}" reply="" + case "${TEXERA_INSTALL_MISSING:-ask}" in + 1|y|yes|true) return 0 ;; + 0|n|no|false) return 1 ;; + esac + [[ -t 0 && -t 1 ]] || return 1 + printf "\n %s is missing. I can run:\n\n" "$tool" >&2 + printf " %s\n\n" "$cmd" >&2 + printf " Run it now? [y/N] " >&2 + read -r reply || return 1 + [[ "$reply" == [Yy]* ]] +} + +# Offer to install $1, then report whether it's actually there afterwards. +# Runs through `bash -lc` so shell functions like nvm / pyenv / sdk resolve. +_offer_install() { + local tool="${1:-}" cmd="" + cmd=$(_install_cmd_for "$tool") || return 1 + _consent_to_install "$tool" "$cmd" || return 1 + printf " installing %s ...\n" "$tool" >&2 + if ! bash -lc "$cmd" >&2; then + printf " %s: install command failed\n" "$tool" >&2 + return 1 + fi + printf " %s: installed\n" "$tool" >&2 + return 0 +} + # --------- toolchain (JDK 17 + node) --------- # Detect a JDK 17 installation rather than pinning one path. We try, in # order: (1) caller-set $JAVA_HOME if it really is 17, (2) macOS's official @@ -423,7 +544,13 @@ _diagnose_jdk17() { echo "" >&2 } -JAVA_HOME_DETECTED="$(_find_jdk17)" || { +JAVA_HOME_DETECTED="$(_find_jdk17)" || JAVA_HOME_DETECTED="" +# Offer to install it rather than only describing how. Declines, and every +# non-interactive run, fall through to the hint-and-exit below unchanged. +if [[ -z "$JAVA_HOME_DETECTED" ]] && _offer_install java; then + JAVA_HOME_DETECTED="$(_find_jdk17)" || JAVA_HOME_DETECTED="" +fi +if [[ -z "$JAVA_HOME_DETECTED" ]]; then echo "FATAL: could not find a JDK 17 install." >&2 _diagnose_jdk17 echo " fix:" >&2 @@ -433,7 +560,7 @@ JAVA_HOME_DETECTED="$(_find_jdk17)" || { echo " or set JAVA_HOME=/path/to/jdk-17 explicitly" >&2 echo "" >&2 exit 1 -} +fi export JAVA_HOME="$JAVA_HOME_DETECTED" export PATH="$JAVA_HOME/bin:$PATH" @@ -573,7 +700,10 @@ export STORAGE_LAKEFS_ENDPOINT="${STORAGE_LAKEFS_ENDPOINT:-http://localhost:8000 export STORAGE_LAKEFS_AUTH_USERNAME="${STORAGE_LAKEFS_AUTH_USERNAME:-AKIAIOSFOLKFSSAMPLES}" export STORAGE_LAKEFS_AUTH_PASSWORD="${STORAGE_LAKEFS_AUTH_PASSWORD:-wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY}" export STORAGE_LAKEFS_AUTH_API_SECRET="${STORAGE_LAKEFS_AUTH_API_SECRET:-random_string_for_lakefs}" -export UDF_PYTHON_PATH="${UDF_PYTHON_PATH:-$(command -v python3 2>/dev/null || command -v python 2>/dev/null)}" +# UDF_PYTHON_PATH is resolved lazily by _require_udf_python (see below) rather +# than defaulted to `command -v python3` here: the system interpreter almost +# never has amber/requirements.txt installed, and picking it silently turned a +# toolchain problem into import errors inside a Python worker. export TEXERA_DASHBOARD_SERVICE_ENDPOINT="${TEXERA_DASHBOARD_SERVICE_ENDPOINT:-http://localhost:8080}"; export WORKFLOW_COMPILING_SERVICE_ENDPOINT="${WORKFLOW_COMPILING_SERVICE_ENDPOINT:-http://localhost:9090}"; export WORKFLOW_EXECUTION_SERVICE_ENDPOINT="${WORKFLOW_EXECUTION_SERVICE_ENDPOINT:-http://localhost:8085}"; @@ -1251,6 +1381,140 @@ _diagnose_node() { printf "\n" } +# --------- python for UDF workers --------- +# "3.12" for an interpreter, or non-zero if it can't be run at all. +_python_version_of() { + local py="${1:-}" + [[ -n "$py" && -x "$py" ]] || return 1 + "$py" -c 'import sys; print("%d.%d" % sys.version_info[:2])' 2>/dev/null +} + +_python_version_ok() { + [[ "$(_python_version_of "${1:-}" 2>/dev/null)" == "$TEXERA_PYTHON_VERSION" ]] +} + +# Can this interpreter actually run a Python UDF worker? pyarrow, pandas and +# loguru are the heavy three from amber/requirements.txt — if they import, the +# environment was populated. +_python_deps_ok() { + local py="${1:-}" + [[ -n "$py" && -x "$py" ]] || return 1 + "$py" -c 'import pyarrow, pandas, loguru' >/dev/null 2>&1 +} + +# Interpreters that might run UDF workers, best first. Prints paths only; +# suitability is the caller's business. Deduplicated, because the sibling-venv +# and pyenv guesses overlap on some layouts. +_udf_python_candidates() { + local seen="" base="" root="" d="" + # Collapse `..` segments so the path we print and export is the one a + # contributor would recognise. `realpath` isn't portable to a stock macOS, + # and only the directory is resolved — the final component stays a symlink, + # which is what we want for a venv's bin/python. + _abspath() { + local dir="" leaf="" + dir=$(dirname "$1"); leaf=$(basename "$1") + [[ -d "$dir" ]] || { printf '%s\n' "$1"; return 0; } + printf '%s/%s\n' "$(cd "$dir" && pwd -P)" "$leaf" + } + _emit() { [[ -n "${1:-}" ]] || return 0; local p=""; p=$(_abspath "$1"); case "$seen" in *"|$p|"*) return 0 ;; esac; seen="$seen|$p|"; printf '%s\n' "$p"; } + # An activated venv is the most explicit statement of intent available. + [[ -n "${VIRTUAL_ENV:-}" ]] && _emit "$VIRTUAL_ENV/bin/python" + # AGENTS.md's layout: <workspace>/{texera, texera-worktrees/<branch>, venv312} + for base in "$REPO_ROOT/.." "$REPO_ROOT/../.." "$SELF_ROOT/.." "$SELF_ROOT/../.."; do + _emit "$base/venv312/bin/python" + done + if command -v pyenv >/dev/null 2>&1; then + root=$(pyenv root 2>/dev/null) || root="$HOME/.pyenv" + while IFS= read -r d; do + [[ -n "$d" ]] && _emit "$d/bin/python" + done < <(shopt -s nullglob; printf '%s\n' "$root/versions/$TEXERA_PYTHON_VERSION"*) + fi + _emit "$(command -v "python$TEXERA_PYTHON_VERSION" 2>/dev/null || true)" + _emit "$(command -v python3 2>/dev/null || true)" +} + +# Lazy resolver, mirroring _require_host_lan_ip: only the subcommands that +# actually launch services pay for it, so `status` / `logs` / `--help` don't +# spawn an interpreter per candidate. +# +# Never fatal. The JVM services run fine without a Python toolchain; only +# Python UDFs don't. So this warns, offers a fix, and carries on. +_require_udf_python() { + [[ -n "${_UDF_PYTHON_RESOLVED:-}" ]] && return 0 + _UDF_PYTHON_RESOLVED=1 + # An explicit UDF_PYTHON_PATH always wins — it's the documented override — + # but say so if it can't import what a worker needs. + if [[ -n "${UDF_PYTHON_PATH:-}" ]]; then + if ! _python_deps_ok "$UDF_PYTHON_PATH"; then + tui_warn "python: UDF_PYTHON_PATH=$UDF_PYTHON_PATH can't import amber's deps" + tui_warn "python: Python UDFs will fail at worker launch" + fi + export UDF_PYTHON_PATH + return 0 + fi + local c="" no_deps="" + while IFS= read -r c; do + [[ -n "$c" ]] || continue + _python_version_ok "$c" || continue + if _python_deps_ok "$c"; then + export UDF_PYTHON_PATH="$c" + tui_ok "python: $c ${DIM}(runs Python UDFs)${RESET}" + return 0 + fi + [[ -z "$no_deps" ]] && no_deps="$c" + done < <(_udf_python_candidates) + # Distinguish "no 3.12 anywhere" from "3.12 without amber's deps": they need + # different fixes, and the old default silently picked the second one. + if [[ -n "$no_deps" ]]; then + tui_warn "python: $no_deps is $TEXERA_PYTHON_VERSION but can't import amber's deps" + printf " ${BOLD}populate it:${RESET}\n" + printf " %s -m pip install -r %s/amber/requirements.txt -r %s/amber/operator-requirements.txt\n" \ + "$no_deps" "$REPO_ROOT" "$REPO_ROOT" + export UDF_PYTHON_PATH="$no_deps" + return 0 + fi + tui_warn "python: no Python $TEXERA_PYTHON_VERSION found — Python UDFs will not run" + # Deliberately not `_install_hint python`: that hint is about the `-i` + # dashboard's textual dependency, which is a different interpreter and a + # different requirements file. + printf " ${BOLD}looked in:${RESET} \$VIRTUAL_ENV, <workspace>/venv312, pyenv, PATH\n" + printf " ${BOLD}or point at one yourself:${RESET} export UDF_PYTHON_PATH=/path/to/python%s\n" \ + "$TEXERA_PYTHON_VERSION" + if _offer_install python; then + while IFS= read -r c; do + [[ -n "$c" ]] || continue + if _python_version_ok "$c"; then + export UDF_PYTHON_PATH="$c" + tui_ok "python: $c" + tui_warn "python: now install amber's deps into it:" + printf " %s -m pip install -r %s/amber/requirements.txt -r %s/amber/operator-requirements.txt\n" \ + "$c" "$REPO_ROOT" "$REPO_ROOT" + return 0 + fi + done < <(_udf_python_candidates) + fi + return 0 +} + +# Translate --install-missing / --no-install into TEXERA_INSTALL_MISSING. The +# flags beat the environment variable; the contradiction is refused rather than +# silently resolved one way. +_set_install_flag() { + local want="" + case "${1:-}" in + --install-missing) want=1 ;; + --no-install) want=0 ;; + *) return 0 ;; + esac + if [[ -n "${_INSTALL_FLAG_SEEN:-}" && "${_INSTALL_FLAG_SEEN}" != "$want" ]]; then + tui_err "--install-missing and --no-install are mutually exclusive" >&2 + exit 2 + fi + _INSTALL_FLAG_SEEN="$want" + export TEXERA_INSTALL_MISSING="$want" +} + # --------- helpers --------- # PID of whatever is listening on a TCP port, or nothing. Always exits 0 — # "empty output" is the contract for "nothing is listening", and callers @@ -2509,6 +2773,7 @@ cmd_up() { --skip-build) BUILD=no ;; --no-build) tui_err "--no-build was renamed to --skip-build" >&2; exit 2 ;; --json) JSON_OUT=true ;; + --install-missing|--no-install) _set_install_flag "$1" ;; # Deploy-target selectors are resolved at startup (they must precede # the build.sbt parse); accept and ignore them here. --worktree=*|--branch=*) selector_seen=true ;; @@ -2541,6 +2806,10 @@ cmd_up() { return $ec fi + # Resolve the interpreter the JVM services hand to Python UDF workers + # before anything is launched with that environment. + _require_udf_python + local n_skip=0 [[ -n "$SKIP_LIST" ]] && n_skip=$(echo "$SKIP_LIST" | tr ',' '\n' | wc -l | tr -d ' ') local skip_label="none" @@ -2701,6 +2970,7 @@ cmd_auto() { case "$1" in --skip=*) SKIP_LIST="${1#--skip=}" ;; --json) JSON_OUT=true ;; + --install-missing|--no-install) _set_install_flag "$1" ;; # Deploy-target selectors are resolved at startup; accept here. --worktree=*|--branch=*) ;; *) tui_err "unknown flag: $1" >&2; exit 2 ;; @@ -2710,6 +2980,8 @@ cmd_auto() { # See cmd_up: human progress to stderr, JSON summary on real stdout (fd 3). if $JSON_OUT; then exec 3>&1 1>&2; fi + _require_udf_python + tui_banner "Texera Local Dev — auto bounce" \ "rebuild + bounce only what changed since last build" if [[ "$REPO_ROOT" != "$SELF_ROOT" ]]; then @@ -3015,6 +3287,8 @@ cmd_up_one() { fi local type="" type=$(amap_get SVC_TYPE "$svc") + # Docker services never launch Python workers; don't pay for the probe. + [[ "$type" == "docker" ]] || _require_udf_python case "$type" in docker) # No source to build — --build degrades to a restart, everything diff --git a/bin/local-dev/tests/test_local_dev_sh.sh b/bin/local-dev/tests/test_local_dev_sh.sh index f92666ab72..75d14af1cc 100755 --- a/bin/local-dev/tests/test_local_dev_sh.sh +++ b/bin/local-dev/tests/test_local_dev_sh.sh @@ -1001,5 +1001,297 @@ else _fail "infra_apply_sql_updates still throws psql stderr away" fi +# -------------------------------------------------------------------------- +# Toolchain detection + consented install (#7066). +# Everything below goes through the *decision* helpers, which are pure: they +# print what would be done and never install anything. That separation is the +# reason this is testable in CI at all. +# -------------------------------------------------------------------------- +_extract_fn() { # $1=function name → its definition, or empty + awk -v fn="$1" 'index($0, fn "()") == 1 {f=1} f{print} f && /^}/{exit}' "$MAIN_SH" +} + +# 38) Which package manager we'd use. On Linux the first of apt-get/dnf/yum/ +# pacman/zypper on PATH wins; with none present it must refuse rather than +# emit a bogus name that a caller would then try to run. +pm_fn=$(_extract_fn _pkg_manager) +if [[ -z "$pm_fn" ]]; then + _fail "_pkg_manager helper missing" +else + _pm_dir=$(mktemp -d 2>/dev/null || mktemp -d -t ldpm) + _pm_check() { # $1=label $2=expected ("" = must refuse) $3..=fake tools + local label="$1" expect="$2"; shift 2 + rm -f "$_pm_dir"/* + local t="" + for t in "$@"; do printf '#!/bin/sh\nexit 0\n' > "$_pm_dir/$t"; chmod +x "$_pm_dir/$t"; done + # PATH is stripped to the fakes so the host's real apt/dnf can't answer, + # but _pkg_manager still needs `uname` to know the platform. + ln -sf "$(command -v uname)" "$_pm_dir/uname" + local got="" rc=0 + got=$(PATH="$_pm_dir"; eval "$pm_fn"; _pkg_manager) || rc=$? + if [[ -z "$expect" ]]; then + if (( rc != 0 )) && [[ -z "$got" ]]; then + _pass "pkg manager: $label (refuses)" + else + _fail "pkg manager: $label should refuse" "rc=$rc got='$got'" + fi + elif [[ "$got" == "$expect" ]]; then + _pass "pkg manager: $label → $got" + else + _fail "pkg manager: $label" "expected '$expect' got '$got'" + fi + } + if [[ "$(uname -s)" == "Linux" ]]; then + _pm_check "apt-get" "apt-get" apt-get + _pm_check "dnf only" "dnf" dnf + _pm_check "pacman only" "pacman" pacman + _pm_check "apt-get preferred over dnf" "apt-get" apt-get dnf + _pm_check "nothing installed" "" + else + _pass "skip: package-manager PATH probing is Linux-only on this host" + fi + rm -rf "$_pm_dir" + if [[ "$pm_fn" == *brew* && "$pm_fn" == *Darwin* ]]; then + _pass "_pkg_manager keeps the Darwin/brew branch" + else + _fail "_pkg_manager lost the Darwin/brew branch" + fi +fi + +# 39) The command we'd run per tool. Java goes through the distro package +# manager first with SDKMAN as the no-sudo fallback; node uses nvm; python +# uses pyenv and must pin 3.12, the version AGENTS.md and the CI matrix +# both specify. Unknown or deliberately-unsupported tools must refuse +# instead of printing something a caller would run. +cmd_fn=$(_extract_fn _install_cmd_for) +if [[ -z "$cmd_fn" || -z "$pm_fn" ]]; then + _fail "_install_cmd_for helper missing" +else + _ic_dir=$(mktemp -d 2>/dev/null || mktemp -d -t ldic) + _ic() { # $1=tool $2..=fake pkg managers on PATH → prints the command + local tool="$1"; shift + rm -f "$_ic_dir"/* + local t="" + for t in "$@"; do printf '#!/bin/sh\nexit 0\n' > "$_ic_dir/$t"; chmod +x "$_ic_dir/$t"; done + ln -sf "$(command -v uname)" "$_ic_dir/uname" + ( PATH="$_ic_dir" + TEXERA_NODE_VERSION=24 TEXERA_PYTHON_VERSION=3.12 + eval "$pm_fn"; eval "$cmd_fn"; _install_cmd_for "$tool" ) 2>/dev/null + } + _ic_expect() { # $1=label $2=needle $3=tool $4..=fake managers + local label="$1" needle="$2" tool="$3"; shift 3 + local got="" + got=$(_ic "$tool" "$@") + if [[ "$got" == *"$needle"* ]]; then + _pass "install cmd: $label" + else + _fail "install cmd: $label" "expected to contain '$needle', got '$got'" + fi + } + # Which manager answers is platform-dependent: _pkg_manager's Darwin branch + # only ever looks for brew, so a faked apt-get on PATH is correctly ignored + # there. Assert each platform's own branch. + if [[ "$(uname -s)" == "Linux" ]]; then + _ic_expect "java via apt names openjdk-17-jdk" "openjdk-17-jdk" java apt-get + _ic_expect "java via apt asks for sudo explicitly" "sudo" java apt-get + _ic_expect "java via dnf names java-17-openjdk-devel" "java-17-openjdk-devel" java dnf + else + _ic_expect "java via brew" "brew install openjdk@17" java brew + fi + # Platform-independent: with no manager at all we fall back to SDKMAN. + _ic_expect "java with no package manager falls back to SDKMAN" "sdk" java + _ic_expect "node uses nvm" "nvm install" node apt-get + _ic_expect "python uses pyenv" "pyenv install" python apt-get + _ic_expect "python pins 3.12" "3.12" python apt-get + for tool in docker sbt definitely-not-a-tool; do + got=""; rc=0 + got=$( PATH="$_ic_dir" + TEXERA_NODE_VERSION=24 TEXERA_PYTHON_VERSION=3.12 + eval "$pm_fn"; eval "$cmd_fn"; _install_cmd_for "$tool" 2>/dev/null ) || rc=$? + if (( rc != 0 )) && [[ -z "$got" ]]; then + _pass "install cmd: refuses '$tool'" + else + _fail "install cmd: should refuse '$tool'" "rc=$rc got='$got'" + fi + done + rm -rf "$_ic_dir" +fi + +# 40) Consent. The prompt is for humans; a non-interactive run (CI, cron, an +# agent following AGENTS.md's non-interactive subcommands) must keep the old +# behaviour and never block on a read. This is the most important test in +# this PR: a regression here hangs every automated `up`. +consent_fn=$(_extract_fn _consent_to_install) +if [[ -z "$consent_fn" ]]; then + _fail "_consent_to_install helper missing" +else + rc=0 + ( eval "$consent_fn"; _consent_to_install java "sudo apt-get install -y openjdk-17-jdk" ) \ + </dev/null >/dev/null 2>&1 || rc=$? + if (( rc != 0 )); then + _pass "consent: refuses without a TTY (never prompts)" + else + _fail "consent: granted install without a TTY" + fi + rc=0 + ( TEXERA_INSTALL_MISSING=1; eval "$consent_fn"; _consent_to_install java "cmd" ) \ + </dev/null >/dev/null 2>&1 || rc=$? + if (( rc == 0 )); then + _pass "consent: TEXERA_INSTALL_MISSING=1 assumes yes" + else + _fail "consent: TEXERA_INSTALL_MISSING=1 didn't assume yes" "rc=$rc" + fi + rc=0 + ( TEXERA_INSTALL_MISSING=0; eval "$consent_fn"; _consent_to_install java "cmd" ) \ + </dev/null >/dev/null 2>&1 || rc=$? + if (( rc != 0 )); then + _pass "consent: TEXERA_INSTALL_MISSING=0 refuses" + else + _fail "consent: TEXERA_INSTALL_MISSING=0 didn't refuse" + fi +fi + +# 41) Picking the interpreter that runs Python UDFs. The old default was +# `command -v python3` — the system interpreter, which has none of +# amber/requirements.txt — so UDFs died at worker launch on import errors +# that pointed nowhere near the interpreter choice. +ver_fn=$(_extract_fn _python_version_of) +vok_fn=$(_extract_fn _python_version_ok) +dok_fn=$(_extract_fn _python_deps_ok) +if [[ -z "$ver_fn" || -z "$vok_fn" || -z "$dok_fn" ]]; then + _fail "python probe helpers missing (_python_version_of/_python_version_ok/_python_deps_ok)" +else + _py_dir=$(mktemp -d 2>/dev/null || mktemp -d -t ldpy) + # `python -c ...` is only ever asked for the version string or an import + # check, so scripts that echo a version / set an exit code drive both probes. + printf '#!/bin/sh\necho 3.12\n' > "$_py_dir/py312" + printf '#!/bin/sh\necho 3.11\n' > "$_py_dir/py311" + printf '#!/bin/sh\nexit 0\n' > "$_py_dir/deps-ok" + printf '#!/bin/sh\nexit 1\n' > "$_py_dir/deps-missing" + chmod +x "$_py_dir"/* + _py_probe() { # $1=fn $2=path → rc + local rc=0 + ( TEXERA_PYTHON_VERSION=3.12 + eval "$ver_fn"; eval "$vok_fn"; eval "$dok_fn" + "$1" "$2" ) >/dev/null 2>&1 || rc=$? + return $rc + } + if _py_probe _python_version_ok "$_py_dir/py312"; then + _pass "python probe: 3.12 accepted" + else + _fail "python probe: 3.12 rejected" + fi + if ! _py_probe _python_version_ok "$_py_dir/py311"; then + _pass "python probe: 3.11 rejected" + else + _fail "python probe: 3.11 accepted (must pin 3.12)" + fi + if ! _py_probe _python_version_ok "$_py_dir/nonexistent"; then + _pass "python probe: missing interpreter rejected" + else + _fail "python probe: missing interpreter accepted" + fi + if _py_probe _python_deps_ok "$_py_dir/deps-ok"; then + _pass "python probe: importable deps accepted" + else + _fail "python probe: importable deps rejected" + fi + if ! _py_probe _python_deps_ok "$_py_dir/deps-missing"; then + _pass "python probe: missing deps rejected" + else + _fail "python probe: missing deps accepted" + fi + if ! _py_probe _python_deps_ok ""; then + _pass "python probe: empty path rejected" + else + _fail "python probe: empty path accepted" + fi + rm -rf "$_py_dir" +fi + +# 42) Candidate order. An activated venv must beat the sibling venv312 from +# AGENTS.md's layout, and both must beat whatever `python3` happens to be on +# PATH — that last one being the old, broken default. +cand_fn=$(_extract_fn _udf_python_candidates) +if [[ -z "$cand_fn" ]]; then + _fail "_udf_python_candidates helper missing" +else + _cd_dir=$(mktemp -d 2>/dev/null || mktemp -d -t ldcd) + mkdir -p "$_cd_dir/active/bin" "$_cd_dir/ws/venv312/bin" "$_cd_dir/ws/texera" + : > "$_cd_dir/active/bin/python"; chmod +x "$_cd_dir/active/bin/python" + : > "$_cd_dir/ws/venv312/bin/python"; chmod +x "$_cd_dir/ws/venv312/bin/python" + order=$( VIRTUAL_ENV="$_cd_dir/active" REPO_ROOT="$_cd_dir/ws/texera" \ + SELF_ROOT="$_cd_dir/ws/texera" TEXERA_PYTHON_VERSION=3.12 + eval "$cand_fn"; _udf_python_candidates 2>/dev/null ) + pos_active=$(printf '%s\n' "$order" | grep -n "active/bin/python" | head -1 | cut -d: -f1) + pos_venv=$(printf '%s\n' "$order" | grep -n "venv312/bin/python" | head -1 | cut -d: -f1) + pos_sys=$(printf '%s\n' "$order" | grep -nE "/python3$" | tail -1 | cut -d: -f1) + if [[ -n "$pos_active" && -n "$pos_venv" ]] && (( pos_active < pos_venv )); then + _pass "python candidates: \$VIRTUAL_ENV before sibling venv312" + else + _fail "python candidates: wrong order" \ + "active=$pos_active venv312=$pos_venv order=$(printf '%s' "$order" | tr '\n' '|')" + fi + if [[ -z "$pos_sys" ]] || { [[ -n "$pos_venv" ]] && (( pos_venv < pos_sys )); }; then + _pass "python candidates: venv312 before bare python3 on PATH" + else + _fail "python candidates: bare python3 outranks venv312" \ + "venv312=$pos_venv sys=$pos_sys order=$(printf '%s' "$order" | tr '\n' '|')" + fi + rm -rf "$_cd_dir" +fi + +# 43) The resolver must be wired into the paths that launch services, and must +# NOT run at source time — `status` / `--help` shouldn't pay for spawning +# interpreters. +if [[ -n "$(_extract_fn _require_udf_python)" ]]; then + _pass "_require_udf_python helper is defined" +else + _fail "_require_udf_python helper missing" +fi +for fn in cmd_up cmd_auto cmd_up_one; do + body=$(_extract_fn "$fn") + if [[ "$body" == *"_require_udf_python"* ]]; then + _pass "$fn resolves the UDF interpreter before launching" + else + _fail "$fn doesn't call _require_udf_python" + fi +done +# The UDF interpreter must not be advertised with the `-i` dashboard's hint: +# that one is about textual and amber/dev-requirements.txt, a different +# interpreter and a different requirements file. +# Comments are stripped: the code explains *why* it doesn't use that hint, and +# the explanation naturally names it. +udf_body=$(_extract_fn _require_udf_python | sed 's/^[[:space:]]*#.*$//') +if [[ "$udf_body" != *"_install_hint python"* ]] \ + && [[ "$udf_body" == *"UDF_PYTHON_PATH"* ]] \ + && [[ "$udf_body" == *"amber/requirements.txt"* ]]; then + _pass "_require_udf_python points at amber's requirements, not the TUI hint" +else + _fail "_require_udf_python reuses the TUI python hint or omits amber's requirements" +fi +if ! grep -qE '^export UDF_PYTHON_PATH="\$\{UDF_PYTHON_PATH:-\$\(command -v python3' "$MAIN_SH"; then + _pass "UDF_PYTHON_PATH no longer defaults to bare python3 at source time" +else + _fail "UDF_PYTHON_PATH still defaults to \`command -v python3\` at source time" +fi + +# 44) The new knobs are discoverable, and contradictory ones are refused rather +# than silently resolved one way. +help_out=$("$SCRIPT" --help 2>&1) +if [[ "$help_out" == *"--install-missing"* && "$help_out" == *"--no-install"* ]]; then + _pass "--help documents --install-missing / --no-install" +else + _fail "--help doesn't document the install flags" +fi +_im_state=$(mktemp -d 2>/dev/null || mktemp -d -t ldim) +out=$(TEXERA_LOCAL_DEV_DIR="$_im_state" "$SCRIPT" up --install-missing --no-install 2>&1); rc=$? +if (( rc == 2 )); then + _pass "up rejects --install-missing together with --no-install (rc=2)" +else + _fail "up accepted contradictory install flags" "rc=$rc out=$(echo "$out" | head -1)" +fi +rm -rf "$_im_state" + printf "\n%d passed, %d failed\n" "$PASS" "$FAIL" (( FAIL == 0 ))
