Yicong-Huang opened a new pull request, #7073:
URL: https://github.com/apache/texera/pull/7073
> **Stacked on #7070** (`fix/local-dev-linux`), which it needs for the
> platform detection. Review that one first; this PR's own diff is the second
> commit. I'll rebase onto `main` once #7070 lands.
### What changes were proposed in this PR?
Two halves of the same fresh-machine problem.
**The interpreter that runs Python UDFs was picked blindly.**
`UDF_PYTHON_PATH` defaulted to `command -v python3` at source time — the
system
interpreter, which almost never has `amber/requirements.txt` installed.
Python
UDFs then failed at worker launch on a stack of import errors that pointed
nowhere near the interpreter choice. The venv AGENTS.md tells contributors to
create (`<workspace>/venv312`) was ignored, as was an already-activated
`$VIRTUAL_ENV`.
Resolution is now lazy, and takes the first candidate that is Python 3.12
**and** can import what a worker needs:
```
$UDF_PYTHON_PATH -> $VIRTUAL_ENV -> <workspace>/venv312 -> pyenv 3.12
-> python3.12 on PATH -> python3 on PATH
```
The choice is reported the way the JDK probe reports `JAVA_HOME`. An explicit
`UDF_PYTHON_PATH` still wins — it is the documented override — but now says
so
when it can't import the deps. "No 3.12 anywhere" and "3.12 without amber's
deps" are reported differently, because they need different fixes and the old
default silently produced the second one. Lazy so `status` / `logs` /
`--help`
don't spawn an interpreter per candidate, and never fatal: the JVM services
run
fine without a Python toolchain, only UDFs don't.
**A missing tool was described, never offered.** `_install_hint` printed a
suggestion and gave up, so a fresh machine meant hand-installing five or six
things, re-running `up` after each to discover the next gap. A missing JDK
17,
Node or Python 3.12 is now offered for install, after showing the exact
command:
| Tool | Installer |
| --- | --- |
| JDK 17 | distro package manager (`apt-get`/`dnf`/`yum`/`pacman`/`zypper`),
`brew` on macOS, SDKMAN when there is none |
| Node 24 | `nvm` (bootstraps nvm first if absent) |
| Python 3.12 | `pyenv` (bootstraps pyenv first if absent) |
`jenv` is not used: it switches between JDKs, it cannot install one.
The decision helpers (`_pkg_manager`, `_install_cmd_for`) only ever *print*
what would run; a separate step executes it. That keeps the choice
unit-testable without installing anything, and it means the user sees the
command — `sudo` included — before being asked.
Behaviour deliberately preserved for non-interactive callers: **without a TTY
nothing is ever prompted** and the old print-a-hint-and-fail path stands, so
scripted and CI runs cannot hang on a `read`. `--install-missing` answers yes
without asking, `--no-install` only ever prints the hint, and the
contradiction
is refused rather than silently resolved. docker and sbt stay hint-only —
docker needs a daemon, group membership and a re-login, which is not
something
to do behind a y/n prompt.
### Any related issues, documentation, discussions?
Closes #7066
Stacked on #7070 (#7065).
### How was this PR tested?
New coverage in the existing `infra`-job suite. Everything goes through the
pure decision helpers, so CI asserts on *what would be installed* without
installing anything; the Python probes are driven by fake interpreters that
echo a version or set an exit code:
```
$ bash bin/local-dev/tests/test_local_dev_sh.sh
...
✓ pkg manager: apt-get → apt-get
✓ pkg manager: dnf only → dnf
✓ pkg manager: pacman only → pacman
✓ pkg manager: apt-get preferred over dnf → apt-get
✓ pkg manager: nothing installed (refuses)
✓ _pkg_manager keeps the Darwin/brew branch
✓ install cmd: java via apt names openjdk-17-jdk
✓ install cmd: java via apt asks for sudo explicitly
✓ install cmd: java via dnf names java-17-openjdk-devel
✓ install cmd: java with no package manager falls back to SDKMAN
✓ install cmd: node uses nvm
✓ install cmd: python uses pyenv
✓ install cmd: python pins 3.12
✓ install cmd: refuses 'docker'
✓ install cmd: refuses 'sbt'
✓ install cmd: refuses 'definitely-not-a-tool'
✓ consent: refuses without a TTY (never prompts)
✓ consent: TEXERA_INSTALL_MISSING=1 assumes yes
✓ consent: TEXERA_INSTALL_MISSING=0 refuses
✓ python probe: 3.12 accepted
✓ python probe: 3.11 rejected
✓ python probe: missing interpreter rejected
✓ python probe: importable deps accepted
✓ python probe: missing deps rejected
✓ python probe: empty path rejected
✓ python candidates: $VIRTUAL_ENV before sibling venv312
✓ python candidates: venv312 before bare python3 on PATH
✓ _require_udf_python helper is defined
✓ cmd_up resolves the UDF interpreter before launching
✓ cmd_auto resolves the UDF interpreter before launching
✓ cmd_up_one resolves the UDF interpreter before launching
✓ _require_udf_python points at amber's requirements, not the TUI hint
✓ UDF_PYTHON_PATH no longer defaults to bare python3 at source time
✓ --help documents --install-missing / --no-install
✓ up rejects --install-missing together with --no-install (rc=2)
103 passed, 0 failed
```
```
$ python -m pytest bin/local-dev/tests/ -q
1 failed, 42 passed
```
`test_is_dirty_after_seed_then_edit` is **pre-existing and unrelated** — it
reproduces identically on a pristine `upstream/main` checkout on this
machine,
and this PR touches neither `tui.py` nor that test. Filed separately.
Manually, on Ubuntu 24.04.4 with the amber deps installed into a sibling
`venv312` and nothing in `UDF_PYTHON_PATH` — it finds the venv rather than
the
system interpreter the old default would have taken:
```
$ bin/local-dev.sh auto
✓ python: /home/…/Repos/venv312/bin/python (runs Python UDFs)
$ command -v python3 # what the old default resolved to
/usr/bin/python3
$ /usr/bin/python3 -c "import pyarrow"
ModuleNotFoundError: No module named 'pyarrow'
```
The consent path, forced by pinning a version that cannot exist
(`TEXERA_PYTHON_VERSION=3.99`). With a TTY it shows the command and asks; the
answer here was `n`, and the run continued:
```
⚠ python: no Python 3.99 found — Python UDFs will not run
looked in: $VIRTUAL_ENV, <workspace>/venv312, pyenv, PATH
or point at one yourself: export UDF_PYTHON_PATH=/path/to/python3.99
python is missing. I can run:
curl -fsSL https://pyenv.run | bash && export
PYENV_ROOT="$HOME/.pyenv" && … && pyenv install -s 3.99
Run it now? [y/N]
```
Without a TTY (`</dev/null`) the same run prints the warning, never prompts,
does not hang, and exits 0. Both flags are accepted on their own
(`auto --no-install`, `auto --install-missing` → rc 0) and rejected together
(rc 2).
Not exercised end-to-end: actually running the JDK / Node / pyenv installers,
since this machine already has all three. The commands themselves are
asserted
by the tests above; the execution step is a `bash -lc` of exactly the string
shown to the user.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
