linrrzqqq opened a new pull request, #70786: URL: https://github.com/apache/doris/pull/70786
### Background The backend discovers Python environments during startup. In venv mode, it uses the full version of each configured interpreter to resolve an environment directory. `SHOW PYTHON PACKAGES` asks each live backend to list the packages for the requested interpreter. ### What problem does this PR solve? Issue Number: None Related PR: #59543 (introduced venv discovery), #60751 (introduced the package-list command) PR Type: Bugfix Problem Summary: Venv discovery provisioned another environment when the expected path was missing. With verbose logging enabled, pip's upgrade check appended debug output to the package list and Doris rejected the JSON. Discovery now reports the missing executable and its expected environment directory. Package listing disables the upgrade check. ### Details - **Frequency** — Venv discovery took the creation path whenever the versioned interpreter was missing. With pip 23.2.1 and `PIP_VERBOSE=2`, the enabled upgrade check adds debug output to stdout. - **Root cause** — `scan_from_venv_root_path` created the versioned directory and invoked the configured interpreter's venv module. `list_installed_packages` ran pip with its upgrade check enabled. - **Why existing tests missed it** — The previous venv unit test expected automatic provisioning. The package-list unit tests supply pip output, and the SQL suite checks the returned package rows. - **Fix** — Require the versioned interpreter to exist and pass `--disable-pip-version-check` on the JSON listing command. - **Other paths and branches** — Shared-nothing and cloud backends use the same scanner and package listing. Both original code paths appear in the local `origin/branch-4.1` and `origin/branch-4.2` refs. - **Behavior change** — Each configured venv must exist at the versioned path before BE startup. ### Testing The local comparison used Python 3.12.12, pip 23.2.1, `PIP_VERBOSE=2`, PyPI, and a fresh pip cache in each run. The BE used `python_env_mode = venv`, `python_venv_interpreter_paths = /usr/bin/python3`, and `python_venv_root_path` pointing to the directory shown as `$VENV_ROOT` below. The same SQL ran in each case. ```sql SHOW PYTHON PACKAGES IN '3.12.12'; ``` **Before the fix and an empty venv root (`6e7bea93c53`)** The BE created `$VENV_ROOT/python3.12.12` during startup. `SHOW PYTHON VERSIONS;` returned the environment it had created. ```text Version EnvName EnvType BasePath ExecutablePath 3.12.12 python3.12.12 venv $VENV_ROOT/python3.12.12 $VENV_ROOT/python3.12.12/bin/python ``` `SHOW PYTHON PACKAGES` returned this error. The BE also logged the JSON parse failure. ```text ERROR 1105 (HY000) at line 1: errCode = 2, detailMessage = [E6] Failed to parse pip list json output for python version: 3.12.12 ``` In a direct run of `python -m pip list --format=json`, the first line after the JSON array was `1 location(s) to search for versions of pip:`. `python -m json.tool` reported `Extra data: line 2 column 1`. With `--disable-pip-version-check`, the output parsed as JSON. **With the fix and an empty venv root (`5a0adb52cf2`)** The expected environment directory remained absent. The startup log reported the missing interpreter, with the local venv root abbreviated here. ```text Failed to initialize python version manager: [NOT_FOUND]Python virtual environment executable not found: $VENV_ROOT/python3.12.12/bin/python. Create the environment at $VENV_ROOT/python3.12.12 before starting BE. ``` ```text ERROR 1105 (HY000) at line 1: errCode = 2, detailMessage = No alive backends found to get python packages ``` **With the fix and a provisioned venv (`5a0adb52cf2`)** Provisioning the environment and restarting BE let the same SQL return these rows. The pip self-check cache remained absent. ```bash /usr/bin/python3 -m venv --system-site-packages "$VENV_ROOT/python3.12.12" ``` ```text Package Version pip 23.2.1 setuptools 68.2.2 ``` **Other local checks** - The full FE and BE builds passed for both commits. clang-format 16, clang-tidy, and the backend build hygiene checks passed for the fix. - Both targeted `PythonEnvTest` cases passed with the coverage-enabled runner. - `test_show_python_packages_command` passed on the isolated cluster. The run executed one suite with zero failures and zero skips. ### Release note Provision every configured Python virtual environment under `python_venv_root_path/python<full-version>` before starting the backend. `SHOW PYTHON PACKAGES` skips pip's upgrade check. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No. The existing Python UDF documentation directs operators to create the versioned venv and install dependencies before use. - Pre-submission status: Ready ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Backend changes, unit test changes, and this draft - Human review: not stated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
