GitHub user aicam created a discussion: Deprecate and remove per-user Python
Virtual Environments (PVE) in favor of curated images?
## Context
Texera currently has two mechanisms for providing the Python packages a Python
UDF needs at execution time:
1. **Per-user Python Virtual Environments (PVE).** A user creates named
environments from the dashboard ("Environments" tab) and installs packages into
them; a Python UDF can then select an environment instead of the default one.
Implemented in `amber/.../web/resource/pythonvirtualenvironment/`
(`PveManager`, `PveResource`, `PveWebsocketResource`), the
`virtual-environment` frontend service and `user-venv` dashboard page, and the
"Default Python Environment" option on the Python UDF operators.
2. **Curated computing-unit images.** An admin registers and validates a
container image, and users select it when creating a computing unit.
Implemented in `CuratedImageResource` and the admin `cu-image` UI.
These overlap in purpose: both exist so a workflow can run against a Python
environment that has the packages it needs.
## Proposal
Consider deprecating and eventually removing PVE, standardizing on curated
images as the way to provide custom Python dependencies.
Reasons this may be worth doing:
- Two overlapping subsystems to maintain, document and test (PVE is roughly 900
lines of backend, ~1,500 lines of frontend, and ~2,300 lines of tests).
- Curated images produce reproducible, pre-built environments rather than
packages installed ad hoc at runtime.
- Fewer moving parts in the execution path (no per-run package installation
step).
## Open questions / trade-offs
- **Self-service gap.** PVE is self-service; curated images are admin-managed.
Do we want a self-service story before removing PVE, or is admin curation
acceptable (and arguably preferable) for most deployments?
- **Deployment coverage.** Curated images are oriented around Kubernetes. What
is the recommended path for single-node users who rely on PVE today?
- **Migration.** Are there active PVE users who would need a migration path or
a deprecation notice period? Note the "Default Python Environment" /
environment-name fields are part of saved workflow JSON, so removal touches
stored workflows.
- **Scope.** Removal would span the backend resources, the dashboard
"Environments" page, the operator-level option, and the `virtual_environments`
table.
## Request for comments
Is standardizing on curated images the right direction? If so, what is the
minimum we would want in place (docs, a single-node story, a deprecation
notice) before deprecating PVE? If not, how should the two features relate
going forward?
GitHub link: https://github.com/apache/texera/discussions/9001
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
