GitHub user bobbai00 created a discussion: Proposal: Unify Backend Resource 
Management On Texera

## How Texera manages resources today

Texera manages six kinds of resources: workflows, datasets, models, computing 
units, environments, and warehouses. By managing a resource I mean three 
things: creating, reading, updating, and deleting it (CRUD), keeping its 
versions, and controlling who can access it. Today each kind is managed by its 
own code: workflows and warehouses in the Dashboard Service, datasets and 
models in the File Service, computing units in the Computing Unit Managing 
Service, and environments in the computing unit itself.

## Major Limitation: Resources Are Managed Ad-hoc

I think Texera's resource management has one major limitation: resources are 
managed ad-hoc. Each kind was built on its own, with its own tables, endpoints, 
and rules, and the codebase has no single definition of a resource that every 
kind follows. As a result, the same three things are done differently for each 
kind:

- **CRUD is defined per kind**: each kind has its own endpoints with its own 
conventions. For example, a workflow is created with `POST /workflow/create`, 
but an environment with `POST /pve/db`. A dataset is deleted with `DELETE 
/dataset/{did}`, but a workflow with `POST /workflow/delete`.
- **Versions are defined per kind**: each versioned kind has its own version 
logic. For example, the Dashboard Service stores a workflow version as a JSON 
patch on every save that changes the workflow. The File Service instead stores 
a dataset version as a lakeFS commit when the user asks for one, and keeps a 
second copy of that code for model versions.
- **Access control is defined per kind**: each kind has its own way to record 
who owns a resource and who can access it. For example, ownership is recorded 
in a `workflow_of_user` table for workflows, in an `owner_uid` column for 
datasets and models, and in a `uid` column for the other kinds. The rules 
differ too: a user with read access can remove themselves from a shared 
workflow, but not from a shared dataset, and environments and warehouses cannot 
be shared at all.

The codebase itself shows how far the kinds have drifted apart. 
`ResourceTables.scala` was added so that the access rules could be written 
once, but its own comment says that it fits only datasets and models, because 
workflows record ownership in a separate table and computing units have no 
public flag.

## Consequences: Backend Code Is Duplicated, Hard to Maintain, and Hard to 
Extend 

The current backend codebase suffers from three problems:

- **Duplicate code**: the same logic is written once per kind. For example, 
`ModelResource.scala` repeats 31 of the 32 endpoints of 
`DatasetResource.scala`, and the access rules are implemented three times: for 
workflows, for datasets and models, and for computing units. The database 
repeats itself in the same way: 17 of its 42 tables are per-kind copies.
- **Hard to add new resource kinds**: a new kind has to repeat the tables, the 
endpoints, the version logic, and the access rules. For example, adding models 
took 12 backend pull requests and 5 database migrations, and 5 of those pull 
requests were refactors that made dataset code reusable for models. Down the 
road I expect Texera to add more kinds, such as agents, conversations, and 
operators, and each of them will pay this cost again.
- **Hard to develop new features on top of resources**: a feature that works 
across kinds has to be connected to each kind separately. For example, search 
maps the columns of each searchable kind by hand into one schema of 28 columns, 
and the agent service can read and save only workflows, because that is the 
only kind it has a client for. A new feature that works with every kind, such 
as an agent or an MCP server, would need the same boilerplate for each kind.

## Proposal: Unify the Resource Management

I propose to unify resource management. By unify I mean: designing and 
implementing common primitives that are shared across all kinds of resources.

Based on current Texera's position, I see three major primitives:

**P1: Resource CRUD**: every kind has an id, a name, and an owner, and every 
kind needs the same operations: create, list, read, update, and delete.

**P2: Resource version CRUD**: a workflow version, a dataset version, and a 
model version are the same concept, which is an immutable snapshot of a 
resource. The `dataset_version` and `model_version` tables already have the 
same columns.

**P3: Resource access control**: every kind needs the same decision, which is 
whether a user may read or write a resource. The four `*_user_access` tables 
already have the same three columns: the user, the resource, and the privilege.

[image: Screenshot 2026-11-09 at 4.51.03 PM.png\]

What stays specific to a kind is its own fields and actions, such as executing 
a workflow or uploading a file to a dataset. Once the common parts exist in one 
copy, a new kind defines only what is specific to it, and a feature built on 
the common parts works for every kind.

## Implementation Proposal

I see at least two layers whose implementation needs to be refactored: the 
database schema layer and the API definition layer. Each layer needs its own 
detailed discussion, so below I only give my proposal for each.

### Layer 1: Database Schema Layer

At this layer, my proposal is to store the common parts once, under two 
supertypes: `Resource` and `Resource Version`.

**Before**: each kind has its own tables for the common parts. Each kind keeps 
its own relationships to `User`, ten `own` and `access` relationships in total, 
and each versioned kind keeps its own version table.

[image: Animation of today's database schema: the plain diagram first, then 
redundant access tables, redundant version tables, and what adding one kind 
costs, highlighted one at a time\]

**After**: every kind is a subtype of `Resource`, which has a single `access` 
relationship to `User` with the levels own, read, and write. Every version is a 
subtype of `Resource Version`, keyed by `(rid, vno)`, so one `lineage` table 
can link a version to the versions it was derived from. A new kind adds only 
its own fields under `Resource`.

[image: Animation of the proposed database schema: the plain diagram first, 
then one access relationship, one version model, easy to record resource 
lineage, and easy to add new resource kinds, highlighted one at a time\]

### Layer 2: API Definition Layer

My proposal at this layer is to give every kind the same endpoints for the 
three common parts. For each part, the table below lists the endpoints before 
and after. In the After column, `{kind}` is the plural name of a kind, such as 
`workflows` or `datasets`, `{rid}` is the id of a resource, `{vno}` is the 
number of a version, and `{uid}` is the id of a user.

#### Resource CRUD

Before, each kind spells the same operation in its own way, and some kinds lack 
an operation. After, every kind has the same five operations.

| Operation | Before | After |
| --- | --- | --- |
| Create | `POST /workflow/create`, `POST /dataset/create`, `POST 
/model/create`, `POST /computing-unit/create`, `POST /pve/db`, `POST 
/warehouse` | `POST /{kind}` |
| List | `GET /workflow/list`, `GET /dataset/list`, `GET /model/list`, `GET 
/computing-unit`, `GET /pve/db`, `GET /warehouse/status` | `GET /{kind}` |
| Read | `GET /workflow/{wid}`, `GET /dataset/{did}`, `GET /model/{mid}`, `GET 
/computing-unit/{cuid}`; none for environments and warehouses | `GET 
/{kind}/{rid}` |
| Update (rename) | `POST /workflow/update/name`, `POST /dataset/update/name`, 
`POST /model/update/name`, `PUT /computing-unit/{cuid}/rename/{name}`, `PUT 
/pve/db/{veid}`; none for warehouses | `PATCH /{kind}/{rid}` |
| Delete | `POST /workflow/delete`, `DELETE /dataset/{did}`, `DELETE 
/model/{mid}`, `DELETE /computing-unit/{cuid}/terminate`, `DELETE 
/pve/db/{veid}`, `DELETE /warehouse/{whid}` | `DELETE /{kind}/{rid}` |

#### Resource Version CRUD

Before, workflow versions have their own root path and no endpoint to create 
one, because a version is made whenever a workflow is saved, while dataset and 
model versions put the operation in the path. After, every versioned kind has 
the same three operations. A version is numbered within its resource and 
immutable, so it has no update or delete.

| Operation | Before | After |
| --- | --- | --- |
| Create | `POST /dataset/{did}/version/create`, `POST 
/model/{mid}/version/create`; none for workflows, where `POST 
/workflow/persist` makes a version as a side effect | `POST 
/{kind}/{rid}/versions` |
| List | `GET /version/{wid}`, `GET /dataset/{did}/version/list`, `GET 
/model/{mid}/version/list` | `GET /{kind}/{rid}/versions` |
| Read | `GET /version/{wid}/{vid}`, `GET 
/dataset/{did}/version/{dvid}/rootFileNodes`, `GET 
/model/{mid}/version/{mvid}/rootFileNodes` | `GET /{kind}/{rid}/versions/{vno}` 
|

#### Resource Access Control

Before, the same sharing endpoints are defined four times, in three services. 
Environments and warehouses have none, and no endpoint checks access, so each 
service checks it in its own code. After, every kind is shared and checked in 
the same way.

| Operation | Before | After |
| --- | --- | --- |
| List | `GET /access/workflow/list/{wid}`, plus a copy each for datasets, 
models, and computing units | `GET /{kind}/{rid}/access` |
| Grant or change | `PUT /access/workflow/grant/{wid}/{email}/{privilege}`, 
plus the same three copies | `PUT /{kind}/{rid}/access/{uid}` |
| Revoke | `DELETE /access/workflow/revoke/{wid}/{email}`, plus the same three 
copies | `DELETE /{kind}/{rid}/access/{uid}` |
| Check | No endpoint: each service checks access in its own code | `POST 
/access/evaluation` |


GitHub link: https://github.com/apache/texera/discussions/9395

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to