The GitHub Actions job "Required Checks" on texera.git/main has succeeded. Run started by GitHub user github-merge-queue[bot] (triggered by github-merge-queue[bot]).
Head commit for run: 321fe45505ecadf3aba184c2caee2b948c0d86dc / Tanishq Gandhi <[email protected]> feat(storage): add datasets resource-type prefix to logical paths (#6502) ### What changes were proposed in this PR? Adds an explicit resource-type prefix to asset logical file paths, changing the format from `/<owner>/<name>/<version>/<file>` to `/datasets/<owner>/<name>/<version>/<file>`. Makes the `datasets` resource-type prefix **required** on dataset logical file paths (`/datasets/<owner>/<name>/<version>/<file>`), so other resource types (e.g. models) can be told apart by the prefix and routed to their own table. Unlike the initial approach, an **unprefixed path no longer resolves** — the prefix is what selects the resource's table. - **Path resolver (`FileResolver`):** a dataset path must start with `datasets`; the previous "parse unprefixed as-is" fallback is removed. - **Python file API (`DatasetFileDocument`):** same rule, mirroring the backend. - **File Lister operator:** parses the now-prefixed `datasetVersionPath` (the second dataset-path property, alongside scan sources' `fileName`). - **File tree / frontend:** tree rooted at a `datasets` node; the selection modal emits prefixed paths; relative-path extraction strips the 4-segment prefix; unused client-side parser removed. - **Cover images (`DatasetResource`):** cover-image handlers now build prefixed paths. - **Example workflows:** updated to prefixed paths. - **Migration (`sql/updates/36.sql`):** one-time, Liquibase-run rewrite that prepends `datasets/` to legacy paths in `workflow.content` and `workflow_version.content` (both `fileName` and `datasetVersionPath`). Only values whose first two segments match an existing `(user.email, dataset.name)` are rewritten (local paths/URLs untouched; email format is irrelevant); idempotent. **Known migration limits:** - A path whose dataset was **renamed or deleted** since the workflow was saved won't match `(email, name)`, so it stays unprefixed and will fail to resolve (it was already unusable). - **Hardcoded paths inside user code are not migrated (breaking).** The migration rewrites only the `fileName` and `datasetVersionPath` operator properties, so a path written by hand inside a Python UDF — e.g. `DatasetFileDocument("/[email protected]/ds/v1/f.csv")`, stored in the operator's `code` property — is left untouched and now raises `ValueError: Invalid file path format. Expected: /datasets/ownerEmail/datasetName/versionName/fileRelativePath`. Unlike the renamed/deleted case above, these paths **were working before this change**. Users must add the `datasets/` prefix in their UDF code; the error message states the expected format. Rewriting arbitrary user source in a SQL migration would risk corrupting code, so this is documented rather than automated — it needs a release note. - The migration assumes `content` is valid JSON (an app invariant) and aborts on a malformed row rather than skipping, so a bad row rolls the whole migration back instead of applying partially. ### Any related issues, documentation, discussions? Closes #6495. ### How was this PR tested? New and updated unit tests, all passing locally: - `FileResolverSpec`: an unprefixed path (and an unknown resource-type prefix) no longer resolves; a valid prefixed path resolves; too-few-segments is rejected. - `FileListerSourceOpExecSpec` (new): a prefixed `datasetVersionPath` parses; unprefixed / unknown-resource-type / too-few rejected. - Frontend: `datasetVersionFileTree` and `dataset-selection-modal` specs updated for the prefix. - Python: `test_dataset_file_document.py` — prefix required, presign re-emits it. The migration (`36.sql`) was verified manually against sample data: unprefixed→prefixed; already-prefixed left unchanged (idempotent); local paths/URLs untouched; dangling (renamed/deleted) datasets untouched; operators without the property get no spurious key added; both `fileName` and `datasetVersionPath` covered. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) --------- Co-authored-by: ali <[email protected]> Co-authored-by: Claude Opus 4.8 <[email protected]> Report URL: https://github.com/apache/texera/actions/runs/31649256024 With regards, GitHub Actions via GitBox
