This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7789-4842e93f2d274399eb9baa9594d3f231b00ec519 in repository https://gitbox.apache.org/repos/asf/texera.git
commit bc9479618953a24b057b4e7ce4d13e9af71775a2 Author: Tanishq Gandhi <[email protected]> AuthorDate: Fri Aug 21 00:31:39 2026 +0000 feat(storage): singularize the resource-type prefix on logical paths (#7789) ### What changes were proposed in this PR? The leading segment of a versioned-resource logical path was plural while every name it refers to is singular — the backing tables (`dataset`, `model`), the dashboard search's resource type, and the hub's entity type. This makes the prefix match them. ``` Before: /datasets/[email protected]/twitter/v1/f.csv table: dataset /models/[email protected]/resnet/v1/w.pt table: model After: /dataset/[email protected]/twitter/v1/f.csv table: dataset /model/[email protected]/resnet/v1/w.pt table: model ``` | Layer | Change | | --- | --- | | Enums | `ResourceType` in Scala (`Datasets`/`Models` → `Dataset`/`Model`), TypeScript (`Datasets` → `Dataset`), Python (`DATASETS` → `DATASET`) | | Call sites | `FileResolver`, `FileListerSourceOpExec`, `DocumentFactory`, `LakeFSFileDocument`, `DatasetResource`, `DatasetFileNode`, `dataset-selection-modal`, `dataset_file_document` | | Migration | `36.sql` restored as it shipped; normalization added in a new `39.sql` | | Examples | the two shipped `bin/single-node/examples/workflows/*.json` carried the plural prefix | Two hardcoded `"dataset"` string literals on the DB side now read from the enum, since the prefix and the type name are the same word again: - `SearchQueryBuilder.DATASET_RESOURCE_TYPE`, inlined into the unified search SQL - `EntityType.Dataset.value`, written to `user_action.resource_type` **On the migration.** `36.sql` introduced the prefix in its plural form and is restored here byte-for-byte, so its checksum stays intact; the normalization lives in a new `39.sql` (38 is taken). Correcting 36 in place could not reach the databases that need it: liquibase selects changeSets by id — `bin/local-dev/main.sh:1824` skips on a match in `databasechangelog` without reading the file — so the databases holding plural paths are exactly the ones that recorded id 36 and will never run it again, and `MD5SUM` is left NULL by design, so nothing reports a mismatch and the stale paths surface only when someone opens one of those workflows. `39.sql` normalizes both shapes, so it converges whether or not 36 ran: | Stored value | Action | | --- | --- | | `/[email protected]/ds/v1/f.csv` (unprefixed legacy) | prepend `dataset` | | `/datasets/[email protected]/ds/v1/f.csv` (plural prefix) | rewrite the leading segment | | `/dataset/...` | already current — untouched | Both cases apply only when the path's owner and name segments match a real `(user.email, dataset.name)` pair — read at parts 1 and 2 unprefixed, 2 and 3 prefixed. That guard is what keeps the migration off the plain filesystem paths and URLs this column also holds: `/datasets` is an ordinary directory name, and `FileResolver` tries `localResolveFunc` first, so a `fileName` of `/datasets/imdb/movies.csv` can be a working local mount and is left alone. A path naming a dataset that no longer exists therefore keeps its old form; it cannot resolve either way. `/models/` is out of scope, since `37.sql` creates that table and runs after this changeset. ### Any related issues, documentation, discussions? Additional work on #6495, which introduced the resource-type prefix. ### How was this PR tested? Existing suites, updated where they pin the path form. ```bash sbt "WorkflowCore/testOnly org.apache.texera.amber.storage.FileResolverSpec org.apache.texera.amber.core.storage.DocumentFactorySpec org.apache.texera.amber.core.storage.model.LakeFSFileDocumentSpec" \ "WorkflowOperator/testOnly org.apache.texera.amber.operator.source.dataset.FileListerSourceOpExecSpec" \ "FileService/testOnly org.apache.texera.service.type.DatasetFileNodeSpec" \ "WorkflowExecutionService/testOnly org.apache.texera.web.resource.dashboard.DatasetSearchQueryBuilderSpec org.apache.texera.web.resource.dashboard.DashboardResourceSpec org.apache.texera.web.resource.dashboard.UnifiedResourceSchemaSpec org.apache.texera.web.resource.dashboard.hub.HubResourceSpec org.apache.texera.web.resource.dashboard.user.workflow.WorkflowExecutionsResourceSpec org.apache.texera.web.resource.dashboard.file.DatasetResourceSpec" ``` ```bash pytest amber/src/test/python/pytexera/storage/test_dataset_file_document.py ``` ```bash cd frontend && yarn ng test --watch=false --include='**/datasetVersionFileTree.spec.ts' --include='**/dataset-selection-modal.component.spec.ts' --include='**/dataset-version-selector.component.spec.ts' --include='**/user-dataset-version-filetree.component.spec.ts' --include='**/dataset-detail.component.spec.ts' ``` `sbt Test/compile` and `tsc` on both app and spec sources are clean, so the enum rename has no missed call sites. `39.sql` was run against a seeded Postgres over 13 fixtures, one per case: legacy unprefixed, plural prefix, `/datasets`-rooted local mounts (3-segment, 4-segment, and as `datasetVersionPath`), plural naming a deleted dataset, `/models/`, already-singular, a URL, `/datasets/` appearing mid-path, non-array `operators`, absent property, and a row checking other operator properties, links and operator order survive. Correct in every case, on Postgres 18 and 15.11, and idempotent — a second run reports 0 rows. Tested through liquibase's id tracking rather than only by piping the SQL in, emulating `main.sh` against a real `databasechangelog` table: ``` fresh DB: changeSet 36 ran, 39 rewrote -> /dataset/[email protected]/ds/v1/a.csv recorded id 36: changeSet 36 skipped, 39 rewrote -> /dataset/[email protected]/ds/v1/a.csv ``` Also covered: `39` alone with no `36`, `39` twice, and `39` before `36` out of order — all four converge on the same state, with the local mount untouched throughout. The same statements were also run inside a Kubernetes deployment: `bin/k8s` rendered with `values-development.yaml`, the postgresql resources applied to minikube, and the SQL piped into the pod against the real 39-table schema the chart bootstraps. Row-for-row identical to local, including the local-mount cases staying untouched. Note that the chart's postgres init runs only `texera_ddl.sql` and the three catalog scripts, and `texera_db` has no `databasechangelog` table — no liquibase changeSet executes on a Kubernetes deployment. `sbt scalafmtCheckAll`, `ruff check`, `ruff format --check`, and `prettier --check` are clean. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) --------- Co-authored-by: Claude Opus 5 <[email protected]> Co-authored-by: ali <[email protected]> --- .../pytexera/storage/dataset_file_document.py | 10 +- .../main/python/pytexera/storage/resource_type.py | 2 +- .../resource/dashboard/SearchQueryBuilder.scala | 11 +- .../web/resource/dashboard/hub/EntityType.scala | 3 +- .../pytexera/storage/test_dataset_file_document.py | 34 ++- .../workflow/WorkflowExecutionsResourceSpec.scala | 6 +- ...xample] Data Exploration on Movies Dataset.json | 2 +- ...[Example] Machine Learning on Iris Dataset.json | 2 +- .../amber/core/storage/DocumentFactory.scala | 8 +- .../texera/amber/core/storage/FileResolver.scala | 16 +- .../texera/amber/core/storage/ResourceType.scala | 4 +- .../core/storage/model/LakeFSFileDocument.scala | 4 +- .../amber/core/storage/DocumentFactorySpec.scala | 8 +- .../storage/model/LakeFSFileDocumentSpec.scala | 10 +- .../texera/amber/storage/FileResolverSpec.scala | 24 +- .../source/dataset/FileListerSourceOpExec.scala | 6 +- .../dataset/FileListerSourceOpExecSpec.scala | 26 +-- .../texera/service/resource/DatasetResource.scala | 2 +- .../service/type/dataset/DatasetFileNode.scala | 16 +- .../service/type/dataset/DatasetFileNodeSpec.scala | 12 +- .../app/common/type/datasetVersionFileTree.spec.ts | 8 +- .../src/app/common/type/datasetVersionFileTree.ts | 2 +- frontend/src/app/common/type/resource-type.ts | 4 +- .../dataset-detail.component.spec.ts | 4 +- ...user-dataset-version-filetree.component.spec.ts | 6 +- .../dataset-selection-modal.component.spec.ts | 8 +- .../dataset-selection-modal.component.ts | 4 +- .../dataset-version-selector.component.spec.ts | 8 +- sql/changelog.xml | 5 + sql/updates/39.sql | 254 +++++++++++++++++++++ 30 files changed, 388 insertions(+), 121 deletions(-) diff --git a/amber/src/main/python/pytexera/storage/dataset_file_document.py b/amber/src/main/python/pytexera/storage/dataset_file_document.py index b8d778d342..2d14f858e4 100644 --- a/amber/src/main/python/pytexera/storage/dataset_file_document.py +++ b/amber/src/main/python/pytexera/storage/dataset_file_document.py @@ -59,18 +59,18 @@ class DatasetFileDocument: :param file_path: Expected format - - "/datasets/ownerEmail/datasetName/versionName/fileRelativePath" + "/dataset/ownerEmail/datasetName/versionName/fileRelativePath" Example: - "/datasets/[email protected]/twitterDataset/v1/california/tw1.csv" + "/dataset/[email protected]/twitterDataset/v1/california/tw1.csv" """ parts = file_path.strip("/").split("/") invalid_format = ValueError( "Invalid file path format. Expected: " - "/datasets/ownerEmail/datasetName/versionName/fileRelativePath" + "/dataset/ownerEmail/datasetName/versionName/fileRelativePath" ) - # TODO(datasets-prefix): require the prefix once all stored paths are migrated (36.sql) and ml model support work is completed. + # TODO(dataset-prefix): require the prefix once all stored paths are migrated (36.sql) and ml model support work is completed. if parts and parts[0] in {t.value for t in ResourceType}: if len(parts) < 5: raise invalid_format @@ -80,7 +80,7 @@ class DatasetFileDocument: self.version_name = parts[3] self.file_relative_path = "/".join(parts[4:]) elif len(parts) >= 4: - self.resource_type = ResourceType.DATASETS + self.resource_type = ResourceType.DATASET self.owner_email = parts[0] self.dataset_name = parts[1] self.version_name = parts[2] diff --git a/amber/src/main/python/pytexera/storage/resource_type.py b/amber/src/main/python/pytexera/storage/resource_type.py index 596fb5d745..e77ec9dd49 100644 --- a/amber/src/main/python/pytexera/storage/resource_type.py +++ b/amber/src/main/python/pytexera/storage/resource_type.py @@ -21,4 +21,4 @@ from enum import Enum class ResourceType(str, Enum): """The leading segment of a logical file path, identifying the resource kind""" - DATASETS = "datasets" + DATASET = "dataset" diff --git a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/SearchQueryBuilder.scala b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/SearchQueryBuilder.scala index e19755e611..e8a651cb82 100644 --- a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/SearchQueryBuilder.scala +++ b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/SearchQueryBuilder.scala @@ -19,6 +19,7 @@ package org.apache.texera.web.resource.dashboard +import org.apache.texera.amber.core.storage.ResourceType import org.apache.texera.dao.SqlServer import org.apache.texera.web.resource.dashboard.DashboardResource.{ DashboardClickableFileEntry, @@ -33,10 +34,18 @@ object SearchQueryBuilder { SqlServer .getInstance() .createDSLContext() + // Resource-type discriminators for dashboard search. They are inlined into the search SQL, + // pattern-matched in DashboardResource, and duplicated as bare literals in the frontend. + // DATASET_RESOURCE_TYPE aliases ResourceType.Dataset, whose value doubles as the leading + // segment of a storage logical path, so renaming that prefix silently changes search results + // with no compile error here. + // TODO: give every resource type (file, workflow, project, dataset, model) a single shared + // constant/enum instead of these raw strings, so the search contract has one source of + // truth and no longer borrows an unrelated storage constant. val FILE_RESOURCE_TYPE = "file" val WORKFLOW_RESOURCE_TYPE = "workflow" val PROJECT_RESOURCE_TYPE = "project" - val DATASET_RESOURCE_TYPE = "dataset" + val DATASET_RESOURCE_TYPE = ResourceType.Dataset.toString val ALL_RESOURCE_TYPE = "" } diff --git a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/EntityType.scala b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/EntityType.scala index 0cf1bd4cc8..bd83b7837e 100644 --- a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/EntityType.scala +++ b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/EntityType.scala @@ -20,6 +20,7 @@ package org.apache.texera.web.resource.dashboard.hub import com.fasterxml.jackson.annotation.{JsonCreator, JsonValue} +import org.apache.texera.amber.core.storage.ResourceType /** * Defines all supported entity types for Hub resources. @@ -34,7 +35,7 @@ sealed trait EntityType { object EntityType { case object Workflow extends EntityType { val value = "workflow" } - case object Dataset extends EntityType { val value = "dataset" } + case object Dataset extends EntityType { val value: String = ResourceType.Dataset.toString } private val values = Seq(Workflow, Dataset) diff --git a/amber/src/test/python/pytexera/storage/test_dataset_file_document.py b/amber/src/test/python/pytexera/storage/test_dataset_file_document.py index bef134bc74..b90dcce81b 100644 --- a/amber/src/test/python/pytexera/storage/test_dataset_file_document.py +++ b/amber/src/test/python/pytexera/storage/test_dataset_file_document.py @@ -47,18 +47,18 @@ def make_response(status_code: int, body=None, content: bytes = b""): class TestDatasetFileDocumentInit: def test_parses_prefixed_path(self, auth_env): - doc = DatasetFileDocument("/datasets/[email protected]/ds/v1/file.csv") + doc = DatasetFileDocument("/dataset/[email protected]/ds/v1/file.csv") assert doc.owner_email == "[email protected]" assert doc.dataset_name == "ds" assert doc.version_name == "v1" assert doc.file_relative_path == "file.csv" def test_joins_nested_relative_path_back_with_slashes(self, auth_env): - doc = DatasetFileDocument("/datasets/[email protected]/ds/v1/a/b/c/file.csv") + doc = DatasetFileDocument("/dataset/[email protected]/ds/v1/a/b/c/file.csv") assert doc.file_relative_path == "a/b/c/file.csv" def test_strips_leading_and_trailing_slashes_before_parsing(self, auth_env): - doc = DatasetFileDocument("///datasets/[email protected]/ds/v1/file.csv///") + doc = DatasetFileDocument("///dataset/[email protected]/ds/v1/file.csv///") assert doc.owner_email == "[email protected]" assert doc.file_relative_path == "file.csv" @@ -80,7 +80,7 @@ class TestDatasetFileDocumentInit: def test_rejects_prefixed_path_with_too_few_segments(self, auth_env): with pytest.raises(ValueError, match="Invalid file path format"): - DatasetFileDocument("/datasets/[email protected]/ds/v1") + DatasetFileDocument("/dataset/[email protected]/ds/v1") def test_rejects_legacy_path_with_too_few_segments(self, auth_env): with pytest.raises(ValueError, match="Invalid file path format"): @@ -92,29 +92,29 @@ class TestDatasetFileDocumentInit: "FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", CUSTOM_ENDPOINT ) with pytest.raises(ValueError, match="JWT token is required"): - DatasetFileDocument("/datasets/[email protected]/ds/v1/file.csv") + DatasetFileDocument("/dataset/[email protected]/ds/v1/file.csv") def test_treats_empty_jwt_as_missing(self, monkeypatch): # An empty string is falsy and should be rejected just like an unset var. monkeypatch.setenv("USER_JWT_TOKEN", "") with pytest.raises(ValueError, match="JWT token is required"): - DatasetFileDocument("/datasets/[email protected]/ds/v1/file.csv") + DatasetFileDocument("/dataset/[email protected]/ds/v1/file.csv") def test_falls_back_to_default_endpoint_when_env_missing(self, monkeypatch): monkeypatch.setenv("USER_JWT_TOKEN", "tok") monkeypatch.delenv( "FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", raising=False ) - doc = DatasetFileDocument("/datasets/[email protected]/ds/v1/file.csv") + doc = DatasetFileDocument("/dataset/[email protected]/ds/v1/file.csv") assert doc.presign_endpoint == DEFAULT_ENDPOINT def test_uses_explicit_endpoint_from_environment(self, auth_env): - doc = DatasetFileDocument("/datasets/[email protected]/ds/v1/file.csv") + doc = DatasetFileDocument("/dataset/[email protected]/ds/v1/file.csv") assert doc.presign_endpoint == CUSTOM_ENDPOINT class TestGetPresignedUrl: - def _make_doc(self, monkeypatch, path="/datasets/[email protected]/ds/v1/file.csv"): + def _make_doc(self, monkeypatch, path="/dataset/[email protected]/ds/v1/file.csv"): monkeypatch.setenv("USER_JWT_TOKEN", "test-jwt-token") monkeypatch.setenv( "FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", CUSTOM_ENDPOINT @@ -144,9 +144,7 @@ class TestGetPresignedUrl: def test_url_encodes_filepath_query_parameter(self, monkeypatch): # urllib.parse.quote keeps "/" as safe by default, but encodes "@" # and " " — pin both pieces so the contract is explicit. - doc = self._make_doc( - monkeypatch, path="/datasets/[email protected]/ds/v1/data file.csv" - ) + doc = self._make_doc(monkeypatch, path="/dataset/[email protected]/ds/v1/data file.csv") with patch( "pytexera.storage.dataset_file_document.requests.Session.get" ) as mock_get: @@ -158,16 +156,16 @@ class TestGetPresignedUrl: assert "bob%40x.com" in file_path assert file_path.startswith("/") - def test_sends_datasets_prefixed_filepath(self, monkeypatch): - # The reconstructed filePath sent to the file-service carries the "datasets" prefix. - doc = self._make_doc(monkeypatch, path="/datasets/[email protected]/ds/v1/file.csv") + def test_sends_dataset_prefixed_filepath(self, monkeypatch): + # The reconstructed filePath sent to the file-service carries the "dataset" prefix. + doc = self._make_doc(monkeypatch, path="/dataset/[email protected]/ds/v1/file.csv") with patch( "pytexera.storage.dataset_file_document.requests.Session.get" ) as mock_get: mock_get.return_value = make_response(200, body={"presignedUrl": "u"}) doc.get_presigned_url() _, kwargs = mock_get.call_args - assert kwargs["params"]["filePath"].startswith("/datasets/") + assert kwargs["params"]["filePath"].startswith("/dataset/") def test_calls_configured_endpoint(self, monkeypatch): doc = self._make_doc(monkeypatch) @@ -235,7 +233,7 @@ class TestReadFile: monkeypatch.setenv( "FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", CUSTOM_ENDPOINT ) - return DatasetFileDocument("/datasets/[email protected]/ds/v1/file.csv") + return DatasetFileDocument("/dataset/[email protected]/ds/v1/file.csv") def test_returns_bytesio_with_downloaded_content(self, monkeypatch): doc = self._make_doc(monkeypatch) @@ -291,7 +289,7 @@ class TestTimeoutsAndRetries: monkeypatch.setenv( "FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", CUSTOM_ENDPOINT ) - return DatasetFileDocument("/datasets/[email protected]/ds/v1/file.csv") + return DatasetFileDocument("/dataset/[email protected]/ds/v1/file.csv") def test_presigned_url_request_passes_request_timeout(self, monkeypatch): doc = self._make_doc(monkeypatch) diff --git a/amber/src/test/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowExecutionsResourceSpec.scala b/amber/src/test/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowExecutionsResourceSpec.scala index 7dc5742a18..7cc549004d 100644 --- a/amber/src/test/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowExecutionsResourceSpec.scala +++ b/amber/src/test/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowExecutionsResourceSpec.scala @@ -910,7 +910,7 @@ class WorkflowExecutionsResourceSpec val content = """{ | "operators": [ - | {"operatorID": "scanA", "operatorProperties": {"fileName": "/datasets/[email protected]/LockedDS/v1/data.csv"}}, + | {"operatorID": "scanA", "operatorProperties": {"fileName": "/dataset/[email protected]/LockedDS/v1/data.csv"}}, | {"operatorID": "downstreamB", "operatorProperties": {}} | ], | "links": [ @@ -969,7 +969,7 @@ class WorkflowExecutionsResourceSpec val content = """{ | "operators": [ - | {"operatorID": "scan", "operatorProperties": {"fileName": "/datasets/[email protected]/MyDS/v1/data.csv"}} + | {"operatorID": "scan", "operatorProperties": {"fileName": "/dataset/[email protected]/MyDS/v1/data.csv"}} | ], | "links": [] |}""".stripMargin @@ -1408,7 +1408,7 @@ class WorkflowExecutionsResourceSpec private def scanOperator(operatorId: String, datasetName: String): String = s"""{"operatorID": "$operatorId", "operatorProperties": """ + - s"""{"fileName": "/datasets/$foreignOwnerEmail/$datasetName/v1/data.csv"}}""" + s"""{"fileName": "/dataset/$foreignOwnerEmail/$datasetName/v1/data.csv"}}""" // Seeds three datasets owned by somebody other than testUser and wires the workflow as // diff --git a/bin/single-node/examples/workflows/[Example] Data Exploration on Movies Dataset.json b/bin/single-node/examples/workflows/[Example] Data Exploration on Movies Dataset.json index cdc9a0e870..5df44f5382 100644 --- a/bin/single-node/examples/workflows/[Example] Data Exploration on Movies Dataset.json +++ b/bin/single-node/examples/workflows/[Example] Data Exploration on Movies Dataset.json @@ -8,7 +8,7 @@ "fileEncoding": "UTF_8", "customDelimiter": ",", "hasHeader": true, - "fileName": "/datasets/texera/popular-movies-of-imdb/v1/TMDb_updated.csv", + "fileName": "/dataset/texera/popular-movies-of-imdb/v1/TMDb_updated.csv", "offset": 0, "limit": 1000 }, diff --git a/bin/single-node/examples/workflows/[Example] Machine Learning on Iris Dataset.json b/bin/single-node/examples/workflows/[Example] Machine Learning on Iris Dataset.json index 7e9a5d11fd..e4cdc3a8af 100644 --- a/bin/single-node/examples/workflows/[Example] Machine Learning on Iris Dataset.json +++ b/bin/single-node/examples/workflows/[Example] Machine Learning on Iris Dataset.json @@ -8,7 +8,7 @@ "fileEncoding": "UTF_8", "customDelimiter": ",", "hasHeader": true, - "fileName": "/datasets/texera/iris-species/v1/Iris.csv" + "fileName": "/dataset/texera/iris-species/v1/Iris.csv" }, "inputPorts": [], "outputPorts": [ diff --git a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/DocumentFactory.scala b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/DocumentFactory.scala index 8481906c9c..4915d57897 100644 --- a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/DocumentFactory.scala +++ b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/DocumentFactory.scala @@ -60,8 +60,8 @@ object DocumentFactory { */ def openReadonlyDocument(fileUri: URI): ReadonlyVirtualDocument[_] = { fileUri.getScheme match { - case DATASET_FILE_URI_SCHEME => new LakeFSFileDocument(fileUri, ResourceType.Datasets) - case MODEL_FILE_URI_SCHEME => new LakeFSFileDocument(fileUri, ResourceType.Models) + case DATASET_FILE_URI_SCHEME => new LakeFSFileDocument(fileUri, ResourceType.Dataset) + case MODEL_FILE_URI_SCHEME => new LakeFSFileDocument(fileUri, ResourceType.Model) case "file" => new ReadonlyLocalFileDocument(fileUri) case unsupportedScheme => throw new UnsupportedOperationException( @@ -184,8 +184,8 @@ object DocumentFactory { */ def openDocument(uri: URI): (VirtualDocument[_], Option[Schema]) = { uri.getScheme match { - case DATASET_FILE_URI_SCHEME => (new LakeFSFileDocument(uri, ResourceType.Datasets), None) - case MODEL_FILE_URI_SCHEME => (new LakeFSFileDocument(uri, ResourceType.Models), None) + case DATASET_FILE_URI_SCHEME => (new LakeFSFileDocument(uri, ResourceType.Dataset), None) + case MODEL_FILE_URI_SCHEME => (new LakeFSFileDocument(uri, ResourceType.Model), None) case VFS_FILE_URI_SCHEME => val IcebergLocation(warehouse, namespace, storageKey) = resolveIcebergLocation(uri) diff --git a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/FileResolver.scala b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/FileResolver.scala index 1a2e4fcf2a..95127e234d 100644 --- a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/FileResolver.scala +++ b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/FileResolver.scala @@ -93,7 +93,7 @@ object FileResolver { * caller can dispatch to the right backing table. * - Legacy unprefixed (datasets only, backward compat): /ownerEmail/datasetName/versionName/ * fileRelativePath (>= 4 segments), resolved as a dataset. Models are new and always require - * the /models/ prefix. + * the /model/ prefix. * * @param fileName the file path to parse * @return Some((resourceType, ownerEmail, resourceName, versionName, fileRelativePath)) if valid, @@ -115,11 +115,11 @@ object FileResolver { ) case None => // Legacy unprefixed dataset path (backward compat): /ownerEmail/datasetName/versionName/<file>. - // TODO(datasets-prefix): require the prefix once all stored paths are migrated (36.sql). + // TODO(dataset-prefix): require the prefix once all stored paths are migrated (36.sql). if (pathSegments.length >= 4) Some( ( - ResourceType.Datasets, + ResourceType.Dataset, pathSegments(0), pathSegments(1), pathSegments(2), @@ -139,8 +139,8 @@ object FileResolver { * * Input: /<prefix>/ownerEmail/resourceName/versionName/fileRelativePath (or legacy unprefixed) * Output: {scheme}:///{repositoryName}/{versionHash}/fileRelativePath - * e.g. /datasets/[email protected]/twitter/v1/dir/f.csv -> dataset:///dataset-15/adeq233td/dir/f.csv - * /models/[email protected]/resnet/v1/weights/m.pt -> model:///model-15/adeq233td/weights/m.pt + * e.g. /dataset/[email protected]/twitter/v1/dir/f.csv -> dataset:///dataset-15/adeq233td/dir/f.csv + * /model/[email protected]/resnet/v1/weights/m.pt -> model:///model-15/adeq233td/weights/m.pt * * @throws java.io.FileNotFoundException if the path is not a valid versioned-resource path, the * resource/version does not exist, or the URI is malformed @@ -152,10 +152,10 @@ object FileResolver { ) val (scheme, repositoryName, versionHash) = resourceType match { - case ResourceType.Datasets => + case ResourceType.Dataset => val (repo, hash) = lookupDataset(ownerEmail, resourceName, versionName, fileName) (DATASET_FILE_URI_SCHEME, repo, hash) - case ResourceType.Models => + case ResourceType.Model => val (repo, hash) = lookupModel(ownerEmail, resourceName, versionName, fileName) (MODEL_FILE_URI_SCHEME, repo, hash) case other => @@ -306,7 +306,7 @@ object FileResolver { return None } parsePrefixedPath(path).collect { - case (ResourceType.Datasets, ownerEmail, datasetName, _, _) => (ownerEmail, datasetName) + case (ResourceType.Dataset, ownerEmail, datasetName, _, _) => (ownerEmail, datasetName) } } } diff --git a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/ResourceType.scala b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/ResourceType.scala index 087dc6ef87..47e90a1818 100644 --- a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/ResourceType.scala +++ b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/ResourceType.scala @@ -26,8 +26,8 @@ package org.apache.texera.amber.core.storage * Path shape: /<prefix>/ownerEmail/resourceName/versionName/fileRelativePath */ object ResourceType extends Enumeration { - val Datasets: Value = Value("datasets") - val Models: Value = Value("models") + val Dataset: Value = Value("dataset") + val Model: Value = Value("model") /** * Returns the resource type named by the given path segment, or None if it is not a known diff --git a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocument.scala b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocument.scala index b821ff72f5..55fab33ce2 100644 --- a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocument.scala +++ b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocument.scala @@ -60,8 +60,8 @@ object LakeFSFileDocument { */ def presignEndpointOf(resourceType: ResourceType.Value): String = resourceType match { - case ResourceType.Datasets => datasetPresignEndpoint - case ResourceType.Models => modelPresignEndpoint + case ResourceType.Dataset => datasetPresignEndpoint + case ResourceType.Model => modelPresignEndpoint } } diff --git a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/DocumentFactorySpec.scala b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/DocumentFactorySpec.scala index a957819b31..f219383f25 100644 --- a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/DocumentFactorySpec.scala +++ b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/DocumentFactorySpec.scala @@ -137,7 +137,7 @@ class DocumentFactorySpec extends AnyFlatSpec with Matchers with BeforeAndAfterA val datasetUri = new URI(s"dataset:///repo/$versionHash/file.txt") val doc = DocumentFactory.openReadonlyDocument(datasetUri) doc shouldBe a[LakeFSFileDocument] - doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Datasets + doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Dataset doc.getURI shouldBe datasetUri } @@ -145,7 +145,7 @@ class DocumentFactorySpec extends AnyFlatSpec with Matchers with BeforeAndAfterA val modelUri = new URI(s"model:///model-1/$versionHash/weights/model.pt") val doc = DocumentFactory.openReadonlyDocument(modelUri) doc shouldBe a[LakeFSFileDocument] - doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Models + doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Model doc.getURI shouldBe modelUri val resource = doc.asInstanceOf[OnVersionedFileResource] @@ -181,7 +181,7 @@ class DocumentFactorySpec extends AnyFlatSpec with Matchers with BeforeAndAfterA val datasetUri = new URI(s"dataset:///repo/$versionHash/file.txt") val (doc, schemaOpt) = DocumentFactory.openDocument(datasetUri) doc shouldBe a[LakeFSFileDocument] - doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Datasets + doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Dataset schemaOpt shouldBe None } @@ -189,7 +189,7 @@ class DocumentFactorySpec extends AnyFlatSpec with Matchers with BeforeAndAfterA val modelUri = new URI(s"model:///model-1/$versionHash/weights/model.pt") val (doc, schemaOpt) = DocumentFactory.openDocument(modelUri) doc shouldBe a[LakeFSFileDocument] - doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Models + doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe ResourceType.Model schemaOpt shouldBe None } diff --git a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala index 56d5cd36ce..fa569f97ca 100644 --- a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala +++ b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala @@ -35,7 +35,7 @@ class LakeFSFileDocumentSpec extends AnyFlatSpec with Matchers { private val versionHash = "97fd4c2a755b69b7c66d322eab40b7e5c2ad5d10" // URI parsing is shared by every resource type; exercise it through the dataset type. - private def datasetDoc(uri: URI) = new LakeFSFileDocument(uri, ResourceType.Datasets) + private def datasetDoc(uri: URI) = new LakeFSFileDocument(uri, ResourceType.Dataset) "LakeFSFileDocument" should "parse a valid 3-segment dataset URI into its components" in { val uri = new URI(s"dataset:///test_dataset/$versionHash/1.txt") @@ -160,7 +160,7 @@ class LakeFSFileDocumentSpec extends AnyFlatSpec with Matchers { "http://localhost:9092/api/dataset/presign-download"; ) .trim - LakeFSFileDocument.presignEndpointOf(ResourceType.Datasets) shouldBe expected + LakeFSFileDocument.presignEndpointOf(ResourceType.Dataset) shouldBe expected } // Each resource type resolves its own endpoint: a dataset grant must not authorize a model @@ -173,9 +173,9 @@ class LakeFSFileDocumentSpec extends AnyFlatSpec with Matchers { "http://localhost:9092/api/model/presign-download"; ) .trim - LakeFSFileDocument.presignEndpointOf(ResourceType.Models) shouldBe expected - LakeFSFileDocument.presignEndpointOf(ResourceType.Models) should not be - LakeFSFileDocument.presignEndpointOf(ResourceType.Datasets) + LakeFSFileDocument.presignEndpointOf(ResourceType.Model) shouldBe expected + LakeFSFileDocument.presignEndpointOf(ResourceType.Model) should not be + LakeFSFileDocument.presignEndpointOf(ResourceType.Dataset) } // The user JWT token is shared by every LakeFS-backed document, whatever its resource type. diff --git a/common/workflow-core/src/test/scala/org/apache/texera/amber/storage/FileResolverSpec.scala b/common/workflow-core/src/test/scala/org/apache/texera/amber/storage/FileResolverSpec.scala index e414371222..2649a3d860 100644 --- a/common/workflow-core/src/test/scala/org/apache/texera/amber/storage/FileResolverSpec.scala +++ b/common/workflow-core/src/test/scala/org/apache/texera/amber/storage/FileResolverSpec.scala @@ -122,13 +122,13 @@ class FileResolverSpec private val localCsvFilePath = "common/workflow-core/src/test/resources/country_sales_small.csv" - private val datasetACsvFilePath = "/datasets/[email protected]/test_dataset/v2/directory/a.csv" + private val datasetACsvFilePath = "/dataset/[email protected]/test_dataset/v2/directory/a.csv" - private val dataset1TxtFilePath = "/datasets/[email protected]/test_dataset/v1/1.txt" + private val dataset1TxtFilePath = "/dataset/[email protected]/test_dataset/v1/1.txt" - private val modelWeightsFilePath = "/models/[email protected]/test_model/v2/weights/model.pt" + private val modelWeightsFilePath = "/model/[email protected]/test_model/v2/weights/model.pt" - private val modelReadmeFilePath = "/models/[email protected]/test_model/v1/README.md" + private val modelReadmeFilePath = "/model/[email protected]/test_model/v1/README.md" // Legacy unprefixed form — still resolvable as a dataset (backward compat). private val unprefixedDataset1TxtFilePath = "/[email protected]/test_dataset/v1/1.txt" @@ -137,11 +137,11 @@ class FileResolverSpec private val unknownResourceTypeFilePath = "/notAResourceType/[email protected]/test_dataset/v1/1.txt" - // A model name presented under the datasets prefix must not resolve as a dataset. - private val modelNameUnderDatasetPrefix = "/datasets/[email protected]/test_model/v1/README.md" + // A model name presented under the dataset prefix must not resolve as a dataset. + private val modelNameUnderDatasetPrefix = "/dataset/[email protected]/test_model/v1/README.md" - // A dataset name presented under the models prefix must not resolve as a model. - private val datasetNameUnderModelPrefix = "/models/[email protected]/test_dataset/v1/1.txt" + // A dataset name presented under the model prefix must not resolve as a model. + private val datasetNameUnderModelPrefix = "/model/[email protected]/test_dataset/v1/1.txt" override protected def beforeAll(): Unit = { initializeDBAndReplaceDSLContext() @@ -225,7 +225,7 @@ class FileResolverSpec "FileResolver" should "throw not found exception when a prefixed path has too few segments" in { assertThrows[FileNotFoundException] { - FileResolver.resolve("/datasets/[email protected]/test_dataset") + FileResolver.resolve("/dataset/[email protected]/test_dataset") } } @@ -254,12 +254,12 @@ class FileResolverSpec "parseDatasetOwnerAndName" should "extract owner email and dataset name from a valid path" in { assert( - FileResolver.parseDatasetOwnerAndName("/datasets/[email protected]/test_dataset/v1/1.txt") + FileResolver.parseDatasetOwnerAndName("/dataset/[email protected]/test_dataset/v1/1.txt") == Some(("[email protected]", "test_dataset")) ) // extra segments beyond the file-relative path are ignored assert( - FileResolver.parseDatasetOwnerAndName("/datasets/[email protected]/ds/v2/directory/nested/a.csv") + FileResolver.parseDatasetOwnerAndName("/dataset/[email protected]/ds/v2/directory/nested/a.csv") == Some(("[email protected]", "ds")) ) } @@ -272,7 +272,7 @@ class FileResolverSpec } it should "return None when the prefixed path has too few segments" in { - assert(FileResolver.parseDatasetOwnerAndName("/datasets/[email protected]/ds").isEmpty) + assert(FileResolver.parseDatasetOwnerAndName("/dataset/[email protected]/ds").isEmpty) assert(FileResolver.parseDatasetOwnerAndName("owner/dataset").isEmpty) } diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExec.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExec.scala index fb27257865..26a6496a24 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExec.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExec.scala @@ -32,7 +32,7 @@ import org.apache.texera.dao.jooq.generated.tables.User.USER object FileListerSourceOpExec { /** - * Parses a dataset version path (/datasets/ownerEmail/datasetName/versionName) into its + * Parses a dataset version path (/dataset/ownerEmail/datasetName/versionName) into its * (resourceTypePrefix, ownerEmail, datasetName, versionName) components. * * @throws IllegalArgumentException if the path is not a well-formed dataset version path @@ -42,14 +42,14 @@ object FileListerSourceOpExec { ): (String, String, String, String) = { val segments = datasetVersionPath.split("/").filter(_.nonEmpty) val invalidPath = s"Invalid dataset version path '$datasetVersionPath'; " + - "expected /datasets/ownerEmail/datasetName/versionName" + "expected /dataset/ownerEmail/datasetName/versionName" if (segments.headOption.exists(ResourceType.isValidPrefix)) { require(segments.length >= 4, invalidPath) (segments(0), segments(1), segments(2), segments(3)) } else { require(segments.length >= 3, invalidPath) - (ResourceType.Datasets.toString, segments(0), segments(1), segments(2)) + (ResourceType.Dataset.toString, segments(0), segments(1), segments(2)) } } diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExecSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExecSpec.scala index 19fb7513d5..ec0ffd9a29 100644 --- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExecSpec.scala +++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExecSpec.scala @@ -23,10 +23,10 @@ import org.scalatest.flatspec.AnyFlatSpec class FileListerSourceOpExecSpec extends AnyFlatSpec { - "parseDatasetVersionPath" should "extract components from a datasets-prefixed path" in { + "parseDatasetVersionPath" should "extract components from a dataset-prefixed path" in { val (prefix, owner, name, version) = - FileListerSourceOpExec.parseDatasetVersionPath("/datasets/[email protected]/twitterDataset/v1") - assert(prefix == "datasets") + FileListerSourceOpExec.parseDatasetVersionPath("/dataset/[email protected]/twitterDataset/v1") + assert(prefix == "dataset") assert(owner == "[email protected]") assert(name == "twitterDataset") assert(version == "v1") @@ -34,7 +34,7 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec { it should "work when the owner segment is a username without an '@'" in { val (prefix, owner, name, version) = - FileListerSourceOpExec.parseDatasetVersionPath("/datasets/texera/test-ds/v1") + FileListerSourceOpExec.parseDatasetVersionPath("/dataset/texera/test-ds/v1") assert(owner == "texera") assert(name == "test-ds") assert(version == "v1") @@ -42,8 +42,8 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec { it should "ignore trailing slashes and extra segments" in { val (prefix, owner, name, version) = - FileListerSourceOpExec.parseDatasetVersionPath("/datasets/alice/ds/v2/extra/") - assert(prefix == "datasets") + FileListerSourceOpExec.parseDatasetVersionPath("/dataset/alice/ds/v2/extra/") + assert(prefix == "dataset") assert(owner == "alice") assert(name == "ds") assert(version == "v2") @@ -52,11 +52,11 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec { "canonicalVersionPath" should "rebuild the prefixed version path from its components" in { assert( FileListerSourceOpExec.canonicalVersionPath( - "datasets", + "dataset", "[email protected]", "twitterDataset", "v1" - ) == "/datasets/[email protected]/twitterDataset/v1" + ) == "/dataset/[email protected]/twitterDataset/v1" ) } @@ -64,17 +64,17 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec { // Emitted paths must be rooted at the parsed components, not the raw configured path: // a stray "extra" segment would otherwise leak into every emitted file path. val (prefix, owner, name, version) = - FileListerSourceOpExec.parseDatasetVersionPath("/datasets/alice/ds/v2/extra/") + FileListerSourceOpExec.parseDatasetVersionPath("/dataset/alice/ds/v2/extra/") assert( FileListerSourceOpExec.canonicalVersionPath(prefix, owner, name, version) - == "/datasets/alice/ds/v2" + == "/dataset/alice/ds/v2" ) } it should "still accept a legacy unprefixed path" in { val (prefix, owner, name, version) = FileListerSourceOpExec.parseDatasetVersionPath("/alice/ds/v1") - assert(prefix == "datasets") + assert(prefix == "dataset") assert(owner == "alice") assert(name == "ds") assert(version == "v1") @@ -85,7 +85,7 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec { FileListerSourceOpExec.parseDatasetVersionPath("/alice/ds/v1") assert( FileListerSourceOpExec.canonicalVersionPath(prefix, owner, name, version) - == "/datasets/alice/ds/v1" + == "/dataset/alice/ds/v1" ) } @@ -99,7 +99,7 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec { it should "reject a prefixed path with too few segments" in { assertThrows[IllegalArgumentException] { - FileListerSourceOpExec.parseDatasetVersionPath("/datasets/alice/ds") + FileListerSourceOpExec.parseDatasetVersionPath("/dataset/alice/ds") } } diff --git a/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala b/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala index 8bb1a70661..4c8243b850 100644 --- a/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala +++ b/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala @@ -353,7 +353,7 @@ class DatasetResource extends LazyLogging { private val COVER_IMAGE_SIZE_LIMIT_BYTES: Long = 10 * 1024 * 1024 // 10 MB private val ALLOWED_IMAGE_EXTENSIONS: Set[String] = Set(".jpg", ".jpeg", ".png", ".gif", ".webp") - private val resourceType = ResourceType.Datasets + private val resourceType = ResourceType.Dataset /** * Helper function to get the dataset from DB with additional information including user access privilege and owner email diff --git a/file-service/src/main/scala/org/apache/texera/service/type/dataset/DatasetFileNode.scala b/file-service/src/main/scala/org/apache/texera/service/type/dataset/DatasetFileNode.scala index 821d02045c..85709b8e5b 100644 --- a/file-service/src/main/scala/org/apache/texera/service/type/dataset/DatasetFileNode.scala +++ b/file-service/src/main/scala/org/apache/texera/service/type/dataset/DatasetFileNode.scala @@ -25,8 +25,8 @@ import org.apache.texera.amber.core.storage.ResourceType import scala.collection.mutable // DatasetFileNode represents a unique file in dataset, its full path is in the format of: -// /datasets/ownerEmail/datasetName/versionName/fileRelativePath -// e.g. /datasets/[email protected]/twitterDataset/v1/california/irvine/tw1.csv +// /dataset/ownerEmail/datasetName/versionName/fileRelativePath +// e.g. /dataset/[email protected]/twitterDataset/v1/california/irvine/tw1.csv class DatasetFileNode( val name: String, // direct name of this node val nodeType: String, // "file" or "directory" @@ -79,10 +79,10 @@ object DatasetFileNode { ): List[DatasetFileNode] = { val rootNode = new DatasetFileNode("/", "directory", null, "") - // Root the tree at the datasets prefix node (a directory node named "datasets"). - val datasetsNode = - new DatasetFileNode(ResourceType.Datasets.toString, "directory", rootNode, "") - rootNode.children = Some(List(datasetsNode)) + // Root the tree at the dataset prefix node (a directory node named "dataset"). + val prefixNode = + new DatasetFileNode(ResourceType.Dataset.toString, "directory", rootNode, "") + rootNode.children = Some(List(prefixNode)) // Owner level nodes map val ownerNodes = mutable.Map[String, DatasetFileNode]() @@ -91,8 +91,8 @@ object DatasetFileNode { case ((ownerEmail, datasetName, versionName), objects) => val ownerNode = ownerNodes.getOrElseUpdate( ownerEmail, { - val newNode = new DatasetFileNode(ownerEmail, "directory", datasetsNode, ownerEmail) - datasetsNode.children = Some(datasetsNode.getChildren :+ newNode) + val newNode = new DatasetFileNode(ownerEmail, "directory", prefixNode, ownerEmail) + prefixNode.children = Some(prefixNode.getChildren :+ newNode) newNode } ) diff --git a/file-service/src/test/scala/org/apache/texera/service/type/dataset/DatasetFileNodeSpec.scala b/file-service/src/test/scala/org/apache/texera/service/type/dataset/DatasetFileNodeSpec.scala index 160412f8b9..12321d4180 100644 --- a/file-service/src/test/scala/org/apache/texera/service/type/dataset/DatasetFileNodeSpec.scala +++ b/file-service/src/test/scala/org/apache/texera/service/type/dataset/DatasetFileNodeSpec.scala @@ -99,13 +99,13 @@ class DatasetFileNodeSpec extends AnyFlatSpec with Matchers { Map(("[email protected]", "twitter", "v1") -> objects) ) - // The tree is rooted at a single "datasets" prefix node; owners nest under it. + // The tree is rooted at a single "dataset" prefix node; owners nest under it. roots should have size 1 - val datasetsNode = roots.head - datasetsNode.getName shouldBe "datasets" - datasetsNode.getNodeType shouldBe "directory" + val prefixNode = roots.head + prefixNode.getName shouldBe "dataset" + prefixNode.getNodeType shouldBe "directory" - val ownerNode = datasetsNode.getChildren.find(_.getName == "[email protected]").get + val ownerNode = prefixNode.getChildren.find(_.getName == "[email protected]").get ownerNode.getNodeType shouldBe "directory" val datasetNode = ownerNode.getChildren.find(_.getName == "twitter").get @@ -123,7 +123,7 @@ class DatasetFileNodeSpec extends AnyFlatSpec with Matchers { val file1 = bDir.getChildren.find(_.getName == "1.csv").get file1.getNodeType shouldBe "file" file1.getSize shouldBe Some(2L) - file1.getFilePath shouldBe "/datasets/[email protected]/twitter/v1/b/1.csv" + file1.getFilePath shouldBe "/dataset/[email protected]/twitter/v1/b/1.csv" // Total size equals the sum of the three files. DatasetFileNode.calculateTotalSize(roots) shouldBe 6L diff --git a/frontend/src/app/common/type/datasetVersionFileTree.spec.ts b/frontend/src/app/common/type/datasetVersionFileTree.spec.ts index 64231dd156..207182e5fd 100644 --- a/frontend/src/app/common/type/datasetVersionFileTree.spec.ts +++ b/frontend/src/app/common/type/datasetVersionFileTree.spec.ts @@ -41,7 +41,7 @@ describe("getRelativePathFromDatasetFileNode", () => { const node: DatasetFileNode = { name: "tw1.csv", type: "file", - parentDir: "/datasets/[email protected]/twitterDataset/v1/california/irvine", + parentDir: "/dataset/[email protected]/twitterDataset/v1/california/irvine", }; expect(getRelativePathFromDatasetFileNode(node)).toBe("california/irvine/tw1.csv"); }); @@ -50,7 +50,7 @@ describe("getRelativePathFromDatasetFileNode", () => { const node: DatasetFileNode = { name: "readme.txt", type: "file", - parentDir: "/datasets/[email protected]/twitterDataset/v1", + parentDir: "/dataset/[email protected]/twitterDataset/v1", }; expect(getRelativePathFromDatasetFileNode(node)).toBe("readme.txt"); }); @@ -59,7 +59,7 @@ describe("getRelativePathFromDatasetFileNode", () => { const node: DatasetFileNode = { name: "v1", type: "directory", - parentDir: "/datasets/[email protected]/twitterDataset", + parentDir: "/dataset/[email protected]/twitterDataset", }; expect(getRelativePathFromDatasetFileNode(node)).toBe(""); }); @@ -68,7 +68,7 @@ describe("getRelativePathFromDatasetFileNode", () => { const node: DatasetFileNode = { name: "f.csv", type: "file", - parentDir: "/datasets/[email protected]/twitterDataset//v1/sub", + parentDir: "/dataset/[email protected]/twitterDataset//v1/sub", }; expect(getRelativePathFromDatasetFileNode(node)).toBe("sub/f.csv"); }); diff --git a/frontend/src/app/common/type/datasetVersionFileTree.ts b/frontend/src/app/common/type/datasetVersionFileTree.ts index 98f898d443..efbb875128 100644 --- a/frontend/src/app/common/type/datasetVersionFileTree.ts +++ b/frontend/src/app/common/type/datasetVersionFileTree.ts @@ -32,7 +32,7 @@ export function getFullPathFromDatasetFileNode(node: DatasetFileNode): string { /** * Returns the relative path of a DatasetFileNode by stripping the first four segments - * (datasets/ownerEmail/datasetName/versionName). + * (dataset/ownerEmail/datasetName/versionName). * @param node The DatasetFileNode whose relative path is needed. * @returns The relative path (without the first four segments and without a leading slash). */ diff --git a/frontend/src/app/common/type/resource-type.ts b/frontend/src/app/common/type/resource-type.ts index a129c917e2..22c4a8592a 100644 --- a/frontend/src/app/common/type/resource-type.ts +++ b/frontend/src/app/common/type/resource-type.ts @@ -17,8 +17,8 @@ */ /** - * The leading segment of a logical file path, identifying the resource kind (e.g. /datasets/...). + * The leading segment of a logical file path, identifying the resource kind (e.g. /dataset/...). */ export enum ResourceType { - Datasets = "datasets", + Dataset = "dataset", } diff --git a/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/dataset-detail.component.spec.ts b/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/dataset-detail.component.spec.ts index 78c35761a4..7bb6af6e0d 100644 --- a/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/dataset-detail.component.spec.ts +++ b/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/dataset-detail.component.spec.ts @@ -2694,7 +2694,7 @@ describe("DatasetDetailComponent rendered template", () => { const leaf = (name: string): DatasetFileNode => ({ name, type: "file", - parentDir: `/datasets/${OWNER}/ds/v1/nested`, + parentDir: `/dataset/${OWNER}/ds/v1/nested`, size: 2048, }); @@ -2719,7 +2719,7 @@ describe("DatasetDetailComponent rendered template", () => { // The heading is the full path — the copy-path button beside it copies // exactly this string — not the bare file name or the relative path. expect(text(q<HTMLElement>(fixture.nativeElement, ".file-title-main"))).toBe( - `/datasets/${OWNER}/ds/v1/nested/b.csv` + `/dataset/${OWNER}/ds/v1/nested/b.csv` ); // 2048 bytes reaches the reader as a human-readable size, not as a raw count. expect(text(q<HTMLElement>(fixture.nativeElement, ".file-size"))).toBe("2.00 KB"); diff --git a/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/user-dataset-version-filetree/user-dataset-version-filetree.component.spec.ts b/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/user-dataset-version-filetree/user-dataset-version-filetree.component.spec.ts index 806d2a6f25..ff9e82609a 100644 --- a/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/user-dataset-version-filetree/user-dataset-version-filetree.component.spec.ts +++ b/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/user-dataset-version-filetree/user-dataset-version-filetree.component.spec.ts @@ -183,7 +183,7 @@ describe("UserDatasetVersionFiletreeComponent", () => { const file = (name: string): DatasetFileNode => ({ name, type: "file", - parentDir: "/datasets/owner/dataset/v1", + parentDir: "/dataset/owner/dataset/v1", }); // Deliberately synchronous: the tree keeps a pending timer, so awaiting whenStable() @@ -262,9 +262,9 @@ describe("UserDatasetVersionFiletreeComponent", () => { const emitted: string[] = []; component.setCoverImage.subscribe((path: string) => emitted.push(path)); - // parentDir has exactly the four stripped segments (datasets/owner/dataset/version), + // parentDir has exactly the four stripped segments (dataset/owner/dataset/version), // so the relative path is just the file name. - component.onSetCover({ name: "photo.png", type: "file", parentDir: "/datasets/owner/dataset/v1" }); + component.onSetCover({ name: "photo.png", type: "file", parentDir: "/dataset/owner/dataset/v1" }); expect(emitted).toEqual(["photo.png"]); }); diff --git a/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.spec.ts b/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.spec.ts index 604d86a9f9..f42ae00a81 100644 --- a/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.spec.ts +++ b/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.spec.ts @@ -111,7 +111,7 @@ describe("DatasetSelectionModalComponent", () => { it("ngOnInit initializes selectedDataset and selectedVersion from data.selectedPath", () => { modalData.fileMode = true; - modalData.selectedPath = `/datasets/${OWNER}/myds/v1`; + modalData.selectedPath = `/dataset/${OWNER}/myds/v1`; build(); @@ -154,7 +154,7 @@ describe("DatasetSelectionModalComponent", () => { expect(datasetService.retrieveDatasetVersionFileTree).toHaveBeenCalledWith(10, 100); expect(component.fileTree).toEqual([fileNode]); - expect(component.selectedPath).toBe(`/datasets/${OWNER}/myds/v1`); + expect(component.selectedPath).toBe(`/dataset/${OWNER}/myds/v1`); }); it("onFileSelected sets selectedPath to the node's full path in file mode", () => { @@ -303,12 +303,12 @@ describe("DatasetSelectionModalComponent", () => { confirmButton().click(); - expect(modalRef.close).toHaveBeenCalledWith(`/datasets/${OWNER}/myds/v1`); + expect(modalRef.close).toHaveBeenCalledWith(`/dataset/${OWNER}/myds/v1`); }); it("takes the path from the file tree's selection in file mode", () => { modalData.fileMode = true; - modalData.selectedPath = `/datasets/${OWNER}/myds/v1`; + modalData.selectedPath = `/dataset/${OWNER}/myds/v1`; build(); fixture.debugElement diff --git a/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.ts b/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.ts index 97ecf579ca..b5768aa685 100644 --- a/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.ts +++ b/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.ts @@ -85,7 +85,7 @@ export class DatasetSelectionModalComponent implements OnInit { const selectedPath = this.data.selectedPath; if (selectedPath) { const segments = selectedPath.split("/").filter(part => part.length > 0); - // TODO(datasets-prefix): require the prefix once all ml model support PRs are done. + // TODO(dataset-prefix): require the prefix once all ml model support PRs are done. if ((Object.values(ResourceType) as string[]).includes(segments[0])) { segments.shift(); } @@ -124,7 +124,7 @@ export class DatasetSelectionModalComponent implements OnInit { this.fileTree = data.fileNodes; }); if (!this.data.fileMode) { - this.selectedPath = `/${ResourceType.Datasets}/${this.selectedDataset.ownerEmail}/${this.selectedDataset.dataset.name}/${this.selectedVersion.name}`; + this.selectedPath = `/${ResourceType.Dataset}/${this.selectedDataset.ownerEmail}/${this.selectedDataset.dataset.name}/${this.selectedVersion.name}`; } } } diff --git a/frontend/src/app/workspace/component/dataset-version-selector/dataset-version-selector.component.spec.ts b/frontend/src/app/workspace/component/dataset-version-selector/dataset-version-selector.component.spec.ts index 394e9773cc..f5ba4045e8 100644 --- a/frontend/src/app/workspace/component/dataset-version-selector/dataset-version-selector.component.spec.ts +++ b/frontend/src/app/workspace/component/dataset-version-selector/dataset-version-selector.component.spec.ts @@ -101,11 +101,11 @@ describe("DatasetVersionSelectorComponent", () => { }); it("shows the chosen path in a read-only box", () => { - setFormControl("/datasets/[email protected]/myds/v1"); + setFormControl("/dataset/[email protected]/myds/v1"); fixture.detectChanges(); const box = selectionBox()!; - expect(box.value).toBe("/datasets/[email protected]/myds/v1"); + expect(box.value).toBe("/dataset/[email protected]/myds/v1"); // The path is only editable through the picker, never by typing. expect(box.readOnly).toBe(true); expect(box.required).toBe(true); @@ -114,13 +114,13 @@ describe("DatasetVersionSelectorComponent", () => { it("opens the picker from the button and shows whatever it returns", () => { setFormControl(""); fixture.detectChanges(); - modalServiceSpy.create.mockReturnValue({ afterClose: of("/datasets/[email protected]/myds/v2") }); + modalServiceSpy.create.mockReturnValue({ afterClose: of("/dataset/[email protected]/myds/v2") }); openButton().click(); fixture.detectChanges(); expect(modalServiceSpy.create).toHaveBeenCalledTimes(1); - expect(selectionBox()!.value).toBe("/datasets/[email protected]/myds/v2"); + expect(selectionBox()!.value).toBe("/dataset/[email protected]/myds/v2"); }); }); }); diff --git a/sql/changelog.xml b/sql/changelog.xml index c0f735a09c..b9c96431a5 100644 --- a/sql/changelog.xml +++ b/sql/changelog.xml @@ -104,6 +104,11 @@ <sqlFile path="sql/updates/38.sql"/> </changeSet> + <!-- Singularize the resource-type prefix on stored dataset paths --> + <changeSet id="39" author="tanishqgandhi1908"> + <sqlFile path="sql/updates/39.sql"/> + </changeSet> + <!-- example changeSet <changeSet id="1" author="author"> <sqlFile path="sql/updates/1.sql"/> diff --git a/sql/updates/39.sql b/sql/updates/39.sql new file mode 100644 index 0000000000..4c48244bdd --- /dev/null +++ b/sql/updates/39.sql @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +\c texera_db + +SET search_path TO texera_db; + +BEGIN; + +-- The resource-type prefix on dataset logical paths is singular, matching the table +-- it names: /dataset/ownerEmail/datasetName/versionName/... This supersedes 36.sql, +-- which introduced the prefix in its plural form; that changeSet is left as it +-- shipped, so databases that already recorded it are brought forward here instead. +-- +-- Stored workflows carry such paths inside workflow.content and +-- workflow_version.content, in two operator properties: +-- * fileName (scan-source operators): /owner/name/version/file +-- * datasetVersionPath (file-lister operator): /owner/name/version +-- +-- Both are normalized to that form: a "/datasets/" prefix has its leading segment +-- rewritten, and a path with no prefix gets "dataset" prepended. The prefixed case +-- is tested first, since such a path looks unprefixed to the other branch and must +-- not be prefixed twice. +-- +-- Either case applies only when the path's owner and name segments match an +-- existing (user.email, dataset.name) pair -- unique, and read at parts 1 and 2 +-- unprefixed, 2 and 3 prefixed. That guard is what keeps the migration off the +-- plain filesystem paths and URLs this column also holds: "/datasets" is an +-- ordinary directory name, and FileResolver tries localResolveFunc first, so a +-- fileName of /datasets/imdb/movies.csv can be a working local mount. It matches +-- no dataset, so it is left alone. +-- +-- Values already in the target form match neither case, making this idempotent. +-- jsonb_set uses create_missing = false so absent properties are never added. + +DO $$ +DECLARE + wf_count INT := 0; + wv_count INT := 0; +BEGIN + WITH affected AS ( + SELECT w.wid + FROM workflow w, + jsonb_array_elements( + CASE + WHEN jsonb_typeof(w.content::jsonb -> 'operators') = 'array' + THEN w.content::jsonb -> 'operators' + ELSE '[]'::jsonb + END + ) AS op, + LATERAL ( + SELECT op #>> '{operatorProperties,fileName}' AS fn, + op #>> '{operatorProperties,datasetVersionPath}' AS dvp + ) f + WHERE jsonb_typeof(w.content::jsonb -> 'operators') = 'array' + AND ( + (f.fn ~ '^/datasets/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 2) + AND d.name = split_part(ltrim(f.fn, '/'), '/', 3))) + OR + (f.dvp ~ '^/datasets/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 2) + AND d.name = split_part(ltrim(f.dvp, '/'), '/', 3))) + OR + (f.fn IS NOT NULL AND left(f.fn, 9) <> '/dataset/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 1) + AND d.name = split_part(ltrim(f.fn, '/'), '/', 2))) + OR + (f.dvp IS NOT NULL AND left(f.dvp, 9) <> '/dataset/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 1) + AND d.name = split_part(ltrim(f.dvp, '/'), '/', 2))) + ) + GROUP BY w.wid + ), + updated AS ( + UPDATE workflow w + SET content = jsonb_set(w.content::jsonb, '{operators}', ( + SELECT jsonb_agg( + jsonb_set( + jsonb_set( + op, + '{operatorProperties,fileName}', + CASE + WHEN f.fn ~ '^/datasets/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 2) + AND d.name = split_part(ltrim(f.fn, '/'), '/', 3)) + THEN to_jsonb(regexp_replace(f.fn, '^/datasets/', '/dataset/')) + WHEN f.fn IS NOT NULL AND left(f.fn, 9) <> '/dataset/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 1) + AND d.name = split_part(ltrim(f.fn, '/'), '/', 2)) + THEN to_jsonb('/dataset/' || ltrim(f.fn, '/')) + ELSE COALESCE(op #> '{operatorProperties,fileName}', 'null'::jsonb) + END, + false + ), + '{operatorProperties,datasetVersionPath}', + CASE + WHEN f.dvp ~ '^/datasets/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 2) + AND d.name = split_part(ltrim(f.dvp, '/'), '/', 3)) + THEN to_jsonb(regexp_replace(f.dvp, '^/datasets/', '/dataset/')) + WHEN f.dvp IS NOT NULL AND left(f.dvp, 9) <> '/dataset/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 1) + AND d.name = split_part(ltrim(f.dvp, '/'), '/', 2)) + THEN to_jsonb('/dataset/' || ltrim(f.dvp, '/')) + ELSE COALESCE(op #> '{operatorProperties,datasetVersionPath}', 'null'::jsonb) + END, + false + ) + ORDER BY ord + ) + FROM jsonb_array_elements( + CASE + WHEN jsonb_typeof(w.content::jsonb -> 'operators') = 'array' + THEN w.content::jsonb -> 'operators' + ELSE '[]'::jsonb + END + ) WITH ORDINALITY AS t(op, ord), + LATERAL ( + SELECT op #>> '{operatorProperties,fileName}' AS fn, + op #>> '{operatorProperties,datasetVersionPath}' AS dvp + ) f + ))::text + FROM affected a + WHERE w.wid = a.wid + RETURNING 1 + ) + SELECT count(*) INTO wf_count FROM updated; + + WITH affected AS ( + SELECT wv.vid + FROM workflow_version wv, + jsonb_array_elements( + CASE + WHEN jsonb_typeof(wv.content::jsonb -> 'operators') = 'array' + THEN wv.content::jsonb -> 'operators' + ELSE '[]'::jsonb + END + ) AS op, + LATERAL ( + SELECT op #>> '{operatorProperties,fileName}' AS fn, + op #>> '{operatorProperties,datasetVersionPath}' AS dvp + ) f + WHERE jsonb_typeof(wv.content::jsonb -> 'operators') = 'array' + AND ( + (f.fn ~ '^/datasets/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 2) + AND d.name = split_part(ltrim(f.fn, '/'), '/', 3))) + OR + (f.dvp ~ '^/datasets/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 2) + AND d.name = split_part(ltrim(f.dvp, '/'), '/', 3))) + OR + (f.fn IS NOT NULL AND left(f.fn, 9) <> '/dataset/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 1) + AND d.name = split_part(ltrim(f.fn, '/'), '/', 2))) + OR + (f.dvp IS NOT NULL AND left(f.dvp, 9) <> '/dataset/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 1) + AND d.name = split_part(ltrim(f.dvp, '/'), '/', 2))) + ) + GROUP BY wv.vid + ), + updated AS ( + UPDATE workflow_version wv + SET content = jsonb_set(wv.content::jsonb, '{operators}', ( + SELECT jsonb_agg( + jsonb_set( + jsonb_set( + op, + '{operatorProperties,fileName}', + CASE + WHEN f.fn ~ '^/datasets/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 2) + AND d.name = split_part(ltrim(f.fn, '/'), '/', 3)) + THEN to_jsonb(regexp_replace(f.fn, '^/datasets/', '/dataset/')) + WHEN f.fn IS NOT NULL AND left(f.fn, 9) <> '/dataset/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.fn, '/'), '/', 1) + AND d.name = split_part(ltrim(f.fn, '/'), '/', 2)) + THEN to_jsonb('/dataset/' || ltrim(f.fn, '/')) + ELSE COALESCE(op #> '{operatorProperties,fileName}', 'null'::jsonb) + END, + false + ), + '{operatorProperties,datasetVersionPath}', + CASE + WHEN f.dvp ~ '^/datasets/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 2) + AND d.name = split_part(ltrim(f.dvp, '/'), '/', 3)) + THEN to_jsonb(regexp_replace(f.dvp, '^/datasets/', '/dataset/')) + WHEN f.dvp IS NOT NULL AND left(f.dvp, 9) <> '/dataset/' + AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON d.owner_uid = u.uid + WHERE u.email = split_part(ltrim(f.dvp, '/'), '/', 1) + AND d.name = split_part(ltrim(f.dvp, '/'), '/', 2)) + THEN to_jsonb('/dataset/' || ltrim(f.dvp, '/')) + ELSE COALESCE(op #> '{operatorProperties,datasetVersionPath}', 'null'::jsonb) + END, + false + ) + ORDER BY ord + ) + FROM jsonb_array_elements( + CASE + WHEN jsonb_typeof(wv.content::jsonb -> 'operators') = 'array' + THEN wv.content::jsonb -> 'operators' + ELSE '[]'::jsonb + END + ) WITH ORDINALITY AS t(op, ord), + LATERAL ( + SELECT op #>> '{operatorProperties,fileName}' AS fn, + op #>> '{operatorProperties,datasetVersionPath}' AS dvp + ) f + ))::text + FROM affected a + WHERE wv.vid = a.vid + RETURNING 1 + ) + SELECT count(*) INTO wv_count FROM updated; + + RAISE NOTICE 'Normalized the resource-type path prefix in % workflow and % workflow_version row(s).', wf_count, wv_count; +END $$; + +COMMIT;
