tanishqgandhi1908 opened a new pull request, #7622:
URL: https://github.com/apache/texera/pull/7622
### What changes were proposed in this PR?
#6502 made the `datasets` resource-type prefix **required** on dataset
logical paths. This PR makes the readers accept **both** forms again, so the
prefix becomes required only once the ML-model work has landed and every stored
path has been migrated.
prefixed (target): /datasets/<owner>/<name>/<version>/<file>
legacy (accepted): /<owner>/<name>/<version>/<file>
Motivation: `sql/updates/36.sql` rewrites the `fileName` and
`datasetVersionPath` operator properties, but it cannot rewrite a path a user
hardcoded inside a Python UDF (that lives in the operator's `code` property).
Those paths worked before #6502 and started raising afterwards. Rewriting user
source in a migration would be unsafe, so the readers tolerate the legacy form
during the transition instead.
Disambiguation: a leading segment that names a known `ResourceType` commits
to the prefixed form, so `/datasets/<owner>/<name>/<version>` (too few
segments) is rejected rather than silently re-read as a legacy path with owner
`datasets`.
Each tolerant branch carries a `TODO(datasets-prefix)` marker so the
fallback can be removed in one pass.
### Any related issues, documentation, discussions?
Follow-up to #6502 Part of the ML-model resource work tracked in #6495.
### How was this PR tested?
Updated and added unit tests, all passing locally:
Also verified end-to-end against a local instance, using workflows whose
stored paths had the prefix stripped:
- CSV File Scan on `/<owner>/reviews/v1/reviews.csv` ran green and emitted
rows, as did two further scan workflows.
- A Python UDF calling `DatasetFileDocument("/<owner>/iris/v2/Iris.csv")`
parsed the legacy path successfully
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
