mattcasters opened a new issue, #7567:
URL: https://github.com/apache/hop/issues/7567
## Feature request
Add an Apache Commons VFS provider for **Databricks Unity Catalog Volumes**
(and optionally Workspace files), so Hop can browse, read, write, list, and
delete volume paths the same way it does for MinIO, Azure, S3, Google Storage,
etc.
### Motivation
- Modern Databricks workspaces often have **public DBFS root disabled**.
Artifact storage for Jobs / native Spark deploy is typically a **UC Volume**
(`/Volumes/<catalog>/<schema>/<volume>/…`).
- Hop’s **Databricks Job Run** (Deploy & run) already uploads to Volumes via
the **Files API** (`PUT /api/2.0/fs/files/…`). That is a special-case client,
not reusable Hop VFS.
- A proper VFS scheme would allow:
- File dialogs / browse volumes from Hop GUI
- File Input/Output and other VFS-aware transforms/actions against volumes
- Optional reuse for deploy (source fat jar or target volume URIs)
- Consistent auth via existing **Databricks Connection** metadata (host +
PAT)
### Background: API already exists
Databricks exposes volumes and workspace files through the workspace **Files
API**:
- Docs: https://docs.databricks.com/api/workspace/files
- Volume file guide: https://docs.databricks.com/aws/en/volumes/volume-files
| Operation | Endpoint (typical) |
|-----------|-------------------|
| Upload | `PUT /api/2.0/fs/files{path}?overwrite=true`
(`application/octet-stream`) |
| Download | `GET /api/2.0/fs/files{path}` (HTTP Range supported) |
| Create directory | `PUT /api/2.0/fs/directories{path}/` |
| List directory | `GET /api/2.0/fs/directories{path}` |
| Delete | `DELETE` on files/directories |
Auth: Bearer PAT (same as Jobs API / Hop Databricks Connection).
**Note:** Legacy DBFS API (`/api/2.0/dbfs/*`) is separate and often blocked
for public root; VFS should target the **Files API**, not DBFS root.
Hop already implements Volume **upload** in the Databricks tech plugin
(`RestDatabricksJobsClient` / Files API path encoding) for Deploy & run.
### Proposed design (high level)
Align with existing Hop VFS plugins (e.g. MinIO named connections):
1. Extract a small **Databricks Files client** (upload / download / list /
mkdir / delete) shared by Jobs deploy and VFS.
2. Add VFS under `plugins/tech/databricks` (or sibling module):
- `@VfsPlugin` + `IVfs`
- `AbstractOriginatingFileProvider` → FileSystem → FileObject
3. **Auth:** reuse **Databricks Connection** metadata (host, token, optional
API base).
4. **URI scheme (suggested):** connection-named scheme, e.g.
```text
databricks-free:///Volumes/apache-hop/default/jars/hop-native.jar
```
Path = absolute UC/Workspace path. Alternative:
`dbxvol://{connection}/Volumes/...`.
5. Capabilities (MVP): READ/WRITE content, LIST, CREATE/DELETE, type/size;
optional Range GET later.
6. WireMock unit tests; docs under native Spark / Databricks.
### Out of scope (MVP)
- Full Databricks Java SDK (pure `HttpClient` is enough, consistent with
Jobs client)
- Delta table APIs (tabular data)
- Replacing cloud object-store VFS (s3a/abfss) that volumes already sit on
- Spark-side path dialect (`/Volumes/...` on cluster remains separate from
Hop VFS URIs)
### Path dialect note
Hop VFS URIs are for **Hop-side** I/O only. Cluster Spark / Jobs still use
`/Volumes/...` (or cloud URIs). Document this clearly (same class of issue as
Hop `s3://` vs Spark `s3a://`).
### Acceptance criteria
- [ ] Named Databricks Connection can be used as a VFS scheme
- [ ] Browse / list UC Volume directories in Hop file dialogs
- [ ] Read and write files via VFS (including large jars within Files API
limits)
- [ ] Deploy path can optionally share the same Files client (no behaviour
regression for Job Run)
- [ ] Unit tests (WireMock) + short user docs
### Related
- Databricks Job Run Deploy & run (existing Files API upload to Volumes)
- Metadata: Databricks Connection (Jobs API / PAT)
- Prior art: MinIO / WebDAV / Azure VFS plugins in Hop
### Priority / when
Useful after classic-cluster Deploy & run is solid on
Premium/classic-capable workspaces; not blocking first JAR deploy if Volume
base path + Files API upload already work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
