williamhyun opened a new pull request, #17332:
URL: https://github.com/apache/iceberg/pull/17332
## Read tables via catalog-vended pre-signed URLs
Lets execution engines read Iceberg tables entirely through catalog-vended,
ephemeral pre-signed HTTP(S) URLs — bypassing cloud-specific SDKs and
credentials — and refreshes those URLs, both reactively (on a 403/401 mid-read)
and proactively so long-running jobs don't fail partway through.
This PR makes additive REST spec changes mirroring the Delta Sharing model:
pre-signed URLs travel as a `path -> URL` map at the root of the scan-planning
response alongside an expiration hint, while `DataFile`/`FileScanTask`
locations stay untouched as the immutable logical path — no manifest rewriting,
no new fields on `DataFile`/`FileScanTask`.
### REST spec changes (`open-api/rest-catalog-open-api.yaml`)
- `X-Iceberg-Access-Delegation` gains a third option, `pre-signed-urls`,
alongside `vended-credentials` and `remote-signing`.
- `CompletedPlanningResult` (and by inheritance
`CompletedPlanningWithIDResult`) gains two fields, populated only when
`pre-signed-urls` was negotiated:
- `pre-signed-urls`: a map of a file's logical `path` (matching the
`DataFile`/`DeleteFile` it belongs to) to a pre-signed HTTP URL.
- `url-expiration-timestamp-ms`: when those URLs expire.
- New endpoint `POST
/v1/{prefix}/namespaces/{namespace}/tables/{table}/presign` for lightweight,
proactive renewal without re-evaluating manifests or filters: request is
`PresignRequest {plan-id, file-paths}`, response is `PresignResponse
{pre-signed-urls, url-expiration-timestamp-ms}`.
- The pre-existing `remote-signing` `/sign` endpoint and
`RemoteSignRequest`/`RemoteSignResponse` schemas are untouched.
### Implementation
- https://github.com/williamhyun/iceberg/pull/206
---
**AI Disclosure**
- Model: Claude Sonnet 5 (Thinking)
- Platform/Tool: Cursor
- Human Oversight: Fully Reviewed
- Prompt Summary: Implement and iterate on a POC for reading Iceberg tables
via catalog-vended pre-signed URLs (server-side scan planning, `HTTPFileIO`,
reactive and proactive URL refresh), then migrate it to a Delta-Sharing-style
REST spec change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
