kevinjqliu opened a new issue, #3495:
URL: https://github.com/apache/iceberg-python/issues/3495
### Summary
PyIceberg currently supports inline REST `storage-credentials` on table
responses, but does not appear to support all vended credential paths defined
by the current REST Catalog OpenAPI spec.
### Current support
- The REST catalog sends `X-Iceberg-Access-Delegation: vended-credentials`
by default.
- `LoadTableResult` / `TableResponse` can parse `storage-credentials`.
- Table construction applies the longest-prefix matching credential config
before falling back to response `config`.
- This covers the common inline credential path for `loadTable`,
`createTable`, and `registerTable` responses.
### Gaps to track
- Implement the explicit `loadCredentials` endpoint:
- `GET /v1/{prefix}/namespaces/{namespace}/tables/{table}/credentials`
- Parse `LoadCredentialsResponse` and expose/apply returned
`storage-credentials`.
- Support optional `planId` and `referenced-by` query parameters where
applicable.
- Apply `storage-credentials` returned from completed server-side scan
planning responses:
- `planTableScan` completed responses can include credentials for the
returned `FileScanTask`s.
- PyIceberg currently parses these credentials, but `plan_scan` converts
REST tasks without threading credentials into the FileIO used to read task
files.
- Support async scan-planning credential paths:
- `fetchPlanningResult` completed responses can also include
`storage-credentials`.
- PyIceberg currently raises `NotImplementedError` for submitted async
scan planning and has no `fetchPlanningResult` flow.
- Revisit staged create behavior:
- The spec allows `metadata-location` to be null for staged table metadata.
- Current credential resolution is based on `metadata-location`, which may
skip credentials whose prefix matches the staged table location instead.
### Acceptance criteria
- Add REST models and catalog methods for `LoadCredentialsResponse` /
`loadCredentials`.
- Add tests proving longest-prefix credential selection works for explicit
`loadCredentials` responses.
- Add tests proving completed scan-planning `storage-credentials` are used
when reading planned data/delete files.
- Add tests for `fetchPlanningResult` once async scan planning is
implemented, including credential handling.
- Preserve existing inline table-response credential precedence:
`storage-credentials` should win over `config` for matching storage locations.
### References
- REST OpenAPI: `open-api/rest-catalog-open-api.yaml`
- Local implementation areas:
- `pyiceberg/catalog/rest/__init__.py`
- `pyiceberg/catalog/rest/scan_planning.py`
- `tests/catalog/test_rest.py`
- `tests/catalog/test_scan_planning_models.py`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
