talatuyarer opened a new pull request, #18073:
URL: https://github.com/apache/iceberg/pull/18073
Closes #17830. Supersedes and incorporates #17832 (thanks @waterWang — the
refresh-path fix and its test come from that PR; co-authored).
### Problem
With `snapshot-loading-mode=refs`, lazy snapshot loading only helps readers.
Every commit forced full snapshot loads three ways: `TableMetadata.buildFrom`
eagerly copied `base.snapshots()`, the snapshot-id probes in `SnapshotProducer`
fault-loaded on any unloaded id, and `refresh()` requested metadata without the
`snapshots` param and dropped the lazy supplier (#17830). A single `fastAppend`
transferred the complete snapshot list **four times**, one of which was
downloaded and immediately discarded:
| # | Request | Payload
| Why |
|---|----------------------|------------------------------------------------------------|-----------------------------------------------------------------|
| 1 | `GET /tables/{t}` | all snapshots — **discarded** (metadata
location unchanged) | refresh at the start of `apply()`
|
| 2 | `GET ?snapshots=all` | all snapshots
| lazy supplier fired by `TableMetadata.buildFrom` / id probes |
| 3 | `POST /tables/{t}` | all snapshots (response)
| the commit itself |
| 4 | `GET /tables/{t}` | all snapshots
| post-commit saved-snapshot check |
### After this change
| # | Request | Payload
| Why |
|---|-----------------------|------------------------------------------------|-----------------------------------|
| 1 | `GET ?snapshots=refs` | ref-pointed snapshots only
| refresh at the start of `apply()` |
| 2 | `POST /tables/{t}` | all snapshots (response; installed as current)
| the commit itself |
| 3 | `GET ?snapshots=refs` | ref-pointed snapshots only
| post-commit saved-snapshot check |
The two refreshes are the same calls as before, ***shrunk to refs-sized
responses***. The `snapshots=all` fetch is gone entirely, the builder and id
probes no longer force loading. The `POST` is unchanged; its full response is
installed as current metadata, which is why the post-commit check needs no
extra fetch. Applies to normal and WAP commits alike. New tests pin these exact
request sequences and assert `snapshots=all` is never requested by a commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
