laserninja opened a new pull request, #12470: URL: https://github.com/apache/gravitino/pull/12470
### What changes were proposed in this pull request? Phase 1 of the Idempotency-Key design (`design-docs/iceberg-idempotency-key.md`): the pluggable storage SPI, a node-local in-memory store, and header handling on every mutation endpoint. - **SPI** (`iceberg-common`): `IdempotencyStore` (`reserve` / `load` / `finalizeRecord` / `release` / `purgeExpired`), the immutable `IdempotencyRecord`, and `IdempotencyKeys` for RFC 9562 UUIDv7 validation. `reserve` is the only atomicity the server depends on, so a database-backed store drops in behind the same interface. - **`InMemoryIdempotencyStore`**: Caffeine, bounded by the key lifetime and `idempotency-max-entries`, warning when the size bound evicts a record before its reuse window elapses. - **`IcebergIdempotencyManager`**: wraps create/update/drop/register/rename for tables, namespaces, and views. Retries replay the stored response; a key reused for another operation, or one whose first request is still in flight, gets `409 Conflict` (the latter with `Retry-After`); `5xx` releases the key so the client can retry with it. - **Config endpoint**: advertises `idempotency-key-lifetime` only when idempotency is enabled, since the spec reads its absence as "not supported". - Disabled by default behind `gravitino.iceberg-rest.idempotency-enabled`, matching Polaris. Two deliberate deviations from the design document: 1. `idempotency-key-lifetime` is emitted as the **top-level `CatalogConfig` field** the spec defines, not a `defaults` entry as the document's example showed. Iceberg 1.11's `ConfigResponse.Builder.withIdempotencyKeyLifetime()` already supports it. 2. `401`, `403`, and `419` **release** the key rather than finalizing it. The spec finalizes deterministic terminal `4xx`, but auth outcomes depend on the caller's credentials rather than catalog state, so replaying them would keep failing a retry that has since obtained a valid token. Known limitation, documented in the manager's Javadoc: a replayed response carries the original status and body but not headers, so a replayed create/update does not carry the `ETag` the first response did. Phase 2 (the durable, multi-node JDBC store on Gravitino's relational backend) follows in a separate PR. ### Why are the changes needed? Epic #12049. Without it, a client retry after a network failure re-executes the mutation and produces duplicate table/namespace/view creates and drops. Supersedes #10751, which used a node-local Caffeine cache with no storage seam. Fix: #12049 ### Does this PR introduce _any_ user-facing change? Yes. - Clients can send `Idempotency-Key: <UUIDv7>` on Iceberg REST mutation requests for safe retries. - `GET /v1/config` includes `idempotency-key-lifetime` when the feature is enabled. - New properties, all optional: `gravitino.iceberg-rest.idempotency-enabled` (`false`), `idempotency-key-lifetime` (`PT30M`), `idempotency-store-type` (`in-memory`), `idempotency-max-entries` (`10000`). Default behavior is unchanged: with the feature off, the header is ignored and nothing is advertised. ### How was this patch tested? New unit tests (42): - `TestIdempotencyKeys` - accepts UUIDv7 in both cases and across the RFC 9562 variant range; rejects UUIDv1/v4, the NCS and Microsoft variants, the nil UUID, non-canonical lengths, and non-hex input. - `TestInMemoryIdempotencyStore` - single-winner `reserve` under 16 concurrent threads, `load` empty until finalized, `release` freeing a reservation but preserving a finalized record, no resurrection of a purged record, `purgeExpired` removing only elapsed rows, size-bound eviction. - `TestIcebergIdempotencyManager` - replay, distinct keys, bodiless `204` replay, terminal `4xx` replay, `5xx`/`401`/thrown-exception release, invalid key rejected without executing, operation-binding mismatch, in-flight `409` with `Retry-After`, query-parameter-sorted bindings. - `TestIcebergIdempotencyKey` - end-to-end through Jersey: a replayed create fires no second `IcebergCreateTableEvent`, a fresh key re-executes into `409 AlreadyExists`, a stored `409` replays, a key reused on `DELETE` is rejected and the table survives, a drop replays `204` where a retry would otherwise `404`, and `GET /v1/config` carries `PT30M`. - `TestIcebergConfig` - the lifetime is absent when the feature is disabled. Full suites pass on JDK 17: `iceberg-common` 91 tests, `iceberg-rest-server` 418 tests, 0 failures. ```bash ./gradlew :iceberg:iceberg-common:test :iceberg:iceberg-rest-server:test -PskipITs ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
