laserninja opened a new pull request, #13608:
URL: https://github.com/apache/gravitino/pull/13608
### What changes were proposed in this pull request?
Implement the server-side scan planning design merged in #12241. Split
completed plans into an inline batch and stateless plan tasks, then resolve
those tasks through the existing `/tasks` endpoint using a pinned snapshot and
deterministic task ordering. Reuse full cached plans and reject malformed,
foreign, out-of-range, or expired plan tasks.
Forward federated plan tasks unchanged, preserve remote residual filters and
delete references, and advertise the fetch endpoint only when the backend
supports it. Document the configuration and behavior.
### Why are the changes needed?
The existing fetch handler rejects every plan task, and large scans return
all tasks in one response. Completing the two-step protocol makes server-side
planning usable by clients that require both endpoints and bounds the file
tasks returned per batch.
Fix: #11284
### Does this PR introduce _any_ user-facing change?
Yes. `POST .../plan` returns up to 100 file scan tasks inline by default,
with remaining batches retrieved through `POST .../tasks`. The new
`gravitino.iceberg-rest.scan-plan-task-batch-size` property controls the batch
size; `0` keeps all tasks inline. Eligible catalogs advertise the fetch
endpoint in `/v1/config`.
### How was this patch tested?
Added HTTP and unit coverage for complete task coverage, retries and events,
duplicate file entries, pinned and incremental snapshots, cache hits and
expired snapshots, replica-independent redemption, delete references, malformed
tokens, integer overflow, configuration, and federation including
case-sensitive residuals.
Validation uses Java 17:
```sh
./gradlew spotlessApply \
:iceberg:iceberg-rest-server:check \
:iceberg:iceberg-common:check \
:catalogs:catalog-common:check -PskipITs
```
All module checks passed: 510 Iceberg REST server tests, 80 Iceberg common
tests, and 39 catalog common tests (629 total; zero failures or skips).
Integration and Docker-tagged tests are excluded from this run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
