lasdf1234 commented on code in PR #12194:
URL: https://github.com/apache/gravitino/pull/12194#discussion_r3662630664
##########
docs/iceberg-rest-service.md:
##########
@@ -697,13 +697,20 @@ Gravitino provides the build-in
`org.apache.gravitino.iceberg.common.cache.Local
Gravitino caches scan plan results to speed up repeated queries with identical
parameters. The cache uses snapshot ID as part of the cache key, so queries
against different snapshots will not use stale cached data.
-Plan scan responses follow the Iceberg 1.11 REST API: completed plans return
structured `file-scan-tasks` only. Legacy `plan-tasks` JSON strings (used by
some Iceberg 1.9.x–1.10.x clients) are not emitted.
+Plan scan responses follow the Iceberg 1.11 REST API: `file-scan-tasks` are
returned as structured tasks rather than as the JSON strings used by some
Iceberg 1.9.x–1.10.x clients.
-| Configuration item | Description
| Default value | Required | Since
Version |
-|------------------------------------------------------------|----------------------------------------------------------|---------------|----------|---------------|
-| `gravitino.iceberg-rest.scan-plan-cache-impl` | The
implementation of the scan plan cache. | (none) | No
| 1.2.0 |
-| `gravitino.iceberg-rest.scan-plan-cache-capacity` | The capacity of
the scan plan cache. | 200 | No | 1.2.0
|
-| `gravitino.iceberg-rest.scan-plan-cache-expire-minutes` | The expiration
time (in minutes) of the scan plan cache. | 60 | No | 1.2.0
|
+Scan planning is synchronous: `POST
/v1/{prefix}/namespaces/{namespace}/tables/{table}/plan` always returns status
`COMPLETED`, never `SUBMITTED`, so a plan is never left running in the
background.
+
+A plan is handed to the client in batches of at most
`scan-plan-task-batch-size` file scan tasks. The first batch is returned inline
in the plan response; each remaining batch is offered as a `plan-task` token
that the client exchanges for its tasks through `POST
/v1/{prefix}/namespaces/{namespace}/tables/{table}/tasks`, the second step of
the Iceberg REST scan planning protocol. A plan that fits in one batch carries
no `plan-task` tokens, so most scans complete in a single round trip.
+
+`plan-task` tokens are self-describing: each one carries the scan it was
planned from, with the snapshot pinned at planning time, plus the range of
tasks it covers. Nothing is stored server side between the two calls, so a
token stays redeemable after a server restart and on any Gravitino instance
serving the same catalog. Redeeming a token replans the pinned snapshot unless
the plan is still in the scan plan cache, so enabling the cache is recommended
when planning large tables. A token that this server did not issue, was issued
for another table, or refers to a plan that can no longer be reproduced (for
example because its snapshot expired) returns `404` with a
`NoSuchPlanTaskException` error.
+
+| Configuration item | Description
| Default value | Required | Since Version |
+|------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|---------------|
+| `gravitino.iceberg-rest.scan-plan-cache-impl` | The
implementation of the scan plan cache.
| (none) | No | 1.2.0 |
+| `gravitino.iceberg-rest.scan-plan-cache-capacity` | The capacity of
the scan plan cache.
| 200 | No | 1.2.0 |
+| `gravitino.iceberg-rest.scan-plan-cache-expire-minutes` | The expiration
time (in minutes) of the scan plan cache.
| 60 | No | 1.2.0 |
+| `gravitino.iceberg-rest.scan-plan-task-batch-size` | Maximum number
of file scan tasks returned inline by one scan planning response. Tasks beyond
this limit are offered as `plan-task` tokens. Set to 0 to always return every
task inline. | 1000 | No | 1.3.0 |
Review Comment:
Why is it 1000? Is this value setting not quite reasonable? The
implementation of iceberg itself is 100. @roryqi WDYT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
