laserninja commented on PR #12194: URL: https://github.com/apache/gravitino/pull/12194#issuecomment-5109468146
@lasdf1234 design document written up as a separate PR so it can be reviewed on its own: https://github.com/apache/gravitino/pull/12241 It covers the `plan-task` token (payload, encoding, validation, and why it is not a capability), the new `POST .../tasks` interface, what changes about `POST .../plan` and the migration impact per client, the internal dispatcher interfaces added, and the determinism the token depends on - the planned snapshot is pinned and planned tasks are totally ordered, because Iceberg plans manifests in parallel and an index range only identifies tasks if the plan is reproducible. It also records the alternatives I rejected, with reasons: an in-memory plan-task store keyed by a random id (a token dies with the process and is meaningless on another replica, and the Iceberg client fetches plan tasks concurrently, so a scan would fail for timing reasons), and asynchronous planning with server-side plan state (that is #11635's scope and does not remove the need for `.../tasks`). Three questions in ยง8 are genuinely open and I would value your opinion, since they are design calls rather than implementation details: 1. Should batching require the scan plan cache to be enabled? With the cache off (today's default) an N-batch plan costs N full plans, since a token is resolved by replanning its pinned snapshot. The alternative is to keep everything inline unless a cache is configured. 2. Is 100 the right default? It matches the Iceberg side, but it makes a 100,000-file scan 1,000 requests. Sizing batches by serialized bytes rather than task count may serve large tables better. 3. Should tokens carry an expiry? Today a token is valid as long as its snapshot exists. Happy to fold any of these back into #12194. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
