jarredhj0214 commented on PR #11906:
URL: https://github.com/apache/gravitino/pull/11906#issuecomment-4899173874
> Is the bulk interface better when processing data one by one or in
batches? Could you please share your research findings?
Thanks for the review.
For the first question:
I added a new section named "Bulk Access Control Operations" in
`docs/security/access-control.md`.
The new documentation explains:
- Usage scenarios for these APIs, such as team onboarding/offboarding,
synchronizing users and groups from an external identity provider,
bootstrapping roles for a new environment, and cleaning up obsolete principals.
- Required privileges for each bulk API.
- Request body fields for users, groups, and roles.
- Shell examples for all six bulk APIs.
- The response format with `succeeded` and `failed`.
- The processing semantics: validate the whole request first, then process
each item independently after validation succeeds.
For the second question:
I did some research on common batch/bulk API designs. There does not seem to
be a single universal rule, but there are several common patterns.
1. Per-item result with partial success
- Microsoft Graph JSON batching returns one top-level batch response, but
each individual request has its own status. A top-level 200 only means the
batch payload was accepted; clients still need to inspect each item result.
https://learn.microsoft.com/en-us/graph/json-batching
- Microsoft Graph throttling documentation also says requests in a batch are
evaluated individually, and failed requests should be retried individually.
https://learn.microsoft.com/en-us/graph/throttling
- Elasticsearch Bulk API returns a top-level `errors` flag and an `items`
array. Each operation has its own result, and successful operations are not
rolled back when other items fail.
https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk
- Amazon SQS `SendMessageBatch` returns `Successful` and `Failed` lists. AWS
explicitly says the batch can contain both successful and unsuccessful actions,
so clients should check batch errors even when HTTP status is 200.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessageBatch.html
- Criteo documents a similar model: authentication, authorization, or
deserialization errors return 4xx; otherwise the bulk call can return 200
regardless of individual operation success or failure, and clients should
inspect the response errors.
https://developers.criteo.com/marketing-solutions/docs/partial-success-for-bulk-operations
2. Batch request accepted, failed items returned for retry
- Amazon DynamoDB `BatchWriteItem` says each individual put/delete operation
is atomic, but the batch as a whole is not. Failed operations are returned as
`UnprocessedItems`, and clients should retry those items, usually with
exponential backoff.
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html
3. Atomic bulk operation
- Some API guidelines recommend atomic bulk operations. For example, Adidas
API guidelines say a bulk operation should be atomic and should not commit
partial changes if any item fails.
https://github.com/adidas/api-guidelines/blob/master/rest-api-guidelines/execution/batch-operations.md
For Gravitino access control APIs, I think the current design is closer to
the first pattern and fits this feature better:
- We validate the whole request before processing, including empty input,
duplicated names, and authorization.
- If request-level validation fails, no item is processed.
- After the request is valid, each user/group/role operation is processed
independently.
- The response returns successful items in `succeeded` and per-item business
failures in `failed`.
This design avoids partial execution for malformed or unauthorized requests,
while still allowing clients to make progress and retry only failed items for
business-level failures, such as removing a non-existent user.
The current implementation batches at the API contract level and reuses
existing per-item access-control managers internally. This keeps the behavior
consistent with existing single-item APIs, including existing authorization
checks, events, and cache invalidation behavior. Internal storage-level
batching can still be optimized later without changing the public API response
contract.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
