mikebridge opened a new pull request, #45532:
URL: https://github.com/apache/superset/pull/45532
### SUMMARY
Adding semantic views in bulk asks the semantic layer to look up each view
before saving it (#45287). Nothing limited how many views one request could
contain or how long those lookups could run in total, so one request could keep
a web worker busy for as long as the provider took, multiplied by the number of
views.
This change adds two limits to `POST /api/v1/semantic_view/`:
- **Batch size.** `SEMANTIC_VIEW_BULK_CREATE_MAX_VIEWS` (default 100).
Larger requests get a 400 with a message naming both numbers, before any lookup
runs. A `views` value that is not a list of objects is also refused with a 400;
before this change, some of those payloads caused a 500.
- **Time budget.** `SEMANTIC_VIEW_BULK_CREATE_TIMEOUT` (default 30 seconds).
The budget is checked before each view's lookup. Once it has run out, each
remaining view is reported as `{"name", "error", "error_type": "TIMEOUT"}`
without a lookup, and a warning is logged. Views added before that point stay
added, which matches the existing partial-success behaviour (201 if anything
was added, 422 otherwise).
Design notes:
- The budget is checked between lookups only. A lookup that is already
running is limited by the semantic layer's own timeouts; this change does not
cancel provider calls. The worst case is therefore the budget plus one lookup.
The default is kept well below the default 60 s web server timeout so that the
per-view results reach the client.
- The lookup still runs before the duplicate-name check, as in #45287.
- `error_type` is a new, optional key. Existing clients that read only
`name` and `error` are unaffected.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Not applicable (API only).
### TESTING INSTRUCTIONS
- `pytest tests/unit_tests/semantic_layers/api_test.py -k post_semantic_view`
- Manually, with `SEMANTIC_LAYERS` enabled: POST more than
`SEMANTIC_VIEW_BULK_CREATE_MAX_VIEWS` views to `/api/v1/semantic_view/` and
expect a 400 with no provider calls. Set `SEMANTIC_VIEW_BULK_CREATE_TIMEOUT =
0` and expect every view to come back with `error_type: "TIMEOUT"` and status
422.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [x] Required feature flags: `SEMANTIC_LAYERS`
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in
[SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [x] Introduces new feature or API
- [ ] Removes existing feature or API
Follow-up to #45287. The Add Semantic View modal sends every selected view
in one request; with more than 100 selected it shows its generic error. Showing
the server's message, or splitting the request, is left for a follow-up that
can build on #45524.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
