mikebridge opened a new pull request, #45553:
URL: https://github.com/apache/superset/pull/45553
### SUMMARY
**TL;DR:** Give semantic-layer extensions a host-provided HTTP transport
with a mandatory deadline and response byte cap. Keep destination policy in
Superset.
**Background:** Providers need HTTP during construction, schema discovery
and query execution. The SDK currently has no portable HTTP capability, so
provider helpers can depend on a particular host's private code.
**Problem:** Moving those helpers without preserving address pinning, TLS,
redirect checks and absolute deadlines would weaken the existing transport. A
timeout on an individual read does not bound DNS, slow framing or a full
multi-request operation.
**What this changes:**
- Add library-neutral SDK request-budget, response, transport and error
types.
- Require a separate `transport=` keyword on provider construction and both
schema-discovery entry points. Update all five host call sites and in-tree
providers/fixtures together; keep static read-only schema access and memo
identity unchanged.
- Resolve once, reject any unsafe answer, connect to a checked numeric
address, and retain the original TLS hostname. GET follows at most three
same-origin redirects. Error and refused-redirect bodies are never read.
- Bound DNS waiting, connect, TLS, headers, framing and body reads by one
monotonic deadline, using a lazy shared resolver and socket deadline scheduler.
Accept identity encoding only and enforce a response-byte cap on every call.
- Declare `SEMANTIC_LAYER_ALLOW_INTERNAL_HOSTS = False`. Opting in permits
internal/HTTP endpoints but retains budgets, TLS verification, redirect checks
and direct connections without environment proxies. Internal mode honors
`REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` for private TLS certificates; neither
mode discovers `.netrc` credentials.
- Include sanitized provider display names in errors/logs, reset lazy
workers after fork, remove completed deadlines lazily, and retain urllib3
socket options. Non-string provider labels use a safe fallback without coercion
or leaking a URL-bearing exception chain.
- Preserve safe transport messages in metadata API failures while keeping
fixed fallback messages for other provider exceptions.
- Add provider-author documentation and an upgrade note for the required
keyword.
This prepares the unreleased SDK contract. Existing downstream provider
bundles must migrate their HTTP helpers and advance their core pin together in
a separate change. This PR does not claim that old bundles work against the new
signatures.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Not applicable: no UI change.
### TESTING INSTRUCTIONS
Run from this checkout using its development environment:
```bash
PYTHONPATH=.:superset-core/src python -m pytest -q -o addopts='' \
tests/unit_tests/semantic_layers tests/unit_tests/commands/semantic_layer \
superset-core/tests tests/unit_tests/utils/test_network.py
```
The tests use local listeners and fakes, with an explicit loopback-only
socket check in the transport suite. They cover all 33 original transport test
functions, plus complete-answer DNS rejection, no second resolution, every
stalled phase, exact/exceeded byte caps, safe logs, unread errors/redirects,
and ten concurrent requests with descriptor cleanup. The connect-stall fixture
blocks real socket I/O at the connect boundary; it is not a kernel
SYN-blackhole experiment.
The same phase/concurrency cases are also exercised in a fresh subprocess
that calls `gevent.monkey.patch_all()` before importing pytest. A real Apache
app factory boot loads the updated local stub against disposable SQLite; no
live provider, real credentials, shared database or container is used.
Final candidate validation: **1,901 semantic-layer tests passed with 100.00%
coverage**, **53 SDK tests passed**, and **7 SDK conformance registration tests
passed**. Changed-file hooks including MyPy passed. Earlier transport
validation also passed 16 isolated-gevent cases; those were not rerun for the
dead-code removal and master merge. All 33 reference transport test functions
are ported. New regression tests cover closed deadlines, shutdown errors, fork
reset and invalid URLs.
Follow-up: harden the standalone `ProviderRequestError` constructor against
string/integer subclasses that override formatting, and validate status fields.
The stacked host boundary validates these fields independently; this
constructor hardening is not included here.
### ADDITIONAL INFORMATION
- [ ] Has associated issue
- [x] Required feature flags: existing `SEMANTIC_LAYERS` availability gate;
provider destination opt-out is a default-off operator configuration setting.
- [ ] Changes UI
- [ ] Includes DB Migration
- [x] Introduces new feature or API: SDK `ProviderTransport` and required
keyword.
- [x] Removes existing feature or API: old entry-point signatures no longer
apply.
Compatibility: this is a first-release SDK gate. Update provider bundles in
lockstep with the host SDK before enabling them. Rollback requires the matching
host/bundle pair; this change makes no metadata database writes or migrations.
OAuth and webhook transport settings are unaffected. OS DNS calls already in
progress cannot be cancelled; caller waiting is bounded and the resolver uses
at most eight workers. Independent security review completed; remote CI and
downstream bundle/query acceptance are still pending. No live-provider
acceptance is claimed.
### Port coverage
All Apache targets are in
`tests/unit_tests/semantic_layers/transport_test.py`. Proxy behavior and error
wording changes below are intentional; every call is bounded.
| Shell case | Apache case | Contract disposition |
|---|---|---|
| `test_https_guard_ignores_background_requests` |
`test_https_guard_ignores_background_requests` | Same outcome; SDK adapter
supplies deadline and byte cap. |
| `test_internal_address_is_refused_before_connecting` |
`test_internal_address_is_refused_before_connecting` | Same outcome; SDK
adapter supplies deadline and byte cap. |
| `test_any_internal_answer_refuses_the_host` |
`test_any_internal_answer_refuses_the_host` | Same outcome; SDK adapter
supplies deadline and byte cap. |
| `test_translated_ipv6_answers_are_checked_as_ipv4` |
`test_translated_ipv6_answers_are_checked_as_ipv4` | Same outcome; SDK adapter
supplies deadline and byte cap. |
| `test_checked_connection_ignores_background_connections` |
`test_checked_connection_ignores_background_connections` | Same outcome; SDK
adapter supplies deadline and byte cap. |
| `test_public_tls_ignores_background_loopback_lookups` |
`test_public_tls_ignores_background_loopback_lookups` | Same outcome; SDK
adapter supplies deadline and byte cap. |
| `test_default_path_verifies_the_configured_hostname` |
`test_default_path_verifies_the_configured_hostname` | Same outcome; SDK
adapter supplies deadline and byte cap. |
| `test_default_path_follows_same_origin_get_redirects_only` |
`test_default_path_follows_same_origin_get_redirects_only` | Same outcome; SDK
adapter supplies deadline and byte cap. |
| `test_flag_allows_local_http_development_endpoints` |
`test_flag_allows_local_http_development_endpoints` | Same outcome; SDK adapter
supplies deadline and byte cap. |
| `test_flag_off_keeps_local_endpoints_refused` |
`test_flag_off_keeps_local_endpoints_refused` | Same outcome; SDK adapter
supplies deadline and byte cap. |
| `test_post_redirects_are_not_followed` |
`test_post_redirects_are_not_followed` | Same outcome; SDK adapter supplies
deadline and byte cap. |
| `test_redirect_chains_are_bounded` | `test_redirect_chains_are_bounded` |
Same outcome; SDK adapter supplies deadline and byte cap. |
| `test_error_message_carries_status_only` |
`test_error_message_carries_provider_and_status_only` | Sanitized provider
display name and status in errors; provider/status/category-only logs; non-2xx
bodies unread. |
| `test_transport_failure_hides_the_destination` |
`test_transport_failure_hides_the_destination` | Same outcome; SDK adapter
supplies deadline and byte cap. |
| `test_failure_logs_never_carry_url_secrets` |
`test_failure_logs_never_carry_url_secrets` | Sanitized provider display name
and status in errors; provider/status/category-only logs; non-2xx bodies
unread. |
| `test_malformed_redirects_raise_the_safe_error` |
`test_malformed_redirects_raise_the_safe_error` | Sanitized provider display
name and status in errors; provider/status/category-only logs; non-2xx bodies
unread. |
| `test_malformed_endpoint_urls_raise_the_safe_error` |
`test_malformed_endpoint_urls_raise_the_safe_error` | Sanitized provider
display name and status in errors; provider/status/category-only logs; non-2xx
bodies unread. |
| `test_bounded_body_is_read_through_the_checked_connection` |
`test_bounded_body_is_read_through_the_checked_connection` | Same outcome; SDK
adapter supplies deadline and byte cap. |
| `test_bounded_body_is_refused_past_its_byte_cap` |
`test_bounded_body_is_refused_past_its_byte_cap` | Same outcome; SDK adapter
supplies deadline and byte cap. |
| `test_bounded_body_stops_at_the_deadline_while_dripping` |
`test_bounded_body_stops_at_the_deadline_while_dripping` | Same outcome; SDK
adapter supplies deadline and byte cap. |
| `test_bounded_read_after_progress_waits_only_for_the_time_left` |
`test_bounded_read_after_progress_waits_only_for_the_time_left` | Same outcome;
SDK adapter supplies deadline and byte cap. |
| `test_bounded_lookup_waits_no_longer_than_the_deadline` |
`test_bounded_lookup_waits_no_longer_than_the_deadline` | Same outcome; SDK
adapter supplies deadline and byte cap. |
| `test_bounded_body_refuses_internal_destinations` |
`test_bounded_body_refuses_internal_destinations` | Same outcome; SDK adapter
supplies deadline and byte cap. |
| `test_bounded_error_bodies_are_not_read_or_logged` |
`test_bounded_error_bodies_are_not_read_or_logged` | Sanitized provider display
name and status in errors; provider/status/category-only logs; non-2xx bodies
unread. |
| `test_bounded_body_refuses_encoded_responses` |
`test_bounded_body_refuses_encoded_responses` | Same outcome; SDK adapter
supplies deadline and byte cap. |
| `test_bounded_post_redirects_are_not_followed` |
`test_bounded_post_redirects_are_not_followed` | Same outcome; SDK adapter
supplies deadline and byte cap. |
| `test_bounded_body_refuses_an_expired_deadline` |
`test_bounded_body_refuses_an_expired_deadline` | Same outcome; SDK adapter
supplies deadline and byte cap. |
| `test_bounded_redirect_bodies_are_never_read` |
`test_bounded_redirect_bodies_are_never_read` | Same outcome; SDK adapter
supplies deadline and byte cap. |
| `test_bounded_framing_obeys_the_absolute_deadline` |
`test_bounded_framing_obeys_the_absolute_deadline` | Same outcome; SDK adapter
supplies deadline and byte cap. |
| `test_bounded_lookup_and_connect_attempts_share_the_deadline` |
`test_bounded_lookup_and_connect_attempts_share_the_deadline` | Same outcome;
SDK adapter supplies deadline and byte cap. |
| `test_bounded_internal_requests_ignore_environment_proxies` |
`test_bounded_internal_requests_ignore_environment_proxies` | Same outcome; SDK
adapter supplies deadline and byte cap. |
| `test_unbounded_internal_requests_keep_environment_proxies` |
`test_former_unbounded_internal_requests_ignore_environment_proxies` |
Intentional policy change: formerly unbounded opt-out also ignores environment
proxies; direct loopback succeeds. |
| `test_dns_failure_is_not_an_address_policy_refusal` |
`test_dns_failure_is_not_an_address_policy_refusal` | Same outcome; SDK adapter
supplies deadline and byte cap. |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
