This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 714816b4c68 [fix](catalog) Probe an http or https catalog endpoint
that names no port at its default port (#70364)
714816b4c68 is described below
commit 714816b4c685c2ecb61fc4058cce93d02aebd6ff
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Mon Nov 16 11:27:25 2026 +0800
[fix](catalog) Probe an http or https catalog endpoint that names no port
at its default port (#70364)
### Background
When an operator configures `security_checker_class_name`, CREATE
CATALOG and ALTER CATALOG ask that checker whether the FE may connect to
each endpoint the catalog's connector declares through
`ConnectorSecuritySurface.ofSsrfEndpoints`, and refuse an intranet
target (#69588). `PluginDrivenExternalCatalog.parseEndpoint` turns each
declared value into the host and port to probe: `scheme://host:port`, or
a bare `host:port`. A value it cannot turn into a concrete host and port
is not probed.
### What problem does this PR solve?
Issue Number: None
Related PR: #69588 (added the gate)
PR Type: Bugfix
Problem Summary:
An http or https endpoint written without a port, such as
`http://10.0.0.1/ns` or `https://10.0.0.1`, passes the gate unprobed,
although an HTTP client connects to port 80 or 443 for such a URI. Today
the Hive and MongoDB connectors declare endpoints, as thrift and
`mongodb://` URIs, so no connector on master reaches this yet; the first
http endpoint will be the Lance REST catalog's `lance.rest.uri`, and an
https service URL usually names no port. The gate now probes such an
endpoint at its scheme's default port.
### Details
- **Frequency:** Not observed; found by code review while declaring the
Lance REST catalog's endpoint. Triggers when an SSRF checker is
configured and a declared endpoint is an http or https URL without a
port.
- **Root cause:** `parseEndpoint` returned null for a `scheme://` value
whose `URI.getPort()` is -1, and `checkEndpointsAgainstSsrf` skips a
null as nothing to probe.
- **Why existing tests missed it:** Unit tests:
`PluginDrivenExternalCatalogSsrfGateTest` covers thrift URIs, bare
`host:port`, IPv6 and URI suffixes, and asserts that a thrift URI
without a port is not probed; no case used an http or https URL.
Regression: no suite reaches the gate, since it needs the operator's
security checker SDK, which CI does not have; the unit tests use a fake.
- **Fix:** For an http or https URI that names no port, `parseEndpoint`
takes 80 or 443, the ports RFC 9110 assigns to the scheme; an explicit
port still wins. A URI of another scheme without a port is still not
probed: its default port, if it has one, belongs to the client, and the
engine knows no connector's client. The fix is in the engine because the
port follows from the scheme alone.
- **Other paths and branches:** CREATE (`checkWhenCreating`) and both
ALTER paths go through `parseEndpoint`, so all are fixed. branch-4.1 and
branch-4.2 have the same rule in `HMSExternalCatalog.parseEndpoint`, but
gate only Hive metastore thrift URIs, so they are not affected.
Introduced by #69588.
### Testing
Local (macOS), on master `f1bc8b3690c` plus this change:
- `PluginDrivenExternalCatalogSsrfGateTest`: the new
`parseEndpointUsesTheDefaultPortOfAWebScheme` (seven URIs: with and
without a path, an upper-case scheme, user-info, query and fragment,
IPv6, an explicit port) and
`alterRejectsAnIntranetWebEndpointWithoutAPort` (an http and an https
endpoint through the real ALTER validation, with `10.0.0.1` flagged as
intranet). On the code before this change 8 of them fail: the six URIs
without a port parse to null, and both ALTER cases accept the endpoint
("Expected DdlException to be thrown, but nothing was thrown"). With it,
all 44 cases of the class pass, as do the 18 of
`PluginDrivenExternalCatalogDriverUrlGateTest`; fe-core checkstyle
passes.
### Notes for Reviewers
- The Javadoc of `ConnectorSecuritySurface.ofSsrfEndpoints` still
describes the value as a `host:port`, optionally scheme-prefixed. It is
left alone here because #70338, still open, edits that file.
---
.../plugin/PluginDrivenExternalCatalog.java | 25 ++++++++++++++++---
.../PluginDrivenExternalCatalogSsrfGateTest.java | 29 ++++++++++++++++++++++
2 files changed, 50 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java
index 95628d1eca6..a51bac974fa 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java
@@ -433,8 +433,9 @@ public class PluginDrivenExternalCatalog extends
ExternalCatalog {
/**
* Parse one endpoint — {@code scheme://host:port} (e.g. {@code
thrift://host:9083}) or a bare
- * {@code host:port} — into host and port for the SSRF probe. Returns null
when no concrete host:port
- * can be extracted (nothing to probe); {@link HostAndPort} handles IPv6
literals in brackets natively.
+ * {@code host:port} — into host and port for the SSRF probe. An http or
https URL that names no port is
+ * probed at its scheme's default port. Returns null when no concrete
host:port can be extracted (nothing to
+ * probe); {@link HostAndPort} handles IPv6 literals in brackets natively.
*/
static HostAndPort parseEndpoint(String value) {
if (value == null || value.isEmpty()) {
@@ -445,10 +446,11 @@ public class PluginDrivenExternalCatalog extends
ExternalCatalog {
// Match the HMS client's URI semantics: query, fragment and
userinfo do not name
// the socket destination. HostAndPort.fromParts also strips
IPv6 brackets.
URI uri = new URI(value);
- if (uri.getHost() == null || uri.getPort() < 0) {
+ int port = uri.getPort() >= 0 ? uri.getPort() :
defaultWebPort(uri.getScheme());
+ if (uri.getHost() == null || port < 0) {
return null;
}
- return HostAndPort.fromParts(uri.getHost(), uri.getPort());
+ return HostAndPort.fromParts(uri.getHost(), port);
}
String authority = value;
int slashIdx = authority.indexOf('/');
@@ -465,6 +467,21 @@ public class PluginDrivenExternalCatalog extends
ExternalCatalog {
}
}
+ /**
+ * The port an http or https client connects to when the URI names none,
80 or 443 (RFC 9110 section 4.2), so a
+ * REST endpoint such as {@code https://host/ns} is probed where it
connects; -1 for any other scheme, whose
+ * default, if it has one, is its client's.
+ */
+ private static int defaultWebPort(String scheme) {
+ if ("http".equalsIgnoreCase(scheme)) {
+ return 80;
+ }
+ if ("https".equalsIgnoreCase(scheme)) {
+ return 443;
+ }
+ return -1;
+ }
+
private void checkHiveParquetTimeZone(CatalogProperty property) throws
DdlException {
String catalogType = getType();
if ("hms".equalsIgnoreCase(catalogType) ||
"hudi".equalsIgnoreCase(catalogType)) {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogSsrfGateTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogSsrfGateTest.java
index 0e744332388..5a3934972f6 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogSsrfGateTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogSsrfGateTest.java
@@ -33,6 +33,7 @@ import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
import org.junit.jupiter.params.provider.ValueSource;
import org.mockito.Mockito;
@@ -124,6 +125,21 @@ public class PluginDrivenExternalCatalogSsrfGateTest {
Assertions.assertEquals(9083, parsed.getPort());
}
+ /**
+ * An http or https client connects to its scheme's default port when the
URI names none, as a REST catalog
+ * URI such as https://host/ns usually does, so that is the destination to
probe.
+ */
+ @ParameterizedTest
+ @CsvSource({"http://host1/ns, host1, 80", "http://host1, host1, 80",
"HTTP://host1/, host1, 80",
+ "https://host1/v1, host1, 443",
"https://user@host1/v1?ignored#ignored, host1, 443",
+ "http://[::1]/ns, ::1, 80", "https://host1:8443/v1, host1, 8443"})
+ public void parseEndpointUsesTheDefaultPortOfAWebScheme(String endpoint,
String host, int port) {
+ HostAndPort parsed =
PluginDrivenExternalCatalog.parseEndpoint(endpoint);
+ Assertions.assertNotNull(parsed, endpoint);
+ Assertions.assertEquals(host, parsed.getHost());
+ Assertions.assertEquals(port, parsed.getPort());
+ }
+
@ParameterizedTest
@ValueSource(strings = {"thrift://[::1]:9083",
"thrift://user@[::1]:9083?ignored#ignored", "[::1]:9083"})
public void parseEndpointPreservesIpv6Destination(String endpoint) {
@@ -170,6 +186,19 @@ public class PluginDrivenExternalCatalogSsrfGateTest {
Assertions.assertEquals(Collections.singletonList("10.0.0.1:9083"),
checker.probed);
}
+ /** A web endpoint that names no port must not slip past the gate: its
default port is what is probed. */
+ @ParameterizedTest
+ @CsvSource({"http://10.0.0.1/ns, 10.0.0.1:80", "https://10.0.0.1,
10.0.0.1:443"})
+ public void alterRejectsAnIntranetWebEndpointWithoutAPort(String endpoint,
String probed) {
+ checker.intranetHost = "10.0.0.1";
+ TestCatalog catalog = new
TestCatalog(hms("thrift://hms.public.example.com:9083"));
+
+ Assertions.assertThrows(DdlException.class,
+ () ->
catalog.validatePropertiesBeforeUpdate(catalog.getProperties(),
+ Collections.singletonMap("uri", endpoint)));
+ Assertions.assertTrue(checker.probed.contains(probed),
checker.probed.toString());
+ }
+
@Test
public void alterChecksEveryEndpointInAnHaUriList() {
checker.intranetHost = "192.168.0.7";
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]