This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new 714816b4c68 [fix](catalog) Probe an http or https catalog endpoint 
that names no port at its default port (#70364)
714816b4c68 is described below

commit 714816b4c685c2ecb61fc4058cce93d02aebd6ff
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Mon Nov 16 11:27:25 2026 +0800

    [fix](catalog) Probe an http or https catalog endpoint that names no port 
at its default port (#70364)
    
    ### Background
    
    When an operator configures `security_checker_class_name`, CREATE
    CATALOG and ALTER CATALOG ask that checker whether the FE may connect to
    each endpoint the catalog's connector declares through
    `ConnectorSecuritySurface.ofSsrfEndpoints`, and refuse an intranet
    target (#69588). `PluginDrivenExternalCatalog.parseEndpoint` turns each
    declared value into the host and port to probe: `scheme://host:port`, or
    a bare `host:port`. A value it cannot turn into a concrete host and port
    is not probed.
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: #69588 (added the gate)
    PR Type: Bugfix
    
    Problem Summary:
    
    An http or https endpoint written without a port, such as
    `http://10.0.0.1/ns` or `https://10.0.0.1`, passes the gate unprobed,
    although an HTTP client connects to port 80 or 443 for such a URI. Today
    the Hive and MongoDB connectors declare endpoints, as thrift and
    `mongodb://` URIs, so no connector on master reaches this yet; the first
    http endpoint will be the Lance REST catalog's `lance.rest.uri`, and an
    https service URL usually names no port. The gate now probes such an
    endpoint at its scheme's default port.
    
    ### Details
    
    - **Frequency:** Not observed; found by code review while declaring the
    Lance REST catalog's endpoint. Triggers when an SSRF checker is
    configured and a declared endpoint is an http or https URL without a
    port.
    - **Root cause:** `parseEndpoint` returned null for a `scheme://` value
    whose `URI.getPort()` is -1, and `checkEndpointsAgainstSsrf` skips a
    null as nothing to probe.
    - **Why existing tests missed it:** Unit tests:
    `PluginDrivenExternalCatalogSsrfGateTest` covers thrift URIs, bare
    `host:port`, IPv6 and URI suffixes, and asserts that a thrift URI
    without a port is not probed; no case used an http or https URL.
    Regression: no suite reaches the gate, since it needs the operator's
    security checker SDK, which CI does not have; the unit tests use a fake.
    - **Fix:** For an http or https URI that names no port, `parseEndpoint`
    takes 80 or 443, the ports RFC 9110 assigns to the scheme; an explicit
    port still wins. A URI of another scheme without a port is still not
    probed: its default port, if it has one, belongs to the client, and the
    engine knows no connector's client. The fix is in the engine because the
    port follows from the scheme alone.
    - **Other paths and branches:** CREATE (`checkWhenCreating`) and both
    ALTER paths go through `parseEndpoint`, so all are fixed. branch-4.1 and
    branch-4.2 have the same rule in `HMSExternalCatalog.parseEndpoint`, but
    gate only Hive metastore thrift URIs, so they are not affected.
    Introduced by #69588.
    
    ### Testing
    
    Local (macOS), on master `f1bc8b3690c` plus this change:
    
    - `PluginDrivenExternalCatalogSsrfGateTest`: the new
    `parseEndpointUsesTheDefaultPortOfAWebScheme` (seven URIs: with and
    without a path, an upper-case scheme, user-info, query and fragment,
    IPv6, an explicit port) and
    `alterRejectsAnIntranetWebEndpointWithoutAPort` (an http and an https
    endpoint through the real ALTER validation, with `10.0.0.1` flagged as
    intranet). On the code before this change 8 of them fail: the six URIs
    without a port parse to null, and both ALTER cases accept the endpoint
    ("Expected DdlException to be thrown, but nothing was thrown"). With it,
    all 44 cases of the class pass, as do the 18 of
    `PluginDrivenExternalCatalogDriverUrlGateTest`; fe-core checkstyle
    passes.
    
    ### Notes for Reviewers
    
    - The Javadoc of `ConnectorSecuritySurface.ofSsrfEndpoints` still
    describes the value as a `host:port`, optionally scheme-prefixed. It is
    left alone here because #70338, still open, edits that file.
---
 .../plugin/PluginDrivenExternalCatalog.java        | 25 ++++++++++++++++---
 .../PluginDrivenExternalCatalogSsrfGateTest.java   | 29 ++++++++++++++++++++++
 2 files changed, 50 insertions(+), 4 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java
 
b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java
index 95628d1eca6..a51bac974fa 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java
@@ -433,8 +433,9 @@ public class PluginDrivenExternalCatalog extends 
ExternalCatalog {
 
     /**
      * Parse one endpoint — {@code scheme://host:port} (e.g. {@code 
thrift://host:9083}) or a bare
-     * {@code host:port} — into host and port for the SSRF probe. Returns null 
when no concrete host:port
-     * can be extracted (nothing to probe); {@link HostAndPort} handles IPv6 
literals in brackets natively.
+     * {@code host:port} — into host and port for the SSRF probe. An http or 
https URL that names no port is
+     * probed at its scheme's default port. Returns null when no concrete 
host:port can be extracted (nothing to
+     * probe); {@link HostAndPort} handles IPv6 literals in brackets natively.
      */
     static HostAndPort parseEndpoint(String value) {
         if (value == null || value.isEmpty()) {
@@ -445,10 +446,11 @@ public class PluginDrivenExternalCatalog extends 
ExternalCatalog {
                 // Match the HMS client's URI semantics: query, fragment and 
userinfo do not name
                 // the socket destination. HostAndPort.fromParts also strips 
IPv6 brackets.
                 URI uri = new URI(value);
-                if (uri.getHost() == null || uri.getPort() < 0) {
+                int port = uri.getPort() >= 0 ? uri.getPort() : 
defaultWebPort(uri.getScheme());
+                if (uri.getHost() == null || port < 0) {
                     return null;
                 }
-                return HostAndPort.fromParts(uri.getHost(), uri.getPort());
+                return HostAndPort.fromParts(uri.getHost(), port);
             }
             String authority = value;
             int slashIdx = authority.indexOf('/');
@@ -465,6 +467,21 @@ public class PluginDrivenExternalCatalog extends 
ExternalCatalog {
         }
     }
 
+    /**
+     * The port an http or https client connects to when the URI names none, 
80 or 443 (RFC 9110 section 4.2), so a
+     * REST endpoint such as {@code https://host/ns} is probed where it 
connects; -1 for any other scheme, whose
+     * default, if it has one, is its client's.
+     */
+    private static int defaultWebPort(String scheme) {
+        if ("http".equalsIgnoreCase(scheme)) {
+            return 80;
+        }
+        if ("https".equalsIgnoreCase(scheme)) {
+            return 443;
+        }
+        return -1;
+    }
+
     private void checkHiveParquetTimeZone(CatalogProperty property) throws 
DdlException {
         String catalogType = getType();
         if ("hms".equalsIgnoreCase(catalogType) || 
"hudi".equalsIgnoreCase(catalogType)) {
diff --git 
a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogSsrfGateTest.java
 
b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogSsrfGateTest.java
index 0e744332388..5a3934972f6 100644
--- 
a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogSsrfGateTest.java
+++ 
b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogSsrfGateTest.java
@@ -33,6 +33,7 @@ import org.junit.jupiter.api.Assertions;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
 import org.junit.jupiter.params.provider.ValueSource;
 import org.mockito.Mockito;
 
@@ -124,6 +125,21 @@ public class PluginDrivenExternalCatalogSsrfGateTest {
         Assertions.assertEquals(9083, parsed.getPort());
     }
 
+    /**
+     * An http or https client connects to its scheme's default port when the 
URI names none, as a REST catalog
+     * URI such as https://host/ns usually does, so that is the destination to 
probe.
+     */
+    @ParameterizedTest
+    @CsvSource({"http://host1/ns, host1, 80", "http://host1, host1, 80", 
"HTTP://host1/, host1, 80",
+            "https://host1/v1, host1, 443", 
"https://user@host1/v1?ignored#ignored, host1, 443",
+            "http://[::1]/ns, ::1, 80", "https://host1:8443/v1, host1, 8443"})
+    public void parseEndpointUsesTheDefaultPortOfAWebScheme(String endpoint, 
String host, int port) {
+        HostAndPort parsed = 
PluginDrivenExternalCatalog.parseEndpoint(endpoint);
+        Assertions.assertNotNull(parsed, endpoint);
+        Assertions.assertEquals(host, parsed.getHost());
+        Assertions.assertEquals(port, parsed.getPort());
+    }
+
     @ParameterizedTest
     @ValueSource(strings = {"thrift://[::1]:9083", 
"thrift://user@[::1]:9083?ignored#ignored", "[::1]:9083"})
     public void parseEndpointPreservesIpv6Destination(String endpoint) {
@@ -170,6 +186,19 @@ public class PluginDrivenExternalCatalogSsrfGateTest {
         Assertions.assertEquals(Collections.singletonList("10.0.0.1:9083"), 
checker.probed);
     }
 
+    /** A web endpoint that names no port must not slip past the gate: its 
default port is what is probed. */
+    @ParameterizedTest
+    @CsvSource({"http://10.0.0.1/ns, 10.0.0.1:80", "https://10.0.0.1, 
10.0.0.1:443"})
+    public void alterRejectsAnIntranetWebEndpointWithoutAPort(String endpoint, 
String probed) {
+        checker.intranetHost = "10.0.0.1";
+        TestCatalog catalog = new 
TestCatalog(hms("thrift://hms.public.example.com:9083"));
+
+        Assertions.assertThrows(DdlException.class,
+                () -> 
catalog.validatePropertiesBeforeUpdate(catalog.getProperties(),
+                        Collections.singletonMap("uri", endpoint)));
+        Assertions.assertTrue(checker.probed.contains(probed), 
checker.probed.toString());
+    }
+
     @Test
     public void alterChecksEveryEndpointInAnHaUriList() {
         checker.intranetHost = "192.168.0.7";


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to