github-actions[bot] commented on code in PR #68879:
URL: https://github.com/apache/doris/pull/68879#discussion_r4254686793
##########
fe/fe-filesystem/fe-filesystem-obs/src/main/java/org/apache/doris/filesystem/obs/ObsFileSystemProvider.java:
##########
@@ -76,14 +80,15 @@ public boolean supportsGuess(Map<String, String>
properties) {
String value = properties.get(name);
if (value != null) {
if (!value.isEmpty()) {
- return value.contains("myhuaweicloud.com");
+ return
StringUtils.containsIgnoreCase(EndpointHosts.hostOf(value),
"myhuaweicloud.com");
Review Comment:
[P2] Normalize the OBS scheme before native client construction. With
`obs.endpoint=HTTPS://OBS.CN-NORTH-4.MYHUAWEICLOUD.COM` and paired OBS
credentials, this new guess selects OBS and binding derives `cn-north-4`, but
the holder preserves the endpoint. `ObsObjStorage` forwards it to the pinned
3.21.11 SDK, whose [ServiceUtils
parser](https://github.com/huaweicloud/huaweicloud-sdk-java-obs/blob/v3.21.11/app/src/main/java/com/obs/services/internal/utils/ServiceUtils.java#L597-L606)
strips only lowercase `http://`/`https://` and then parses everything after
the last colon as a port. It therefore throws `NumberFormatException` on
`//OBS.CN-NORTH-4.MYHUAWEICLOUD.COM` before the first filesystem operation,
while the lowercase endpoint works. Normalize the effective scheme before SDK
construction and cover native client/config creation; the new OBS tests stop at
detection. This is a separate OBS SDK failure from the already-fixed COS suffix
issue.
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/EndpointHosts.java:
##########
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+/**
+ * Host extraction for endpoint / object-storage URI marker matching.
+ *
+ * <p>Cloud dialect detection keys on host markers such as {@code
oss-dls.aliyuncs.com},
+ * {@code myqcloud.com} or {@code amazonaws.com}. Those markers are host
names, so they may only be
+ * matched against the host part: a URI whose object key happens to contain
the same text (for
+ * example {@code s3://bucket/archive/AMAZONAWS.COM/file}) does not point at
that cloud's storage.
+ *
+ * <p>Deliberately string based rather than {@code java.net.URI} based: a
value such as
+ * {@code oss://bucket/my dir/...} does not parse as a URI, and a failed parse
must not silently
+ * turn a real endpoint into "no match".
+ */
+public final class EndpointHosts {
+
+ private EndpointHosts() {
+ }
+
+ /**
+ * Returns the host part of {@code endpointOrUri}: everything after {@code
scheme://}, up to the
+ * first {@code /} that starts the path, with any {@code userinfo@}
removed. A value without a
+ * scheme is a bare host, so it is returned as written (minus any
path/port unaffected parts).
+ */
+ public static String hostOf(String endpointOrUri) {
+ if (endpointOrUri == null) {
+ return "";
+ }
+ String host = endpointOrUri.trim();
+ int schemeIndex = host.indexOf("://");
+ if (schemeIndex >= 0) {
+ host = host.substring(schemeIndex + 3);
+ }
+ int pathIndex = host.indexOf('/');
Review Comment:
[P2] End the authority before processing userinfo. For
`s3.endpoint=https://s3.us-west-2.amazonaws.com#[email protected]`,
`s3.region=us-west-2`, and `s3.credentials_provider_type=INSTANCE_PROFILE`,
this helper returns `minio.internal`: it retains the fragment and mistakes its
`@` for userinfo, discarding the actual AWS host. `bindPrimary` therefore
declines S3, selects MinIO, and fails its HMAC-only credential validation. The
previous AWS match selected S3, and the pinned SDK [accepts and discards
endpoint
fragments](https://github.com/aws/aws-sdk-java-v2/blob/2.29.52/codegen/src/main/resources/software/amazon/awssdk/codegen/rules/ParseUrl.java.resource#L35-L82).
Terminate the authority at the earliest `/`, `?`, or `#` before stripping
userinfo, and cover binding with an AWS-only credential mode. This is a new
false-negative extraction of a real cloud host, distinct from the existing
marker-in-object-key findings.
##########
fe/fe-filesystem/fe-filesystem-oss-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/properties/OssHdfsProperties.java:
##########
@@ -189,12 +194,15 @@ public static boolean guessIsMe(Map<String, String>
props) {
}
return OSS_ENDPOINT_KEY_NAME.stream()
.map(props::get)
- .anyMatch(ep -> StringUtils.isNotBlank(ep) &&
ep.endsWith(OSS_HDFS_ENDPOINT_SUFFIX));
+ .anyMatch(ep -> StringUtils.isNotBlank(ep)
+ &&
ep.toLowerCase(Locale.ROOT).endsWith(OSS_HDFS_ENDPOINT_SUFFIX));
Review Comment:
[P2] Match the OSS-HDFS suffix against the endpoint host. Add
`dlf.catalog.endpoint=https://gateway.internal/proxy/CN-BEIJING.OSS-DLS.ALIYUNCS.COM`
to a valid map with `s3.endpoint=https://s3.us-east-1.amazonaws.com`,
`s3.region=us-east-1`, and paired S3 credentials. This new whole-value
lowercase suffix match selects OSS-HDFS solely from the DLF proxy path.
OSS-HDFS precedes S3 in `bindPrimary`/`bindAll` and its holder rejects the map
because S3 credentials are not its required OSS-HDFS aliases; before this
change the same map binds S3. Extract the host before the suffix test and add
this negative binding case. This is a separate HDFS-family/DLF-alias selector
from the existing native-COS gateway finding; it never calls the host helper
now used by the native cloud providers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
