github-actions[bot] commented on code in PR #70725:
URL: https://github.com/apache/doris/pull/70725#discussion_r4617241505


##########
fe/fe-connector/fe-connector-lance/src/main/java/org/apache/doris/connector/lance/LanceStorageOptions.java:
##########
@@ -79,17 +82,36 @@ private static Map<String, String> 
buildStorageOptions(String datasetUri,
         Map<String, String> normalizedVended = new HashMap<>();
         if (vendedOptions != null && !vendedOptions.isEmpty()) {
             vendedOptions.forEach(LanceStorageOptions::validateVendedOption);
-            // Safe to validate before normalizing: normalization only renames 
a key to a provider
-            // constant or passes it through, so it cannot introduce a NUL 
missed above.
+            // Normalization renames keys to provider constants and may fill 
an absent session token
+            // with an empty string, so it cannot introduce a NUL missed by 
validation above.
             normalizedVended = 
provider.normalizeVendedStorageOptions(vendedOptions);
         }
         result.putAll(normalizedVended);

Review Comment:
   [P2] Reconcile the inherited S3 endpoint after merging the addressing mode
   
   A REST catalog with `s3.endpoint=http://minio:9000` and 
`use_path_style=true` can now receive `virtual_hosted_style_request=true` from 
DescribeTable while retaining the catalog endpoint. The merge produces a 
virtual-hosted request with a bucketless endpoint: the pinned object_store 
builder uses an explicit endpoint verbatim in that mode, so 
`s3://warehouse/...` requests go to the service root instead of the warehouse 
bucket. The reverse style change duplicates the bucket in host and path. 
`vendedCredentialsSupersedeTheCatalogsOnS3` currently asserts the first 
inconsistent map. Convert an inherited catalog endpoint using the final merged 
addressing mode, and verify the resulting native host/path for both style 
changes.



##########
fe/fe-connector/fe-connector-lance/src/main/java/org/apache/doris/connector/lance/LanceOssStorageProvider.java:
##########
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.lance;
+
+import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties;
+import org.apache.doris.filesystem.properties.StorageProperties;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+/** Alibaba Cloud OSS storage, which Lance reaches through its OpenDAL OSS 
provider. */
+final class LanceOssStorageProvider implements LanceStorageProvider {
+
+    static final LanceOssStorageProvider INSTANCE = new 
LanceOssStorageProvider();
+
+    /** The provider name of the engine's OSS storage binding; OSS reports the 
S3 family like S3 itself. */
+    private static final String OSS_PROVIDER = "OSS";
+
+    private static final String ENDPOINT = "oss_endpoint";
+    private static final String ACCESS_KEY_ID = "oss_access_key_id";
+    private static final String SECRET_ACCESS_KEY = "oss_secret_access_key";
+    private static final String REGION = "oss_region";
+    private static final String SECURITY_TOKEN = "oss_security_token";
+    private static final String ADDRESSING_STYLE = "addressing_style";
+    private static final String SKIP_SIGNATURE = "skip_signature";
+    private static final String ALLOW_ANONYMOUS = "allow_anonymous";
+
+    /**
+     * Lance exposes the {@code oss_*} names as its public storage-option 
vocabulary and normalizes
+     * them to OpenDAL's field names before constructing the operator. Both 
spellings are accepted,
+     * so collapse only these known pairs before merging static and 
namespace-vended options.
+     */
+    private static final Map<String, String> PUBLIC_BY_ALIAS = Map.ofEntries(
+            Map.entry("endpoint", ENDPOINT),
+            Map.entry(ENDPOINT, ENDPOINT),
+            Map.entry("access_key_id", ACCESS_KEY_ID),
+            Map.entry(ACCESS_KEY_ID, ACCESS_KEY_ID),
+            Map.entry("access_key_secret", SECRET_ACCESS_KEY),
+            Map.entry(SECRET_ACCESS_KEY, SECRET_ACCESS_KEY),
+            Map.entry("region", REGION),
+            Map.entry(REGION, REGION),
+            Map.entry("security_token", SECURITY_TOKEN),
+            Map.entry(SECURITY_TOKEN, SECURITY_TOKEN),
+            // OpenDAL 0.57 renamed this option to skip_signature, but lance-c 
0.1.6 still uses
+            // OpenDAL 0.56. Emit the old spelling, which newer OpenDAL keeps 
as an alias.
+            Map.entry(SKIP_SIGNATURE, ALLOW_ANONYMOUS),
+            Map.entry(ALLOW_ANONYMOUS, ALLOW_ANONYMOUS));
+
+    private LanceOssStorageProvider() {
+    }
+
+    @Override
+    public Map<String, String> normalizeDorisStorageOptions(
+            String datasetUri, List<StorageProperties> storageProperties) {
+        Map<String, String> result = new HashMap<>();
+        S3CompatibleFileSystemProperties properties = 
selectOss(storageProperties);
+        if (properties == null) {
+            return result;
+        }
+        if (properties.hasAssumeRole()) {
+            throw new IllegalArgumentException("Lance OSS AssumeRole access is 
not supported yet");
+        }
+        LanceStorageOptions.putIfNotEmpty(result, ENDPOINT, 
properties.getEndpoint());
+        LanceStorageOptions.putIfNotEmpty(result, REGION, 
properties.getRegion());
+        LanceStorageOptions.putIfNotEmpty(result, ACCESS_KEY_ID, 
properties.getAccessKey());
+        LanceStorageOptions.putIfNotEmpty(result, SECRET_ACCESS_KEY, 
properties.getSecretKey());
+        LanceStorageOptions.putIfNotEmpty(result, SECURITY_TOKEN, 
properties.getSessionToken());
+
+        // Lance snapshots the host's OSS_*/AWS_*/ALIBABA_CLOUD_* environment 
into the same config
+        // map before storage options are applied, so state both addressing 
styles explicitly the
+        // way the S3 provider does. Leaving the default implicit would let an 
exported
+        // OSS_ADDRESSING_STYLE outrank an explicit oss.use_path_style=false.
+        String usePathStyle = properties.getUsePathStyle();
+        if (StringUtils.isNotEmpty(usePathStyle)) {
+            result.put(ADDRESSING_STYLE, properties.isUsePathStyle() ? "path" 
: "virtual");
+        }
+        return result;
+    }
+
+    @Override
+    public Map<String, String> normalizeVendedStorageOptions(Map<String, 
String> options) {
+        return LanceStorageOptions.normalizeAliases(options, PUBLIC_BY_ALIAS);

Review Comment:
   [P2] Clear the previous OSS token when replacing its key pair
   
   When the catalog has OSS session credentials and DescribeTable vends a 
complete permanent key pair without `security_token`, this normalizer replaces 
only the keys. The merged map retains the catalog's `oss_security_token`, so 
OpenDAL signs the new identity with the previous identity's token and otherwise 
valid reads fail authentication. With no catalog token, Lance can also import 
an unrelated `OSS_SECURITY_TOKEN` or `ALIBABA_CLOUD_SECURITY_TOKEN` from the 
FE/BE environment. Apply the same identity reset as the S3 normalizer: 
explicitly clear a missing token for a complete replacement pair, and prevent a 
permanent catalog pair from inheriting an ambient token.



##########
fe/fe-connector/fe-connector-lance/src/main/java/org/apache/doris/connector/lance/LanceOssStorageProvider.java:
##########
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.lance;
+
+import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties;
+import org.apache.doris.filesystem.properties.StorageProperties;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+/** Alibaba Cloud OSS storage, which Lance reaches through its OpenDAL OSS 
provider. */
+final class LanceOssStorageProvider implements LanceStorageProvider {
+
+    static final LanceOssStorageProvider INSTANCE = new 
LanceOssStorageProvider();
+
+    /** The provider name of the engine's OSS storage binding; OSS reports the 
S3 family like S3 itself. */
+    private static final String OSS_PROVIDER = "OSS";
+
+    private static final String ENDPOINT = "oss_endpoint";
+    private static final String ACCESS_KEY_ID = "oss_access_key_id";
+    private static final String SECRET_ACCESS_KEY = "oss_secret_access_key";
+    private static final String REGION = "oss_region";
+    private static final String SECURITY_TOKEN = "oss_security_token";
+    private static final String ADDRESSING_STYLE = "addressing_style";
+    private static final String SKIP_SIGNATURE = "skip_signature";
+    private static final String ALLOW_ANONYMOUS = "allow_anonymous";
+
+    /**
+     * Lance exposes the {@code oss_*} names as its public storage-option 
vocabulary and normalizes
+     * them to OpenDAL's field names before constructing the operator. Both 
spellings are accepted,
+     * so collapse only these known pairs before merging static and 
namespace-vended options.
+     */
+    private static final Map<String, String> PUBLIC_BY_ALIAS = Map.ofEntries(
+            Map.entry("endpoint", ENDPOINT),
+            Map.entry(ENDPOINT, ENDPOINT),
+            Map.entry("access_key_id", ACCESS_KEY_ID),
+            Map.entry(ACCESS_KEY_ID, ACCESS_KEY_ID),
+            Map.entry("access_key_secret", SECRET_ACCESS_KEY),
+            Map.entry(SECRET_ACCESS_KEY, SECRET_ACCESS_KEY),
+            Map.entry("region", REGION),
+            Map.entry(REGION, REGION),
+            Map.entry("security_token", SECURITY_TOKEN),
+            Map.entry(SECURITY_TOKEN, SECURITY_TOKEN),
+            // OpenDAL 0.57 renamed this option to skip_signature, but lance-c 
0.1.6 still uses
+            // OpenDAL 0.56. Emit the old spelling, which newer OpenDAL keeps 
as an alias.
+            Map.entry(SKIP_SIGNATURE, ALLOW_ANONYMOUS),
+            Map.entry(ALLOW_ANONYMOUS, ALLOW_ANONYMOUS));
+
+    private LanceOssStorageProvider() {
+    }
+
+    @Override
+    public Map<String, String> normalizeDorisStorageOptions(
+            String datasetUri, List<StorageProperties> storageProperties) {
+        Map<String, String> result = new HashMap<>();
+        S3CompatibleFileSystemProperties properties = 
selectOss(storageProperties);
+        if (properties == null) {
+            return result;
+        }
+        if (properties.hasAssumeRole()) {
+            throw new IllegalArgumentException("Lance OSS AssumeRole access is 
not supported yet");
+        }
+        LanceStorageOptions.putIfNotEmpty(result, ENDPOINT, 
properties.getEndpoint());
+        LanceStorageOptions.putIfNotEmpty(result, REGION, 
properties.getRegion());
+        LanceStorageOptions.putIfNotEmpty(result, ACCESS_KEY_ID, 
properties.getAccessKey());
+        LanceStorageOptions.putIfNotEmpty(result, SECRET_ACCESS_KEY, 
properties.getSecretKey());
+        LanceStorageOptions.putIfNotEmpty(result, SECURITY_TOKEN, 
properties.getSessionToken());
+
+        // Lance snapshots the host's OSS_*/AWS_*/ALIBABA_CLOUD_* environment 
into the same config
+        // map before storage options are applied, so state both addressing 
styles explicitly the
+        // way the S3 provider does. Leaving the default implicit would let an 
exported
+        // OSS_ADDRESSING_STYLE outrank an explicit oss.use_path_style=false.
+        String usePathStyle = properties.getUsePathStyle();
+        if (StringUtils.isNotEmpty(usePathStyle)) {
+            result.put(ADDRESSING_STYLE, properties.isUsePathStyle() ? "path" 
: "virtual");
+        }
+        return result;
+    }
+
+    @Override
+    public Map<String, String> normalizeVendedStorageOptions(Map<String, 
String> options) {
+        return LanceStorageOptions.normalizeAliases(options, PUBLIC_BY_ALIAS);
+    }
+
+    @Override
+    public Map<String, String> inferStorageOptions(Map<String, String> 
effectiveOptions) {
+        Map<String, String> inferred = new HashMap<>();
+        boolean hasSigningConfiguration = 
hasSigningConfiguration(effectiveOptions);
+        String allowAnonymous = effectiveOptions.get(ALLOW_ANONYMOUS);
+        if (allowAnonymous != null) {
+            Boolean anonymous = parseOpenDalBoolean(allowAnonymous);
+            if (anonymous == null) {
+                throw new IllegalArgumentException("Unrecognized value for OSS 
storage option '"
+                        + ALLOW_ANONYMOUS + "': '" + allowAnonymous
+                        + "'. Expected one of true, on, false, off");
+            }
+            if (anonymous && hasSigningConfiguration) {
+                throw new IllegalArgumentException(
+                        "Conflicting OSS authentication: anonymous access is 
enabled but signing "
+                                + "credentials are also configured");
+            }
+            return inferred;
+        }
+        inferred.put(ALLOW_ANONYMOUS, 
String.valueOf(!hasSigningConfiguration));

Review Comment:
   [P2] Keep ambient OSS authentication selectors from overriding explicit keys
   
   Lance imports the process's OSS/AWS/Alibaba options before overlaying this 
map. On the native versions actually pinned here, `skip_signature` and 
`allow_anonymous` are separate fields combined with OR, so emitting only 
`allow_anonymous=false` leaves `OSS_SKIP_SIGNATURE=true` effective and sends 
private-bucket reads unsigned. Likewise, an ambient `OSS_ROLE_ARN` or 
`ALIBABA_CLOUD_ROLE_ARN` survives and makes OpenDAL wrap the complete 
configured/vended key pair in an STS AssumeRole provider; keys allowed to read 
the bucket but not assume that unrelated role then fail. A BE-only export can 
let FE planning succeed while execution fails. Explicit identities need to 
neutralize both ambient signing and role selectors, while preserving any 
deliberately supplied namespace options.



##########
fe/fe-connector/fe-connector-lance/src/main/java/org/apache/doris/connector/lance/LanceOssStorageProvider.java:
##########
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.lance;
+
+import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties;
+import org.apache.doris.filesystem.properties.StorageProperties;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+/** Alibaba Cloud OSS storage, which Lance reaches through its OpenDAL OSS 
provider. */
+final class LanceOssStorageProvider implements LanceStorageProvider {
+
+    static final LanceOssStorageProvider INSTANCE = new 
LanceOssStorageProvider();
+
+    /** The provider name of the engine's OSS storage binding; OSS reports the 
S3 family like S3 itself. */
+    private static final String OSS_PROVIDER = "OSS";
+
+    private static final String ENDPOINT = "oss_endpoint";
+    private static final String ACCESS_KEY_ID = "oss_access_key_id";
+    private static final String SECRET_ACCESS_KEY = "oss_secret_access_key";
+    private static final String REGION = "oss_region";
+    private static final String SECURITY_TOKEN = "oss_security_token";
+    private static final String ADDRESSING_STYLE = "addressing_style";
+    private static final String SKIP_SIGNATURE = "skip_signature";
+    private static final String ALLOW_ANONYMOUS = "allow_anonymous";
+
+    /**
+     * Lance exposes the {@code oss_*} names as its public storage-option 
vocabulary and normalizes
+     * them to OpenDAL's field names before constructing the operator. Both 
spellings are accepted,
+     * so collapse only these known pairs before merging static and 
namespace-vended options.
+     */
+    private static final Map<String, String> PUBLIC_BY_ALIAS = Map.ofEntries(
+            Map.entry("endpoint", ENDPOINT),
+            Map.entry(ENDPOINT, ENDPOINT),
+            Map.entry("access_key_id", ACCESS_KEY_ID),
+            Map.entry(ACCESS_KEY_ID, ACCESS_KEY_ID),
+            Map.entry("access_key_secret", SECRET_ACCESS_KEY),
+            Map.entry(SECRET_ACCESS_KEY, SECRET_ACCESS_KEY),
+            Map.entry("region", REGION),
+            Map.entry(REGION, REGION),
+            Map.entry("security_token", SECURITY_TOKEN),
+            Map.entry(SECURITY_TOKEN, SECURITY_TOKEN),
+            // OpenDAL 0.57 renamed this option to skip_signature, but lance-c 
0.1.6 still uses
+            // OpenDAL 0.56. Emit the old spelling, which newer OpenDAL keeps 
as an alias.
+            Map.entry(SKIP_SIGNATURE, ALLOW_ANONYMOUS),
+            Map.entry(ALLOW_ANONYMOUS, ALLOW_ANONYMOUS));

Review Comment:
   [P2] Canonicalize the OSS addressing-style option before merging
   
   `ADDRESSING_STYLE=path` is a valid case-insensitive OpenDAL option, but 
`addressing_style` is missing from this alias table. With an OSS catalog 
binding, normalization has already emitted lowercase 
`addressing_style=virtual`; the vended uppercase spelling therefore survives 
beside it instead of overriding it. OpenDAL lowercases both keys during its 
derived config deserialization and rejects the duplicate `addressing_style` 
field, so this valid namespace override cannot open the table. Include 
`ADDRESSING_STYLE -> ADDRESSING_STYLE` in the canonical map and cover a 
mixed-case override/conflict.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to