github-actions[bot] commented on code in PR #70725:
URL: https://github.com/apache/doris/pull/70725#discussion_r4617241505
##########
fe/fe-connector/fe-connector-lance/src/main/java/org/apache/doris/connector/lance/LanceStorageOptions.java:
##########
@@ -79,17 +82,36 @@ private static Map<String, String>
buildStorageOptions(String datasetUri,
Map<String, String> normalizedVended = new HashMap<>();
if (vendedOptions != null && !vendedOptions.isEmpty()) {
vendedOptions.forEach(LanceStorageOptions::validateVendedOption);
- // Safe to validate before normalizing: normalization only renames
a key to a provider
- // constant or passes it through, so it cannot introduce a NUL
missed above.
+ // Normalization renames keys to provider constants and may fill
an absent session token
+ // with an empty string, so it cannot introduce a NUL missed by
validation above.
normalizedVended =
provider.normalizeVendedStorageOptions(vendedOptions);
}
result.putAll(normalizedVended);
Review Comment:
[P2] Reconcile the inherited S3 endpoint after merging the addressing mode
A REST catalog with `s3.endpoint=http://minio:9000` and
`use_path_style=true` can now receive `virtual_hosted_style_request=true` from
DescribeTable while retaining the catalog endpoint. The merge produces a
virtual-hosted request with a bucketless endpoint: the pinned object_store
builder uses an explicit endpoint verbatim in that mode, so
`s3://warehouse/...` requests go to the service root instead of the warehouse
bucket. The reverse style change duplicates the bucket in host and path.
`vendedCredentialsSupersedeTheCatalogsOnS3` currently asserts the first
inconsistent map. Convert an inherited catalog endpoint using the final merged
addressing mode, and verify the resulting native host/path for both style
changes.
##########
fe/fe-connector/fe-connector-lance/src/main/java/org/apache/doris/connector/lance/LanceOssStorageProvider.java:
##########
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.lance;
+
+import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties;
+import org.apache.doris.filesystem.properties.StorageProperties;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+/** Alibaba Cloud OSS storage, which Lance reaches through its OpenDAL OSS
provider. */
+final class LanceOssStorageProvider implements LanceStorageProvider {
+
+ static final LanceOssStorageProvider INSTANCE = new
LanceOssStorageProvider();
+
+ /** The provider name of the engine's OSS storage binding; OSS reports the
S3 family like S3 itself. */
+ private static final String OSS_PROVIDER = "OSS";
+
+ private static final String ENDPOINT = "oss_endpoint";
+ private static final String ACCESS_KEY_ID = "oss_access_key_id";
+ private static final String SECRET_ACCESS_KEY = "oss_secret_access_key";
+ private static final String REGION = "oss_region";
+ private static final String SECURITY_TOKEN = "oss_security_token";
+ private static final String ADDRESSING_STYLE = "addressing_style";
+ private static final String SKIP_SIGNATURE = "skip_signature";
+ private static final String ALLOW_ANONYMOUS = "allow_anonymous";
+
+ /**
+ * Lance exposes the {@code oss_*} names as its public storage-option
vocabulary and normalizes
+ * them to OpenDAL's field names before constructing the operator. Both
spellings are accepted,
+ * so collapse only these known pairs before merging static and
namespace-vended options.
+ */
+ private static final Map<String, String> PUBLIC_BY_ALIAS = Map.ofEntries(
+ Map.entry("endpoint", ENDPOINT),
+ Map.entry(ENDPOINT, ENDPOINT),
+ Map.entry("access_key_id", ACCESS_KEY_ID),
+ Map.entry(ACCESS_KEY_ID, ACCESS_KEY_ID),
+ Map.entry("access_key_secret", SECRET_ACCESS_KEY),
+ Map.entry(SECRET_ACCESS_KEY, SECRET_ACCESS_KEY),
+ Map.entry("region", REGION),
+ Map.entry(REGION, REGION),
+ Map.entry("security_token", SECURITY_TOKEN),
+ Map.entry(SECURITY_TOKEN, SECURITY_TOKEN),
+ // OpenDAL 0.57 renamed this option to skip_signature, but lance-c
0.1.6 still uses
+ // OpenDAL 0.56. Emit the old spelling, which newer OpenDAL keeps
as an alias.
+ Map.entry(SKIP_SIGNATURE, ALLOW_ANONYMOUS),
+ Map.entry(ALLOW_ANONYMOUS, ALLOW_ANONYMOUS));
+
+ private LanceOssStorageProvider() {
+ }
+
+ @Override
+ public Map<String, String> normalizeDorisStorageOptions(
+ String datasetUri, List<StorageProperties> storageProperties) {
+ Map<String, String> result = new HashMap<>();
+ S3CompatibleFileSystemProperties properties =
selectOss(storageProperties);
+ if (properties == null) {
+ return result;
+ }
+ if (properties.hasAssumeRole()) {
+ throw new IllegalArgumentException("Lance OSS AssumeRole access is
not supported yet");
+ }
+ LanceStorageOptions.putIfNotEmpty(result, ENDPOINT,
properties.getEndpoint());
+ LanceStorageOptions.putIfNotEmpty(result, REGION,
properties.getRegion());
+ LanceStorageOptions.putIfNotEmpty(result, ACCESS_KEY_ID,
properties.getAccessKey());
+ LanceStorageOptions.putIfNotEmpty(result, SECRET_ACCESS_KEY,
properties.getSecretKey());
+ LanceStorageOptions.putIfNotEmpty(result, SECURITY_TOKEN,
properties.getSessionToken());
+
+ // Lance snapshots the host's OSS_*/AWS_*/ALIBABA_CLOUD_* environment
into the same config
+ // map before storage options are applied, so state both addressing
styles explicitly the
+ // way the S3 provider does. Leaving the default implicit would let an
exported
+ // OSS_ADDRESSING_STYLE outrank an explicit oss.use_path_style=false.
+ String usePathStyle = properties.getUsePathStyle();
+ if (StringUtils.isNotEmpty(usePathStyle)) {
+ result.put(ADDRESSING_STYLE, properties.isUsePathStyle() ? "path"
: "virtual");
+ }
+ return result;
+ }
+
+ @Override
+ public Map<String, String> normalizeVendedStorageOptions(Map<String,
String> options) {
+ return LanceStorageOptions.normalizeAliases(options, PUBLIC_BY_ALIAS);
Review Comment:
[P2] Clear the previous OSS token when replacing its key pair
When the catalog has OSS session credentials and DescribeTable vends a
complete permanent key pair without `security_token`, this normalizer replaces
only the keys. The merged map retains the catalog's `oss_security_token`, so
OpenDAL signs the new identity with the previous identity's token and otherwise
valid reads fail authentication. With no catalog token, Lance can also import
an unrelated `OSS_SECURITY_TOKEN` or `ALIBABA_CLOUD_SECURITY_TOKEN` from the
FE/BE environment. Apply the same identity reset as the S3 normalizer:
explicitly clear a missing token for a complete replacement pair, and prevent a
permanent catalog pair from inheriting an ambient token.
##########
fe/fe-connector/fe-connector-lance/src/main/java/org/apache/doris/connector/lance/LanceOssStorageProvider.java:
##########
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.lance;
+
+import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties;
+import org.apache.doris.filesystem.properties.StorageProperties;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+/** Alibaba Cloud OSS storage, which Lance reaches through its OpenDAL OSS
provider. */
+final class LanceOssStorageProvider implements LanceStorageProvider {
+
+ static final LanceOssStorageProvider INSTANCE = new
LanceOssStorageProvider();
+
+ /** The provider name of the engine's OSS storage binding; OSS reports the
S3 family like S3 itself. */
+ private static final String OSS_PROVIDER = "OSS";
+
+ private static final String ENDPOINT = "oss_endpoint";
+ private static final String ACCESS_KEY_ID = "oss_access_key_id";
+ private static final String SECRET_ACCESS_KEY = "oss_secret_access_key";
+ private static final String REGION = "oss_region";
+ private static final String SECURITY_TOKEN = "oss_security_token";
+ private static final String ADDRESSING_STYLE = "addressing_style";
+ private static final String SKIP_SIGNATURE = "skip_signature";
+ private static final String ALLOW_ANONYMOUS = "allow_anonymous";
+
+ /**
+ * Lance exposes the {@code oss_*} names as its public storage-option
vocabulary and normalizes
+ * them to OpenDAL's field names before constructing the operator. Both
spellings are accepted,
+ * so collapse only these known pairs before merging static and
namespace-vended options.
+ */
+ private static final Map<String, String> PUBLIC_BY_ALIAS = Map.ofEntries(
+ Map.entry("endpoint", ENDPOINT),
+ Map.entry(ENDPOINT, ENDPOINT),
+ Map.entry("access_key_id", ACCESS_KEY_ID),
+ Map.entry(ACCESS_KEY_ID, ACCESS_KEY_ID),
+ Map.entry("access_key_secret", SECRET_ACCESS_KEY),
+ Map.entry(SECRET_ACCESS_KEY, SECRET_ACCESS_KEY),
+ Map.entry("region", REGION),
+ Map.entry(REGION, REGION),
+ Map.entry("security_token", SECURITY_TOKEN),
+ Map.entry(SECURITY_TOKEN, SECURITY_TOKEN),
+ // OpenDAL 0.57 renamed this option to skip_signature, but lance-c
0.1.6 still uses
+ // OpenDAL 0.56. Emit the old spelling, which newer OpenDAL keeps
as an alias.
+ Map.entry(SKIP_SIGNATURE, ALLOW_ANONYMOUS),
+ Map.entry(ALLOW_ANONYMOUS, ALLOW_ANONYMOUS));
+
+ private LanceOssStorageProvider() {
+ }
+
+ @Override
+ public Map<String, String> normalizeDorisStorageOptions(
+ String datasetUri, List<StorageProperties> storageProperties) {
+ Map<String, String> result = new HashMap<>();
+ S3CompatibleFileSystemProperties properties =
selectOss(storageProperties);
+ if (properties == null) {
+ return result;
+ }
+ if (properties.hasAssumeRole()) {
+ throw new IllegalArgumentException("Lance OSS AssumeRole access is
not supported yet");
+ }
+ LanceStorageOptions.putIfNotEmpty(result, ENDPOINT,
properties.getEndpoint());
+ LanceStorageOptions.putIfNotEmpty(result, REGION,
properties.getRegion());
+ LanceStorageOptions.putIfNotEmpty(result, ACCESS_KEY_ID,
properties.getAccessKey());
+ LanceStorageOptions.putIfNotEmpty(result, SECRET_ACCESS_KEY,
properties.getSecretKey());
+ LanceStorageOptions.putIfNotEmpty(result, SECURITY_TOKEN,
properties.getSessionToken());
+
+ // Lance snapshots the host's OSS_*/AWS_*/ALIBABA_CLOUD_* environment
into the same config
+ // map before storage options are applied, so state both addressing
styles explicitly the
+ // way the S3 provider does. Leaving the default implicit would let an
exported
+ // OSS_ADDRESSING_STYLE outrank an explicit oss.use_path_style=false.
+ String usePathStyle = properties.getUsePathStyle();
+ if (StringUtils.isNotEmpty(usePathStyle)) {
+ result.put(ADDRESSING_STYLE, properties.isUsePathStyle() ? "path"
: "virtual");
+ }
+ return result;
+ }
+
+ @Override
+ public Map<String, String> normalizeVendedStorageOptions(Map<String,
String> options) {
+ return LanceStorageOptions.normalizeAliases(options, PUBLIC_BY_ALIAS);
+ }
+
+ @Override
+ public Map<String, String> inferStorageOptions(Map<String, String>
effectiveOptions) {
+ Map<String, String> inferred = new HashMap<>();
+ boolean hasSigningConfiguration =
hasSigningConfiguration(effectiveOptions);
+ String allowAnonymous = effectiveOptions.get(ALLOW_ANONYMOUS);
+ if (allowAnonymous != null) {
+ Boolean anonymous = parseOpenDalBoolean(allowAnonymous);
+ if (anonymous == null) {
+ throw new IllegalArgumentException("Unrecognized value for OSS
storage option '"
+ + ALLOW_ANONYMOUS + "': '" + allowAnonymous
+ + "'. Expected one of true, on, false, off");
+ }
+ if (anonymous && hasSigningConfiguration) {
+ throw new IllegalArgumentException(
+ "Conflicting OSS authentication: anonymous access is
enabled but signing "
+ + "credentials are also configured");
+ }
+ return inferred;
+ }
+ inferred.put(ALLOW_ANONYMOUS,
String.valueOf(!hasSigningConfiguration));
Review Comment:
[P2] Keep ambient OSS authentication selectors from overriding explicit keys
Lance imports the process's OSS/AWS/Alibaba options before overlaying this
map. On the native versions actually pinned here, `skip_signature` and
`allow_anonymous` are separate fields combined with OR, so emitting only
`allow_anonymous=false` leaves `OSS_SKIP_SIGNATURE=true` effective and sends
private-bucket reads unsigned. Likewise, an ambient `OSS_ROLE_ARN` or
`ALIBABA_CLOUD_ROLE_ARN` survives and makes OpenDAL wrap the complete
configured/vended key pair in an STS AssumeRole provider; keys allowed to read
the bucket but not assume that unrelated role then fail. A BE-only export can
let FE planning succeed while execution fails. Explicit identities need to
neutralize both ambient signing and role selectors, while preserving any
deliberately supplied namespace options.
##########
fe/fe-connector/fe-connector-lance/src/main/java/org/apache/doris/connector/lance/LanceOssStorageProvider.java:
##########
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.lance;
+
+import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties;
+import org.apache.doris.filesystem.properties.StorageProperties;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+/** Alibaba Cloud OSS storage, which Lance reaches through its OpenDAL OSS
provider. */
+final class LanceOssStorageProvider implements LanceStorageProvider {
+
+ static final LanceOssStorageProvider INSTANCE = new
LanceOssStorageProvider();
+
+ /** The provider name of the engine's OSS storage binding; OSS reports the
S3 family like S3 itself. */
+ private static final String OSS_PROVIDER = "OSS";
+
+ private static final String ENDPOINT = "oss_endpoint";
+ private static final String ACCESS_KEY_ID = "oss_access_key_id";
+ private static final String SECRET_ACCESS_KEY = "oss_secret_access_key";
+ private static final String REGION = "oss_region";
+ private static final String SECURITY_TOKEN = "oss_security_token";
+ private static final String ADDRESSING_STYLE = "addressing_style";
+ private static final String SKIP_SIGNATURE = "skip_signature";
+ private static final String ALLOW_ANONYMOUS = "allow_anonymous";
+
+ /**
+ * Lance exposes the {@code oss_*} names as its public storage-option
vocabulary and normalizes
+ * them to OpenDAL's field names before constructing the operator. Both
spellings are accepted,
+ * so collapse only these known pairs before merging static and
namespace-vended options.
+ */
+ private static final Map<String, String> PUBLIC_BY_ALIAS = Map.ofEntries(
+ Map.entry("endpoint", ENDPOINT),
+ Map.entry(ENDPOINT, ENDPOINT),
+ Map.entry("access_key_id", ACCESS_KEY_ID),
+ Map.entry(ACCESS_KEY_ID, ACCESS_KEY_ID),
+ Map.entry("access_key_secret", SECRET_ACCESS_KEY),
+ Map.entry(SECRET_ACCESS_KEY, SECRET_ACCESS_KEY),
+ Map.entry("region", REGION),
+ Map.entry(REGION, REGION),
+ Map.entry("security_token", SECURITY_TOKEN),
+ Map.entry(SECURITY_TOKEN, SECURITY_TOKEN),
+ // OpenDAL 0.57 renamed this option to skip_signature, but lance-c
0.1.6 still uses
+ // OpenDAL 0.56. Emit the old spelling, which newer OpenDAL keeps
as an alias.
+ Map.entry(SKIP_SIGNATURE, ALLOW_ANONYMOUS),
+ Map.entry(ALLOW_ANONYMOUS, ALLOW_ANONYMOUS));
Review Comment:
[P2] Canonicalize the OSS addressing-style option before merging
`ADDRESSING_STYLE=path` is a valid case-insensitive OpenDAL option, but
`addressing_style` is missing from this alias table. With an OSS catalog
binding, normalization has already emitted lowercase
`addressing_style=virtual`; the vended uppercase spelling therefore survives
beside it instead of overriding it. OpenDAL lowercases both keys during its
derived config deserialization and rejects the duplicate `addressing_style`
field, so this valid namespace override cannot open the table. Include
`ADDRESSING_STYLE -> ADDRESSING_STYLE` in the canonical map and cover a
mixed-case override/conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]