github-actions[bot] commented on code in PR #70443:
URL: https://github.com/apache/doris/pull/70443#discussion_r4597719869
##########
be/src/cloud/cloud_meta_mgr.cpp:
##########
@@ -1909,42 +1907,21 @@ Status
CloudMetaMgr::get_storage_vault_info(StorageVaultInfos* vault_infos, bool
*is_vault_mode = resp.enable_storage_vault();
- auto add_obj_store = [&vault_infos](const auto& obj_store) {
- vault_infos->emplace_back(obj_store.id(),
S3Conf::get_s3_conf(obj_store),
- StorageVaultPB_PathFormat {});
- };
-
- std::ranges::for_each(resp.obj_info(), add_obj_store);
- std::ranges::for_each(resp.storage_vault(), [&](const auto& vault) {
+ for (const auto& info : resp.obj_info()) {
+ vault_infos->emplace_back(info.id(), info, StorageVaultPB_PathFormat
{});
+ }
+ for (const auto& vault : resp.storage_vault()) {
if (vault.has_hdfs_info()) {
vault_infos->emplace_back(vault.id(), vault.hdfs_info(),
vault.path_format());
+ } else if (vault.has_obj_info()) {
+ const auto& id = vault.id().empty() ? vault.obj_info().id() :
vault.id();
+ vault_infos->emplace_back(id, vault.obj_info(),
vault.path_format());
Review Comment:
[P1] Preserve the legacy object-vault layout during upgrade
An operator-provisioned built-in S3 vault can persist
`path_format={path_version:1,shard_num:1000}` through `create_instance(vault)`
-> `add_vault_into_instance`. Older BEs ignored that field for every object
vault and wrote layout 0. Forwarding it here makes an upgraded BE construct
`StorageResource` with layout 1, so `Rowset::segment_path` looks for
`data/<shard>/<tablet>/<rowset>/<segment>.dat` instead of the existing
`data/<tablet>/<rowset>_<segment>.dat`. Existing committed rowsets become
unreadable after restart/upgrade, and mixed-version BEs disagree about new
paths. Preserve the historical S3 layout in this refactor or introduce an
explicit compatible migration/layout selector.
##########
common/cpp/obj-storage/client/s3_obj_storage_client.cpp:
##########
@@ -117,6 +191,9 @@ ObjStorageUploadResult
S3ObjStorageClient::create_multipart_upload(const ObjStor
ObjStorageResponse S3ObjStorageClient::put_object(const ObjStoragePath& opts,
std::string_view stream) {
Aws::S3::Model::PutObjectRequest request;
+ if (_supports_s3_encryption && opts.sse_enabled) {
+
request.SetServerSideEncryption(Aws::S3::Model::ServerSideEncryption::AES256);
Review Comment:
[P1] Avoid enabling SSE on existing MinIO vaults by default
Built-in object vaults already have `sse_enabled=true` because
`add_vault_into_instance` hardcodes it, even when `create_instance` has no SSE
opt-in. The repository's `docker/runtime/all-in-one/4.1/resource/cloud_init.sh`
uses that default vault mode, and its `compose/cloud.yml` starts MinIO without
KMS. The newly propagated flag now adds `AES256` here and during multipart
initiation; MinIO [rejects SSE-S3 without configured
KMS](https://github.com/minio/minio/blob/RELEASE.2025-01-20T14-49-07Z/cmd/encryption-v1.go#L363-L369),
so previously working built-in MinIO vaults can no longer accept data writes.
Preserve prior upload behavior unless encryption was explicitly enabled and
supported, rather than turning this historically forced metadata flag into a
new storage requirement.
##########
be/src/cloud/cloud_storage_engine.cpp:
##########
@@ -456,6 +448,9 @@ Status
CloudStorageEngine::start_bg_threads(std::shared_ptr<WorkloadGroup> wg_sp
}
void CloudStorageEngine::sync_storage_vault() {
+ // The periodic refresh and a lookup for a missing vault can overlap.
Serialize the fetch as
+ // well as publication, so an older metadata snapshot cannot overwrite
newer credentials.
+ std::lock_guard lock(_sync_storage_vault_mutex);
Review Comment:
[P1] Use a bthread-aware mutex for vault refresh
Missing-vault lookups run in `CloudDeltaWriter::batch_init`'s
`bthread_fork_join` tasks (`writer->init` -> `CloudRowsetBuilder::init` ->
`get_storage_resource`). This `std::mutex` remains locked while
`get_storage_vault_info` waits for its synchronous RPC, including `retry_rpc`'s
`bthread_usleep`. When enough tablets initialize against a newly created vault
concurrently, mutex waiters can park all bthread worker pthreads while the
owner is suspended waiting for RPC/retry completion; it then cannot resume to
release the lock, hanging refresh and loads. Keep serialization with
`bthread::Mutex` or an ordinary-thread refresh executor.
##########
common/cpp/obj-storage/parser/obj_storage_config_parser.cpp:
##########
@@ -0,0 +1,686 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cpp/obj-storage/parser/obj_storage_config_parser.h"
+
+#include <brpc/uri.h>
+#include <fmt/format.h>
+#include <gen_cpp/AgentService_types.h>
+
+#include <algorithm>
+#include <cctype>
+#include <charconv>
+#include <type_traits>
+#include <utility>
+
+#include "cpp/obj-storage/auth/gcp/gcp_auth.h"
+
+namespace doris {
+namespace {
+using Vault = cloud::ObjectStoreInfoPB;
+
+struct ProviderMapping {
+ std::string_view name;
+ TObjStorageType::type resource;
+ cloud::ObjectStoreInfoPB::Provider vault;
+};
+
+constexpr ProviderMapping PROVIDERS[] = {
+ {"AWS", TObjStorageType::AWS, cloud::ObjectStoreInfoPB::S3},
+ {"S3", TObjStorageType::AWS, cloud::ObjectStoreInfoPB::S3},
+ {"OSS", TObjStorageType::OSS, cloud::ObjectStoreInfoPB::OSS},
+ {"COS", TObjStorageType::COS, cloud::ObjectStoreInfoPB::COS},
+ {"BOS", TObjStorageType::BOS, cloud::ObjectStoreInfoPB::BOS},
+ {"OBS", TObjStorageType::OBS, cloud::ObjectStoreInfoPB::OBS},
+ {"GCP", TObjStorageType::GCP, cloud::ObjectStoreInfoPB::GCP},
+ {"GCS", TObjStorageType::GCP, cloud::ObjectStoreInfoPB::GCP},
+ {"TOS", TObjStorageType::TOS, cloud::ObjectStoreInfoPB::TOS},
+ {"AZURE", TObjStorageType::AZURE, cloud::ObjectStoreInfoPB::AZURE},
+};
+
+std::optional<std::string_view>
credential_provider_name(TCredProviderType::type provider) {
+ switch (provider) {
+ case TCredProviderType::DEFAULT:
+ return "DEFAULT";
+ case TCredProviderType::SIMPLE:
+ return "SIMPLE";
+ case TCredProviderType::INSTANCE_PROFILE:
+ return "INSTANCE_PROFILE";
+ case TCredProviderType::ENV:
+ return "ENV";
+ case TCredProviderType::SYSTEM_PROPERTIES:
+ return "SYSTEM_PROPERTIES";
+ case TCredProviderType::WEB_IDENTITY:
+ return "WEB_IDENTITY";
+ case TCredProviderType::CONTAINER:
+ return "CONTAINER";
+ case TCredProviderType::ANONYMOUS:
+ return "ANONYMOUS";
+ default:
+ return std::nullopt;
+ }
+}
+
+std::optional<std::string_view> gcp_credential_provider_name(
+ TGcpCredentialProviderType::type provider) {
+ switch (provider) {
+ case TGcpCredentialProviderType::DEFAULT:
+ return "DEFAULT";
+ case TGcpCredentialProviderType::COMPUTE_ENGINE:
+ return "COMPUTE_ENGINE";
+ default:
+ return std::nullopt;
+ }
+}
+
+std::string to_upper(std::string text) {
+ std::transform(text.begin(), text.end(), text.begin(),
+ [](unsigned char c) { return
static_cast<char>(std::toupper(c)); });
+ return text;
+}
+
+std::string normalize_provider(std::string text) {
+ const auto non_space = [](unsigned char c) { return !std::isspace(c); };
+ text.erase(text.begin(), std::find_if(text.begin(), text.end(),
non_space));
+ text.erase(std::find_if(text.rbegin(), text.rend(), non_space).base(),
text.end());
+ return to_upper(std::move(text));
+}
+
+std::string_view pop_dns_label(std::string_view& name) {
+ const auto dot = name.rfind('.');
+ if (dot == std::string_view::npos) {
+ return std::exchange(name, std::string_view {});
+ }
+ auto label = name.substr(dot + 1);
+ name = name.substr(0, dot);
+ return label;
+}
+
+} // namespace
+
+cloud::ObjectStoreInfoPB::Provider vault_storage_provider(const
cloud::ObjectStoreInfoPB& info) {
+ if (info.has_provider() || info.endpoint().empty()) {
+ return info.provider();
+ }
+ return storage_provider_from_endpoint(info.endpoint());
+}
+
+cloud::ObjectStoreInfoPB::Provider
storage_provider_from_endpoint(std::string_view endpoint) {
+ std::string url(endpoint);
+ std::transform(url.begin(), url.end(), url.begin(),
+ [](unsigned char c) { return std::tolower(c); });
+ brpc::URI uri;
+ if (url.find('\0') != std::string::npos || uri.SetHttpURL(url.c_str()) !=
0 ||
+ (!uri.scheme().empty() && uri.scheme() != "http" && uri.scheme() !=
"https")) {
+ return Vault::S3;
+ }
+ std::string_view host = uri.host();
+ if (host.ends_with('.')) {
+ host.remove_suffix(1);
+ }
+ if (host.empty() ||
host.find_first_not_of("abcdefghijklmnopqrstuvwxyz0123456789-.") !=
+ std::string_view::npos) {
+ return Vault::S3;
+ }
+
+ // Match from the DNS root so virtual-hosted buckets cannot change the
service or cloud.
+ auto labels = host;
+ auto tld = pop_dns_label(labels);
+ auto domain = pop_dns_label(labels);
+ auto last = pop_dns_label(labels);
+ auto previous = pop_dns_label(labels);
+ auto is_region = [](std::string_view name) {
+ return name.size() > 3 && name[2] == '-' && name[0] >= 'a' && name[0]
<= 'z' &&
+ name[1] >= 'a' && name[1] <= 'z';
+ };
+ if (domain == "aliyuncs" && tld == "com") {
+ if ((last.starts_with("oss-") && is_region(last.substr(4))) || last ==
"oss-accelerate" ||
+ last == "oss-accelerate-overseas" || (previous == "s3" &&
is_region(last)) ||
+ (last == "oss" && is_region(previous))) {
+ return Vault::OSS;
+ }
+ } else if (domain == "googleapis" && tld == "com" && last == "storage") {
+ return Vault::GCP;
+ } else if (domain == "myqcloud" && tld == "com" && previous == "cos" &&
!last.empty()) {
+ return Vault::COS;
+ } else if (domain == "myhuaweicloud" && tld == "com" && previous == "obs"
&& !last.empty()) {
+ return Vault::OBS;
+ } else if (domain == "bcebos" && tld == "com" && !last.empty()) {
+ return Vault::BOS;
+ } else if ((domain == "volces" || domain == "ivolces") && tld == "com" &&
+ ((last.starts_with("tos-") && is_region(last.substr(4))) ||
+ (last.starts_with("tos-s3-") && is_region(last.substr(7))))) {
+ return Vault::TOS;
+ } else if (last == "core" && (previous == "blob" || previous == "dfs") &&
!labels.empty() &&
+ ((domain == "windows" && tld == "net") ||
+ (domain == "chinacloudapi" && tld == "cn") ||
+ (domain == "usgovcloudapi" && tld == "net") ||
+ (domain == "cloudapi" && tld == "de"))) {
+ return Vault::AZURE;
+ }
+ return Vault::S3;
+}
+
+AwsCredentialImplementation aws_credential_implementation(std::string_view
version) {
+ return version == "v2" ? AwsCredentialImplementation::V2 :
AwsCredentialImplementation::V1;
+}
+
+void normalize_obj_storage_endpoint(ObjStorageClientConfig& client,
std::string_view scheme) {
+ auto& endpoint = obj_storage_connection(client).endpoint;
+ if (auto* aws = std::get_if<AwsObjStorageConfig>(&client); aws &&
aws->use_sdk_endpoint) {
+ endpoint.clear();
+ aws->sdk_endpoint_http = scheme == "http";
+ return;
+ }
+ if (!endpoint.empty() && endpoint.find("://") == std::string::npos) {
+ endpoint.insert(0,
+ scheme == "http" &&
!std::holds_alternative<AzureObjStorageConfig>(client)
+ ? "http://"
+ : "https://");
+ }
+}
+
+namespace {
+using Props = ObjStorageProperties;
+using Error = std::optional<std::string>;
+template <class T>
+using Parsed = ObjStorageConfigParseResult<T>;
+
+std::string value(const Props& properties, const char* key) {
+ auto it = properties.find(key);
+ return it == properties.end() ? std::string {} : it->second;
+}
+
+template <class T>
+Error read_number(const Props& props, const char* key, std::optional<T>&
output) {
+ auto it = props.find(key);
+ if (it == props.end()) {
+ return {};
+ }
+ const auto& text = it->second;
+ int number;
+ auto [end, error] = std::from_chars(text.data(), text.data() +
text.size(), number);
+ if (error != std::errc {} || end != text.data() + text.size() || number <
-1) {
+ return fmt::format("invalid {} value \"{}\"", key, text);
+ }
+ // Legacy resources use -1 or 0 for host defaults. Passing a zero request
timeout
+ // to the SDK would disable stalled-transfer detection instead of keeping
the BE default.
+ if (number > 0) {
+ output = T(number);
+ }
+ return {};
+}
+Error read_timeouts(const Props& props, ObjStorageConnectionConfig& conn) {
+ if (auto error = read_number(props, "AWS_REQUEST_TIMEOUT_MS",
conn.request_timeout)) {
+ return error;
+ }
+ return read_number(props, "AWS_CONNECTION_TIMEOUT_MS",
conn.connect_timeout);
+}
+Error read_bool(const Props& props, const char* key, bool& output) {
+ auto it = props.find(key);
+ if (it == props.end()) {
+ return {};
+ }
+ auto text = it->second;
+ std::transform(text.begin(), text.end(), text.begin(),
+ [](unsigned char c) { return
static_cast<char>(std::tolower(c)); });
+ if (text == "true") {
+ output = true;
+ } else if (text == "false") {
+ output = false;
+ } else {
+ return fmt::format("invalid {} value \"{}\"", key, it->second);
+ }
+ return {};
+}
+template <class C>
+Error read_regional_options(const Props& props, C& client) {
+ client.region = value(props, "AWS_REGION");
+ client.connection.endpoint = value(props, "AWS_ENDPOINT");
+ if (auto error = read_bool(props, "use_path_style",
client.use_path_style)) {
+ return error;
+ }
+ bool override_endpoint = true;
+ if (auto error = read_bool(props, "AWS_NEED_OVERRIDE_ENDPOINT",
override_endpoint)) {
+ return error;
+ }
+ if (!override_endpoint) {
+ if constexpr (std::is_same_v<C, AwsObjStorageConfig>) {
+ client.use_sdk_endpoint = true;
+ } else {
+ return "AWS SDK automatic endpoint selection requires
provider=AWS";
+ }
+ }
+ if (auto error = read_number(props, "AWS_MAX_CONNECTIONS",
client.connection.max_connections)) {
+ return error;
+ }
+ return read_timeouts(props, client.connection);
+}
+
+AwsCredentialChainType chain_source(CredProviderType mode) {
+ switch (mode) {
+ case CredProviderType::Env:
+ return AwsCredentialChainType::Environment;
+ case CredProviderType::SystemProperties:
+ return AwsCredentialChainType::Profile;
+ case CredProviderType::WebIdentity:
+ return AwsCredentialChainType::WebIdentity;
+ case CredProviderType::Container:
+ return AwsCredentialChainType::Container;
+ case CredProviderType::InstanceProfile:
+ return AwsCredentialChainType::InstanceProfile;
+ default:
+ return AwsCredentialChainType::Default;
+ }
+}
+
+Parsed<AccessKeyOrAnonymousObjStorageAuth> parse_access_key_credentials(const
Props& props) {
+ AccessKeyObjStorageAuth keys {value(props, "AWS_ACCESS_KEY"), value(props,
"AWS_SECRET_KEY"),
+ value(props, "AWS_TOKEN")};
+ if (!keys.access_key.empty() && !keys.secret_key.empty()) {
+ // Some persisted resources also carry AWS role/provider fields.
Static keys
+ // have always taken precedence; they include temporary credentials
issued by FE.
+ return AccessKeyOrAnonymousObjStorageAuth {std::move(keys)};
+ }
+ if (keys.access_key.empty() != keys.secret_key.empty()) {
+ return std::string("Incomplete access key credentials");
+ }
+ if (!keys.session_token.empty()) {
+ return std::string("Session token requires access key credentials");
+ }
+ const auto mode = to_upper(value(props, "AWS_CREDENTIALS_PROVIDER_TYPE"));
+ if ((!mode.empty() && mode != "DEFAULT" && mode != "SIMPLE" && mode !=
"ANONYMOUS") ||
+ !value(props, "AWS_ROLE_ARN").empty() || !value(props,
"AWS_EXTERNAL_ID").empty()) {
+ return std::string(
+ "AWS credential chains and AssumeRole are not supported for
this provider");
+ }
+ return AccessKeyOrAnonymousObjStorageAuth {AnonymousObjStorageAuth {}};
Review Comment:
[P1] Preserve the GCS DEFAULT HMAC credential chain
With `provider=GCP`, `AWS_CREDENTIALS_PROVIDER_TYPE=DEFAULT` and no explicit
AK/SK or native `gs` credential, FE intentionally requests ambient HMAC
credentials: `GcsAuthResolver` and `AbstractDelegatingS3Properties` explicitly
preserve this legacy mode, and
`S3ResourceTest.testLegacyGcpDefaultChainSurvivesReplayAndPolicyPush` covers
its producer. The default v2 BE previously built
`CustomAwsCredentialsProviderChain` and could read HMAC keys from
environment/profile. The GCP parser now reaches this unconditional anonymous
result, and `GcpObjStorageClient` constructs `AnonymousAWSCredentialsProvider`,
so existing private GCS reads/loads become unsigned and fail after upgrade.
Keep `DEFAULT` distinct from `ANONYMOUS` for this supported mode.
##########
be/src/agent/task_worker_pool.cpp:
##########
@@ -1792,22 +1793,26 @@ void update_s3_resource(const TStorageResource& param,
io::RemoteFileSystemSPtr
Status st;
io::RemoteFileSystemSPtr fs;
- if (!existed_fs) {
- // No such FS instance on BE
- auto res =
io::S3FileSystem::create(S3Conf::get_s3_conf(param.s3_storage_param),
- std::to_string(param.id));
+ auto parsed = parse_be_obj_storage_config(param.s3_storage_param);
+ if (!parsed.has_value()) {
+ LOG(WARNING) << "invalid S3 storage resource " << param.id << ": " <<
parsed.error();
+ return;
+ }
+ auto storage_config = std::move(parsed).value();
+ auto current =
std::static_pointer_cast<io::ObjStorageFileSystem>(existed_fs);
+ DCHECK(!existed_fs || existed_fs->type() ==
io::FileSystemType::OBJECT_STORAGE);
+ // A changed target needs its own filesystem. Open readers and writers
still own the old holder.
+ if (current && current->same_target(storage_config)) {
Review Comment:
[P2] Keep existing holders connected after provider changes
A provider-only `ALTER RESOURCE` from S3 to GCP can keep the same
`storage.googleapis.com` endpoint, `us-east1` region, bucket, prefix and HMAC
keys; FE permits this even for policy resources. `same_client_target` rejects
the variant change, so this code replaces the filesystem.
`RowsetMeta::remote_storage_resource` caches its filesystem after the first
lookup, and open readers retain that filesystem's holder. A later AK/SK
rotation updates only the replacement filesystem in the registry, leaving old
rowsets using revoked keys until reload/restart. Preserve or redirect the
existing shared holder for same-physical-target provider changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]