This is an automated email from the ASF dual-hosted git repository.
wyxxxcat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ff6e7476efa [fix](cloud) Run checker inverted check for instances with
multiple storage vaults (#69290)
ff6e7476efa is described below
commit ff6e7476efa81e589f32366049826495dd43100a
Author: matanper <[email protected]>
AuthorDate: Wed Nov 18 08:53:05 2026 +0200
[fix](cloud) Run checker inverted check for instances with multiple storage
vaults (#69290)
### What problem does this PR solve?
Issue Number: close #69289
Related PR: #69278
Problem Summary:
#### What problem was fixed, and how
The cloud checker's inverted check (`enable_inverted_check = true`) is
the leak detector. It lists the objects of each storage location and
reports objects that no longer belong to any rowset, i.e. data the
recycler failed to delete.
1. **It never ran for instances with more than one storage vault.**
`InstanceChecker::do_inverted_check` started with:
```cpp
if (accessor_map_.size() > 1) {
LOG(INFO) << "currently not support inverted check for multi accessor.
instance_id=" << instance_id_;
return 0;
}
```
The checker creates one accessor per storage vault, plus the built-in
one, so as soon as an instance had a second vault, nothing was
inverted-checked for any vault and `num_inverted_check_failed` stayed 0.
The guard came with #33280 without a stated reason; nothing in the check
depends on a single accessor (objects are matched to rowset meta by
tablet id and rowset id).
- Fix: remove the guard and list each **storage location** once.
2. **Locations are compared as configured, not by the display `uri()`.**
Deduplicating by `uri()` was wrong in two ways:
- `S3Accessor` normalizes `uri()` for display (collapsing repeated `/`),
so vaults on prefixes `a/b` and `a//b` (different objects) would share a
uri and one would be skipped.
- `HdfsAccessor` declared its own `uri_` that shadowed
`StorageVaultAccessor::uri_`, so the base `uri()` was empty for every
HDFS accessor and all HDFS vaults would collapse into one.
- Fix: `StorageVaultAccessor` takes its **location** as a constructor
argument, and `location()` is a non-virtual getter, so every accessor
must provide it and cannot shadow or forget it. Each accessor builds it
from everything in its configuration that may select a namespace,
without normalization, separated by `\n`:
- S3: endpoint, bucket and prefix as configured, plus the credential
identity (provider, ak, role_arn, external_id, credential provider type;
never the sk). S3-compatible gateways such as Ceph RGW multi-tenancy
resolve one bucket name to different tenants' buckets by the user.
- HDFS: `fs_name`, the prefix, and all `hdfs_confs` in their configured
order. A logical nameservice like `hdfs://ns` resolves through
`dfs.namenode.rpc-address.ns.*`, and the order is kept because libhdfs
applies the first value of a repeated key and libhdfs3 the last.
`hdfs://ns/a` + `/b` differs from `hdfs://ns` + `/a/b`.
- Equal locations surely see the same objects, so the second one is
skipped. Different locations may still reach the same storage, which
only costs a second listing.
- The shadowing `HdfsAccessor::uri_` is removed, so HDFS log lines that
print `uri()` show the real location too.
3. **Segments are matched to the storage they belong to.** Every
accessor also has a **storage root**: the S3 bucket (empty for HDFS)
plus the prefix without empty path components. The endpoint and
`fs_name` are left out because they have aliases (http/https and
internal/public endpoints; host, FQDN, IP or nameservice). So a
different root surely means different paths, while an equal root may
still be different storage. The root errs toward "same", and the
location toward "different", so any remaining ambiguity can only hide a
leak, never report live data.
- **Ownership:** the tablet cache keeps each rowset's `resource_id`. A
segment is covered by its rowset only if the rowset's vault has the root
the segment was listed from. A copy of a vault A segment left in vault B
is a leak. A resource without an accessor (an HDFS vault in a build
without HDFS) is trusted.
- **Nested vaults:** a vault root can lie inside another vault's `data/`
directory, e.g. prefix `root/data` inside `root` on one bucket. The
parent then lists the child's objects as `data/data/...`. The parent
skips that subtree, and the objects are checked once, when the child
vault is listed.
- **List errors:** a vault whose `data/` listing fails makes the check
fail (`-1`, unless leaks were found) but no longer stops the other
vaults from being checked.
4. **Leak accounting.**
- A leaked segment was reported as a leak (`1`) only on a tablet-cache
miss; on a cache hit it returned `-1` ("rowset not exists"), a check
error. Whether a leak was counted depended on listing order. Now both
paths report it as a leak.
- The tablet cache claimed the tablet before its rowset meta was read. A
failed read (txn, range get or parse error) left an empty or partial
cache, and with the change above the following objects of that tablet
would have been reported as leaks. The cache now claims the tablet only
after a complete, successful read.
- A cached absence is not trusted: a load can commit a new rowset of the
tablet after the cache was loaded, and its segments are listed later in
the same scan. An object whose rowset is missing from the cache is
checked against a fresh read of the tablet's rowsets before it is
counted as a leak. Present rowsets still hit the cache.
5. **Cancellation.** A stop during a tablet's rowset read leaves the
cache invalid, and the object loop used to keep going, starting a new
rowset meta read for each remaining object of that tablet. The traversal
now ends as soon as the checker is stopped.
#### Which behaviors were modified
| Before | Now | Impact |
|---|---|---|
| Inverted check skipped on instances with more than one accessor (every
multi-vault instance). | Runs for every distinct storage location of the
instance. | Leaks on multi-vault instances start being reported
(`num_inverted_check_failed`). The check lists every vault's `data/`
directory, so its cost grows with the number of vaults; HDFS listing in
particular is recursive. It is still gated by `enable_inverted_check`
and `check_object_interval_seconds`. |
| A leaked segment after a live rowset of the same tablet: check error
(`-1`). | Leak (`1`), after a fresh read confirms the rowset is absent.
| `num_file_leak` counts it, and the check result is `1` instead of
`-1`. |
| An object whose rowset is missing from the cache: leak on a miss,
error on a hit. | Leak only if a fresh read of the tablet's rowsets also
lacks it. | Rowsets committed during the scan are no longer misreported.
Objects of loads whose rowset is not committed yet are still reported,
as before. An extra rowset read happens for every such object. |
| A segment whose rowset exists in another vault: accepted. | Leak, when
the rowset's vault has a different storage root. | Segment copies left
in the wrong vault are reported. Vaults that may be the same storage
(same bucket and prefix) accept each other's segments. |
| (multi-vault was skipped) | Objects of a vault nested in another
vault's `data/` are checked only by their own vault. | No false leaks
for nested vault prefixes. |
| A failing vault listing ends the check. | It fails the check, and the
other vaults are still checked. | One unreachable vault does not hide
the leaks of the others. |
| `uri()` of an HDFS accessor: empty. | `fs_name + prefix`. | HDFS log
lines show the location. |
#### What features were added, and why
None besides the fix. `MockAccessor` takes an optional uri (also its
location) and storage root, so tests can model distinct, shared and
nested storage. It also has a `MockAccessor::list_directory` sync point.
#### Which code was refactored, and why
`StorageVaultAccessor` takes the location and the storage root in its
constructor (`S3Accessor`, `HdfsAccessor` and the test `MockAccessor`
pass theirs), so a new accessor type cannot compile without them. See
points 2 and 3.
#### Which functions were optimized
None.
#### Relation to #69278
#69278 makes the recycler and checker understand the `path_version = 1`
object layout per vault. This PR is independent and based on master. If
both are merged, the checker inverted-checks v1 vaults too, which the
guard prevented. The two touch neighbouring lines of
`do_inverted_check`; the conflict is trivial.
### Release note
The cloud checker's inverted check (`enable_inverted_check`) now runs
for instances with multiple storage vaults (it was silently skipped
before), reports segment files left in a vault other than their
rowset's, and counts leaked segment files consistently.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
**Unit tests (ASAN, `run-cloud-ut.sh`):**
- `recycler_test:InvertedCheckTest.*` (9 tests, new):
- `multi_vault`: two vaults, every object checked; a leak in each (one
on the cache-hit path) gives result 1 and `num_inverted_check_failed =
2`.
- `same_location_once`: two resources on one location are listed once.
- `nested_vault`: a vault on `root/data` inside one on `root`; its
objects (and its leak) are checked only by the child.
- `segment_in_another_vault_is_leak`: a copy of a vault1 segment in
vault2 is a leak; a segment listed through another vault with the same
root is not.
- `list_error_does_not_stop_other_vaults`: a failing listing of one
vault, the other is still checked.
- `orphans_in_both_listing_orders`: orphans before and after live
rowsets of a tablet.
- `rowset_meta_read_error_is_not_leak`: a failing rowset meta read is an
error, never a leak.
- `rowset_committed_during_scan_is_not_leak`: a rowset committed between
the cache load and the listing of its segment.
- `stops_when_stopped`: one rowset read, then the traversal ends.
- `recycler_test:CheckerTest.*`: all 39 pass.
- `HdfsAccessorLocationTest.location` (in `recycler_test`, so it runs
without a test HDFS): distinct fs_name, prefix or address mapping give
distinct locations; a different conf order or repeated-key order gives a
different location; the fs_name/prefix split; the root ignores confs and
fs_name.
- `S3AccessorLocationTest` (`s3_accessor_mock_test`): `a/b` and `a//b`
are different locations; another ak or role gives another location while
the sk is not part of it; endpoint aliases share a root.
- Full `recycler_test`: all pass except
`RecyclerBenchmarkTest.RecycleRowsets`, which needs a live FoundationDB
(it builds `FdbTxnKv` from `fdb.cluster`) and fails the same way on
master.
**Manual test** (run at `469f256`; the later commits, i.e. the
cached-absence recheck, the location and storage root changes, are
covered by the unit tests above):
- Setup: cloud-mode cluster with this branch's `doris_cloud`
(meta-service + recycler, `enable_checker = true`,
`enable_inverted_check = true`, `check_object_interval_seconds = 60`).
Two S3 storage vaults plus the built-in vault, one table per vault, 3
loads each, `enable_packed_file = false` so segments are plain objects.
- Before (master): every checker round logs `currently not support
inverted check for multi accessor` and checks nothing.
- After, with live data: `inverted check instance objects finished ...
num_scanned=10 num_file_leak=0`, covering all 5 + 5 objects of the two
vaults, and the guard message is gone.
- After putting `data/<tablet_id>/zzzleak_0.dat` into each vault (both
tablets also have live rowsets): `num_scanned=12 num_file_leak=2`, with
`rowset should be recycled, key=data/<tablet_id>/zzzleak_0.dat` for
both, and no `rowset not exists` lines.
- Objects left over from an earlier run whose metadata was wiped were
correctly reported as leaks across both vaults.
- Behavior changed:
- [ ] No.
- [x] Yes. See "Which behaviors were modified" above.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
cloud/src/recycler/checker.cpp | 76 +++++--
cloud/src/recycler/hdfs_accessor.cpp | 38 +++-
cloud/src/recycler/hdfs_accessor.h | 5 +-
cloud/src/recycler/s3_accessor.cpp | 18 +-
cloud/src/recycler/storage_vault_accessor.h | 44 +++-
cloud/test/mock_accessor.h | 16 +-
cloud/test/recycler_test.cpp | 315 ++++++++++++++++++++++++++++
cloud/test/s3_accessor_mock_test.cpp | 55 +++++
8 files changed, 533 insertions(+), 34 deletions(-)
diff --git a/cloud/src/recycler/checker.cpp b/cloud/src/recycler/checker.cpp
index f1caa677b75..40198a9819d 100644
--- a/cloud/src/recycler/checker.cpp
+++ b/cloud/src/recycler/checker.cpp
@@ -1053,12 +1053,6 @@ int InstanceChecker::get_bucket_lifecycle(int64_t*
lifecycle_days) {
}
int InstanceChecker::do_inverted_check() {
- if (accessor_map_.size() > 1) {
- LOG(INFO) << "currently not support inverted check for multi accessor.
instance_id="
- << instance_id_;
- return 0;
- }
-
LOG(INFO) << "begin to inverted check objects instance_id=" <<
instance_id_;
int check_ret = 0;
long num_scanned = 0;
@@ -1076,15 +1070,23 @@ int InstanceChecker::do_inverted_check() {
struct TabletRowsets {
int64_t tablet_id {0};
- std::unordered_set<std::string> rowset_ids;
+ std::unordered_map<std::string /* rowset_id */, std::string /*
resource_id */> rowsets;
};
TabletRowsets tablet_rowsets_cache;
RowsetIndexesFormatV1 rowset_index_cache_v1;
RowsetIndexesFormatV2 rowset_index_cache_v2;
- // Return 0 if check success, return 1 if file is garbage data, negative
if error occurred
- auto check_segment_file = [&](const std::string& obj_key) {
+ // A rowset only covers segments under the root of its own vault, a copy
in another vault is a
+ // leak. A resource without accessor (an HDFS vault in a build without
HDFS) is trusted.
+ auto stored_under = [&](const std::string& resource_id, const StorageRoot&
root) {
+ auto it = accessor_map_.find(resource_id);
+ return it == accessor_map_.end() || it->second->root() == root;
+ };
+
+ // Return 0 if check success, return 1 if file is garbage data, negative
if error occurred.
+ // `root` is the storage root the object was listed from.
+ auto check_segment_file = [&](const std::string& obj_key, const
StorageRoot& root) {
std::vector<std::string> str;
butil::SplitString(obj_key, '/', &str);
// data/{tablet_id}/{rowset_id}_{seg_num}.dat
@@ -1116,16 +1118,16 @@ int InstanceChecker::do_inverted_check() {
}
if (tablet_rowsets_cache.tablet_id == tablet_id) {
- if (tablet_rowsets_cache.rowset_ids.contains(rowset_id)) {
+ auto it = tablet_rowsets_cache.rowsets.find(rowset_id);
+ if (it != tablet_rowsets_cache.rowsets.end() &&
stored_under(it->second, root)) {
return 0;
- } else {
- LOG(WARNING) << "rowset not exists, key=" << obj_key;
- return -1;
}
}
- // Get all rowset id of this tablet
- tablet_rowsets_cache.tablet_id = tablet_id;
- tablet_rowsets_cache.rowset_ids.clear();
+ // A cache miss is read again, a rowset may have committed after the
cache was loaded. The
+ // cache claims the tablet only after a complete read, a failed read
must not leave a
+ // partial rowset set that reports the next objects of the tablet as
leaks.
+ tablet_rowsets_cache.tablet_id = 0;
+ tablet_rowsets_cache.rowsets.clear();
std::unique_ptr<Transaction> txn;
TxnErrorCode err = txn_kv_->create_txn(&txn);
if (err != TxnErrorCode::TXN_OK) {
@@ -1137,6 +1139,7 @@ int InstanceChecker::do_inverted_check() {
auto end = meta_rowset_key({instance_id_, tablet_id, INT64_MAX});
while (it == nullptr /* may be not init */ || (it->more() &&
!stopped())) {
TxnErrorCode err = txn->get(begin, end, &it);
+
TEST_SYNC_POINT_CALLBACK("InstanceChecker::do_inverted_check.get_rowsets_err",
&err);
if (err != TxnErrorCode::TXN_OK) {
LOG(WARNING) << "failed to get rowset kv, err=" << err;
return -1;
@@ -1152,7 +1155,7 @@ int InstanceChecker::do_inverted_check() {
LOG(WARNING) << "malformed rowset meta value, key=" <<
hex(k);
return -1;
}
- tablet_rowsets_cache.rowset_ids.insert(rowset.rowset_id_v2());
+ tablet_rowsets_cache.rowsets.emplace(rowset.rowset_id_v2(),
rowset.resource_id());
if (!it->has_next()) {
begin = k;
begin.push_back('\x00'); // Update to next smallest key
for iteration
@@ -1160,12 +1163,23 @@ int InstanceChecker::do_inverted_check() {
}
}
}
+ if (stopped()) {
+ // The loop above stops early when the checker is stopped, the
rowsets are incomplete
+ return -1;
+ }
+ tablet_rowsets_cache.tablet_id = tablet_id;
- if (!tablet_rowsets_cache.rowset_ids.contains(rowset_id)) {
+ auto owner = tablet_rowsets_cache.rowsets.find(rowset_id);
+ if (owner == tablet_rowsets_cache.rowsets.end()) {
// Garbage data leak
LOG(WARNING) << "rowset should be recycled, key=" << obj_key;
return 1;
}
+ if (!stored_under(owner->second, root)) {
+ LOG(WARNING) << "segment is not in the storage vault of its
rowset, key=" << obj_key
+ << " resource_id=" << owner->second;
+ return 1;
+ }
return 0;
};
@@ -1218,20 +1232,38 @@ int InstanceChecker::do_inverted_check() {
// so we choose to skip here.
TEST_SYNC_POINT_RETURN_WITH_VALUE("InstanceChecker::do_inverted_check",
(int)0);
- for (auto& [_, accessor] : accessor_map_) {
+ // List each storage location once, resources sharing one would count its
leaks repeatedly.
+ // Compare `location()`, not the normalized `uri()` ("a/b" and "a//b" hold
different objects).
+ std::unordered_set<std::string> checked_locations;
+ for (auto& [resource_id, accessor] : accessor_map_) {
+ if (!checked_locations.insert(accessor->location()).second) {
+ LOG(INFO) << "skip inverted check of a storage location already
checked, instance_id="
+ << instance_id_ << " resource_id=" << resource_id
+ << " uri=" << accessor->uri();
+ continue;
+ }
+ const StorageRoot& root = accessor->root();
std::unique_ptr<ListIterator> list_iter;
+ // A vault that fails to list does not stop the check of the others
int ret = accessor->list_directory("data", &list_iter);
if (ret != 0) {
- return -1;
+ LOG(WARNING) << "failed to list data directory. uri=" <<
accessor->uri();
+ check_ret = -1;
+ continue;
}
for (auto file = list_iter->next(); file.has_value(); file =
list_iter->next()) {
+ if (stopped()) {
+ // Otherwise each remaining object would start a rowset read
cut short by the stop
+ LOG(INFO) << "inverted check stopped, instance_id=" <<
instance_id_;
+ return -1;
+ }
const auto& path = file->path;
if (path == "data/packed_file" ||
path.starts_with("data/packed_file/")) {
continue; // packed_file has dedicated check logic
}
++num_scanned;
- int ret = check_segment_file(path);
+ int ret = check_segment_file(path, root);
if (ret != 0) {
LOG(WARNING) << "failed to check segment file, uri=" <<
accessor->uri()
<< " path=" << path;
@@ -1255,7 +1287,7 @@ int InstanceChecker::do_inverted_check() {
if (!list_iter->is_valid()) {
LOG(WARNING) << "failed to list data directory. uri=" <<
accessor->uri();
- return -1;
+ check_ret = -1;
}
}
return num_file_leak > 0 ? 1 : check_ret;
diff --git a/cloud/src/recycler/hdfs_accessor.cpp
b/cloud/src/recycler/hdfs_accessor.cpp
index 4fddeab693f..556cd4b4773 100644
--- a/cloud/src/recycler/hdfs_accessor.cpp
+++ b/cloud/src/recycler/hdfs_accessor.cpp
@@ -29,7 +29,9 @@
#include <hdfs/hdfs.h> // IWYU pragma: export
#endif
+#include <string>
#include <string_view>
+#include <vector>
#include "common/config.h"
#include "common/defer.h"
@@ -325,11 +327,39 @@ private:
std::vector<DirEntries> level_dirs_;
};
-HdfsAccessor::HdfsAccessor(const HdfsVaultInfo& info)
- : StorageVaultAccessor(AccessorType::HDFS), info_(info),
prefix_(info.prefix()) {
- if (!prefix_.empty() && prefix_[0] != '/') {
- prefix_.insert(prefix_.begin(), '/');
+namespace {
+
+// Either empty or starts with '/', e.g. "p1" and "/p1" are the same directory
+std::string accessor_prefix(const HdfsVaultInfo& info) {
+ std::string prefix = info.prefix();
+ if (!prefix.empty() && prefix[0] != '/') {
+ prefix.insert(prefix.begin(), '/');
}
+ return prefix;
+}
+
+// A nameservice in fs_name resolves through hdfs_confs, e.g. hdfs://ns of two
vaults are different
+// clusters when their dfs.namenode.rpc-address.ns.* differ. All confs are
kept in order, a client
+// may apply the first or the last value of a repeated key. '\n' separates the
parts, so
+// hdfs://ns/a + /b and hdfs://ns + /a/b differ.
+std::string accessor_location(const HdfsVaultInfo& info) {
+ std::string location = info.build_conf().fs_name() + '\n' +
accessor_prefix(info);
+ for (const auto& conf : info.build_conf().hdfs_confs()) {
+ location += '\n';
+ location += conf.key();
+ location += '=';
+ location += conf.value();
+ }
+ return location;
+}
+
+} // namespace
+
+HdfsAccessor::HdfsAccessor(const HdfsVaultInfo& info)
+ : StorageVaultAccessor(AccessorType::HDFS, accessor_location(info),
+ storage_root("", accessor_prefix(info))),
+ info_(info),
+ prefix_(accessor_prefix(info)) {
uri_ = info.build_conf().fs_name() + prefix_;
}
diff --git a/cloud/src/recycler/hdfs_accessor.h
b/cloud/src/recycler/hdfs_accessor.h
index 7eb182f9122..0de1748c058 100644
--- a/cloud/src/recycler/hdfs_accessor.h
+++ b/cloud/src/recycler/hdfs_accessor.h
@@ -77,10 +77,9 @@ private:
HdfsSPtr fs_;
+ // The inherited `uri_` is fs_name/prefix, e.g.
hdfs://localhost:8020/test_prefix. It must not
+ // be shadowed here, otherwise `uri()` of the base class returns an empty
string.
std::string prefix_; // Either be empty or start with '/' if length > 1
- // format: fs_name/prefix/
- // e.g.: hdfs://localhost:8020/test_prefix/
- std::string uri_;
};
} // namespace doris::cloud
diff --git a/cloud/src/recycler/s3_accessor.cpp
b/cloud/src/recycler/s3_accessor.cpp
index b84a2e4c64a..5d90137e88d 100644
--- a/cloud/src/recycler/s3_accessor.cpp
+++ b/cloud/src/recycler/s3_accessor.cpp
@@ -56,6 +56,7 @@
#include "cpp/sync_point.h"
#include "cpp/token_bucket_rate_limiter.h"
#include "cpp/util.h"
+#include "meta-service/meta_service_helper.h"
#include "recycler/storage_vault_accessor.h"
#include "recycler/sync_executor.h"
@@ -314,8 +315,23 @@ std::optional<S3Conf> S3Conf::from_obj_store_info(const
ObjectStoreInfoPB& obj_i
return s3_conf;
}
+namespace {
+
+// Endpoint, bucket and prefix as configured, plus the credential identity:
gateways such as Ceph
+// RGW resolve one bucket name per tenant. The ak is hidden as in logs, the sk
is never part of it.
+// '\n' never occurs in an endpoint or a bucket name, so different confs never
collide.
+std::string s3_location(const S3Conf& conf) {
+ return fmt::format("{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", conf.endpoint,
conf.bucket, conf.prefix,
+ static_cast<int>(conf.provider),
hide_access_key(conf.ak), conf.role_arn,
+ conf.external_id,
static_cast<int>(conf.cred_provider_type));
+}
+
+} // namespace
+
S3Accessor::S3Accessor(S3Conf conf)
- : StorageVaultAccessor(AccessorType::S3), conf_(std::move(conf)) {}
+ : StorageVaultAccessor(AccessorType::S3, s3_location(conf),
+ storage_root(conf.bucket, conf.prefix)),
+ conf_(std::move(conf)) {}
S3Accessor::~S3Accessor() = default;
diff --git a/cloud/src/recycler/storage_vault_accessor.h
b/cloud/src/recycler/storage_vault_accessor.h
index 259148e544d..2d142d969c4 100644
--- a/cloud/src/recycler/storage_vault_accessor.h
+++ b/cloud/src/recycler/storage_vault_accessor.h
@@ -17,9 +17,12 @@
#pragma once
+#include <algorithm>
#include <memory>
#include <optional>
#include <string>
+#include <string_view>
+#include <utility>
#include <vector>
namespace Aws::S3 {
@@ -48,9 +51,38 @@ public:
virtual std::optional<FileMeta> next() = 0;
};
+// Where the objects of an accessor live, as far as it can be known: different
roots surely hold
+// different paths, equal roots may still be different storage. So `container`
is only the S3
+// bucket, an endpoint has aliases (http/https, internal/public), and is empty
for HDFS, whose
+// fs_name has aliases (host, FQDN, IP, nameservice). `prefix` is the
directory without empty path
+// components.
+struct StorageRoot {
+ std::string container;
+ std::string prefix;
+
+ bool operator==(const StorageRoot&) const = default;
+};
+
+// E.g. prefix "/a//b/" is "a/b"
+inline StorageRoot storage_root(std::string container, std::string_view
prefix) {
+ StorageRoot root {.container = std::move(container)};
+ for (size_t begin = 0; begin < prefix.size();) {
+ size_t end = std::min(prefix.find('/', begin), prefix.size());
+ if (end > begin) {
+ if (!root.prefix.empty()) {
+ root.prefix += '/';
+ }
+ root.prefix.append(prefix.substr(begin, end - begin));
+ }
+ begin = end + 1;
+ }
+ return root;
+}
+
class StorageVaultAccessor {
public:
- explicit StorageVaultAccessor(AccessorType type) : type_(type) {}
+ StorageVaultAccessor(AccessorType type, std::string location, StorageRoot
root)
+ : type_(type), location_(std::move(location)),
root_(std::move(root)) {}
virtual ~StorageVaultAccessor() = default;
@@ -61,6 +93,14 @@ public:
const std::string& uri() const { return uri_; }
+ // The storage this accessor lists: equal locations surely see the same
objects. It keeps the
+ // whole configuration that may select a namespace, unnormalized, e.g. S3
credentials (a gateway
+ // may map one bucket name per tenant) and HDFS confs in order. A false
difference only costs a
+ // second listing.
+ const std::string& location() const { return location_; }
+
+ const StorageRoot& root() const { return root_; }
+
// If `expiration_time` > 0, only delete objects with mtime earlier than
`expiration_time`.
// returns 0 for success otherwise error
virtual int delete_prefix(const std::string& path_prefix, int64_t
expiration_time = 0) = 0;
@@ -98,6 +138,8 @@ public:
private:
const AccessorType type_;
+ const std::string location_;
+ const StorageRoot root_;
protected:
std::string uri_;
diff --git a/cloud/test/mock_accessor.h b/cloud/test/mock_accessor.h
index 03fed7caeaf..fa939c4ed54 100644
--- a/cloud/test/mock_accessor.h
+++ b/cloud/test/mock_accessor.h
@@ -57,7 +57,10 @@ private:
class MockAccessor final : public StorageVaultAccessor {
public:
- explicit MockAccessor();
+ // `uri` is also the location, accessors with the same uri model one
storage location. The root
+ // defaults to the container `uri` with an empty prefix, so distinct mocks
are disjoint.
+ explicit MockAccessor(std::string uri = "mock");
+ MockAccessor(std::string uri, StorageRoot root);
~MockAccessor() override;
@@ -90,8 +93,14 @@ private:
std::set<std::string> objects_;
};
-inline MockAccessor::MockAccessor() : StorageVaultAccessor(AccessorType::MOCK)
{
- uri_ = "mock";
+inline MockAccessor::MockAccessor(std::string uri)
+ : StorageVaultAccessor(AccessorType::MOCK, uri, StorageRoot
{.container = uri}) {
+ uri_ = std::move(uri);
+}
+
+inline MockAccessor::MockAccessor(std::string uri, StorageRoot root)
+ : StorageVaultAccessor(AccessorType::MOCK, uri, std::move(root)) {
+ uri_ = std::move(uri);
}
inline MockAccessor::~MockAccessor() = default;
@@ -195,6 +204,7 @@ inline int MockAccessor::list_directory(const std::string&
dir_path,
return -1;
}
+ TEST_SYNC_POINT_RETURN_WITH_VALUE("MockAccessor::list_directory", (int)0,
this);
std::vector<std::string> entries;
{
diff --git a/cloud/test/recycler_test.cpp b/cloud/test/recycler_test.cpp
index 1b65192150f..2456d53ede4 100644
--- a/cloud/test/recycler_test.cpp
+++ b/cloud/test/recycler_test.cpp
@@ -5205,6 +5205,321 @@ TEST(CheckerTest, normal_inverted_check) {
ASSERT_EQ(checker.do_inverted_check(), 0);
}
+// A checker of an instance whose storage vaults are the mock vaults added by
each test
+class InvertedCheckTest : public testing::Test {
+protected:
+ void SetUp() override {
+ ASSERT_EQ(txn_kv->init(), 0);
+ InstanceInfoPB instance;
+ instance.set_instance_id(instance_id);
+ ASSERT_EQ(checker.init(instance), 0);
+ }
+
+ std::shared_ptr<MockAccessor> add_vault(const std::string& id,
+ std::optional<StorageRoot> root =
std::nullopt) {
+ auto vault = root ? std::make_shared<MockAccessor>("mock://" + id,
*root)
+ : std::make_shared<MockAccessor>("mock://" + id);
+ checker.TEST_add_accessor(id, vault);
+ return vault;
+ }
+
+ int64_t num_scanned() {
+ return read_status_bvar_value("checker", "num_inverted_scanned",
instance_id);
+ }
+ int64_t num_leaks() {
+ return read_status_bvar_value("checker", "num_inverted_check_failed",
instance_id);
+ }
+
+ std::shared_ptr<MemTxnKv> txn_kv = std::make_shared<MemTxnKv>();
+ InstanceChecker checker {txn_kv, instance_id};
+};
+
+// Puts the data objects of `from` into `to`, as storage seen by both vaults
+static void copy_data_objects(StorageVaultAccessor* from,
StorageVaultAccessor* to) {
+ std::unique_ptr<ListIterator> list_iter;
+ ASSERT_EQ(from->list_directory("data", &list_iter), 0);
+ for (auto file = list_iter->next(); file.has_value(); file =
list_iter->next()) {
+ ASSERT_EQ(to->put_file(file->path, ""), 0);
+ }
+}
+
+TEST_F(InvertedCheckTest, multi_vault) {
+ auto vault1 = add_vault("vault1");
+ auto vault2 = add_vault("vault2");
+ for (int t = 10001; t <= 10010; ++t) {
+ for (int v = 0; v < 3; ++v) {
+ create_committed_rowset(txn_kv.get(), vault1.get(), "vault1", t,
v, t * 2, 1, 0);
+ }
+ }
+ for (int t = 10011; t <= 10020; ++t) {
+ for (int v = 0; v < 3; ++v) {
+ create_committed_rowset(txn_kv.get(), vault2.get(), "vault2", t,
v, t * 2, 1, 0);
+ }
+ }
+ // Objects of both vaults are checked
+ ASSERT_EQ(checker.do_inverted_check(), 0);
+ ASSERT_EQ(num_scanned(), 60);
+ ASSERT_EQ(num_leaks(), 0);
+
+ // A leaked segment in each vault: one of a tablet without rowsets, one
listed after the live
+ // rowsets of its tablet ("zzz" sorts after the numeric test rowset ids)
+ ASSERT_EQ(vault1->put_file(segment_path(10030, next_rowset_id(), 0), ""),
0);
+ ASSERT_EQ(vault2->put_file(segment_path(10015, "zzz", 0), ""), 0);
+ ASSERT_EQ(checker.do_inverted_check(), 1);
+ ASSERT_EQ(num_scanned(), 62);
+ ASSERT_EQ(num_leaks(), 2);
+}
+
+TEST_F(InvertedCheckTest, same_location_once) {
+ // Two resources backed by the same storage location
+ auto accessor = add_vault("vault1");
+ checker.TEST_add_accessor("vault2", accessor);
+ for (int t = 10001; t <= 10010; ++t) {
+ create_committed_rowset(txn_kv.get(), accessor.get(), "vault1", t, 0,
t * 2, 1, 0);
+ }
+ ASSERT_EQ(accessor->put_file(segment_path(10030, next_rowset_id(), 0),
""), 0);
+ ASSERT_EQ(checker.do_inverted_check(), 1);
+ ASSERT_EQ(num_scanned(), 11);
+ ASSERT_EQ(num_leaks(), 1);
+}
+
+TEST_F(InvertedCheckTest, segment_in_another_vault_is_leak) {
+ auto vault1 = add_vault("vault1");
+ auto vault2 = add_vault("vault2");
+ // Vaults 3 and 4 address the same paths with different credentials, so
both are listed
+ StorageRoot root {.container = "bucket", .prefix = "p"};
+ auto vault3 = add_vault("vault3", root);
+ auto vault4 = add_vault("vault4", root);
+ create_committed_rowset(txn_kv.get(), vault1.get(), "vault1", 10001, 0,
20002, 1, 0);
+ create_committed_rowset(txn_kv.get(), vault3.get(), "vault3", 10002, 0,
20004, 1, 0);
+ // A copy of a vault1 segment in vault2 is not covered by its rowset
+ copy_data_objects(vault1.get(), vault2.get());
+ // A vault3 segment listed through vault4 is in the storage of its rowset
+ copy_data_objects(vault3.get(), vault4.get());
+
+ ASSERT_EQ(checker.do_inverted_check(), 1);
+ ASSERT_EQ(num_scanned(), 4);
+ ASSERT_EQ(num_leaks(), 1);
+}
+
+TEST_F(InvertedCheckTest, list_error_does_not_stop_other_vaults) {
+ auto vault1 = add_vault("vault1");
+ auto vault2 = add_vault("vault2");
+ create_committed_rowset(txn_kv.get(), vault2.get(), "vault2", 10001, 0,
20002, 1, 0);
+ ASSERT_EQ(vault2->put_file(segment_path(10030, next_rowset_id(), 0), ""),
0);
+
+ auto sp = SyncPoint::get_instance();
+ SyncPoint::CallbackGuard guard;
+ sp->set_call_back(
+ "MockAccessor::list_directory",
+ [&](auto&& args) {
+ if (try_any_cast<MockAccessor*>(args[0]) == vault1.get()) {
+ *try_any_cast_ret<int>(args) = {-1, true};
+ }
+ },
+ &guard);
+ sp->enable_processing();
+ DORIS_CLOUD_DEFER {
+ SyncPoint::get_instance()->disable_processing();
+ };
+
+ // vault2 is checked whichever vault is listed first, its leak is the
result
+ ASSERT_EQ(checker.do_inverted_check(), 1);
+ ASSERT_EQ(num_scanned(), 2);
+ ASSERT_EQ(num_leaks(), 1);
+}
+
+TEST_F(InvertedCheckTest, orphans_in_both_listing_orders) {
+ auto accessor = add_vault("1");
+ // Tablet 10001: a live rowset listed before two orphans ("zzz*" sort
after the numeric test
+ // rowset ids), the orphans are checked against the cached rowsets of the
tablet
+ create_committed_rowset(txn_kv.get(), accessor.get(), "1", 10001, 0,
20002, 1, 0);
+ ASSERT_EQ(accessor->put_file(segment_path(10001, "zzz1", 0), ""), 0);
+ ASSERT_EQ(accessor->put_file(segment_path(10001, "zzz2", 0), ""), 0);
+ // Tablet 10002: an orphan listed before the live rowset ("0000" sorts
first), the orphan loads
+ // the rowsets of the tablet and the live rowset is checked against the
cache
+ create_committed_rowset(txn_kv.get(), accessor.get(), "1", 10002, 0,
20004, 1, 0);
+ ASSERT_EQ(accessor->put_file(segment_path(10002, "0000", 0), ""), 0);
+
+ ASSERT_EQ(checker.do_inverted_check(), 1);
+ ASSERT_EQ(num_scanned(), 5);
+ ASSERT_EQ(num_leaks(), 3);
+}
+
+TEST_F(InvertedCheckTest, rowset_meta_read_error_is_not_leak) {
+ auto accessor = add_vault("1");
+ for (int v = 0; v < 2; ++v) {
+ create_committed_rowset(txn_kv.get(), accessor.get(), "1", 10001, v,
20002, 1, 0);
+ }
+
+ auto sp = SyncPoint::get_instance();
+ SyncPoint::CallbackGuard guard;
+ sp->set_call_back(
+ "InstanceChecker::do_inverted_check.get_rowsets_err",
+ [](auto&& args) { *try_any_cast<TxnErrorCode*>(args[0]) =
TxnErrorCode::TXN_CONFLICT; },
+ &guard);
+ sp->enable_processing();
+ DORIS_CLOUD_DEFER {
+ SyncPoint::get_instance()->disable_processing();
+ };
+
+ // Failing to read the rowsets of the tablet is an error for each of its
objects, never a leak
+ ASSERT_EQ(checker.do_inverted_check(), -1);
+ ASSERT_EQ(num_scanned(), 2);
+ ASSERT_EQ(num_leaks(), 0);
+}
+
+TEST_F(InvertedCheckTest, rowset_committed_during_scan_is_not_leak) {
+ auto accessor = add_vault("1");
+ create_committed_rowset(txn_kv.get(), accessor.get(), "1", 10001, 0,
20002, 1, 0);
+ // The segment of rowset "zzz" is uploaded, and its meta commits after the
checker loaded the
+ // rowsets of the tablet for the first object ("zzz" sorts after the
numeric rowset ids)
+ ASSERT_EQ(accessor->put_file(segment_path(10001, "zzz", 0), ""), 0);
+
+ int rowset_reads = 0;
+ auto sp = SyncPoint::get_instance();
+ SyncPoint::CallbackGuard guard;
+ sp->set_call_back(
+ "InstanceChecker::do_inverted_check.get_rowsets_err",
+ [&](auto&& args) {
+ if (++rowset_reads != 1) {
+ return;
+ }
+ doris::RowsetMetaCloudPB rowset;
+ rowset.set_rowset_id(0);
+ rowset.set_rowset_id_v2("zzz");
+ rowset.set_tablet_id(10001);
+ rowset.set_num_segments(1);
+ rowset.set_resource_id("1");
+ std::unique_ptr<Transaction> txn;
+ ASSERT_EQ(txn_kv->create_txn(&txn), TxnErrorCode::TXN_OK);
+ txn->put(meta_rowset_key({instance_id, 10001, 1}),
rowset.SerializeAsString());
+ ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
+ },
+ &guard);
+ sp->enable_processing();
+ DORIS_CLOUD_DEFER {
+ SyncPoint::get_instance()->disable_processing();
+ };
+
+ // The rowset missing from the cache is read again and found, so it is not
a leak
+ ASSERT_EQ(checker.do_inverted_check(), 0);
+ ASSERT_EQ(rowset_reads, 2);
+ ASSERT_EQ(num_scanned(), 2);
+ ASSERT_EQ(num_leaks(), 0);
+}
+
+TEST_F(InvertedCheckTest, stops_when_stopped) {
+ auto accessor = add_vault("1");
+ for (int v = 0; v < 3; ++v) {
+ create_committed_rowset(txn_kv.get(), accessor.get(), "1", 10001, v,
20002, 1, 0);
+ }
+
+ // The checker is stopped while the rowsets of the first object's tablet
are being read
+ int rowset_reads = 0;
+ auto sp = SyncPoint::get_instance();
+ SyncPoint::CallbackGuard guard;
+ sp->set_call_back(
+ "InstanceChecker::do_inverted_check.get_rowsets_err",
+ [&](auto&& args) {
+ ++rowset_reads;
+ checker.stop();
+ },
+ &guard);
+ sp->enable_processing();
+ DORIS_CLOUD_DEFER {
+ SyncPoint::get_instance()->disable_processing();
+ };
+
+ // No more objects are checked, so no more rowset reads are started
+ ASSERT_EQ(checker.do_inverted_check(), -1);
+ ASSERT_EQ(rowset_reads, 1);
+ ASSERT_EQ(num_scanned(), 1);
+ ASSERT_EQ(num_leaks(), 0);
+}
+
+#ifdef ENABLE_HDFS_STORAGE_VAULT
+TEST(HdfsAccessorLocationTest, location) {
+ auto make_info = [](const std::string& fs_name, const std::string& prefix)
{
+ HdfsVaultInfo info;
+ info.set_prefix(prefix);
+ info.mutable_build_conf()->set_fs_name(fs_name);
+ return info;
+ };
+ // No init and no HDFS needed: the location only depends on the
configuration. It lives here
+ // because hdfs_accessor_test skips everything without a test HDFS.
+ auto info1 = make_info("hdfs://nn1:8020", "p1");
+ auto info1_again = make_info("hdfs://nn1:8020", "p1");
+ auto info2 = make_info("hdfs://nn1:8020", "p2");
+ auto info3 = make_info("hdfs://nn2:8020", "p1");
+ HdfsAccessor accessor1(info1);
+ HdfsAccessor accessor1_again(info1_again);
+ HdfsAccessor accessor2(info2);
+ HdfsAccessor accessor3(info3);
+
+ EXPECT_EQ(accessor1.uri(), "hdfs://nn1:8020/p1");
+ EXPECT_EQ(accessor1.location(), "hdfs://nn1:8020\n/p1");
+ EXPECT_EQ(accessor1.location(), accessor1_again.location());
+ EXPECT_NE(accessor1.location(), accessor2.location());
+ EXPECT_NE(accessor1.location(), accessor3.location());
+
+ // HA: the same logical nameservice and prefix resolve through each
vault's hdfs_confs
+ auto make_ha_info = [&](std::vector<std::pair<std::string, std::string>>
confs) {
+ auto info = make_info("hdfs://ns", "/warehouse");
+ for (auto& [key, value] : confs) {
+ auto* conf = info.mutable_build_conf()->add_hdfs_confs();
+ conf->set_key(key);
+ conf->set_value(value);
+ }
+ return info;
+ };
+ auto cluster_a = make_ha_info({{"dfs.ha.namenodes.ns", "nn1,nn2"},
+ {"dfs.namenode.rpc-address.ns.nn1",
"a1:8020"},
+ {"dfs.namenode.rpc-address.ns.nn2",
"a2:8020"}});
+ auto cluster_a_reordered =
make_ha_info({{"dfs.namenode.rpc-address.ns.nn2", "a2:8020"},
+ {"dfs.ha.namenodes.ns",
"nn1,nn2"},
+
{"dfs.namenode.rpc-address.ns.nn1", "a1:8020"}});
+ auto cluster_b = make_ha_info({{"dfs.ha.namenodes.ns", "nn1,nn2"},
+ {"dfs.namenode.rpc-address.ns.nn1",
"b1:8020"},
+ {"dfs.namenode.rpc-address.ns.nn2",
"b2:8020"}});
+ HdfsAccessor accessor_a(cluster_a);
+ HdfsAccessor accessor_a_reordered(cluster_a_reordered);
+ HdfsAccessor accessor_b(cluster_b);
+ EXPECT_EQ(accessor_a.uri(), accessor_b.uri());
+ EXPECT_NE(accessor_a.location(), accessor_b.location());
+ // The confs are kept in order, a different order may resolve differently
and is listed again
+ EXPECT_NE(accessor_a.location(), accessor_a_reordered.location());
+ // The root ignores the confs and fs_name, which has aliases (host, FQDN,
IP, nameservice)
+ EXPECT_EQ(accessor_a.root(), accessor_b.root());
+ EXPECT_EQ(accessor_a.root(), (StorageRoot {.container = "", .prefix =
"warehouse"}));
+ EXPECT_EQ(accessor1.root(), accessor3.root());
+ EXPECT_NE(accessor1.root(), accessor2.root());
+
+ // fs_name and prefix do not merge: same uri, different directories
+ auto info_split_1 = make_info("hdfs://ns/a", "b");
+ auto info_split_2 = make_info("hdfs://ns", "a/b");
+ HdfsAccessor accessor_split_1(info_split_1);
+ HdfsAccessor accessor_split_2(info_split_2);
+ EXPECT_EQ(accessor_split_1.uri(), accessor_split_2.uri());
+ EXPECT_NE(accessor_split_1.location(), accessor_split_2.location());
+
+ // A repeated key is kept with all its values in order: which one takes
effect depends on the
+ // HDFS client, so neither value is dropped
+ auto repeated = make_ha_info({{"dfs.ha.namenodes.ns", "nn1,nn2"},
+ {"dfs.namenode.rpc-address.ns.nn1",
"a1:8020"},
+ {"dfs.namenode.rpc-address.ns.nn2",
"a2:8020"},
+ {"dfs.namenode.rpc-address.ns.nn1",
"b1:8020"}});
+ auto repeated_reversed = make_ha_info({{"dfs.ha.namenodes.ns", "nn1,nn2"},
+ {"dfs.namenode.rpc-address.ns.nn1",
"b1:8020"},
+ {"dfs.namenode.rpc-address.ns.nn2",
"a2:8020"},
+ {"dfs.namenode.rpc-address.ns.nn1",
"a1:8020"}});
+ HdfsAccessor accessor_repeated(repeated);
+ HdfsAccessor accessor_repeated_reversed(repeated_reversed);
+ EXPECT_NE(accessor_repeated.location(), accessor_a.location());
+ EXPECT_NE(accessor_repeated.location(),
accessor_repeated_reversed.location());
+}
+#endif
+
// TODO(Xiaocc): We need one mocked accessor which provides one async stream
like list function
// to do the following test
TEST(CheckerTest, DISABLED_abnormal_inverted_check) {
diff --git a/cloud/test/s3_accessor_mock_test.cpp
b/cloud/test/s3_accessor_mock_test.cpp
index 5e9d985777e..2225adb3eef 100644
--- a/cloud/test/s3_accessor_mock_test.cpp
+++ b/cloud/test/s3_accessor_mock_test.cpp
@@ -34,6 +34,7 @@
#include "cpp/obj-client/s3_obj_storage_client.h"
#include "cpp/sync_point.h"
#include "cpp/test/container_credentials_test_util.h"
+#include "meta-service/meta_service_helper.h"
#include "recycler/s3_accessor.h"
using namespace doris;
@@ -160,4 +161,58 @@ TEST_F(S3AccessorMockTest,
container_provider_is_unusable_without_any_uri) {
EXPECT_EQ(as_valid_http_provider(provider), nullptr);
}
+TEST(S3AccessorLocationTest, location_is_the_configured_location) {
+ auto location = [](std::string endpoint, std::string bucket, std::string
prefix) {
+ return S3Accessor(S3Conf {.endpoint = std::move(endpoint),
+ .bucket = std::move(bucket),
+ .prefix = std::move(prefix)})
+ .location();
+ };
+ EXPECT_EQ(location("endpoint", "bucket", "a/b"), location("endpoint",
"bucket", "a/b"));
+ // Prefixes with repeated '/' hold different objects although their
normalized uri is the same
+ EXPECT_NE(location("endpoint", "bucket", "a/b"), location("endpoint",
"bucket", "a//b"));
+ EXPECT_NE(location("endpoint", "bucket", "a/b"), location("endpoint",
"bucket2", "a/b"));
+ EXPECT_NE(location("endpoint", "bucket", "a/b"), location("endpoint2",
"bucket", "a/b"));
+}
+
+TEST(S3AccessorLocationTest, location_includes_the_credential_identity) {
+ auto accessor = [](std::string ak, std::string sk, std::string role_arn) {
+ return std::make_unique<S3Accessor>(S3Conf {.ak = std::move(ak),
+ .sk = std::move(sk),
+ .endpoint = "endpoint",
+ .bucket = "bucket",
+ .prefix = "/a/b/",
+ .role_arn =
std::move(role_arn)});
+ };
+ const std::string ak1 = "AKIAIOSFODNN7EXAMPLE";
+ const std::string ak2 = "AKIAI44QH8DHBEXAMPLE";
+ auto base = accessor(ak1, "sk1", "");
+ // An S3-compatible gateway may map one bucket name to the buckets of
different tenants by the
+ // credentials, so another identity is listed again
+ EXPECT_NE(base->location(), accessor(ak2, "sk1", "")->location());
+ EXPECT_NE(base->location(), accessor(ak1, "sk1", "role")->location());
+ // The secret key is not part of the location, it identifies no namespace
of its own
+ EXPECT_EQ(base->location(), accessor(ak1, "sk2", "")->location());
+ EXPECT_EQ(base->location().find("sk1"), std::string::npos);
+ // The ak is hidden as in logs
+ EXPECT_EQ(base->location().find(ak1), std::string::npos);
+ EXPECT_NE(base->location().find(hide_access_key(ak1)), std::string::npos);
+ // The root ignores the credentials
+ EXPECT_EQ(base->root(), accessor(ak2, "sk2", "role")->root());
+ EXPECT_EQ(base->root(), (StorageRoot {.container = "bucket", .prefix =
"a/b"}));
+}
+
+TEST(S3AccessorLocationTest, root_ignores_endpoint_aliases) {
+ auto accessor = [](std::string endpoint, std::string prefix) {
+ return std::make_unique<S3Accessor>(S3Conf {
+ .endpoint = std::move(endpoint), .bucket = "bucket", .prefix =
std::move(prefix)});
+ };
+ // Different locations, which are listed separately, may be the same
bucket: they share a root
+ auto http = accessor("http://s3.example.com", "a/b");
+ auto https = accessor("https://s3.example.com", "a//b/");
+ EXPECT_NE(http->location(), https->location());
+ EXPECT_EQ(http->root(), https->root());
+ EXPECT_NE(http->root(), accessor("http://s3.example.com", "a/c")->root());
+}
+
} // namespace doris::cloud
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]