This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 3b417f25f67 branch-4.0:[enhancement](s3)Add TotalGetRequestTime
profile metric to the S3 reader. (#57519) (#57635)
3b417f25f67 is described below
commit 3b417f25f6755ce8607a4a582f861d1f1415f5bd
Author: daidai <[email protected]>
AuthorDate: Wed Nov 5 23:52:10 2025 +0800
branch-4.0:[enhancement](s3)Add TotalGetRequestTime profile metric to the
S3 reader. (#57519) (#57635)
bp #57519
Problem Summary:
This PR adds a `TotalGetRequestTime` profile metric to the S3 reader.
The new metric records the total time spent on all `GetObject` requests,
which will be useful for future performance analysis and optimization.
Example profile (Some less relevant information was manually removed.):
```
FILE_SCAN_OPERATOR (id=0. nereids_id=104. table name = lineitem):
CommonCounters:
- ExecTime: 11sec669ms
- OpenTime: 7.395ms
- WaitForDependency[FILE_SCAN_OPERATOR_DEPENDENCY]Time:
11sec634ms
....
CustomCounters:
....
Scanner:
....
- FileReadTime: 4min8sec
- FileScannerGetBlockTime: 4min9sec
- MergedSmallIO: 0ns
- ReadTime: 4min8sec
- RequestBytes: 2.05 MB
- RequestIO: 20.0K (20000)
- ParquetReader: 0ns
- ColumnReadTime: 4min9sec
- FileOpenTime: 365.293ms
- ParseFooterTime: 5min6sec
- ParseMetaTime: 126.383ms
.....
- S3Profile: 0ns
- TooManyRequestErr: 0
- TooManyRequestSleepTime: 0ms
- TotalBytesRead: 199.14 MB
- TotalGetRequestTime: 9min12sec
- TotalGetRequest: 40.0K (40000)
- ScannerCpuTime: 20sec946ms
- ScannerGetBlockTime: 9min19sec
```
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---
be/src/io/fs/s3_file_reader.cpp | 4 ++++
be/src/io/fs/s3_file_reader.h | 1 +
2 files changed, 5 insertions(+)
diff --git a/be/src/io/fs/s3_file_reader.cpp b/be/src/io/fs/s3_file_reader.cpp
index eede868468f..f89c3f0196e 100644
--- a/be/src/io/fs/s3_file_reader.cpp
+++ b/be/src/io/fs/s3_file_reader.cpp
@@ -138,6 +138,7 @@ Status S3FileReader::read_at_impl(size_t offset, Slice
result, size_t* bytes_rea
.tag("key", _key);
std::this_thread::sleep_for(std::chrono::seconds(sleep_time));
});
+ SCOPED_RAW_TIMER(&_s3_stats.total_get_request_time_ns);
int total_sleep_time = 0;
while (retry_count <= max_retries) {
@@ -204,11 +205,14 @@ void S3FileReader::_collect_profile_before_close() {
_profile, "TooManyRequestSleepTime", TUnit::TIME_MS,
s3_profile_name);
RuntimeProfile::Counter* total_bytes_read =
ADD_CHILD_COUNTER(_profile, "TotalBytesRead", TUnit::BYTES,
s3_profile_name);
+ RuntimeProfile::Counter* total_get_request_time_ns =
+ ADD_CHILD_TIMER(_profile, "TotalGetRequestTime",
s3_profile_name);
COUNTER_UPDATE(total_get_request_counter,
_s3_stats.total_get_request_counter);
COUNTER_UPDATE(too_many_request_err_counter,
_s3_stats.too_many_request_err_counter);
COUNTER_UPDATE(too_many_request_sleep_time,
_s3_stats.too_many_request_sleep_time_ms);
COUNTER_UPDATE(total_bytes_read, _s3_stats.total_bytes_read);
+ COUNTER_UPDATE(total_get_request_time_ns,
_s3_stats.total_get_request_time_ns);
}
}
diff --git a/be/src/io/fs/s3_file_reader.h b/be/src/io/fs/s3_file_reader.h
index 36fe67b342c..58294ec1891 100644
--- a/be/src/io/fs/s3_file_reader.h
+++ b/be/src/io/fs/s3_file_reader.h
@@ -65,6 +65,7 @@ private:
int64_t too_many_request_err_counter = 0;
int64_t too_many_request_sleep_time_ms = 0;
int64_t total_bytes_read = 0;
+ int64_t total_get_request_time_ns = 0;
};
Path _path;
size_t _file_size;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
