yiguolei commented on code in PR #69635:
URL: https://github.com/apache/doris/pull/69635#discussion_r4533785021
##########
be/src/exec/spill/spill_file_manager.cpp:
##########
@@ -152,15 +215,31 @@ void SpillFileManager::stop() {
}
// The GC thread may observe the stop latch before processing a recently
queued failed deletion.
// Retry the pending directories after the thread exits; later failures
get one final retry in
- // the destructor.
- _retry_pending_query_spill_directories();
+ // the destructor. Neither retries remote directories, so that an
unavailable object store does
+ // not delay shutdown; they stay queued for a later gc() and are otherwise
left to the
+ // lifecycle rule of the bucket.
+ _retry_pending_local_spill_directories();
}
Status SpillFileManager::init() {
LOG(INFO) << "init spill stream manager";
+ if (config::spill_storage_type == "s3") {
+ if (!config::is_cloud_mode()) {
+ return Status::InvalidArgument("spill_storage_type=s3 is only
supported in cloud mode");
+ }
+ if (config::spill_file_part_size_bytes < config::s3_write_buffer_size)
{
+ // Every part ends with a partially filled upload buffer that
still allocates
+ // s3_write_buffer_size, so tiny parts waste upload memory.
Review Comment:
如果这里，你觉得有问题。 我们应该打印warming log，然后改变一下，比如 std::max spill_file_part_size_bytes
s3_write_buffer_size
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
