yiguolei commented on code in PR #69635:
URL: https://github.com/apache/doris/pull/69635#discussion_r4534755975
##########
be/src/exec/spill/spill_file.cpp:
##########
@@ -35,47 +40,122 @@
namespace doris {
#include "common/compile_check_begin.h"
-SpillFile::SpillFile(SpillDataDir* data_dir, std::string relative_path)
+SpillFile::SpillFile(SpillDataDir* data_dir, io::FileSystemSPtr fs,
std::string relative_path)
: _data_dir(data_dir),
- _spill_dir(data_dir->get_spill_data_path() + "/" +
std::move(relative_path)) {}
+ _fs(std::move(fs)),
+ _query_id(relative_path.substr(0, relative_path.find('/'))),
+ _spill_dir(data_dir->get_spill_data_path() + "/" +
std::move(relative_path)),
+ _log_path(_spill_dir) {
+ if (_data_dir->is_remote()) {
+ auto* s3_fs = dynamic_cast<io::S3FileSystem*>(_fs.get());
+ DORIS_CHECK(s3_fs != nullptr);
+ _log_path = fmt::format("s3://{}/{}", s3_fs->bucket(),
+ (io::Path(s3_fs->prefix()) /
_spill_dir).native());
+ }
+ _data_dir->record_spill_file_created();
+ LOG(INFO) << "[spill][create] " << _log_context() << ", status=OK";
+}
+
+SpillFile::SpillFile(SpillDataDir* data_dir, std::string relative_path)
+ : SpillFile(data_dir, data_dir->fs(), std::move(relative_path)) {
+ DORIS_CHECK(!data_dir->is_remote());
+}
SpillFile::~SpillFile() {
gc();
Review Comment:
这里把gc 整体变成异步吧，不要再gc 里搞事情了，容易把query context 析构变慢，从而内存释放变慢
##########
be/src/exec/spill/spill_file_manager.cpp:
##########
@@ -345,278 +499,300 @@ void SpillFileManager::delete_spill_file(SpillFileSPtr
spill_file) {
void SpillFileManager::delete_query_spill_directory(const std::string&
query_id,
SpillDataDir* data_dir) {
- PendingQuerySpillDirectory pending_directory {
- .query_dir = data_dir->get_spill_data_path(query_id),
+ if (data_dir->is_remote()) {
Review Comment:
而且我有一个问题，我们当一个query context 析构的时候，对应的SpillFile
对象析构，在这对象析构的时候，就会删除对应的文件啊，为什么在query context 中需要单独处理呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
