vagetablechicken commented on a change in pull request #3260: [Storage] open
data dirs parallelly
URL: https://github.com/apache/incubator-doris/pull/3260#discussion_r403407340
##########
File path: be/src/olap/storage_engine.cpp
##########
@@ -148,16 +148,33 @@ void StorageEngine::load_data_dirs(const
std::vector<DataDir*>& data_dirs) {
OLAPStatus StorageEngine::_open() {
// init store_map
+ std::vector<std::pair<DataDir*,std::future<Status>>> tmp_vec;
for (auto& path : _options.store_paths) {
+ LOG(INFO) << "store_path " << path.path;
DataDir* store = new DataDir(path.path, path.capacity_bytes,
path.storage_medium,
_tablet_manager.get(),
_txn_manager.get());
- auto st = store->init();
- if (!st.ok()) {
- LOG(WARNING) << "Store load failed, path=" << path.path;
- return OLAP_ERR_INVALID_ROOT_PATH;
+ tmp_vec.emplace_back(std::make_pair(store,std::async([store](){ return
store->init();})));
+ }
+
+ try {
+ for (auto& pair : tmp_vec) {
+ DataDir* store = pair.first;
+ auto st = pair.second.get();
+ if (!st.ok()) {
+ throw std::runtime_error("Store load failed, path=" +
store->path());
Review comment:
Yes, I forgot it's not safe to delete stores in catch block.
I think using exception in the phase of storage initialization should be no
problem. But we should wait all threads finished, so catching exceptions can't
make the code clean.
So I will modify the code like
```
std::vector<status> sts;
std::vector<std::thread> threads;
for dir in data dirs:
threads.emplace_back([&sts[i],dir](){ sts[i}=dir->init() });
for_each(threads, [](t){ t.join();})
for st in sts:
if(st.ok()) { ... }
else { init_error=true; break; }
if(init_error) {
// clean up
return error;
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
