Re: [PR] [branch-4.0] [Opt](cloud) Support packed file for delete bitmap storage [doris]
yiguolei merged PR #61007: URL: https://github.com/apache/doris/pull/61007 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [branch-4.0] [Opt](cloud) Support packed file for delete bitmap storage [doris]
github-actions[bot] commented on PR #61007: URL: https://github.com/apache/doris/pull/61007#issuecomment-4044555914 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [branch-4.0] [Opt](cloud) Support packed file for delete bitmap storage [doris]
github-actions[bot] commented on PR #61007: URL: https://github.com/apache/doris/pull/61007#issuecomment-4044556110 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [branch-4.0] [Opt](cloud) Support packed file for delete bitmap storage [doris]
hello-stephen commented on PR #61007: URL: https://github.com/apache/doris/pull/61007#issuecomment-3993068173 # BE UT Coverage Report Increment line coverage `5.56% (6/108)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/575a29da8786088026407f0606085e150193f9e1_575a29da8786088026407f0606085e150193f9e1/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/575a29da8786088026407f0606085e150193f9e1_575a29da8786088026407f0606085e150193f9e1/report/index.html) | Category | Coverage | |---|| | Function Coverage | 52.98% (19139/36124) | | Line Coverage | 36.18% (178244/492685) | | Region Coverage | 32.75% (138050/421523) | | Branch Coverage | 33.74% (59976/177753) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [branch-4.0] [Opt](cloud) Support packed file for delete bitmap storage [doris]
hello-stephen commented on PR #61007: URL: https://github.com/apache/doris/pull/61007#issuecomment-3992368948 # FE UT Coverage Report Increment line coverage `` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61007_575a29da8786088026407f0606085e150193f9e1/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61007_575a29da8786088026407f0606085e150193f9e1/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [branch-4.0] [Opt](cloud) Support packed file for delete bitmap storage [doris]
doris-robot commented on PR #61007: URL: https://github.com/apache/doris/pull/61007#issuecomment-3992123794 # Cloud UT Coverage Report Increment line coverage `22.43% (48/214)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/575a29da8786088026407f0606085e150193f9e1_575a29da8786088026407f0606085e150193f9e1_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/575a29da8786088026407f0606085e150193f9e1_575a29da8786088026407f0606085e150193f9e1_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 79.15% (1788/2259) | | Line Coverage | 64.49% (31945/49534) | | Region Coverage | 65.31% (15983/24471) | | Branch Coverage | 55.88% (8503/15216) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [branch-4.0] [Opt](cloud) Support packed file for delete bitmap storage [doris]
Copilot commented on code in PR #61007:
URL: https://github.com/apache/doris/pull/61007#discussion_r2879052338
##
cloud/src/recycler/recycler.cpp:
##
@@ -3224,6 +3238,225 @@ int
InstanceRecycler::decrement_packed_file_ref_counts(const doris::RowsetMetaCl
return ret;
}
+int InstanceRecycler::decrement_delete_bitmap_packed_file_ref_counts(int64_t
tablet_id,
+ const
std::string& rowset_id,
+ bool*
out_is_packed) {
+if (out_is_packed) {
+*out_is_packed = false;
+}
+
+// Get delete bitmap storage info from FDB
+std::string dbm_key = versioned::meta_delete_bitmap_key({instance_id_,
tablet_id, rowset_id});
+std::unique_ptr txn;
+TxnErrorCode err = txn_kv_->create_txn(&txn);
+if (err != TxnErrorCode::TXN_OK) {
+LOG_WARNING("failed to create txn when getting delete bitmap storage")
+.tag("instance_id", instance_id_)
+.tag("tablet_id", tablet_id)
+.tag("rowset_id", rowset_id)
+.tag("err", err);
+return -1;
+}
+
+std::string dbm_val;
+err = txn->get(dbm_key, &dbm_val);
+if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+// No delete bitmap for this rowset, nothing to do
+LOG_INFO("delete bitmap not found, skip packed file ref count
decrement")
+.tag("instance_id", instance_id_)
+.tag("tablet_id", tablet_id)
+.tag("rowset_id", rowset_id);
+return 0;
+}
+if (err != TxnErrorCode::TXN_OK) {
+LOG_WARNING("failed to get delete bitmap storage")
+.tag("instance_id", instance_id_)
+.tag("tablet_id", tablet_id)
+.tag("rowset_id", rowset_id)
+.tag("err", err);
+return -1;
+}
+
+DeleteBitmapStoragePB storage;
+if (!storage.ParseFromString(dbm_val)) {
+LOG_WARNING("failed to parse delete bitmap storage")
+.tag("instance_id", instance_id_)
+.tag("tablet_id", tablet_id)
+.tag("rowset_id", rowset_id);
+return -1;
+}
+
+// Check if delete bitmap is stored in packed file
+if (!storage.has_packed_slice_location() ||
+storage.packed_slice_location().packed_file_path().empty()) {
+// Not stored in packed file, nothing to do
+return 0;
+}
+
+if (out_is_packed) {
+*out_is_packed = true;
+}
+
+const auto& packed_loc = storage.packed_slice_location();
+const std::string& packed_file_path = packed_loc.packed_file_path();
+
+LOG_INFO("decrementing delete bitmap packed file ref count")
+.tag("instance_id", instance_id_)
+.tag("tablet_id", tablet_id)
+.tag("rowset_id", rowset_id)
+.tag("packed_file_path", packed_file_path);
+
+const int max_retry_times = std::max(1,
config::decrement_packed_file_ref_counts_retry_times);
+for (int attempt = 1; attempt <= max_retry_times; ++attempt) {
+std::unique_ptr update_txn;
+err = txn_kv_->create_txn(&update_txn);
+if (err != TxnErrorCode::TXN_OK) {
+LOG_WARNING("failed to create txn for delete bitmap packed file
update")
+.tag("instance_id", instance_id_)
+.tag("tablet_id", tablet_id)
+.tag("rowset_id", rowset_id)
+.tag("err", err);
+return -1;
+}
+
+std::string packed_key = packed_file_key({instance_id_,
packed_file_path});
+std::string packed_val;
+err = update_txn->get(packed_key, &packed_val);
+if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+LOG_WARNING("packed file info not found for delete bitmap")
+.tag("instance_id", instance_id_)
+.tag("tablet_id", tablet_id)
+.tag("rowset_id", rowset_id)
+.tag("packed_file_path", packed_file_path);
+return 0;
+}
+if (err != TxnErrorCode::TXN_OK) {
+LOG_WARNING("failed to get packed file info for delete bitmap")
+.tag("instance_id", instance_id_)
+.tag("tablet_id", tablet_id)
+.tag("rowset_id", rowset_id)
+.tag("packed_file_path", packed_file_path)
+.tag("err", err);
+return -1;
+}
+
+cloud::PackedFileInfoPB packed_info;
+if (!packed_info.ParseFromString(packed_val)) {
+LOG_WARNING("failed to parse packed file info for delete bitmap")
+.tag("instance_id", instance_id_)
+.tag("tablet_id", tablet_id)
+.tag("rowset_id", rowset_id)
+.tag("packed_file_path", packed_file_path);
+return -1;
+
Re: [PR] [branch-4.0] [Opt](cloud) Support packed file for delete bitmap storage [doris]
liaoxin01 commented on PR #61007: URL: https://github.com/apache/doris/pull/61007#issuecomment-3991915536 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [branch-4.0] [Opt](cloud) Support packed file for delete bitmap storage [doris]
Thearas commented on PR #61007: URL: https://github.com/apache/doris/pull/61007#issuecomment-3991870721 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[PR] [branch-4.0] [Opt](cloud) Support packed file for delete bitmap storage [doris]
liaoxin01 opened a new pull request, #61007: URL: https://github.com/apache/doris/pull/61007 ## Summary - Cherry-pick merged master commit 59e5f6bbf7f (from #60411) to branch-4.0. - Backport packed-file support for delete bitmap writer/reader and related regression tests. ## Conflict Resolution - Resolved conflict in be/src/cloud/cloud_meta_mgr.cpp by keeping branch-4.0 structure and porting packed delete bitmap storage logic, including packed slice location persistence. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
