advisedy commented on issue #3100: URL: https://github.com/apache/kvrocks/issues/3100#issuecomment-4956249489
@PragmaTwice ，I've been looking into this issue and believe the crash comes from undefined behavior in GetFullReplDataInfo: when the checkpoint file list is empty, the code still calls files->pop_back() (you can see [std::basic_string::pop_back](https://en.cppreference.com/cpp/string/basic_string/pop_back)). This can corrupt the string's internal state, so that later when _fetch_meta tries to concatenate files + CRLF, it attempts a huge memory allocation, which trips TSan. The list becomes empty because of a mutex race on the checkpoint directory: GetFullReplDataInfo releases checkpoint_mu_ before calling GetChildren, while the cron path calls rocksdb::DestroyDB(checkpoint_dir) without holding that mutex. As a result, GetChildren may observe an empty directory (this issue). I also realize that in this scenario, it could also observe a partial file list — although unlike the "list then delete during _fetch_file" case (which I believe is already handled internally), a partial list is still problematic: an incomplete file set could be mistaken for a complete one. I plan to try the following fix: When purging the checkpoint, under checkpoint_mu_, rename checkpoint_dir aside, then run DestroyDB on the trash path outside the lock. That way _fetch_meta should not list a half-deleted live directory. Additionally, if the effective file list is empty, return an error instead of calling pop_back() on an empty string. If this approach looks reasonable to you, please let me know and I'll open a PR along these lines. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
