cauchy1988 opened a new pull request, #2404: URL: https://github.com/apache/incubator-pegasus/pull/2404
## Problem When a replica server crashes during `DB::Open` (e.g. OOM, `kill -9`), it may leave an incomplete rdb directory — the MANIFEST contains only partial column families (e.g. only `pegasus_data` without `pegasus_meta`). On restart, the server hits: ``` CHECK_PREFIX_MSG(!missing_meta_cf, "You must upgrade Pegasus server from 2.0"); ``` and aborts immediately, making the replica unrecoverable. ## Root Cause - The crash leaves a partially created RocksDB directory (`rdb_path` exists but is incomplete) - On restart, `db_exist = true` and the code enters the "open existing DB" path - `check_column_families()` detects `missing_meta_cf = true` or `missing_data_cf = true` - The `CHECK_PREFIX_MSG` causes a hard abort An incomplete DB is unusable anyway (cannot read decree, data version, etc. from it), so crashing is unnecessary. ## Fix When `missing_meta_cf || missing_data_cf` is detected: 1. Log a WARNING with details about the incomplete state 2. Remove the corrupted `rdb` directory 3. Set `db_exist = false` 4. Fall through to create a fresh DB — the replica will re-sync data from the primary The existing `db_exist` branch (LoadLatestOptions etc.) is wrapped in `if (db_exist)` to avoid operating on the deleted directory. ## Testing - Verified that the fix resolves the crash in Docker development environment (Docker for Mac ARM64) - The replica successfully creates a new DB and re-syncs from primary after the corrupted directory is removed ## Related Closes #2215 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
