Dialong commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5585092782
## Deterministic reproduction of the BE SIGSEGV via out-of-range dictionary
codes (CRC-valid corruption)
We hit the same class of crash in a production incident (BE crash loop when
reading corrupted tablets, triggered via **base compaction**, build
`doris-4.0.2-rc02`) and have now built a **deterministic, software-only
reproducer** for it — no real corrupted hardware data needed. We hope this
helps close the loop on the investigation and on PR #66832.
### Environment
- Apache Doris 4.0.2-rc02 (`doris-4.0.2-rc02-30d2df04594`), 1 FE + 1 BE,
single replica, local Docker (arm64; the defect is architecture-independent)
- Table: 4 × `VARCHAR` columns (all written with `DICT_ENCODING`), 5 rows, 1
tablet
### Method
A segment file is `[page]* [footer][footer_size][crc32c]["D0R1"]`; each
page's CRC32C covers `body + footer_buf + footer_size`. We keep **every CRC
valid** while making the content invalid ("parseable but illegal", the
torn-write shape):
1. For a dict-encoded VARCHAR column, the data page stores int32 dictionary
codes as `4B marker + bitshuffle+LZ4 stream` (the LZ4 payload has a 4-byte
big-endian block-length prefix), and the dict page stores the strings + offsets
+ num_values (BinaryPlain).
2. We decompress the payload, change **one dictionary code to
`0x7FFFFFFF`**, re-encode bitshuffle+LZ4. (`0x7FFFFFFF` expands to 31
consecutive `0x01` bytes in bit-plane layout, so the LZ4 output has exactly the
same length as the original — the replacement is in-place; only 12 bytes differ
in the whole file.)
3. Recompute the page CRC32C. Page size, all page pointers and the footer
stay untouched.
Result: the corrupted page **passes all checksum validation**, but the first
dictionary code is 8 GB past the end of the dictionary.
### Result: SIGSEGV on 4/4 read paths
| Query | Outcome |
|---|---|
| `SELECT * FROM t` | **BE SIGSEGV** |
| `SELECT * FROM t WHERE dict_col = 'x'` | **BE SIGSEGV** |
| `SELECT DISTINCT dict_col FROM t` | **BE SIGSEGV** |
| `SELECT dict_col, COUNT(*) FROM t GROUP BY dict_col` | **BE SIGSEGV** |
Crash stack (stable across crashes):
```
#4 doris::vectorized::ColumnStr<unsigned int>::insert_many_dict_data(
int const*, unsigned long, doris::StringRef const*, unsigned long,
unsigned int)
at column_string.h:341
#5 doris::segment_v2::BinaryDictPageDecoder::next_batch
binary_dict_page.cpp:292
#6 doris::segment_v2::FileColumnIterator::next_batch
column_reader.cpp:1367
#7 doris::segment_v2::SegmentIterator::_read_columns_by_index
segment_iterator.cpp:2078
...
#17 doris::vectorized::OlapScanner::open
olap_scanner.cpp:287
#18 doris::vectorized::ScannerScheduler::_scanner_scan
```
Root cause: `BinaryDictPageDecoder::next_batch` hands the raw int32 codes to
`insert_many_dict_data`, which indexes `dict[codeword]` with **no bounds check
against `_num_dict_items`** (note the parameter is passed but commented out:
`uint32_t /*dict_num*/`). An out-of-range code produces a wild `StringRef`; the
subsequent `memcpy` segfaults.
### Relation to this issue's two signatures
- Signature 2 (`ColumnDictI32::filter_by_selector → SegmentIterator →
OlapScanner`): same root cause, different use site (predicate path vs.
materialization path). Our stack confirms the suspected **missing
dictionary-code bounds check** — this is the first deterministic experimental
confirmation of that hypothesis, as far as we know.
- Signature 1 (`DistinctStreamingAgg → ColumnStr::serialize_impl → memcpy`):
in a separate experiment (dict `num_values` shrunk 5→2, CRC-valid), `SELECT
DISTINCT dict_col` **silently returned corrupted garbage to the client**
instead of failing — i.e. the aggregation path also forwards invalid column
state without validation. Whether it segfaults or returns garbage just depends
on whether the wild `Slice` lands in unmapped memory.
### What already protects us (4.0.2 measured)
| Layer | Status |
|---|---|
| Page CRC32C | works — naive bit-flip → graceful `checksum mismatch` |
| Dict page offsets range check at init | works — graceful `offsets pos
beyonds data_size` |
| `ColumnString` insert length guard | works — graceful `[E6] ColumnString
insert size out of range` |
| **Dictionary-code vs. dict-size bounds check** | **missing → SIGSEGV (this
issue)** |
| Aggregation input validation (DISTINCT) | partially missing → silent
garbage |
### Assessment of PR #66832
The fix direction is **exactly right**: validating codes in
`BinaryDictPageDecoder::next_batch` / `read_by_rowids` sits directly upstream
of our observed crash site and would have turned all 4 of our SIGSEGVs into
`Status::Corruption`. Two residual gaps it does **not** cover, which may
deserve follow-ups:
1. The DISTINCT/aggregation path silently returning garbage for invalid
column state (our experiment above).
2. Signature 1's `serialize_impl` crash — same "invalid `ColumnString` state
propagated downstream" family, but the propagation point is not the dict-code
read.
### Additional production evidence (compaction-triggered)
Answering the maintainer's earlier question — yes, in our production
incident the crash was produced by **compaction, not queries**: a BE crash
looped 41 times on `vertical compaction` over corrupted tablets (stack in
`merger.cpp`), until the corrupted data directory was removed. We also
confirmed experimentally that with a corrupted tablet present, **FE
auto-analyze internal queries** (`MIN/MAX/NDV(col)` full-table scans) will
crash the BE within ~1–2 minutes of startup, forming a crash loop even when
compaction is not scheduled — i.e. any periodic automatic read path can act as
the re-trigger.
### Attachments
- `exp_d_code_oob.dat` — the 1313-byte reproducer segment (12 bytes
different from a healthy one; contains only synthetic test rows)
- `fresh.dat` — the healthy baseline for diffing
- `all_crash_stacks.txt` — full `be.out` failure-handler stacks
- `scan_pages2.py` / `bshuf_tool.py` — page-layout scanner and
bitshuffle+LZ4 round-trip tooling used to build the reproducer
Happy to provide any further detail (full experiment matrix, the production
incident's tablet list and timeline, or a private upload channel for the real
corrupted tablet directory from our production incident, which reproduces the
compaction-path crash 100% on the same build).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
