YanzhiJin5 commented on issue #63609: URL: https://github.com/apache/doris/issues/63609#issuecomment-5032630084
Hi @ShummGen, thank you for reporting this issue and for providing the additional crash and tablet information. I am interested in investigating this problem and, if we can identify a reproducible root cause, trying to submit a fix. I performed an initial review of the relevant Doris code. The crash location in `ColumnStr::serialize_impl()` appears to be where the invalid state is finally exposed, but it may not be the original source of the problem. At the segment page-reading layer, Doris is expected to return an error immediately when it detects: - A checksum mismatch - A decompression failure - An invalid page footer - A mismatch between the expected and actual decompressed size Under the normal error path, a corrupted page should therefore not produce a block that continues into `DistinctStreamingAgg`. Based on the information currently available, some possible explanations are: 1. A column or block is partially modified before the read error is returned, and that partial result is later reused. 2. A scanner or iterator continues forwarding a block after receiving a read error. 3. A corrupted offset, dictionary, or page metadata value reaches a path that is not fully covered by checksum validation. 4. The crash is caused by an independent invalid `ColumnString` state or serialized-key buffer problem, while the corruption messages happen concurrently. 5. A `ColumnString` offset overflow or an incorrect serialized-key size calculation causes the final invalid `memcpy`. These are only hypotheses. The current abbreviated stack trace is not sufficient to determine which one is correct, so I do not want to propose a speculative guard around `memcpy` without locating the first invalid state. Could you please provide the following information if it is available? ## 1. Exact Doris build information Please provide the exact commit hash or build information for the crashed BE, rather than only `4.1.0`. For example: ```sql SELECT VERSION(); SHOW FRONTENDS; SHOW BACKENDS; ``` If the BE binary was built from source, the Git commit used for that build would be especially helpful. ## 2. Full GDB backtrace For at least one of the core dumps, could you decompress it and collect: ```gdb set pagination off set print pretty on thread apply all bt full ``` It would be particularly helpful to inspect the frame corresponding to: ```text doris::vectorized::ColumnStr<unsigned int>::serialize_impl ``` In that frame, if debug symbols are available, please also collect: ```gdb info args info locals p row p this p offsets.size() p chars.size() p offsets[row] p offsets[row - 1] ``` If some expressions use different field names or cannot be evaluated due to optimized code, please provide the resulting GDB messages as well. For the caller that owns the serialized key, the following values would also be useful if accessible: ```gdb p keys[row] p keys[row].data p keys[row].size ``` Please redact any actual user data contained in strings or memory buffers. Pointer values, sizes, offsets, symbols, and stack frames are sufficient. ## 3. Complete BE log around one crash Could you provide the complete log interval covering approximately 1–2 minutes before and after one crash? It would help to preserve: - Thread IDs - Query ID - Fragment instance ID - Tablet ID - Rowset ID - Segment path - The first checksum or decompression error - Scanner or iterator errors - Query cancellation messages - Compaction task messages - The final crash stack This will help determine whether the corruption error and `DistinctStreamingAgg` crash occurred in the same thread and query, or whether they were concurrent but unrelated events. ## 4. Query and execution plan If the crash was triggered by a query, please provide: - The SQL statement, with table and column names anonymized if necessary - `EXPLAIN` output - Table schema - Key model (`DUPLICATE KEY`, `UNIQUE KEY`, or `AGGREGATE KEY`) - The data types of the `DISTINCT` or grouping columns - Whether the query contains `DISTINCT`, `GROUP BY`, or distinct aggregate functions - Any relevant session variables The values stored in the table are not required; the schema and data types are the important parts. ## 5. Relationship between the five core dumps Could you clarify whether all five crashes have: - The same stack trace - The same query - The same tablet or segment - The same BE node - The same triggering operation In particular, it would be useful to know whether any core dump was definitely produced by cumulative compaction rather than query execution. The shown `DistinctStreamingAgg` path looks like a query execution path, so query-triggered and compaction-triggered failures may need to be investigated separately. ## 6. Corrupted segment sample and metadata If your data handling policy permits it, the most useful reproducer would be one affected segment together with its metadata and schema. Possible materials include: - One corrupted `.dat` segment file - The corresponding tablet metadata - Rowset metadata - Tablet schema - Segment ID and rowset ID - The file size and checksum - A healthy replica of the same segment for comparison, if available The actual file does not need to be attached publicly. If it contains sensitive data, please do not upload it directly to the public issue. We can first determine whether a smaller sanitized reproduction can be created. ## 7. Relevant BE configuration Please provide the values, if applicable, for: - File cache settings - Storage page cache settings - Checksum verification settings - `enable_query_memory_overcommit` - `mem_limit` - The query memory limit - Whether this is a local-storage or cloud-mode deployment Please also mention whether the segment files were ever copied, restored, replaced, or modified outside Doris after they were created. ## Timestamp clarification There may be a small inconsistency in the reported dates: ```text Aborted at 1748134471 ``` Unix timestamp `1748134471` corresponds to **May 25, 2025**, while the issue describes the crash as occurring on **May 25, 2026**. Could you confirm which date is correct? This may be important when matching the running binary to the corresponding source revision. Also, some example timestamps and source line numbers, such as `09:14:32.123456` and `tablet.cpp:1234`, look like anonymized placeholders. That is completely understandable, but please indicate which log lines are exact and which ones were reconstructed or anonymized so they are not accidentally used for source-version matching. ## Proposed investigation approach Once enough information is available, I plan to: 1. Reproduce checksum and ZSTD failures using an intentionally corrupted segment. 2. Verify whether the destination `Block` or `ColumnString` is partially modified before an error is returned. 3. Trace whether any error path still forwards that block to downstream operators. 4. Run the reproducer under ASAN to find the earliest invalid access or broken invariant. 5. Fix the problem at the layer that creates or propagates the invalid state. 6. Add a BE test proving that a corrupted segment returns a safe corruption error without crashing. 7. Verify the query and cumulative-compaction paths separately if both are affected. At this point, I would treat this primarily as a BE correctness and robustness issue. Regardless of what originally caused the historical file corruption, Doris should not continue with an invalid block or terminate with `SIGSEGV` while handling it. Thank you again for offering to provide additional diagnostic material. The full backtrace, exact build commit, complete log window, and one corrupted segment sample would be the most useful next items. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
