YanzhiJin5 commented on issue #63609: URL: https://github.com/apache/doris/issues/63609#issuecomment-5379565592
Hi @ShummGen, First, I apologize for the long gap since your last update, and thank you again for providing such detailed information from the July 27 incident. I have spent some time going back through the issue and tracing the relevant code on the exact production revision you provided (`b10073ad9ca17cd5685c4dd3b3ef650f256376d0`) before asking you for any additional data. A brief update on the investigation so far: * I traced the storage-read error path from `PageIO` through `ColumnReader`, `FileColumnIterator`, `SegmentIterator`, and `OlapScanner`. * On the normal error path I reviewed, a block from a storage read that returns an error is not forwarded downstream to aggregation. Therefore, I do not currently have evidence that the nearby checksum/ZSTD errors directly produced the block involved in the earlier `DistinctStreamingAgg -> ColumnStr::serialize_impl` crash. * For that reason, I am treating the earlier `ColumnStr` crash and the more recent `ColumnDictI32::filter_by_selector -> SegmentIterator -> OlapScanner` crash as separate signatures unless further evidence connects them. * I also investigated a missing dictionary-code bounds check and prototyped a defensive validation change with regression tests. However, after revisiting the evidence, I do **not** think the available production data is sufficient to claim that invalid dictionary codes caused your crash. I would rather confirm or rule that out from the actual crash state than rely on a speculative fix. The good news is that the source investigation has now narrowed the recent `ColumnDictI32::filter_by_selector` crash to a much more specific question: at the failing iteration, were the selector and dictionary code actually within their expected ranges? If the core from the recent July 27 crash and its matching `doris_be` binary are still retained, please use whichever option is easiest for you. ### Option 1: minimal GDB output If opening the core in GDB is convenient, a small amount of output would be enough for the next step. After opening the matching binary and core, the following commands should be sufficient initially: ```gdb set pagination off bt ``` If the backtrace contains a `ColumnDictI32::filter_by_selector` frame, select its frame number as shown by `bt`, for example: ```gdb frame <N> info args info locals ``` There is no need to spend time manually interpreting the values. Please just send the output that GDB is able to provide. Optimized production binaries may report some variables as `<optimized out>`; that is completely fine, and there is no need to work around it. Please also feel free to redact any business data, paths, SQL text, URLs, or other sensitive information before posting the output publicly. ### Option 2: existing core file If running GDB and extracting variables yourself would be more work, I would be equally happy to inspect the existing core and matching `doris_be` binary directly. Since you previously mentioned that the raw core/segment may contain sensitive business data, I do **not** suggest uploading them to the public GitHub issue. If a Doris maintainer can recommend an appropriate private transfer channel, this may actually be the lower-effort option for you: you could provide the existing core and matching binary (and separate debug symbols, if any), and I can do the GDB investigation from there. If neither option is convenient or the recent core is no longer retained, that is also completely fine. I can continue from the source side and try to construct a local reproducer using the information you have already provided. Thank you again for preserving and sharing the diagnostic information. It has been very helpful in narrowing the investigation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
