my-ship-it commented on issue #1708:
URL: https://github.com/apache/cloudberry/issues/1708#issuecomment-4343029472
Hey @adnanhamdussalam, thanks for the report — and for including the stack,
that's already a useful starting point.
I took a look, but I don't think we can call this a bug yet — at least not
from what's here. The stack you pasted is actually from the coordinator side
(`mppExecutorFinishup` → `cdbdispatchresult.c:510`). That's the path the QD
goes down when it tries to collect results from a segment and finds the
connection has already dropped. So the coordinator itself is fine — it's
basically just telling us "one of my segments died on me." The real crash
happened on a segment, and that part isn't in what you shared.
The upside: the stack does rule out a lot. It's not a parser/planner issue,
not a cancellation or deadlock, not a resource-group admission thing, and not a
QD bug. What's left is roughly:
- a segment getting OOM-killed (honestly my first guess given the 90-day
window plus those `array_agg(distinct concat(...) order by ...)` aggregates),
- a segfault inside a QE — often a custom UDF, and `concat_my` jumps out at
me here,
- or some PANIC path that fired after interconnect teardown (less likely).
To tell which one it is, would you mind grabbing a few extra things when you
get a chance?
The two that would help the most:
- **Segment logs around the time it failed.** Something like \`grep -E
'PANIC|SIGSEGV|signal|terminating' pg_log/cbdb-*.log\` on the segment hosts —
the PANIC/signal line and the stack right after it is exactly what's missing.
- **\`dmesg -T | grep -iE 'killed process|out of memory'\`** on the segment
hosts. If the OOM-killer was involved, this is where it'll show up.
And if it's easy:
- The Cloudberry version (\`SELECT version();\`).
- The definition of \`concat_my\` (\`\df+ concat_my\`). If it's a custom C
function, could you try swapping it for \`concat_ws(',', h1.agent_skilling,
h2.agent_skilling)\` once and see if the crash still happens? That alone would
tell us a lot.
- Does it fail every time, or only on certain data sizes? Does a smaller
window (say 7 days instead of 90) still reproduce it?
- Values for \`statement_mem\`, \`max_statement_mem\`,
\`gp_workfile_limit_per_segment\`, and which resource group this is running
under.
A coredump from a segment would be ideal if you happen to have
\`kernel.core_pattern\` set up, but no worries if not — even just the segment
log line and \`dmesg\` output should let us classify this pretty quickly.
Thanks again, and looking forward to whatever you can share!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
