[
https://issues.apache.org/jira/browse/YUNIKORN-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833467#comment-17833467
]
Peter Bacsko edited comment on YUNIKORN-2521 at 4/3/24 9:36 AM:
----------------------------------------------------------------
I looked at the goroutine dump and it's weird to me. Probably I'm missing
something... for example, I don't know if this is important or not, these are
the memory addresses that are passed to an internal function:
{noformat}
sync.runtime_SemacquireRWMutexR(0xffffffffffffffff?, 0x7?, 0xc00a2fa8a0?)
sync.runtime_SemacquireRWMutexR(0x20?, 0xc0?, 0x24?)
sync.runtime_SemacquireRWMutexR(0xc004146f30?, 0x48?, 0x18?)
sync.runtime_SemacquireRWMutexR(0x6aa100000?, 0x90?, 0xc00a703f40?)
{noformat}
Not sure if what 0xffffffffffffffff is about or is that even normal (on the
other hand, 0x20 is a very low address). Could be just a complier optimization.
I don't want to go down the rabbit hole just yet.
I think we need multiple goroutine dumps (maybe 3-4) taken with regular
intervals to see what is stuck exactly. You might have captured a moment which
is perfectly normal.
[~nyoshida] you mentioned that the state dump never returns. It would be very
useful to retrieve it and then make a goroutine dump while it's blocking. That
can also reveal something.
was (Author: pbacsko):
I looked at the goroutine dump and it's weird to me. Probably I'm missing
something... for example, I don't know if this is important or not, these are
the memory addresses that are passed to an internal function:
{noformat}
sync.runtime_SemacquireRWMutexR(0xffffffffffffffff?, 0x7?, 0xc00a2fa8a0?)
sync.runtime_SemacquireRWMutexR(0x20?, 0xc0?, 0x24?)
sync.runtime_SemacquireRWMutexR(0xc004146f30?, 0x48?, 0x18?)
sync.runtime_SemacquireRWMutexR(0x6aa100000?, 0x90?, 0xc00a703f40?)
{noformat}
Not sure if what 0xffffffffffffffff is about or is that even normal (on the
other hand, 0x20 is a very low address). Could be just a complier optimization.
I don't want to go down the rabbit hole just yet.
I think we need multiple gorotuine dumps (maybe 3-4) taken with regular
intervals to see what is stuck exactly. You might have captured a moment which
is perfectly normal.
[~nyoshida] you mentioned that the state dump never returns. It would be very
useful to retrieve it and then make a goroutine dump while it's blocking. That
can also reveal something.
> Scheduler deadlock
> ------------------
>
> Key: YUNIKORN-2521
> URL: https://issues.apache.org/jira/browse/YUNIKORN-2521
> Project: Apache YuniKorn
> Issue Type: Bug
> Affects Versions: 1.5.0
> Environment: Yunikorn: 1.5
> AWS EKS: v1.28.6-eks-508b6b3
> Reporter: Noah Yoshida
> Priority: Major
> Attachments: goroutine-dump.txt
>
>
> Discussion on Yunikorn slack:
> [https://yunikornworkspace.slack.com/archives/CLNUW68MU/p1711048995187179]
> Occasionally, Yunikorn will deadlock and prevent any new pods from starting.
> All pods stay in Pending. There are no error logs inside of the Yunikorn
> scheduler indicating any issue.
> Additionally, the pods all have the correct annotations / labels from the
> admission service, so they are at least getting put into k8s correctly.
> The issue was seen intermittently on Yunikorn version 1.5 in EKS, using
> version `v1.28.6-eks-508b6b3`.
> At least for me, we run about 25-50 nodes and 200-400 pods. Pods and nodes
> are added and removed pretty frequently as we do ML workloads.
> Attached is the goroutine dump. We were not able to get a statedump as the
> endpoint kept timing out.
> You can fix it by restarting the Yunikorn scheduler pod. Sometimes you also
> have to delete any "Pending" pods that got stuck while the scheduler was
> deadlocked as well, for them to get picked up by the new scheduler pod.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
