[
https://issues.apache.org/jira/browse/YUNIKORN-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881159#comment-17881159
]
Wilfred Spiegelenburg commented on YUNIKORN-2860:
-------------------------------------------------
This is partially a K8s issue and partially ours. There is no guarantee that
the pods we create for gang scheduling placeholders come back in a pre-defined
order. Even if and when we process thing serially there is no guarantee that we
get them back in that order.
I have seen before that we create 10 placeholders and the first pod that K8s
returns to us is the last pod we created. That becomes worse when multiple
origins (applications) are involved. We pass on from the k8shim to the core
serially.
I think we can improve this if we track the applications that have requested
gangs in a queue in order of creation and only service the ones that fit in the
queue or just the first one out of that list until all placeholders are
allocated. That could be investigated for a next release. Not sure if it can
work as something like this might have side effects we do not want or cause
bigger issues. For instance if the placeholder pods continually fail placements
due to predicates or something we need to "escape" this.
> submit gang applications Simultaneously may cause unexpected pending apps
> ---------------------------------------------------------------------------
>
> Key: YUNIKORN-2860
> URL: https://issues.apache.org/jira/browse/YUNIKORN-2860
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Affects Versions: 1.3.0, 1.4.0, 1.5.0, 1.5.1, 1.5.2
> Reporter: shawn
> Priority: Major
> Attachments: image-2024-09-11-15-41-12-142.png,
> image-2024-09-11-15-42-07-739.png, state-dump.txt, yunikorn-scheduler.txt
>
>
>
> I Simultaneously submit 4 gang apps to yunikorn，sometimes 4 apps get
> pending, while two pgs get running, that's not expected.
> It can be reproduced as follows：
> queues
> 1.kubectl create configmap yunikorn-configs --from-file=queues.yaml -n
> yunikorn
> * queues.yaml
> {code:java}
> partitions:
> - name: default
> queues:
> - name: root
> queues:
> - name: my-dev
> submitacl: "*"
> resources:
> guaranteed: { memory: 1G, vcore: 1 }
> max: { memory: 2G, vcore: 2 }{code}
> 2.Simultaneously submit gang-scheduling-job-example1-4.yaml, while
> gang-scheduling-job-example1-4.yaml only differ in name and applicationId
> {code:java}
> apiVersion: batch/v1
> kind: Job
> metadata:
> name: gang-scheduling-job-example1
> spec:
> completions: 2
> parallelism: 2
> template:
> metadata:
> labels:
> app: sleep
> applicationId: "gang-scheduling-job-example1"
> queue: root.my-dev
> annotations:
> yunikorn.apache.org/task-group-name: task-group-example-0
> yunikorn.apache.org/task-groups: |-
> [{
> "name": "task-group-example-0",
> "minMember": 2,
> "minResource": {
> "cpu": "1",
> "memory": "1G"
> },
> "nodeSelector": {},
> "tolerations": [],
> "affinity": {}
> }]
> spec:
> schedulerName: yunikorn
> restartPolicy: Never
> containers:
> - name: sleep30
> image: "nginx:latest"
> command: ["sleep", "999999999"]
> resources:
> requests:
> cpu: "1"
> memory: "1G" {code}
> finally，kubectl get pods -n default gets unexpected result(not always
> reproducible)
> !image-2024-09-11-15-41-12-142.png!
>
> app state as follows
> !image-2024-09-11-15-42-07-739.png|width=754,height=280!
> full state dump as state-dump.txt, yunikorn scheduler logs are in
> yunikorn-scheduler.txt
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
