It looks like I ran into this again. A job has now been pending for 20
minutes. I tried another job with 3 CPUs earlier today and that worked
fine.
When I ran `kubectl describe pods` I got a lot of info I don't understand.
There is however a warning:
> Warning FailedScheduling 3m58s (x62 over 14m) default-scheduler 0/81
> nodes are available: 11 node(s) didn't match Pod's node affinity/selector,
> 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane:
> }, 3 node(s) had untolerated taint {toolforge.org/gateway: true}, 64
> Insufficient cpu. preemption: 0/81 nodes are available: 17 Preemption is
> not helpful for scheduling, 64 No preemption victims found for incoming pod.
I attached the full output. The first job is the one not wanting to start.
*Sebastian Berlin*
Utvecklare/*Developer*
Wikimedia Sverige (WMSE)
E-post/*E-Mail*: [email protected]
Telefon/*Phone*: (+46) 0707 - 92 03 84
On Mon, 23 Mar 2026 at 15:27, David Caro <[email protected]> wrote:
>
> The workers currently have 8 CPU each, and from what I see there's more
> than
> enough cpu free to run a 2 cpu job (just did).
>
> It is true though that the more resources you request, the harder it will
> be
> to allocate (it might even fail to allocate), but I'm not seeing that the
> cluster was out of capacity at that moment.
>
> Next time it happens, you can also try running `kubectl describe pods` to
> show
> the events and some more details on why the job was stuck.
>
> We are working on a patch to add also more information to the state of the
> jobs
> that will help debug quota issues and similar things too, though not yet
> there.
>
> On 03/23 14:34, Sebastian Berlin via Cloud wrote:
> > Thanks. The job started when I lowered to 2 CPUs.
> >
> > This will probably have some impact on the runtime, which is a factor.
> I'd
> > still like to know if it's as simple as the extra CPU preventing it from
> > starting altogether. If so, does that mean the limit is set too high to
> in
> > practice?
> >
> > *Sebastian Berlin*
> > Utvecklare/*Developer*
> > Wikimedia Sverige (WMSE)
> >
> > E-post/*E-Mail*: [email protected]
> > Telefon/*Phone*: (+46) 0707 - 92 03 84
> >
> >
> > On Mon, 23 Mar 2026 at 12:44, Nicolas Vervelle <[email protected]>
> wrote:
> >
> > > Hi Sebastian,
> > >
> > > I also had this kind of problem, it was with jobs requesting a lot of
> cpu
> > > resources, 3 like you (which is the default maximum). When I reduced
> the
> > > CPU request, it started a lot faster.
> > >
> > > So I guess the cluster is too busy and never find a node with enough
> CPU
> > > available.
> > >
> > > Nicolas
> > >
> > > Le lun. 23 mars 2026 à 12:15, Sebastian Berlin via Cloud <
> > > [email protected]> a écrit :
> > >
> > >> A Toolforge job I'm running is stuck. I started it last Friday after
> > >> having stopped once before for now apparent reason. When I checked
> today I
> > >> got:
> > >>
> > >>> tools.commonsdb-registry@tools-bastion-15:~$ toolforge jobs show
> > >>> make-declarations-wlm-2013
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Job name: | make-declarations-wlm-2013
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Command: | src/make_declaration.py -t mem:1.0Gi -t cpu:3.0 -t
> > >>> version:0.1.11 --verbose batch:category-26152515 |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Job type: | one-off
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Image: |
> tool-commonsdb-registry/tool-commonsdb-registry:latest
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Port: | none
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | File log: | yes
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Output log: |
> > >>> /data/project/commonsdb-registry/make-declarations-wlm-2013.out
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Error log: |
> > >>> /data/project/commonsdb-registry/make-declarations-wlm-2013.err
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Emails: | all
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Resources: | mem: 1.0Gi, cpu: 3.0
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Replicas: |
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Mounts: | all
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Retry: | no
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Timeout: | no
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Health check: | none
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Status: | Running for 3d23m
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>> | Hints: | Run not attempted yet. Pod in 'Pending' phase.
> > >>> |
> > >>>
> > >>>
> +---------------+-----------------------------------------------------------------------------------------------------+
> > >>
> > >>
> > >> As far as I can tell that means it hasn't actually started. It's still
> > >> waiting for something, but it's unclear what. I restarted it a couple
> of
> > >> times today so far, but it never gets past that stage.
> > >>
> > >> When I've run similar jobs in the past they sometimes take a while to
> > >> start. Eventually they have though, so I though maybe they were
> waiting for
> > >> resources or something.
> > >>
> > >> Can anyone help me get this started or explain if I've misunderstood
> > >> something?
> > >>
> > >> *Sebastian Berlin*
> > >> Utvecklare/*Developer*
> > >> Wikimedia Sverige (WMSE)
> > >>
> > >> E-post/*E-Mail*: [email protected]
> > >> Telefon/*Phone*: (+46) 0707 - 92 03 84
> > >> _______________________________________________
> > >> Cloud mailing list -- [email protected]
> > >> List information:
> > >>
> https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
> > >>
> > >
>
> > _______________________________________________
> > Cloud mailing list -- [email protected]
> > List information:
> https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
>
>
> --
> David Caro
> SRE - Cloud Services
> Wikimedia Foundation <https://wikimediafoundation.org/>
> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
>
> "Imagine a world in which every single human being can freely share in the
> sum of all knowledge. That's our commitment."
>
Name: make-declarations-valued-images-prepare-cmwct
Namespace: tool-commonsdb-registry
Priority: 0
Service Account: default
Node: <none>
Labels: app.kubernetes.io/component=jobs
app.kubernetes.io/created-by=commonsdb-registry
app.kubernetes.io/managed-by=toolforge-jobs-framework
app.kubernetes.io/name=make-declarations-valued-images-prepare
app.kubernetes.io/version=2
batch.kubernetes.io/controller-uid=b88af75a-c2ea-4bd1-b059-0a56d25cbeee
batch.kubernetes.io/job-name=make-declarations-valued-images-prepare
controller-uid=b88af75a-c2ea-4bd1-b059-0a56d25cbeee
job-name=make-declarations-valued-images-prepare
jobs.toolforge.org/emails=all
jobs.toolforge.org/filelog=yes
toolforge=tool
toolforge.org/mount-storage=all
Annotations: <none>
Status: Pending
SeccompProfile: RuntimeDefault
IP:
IPs: <none>
Controlled By: Job/make-declarations-valued-images-prepare
Containers:
job:
Image:
tools-harbor.wmcloud.org/tool-commonsdb-registry/tool-commonsdb-registry:latest
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
--
exec
1>>/data/project/commonsdb-registry/make-declarations-valued-images-prepare.out;exec
2>>/data/project/commonsdb-registry/make-declarations-valued-images-prepare.err;launcher
env HOME=/workspace src/make_declaration.py -t mem:2.0Gi -t cpu:3.0 -t
version:0.1.14 --verbose --prepare "Category:Valued images sorted by promotion
date"
Limits:
cpu: 3
memory: 2Gi
Requests:
cpu: 3
memory: 1073741824
Environment:
NO_HOME: a buildservice pod does not need a home env
API_ENDPOINT: <set to the key 'API_ENDPOINT' in secret
'toolforge.envvar.v1.api-endpoint'> Optional: false
API_KEY: <set to the key 'API_KEY' in secret
'toolforge.envvar.v1.api-key'> Optional: false
DECLARATION_JOURNAL_URL: <set to the key 'DECLARATION_JOURNAL_URL' in
secret 'toolforge.envvar.v1.declaration-journal-url'> Optional: false
MEMBER_CREDENTIALS_FILE: <set to the key 'MEMBER_CREDENTIALS_FILE' in
secret 'toolforge.envvar.v1.member-credentials-file'> Optional: false
PRIVATE_KEY_FILE: <set to the key 'PRIVATE_KEY_FILE' in secret
'toolforge.envvar.v1.private-key-file'> Optional: false
PUBLIC_KEY_FILE: <set to the key 'PUBLIC_KEY_FILE' in secret
'toolforge.envvar.v1.public-key-file'> Optional: false
TOOL_REPLICA_PASSWORD: <set to the key 'TOOL_REPLICA_PASSWORD' in
secret 'toolforge.envvar.v1.tool-replica-password'> Optional: false
TOOL_REPLICA_USER: <set to the key 'TOOL_REPLICA_USER' in secret
'toolforge.envvar.v1.tool-replica-user'> Optional: false
TOOL_TOOLSDB_PASSWORD: <set to the key 'TOOL_TOOLSDB_PASSWORD' in
secret 'toolforge.envvar.v1.tool-toolsdb-password'> Optional: false
TOOL_TOOLSDB_USER: <set to the key 'TOOL_TOOLSDB_USER' in secret
'toolforge.envvar.v1.tool-toolsdb-user'> Optional: false
TSA_SKIP_VERIFY: <set to the key 'TSA_SKIP_VERIFY' in secret
'toolforge.envvar.v1.tsa-skip-verify'> Optional: false
TSA_URL: <set to the key 'TSA_URL' in secret
'toolforge.envvar.v1.tsa-url'> Optional: false
TOOL_TOOLFORGE_API_URL:
https://api.svc.tools.eqiad1.wikimedia.cloud:30003
TOOL_REDIS_URI:
redis://redis.svc.tools.eqiad1.wikimedia.cloud:6379
TOOL_ELASTICSEARCH_URL:
http://elasticsearch.svc.tools.eqiad1.wikimedia.cloud:80
TOOL_DATA_DIR: /data/project/commonsdb-registry
Mounts:
/data/project from home (rw)
/data/scratch from scratch (rw)
/etc/ldap.conf from etcldap-conf (ro)
/etc/ldap.yaml from etcldap-yaml (ro)
/etc/novaobserver.yaml from etcnovaobserver-yaml (ro)
/etc/openstack/clouds.yaml from etcopenstack-clouds (ro)
/etc/wmcs-project from wmcs-project (ro)
/mnt/nfs/dumps-clouddumps1001.wikimedia.org from dumpsrc-clouddumps1001
(ro)
/mnt/nfs/dumps-clouddumps1002.wikimedia.org from dumpsrc-clouddumps1002
(ro)
/public/dumps from dumps (ro)
/var/lib/sss/pipes from sssd-pipes (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-65fpj
(ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-65fpj:
Type: Projected (a volume that contains injected data
from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
dumps:
Type: HostPath (bare host directory volume)
Path: /public/dumps
HostPathType: Directory
dumpsrc-clouddumps1001:
Type: HostPath (bare host directory volume)
Path: /mnt/nfs/dumps-clouddumps1001.wikimedia.org
HostPathType: Directory
dumpsrc-clouddumps1002:
Type: HostPath (bare host directory volume)
Path: /mnt/nfs/dumps-clouddumps1002.wikimedia.org
HostPathType: Directory
home:
Type: HostPath (bare host directory volume)
Path: /data/project
HostPathType: Directory
wmcs-project:
Type: HostPath (bare host directory volume)
Path: /etc/wmcs-project
HostPathType: File
scratch:
Type: HostPath (bare host directory volume)
Path: /data/scratch
HostPathType: Directory
etcldap-conf:
Type: HostPath (bare host directory volume)
Path: /etc/ldap.conf
HostPathType: File
etcldap-yaml:
Type: HostPath (bare host directory volume)
Path: /etc/ldap.yaml
HostPathType: File
etcnovaobserver-yaml:
Type: HostPath (bare host directory volume)
Path: /etc/novaobserver.yaml
HostPathType: File
etcopenstack-clouds:
Type: HostPath (bare host directory volume)
Path: /etc/openstack/clouds.yaml
HostPathType: File
sssd-pipes:
Type: HostPath (bare host directory volume)
Path: /var/lib/sss/pipes
HostPathType: Directory
QoS Class: Burstable
Node-Selectors: kubernetes.wmcloud.org/nfs-mounted=true
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists
for 300s
node.kubernetes.io/unreachable:NoExecute
op=Exists for 300s
Topology Spread Constraints: kubernetes.io/hostname:ScheduleAnyway when max
skew 1 is exceeded for selector
app.kubernetes.io/created-by=commonsdb-registry,app.kubernetes.io/managed-by=toolforge-jobs-framework,app.kubernetes.io/name=make-declarations-valued-images-prepare,toolforge=tool
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m58s (x62 over 14m) default-scheduler 0/81
nodes are available: 11 node(s) didn't match Pod's node affinity/selector, 3
node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3
node(s) had untolerated taint {toolforge.org/gateway: true}, 64 Insufficient
cpu. preemption: 0/81 nodes are available: 17 Preemption is not helpful for
scheduling, 64 No preemption victims found for incoming pod.
Name: make-declarations-wlm-2015-9p9q4
Namespace: tool-commonsdb-registry
Priority: 0
Service Account: default
Node: tools-k8s-worker-nfs-1/172.16.0.87
Start Time: Mon, 18 May 2026 06:12:14 +0000
Labels: app.kubernetes.io/component=jobs
app.kubernetes.io/created-by=commonsdb-registry
app.kubernetes.io/managed-by=toolforge-jobs-framework
app.kubernetes.io/name=make-declarations-wlm-2015
app.kubernetes.io/version=2
batch.kubernetes.io/controller-uid=259c7b1f-4a05-45b9-b047-c85ef1c05e1e
batch.kubernetes.io/job-name=make-declarations-wlm-2015
controller-uid=259c7b1f-4a05-45b9-b047-c85ef1c05e1e
job-name=make-declarations-wlm-2015
jobs.toolforge.org/emails=all
jobs.toolforge.org/filelog=yes
toolforge=tool
toolforge.org/mount-storage=all
Annotations: cni.projectcalico.org/containerID:
5f5b74f0ca06738d2fa49e8070dc60223e85cd8437a529d7bf4ec1fe624b1bf7
cni.projectcalico.org/podIP: 192.168.20.38/32
cni.projectcalico.org/podIPs: 192.168.20.38/32
Status: Running
SeccompProfile: RuntimeDefault
IP: 192.168.20.38
IPs:
IP: 192.168.20.38
Controlled By: Job/make-declarations-wlm-2015
Containers:
job:
Container ID:
containerd://b2e910285e7fc3afc1faa7dffe3f528fc07196a88f22db4fe4f25e99cd1c2e27
Image:
tools-harbor.wmcloud.org/tool-commonsdb-registry/tool-commonsdb-registry:latest
Image ID:
tools-harbor.wmcloud.org/tool-commonsdb-registry/tool-commonsdb-registry@sha256:888dfe23775d28a9fe00ef0cf59a6336ef7ae0fc0ffe6367ec2de3314a8db7e8
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
--
exec
1>>/data/project/commonsdb-registry/make-declarations-wlm-2015.out;exec
2>>/data/project/commonsdb-registry/make-declarations-wlm-2015.err;launcher env
HOME=/workspace src/make_declaration.py -t mem:2.0Gi -t cpu:2.0 -t
version:0.1.13 --verbose batch:category-42097914
State: Running
Started: Mon, 18 May 2026 06:12:25 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 2
memory: 2Gi
Requests:
cpu: 2
memory: 1073741824
Environment:
NO_HOME: a buildservice pod does not need a home env
API_ENDPOINT: <set to the key 'API_ENDPOINT' in secret
'toolforge.envvar.v1.api-endpoint'> Optional: false
API_KEY: <set to the key 'API_KEY' in secret
'toolforge.envvar.v1.api-key'> Optional: false
DECLARATION_JOURNAL_URL: <set to the key 'DECLARATION_JOURNAL_URL' in
secret 'toolforge.envvar.v1.declaration-journal-url'> Optional: false
MEMBER_CREDENTIALS_FILE: <set to the key 'MEMBER_CREDENTIALS_FILE' in
secret 'toolforge.envvar.v1.member-credentials-file'> Optional: false
PRIVATE_KEY_FILE: <set to the key 'PRIVATE_KEY_FILE' in secret
'toolforge.envvar.v1.private-key-file'> Optional: false
PUBLIC_KEY_FILE: <set to the key 'PUBLIC_KEY_FILE' in secret
'toolforge.envvar.v1.public-key-file'> Optional: false
TOOL_REPLICA_PASSWORD: <set to the key 'TOOL_REPLICA_PASSWORD' in
secret 'toolforge.envvar.v1.tool-replica-password'> Optional: false
TOOL_REPLICA_USER: <set to the key 'TOOL_REPLICA_USER' in secret
'toolforge.envvar.v1.tool-replica-user'> Optional: false
TOOL_TOOLSDB_PASSWORD: <set to the key 'TOOL_TOOLSDB_PASSWORD' in
secret 'toolforge.envvar.v1.tool-toolsdb-password'> Optional: false
TOOL_TOOLSDB_USER: <set to the key 'TOOL_TOOLSDB_USER' in secret
'toolforge.envvar.v1.tool-toolsdb-user'> Optional: false
TSA_SKIP_VERIFY: <set to the key 'TSA_SKIP_VERIFY' in secret
'toolforge.envvar.v1.tsa-skip-verify'> Optional: false
TSA_URL: <set to the key 'TSA_URL' in secret
'toolforge.envvar.v1.tsa-url'> Optional: false
TOOL_TOOLFORGE_API_URL:
https://api.svc.tools.eqiad1.wikimedia.cloud:30003
TOOL_REDIS_URI:
redis://redis.svc.tools.eqiad1.wikimedia.cloud:6379
TOOL_ELASTICSEARCH_URL:
http://elasticsearch.svc.tools.eqiad1.wikimedia.cloud:80
TOOL_DATA_DIR: /data/project/commonsdb-registry
Mounts:
/data/project from home (rw)
/data/scratch from scratch (rw)
/etc/ldap.conf from etcldap-conf (ro)
/etc/ldap.yaml from etcldap-yaml (ro)
/etc/novaobserver.yaml from etcnovaobserver-yaml (ro)
/etc/openstack/clouds.yaml from etcopenstack-clouds (ro)
/etc/wmcs-project from wmcs-project (ro)
/mnt/nfs/dumps-clouddumps1001.wikimedia.org from dumpsrc-clouddumps1001
(ro)
/mnt/nfs/dumps-clouddumps1002.wikimedia.org from dumpsrc-clouddumps1002
(ro)
/public/dumps from dumps (ro)
/var/lib/sss/pipes from sssd-pipes (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lmvb6
(ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-lmvb6:
Type: Projected (a volume that contains injected data
from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
dumps:
Type: HostPath (bare host directory volume)
Path: /public/dumps
HostPathType: Directory
dumpsrc-clouddumps1001:
Type: HostPath (bare host directory volume)
Path: /mnt/nfs/dumps-clouddumps1001.wikimedia.org
HostPathType: Directory
dumpsrc-clouddumps1002:
Type: HostPath (bare host directory volume)
Path: /mnt/nfs/dumps-clouddumps1002.wikimedia.org
HostPathType: Directory
home:
Type: HostPath (bare host directory volume)
Path: /data/project
HostPathType: Directory
wmcs-project:
Type: HostPath (bare host directory volume)
Path: /etc/wmcs-project
HostPathType: File
scratch:
Type: HostPath (bare host directory volume)
Path: /data/scratch
HostPathType: Directory
etcldap-conf:
Type: HostPath (bare host directory volume)
Path: /etc/ldap.conf
HostPathType: File
etcldap-yaml:
Type: HostPath (bare host directory volume)
Path: /etc/ldap.yaml
HostPathType: File
etcnovaobserver-yaml:
Type: HostPath (bare host directory volume)
Path: /etc/novaobserver.yaml
HostPathType: File
etcopenstack-clouds:
Type: HostPath (bare host directory volume)
Path: /etc/openstack/clouds.yaml
HostPathType: File
sssd-pipes:
Type: HostPath (bare host directory volume)
Path: /var/lib/sss/pipes
HostPathType: Directory
QoS Class: Burstable
Node-Selectors: kubernetes.wmcloud.org/nfs-mounted=true
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists
for 300s
node.kubernetes.io/unreachable:NoExecute
op=Exists for 300s
Topology Spread Constraints: kubernetes.io/hostname:ScheduleAnyway when max
skew 1 is exceeded for selector
app.kubernetes.io/created-by=commonsdb-registry,app.kubernetes.io/managed-by=toolforge-jobs-framework,app.kubernetes.io/name=make-declarations-wlm-2015,toolforge=tool
Events: <none>
_______________________________________________
Cloud mailing list -- [email protected]
List information:
https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
