Public bug reported:

Note that I'm not entirely sure where to report this. To my mind
upgrading podman makes most sense, but I don't know.

On AWS at least, Ubuntu upgraded kernel to 7.0. Since then, the kernel loads 
`nvidia_drm` module:
```
$ uname -a
Linux ip-172-22-29-16 7.0.0-1011-aws #11~24.04.1-Ubuntu SMP PREEMPT Mon Aug 10 
15:20:57 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
$ lsmod | grep nvidia_drm
nvidia_drm            159744  0
nvidia_modeset       1544192  1 nvidia_drm
drm_ttm_helper         20480  1 nvidia_drm
```

which was not present with the previous 6.17 kernel:
```
$ uname -a
Linux ip-172-22-29-13 6.17.0-1019-aws #19~24.04.1-Ubuntu SMP Tue Jun 23 
18:53:06 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
$ lsmod | grep nvidia_drm
```


That creates new devices:
```
$ ls -al /dev/dri/
total 0
drwxr-xr-x  3 root root        120 Sep  3 13:07 .
drwxr-xr-x 17 root root       3600 Sep  3 08:04 ..
drwxr-xr-x  2 root root        100 Sep  3 13:07 by-path
crw-rw----  1 root video  226,   0 Sep  3 08:04 card0
crw-rw----  1 root video  226,   1 Sep  3 13:07 card1
crw-rw----  1 root render 226, 128 Sep  3 13:07 renderD128
```

vs the old kernel:
```
$ ls -al /dev/dri/
total 0
drwxr-xr-x  3 root root      80 Sep  3 09:26 .
drwxr-xr-x 17 root root    3520 Sep  3 09:26 ..
drwxr-xr-x  2 root root      60 Sep  3 09:26 by-path
crw-rw----  1 root video 226, 0 Sep  3 09:26 card0
```

and that causes nvidia-ctk to generate a different version of the spec, one 
that podman 4.9.3 doesn't understand:
```
$ nvidia-ctk --quiet cdi generate | head -2
---
cdiVersion: 0.7.0
```

vs the old kernel (or new kernel with `rmmod nvidia_drm`):
```
$ nvidia-ctk --quiet cdi generate | head -2
---
cdiVersion: 0.5.0
```

and that makes podman unable to work with the GPU:
```
$ sudo podman --log-level=debug run --rm --device nvidia.com/gpu=0 
docker.io/library/busybox true 2>&1 | grep 'CDI'
time="2026-09-03T13:27:30Z" level=debug msg="Identified CDI device 
nvidia.com/gpu=0"
time="2026-09-03T13:27:30Z" level=debug msg="The following error was triggered 
when refreshing the CDI registry: failed to load CDI Spec failed to parse CDI 
Spec \"/var/run/cdi/nvidia.yaml\": failed to unmarshal CDI Spec: error 
unmarshaling JSON: while decoding JSON: json: unknown field \"additionalGids\""
Error: setting up CDI devices: unresolvable CDI devices nvidia.com/gpu=0
```

because it doesn't understand the 0.7.0 spec.

I won't pretend to fully understand how all this works, and I do realize
upgrading podman will likely raise different demons. It did fall behind
quite a bit though.

** Affects: libpod (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2166357

Title:
  podman 4.9.3 in 24.04 can't parse Nvidia's CDI 0.7 on new linux-aws
  kernel v7

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2166357/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to