[Kernel-packages] [Bug 1889637] Re: Raspberry Pi 3B hangs - dev_pm_opp_set_rate: failed to find current OPP, Failed to get throttled, Failed to change plib frequency; mmc timeout waiting for hardware
** Attachment added: "`apport-cli linux` output"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1889637/+attachment/5428751/+files/launchpad.bug
** Changed in: linux (Ubuntu)
Status: Incomplete => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1889637
Title:
Raspberry Pi 3B hangs - dev_pm_opp_set_rate: failed to find current
OPP, Failed to get throttled, Failed to change plib frequency; mmc
timeout waiting for hardware interrupt
Status in linux package in Ubuntu:
Confirmed
Bug description:
### uname -a (64-bit ARM, official image):
`Linux ubuntu 5.4.0-1015-raspi #15-Ubuntu SMP Fri Jul 10 05:34:24 UTC 2020
aarch64 aarch64 aarch64 GNU/Linux`
### LSB release (Ubuntu *Server*, focal):
Description: Ubuntu 20.04.1 LTS
### Interesting packages installed
- zfs-dkms (with initramfs support) @ 0.8.3-1ubuntu12.2
* spl-dkms @ 0.8.3-1ubuntu12.2
- dphys-swapfile
### Hardware model:
Raspberry Pi 3 Model B
- 32 GiB SD card with root partition
* had a swap partition; now unused
* migrated to dphys-swapfile
- Attached 32 GiB USB stick as zpool for storage (not root FS)
- Current PSU reportedly outputs 2.4A supply for the Pi
* Still have occasional undervolt warnings (formally requires 2.5A)
* Lightning indicator not present however
- Connected over wireless networking
## Issue
- When under significant computational load at some point, the machine
appears to freeze.
* I usually log in in a headless manner via ssh, so externally the machine
is frozen and I need to pull the power cable
- Connectig the HDMI monitor the following messsages appear, in various
orders each time:
```terminal
cpu cpu0: dev_pm_opp_set_rate: failed to find current OPP for freq
9,223,372,036,854,775,698 ({illegible on my photograph, presumably -110})
hwmon hwmon1: Failed to get throttled (-110)
raspberrypi-clk firmware clocks: Failed to change plib frequency: -110
mmc0: timeout waiting for hardware interrupt
# mmc0 would be the root partition
### ... typically later on in the output
rcu: INFO: rcu_sched detected stalls on CPU/tasks
rcu: $1-...0: (1 GPs behind) idle=.../1/0x4{more 0s...}02
softirq=66377/66378 {or 26106/26107} fqs={this value varies}
INFO: task kworker/{...} blocked for more than 120 seconds
TAINTED: PWC OE 5.4.0-1015-raspi #15-ubuntu
watchdog: BUG: soft lockup - CPU #3 stuck for 22s!
```
The OPP frequency above looks to me like it may be the cause of the
issue, I have added the commas myself to the output but it would
appear to be a rubbish value;
[this](https://lkml.org/lkml/2020/7/24/683) mailing list archive I
found whilst searching for terms found in the messages appears to back
up my belief that we should be seeing a sensible CPU frequency here,
expressed in integer Hz; the above would be 9.2 EHz assuming Hertz are
the base unit, higher still if it's k/M/GHz etc. My most sensible
guess is this value has been brought up somewhere as garbage, and
understandably the system fails to scale the clock speed, with the
resultant crashes presumably due to this.
Beyond this point, there is no kernel panic, however the machine locks
up externally; does not respond to USB keyboard NumLock and is
invisible on the network, with more and more errors gradually being
output to the console via the HDMI display; the most notable being the
SD card is not responding
Just before encountering this issue I had added a swap aprtition, to
the SD card, as I had none by default and the system seemed to be
hanging when it presumably was sending bad_allocs to userland
processes as it failed to allocate memory. As the SD card was
mentioned, I have tried a variety of power supplies (as I was getting
several undervolt warnings) and eventually removed the swap partition
and used a swapfile with `dphys-swapfile` knowing that the way the Pi
accesses the SD card is somewhat different from a typical machine.
However, neither of these two seems to have resolved the issue, giving
further evidence that the frequency scaling may well be the primary
issue and the rest is simply the carnage that ensues.
## Steps to Reproduce
- Seems to happen sporadically when the machine is under stress, within 5-25
minutes
- Currently I am trying to set up a rootless docker compose file
* Attempting to pull the images eventually leads to the issue
* The images are being downloaded to the zpool on the USB stick and *not*
the SD card
- The system seems to hang initially waiting on the SDcard to respond to an
IRQ
- however I believe that the CPU scaling message seems to be the root cause
- Do not have any of the importat messages in the `syslog`, I need an
external HDMI monitor to get the output on screen from the kernel ring buffer
## Links
- [
[Kernel-packages] [Bug 1889637] Re: Raspberry Pi 3B hangs - dev_pm_opp_set_rate: failed to find current OPP, Failed to get throttled, Failed to change plib frequency; mmc timeout waiting for hardware
** Package changed: ubuntu => linux (Ubuntu)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1889637
Title:
Raspberry Pi 3B hangs - dev_pm_opp_set_rate: failed to find current
OPP, Failed to get throttled, Failed to change plib frequency; mmc
timeout waiting for hardware interrupt
Status in linux package in Ubuntu:
New
Bug description:
### uname -a (64-bit ARM, official image):
`Linux ubuntu 5.4.0-1015-raspi #15-Ubuntu SMP Fri Jul 10 05:34:24 UTC 2020
aarch64 aarch64 aarch64 GNU/Linux`
### LSB release (Ubuntu *Server*, focal):
Description: Ubuntu 20.04.1 LTS
### Interesting packages installed
- zfs-dkms (with initramfs support) @ 0.8.3-1ubuntu12.2
* spl-dkms @ 0.8.3-1ubuntu12.2
- dphys-swapfile
### Hardware model:
Raspberry Pi 3 Model B
- 32 GiB SD card with root partition
* had a swap partition; now unused
* migrated to dphys-swapfile
- Attached 32 GiB USB stick as zpool for storage (not root FS)
- Current PSU reportedly outputs 2.4A supply for the Pi
* Still have occasional undervolt warnings (formally requires 2.5A)
* Lightning indicator not present however
- Connected over wireless networking
## Issue
- When under significant computational load at some point, the machine
appears to freeze.
* I usually log in in a headless manner via ssh, so externally the machine
is frozen and I need to pull the power cable
- Connectig the HDMI monitor the following messsages appear, in various
orders each time:
```terminal
cpu cpu0: dev_pm_opp_set_rate: failed to find current OPP for freq
9,223,372,036,854,775,698 ({illegible on my photograph, presumably -110})
hwmon hwmon1: Failed to get throttled (-110)
raspberrypi-clk firmware clocks: Failed to change plib frequency: -110
mmc0: timeout waiting for hardware interrupt
# mmc0 would be the root partition
### ... typically later on in the output
rcu: INFO: rcu_sched detected stalls on CPU/tasks
rcu: $1-...0: (1 GPs behind) idle=.../1/0x4{more 0s...}02
softirq=66377/66378 {or 26106/26107} fqs={this value varies}
INFO: task kworker/{...} blocked for more than 120 seconds
TAINTED: PWC OE 5.4.0-1015-raspi #15-ubuntu
watchdog: BUG: soft lockup - CPU #3 stuck for 22s!
```
The OPP frequency above looks to me like it may be the cause of the
issue, I have added the commas myself to the output but it would
appear to be a rubbish value;
[this](https://lkml.org/lkml/2020/7/24/683) mailing list archive I
found whilst searching for terms found in the messages appears to back
up my belief that we should be seeing a sensible CPU frequency here,
expressed in integer Hz; the above would be 9.2 EHz assuming Hertz are
the base unit, higher still if it's k/M/GHz etc. My most sensible
guess is this value has been brought up somewhere as garbage, and
understandably the system fails to scale the clock speed, with the
resultant crashes presumably due to this.
Beyond this point, there is no kernel panic, however the machine locks
up externally; does not respond to USB keyboard NumLock and is
invisible on the network, with more and more errors gradually being
output to the console via the HDMI display; the most notable being the
SD card is not responding
Just before encountering this issue I had added a swap aprtition, to
the SD card, as I had none by default and the system seemed to be
hanging when it presumably was sending bad_allocs to userland
processes as it failed to allocate memory. As the SD card was
mentioned, I have tried a variety of power supplies (as I was getting
several undervolt warnings) and eventually removed the swap partition
and used a swapfile with `dphys-swapfile` knowing that the way the Pi
accesses the SD card is somewhat different from a typical machine.
However, neither of these two seems to have resolved the issue, giving
further evidence that the frequency scaling may well be the primary
issue and the rest is simply the carnage that ensues.
## Steps to Reproduce
- Seems to happen sporadically when the machine is under stress, within 5-25
minutes
- Currently I am trying to set up a rootless docker compose file
* Attempting to pull the images eventually leads to the issue
* The images are being downloaded to the zpool on the USB stick and *not*
the SD card
- The system seems to hang initially waiting on the SDcard to respond to an
IRQ
- however I believe that the CPU scaling message seems to be the root cause
- Do not have any of the importat messages in the `syslog`, I need an
external HDMI monitor to get the output on screen from the kernel ring buffer
## Links
- [Related AskUbuntu
question](https://askubuntu.com/questions/1241412/ubuntu-20-04-lts-hangs-with-error-hwmon1-failed-to-get-throttled-110)
- [Potentially related bug - the frequency is
[Kernel-packages] [Bug 1718761] Re: It's not possible to use OverlayFS (mount -t overlay) to stack directories on a ZFS volume
I am arriving here as I would also like to run containers (docker) over
a zpool as the storage backend. In my case, I am running Ubuntu Server
(Focal) on a Pi 3B, I would ideally have root on zfs but with it being a
Pi, I have not done this yet. I have an externally attached USB pool for
real data and have left the OS as it is on the SD card, however, as the
above says, I would like to move more of my systems towards ZFS with
time and use Ubuntu-based distros and containers on a lot of them, where
these can't be run under illumos as VMs.
I have installed docker-rootless on this system, however due to kernel
limitations, it is impossible to mount datasets as non-root on linux
(long-standing issue, I am working on a bug report with details of
options for a workaround and will attach), and using a noverlay over the
top of zfs would be fine for me, as I can snapshot and work with the
higher-level system and back up easily, even if not as fast as native
zfs performance, however, the daemon will not start, and I have fallen
back to the "vfs" driver on my data-root filesystem.
Some googling hasn't indicated what feature(s) would be required in zfs for
overlay to be used on top, docker's documentation doesn't seem to clarify
anything, and searching for the "fstype=1" requirement in xfs, I find from the
manual page for `mkfs.xfs`:
```
ftype=value
This feature allows the inode type to be stored in
the directory structure so that the readdir(3) and
getdents(2) do not need to look up the inode to
determine the inode type.
```
which perhaps is not the issue at hand.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1718761
Title:
It's not possible to use OverlayFS (mount -t overlay) to stack
directories on a ZFS volume
Status in zfs-linux package in Ubuntu:
Confirmed
Bug description:
-- Configuration
# echo -e $(grep VERSION= /etc/os-release)\\nSIGNATURE=\"$(cat
/proc/version_signature)\"
VERSION="16.04.3 LTS (Xenial Xerus)"
SIGNATURE="Ubuntu 4.10.0-35.39~16.04.1-generic 4.10.17"
# dpkg --list | grep zfs
ii libzfs2linux0.6.5.6-0ubuntu18
ii zfs-doc 0.6.5.6-0ubuntu18
ii zfs-initramfs 0.6.5.6-0ubuntu18
ii zfs-zed 0.6.5.6-0ubuntu18
ii zfsutils-linux 0.6.5.6-0ubuntu18
-- Fault: Creating an overlay of multiple directories on a ZFS volume
does not work
# df /var/tmp
Filesystem Type 1K-blocks Used Available Use% Mounted on
tank07/var/tmp zfs 129916288 128 129916160 1% /var/tmp
# mkdir /var/tmp/{lower,middle,upper,workdir,overlay}
# mount -t overlay overlay
-olowerdir=/var/tmp/middle:/var/tmp/lower,upperdir=/var/tmp/upper,workdir=/var/tmp/workdir
/var/tmp/overlay
mount: wrong fs type, bad option, bad superblock on overlay,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
# dmesg|tail -1
[276328.438284] overlayfs: filesystem on '/var/tmp/upper' not supported as
upperdir
-- Control test 1: Creating an overlay of multiple directories on
another filesystem works
# df /tmp
Filesystem Type 1K-blocks Used Available Use% Mounted on
tmpfs tmpfs 1048576 133492915084 13% /tmp
# mkdir /tmp/{lower,middle,upper,workdir,overlay}
# mount -t overlay overlay
-olowerdir=/tmp/middle:/tmp/lower,upperdir=/tmp/upper,workdir=/tmp/workdir
/tmp/overlay
# mount | grep overlay
overlay on /tmp/overlay type overlay
(rw,relatime,lowerdir=/tmp/middle:/tmp/lower,upperdir=/tmp/upper,workdir=/tmp/workdir)
-- Control test 2: Creating an overlay using AuFS works on ZFS volume
and elsewhere
# mount -t aufs -obr=/tmp/lower:/tmp/middle:/tmp/upper:/tmp/workdir none
/tmp/overlay
# mount -t aufs
-obr=/var/tmp/lower:/var/tmp/middle:/var/tmp/upper:/var/tmp/workdir none
/var/tmp/overlay
# mount | grep aufs
none on /var/tmp/overlay type aufs (rw,relatime,si=9ead1ecae778b250)
none on /tmp/overlay type aufs (rw,relatime,si=9ead1ec9257d1250)
-- Remark
While the use of AuFS can be used as a workaround in the above scenario, AuFS
in turn will not work with [fuse.]glusterfs mounts (this has been documented
elsewhere). Given that OverlayFS is part of the (upstream) kernel and Ubuntu
now officially supports ZFS, the above should be fixed.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Sep 18 16:12 seq
crw-rw 1 root audio 116, 33 Sep 18 16:12 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.20.1-0ubuntu2.10
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit
