Re: sysupgrade broken on imx nand targets (and maybe others too)
On Tue, Jan 24, 2023 at 11:43 PM Lanchon wrote:
>
>
>
> On 1/24/23 18:08, Koen Vandeputte wrote:
> >
> > Hi,
> >
> > I think our previous mails overlapped a bit as I didn't notice your
> > previous response :)
> >
> > I'll send the data tomorrow.
> > I'm also wondering if adding a sleep before ubiformat in the old way
> > would influence/break it's behaviour?
> possibly yes
> >
> >
> > Piotr,
> > Would you have any idea here?
> >
> >
> > Thanks again for your efforts,
> >
> > Koen
>
>
> if your active watchdog device is /dev/MyWatchdog, then...
>
>
> just before the line:
>
> ${gz}cat "$ubi_file" | ubiformat "/dev/mtd$mtdnum" -y -f - && ubiattach
> -m "$mtdnum"
>
> you could add this to try disable the watchdog:
>
> echo -n V >/dev/MyWatchdog
>
>
> or else you could tickle the watchdog while flashing like this:
>
> ${gz}cat "$ubi_file" | ubiformat "/dev/mtd$mtdnum" -y -f - 2>&1 | tee
> /dev/MyWatchdog && ubiattach -m "$mtdnum"
>
> BUT... this is a hack, cause shell option pipefail would be needed to
> detect a failure of ubiformat then, as normally only the exit code of
> the last piped process gets returned from the pipe expression.
>
>
> so these are just tests, not fixes.
>
>
> another dirty thing you could do is doubling the watchdog timeout period
> for your platform. this at least could maybe be accepted as a stopgap
> band aid in the openwrt tree. but the race condition remains, and it
> will bite back sometime. a real fix for the race should be implemented.
>
> BTW your watchdog seems to be:
> https://www.kernelconfig.io/config_imx2_wdt
>
>
> thanks!
>
>
Hi all,
Hardware:
Gateworks Ventana gw5200, naked without any additional hardware attached.
Full normal bootlog:
https://pastebin.com/raw/nSBnrHxN
Watchdog info:
root@OpenWrt:~# ubus call system watchdog
{
"status": "running",
"timeout": 30,
"frequency": 5,
"magicclose": false
}
Normal reboot command issued as user after boot:
--> a reboot on Gateworks Ventana is handled by waking the watchdog
which triggers a GPIO, controlling a PMIC to reset power
[ 56.791836] br-wan: port 1(eth0) entered disabled state
[ 56.801236] device eth0 left promiscuous mode
[ 56.805828] br-wan: port 1(eth0) entered disabled state
* reboot command issued by me here *
crond[1416]: USER root pid 2191 cmd /usr/sbin/logrotate
/etc/logrotate/logrotate.conf > /var/log/logrotate.log
[ 61.139944] ci_hdrc ci_hdrc.1: remove, state 1
[ 61.144418] usb usb2: USB disconnect, device number 1
[ 61.150316] ci_hdrc ci_hdrc.1: USB bus 2 deregistered
[ 61.155792] ci_hdrc ci_hdrc.0: remove, state 1
[ 61.160257] usb usb1: USB disconnect, device number 1
[ 61.166336] ci_hdrc ci_hdrc.0: USB bus 1 deregistered
[ 61.173094] imx2-wdt 20bc000.watchdog: Device shutdown: Expect reboot!
[ 61.179787] reboot: Restarting system
Full log sysupgrade failing: (cat ubifile | ubiformat -f -)
https://pastebin.com/raw/QPNbe49K
Time from sysupgrade-start to reboot --> 23.38 sec
Full log sysupgrade working: (ubiformat -f /tmp/ubifile)
https://pastebin.com/raw/qtSMnbDh
Time from sysupgrade-start to reboot --> 33.55 sec
Again ..
I'm NOT saying that feeding the ubi directly is a proper fix.
I'm saying that the watchdog itself seems NOT to be the cause.
It is used as a way to powercycle the board when a "reboot" command is issued.
According to me, somewhere during upgrade a 'reboot' command gets
triggered somewhere at a wrong time ..
Kind regards,
Koen
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: sysupgrade broken on imx nand targets (and maybe others too)
On 1/24/23 18:08, Koen Vandeputte wrote:
Hi,
I think our previous mails overlapped a bit as I didn't notice your
previous response :)
I'll send the data tomorrow.
I'm also wondering if adding a sleep before ubiformat in the old way
would influence/break it's behaviour?
possibly yes
Piotr,
Would you have any idea here?
Thanks again for your efforts,
Koen
if your active watchdog device is /dev/MyWatchdog, then...
just before the line:
${gz}cat "$ubi_file" | ubiformat "/dev/mtd$mtdnum" -y -f - && ubiattach
-m "$mtdnum"
you could add this to try disable the watchdog:
echo -n V >/dev/MyWatchdog
or else you could tickle the watchdog while flashing like this:
${gz}cat "$ubi_file" | ubiformat "/dev/mtd$mtdnum" -y -f - 2>&1 | tee
/dev/MyWatchdog && ubiattach -m "$mtdnum"
BUT... this is a hack, cause shell option pipefail would be needed to
detect a failure of ubiformat then, as normally only the exit code of
the last piped process gets returned from the pipe expression.
so these are just tests, not fixes.
another dirty thing you could do is doubling the watchdog timeout period
for your platform. this at least could maybe be accepted as a stopgap
band aid in the openwrt tree. but the race condition remains, and it
will bite back sometime. a real fix for the race should be implemented.
BTW your watchdog seems to be:
https://www.kernelconfig.io/config_imx2_wdt
thanks!
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: sysupgrade broken on imx nand targets (and maybe others too)
Hi Koen, On 24.01.2023 22:08, Koen Vandeputte wrote: [...] Piotr, Would you have any idea here? Not really, very strange. What makes me wondering is that it doesn't look like a h/w related thing (e.g. watchdog fire or some sudden current consumption spike resulting in reset) and more like a clean 'reboot' but delayed. Smells like some race condition. Could you do a test without USB support installed (just a blind shooting but maybe we could get more clear logs just before the reboot)? -- Cheers, Piotr ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: sysupgrade broken on imx nand targets (and maybe others too)
On Tue, Jan 24, 2023 at 9:46 PM Lanchon wrote:
>
>
>
> On 1/24/23 17:35, Lanchon wrote:
> >
> >
> > On 1/24/23 13:25, Koen Vandeputte wrote:
> >> On Tue, Jan 24, 2023 at 4:26 PM Koen Vandeputte
> >> wrote:
> >>> On Tue, Jan 24, 2023 at 7:59 AM Lanchon wrote:
> hi Koen, thanks again.
>
> i copied your log here for ease of reference:
> https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db
>
>
> first let me say:
>
> - ubinized sysupgrade is not used by any of my devices.
>
> - ubinized sysupgrade happens when when an ubi partition image is
> fed as
> an upgrade file. the image contains the complete set of ubi volumes
> that
> are normally stored within the ubi partition on your device: typically
> kernel (raw image), R/O rootfs (sqfs), and R/W overlay (ubifs). during
> said sysupgrade, the current configuration is first copied to RAM,
> then
> the ubi partition image is written, and finally -if current config is
> kept- the RAM contents are written back to the new overlay volume.
>
> - ubinized sysupgrades were known to be broken by other people at the
> time of my commits, and they wanted to remove the feature altogether
> because it was unused (look it up in the relevant pull request for my
> commits on github). as i remember it, it was broken because some ubi
> volumes within the ubi partition were sometimes mounted or R/O block
> devices were created on top of them (/dev/ubiblock*), locking the ubi
> partition and preventing the upgrade.
>
> - although my devices would normally not use such upgrades, i could
> still take a whole ubi partition backup and then test ubinized
> sysupgrade with it on my devices. in fact, if you restore the ubi
> partition image without conserving the current configuration, then
> this
> procedure is the best way to do a backup/restore of the complete state
> of the router: kernel, rootfs, and overlay are completely saved and
> restored. btw, i think this should be documented. (and this is mostly
> the reason why i added gzip support on sysupgrade: ubinized images of
> backed-up ubi partitions compress like crazy.)
>
> - my tests of ubinized sysupgrade worked after these changes but not
> before. specifically the fix is in: af34733593 base-files: fix
> ubinized
> nand sysupgrade
>
>
> regarding your log:
>
> - nand_do_platform_check succeeds:
> https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L438-L469
>
> https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L192
>
>
> - next comes the actual nand_do_flash_file:
> https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L379-L405
>
> https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L2061
>
>
> - it is determined that passed file is a ubi partition image, so
> nand_upgrade_ubinized is invoked:
> https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L260-L269
>
> https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L2088
>
>
> nand_upgrade_ubinized is basically a one-liner:
> ${gz}cat "$ubi_file" | ubiformat "/dev/mtd$mtdnum" -y -f - &&
> ubiattach
> -m "$mtdnum"
>
> cat/zcat the image, feeding that to ubiformat -f - which writes it.
>
>
> and the write does take place, but take a look:
>
> ---
>
> + cat /tmp/nandnew.ubi
> + ubiformat /dev/mtd2 -y -f -
> ubiformat: mtd2 (nand), size 250609664 bytes (239.0 MiB), 1912
> eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 2048 bytes
>
> libscan: scanning eraseblock 0 -- 0 % complete
> libscan: scanning eraseblock 1 -- 0 % complete
> libscan: scanning eraseblock 2 -- 0 % complete
> ...
> libscan: scanning eraseblock 1868 -- 97 % complete
>
> libscan: scanning eraseblock
> [ 207.876200] ci_hdrc ci_hdrc.1: remove, state 1
> 1869 -- 97 % complete
>
> libscan:
> [ 207.883339] usb usb2: USB disconnect, device number 1
> scanning eraseblock 1870 -- 97 %
> [ 207.891238] usb 2-1: USB disconnect, device number 2
> complete
>
> libscan: scanning eras
> [ 207.901522] ci_hdrc ci_hdrc.1: USB bus 2 deregistered
> eblock 1871 -- 97 % complete
>
> libscan: scanning eraseblock 1872
> [ 207.910396] ci_hdrc ci_hdrc.0: remove, state 4
>
Re: sysupgrade broken on imx nand targets (and maybe others too)
On 1/24/23 17:35, Lanchon wrote:
On 1/24/23 13:25, Koen Vandeputte wrote:
On Tue, Jan 24, 2023 at 4:26 PM Koen Vandeputte
wrote:
On Tue, Jan 24, 2023 at 7:59 AM Lanchon wrote:
hi Koen, thanks again.
i copied your log here for ease of reference:
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db
first let me say:
- ubinized sysupgrade is not used by any of my devices.
- ubinized sysupgrade happens when when an ubi partition image is
fed as
an upgrade file. the image contains the complete set of ubi volumes
that
are normally stored within the ubi partition on your device: typically
kernel (raw image), R/O rootfs (sqfs), and R/W overlay (ubifs). during
said sysupgrade, the current configuration is first copied to RAM,
then
the ubi partition image is written, and finally -if current config is
kept- the RAM contents are written back to the new overlay volume.
- ubinized sysupgrades were known to be broken by other people at the
time of my commits, and they wanted to remove the feature altogether
because it was unused (look it up in the relevant pull request for my
commits on github). as i remember it, it was broken because some ubi
volumes within the ubi partition were sometimes mounted or R/O block
devices were created on top of them (/dev/ubiblock*), locking the ubi
partition and preventing the upgrade.
- although my devices would normally not use such upgrades, i could
still take a whole ubi partition backup and then test ubinized
sysupgrade with it on my devices. in fact, if you restore the ubi
partition image without conserving the current configuration, then
this
procedure is the best way to do a backup/restore of the complete state
of the router: kernel, rootfs, and overlay are completely saved and
restored. btw, i think this should be documented. (and this is mostly
the reason why i added gzip support on sysupgrade: ubinized images of
backed-up ubi partitions compress like crazy.)
- my tests of ubinized sysupgrade worked after these changes but not
before. specifically the fix is in: af34733593 base-files: fix
ubinized
nand sysupgrade
regarding your log:
- nand_do_platform_check succeeds:
https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L438-L469
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L192
- next comes the actual nand_do_flash_file:
https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L379-L405
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L2061
- it is determined that passed file is a ubi partition image, so
nand_upgrade_ubinized is invoked:
https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L260-L269
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L2088
nand_upgrade_ubinized is basically a one-liner:
${gz}cat "$ubi_file" | ubiformat "/dev/mtd$mtdnum" -y -f - &&
ubiattach
-m "$mtdnum"
cat/zcat the image, feeding that to ubiformat -f - which writes it.
and the write does take place, but take a look:
---
+ cat /tmp/nandnew.ubi
+ ubiformat /dev/mtd2 -y -f -
ubiformat: mtd2 (nand), size 250609664 bytes (239.0 MiB), 1912
eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 2048 bytes
libscan: scanning eraseblock 0 -- 0 % complete
libscan: scanning eraseblock 1 -- 0 % complete
libscan: scanning eraseblock 2 -- 0 % complete
...
libscan: scanning eraseblock 1868 -- 97 % complete
libscan: scanning eraseblock
[ 207.876200] ci_hdrc ci_hdrc.1: remove, state 1
1869 -- 97 % complete
libscan:
[ 207.883339] usb usb2: USB disconnect, device number 1
scanning eraseblock 1870 -- 97 %
[ 207.891238] usb 2-1: USB disconnect, device number 2
complete
libscan: scanning eras
[ 207.901522] ci_hdrc ci_hdrc.1: USB bus 2 deregistered
eblock 1871 -- 97 % complete
libscan: scanning eraseblock 1872
[ 207.910396] ci_hdrc ci_hdrc.0: remove, state 4
-- 97 % complete
libscan: scan
[ 207.917055] usb usb1: USB disconnect, device number 1
ning eraseblock 1873 -- 98 % comp
[ 207.925651] ci_hdrc ci_hdrc.0: USB bus 1 deregistered
lete
libscan: scanning eraseblo
[ 207.934010] imx2-wdt 20bc000.watchdog: Device shutdown: Expect
reboot!
ck 1874 -- 98 % complete
libsca
[ 207.942382] reboot: Restarting system
---
while sysupgrade is flashing UBI the partition, the system is
rebooted;
i don't know why.
here are the cleaned-up kernel messages:
[ 207.876200] ci_hdrc ci_hdrc.1: remove, state 1
[ 207.883339] usb usb2: USB disconnect, device number 1
[ 207.891238] usb 2-1: USB disconnect, device number 2
[ 207.901522] ci_hdrc ci_hdrc.1: USB bus 2 deregistered
[ 207.910396] ci_hdrc
Re: sysupgrade broken on imx nand targets (and maybe others too)
On 1/24/23 13:25, Koen Vandeputte wrote:
On Tue, Jan 24, 2023 at 4:26 PM Koen Vandeputte
wrote:
On Tue, Jan 24, 2023 at 7:59 AM Lanchon wrote:
hi Koen, thanks again.
i copied your log here for ease of reference:
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db
first let me say:
- ubinized sysupgrade is not used by any of my devices.
- ubinized sysupgrade happens when when an ubi partition image is fed as
an upgrade file. the image contains the complete set of ubi volumes that
are normally stored within the ubi partition on your device: typically
kernel (raw image), R/O rootfs (sqfs), and R/W overlay (ubifs). during
said sysupgrade, the current configuration is first copied to RAM, then
the ubi partition image is written, and finally -if current config is
kept- the RAM contents are written back to the new overlay volume.
- ubinized sysupgrades were known to be broken by other people at the
time of my commits, and they wanted to remove the feature altogether
because it was unused (look it up in the relevant pull request for my
commits on github). as i remember it, it was broken because some ubi
volumes within the ubi partition were sometimes mounted or R/O block
devices were created on top of them (/dev/ubiblock*), locking the ubi
partition and preventing the upgrade.
- although my devices would normally not use such upgrades, i could
still take a whole ubi partition backup and then test ubinized
sysupgrade with it on my devices. in fact, if you restore the ubi
partition image without conserving the current configuration, then this
procedure is the best way to do a backup/restore of the complete state
of the router: kernel, rootfs, and overlay are completely saved and
restored. btw, i think this should be documented. (and this is mostly
the reason why i added gzip support on sysupgrade: ubinized images of
backed-up ubi partitions compress like crazy.)
- my tests of ubinized sysupgrade worked after these changes but not
before. specifically the fix is in: af34733593 base-files: fix ubinized
nand sysupgrade
regarding your log:
- nand_do_platform_check succeeds:
https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L438-L469
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L192
- next comes the actual nand_do_flash_file:
https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L379-L405
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L2061
- it is determined that passed file is a ubi partition image, so
nand_upgrade_ubinized is invoked:
https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L260-L269
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L2088
nand_upgrade_ubinized is basically a one-liner:
${gz}cat "$ubi_file" | ubiformat "/dev/mtd$mtdnum" -y -f - && ubiattach
-m "$mtdnum"
cat/zcat the image, feeding that to ubiformat -f - which writes it.
and the write does take place, but take a look:
---
+ cat /tmp/nandnew.ubi
+ ubiformat /dev/mtd2 -y -f -
ubiformat: mtd2 (nand), size 250609664 bytes (239.0 MiB), 1912
eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 2048 bytes
libscan: scanning eraseblock 0 -- 0 % complete
libscan: scanning eraseblock 1 -- 0 % complete
libscan: scanning eraseblock 2 -- 0 % complete
...
libscan: scanning eraseblock 1868 -- 97 % complete
libscan: scanning eraseblock
[ 207.876200] ci_hdrc ci_hdrc.1: remove, state 1
1869 -- 97 % complete
libscan:
[ 207.883339] usb usb2: USB disconnect, device number 1
scanning eraseblock 1870 -- 97 %
[ 207.891238] usb 2-1: USB disconnect, device number 2
complete
libscan: scanning eras
[ 207.901522] ci_hdrc ci_hdrc.1: USB bus 2 deregistered
eblock 1871 -- 97 % complete
libscan: scanning eraseblock 1872
[ 207.910396] ci_hdrc ci_hdrc.0: remove, state 4
-- 97 % complete
libscan: scan
[ 207.917055] usb usb1: USB disconnect, device number 1
ning eraseblock 1873 -- 98 % comp
[ 207.925651] ci_hdrc ci_hdrc.0: USB bus 1 deregistered
lete
libscan: scanning eraseblo
[ 207.934010] imx2-wdt 20bc000.watchdog: Device shutdown: Expect reboot!
ck 1874 -- 98 % complete
libsca
[ 207.942382] reboot: Restarting system
---
while sysupgrade is flashing UBI the partition, the system is rebooted;
i don't know why.
here are the cleaned-up kernel messages:
[ 207.876200] ci_hdrc ci_hdrc.1: remove, state 1
[ 207.883339] usb usb2: USB disconnect, device number 1
[ 207.891238] usb 2-1: USB disconnect, device number 2
[ 207.901522] ci_hdrc ci_hdrc.1: USB bus 2 deregistered
[ 207.910396] ci_hdrc ci_hdrc.0: remove, state 4
[ 207.917055] usb usb1: USB disconnect,
Re: sysupgrade broken on imx nand targets (and maybe others too)
On Tue, Jan 24, 2023 at 5:49 PM wrote: > > On 24.01.2023 17:13, Lanchon wrote: > > the problem lies elsewhere: on your platform something is rebooting the > > system asynchronously while it is updating. this is very dangerous and must > > be fixed elsewhere in code. > > just a wild guess. faulty power supplies sometimes lead to reboots if the > power supplied does not suffice. maybe a direction to investigate? ..sunny > regards ede > Hi, In this case it should be faulty on all of my 53 boards. Some are powered using a DC supply, others through PoE. I don't think this is the issue here. :-) Regards, Koen ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: sysupgrade broken on imx nand targets (and maybe others too)
On 24.01.2023 17:13, Lanchon wrote: the problem lies elsewhere: on your platform something is rebooting the system asynchronously while it is updating. this is very dangerous and must be fixed elsewhere in code. just a wild guess. faulty power supplies sometimes lead to reboots if the power supplied does not suffice. maybe a direction to investigate? ..sunny regards ede ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: sysupgrade broken on imx nand targets (and maybe others too)
On Tue, Jan 24, 2023 at 4:26 PM Koen Vandeputte
wrote:
>
> On Tue, Jan 24, 2023 at 7:59 AM Lanchon wrote:
> >
> > hi Koen, thanks again.
> >
> > i copied your log here for ease of reference:
> > https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db
> >
> >
> > first let me say:
> >
> > - ubinized sysupgrade is not used by any of my devices.
> >
> > - ubinized sysupgrade happens when when an ubi partition image is fed as
> > an upgrade file. the image contains the complete set of ubi volumes that
> > are normally stored within the ubi partition on your device: typically
> > kernel (raw image), R/O rootfs (sqfs), and R/W overlay (ubifs). during
> > said sysupgrade, the current configuration is first copied to RAM, then
> > the ubi partition image is written, and finally -if current config is
> > kept- the RAM contents are written back to the new overlay volume.
> >
> > - ubinized sysupgrades were known to be broken by other people at the
> > time of my commits, and they wanted to remove the feature altogether
> > because it was unused (look it up in the relevant pull request for my
> > commits on github). as i remember it, it was broken because some ubi
> > volumes within the ubi partition were sometimes mounted or R/O block
> > devices were created on top of them (/dev/ubiblock*), locking the ubi
> > partition and preventing the upgrade.
> >
> > - although my devices would normally not use such upgrades, i could
> > still take a whole ubi partition backup and then test ubinized
> > sysupgrade with it on my devices. in fact, if you restore the ubi
> > partition image without conserving the current configuration, then this
> > procedure is the best way to do a backup/restore of the complete state
> > of the router: kernel, rootfs, and overlay are completely saved and
> > restored. btw, i think this should be documented. (and this is mostly
> > the reason why i added gzip support on sysupgrade: ubinized images of
> > backed-up ubi partitions compress like crazy.)
> >
> > - my tests of ubinized sysupgrade worked after these changes but not
> > before. specifically the fix is in: af34733593 base-files: fix ubinized
> > nand sysupgrade
> >
> >
> > regarding your log:
> >
> > - nand_do_platform_check succeeds:
> > https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L438-L469
> > https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L192
> >
> > - next comes the actual nand_do_flash_file:
> > https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L379-L405
> > https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L2061
> >
> > - it is determined that passed file is a ubi partition image, so
> > nand_upgrade_ubinized is invoked:
> > https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L260-L269
> > https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L2088
> >
> > nand_upgrade_ubinized is basically a one-liner:
> > ${gz}cat "$ubi_file" | ubiformat "/dev/mtd$mtdnum" -y -f - && ubiattach
> > -m "$mtdnum"
> >
> > cat/zcat the image, feeding that to ubiformat -f - which writes it.
> >
> >
> > and the write does take place, but take a look:
> >
> > ---
> >
> > + cat /tmp/nandnew.ubi
> > + ubiformat /dev/mtd2 -y -f -
> > ubiformat: mtd2 (nand), size 250609664 bytes (239.0 MiB), 1912
> > eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 2048 bytes
> >
> > libscan: scanning eraseblock 0 -- 0 % complete
> > libscan: scanning eraseblock 1 -- 0 % complete
> > libscan: scanning eraseblock 2 -- 0 % complete
> > ...
> > libscan: scanning eraseblock 1868 -- 97 % complete
> >
> > libscan: scanning eraseblock
> > [ 207.876200] ci_hdrc ci_hdrc.1: remove, state 1
> > 1869 -- 97 % complete
> >
> > libscan:
> > [ 207.883339] usb usb2: USB disconnect, device number 1
> > scanning eraseblock 1870 -- 97 %
> > [ 207.891238] usb 2-1: USB disconnect, device number 2
> > complete
> >
> > libscan: scanning eras
> > [ 207.901522] ci_hdrc ci_hdrc.1: USB bus 2 deregistered
> > eblock 1871 -- 97 % complete
> >
> > libscan: scanning eraseblock 1872
> > [ 207.910396] ci_hdrc ci_hdrc.0: remove, state 4
> > -- 97 % complete
> >
> > libscan: scan
> > [ 207.917055] usb usb1: USB disconnect, device number 1
> > ning eraseblock 1873 -- 98 % comp
> > [ 207.925651] ci_hdrc ci_hdrc.0: USB bus 1 deregistered
> > lete
> >
> > libscan: scanning eraseblo
> > [ 207.934010] imx2-wdt 20bc000.watchdog: Device shutdown: Expect reboot!
> > ck 1874 -- 98 % complete
> >
> > libsca
> > [ 207.942382] reboot: Restarting system
> >
> > ---
> >
> >
> > while sysupgrade is flashing UBI the partition, the system is
Re: sysupgrade broken on imx nand targets (and maybe others too)
On 1/24/23 12:26, Koen Vandeputte wrote:
On Tue, Jan 24, 2023 at 7:59 AM Lanchon wrote:
hi Koen, thanks again.
i copied your log here for ease of reference:
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db
first let me say:
- ubinized sysupgrade is not used by any of my devices.
- ubinized sysupgrade happens when when an ubi partition image is fed as
an upgrade file. the image contains the complete set of ubi volumes that
are normally stored within the ubi partition on your device: typically
kernel (raw image), R/O rootfs (sqfs), and R/W overlay (ubifs). during
said sysupgrade, the current configuration is first copied to RAM, then
the ubi partition image is written, and finally -if current config is
kept- the RAM contents are written back to the new overlay volume.
- ubinized sysupgrades were known to be broken by other people at the
time of my commits, and they wanted to remove the feature altogether
because it was unused (look it up in the relevant pull request for my
commits on github). as i remember it, it was broken because some ubi
volumes within the ubi partition were sometimes mounted or R/O block
devices were created on top of them (/dev/ubiblock*), locking the ubi
partition and preventing the upgrade.
- although my devices would normally not use such upgrades, i could
still take a whole ubi partition backup and then test ubinized
sysupgrade with it on my devices. in fact, if you restore the ubi
partition image without conserving the current configuration, then this
procedure is the best way to do a backup/restore of the complete state
of the router: kernel, rootfs, and overlay are completely saved and
restored. btw, i think this should be documented. (and this is mostly
the reason why i added gzip support on sysupgrade: ubinized images of
backed-up ubi partitions compress like crazy.)
- my tests of ubinized sysupgrade worked after these changes but not
before. specifically the fix is in: af34733593 base-files: fix ubinized
nand sysupgrade
regarding your log:
- nand_do_platform_check succeeds:
https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L438-L469
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L192
- next comes the actual nand_do_flash_file:
https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L379-L405
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L2061
- it is determined that passed file is a ubi partition image, so
nand_upgrade_ubinized is invoked:
https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L260-L269
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L2088
nand_upgrade_ubinized is basically a one-liner:
${gz}cat "$ubi_file" | ubiformat "/dev/mtd$mtdnum" -y -f - && ubiattach
-m "$mtdnum"
cat/zcat the image, feeding that to ubiformat -f - which writes it.
and the write does take place, but take a look:
---
+ cat /tmp/nandnew.ubi
+ ubiformat /dev/mtd2 -y -f -
ubiformat: mtd2 (nand), size 250609664 bytes (239.0 MiB), 1912
eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 2048 bytes
libscan: scanning eraseblock 0 -- 0 % complete
libscan: scanning eraseblock 1 -- 0 % complete
libscan: scanning eraseblock 2 -- 0 % complete
...
libscan: scanning eraseblock 1868 -- 97 % complete
libscan: scanning eraseblock
[ 207.876200] ci_hdrc ci_hdrc.1: remove, state 1
1869 -- 97 % complete
libscan:
[ 207.883339] usb usb2: USB disconnect, device number 1
scanning eraseblock 1870 -- 97 %
[ 207.891238] usb 2-1: USB disconnect, device number 2
complete
libscan: scanning eras
[ 207.901522] ci_hdrc ci_hdrc.1: USB bus 2 deregistered
eblock 1871 -- 97 % complete
libscan: scanning eraseblock 1872
[ 207.910396] ci_hdrc ci_hdrc.0: remove, state 4
-- 97 % complete
libscan: scan
[ 207.917055] usb usb1: USB disconnect, device number 1
ning eraseblock 1873 -- 98 % comp
[ 207.925651] ci_hdrc ci_hdrc.0: USB bus 1 deregistered
lete
libscan: scanning eraseblo
[ 207.934010] imx2-wdt 20bc000.watchdog: Device shutdown: Expect reboot!
ck 1874 -- 98 % complete
libsca
[ 207.942382] reboot: Restarting system
---
while sysupgrade is flashing UBI the partition, the system is rebooted;
i don't know why.
here are the cleaned-up kernel messages:
[ 207.876200] ci_hdrc ci_hdrc.1: remove, state 1
[ 207.883339] usb usb2: USB disconnect, device number 1
[ 207.891238] usb 2-1: USB disconnect, device number 2
[ 207.901522] ci_hdrc ci_hdrc.1: USB bus 2 deregistered
[ 207.910396] ci_hdrc ci_hdrc.0: remove, state 4
[ 207.917055] usb usb1: USB disconnect, device number 1
[ 207.925651] ci_hdrc ci_hdrc.0: USB bus 1
Re: sysupgrade broken on imx nand targets (and maybe others too)
On Tue, Jan 24, 2023 at 7:59 AM Lanchon wrote:
>
> hi Koen, thanks again.
>
> i copied your log here for ease of reference:
> https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db
>
>
> first let me say:
>
> - ubinized sysupgrade is not used by any of my devices.
>
> - ubinized sysupgrade happens when when an ubi partition image is fed as
> an upgrade file. the image contains the complete set of ubi volumes that
> are normally stored within the ubi partition on your device: typically
> kernel (raw image), R/O rootfs (sqfs), and R/W overlay (ubifs). during
> said sysupgrade, the current configuration is first copied to RAM, then
> the ubi partition image is written, and finally -if current config is
> kept- the RAM contents are written back to the new overlay volume.
>
> - ubinized sysupgrades were known to be broken by other people at the
> time of my commits, and they wanted to remove the feature altogether
> because it was unused (look it up in the relevant pull request for my
> commits on github). as i remember it, it was broken because some ubi
> volumes within the ubi partition were sometimes mounted or R/O block
> devices were created on top of them (/dev/ubiblock*), locking the ubi
> partition and preventing the upgrade.
>
> - although my devices would normally not use such upgrades, i could
> still take a whole ubi partition backup and then test ubinized
> sysupgrade with it on my devices. in fact, if you restore the ubi
> partition image without conserving the current configuration, then this
> procedure is the best way to do a backup/restore of the complete state
> of the router: kernel, rootfs, and overlay are completely saved and
> restored. btw, i think this should be documented. (and this is mostly
> the reason why i added gzip support on sysupgrade: ubinized images of
> backed-up ubi partitions compress like crazy.)
>
> - my tests of ubinized sysupgrade worked after these changes but not
> before. specifically the fix is in: af34733593 base-files: fix ubinized
> nand sysupgrade
>
>
> regarding your log:
>
> - nand_do_platform_check succeeds:
> https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L438-L469
> https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L192
>
> - next comes the actual nand_do_flash_file:
> https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L379-L405
> https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L2061
>
> - it is determined that passed file is a ubi partition image, so
> nand_upgrade_ubinized is invoked:
> https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L260-L269
> https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L2088
>
> nand_upgrade_ubinized is basically a one-liner:
> ${gz}cat "$ubi_file" | ubiformat "/dev/mtd$mtdnum" -y -f - && ubiattach
> -m "$mtdnum"
>
> cat/zcat the image, feeding that to ubiformat -f - which writes it.
>
>
> and the write does take place, but take a look:
>
> ---
>
> + cat /tmp/nandnew.ubi
> + ubiformat /dev/mtd2 -y -f -
> ubiformat: mtd2 (nand), size 250609664 bytes (239.0 MiB), 1912
> eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 2048 bytes
>
> libscan: scanning eraseblock 0 -- 0 % complete
> libscan: scanning eraseblock 1 -- 0 % complete
> libscan: scanning eraseblock 2 -- 0 % complete
> ...
> libscan: scanning eraseblock 1868 -- 97 % complete
>
> libscan: scanning eraseblock
> [ 207.876200] ci_hdrc ci_hdrc.1: remove, state 1
> 1869 -- 97 % complete
>
> libscan:
> [ 207.883339] usb usb2: USB disconnect, device number 1
> scanning eraseblock 1870 -- 97 %
> [ 207.891238] usb 2-1: USB disconnect, device number 2
> complete
>
> libscan: scanning eras
> [ 207.901522] ci_hdrc ci_hdrc.1: USB bus 2 deregistered
> eblock 1871 -- 97 % complete
>
> libscan: scanning eraseblock 1872
> [ 207.910396] ci_hdrc ci_hdrc.0: remove, state 4
> -- 97 % complete
>
> libscan: scan
> [ 207.917055] usb usb1: USB disconnect, device number 1
> ning eraseblock 1873 -- 98 % comp
> [ 207.925651] ci_hdrc ci_hdrc.0: USB bus 1 deregistered
> lete
>
> libscan: scanning eraseblo
> [ 207.934010] imx2-wdt 20bc000.watchdog: Device shutdown: Expect reboot!
> ck 1874 -- 98 % complete
>
> libsca
> [ 207.942382] reboot: Restarting system
>
> ---
>
>
> while sysupgrade is flashing UBI the partition, the system is rebooted;
> i don't know why.
>
> here are the cleaned-up kernel messages:
>
> [ 207.876200] ci_hdrc ci_hdrc.1: remove, state 1
> [ 207.883339] usb usb2: USB disconnect, device number 1
> [ 207.891238] usb 2-1: USB disconnect, device number 2
> [ 207.901522] ci_hdrc ci_hdrc.1: USB bus 2
Re: sysupgrade broken on imx nand targets (and maybe others too)
hi Koen, thanks again.
i copied your log here for ease of reference:
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db
first let me say:
- ubinized sysupgrade is not used by any of my devices.
- ubinized sysupgrade happens when when an ubi partition image is fed as
an upgrade file. the image contains the complete set of ubi volumes that
are normally stored within the ubi partition on your device: typically
kernel (raw image), R/O rootfs (sqfs), and R/W overlay (ubifs). during
said sysupgrade, the current configuration is first copied to RAM, then
the ubi partition image is written, and finally -if current config is
kept- the RAM contents are written back to the new overlay volume.
- ubinized sysupgrades were known to be broken by other people at the
time of my commits, and they wanted to remove the feature altogether
because it was unused (look it up in the relevant pull request for my
commits on github). as i remember it, it was broken because some ubi
volumes within the ubi partition were sometimes mounted or R/O block
devices were created on top of them (/dev/ubiblock*), locking the ubi
partition and preventing the upgrade.
- although my devices would normally not use such upgrades, i could
still take a whole ubi partition backup and then test ubinized
sysupgrade with it on my devices. in fact, if you restore the ubi
partition image without conserving the current configuration, then this
procedure is the best way to do a backup/restore of the complete state
of the router: kernel, rootfs, and overlay are completely saved and
restored. btw, i think this should be documented. (and this is mostly
the reason why i added gzip support on sysupgrade: ubinized images of
backed-up ubi partitions compress like crazy.)
- my tests of ubinized sysupgrade worked after these changes but not
before. specifically the fix is in: af34733593 base-files: fix ubinized
nand sysupgrade
regarding your log:
- nand_do_platform_check succeeds:
https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L438-L469
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L192
- next comes the actual nand_do_flash_file:
https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L379-L405
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L2061
- it is determined that passed file is a ubi partition image, so
nand_upgrade_ubinized is invoked:
https://github.com/openwrt/openwrt/blob/ac21dff5b67698c09f54a4b98d6f9f12af17edd4/package/base-files/files/lib/upgrade/nand.sh#L260-L269
https://gist.github.com/Lanchon/f24ea9c16eda5ffaa5085a7b240238db#file-imx6dl-gw52xx-ubinized-sysupgrade-log-txt-L2088
nand_upgrade_ubinized is basically a one-liner:
${gz}cat "$ubi_file" | ubiformat "/dev/mtd$mtdnum" -y -f - && ubiattach
-m "$mtdnum"
cat/zcat the image, feeding that to ubiformat -f - which writes it.
and the write does take place, but take a look:
---
+ cat /tmp/nandnew.ubi
+ ubiformat /dev/mtd2 -y -f -
ubiformat: mtd2 (nand), size 250609664 bytes (239.0 MiB), 1912
eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 2048 bytes
libscan: scanning eraseblock 0 -- 0 % complete
libscan: scanning eraseblock 1 -- 0 % complete
libscan: scanning eraseblock 2 -- 0 % complete
...
libscan: scanning eraseblock 1868 -- 97 % complete
libscan: scanning eraseblock
[ 207.876200] ci_hdrc ci_hdrc.1: remove, state 1
1869 -- 97 % complete
libscan:
[ 207.883339] usb usb2: USB disconnect, device number 1
scanning eraseblock 1870 -- 97 %
[ 207.891238] usb 2-1: USB disconnect, device number 2
complete
libscan: scanning eras
[ 207.901522] ci_hdrc ci_hdrc.1: USB bus 2 deregistered
eblock 1871 -- 97 % complete
libscan: scanning eraseblock 1872
[ 207.910396] ci_hdrc ci_hdrc.0: remove, state 4
-- 97 % complete
libscan: scan
[ 207.917055] usb usb1: USB disconnect, device number 1
ning eraseblock 1873 -- 98 % comp
[ 207.925651] ci_hdrc ci_hdrc.0: USB bus 1 deregistered
lete
libscan: scanning eraseblo
[ 207.934010] imx2-wdt 20bc000.watchdog: Device shutdown: Expect reboot!
ck 1874 -- 98 % complete
libsca
[ 207.942382] reboot: Restarting system
---
while sysupgrade is flashing UBI the partition, the system is rebooted;
i don't know why.
here are the cleaned-up kernel messages:
[ 207.876200] ci_hdrc ci_hdrc.1: remove, state 1
[ 207.883339] usb usb2: USB disconnect, device number 1
[ 207.891238] usb 2-1: USB disconnect, device number 2
[ 207.901522] ci_hdrc ci_hdrc.1: USB bus 2 deregistered
[ 207.910396] ci_hdrc ci_hdrc.0: remove, state 4
[ 207.917055] usb usb1: USB disconnect, device number 1
[ 207.925651] ci_hdrc ci_hdrc.0: USB bus 1 deregistered
[ 207.934010] imx2-wdt 20bc000.watchdog: Device
Re: sysupgrade broken on imx nand targets (and maybe others too)
hi Koen, thanks for this detailed report, i'll try to take a look at it tonight. thank you again. On 1/23/23 12:37, Koen Vandeputte wrote: Hi Rodrigo, After a long absence and now testing the latest master, I noticed that imx nand flash sysupgrade was broken: expected behaviour: - scan the nand - write image - format empty space -reboot Seen behaviour: - Scan the nand - reboots I traced it back to this batch of commits by you: 9d1e687da3 base-files: verify nand sysupgrade images 9710712120 base-files: accept gzipped nand sysupgrade images af34733593 base-files: fix ubinized nand sysupgrade e25e6d8e54 base-files: fix and clean up nand sysupgrade code It can be easily confirmed by reverting /lib/upgrade/nand.sh with a version before these commits are applied to it. I added a "set -x" to nand.sh to get more detailed logs: https://pastebin.com/raw/yxY0SK1x Any idea? Thanks, Koen ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
sysupgrade broken on imx nand targets (and maybe others too)
Hi Rodrigo, After a long absence and now testing the latest master, I noticed that imx nand flash sysupgrade was broken: expected behaviour: - scan the nand - write image - format empty space -reboot Seen behaviour: - Scan the nand - reboots I traced it back to this batch of commits by you: 9d1e687da3 base-files: verify nand sysupgrade images 9710712120 base-files: accept gzipped nand sysupgrade images af34733593 base-files: fix ubinized nand sysupgrade e25e6d8e54 base-files: fix and clean up nand sysupgrade code It can be easily confirmed by reverting /lib/upgrade/nand.sh with a version before these commits are applied to it. I added a "set -x" to nand.sh to get more detailed logs: https://pastebin.com/raw/yxY0SK1x Any idea? Thanks, Koen ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
