Hello Mark,
added Richard Weinberger to cc...
Am 12.07.2018 um 02:28 schrieb Mark Spieth:
Hi
In the process of investigating a boot failure on one of our devices, the
UBI: fixable bit-flip detected at PEB
message was seen with the following behaviour during kernel load in u-boot.
Read [2285568] bytes
UBI: fixable bit-flip detected at PEB 415
UBI: schedule PEB 415 for scrubbing
UBI: fixable bit-flip detected at PEB 415
UBI: fixable bit-flip detected at PEB 419
UBI: schedule PEB 419 for scrubbing
UBI: fixable bit-flip detected at PEB 419
UBI: fixable bit-flip detected at PEB 420
UBI: schedule PEB 420 for scrubbing
UBI: fixable bit-flip detected at PEB 420
UBI: fixable bit-flip detected at PEB 419
UBI: fixable bit-flip detected at PEB 420
UBI: fixable bit-flip detected at PEB 419
UBI: fixable bit-flip detected at PEB 420
UBI: fixable bit-flip detected at PEB 419
UBI: fixable bit-flip detected at PEB 420
UBI: fixable bit-flip detected at PEB 419
UBI: fixable bit-flip detected at PEB 420
UBI: fixable bit-flip detected at PEB 419
UBI: fixable bit-flip detected at PEB 420
UBI: fixable bit-flip detected at PEB 419
This repeats until reset.
U boot is a patched version of 2010.06 supplied by the chip vendor. No newer version is available
from the vendor to try.
:-(
Can you use current mainline ? It s hard to say something
about a 8 year old vendor U-Boot version ...
The patches include the init eba/wl swap.
What do you mean here?
A more detailed log with debugging available follows:
UBI: fixable bit-flip detected at PEB 419
UBI DBG: schedule_erase: schedule erasure of PEB 419, EC 19, torture 0
UBI DBG: erase_worker: erase PEB 419 EC 19
UBI DBG: sync_erase: erase PEB 419, old EC 19
UBI DBG: do_sync_erase: erase PEB 419
UBI DBG: sync_erase: erased PEB 419, new EC 20
UBI DBG: ubi_io_write_ec_hdr: write EC header to PEB 419
UBI DBG: ubi_io_write: write 2048 bytes to PEB 419:0
UBI DBG: ensure_wear_leveling: schedule scrubbing
UBI DBG: wear_leveling_worker: scrub PEB 420 to PEB 419
UBI DBG: ubi_io_read_vid_hdr: read VID header from PEB 420
UBI DBG: ubi_io_read: read 2048 bytes from PEB 420:2048
UBI DBG: ubi_eba_copy_leb: copy LEB 6:11, PEB 420 to PEB 419
UBI DBG: ubi_eba_copy_leb: read 126976 bytes of data
UBI DBG: ubi_io_read: read 126976 bytes from PEB 420:4096
UBI: fixable bit-flip detected at PEB 420
UBI DBG: ubi_io_write_vid_hdr: write VID header to PEB 419
UBI DBG: ubi_io_write: write 2048 bytes to PEB 419:2048
UBI DBG: ubi_io_read_vid_hdr: read VID header from PEB 419
UBI DBG: ubi_io_read: read 2048 bytes from PEB 419:2048
UBI DBG: ubi_io_write: write 126976 bytes to PEB 419:4096
UBI DBG: ubi_io_read: read 126976 bytes from PEB 419:4096
UBI: fixable bit-flip detected at PEB 419
UBI DBG: schedule_erase: schedule erasure of PEB 419, EC 20, torture 0
UBI DBG: erase_worker: erase PEB 419 EC 20
UBI DBG: sync_erase: erase PEB 419, old EC 20
UBI DBG: do_sync_erase: erase PEB 419
UBI DBG: sync_erase: erased PEB 419, new EC 21
UBI DBG: ubi_io_write_ec_hdr: write EC header to PEB 419
UBI DBG: ubi_io_write: write 2048 bytes to PEB 419:0
UBI DBG: ensure_wear_leveling: schedule scrubbing
UBI DBG: wear_leveling_worker: scrub PEB 420 to PEB 419
UBI DBG: ubi_io_read_vid_hdr: read VID header from PEB 420
UBI DBG: ubi_io_read: read 2048 bytes from PEB 420:2048
UBI DBG: ubi_eba_copy_leb: copy LEB 6:11, PEB 420 to PEB 419
UBI DBG: ubi_eba_copy_leb: read 126976 bytes of data
UBI DBG: ubi_io_read: read 126976 bytes from PEB 420:4096
UBI: fixable bit-flip detected at PEB 420
UBI DBG: ubi_io_write_vid_hdr: write VID header to PEB 419
UBI DBG: ubi_io_write: write 2048 bytes to PEB 419:2048
UBI DBG: ubi_io_read_vid_hdr: read VID header from PEB 419
UBI DBG: ubi_io_read: read 2048 bytes from PEB 419:2048
UBI DBG: ubi_io_write: write 126976 bytes to PEB 419:4096
UBI DBG: ubi_io_read: read 126976 bytes from PEB 419:4096
UBI: fixable bit-flip detected at PEB 419
Investigation showed that a read with correctable bit errors was done returning -EUCLEAN to the ubi
read function.
Having read https://lists.denx.de/pipermail/u-boot/2013-September/161961.html which details a
workaround to not return EUCLEAN from the NAND reader unless the number of fixed bits returned was
75% of the total number of correctable bits was exceeded during the read. This was impleneted in
this version of ubi in uboot 2010.06 and it does hide the bit-flip infinite issue since this is new
NAND FLASH. The original 2010.06 implementation returns EUCLEAN for any number of fixable bit flips
and thus causes the PEB move to the best free one (scrub mode in wear_leveling_worker).
This fix is not a root cause fix though. Investigating further led to the following root cause
solution. The following is AFAICT.
When the scrubber chooses a PEB to move the from the free balanced tree. This tree is sorted by EC
(erase count) and then by PEB number.
The find_wl_entry call uses a max parameter of WL_FREE_MAX_DIFF which is 8192 in this config. So the
find_wl_entry function will find a PEB that is better in error count that the current PEB EC. This
can easily cause it to find the PEB that was just moved from if it is the lowest numbered PEB in the
free tree. Waiting for EC to go above 8192 would take a long time and cause premature aging of the
flash PEBs in question.
The easy solution is to change the max parameter to this call to 0 so it finds a PEB with a smaller
EC than the one being replaced. This means it wont use the previously discarded PEB as its first
choice.
I am not sure if it is so easy ...
This fix was implemented and fixable bit-flip errors no longer hang/freeze the boot process! UBI
erase and reformat was used between re-tests to get consistent results.
Adding the above 75% correctable bitflip threshold is also a good thing as less movement will ensue
when the FLASH is new, but as the flash ages, the root cause will once again be invoked causing
un-recoverable boot failures.
Note this fault is also in the latest kernel drivers for UBI and may also exist in other wear
leveling implementations. The kernel driver issue may be at fault for android devices locking
up/freezing sporadically during FLASH read when scrubbing due to a relatively full flash and
correctable errors causing ping pong PEB moves.
The question is, is my root cause solution sound or have I missed something?
I have to think about, before I write nonsene, but may Richard has
here a deeper insight.
I know an algo change would probably be better or a way to detect move loops to prevent this from
occurring, but this solution does work on all the devices that were failing manufacture tests
previously.
bye,
Heiko
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-52 Fax: +49-8142-66989-80 Email: h...@denx.de
_______________________________________________
U-Boot mailing list
U-Boot@lists.denx.de
https://lists.denx.de/listinfo/u-boot
