Re: [OE-core] wic creates ext4 images that read really slow in u-boot
On Wed, Feb 20, 2019 at 10:55:20AM +, Mike Looijmans wrote: > >>> I am aware of two fixes for U-Boot. I will look them up, and reply again > >>> to this thread. > > Browsing through u-boot commits, I found this gem that's disabled by default: > > CONFIG_BLOCK_CACHE=y > > The commit is this one: > > commit e40cf34a29f1b248643731a11fb1c6f0520d016c > Author: Eric Nelson > Date: Mon Mar 28 10:05:44 2016 -0700 > > drivers: block: add block device cache > > Add a block device cache to speed up repeated reads of block devices by > various filesystems. For the record, it's not that it's disabled by default but rather to enable it, you need to enable other "new" frameworks and so forth. And at least for "legacy" platforms, there's a certain level of pushback against doing the work required to enable that (as to be honest, it is more than just enable a few CONFIG options and re-test). A counter-pushback of "but it enables things that fix some bad performance problems!" would be really helpful. -- Tom signature.asc Description: PGP signature -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] wic creates ext4 images that read really slow in u-boot
On Thu, 21 Feb 2019 at 10:17, Mike Looijmans wrote: > That'd be best. One would expect that creating a filesystem "offline" would > yield something optimal in terms of the way files are allocated, since > everything about all files is already known when it's written. But because we don't run image generation as root in a loopback mount, we're limited to how clever e2fsprogs's debugfs is, *not* the kernel driver. Just a few weeks ago I fixed a bug in debugfs where it was only writing the first 2^31 bits of a file... Ross -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] wic creates ext4 images that read really slow in u-boot
On 19-02-19 21:45, Leon Woestenberg wrote: > Hello all, > > On Tue, Feb 19, 2019 at 8:28 PM Andre McCurdy wrote: >> On Tue, Feb 19, 2019 at 9:13 AM Leon Woestenberg wrote: >>> >>> Hello Mike, >>> >>> sounds familiar. >>> >>> On Tue, 19 Feb 2019 at 17:55, Mike Looijmans >>> wrote: Took me a while to figure out why my board took 90 seconds to boot suddenly. The issue turned out to be the ext4 partition created by wic. >>> >>> I suspect it's not WIC's fault. ZynqMP> load mmc 0:2 0x10 /lib/firmware/fpga.bin 19311092 bytes read in 85529 ms (219.7 KiB/s) Now if I boot the board rename and copy that file onto itself, then it's suddenly normal again when I reboot the board: ZynqMP> load mmc 0:2 0x1 I'm not knowledgeable on ext4, so any ideas what's being passed onto the image creation tool that causes this? >>> >>> I suspect your version of U-Boot does not handle files spread across >>> multiple filesystems (allocation) extends efficiently. >>> >>> Copying the file makes the copy being layed out in one extend probably. >> >> >> If WIC is generating filesystem images from scratch there's no excuse for >> files to be unnecessarily fragmented. >> >> Even if some of all of the boot time can be recovered by a patch to u-boot >> that won't help systems which have already been deployed and don't have a >> way to update the bootloader. >> > I am not sure if "fragmented" is the right term in terms of filesystem > details. Filesystem allocation extents (not "extends" as I mistyped > earlier) are different from heavy file fragmentation. However, I agree > things can be made more optimal. > > So, with adding the above, there are *two* issues at play here: > 1) Most/older versions of U-Boot not able to efficiently load files > from ext4 filesystems, that cross multiple extents. I am aware of two > fixes, see below. > 2) WIC uses mkext4fs in such a way that files can cross multiple > (allocation) extents. This is sub-optimal and know to cause issues > with many U-Boot versions (deployed on existing systems in the field). > The problems shows "randomly" with large files being deployed to the > ext4 filesystem. We (OE/Yocto) may want to fix this. That'd be best. One would expect that creating a filesystem "offline" would yield something optimal in terms of the way files are allocated, since everything about all files is already known when it's written. My current workaround is to just mount the device and unpack the rootfs tar onto that, which yields a filesystem that u-boot can read at normal speed. >>> I am aware of two fixes for U-Boot. I will look them up, and reply again to >>> this thread. > > Google for these two commits. I cherry-picked the first one for my > project and can confirm it brings back the desired performance. (Still > assuming this is the identical cause as for TO/Mike.) > > commit 855b8e4f9f255415a7753819e392e4b5d984f35c > Author: Matt Madison > Date: Sat Aug 19 08:46:46 2017 -0700 > > ext4: cache extent blocks during file reads > > A simpler and less-efficient approach to solving > the problem with extent index blocks than what > was in fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5, > but one that does not break write operations. > > Signed-off-by: Matt Madison I'll give it a try... You mentioned "two" commits? -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] wic creates ext4 images that read really slow in u-boot
>>> I am aware of two fixes for U-Boot. I will look them up, and reply again to >>> this thread. Browsing through u-boot commits, I found this gem that's disabled by default: CONFIG_BLOCK_CACHE=y The commit is this one: commit e40cf34a29f1b248643731a11fb1c6f0520d016c Author: Eric Nelson Date: Mon Mar 28 10:05:44 2016 -0700 drivers: block: add block device cache Add a block device cache to speed up repeated reads of block devices by various filesystems. This small amount of cache can dramatically speed up filesystem operations by skipping repeated reads of common areas of a block device (typically directory structures). This has shown to have some benefit on FAT filesystem operations of loading a kernel and RAM disk, but more dramatic benefits on ext4 filesystems when the kernel and/or RAM disk are spread across multiple extent header structures as described in commit fc0fc50. ... I'll be giving that a test drive soon... -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] wic creates ext4 images that read really slow in u-boot
Hello Mike, all, On Wed, Feb 20, 2019 at 11:42 AM Mike Looijmans wrote: > > On 19-02-19 21:45, Leon Woestenberg wrote: > > Hello all, > > > > On Tue, Feb 19, 2019 at 8:28 PM Andre McCurdy wrote: > >> On Tue, Feb 19, 2019 at 9:13 AM Leon Woestenberg > >> wrote: > >>> > >>> Hello Mike, > >>> > >>> sounds familiar. > >>> > >>> On Tue, 19 Feb 2019 at 17:55, Mike Looijmans > >>> wrote: > > Took me a while to figure out why my board took 90 seconds to boot > suddenly. > The issue turned out to be the ext4 partition created by wic. > >>> > >>> I suspect it's not WIC's fault. > > >>> I am aware of two fixes for U-Boot. I will look them up, and reply again > >>> to this thread. > > > > Google for these two commits. I cherry-picked the first one for my > > project and can confirm it brings back the desired performance. (Still > > assuming this is the identical cause as for TO/Mike.) > > > > commit 855b8e4f9f255415a7753819e392e4b5d984f35c > > Author: Matt Madison > > Date: Sat Aug 19 08:46:46 2017 -0700 > > > > ext4: cache extent blocks during file reads > > > > A simpler and less-efficient approach to solving > > the problem with extent index blocks than what > > was in fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5, > > but one that does not break write operations. > > > > Signed-off-by: Matt Madison > > I'll give it a try... > > You mentioned "two" commits? Yes the commit above, and *another* approach referred to and mentioned in the commit message (also with hash). The block cache feature may solve the problem in a more generic way (also caching things like directory lookups), whereas the ext4 extent patches are more specific to ext4 only. I found the ext4 fixes starting from here: https://github.com/madisongh/meta-tegra/issues/42 Regards, Leon. -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] wic creates ext4 images that read really slow in u-boot
On 19/02/2019 20:45, Leon Woestenberg wrote: > Hello all, > > On Tue, Feb 19, 2019 at 8:28 PM Andre McCurdy wrote: >> On Tue, Feb 19, 2019 at 9:13 AM Leon Woestenberg wrote: >>> >>> Hello Mike, >>> >>> sounds familiar. >>> >>> On Tue, 19 Feb 2019 at 17:55, Mike Looijmans >>> wrote: Took me a while to figure out why my board took 90 seconds to boot suddenly. The issue turned out to be the ext4 partition created by wic. >>> >>> I suspect it's not WIC's fault. ZynqMP> load mmc 0:2 0x10 /lib/firmware/fpga.bin 19311092 bytes read in 85529 ms (219.7 KiB/s) Now if I boot the board rename and copy that file onto itself, then it's suddenly normal again when I reboot the board: ZynqMP> load mmc 0:2 0x1 I'm not knowledgeable on ext4, so any ideas what's being passed onto the image creation tool that causes this? >>> >>> I suspect your version of U-Boot does not handle files spread across >>> multiple filesystems (allocation) extends efficiently. >>> >>> Copying the file makes the copy being layed out in one extend probably. >> >> >> If WIC is generating filesystem images from scratch there's no excuse for >> files to be unnecessarily fragmented. >> >> Even if some of all of the boot time can be recovered by a patch to u-boot >> that won't help systems which have already been deployed and don't have a >> way to update the bootloader. >> > I am not sure if "fragmented" is the right term in terms of filesystem > details. Filesystem allocation extents (not "extends" as I mistyped > earlier) are different from heavy file fragmentation. However, I agree > things can be made more optimal. > > So, with adding the above, there are *two* issues at play here: > 1) Most/older versions of U-Boot not able to efficiently load files > from ext4 filesystems, that cross multiple extents. I am aware of two > fixes, see below. > 2) WIC uses mkext4fs in such a way that files can cross multiple > (allocation) extents. This is sub-optimal and know to cause issues > with many U-Boot versions (deployed on existing systems in the field). > The problems shows "randomly" with large files being deployed to the > ext4 filesystem. We (OE/Yocto) may want to fix this. > >>> I am aware of two fixes for U-Boot. I will look them up, and reply again to >>> this thread. > > Google for these two commits. I cherry-picked the first one for my > project and can confirm it brings back the desired performance. (Still > assuming this is the identical cause as for TO/Mike.) > > commit 855b8e4f9f255415a7753819e392e4b5d984f35c > Author: Matt Madison > Date: Sat Aug 19 08:46:46 2017 -0700 > > ext4: cache extent blocks during file reads > > A simpler and less-efficient approach to solving > the problem with extent index blocks than what > was in fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5, > but one that does not break write operations. > > Signed-off-by: Matt Madison > > Regards, > > Leon. > > p.s. excuse the earlier HTML mail with signature > I can concur that I have also used this patch to fix speed issues with ext4 reads. The problem becomes particularly prominent when you have large files on a small partition which get over-written. I'm sure I have seen a patch go into uboot mainline recently which improved performance but I can't find it or it never actually made it into the tree. -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] wic creates ext4 images that read really slow in u-boot
Hello all, On Tue, Feb 19, 2019 at 8:28 PM Andre McCurdy wrote: > On Tue, Feb 19, 2019 at 9:13 AM Leon Woestenberg wrote: >> >> Hello Mike, >> >> sounds familiar. >> >> On Tue, 19 Feb 2019 at 17:55, Mike Looijmans wrote: >>> >>> Took me a while to figure out why my board took 90 seconds to boot suddenly. >>> The issue turned out to be the ext4 partition created by wic. >> >> I suspect it's not WIC's fault. >>> >>> ZynqMP> load mmc 0:2 0x10 /lib/firmware/fpga.bin >>> 19311092 bytes read in 85529 ms (219.7 KiB/s) >>> >>> Now if I boot the board rename and copy that file onto itself, then it's >>> suddenly normal again when I reboot the board: >>> >>> ZynqMP> load mmc 0:2 0x1 >>> I'm not knowledgeable on ext4, so any ideas what's being passed onto the >>> image >>> creation tool that causes this? >> >> I suspect your version of U-Boot does not handle files spread across >> multiple filesystems (allocation) extends efficiently. >> >> Copying the file makes the copy being layed out in one extend probably. > > > If WIC is generating filesystem images from scratch there's no excuse for > files to be unnecessarily fragmented. > > Even if some of all of the boot time can be recovered by a patch to u-boot > that won't help systems which have already been deployed and don't have a way > to update the bootloader. > I am not sure if "fragmented" is the right term in terms of filesystem details. Filesystem allocation extents (not "extends" as I mistyped earlier) are different from heavy file fragmentation. However, I agree things can be made more optimal. So, with adding the above, there are *two* issues at play here: 1) Most/older versions of U-Boot not able to efficiently load files from ext4 filesystems, that cross multiple extents. I am aware of two fixes, see below. 2) WIC uses mkext4fs in such a way that files can cross multiple (allocation) extents. This is sub-optimal and know to cause issues with many U-Boot versions (deployed on existing systems in the field). The problems shows "randomly" with large files being deployed to the ext4 filesystem. We (OE/Yocto) may want to fix this. >> I am aware of two fixes for U-Boot. I will look them up, and reply again to >> this thread. Google for these two commits. I cherry-picked the first one for my project and can confirm it brings back the desired performance. (Still assuming this is the identical cause as for TO/Mike.) commit 855b8e4f9f255415a7753819e392e4b5d984f35c Author: Matt Madison Date: Sat Aug 19 08:46:46 2017 -0700 ext4: cache extent blocks during file reads A simpler and less-efficient approach to solving the problem with extent index blocks than what was in fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5, but one that does not break write operations. Signed-off-by: Matt Madison Regards, Leon. p.s. excuse the earlier HTML mail with signature -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] wic creates ext4 images that read really slow in u-boot
On Tue, Feb 19, 2019 at 9:13 AM Leon Woestenberg wrote: > Hello Mike, > > sounds familiar. > > On Tue, 19 Feb 2019 at 17:55, Mike Looijmans > wrote: > >> Took me a while to figure out why my board took 90 seconds to boot >> suddenly. >> >> The issue turned out to be the ext4 partition created by wic. > > > I suspect it's not WIC's fault. > > >> ZynqMP> load mmc 0:2 0x10 /lib/firmware/fpga.bin >> 19311092 bytes read in 85529 ms (219.7 KiB/s) >> >> Now if I boot the board rename and copy that file onto itself, then it's >> suddenly normal again when I reboot the board: >> >> ZynqMP> load mmc 0:2 0x1 >> I'm not knowledgeable on ext4, so any ideas what's being passed onto the >> image >> creation tool that causes this? > > > I suspect your version of U-Boot does not handle files spread across > multiple filesystems (allocation) extends efficiently. > > Copying the file makes the copy being layed out in one extend probably. > If WIC is generating filesystem images from scratch there's no excuse for files to be unnecessarily fragmented. Even if some of all of the boot time can be recovered by a patch to u-boot that won't help systems which have already been deployed and don't have a way to update the bootloader. I am aware of two fixes for U-Boot. I will look them up, and reply again to > this thread. > > Regards, Leon > -- > Leon Woestenberg > l...@sidebranch.com > T: +31 40 711 42 76 > M: +31 6 472 30 372 > > Sidebranch > Embedded Systems > Eindhoven, The Netherlands > http://www.sidebranch.com > -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] wic creates ext4 images that read really slow in u-boot
Hello Mike, sounds familiar. On Tue, 19 Feb 2019 at 17:55, Mike Looijmans wrote: > Took me a while to figure out why my board took 90 seconds to boot > suddenly. > > The issue turned out to be the ext4 partition created by wic. I suspect it's not WIC's fault. > ZynqMP> load mmc 0:2 0x10 /lib/firmware/fpga.bin > 19311092 bytes read in 85529 ms (219.7 KiB/s) > > Now if I boot the board rename and copy that file onto itself, then it's > suddenly normal again when I reboot the board: > > ZynqMP> load mmc 0:2 0x1 > I'm not knowledgeable on ext4, so any ideas what's being passed onto the > image > creation tool that causes this? I suspect your version of U-Boot does not handle files spread across multiple filesystems (allocation) extends efficiently. Copying the file makes the copy being layed out in one extend probably. I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread. Regards, Leon -- Leon Woestenberg l...@sidebranch.com T: +31 40 711 42 76 M: +31 6 472 30 372 Sidebranch Embedded Systems Eindhoven, The Netherlands http://www.sidebranch.com -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
