Bug#870185: armel/marvell kernel size

[Roger Shimizu]

On Sun, Aug 27, 2017 at 1:31 AM, Roger Shimizu  wrote:
> On Sun, Aug 27, 2017 at 1:27 AM, Roger Shimizu  wrote:
>> On Tue, Aug 22, 2017 at 2:55 AM, Ben Hutchings  wrote:
>>> For the size check, you're mostly duplicating the existing check_size()
>>> function.  It would be preferable to have a single function with some
>>> extra parameters so that it can do both compressed and uncompressed
>>> size checks.
>
> Forgot to give out result of my test run.
>
> $ debian/bin/buildcheck.py debian/build/build_armel_none_marvell armel
> none marvell
> Can't read ABI reference.  ABI not checked!
> Image size 2084434/2097080, using 99.40%.  Under 1% space in
> UNRELEASED.  Continuing.
> Uncompressed Image size 5934896/7340032, using 80.86%.  Uncompressed
> Image fits.  Continuing.

I changed a bit by adding more error checking, and pushed to the same branch.
now its hash code is cc7783a.

Another question, the main fix for the bug, which is the commit below,
is pushed to master.
Can I cherry-pick this commit to branch sid? Thanks!


commit d0704122dd83e5b4258184a0e92dc57934477c71
Author: Roger Shimizu 
AuthorDate: Tue Aug 22 01:01:21 2017 +0900
Commit: Roger Shimizu 
CommitDate: Tue Aug 22 01:01:21 2017 +0900

[armel] Disable CONFIG_STRICT_KERNEL_RWX

This will save about 3MB on linux Image (before compression).
u-boot loader on some armel devices has limitation that:
 - kernel image size <= 2MB (already described in
debian/config/armel/defines)
 - kernel size after decompression <= 7MB


Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

Bug#870185: armel/marvell kernel size

[Ben Hutchings]

On Mon, 2017-08-28 at 00:34 +0900, Roger Shimizu wrote:
> On Sun, Aug 27, 2017 at 1:31 AM, Roger Shimizu  wrote:
> > On Sun, Aug 27, 2017 at 1:27 AM, Roger Shimizu  
> > wrote:
> > > On Tue, Aug 22, 2017 at 2:55 AM, Ben Hutchings  
> > > wrote:
> > > > For the size check, you're mostly duplicating the existing check_size()
> > > > function.  It would be preferable to have a single function with some
> > > > extra parameters so that it can do both compressed and uncompressed
> > > > size checks.
> > 
> > Forgot to give out result of my test run.
> > 
> > $ debian/bin/buildcheck.py debian/build/build_armel_none_marvell armel
> > none marvell
> > Can't read ABI reference.  ABI not checked!
> > Image size 2084434/2097080, using 99.40%.  Under 1% space in
> > UNRELEASED.  Continuing.
> > Uncompressed Image size 5934896/7340032, using 80.86%.  Uncompressed
> > Image fits.  Continuing.
> 
> I changed a bit by adding more error checking, and pushed to the same branch.
> now its hash code is cc7783a.
> 
> Another question, the main fix for the bug, which is the commit below,
> is pushed to master.
> Can I cherry-pick this commit to branch sid? Thanks!
[...]

Yes, please do.

Ben.

-- 
Ben Hutchings
Teamwork is essential - it allows you to blame someone else.



signature.asc
Description: This is a digitally signed message part

Bug#870185: armel/marvell kernel size

[Roger Shimizu]

On Sun, Aug 27, 2017 at 1:27 AM, Roger Shimizu  wrote:
> On Tue, Aug 22, 2017 at 2:55 AM, Ben Hutchings  wrote:
>> For the size check, you're mostly duplicating the existing check_size()
>> function.  It would be preferable to have a single function with some
>> extra parameters so that it can do both compressed and uncompressed
>> size checks.

Forgot to give out result of my test run.

$ debian/bin/buildcheck.py debian/build/build_armel_none_marvell armel
none marvell
Can't read ABI reference.  ABI not checked!
Image size 2084434/2097080, using 99.40%.  Under 1% space in
UNRELEASED.  Continuing.
Uncompressed Image size 5934896/7340032, using 80.86%.  Uncompressed
Image fits.  Continuing.

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

Bug#870185: armel/marvell kernel size

[Roger Shimizu]

On Tue, Aug 22, 2017 at 2:55 AM, Ben Hutchings  wrote:
> On Tue, 2017-08-22 at 01:22 +0900, Roger Shimizu wrote:
>
> Oh I see, that adds section (1 MiB) alignment in several places.
> Surprisingly, the padding isn't completely zero-filled, so it inflates
> the compressed image size too.

Good to know the details.
Pushed the fix to branch master.

>> Added another commit on branch rosh/check-size2, that check the
>> decompressed image size.
>> I confirmed it works. Please kindly review. Thank you!
>
> Please choose a more meaningful name than 'image2' e.g. 'uncompressed-
> image'.
>
> For the size check, you're mostly duplicating the existing check_size()
> function.  It would be preferable to have a single function with some
> extra parameters so that it can do both compressed and uncompressed
> size checks.

Thanks for your review!
Pushed updated commit to branch rosh/check-uncompressed-size.

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

Bug#870185: armel/marvell kernel size

[Ben Hutchings]

On Tue, 2017-08-22 at 01:22 +0900, Roger Shimizu wrote:
> Control: tag -1 +pending
> 
> On Mon, Aug 21, 2017 at 9:28 AM, Roger Shimizu  wrote:
> > On Mon, Aug 21, 2017 at 12:43 AM, Ben Hutchings  
> > wrote:
> > > 
> > > OK, then try using 4.11.6 source and bisecting the Debian config
> > > changes.
> > 
> > Result:
> > 0905519af414d339f615d7aac974f9a9163cdbd3 is the first bad commit
> 
> So I confirmed the only problem is CONFIG_STRICT_KERNEL_RWX
> Added a commit to disable STRICT_KERNEL_RWX for armel.
> After cherry-pick this commit to 4.11.6-1 and 4.12.6-1, I confirmed
> the kernel boot again on my linkstation device.

Oh I see, that adds section (1 MiB) alignment in several places. 
Surprisingly, the padding isn't completely zero-filled, so it inflates
the compressed image size too.

> Added another commit on branch rosh/check-size2, that check the
> decompressed image size.
> I confirmed it works. Please kindly review. Thank you!

Please choose a more meaningful name than 'image2' e.g. 'uncompressed-
image'.

For the size check, you're mostly duplicating the existing check_size()
function.  It would be preferable to have a single function with some
extra parameters so that it can do both compressed and uncompressed
size checks.

Ben.

-- 
Ben Hutchings
Make three consecutive correct guesses and you will be considered an
expert.



signature.asc
Description: This is a digitally signed message part

Processed: Re: Bug#870185: armel/marvell kernel size

[Debian Bug Tracking System]

Processing control commands:

> tag -1 +pending
Bug #870185 [linux-image-4.11.0-0.bpo.1-marvell] FATAL: kernel 
4.11.0-0.bpo.1-marvell does not boot on QNAP TS-219P II
Added tag(s) pending.

-- 
870185: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=870185
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Bug#870185: armel/marvell kernel size

[Roger Shimizu]

Control: tag -1 +pending

On Mon, Aug 21, 2017 at 9:28 AM, Roger Shimizu  wrote:
> On Mon, Aug 21, 2017 at 12:43 AM, Ben Hutchings  wrote:
>>
>> OK, then try using 4.11.6 source and bisecting the Debian config
>> changes.
>
> Result:
> 0905519af414d339f615d7aac974f9a9163cdbd3 is the first bad commit

So I confirmed the only problem is CONFIG_STRICT_KERNEL_RWX
Added a commit to disable STRICT_KERNEL_RWX for armel.
After cherry-pick this commit to 4.11.6-1 and 4.12.6-1, I confirmed
the kernel boot again on my linkstation device.

Added another commit on branch rosh/check-size2, that check the
decompressed image size.
I confirmed it works. Please kindly review. Thank you!

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

Bug#870185: armel/marvell kernel size

[Roger Shimizu]

On Mon, Aug 21, 2017 at 12:43 AM, Ben Hutchings  wrote:
>
> OK, then try using 4.11.6 source and bisecting the Debian config
> changes.

Result:
0905519af414d339f615d7aac974f9a9163cdbd3 is the first bad commit

Here's the detailed log.

$ git bisect log
git bisect start
# good: [327c328b5435d93c2d3294059882629a6f8f1861] Update to 4.11-rc7
(and credit Lukas for his previous work)
git bisect good 327c328b5435d93c2d3294059882629a6f8f1861
# bad: [aca58f89537f4c470cc1e9102c1249e39a4b13a3] Prepare to release
linux (4.11.3-1~exp1).
git bisect bad aca58f89537f4c470cc1e9102c1249e39a4b13a3
# good: [3f68b53e2c3d425af23d0fbc33ef1e5f932484ab] usbip: Enable
USBIP_VUDC as module
git bisect good 3f68b53e2c3d425af23d0fbc33ef1e5f932484ab
# bad: [0a348a06ca2392fe337580ba007f325bdb111ff1] [armel] udeb: Add
af_packet. firmware_class, nls_base to kernel-image (fixes FTBFS)
git bisect bad 0a348a06ca2392fe337580ba007f325bdb111ff1
# good: [472b038c1a71a837c1cd30d49f6cda385e9dd540] udeb: Move mfd-core
to kernel-image, as both input-modules and mmc-modules need it
git bisect good 472b038c1a71a837c1cd30d49f6cda385e9dd540
# bad: [2d277e1fa20d2fe1fc5414f3806fded57bf9200b] Update to 4.11
git bisect bad 2d277e1fa20d2fe1fc5414f3806fded57bf9200b
# bad: [0905519af414d339f615d7aac974f9a9163cdbd3] Clean up kconfig
using kconfigeditor2
git bisect bad 0905519af414d339f615d7aac974f9a9163cdbd3
# good: [0e0b29ad5ab36d7268f8a50fb5a5c90cea4186c9] [arm64,x86] Replace
securelevel patch set with lockdown patch set
git bisect good 0e0b29ad5ab36d7268f8a50fb5a5c90cea4186c9
# good: [259372e240d0021eb2acdc98bd61b6adb84d645e] [x86] Make
hyperv-modules depends on nic-shared-modules
git bisect good 259372e240d0021eb2acdc98bd61b6adb84d645e
# first bad commit: [0905519af414d339f615d7aac974f9a9163cdbd3] Clean
up kconfig using kconfigeditor2

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

Bug#870185: armel/marvell kernel size

[Ben Hutchings]

On Sun, 2017-08-20 at 21:16 +0900, Roger Shimizu wrote:
> On Sun, Aug 20, 2017 at 6:54 AM, Ben Hutchings 
> wrote:
> > On Sun, 2017-08-20 at 02:35 +0900, Roger Shimizu wrote:
> > > 
> > > The real problem is kernel size (after decompression) increased
> > > from
> > > 5MB to 8MB. (detail is in my previous post)
> > > This looks like a bug, since usually kernel size grows gradually,
> > > not
> > > in +3MB way this time.
> > 
> > Some things to check:
> > 
> > - If you build 4.10 with the config used for 4.11 (whichever is the
> > first larger one), is the size 5 MB or 8 MB?
> 
> I build vanilla kernel with debian config, and here's the result.
> (images are under arch/arm/boot/)
> 
> linux_4.10.7 with debian 4.10.7 config
> -rwxr-xr-x 1 rosh rosh 5757504 Aug 20 06:40 Image
> -rwxr-xr-x 1 rosh rosh 2012496 Aug 20 06:40 zImage
> 
> linux_4.11.6 with debian 4.11.6 config
> -rwxr-xr-x 1 rosh rosh 8763712 Aug 20 07:03 Image
> -rwxr-xr-x 1 rosh rosh 2064576 Aug 20 07:03 zImage
> 
> linux_4.11.6 with debian 4.10.7 config
> -rwxr-xr-x 1 rosh rosh 5891216 Aug 20 07:43 Image
> -rwxr-xr-x 1 rosh rosh 2049216 Aug 20 07:43 zImage
> 
> linux_4.10.7 with debian 4.11.6 config
> -rwxr-xr-x 1 rosh rosh 5811408 Aug 20 07:56 Image
> -rwxr-xr-x 1 rosh rosh 2032704 Aug 20 07:56 zImage
> 
> So it seems the problem is pointing to config changes since 4.11.

OK, then try using 4.11.6 source and bisecting the Debian config
changes.

Ben.

-- 
Ben Hutchings
It is a miracle that curiosity survives formal education. - Albert
Einstein



signature.asc
Description: This is a digitally signed message part

Bug#870185: armel/marvell kernel size

[Roger Shimizu]

On Sun, Aug 20, 2017 at 6:54 AM, Ben Hutchings  wrote:
> On Sun, 2017-08-20 at 02:35 +0900, Roger Shimizu wrote:
>>
>> The real problem is kernel size (after decompression) increased from
>> 5MB to 8MB. (detail is in my previous post)
>> This looks like a bug, since usually kernel size grows gradually, not
>> in +3MB way this time.
>
> Some things to check:
>
> - If you build 4.10 with the config used for 4.11 (whichever is the
> first larger one), is the size 5 MB or 8 MB?

I build vanilla kernel with debian config, and here's the result.
(images are under arch/arm/boot/)

linux_4.10.7 with debian 4.10.7 config
-rwxr-xr-x 1 rosh rosh 5757504 Aug 20 06:40 Image
-rwxr-xr-x 1 rosh rosh 2012496 Aug 20 06:40 zImage

linux_4.11.6 with debian 4.11.6 config
-rwxr-xr-x 1 rosh rosh 8763712 Aug 20 07:03 Image
-rwxr-xr-x 1 rosh rosh 2064576 Aug 20 07:03 zImage

linux_4.11.6 with debian 4.10.7 config
-rwxr-xr-x 1 rosh rosh 5891216 Aug 20 07:43 Image
-rwxr-xr-x 1 rosh rosh 2049216 Aug 20 07:43 zImage

linux_4.10.7 with debian 4.11.6 config
-rwxr-xr-x 1 rosh rosh 5811408 Aug 20 07:56 Image
-rwxr-xr-x 1 rosh rosh 2032704 Aug 20 07:56 zImage

So it seems the problem is pointing to config changes since 4.11.

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

Re: Bug#870185: armel/marvell kernel size

[Ben Hutchings]

On Sun, 2017-08-20 at 02:35 +0900, Roger Shimizu wrote:
> On Sun, Aug 20, 2017 at 12:55 AM, Ian Campbell 
> wrote:
> > On Sat, 2017-08-19 at 12:57 +0900, Roger Shimizu wrote:
> > > I know for bug #870185, Robert fixed his device by modify uboot
> > > params, but I guess it's still possible to keep uboot params and
> > > only
> > > change the boot addresses of kernel/initrd in flash-kernel db
> > > file.
> > 
> > In https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=870185#35 I
> > concluded it wasn't, perhaps I'm wrong though.
> 
> You may read my previous research:
> - https://bugs.debian.org/809476#49
> 
> > Can someone confirm whether the issue is that the u-boot load
> > addresses
> >  for kernel and initrd are conflicting or if it is the kernel's
> > target/decompression address and u-boot's initrd address which are
> > conflicting? The symptoms seem to suggest the kernel is being
> > decompressed over the initrd.
> 
> The real problem is kernel size (after decompression) increased from
> 5MB to 8MB. (detail is in my previous post)
> This looks like a bug, since usually kernel size grows gradually, not
> in +3MB way this time.

Some things to check:

- If you build 4.10 with the config used for 4.11 (whichever is the
first larger one), is the size 5 MB or 8 MB?

- If not, try bisecting the upstream source to find a jump in size.

Ben.

-- 
Ben Hutchings
The generation of random numbers is too important to be left to chance.
   - Robert Coveyou



signature.asc
Description: This is a digitally signed message part

Bug#870185: armel/marvell kernel size

[Roger Shimizu]

On Sun, Aug 20, 2017 at 12:55 AM, Ian Campbell  wrote:
> On Sat, 2017-08-19 at 12:57 +0900, Roger Shimizu wrote:
>> I know for bug #870185, Robert fixed his device by modify uboot
>> params, but I guess it's still possible to keep uboot params and only
>> change the boot addresses of kernel/initrd in flash-kernel db file.
>
> In https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=870185#35 I
> concluded it wasn't, perhaps I'm wrong though.

You may read my previous research:
- https://bugs.debian.org/809476#49

> Can someone confirm whether the issue is that the u-boot load addresses
>  for kernel and initrd are conflicting or if it is the kernel's
> target/decompression address and u-boot's initrd address which are
> conflicting? The symptoms seem to suggest the kernel is being
> decompressed over the initrd.

The real problem is kernel size (after decompression) increased from
5MB to 8MB. (detail is in my previous post)
This looks like a bug, since usually kernel size grows gradually, not
in +3MB way this time.

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

Re: Bug#870185: armel/marvell kernel size

[Ian Campbell]

On Sat, 2017-08-19 at 12:57 +0900, Roger Shimizu wrote:
> I know for bug #870185, Robert fixed his device by modify uboot
> params, but I guess it's still possible to keep uboot params and only
> change the boot addresses of kernel/initrd in flash-kernel db file.

In https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=870185#35 I
concluded it wasn't, perhaps I'm wrong though.

Can someone confirm whether the issue is that the u-boot load addresses
 for kernel and initrd are conflicting or if it is the kernel's
target/decompression address and u-boot's initrd address which are
conflicting? The symptoms seem to suggest the kernel is being
decompressed over the initrd.

Since we don't want to change the u-boot we can't easily influence the
initrd address (since it doesn't use a u-boot header on this platform).
So it would seem the only possibility (other than shrinking the kernel)
would be to change the address at which it decompresses itself so it
doesn't conflict with the initrd.

Not sure how hard that would be -- it might be as simple as tweaking a
constant in the source or a Kconfig option, but I suspect it might
involve changing early boot assembly...

Ian.

Bug#870185: armel/marvell kernel size

[Roger Shimizu]

On Mon, Jul 31, 2017 at 11:43 PM, Roger Shimizu  wrote:
> On Mon, Jul 31, 2017 at 3:05 AM, Roger Shimizu  wrote:
>>
>> While I'm still working on this, but I find the latest kernel in
>> archive, 4.11.11-1+b1, fails to boot on my kirkwood based Linkstation.
>> I tried netconsole, but don't get any log. (netconsole outputs fine on
>> working kernels, such as 4.9 and 4.10 series.)
>> And I trace to latest working kernel is 4.10.7-1~exp1.
>>
>> 4.11-1_exp[12] FTBFS on armel, and 4.11.3-1_exp1 doesn't boot.
>>
>> I checked d-kernel and d-arm list, but didn't find similar issue.
>> Do you have any clue on this? Thanks!
>
> I think my problem is very likely the same as:
>  - https://bugs.debian.org/870185
>
> And I tried Ben's recommendation to make armel kernel smaller,
> which I pushed to branch rosh/strip_armel on alioth, confirmed it worked
> well for 4.10.7-1~exp1 (by cherry-pick) on my kirkwood based Linkstation.

I find some new clue about kernel size issue for armel.
vmlinuz installed under /boot is compressed. It can be decompressed by
"scripts/extract-vmlinux" script under linux git repo. But for armel,
it need at patch I posted:
 - https://patchwork.kernel.org/patch/8120831/

After I decompress the kernel images, I find the size for 4.11
increased significantly:


$ ~/linux/scripts/extract-vmlinux /boot/vmlinuz-4.9.0-3-marvell >
~/vmlinuz-4.9.0-3-marvell.extracted
$ ~/linux/scripts/extract-vmlinux /boot/vmlinuz-4.10.0-trunk-marvell >
~/vmlinuz-4.10.0-trunk-marvell.extracted
$ ~/linux/scripts/extract-vmlinux /boot/vmlinuz-4.11.0-0.bpo.1-marvell
> ~/vmlinuz-4.11.0-0.bpo.1-marvell.extracted

$ ls -l ~/vmlinuz-*
-rw-r--r-- 1 rosh rosh 5782024 Aug 19 12:35
vmlinuz-4.10.0-trunk-marvell.extracted
-rw-r--r-- 1 rosh rosh 8763712 Aug 19 12:36
vmlinuz-4.11.0-0.bpo.1-marvell.extracted
-rw-r--r-- 1 rosh rosh 5844072 Aug 19 12:35 vmlinuz-4.9.0-3-marvell.extracted


I know for bug #870185, Robert fixed his device by modify uboot
params, but I guess it's still possible to keep uboot params and only
change the boot addresses of kernel/initrd in flash-kernel db file.

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1