Re: [PATCH 5/6] powerpc/boot: add xz support to the wrapper script

[Michael Ellerman]

Oliver O'Halloran  writes:

> This modifies the script so that the -Z option takes an argument to
> specify the compression type. It can either be 'gz', 'xz' or 'none'.
> The legazy --no-gzip and -z options are still supported and will set
> the compression to none and gzip respectively, but they are not
> documented.
>
> Only xz -6 is used for compression rather than xz -9. Using compression
> levels higher than 6 requires the decompressor to build a large (64MB)
> dictionary when decompressing and some environments cannot satisfy large
> allocations (e.g. POWER 6 LPAR partition firmware).

This isn't working for me on machines that use uImage.

That's the "uboot" case in wrapper, where we do:

${MKIMAGE} -A ppc -O linux -T kernel -C gzip -a $membase -e $membase \
$uboot_version -d "$vmz" "$ofile"

ie. we tell mkimage that we're using gzip compression, regardless of
whether we actually are.

That leads to something like:

  ## Booting kernel from Legacy Image at 0100 ...
 Image Name:   Linux-4.8.0-rc5-compiler_gcc-6.2
 Image Type:   PowerPC Linux Kernel Image (gzip compressed)
 Data Size:3381044 Bytes = 3.2 MiB
 Load Address: 
 Entry Point:  
 Verifying Checksum ... OK
  ## Flattened Device Tree blob at 0c00
 Booting using the fdt blob at 0xc00
 Uncompressing Kernel Image ... Error: Bad gzipped data
  gzip compressed: uncompress error -1
  Must RESET board to recover


So you'll need to do some juggling so you can pass the right argument
for -C to mkimage.

cheers

[PATCH 5/6] powerpc/boot: add xz support to the wrapper script

[Oliver O'Halloran]

This modifies the script so that the -Z option takes an argument to
specify the compression type. It can either be 'gz', 'xz' or 'none'.
The legazy --no-gzip and -z options are still supported and will set
the compression to none and gzip respectively, but they are not
documented.

Only xz -6 is used for compression rather than xz -9. Using compression
levels higher than 6 requires the decompressor to build a large (64MB)
dictionary when decompressing and some environments cannot satisfy large
allocations (e.g. POWER 6 LPAR partition firmware).

Signed-off-by: Oliver O'Halloran 
---
 arch/powerpc/boot/Makefile |  7 --
 arch/powerpc/boot/wrapper  | 61 ++
 2 files changed, 50 insertions(+), 18 deletions(-)

diff --git a/arch/powerpc/boot/Makefile b/arch/powerpc/boot/Makefile
index 861348c72519..9fb451d0586e 100644
--- a/arch/powerpc/boot/Makefile
+++ b/arch/powerpc/boot/Makefile
@@ -225,10 +225,13 @@ CROSSWRAP := -C "$(CROSS_COMPILE)"
 endif
 endif
 
+compressor-$(CONFIG_KERNEL_GZIP) := gz
+
 # args (to if_changed): 1 = (this rule), 2 = platform, 3 = dts 4=dtb 5=initrd
 quiet_cmd_wrap = WRAP$@
-  cmd_wrap =$(CONFIG_SHELL) $(wrapper) -c -o $@ -p $2 $(CROSSWRAP) \
-   $(if $3, -s $3)$(if $4, -d $4)$(if $5, -i $5) vmlinux
+  cmd_wrap =$(CONFIG_SHELL) $(wrapper) -Z $(compressor-y) -c -o $@ -p $2 \
+   $(CROSSWRAP) $(if $3, -s $3)$(if $4, -d $4)$(if $5, -i $5) \
+   vmlinux
 
 image-$(CONFIG_PPC_PSERIES)+= zImage.pseries
 image-$(CONFIG_PPC_POWERNV)+= zImage.pseries
diff --git a/arch/powerpc/boot/wrapper b/arch/powerpc/boot/wrapper
index 6681ec3625c9..6feacfd87588 100755
--- a/arch/powerpc/boot/wrapper
+++ b/arch/powerpc/boot/wrapper
@@ -20,6 +20,8 @@
 # -D dir   specify directory containing data files used by script
 #  (default ./arch/powerpc/boot)
 # -W dir   specify working directory for temporary files (default .)
+# -z   use gzip (legacy)
+# -Z zsuffixcompression to use (gz, xz or none)
 
 # Stop execution if any command fails
 set -e
@@ -38,7 +40,7 @@ dtb=
 dts=
 cacheit=
 binary=
-gzip=.gz
+compression=.gz
 pie=
 format=
 
@@ -59,7 +61,8 @@ tmpdir=.
 usage() {
 echo 'Usage: wrapper [-o output] [-p platform] [-i initrd]' >&2
 echo '   [-d devtree] [-s tree.dts] [-c] [-C cross-prefix]' >&2
-echo '   [-D datadir] [-W workingdir] [--no-gzip] [vmlinux]' >&2
+echo '   [-D datadir] [-W workingdir] [-Z (gz|xz|none)]' >&2
+echo '   [--no-compression] [vmlinux]' >&2
 exit 1
 }
 
@@ -126,8 +129,24 @@ while [ "$#" -gt 0 ]; do
[ "$#" -gt 0 ] || usage
tmpdir="$1"
;;
+-z)
+   compression=.gz
+   ;;
+-Z)
+   shift
+   [ "$#" -gt 0 ] || usage
+[ "$1" != "gz" -o "$1" != "xz" -o "$1" != "none" ] || usage
+
+   compression=".$1"
+
+if [ $compression = ".none" ]; then
+compression=
+fi
+   ;;
 --no-gzip)
-gzip=
+# a "feature" of the the wrapper script is that it can be used outside
+# the kernel tree. So keeping this around for backwards compatibility.
+compression=
 ;;
 -?)
usage
@@ -140,6 +159,7 @@ while [ "$#" -gt 0 ]; do
 shift
 done
 
+
 if [ -n "$dts" ]; then
 if [ ! -r "$dts" -a -r "$object/dts/$dts" ]; then
dts="$object/dts/$dts"
@@ -212,7 +232,7 @@ miboot|uboot*)
 ;;
 cuboot*)
 binary=y
-gzip=
+compression=
 case "$platform" in
 *-mpc866ads|*-mpc885ads|*-adder875*|*-ep88xc)
 platformo=$object/cuboot-8xx.o
@@ -243,7 +263,7 @@ cuboot*)
 ps3)
 platformo="$object/ps3-head.o $object/ps3-hvcall.o $object/ps3.o"
 lds=$object/zImage.ps3.lds
-gzip=
+compression=
 ext=bin
 objflags="-O binary --set-section-flags=.bss=contents,alloc,load,data"
 ksection=.kernel:vmlinux.bin
@@ -310,27 +330,37 @@ mvme7100)
 esac
 
 vmz="$tmpdir/`basename \"$kernel\"`.$ext"
-if [ -z "$cacheit" -o ! -f "$vmz$gzip" -o "$vmz$gzip" -ot "$kernel" ]; then
-${CROSS}objcopy $objflags "$kernel" "$vmz.$$"
 
-strip_size=$(stat -c %s $vmz.$$)
+# Calculate the vmlinux.strip size
+${CROSS}objcopy $objflags "$kernel" "$vmz.$$"
+strip_size=$(stat -c %s $vmz.$$)
 
-if [ -n "$gzip" ]; then
+if [ -z "$cacheit" -o ! -f "$vmz$compression" -o "$vmz$compression" -ot 
"$kernel" ]; then
+# recompress the image if we need to
+case $compression in
+.xz)
+xz --check=crc32 -f -6 "$vmz.$$"
+;;
+.gz)
 gzip -n -f -9 "$vmz.$$"
-fi
+;;
+*)
+# drop the compression suffix so the stripped vmlinux is used
+compression=
+   ;;
+esac
 
 if [ -n "$cacheit" ]; then
-   mv -f "$vmz.$$$gzip" "$vmz$gzip"
+   mv -f "$vmz.$$$compression" "$vmz$compression"
 else
vmz="$vmz.$$"
 fi
 else
-# Calculate the vmlinux.strip size
-${CROSS}objcopy 

[PATCH 5/6] powerpc/boot: add xz support to the wrapper script

[Oliver O'Halloran]

This modifies the script so that the -Z option takes an argument to
specify the compression type. It can either be 'gz', 'xz' or 'none'.
The legazy --no-gzip and -z options are still supported and will set
the compression to none and gzip respectively, but they are not
documented.

Signed-off-by: Oliver O'Halloran 
---
 arch/powerpc/boot/Makefile |  7 --
 arch/powerpc/boot/wrapper  | 61 ++
 2 files changed, 50 insertions(+), 18 deletions(-)

diff --git a/arch/powerpc/boot/Makefile b/arch/powerpc/boot/Makefile
index 3fdd74ac2fae..482bac2af1ff 100644
--- a/arch/powerpc/boot/Makefile
+++ b/arch/powerpc/boot/Makefile
@@ -212,10 +212,13 @@ CROSSWRAP := -C "$(CROSS_COMPILE)"
 endif
 endif
 
+compressor-$(CONFIG_KERNEL_GZIP) := gz
+
 # args (to if_changed): 1 = (this rule), 2 = platform, 3 = dts 4=dtb 5=initrd
 quiet_cmd_wrap = WRAP$@
-  cmd_wrap =$(CONFIG_SHELL) $(wrapper) -c -o $@ -p $2 $(CROSSWRAP) \
-   $(if $3, -s $3)$(if $4, -d $4)$(if $5, -i $5) vmlinux
+  cmd_wrap =$(CONFIG_SHELL) $(wrapper) -Z $(compressor-y) -c -o $@ -p $2 \
+   $(CROSSWRAP) $(if $3, -s $3)$(if $4, -d $4)$(if $5, -i $5) \
+   vmlinux
 
 image-$(CONFIG_PPC_PSERIES)+= zImage.pseries
 image-$(CONFIG_PPC_POWERNV)+= zImage.pseries
diff --git a/arch/powerpc/boot/wrapper b/arch/powerpc/boot/wrapper
index 6681ec3625c9..cf7631be5007 100755
--- a/arch/powerpc/boot/wrapper
+++ b/arch/powerpc/boot/wrapper
@@ -20,6 +20,8 @@
 # -D dir   specify directory containing data files used by script
 #  (default ./arch/powerpc/boot)
 # -W dir   specify working directory for temporary files (default .)
+# -z   use gzip (legacy)
+# -Z zsuffixcompression to use (gz, xz or none)
 
 # Stop execution if any command fails
 set -e
@@ -38,7 +40,7 @@ dtb=
 dts=
 cacheit=
 binary=
-gzip=.gz
+compression=.gz
 pie=
 format=
 
@@ -59,7 +61,8 @@ tmpdir=.
 usage() {
 echo 'Usage: wrapper [-o output] [-p platform] [-i initrd]' >&2
 echo '   [-d devtree] [-s tree.dts] [-c] [-C cross-prefix]' >&2
-echo '   [-D datadir] [-W workingdir] [--no-gzip] [vmlinux]' >&2
+echo '   [-D datadir] [-W workingdir] [-Z (gz|xz|none)]' >&2
+echo '   [--no-compression] [vmlinux]' >&2
 exit 1
 }
 
@@ -126,8 +129,24 @@ while [ "$#" -gt 0 ]; do
[ "$#" -gt 0 ] || usage
tmpdir="$1"
;;
+-z)
+   compression=.gz
+   ;;
+-Z)
+   shift
+   [ "$#" -gt 0 ] || usage
+[ "$1" != "gz" -o "$1" != "xz" -o "$1" != "none" ] || usage
+
+   compression=".$1"
+
+if [ $compression = ".none" ]; then
+compression=
+fi
+   ;;
 --no-gzip)
-gzip=
+# a "feature" of the the wrapper script is that it can be used outside
+# the kernel tree. So keeping this around for backwards compatibility.
+compression=
 ;;
 -?)
usage
@@ -140,6 +159,7 @@ while [ "$#" -gt 0 ]; do
 shift
 done
 
+
 if [ -n "$dts" ]; then
 if [ ! -r "$dts" -a -r "$object/dts/$dts" ]; then
dts="$object/dts/$dts"
@@ -212,7 +232,7 @@ miboot|uboot*)
 ;;
 cuboot*)
 binary=y
-gzip=
+compression=
 case "$platform" in
 *-mpc866ads|*-mpc885ads|*-adder875*|*-ep88xc)
 platformo=$object/cuboot-8xx.o
@@ -243,7 +263,7 @@ cuboot*)
 ps3)
 platformo="$object/ps3-head.o $object/ps3-hvcall.o $object/ps3.o"
 lds=$object/zImage.ps3.lds
-gzip=
+compression=
 ext=bin
 objflags="-O binary --set-section-flags=.bss=contents,alloc,load,data"
 ksection=.kernel:vmlinux.bin
@@ -310,27 +330,37 @@ mvme7100)
 esac
 
 vmz="$tmpdir/`basename \"$kernel\"`.$ext"
-if [ -z "$cacheit" -o ! -f "$vmz$gzip" -o "$vmz$gzip" -ot "$kernel" ]; then
-${CROSS}objcopy $objflags "$kernel" "$vmz.$$"
 
-strip_size=$(stat -c %s $vmz.$$)
+# Calculate the vmlinux.strip size
+${CROSS}objcopy $objflags "$kernel" "$vmz.$$"
+strip_size=$(stat -c %s $vmz.$$)
 
-if [ -n "$gzip" ]; then
+if [ -z "$cacheit" -o ! -f "$vmz$compression" -o "$vmz$compression" -ot 
"$kernel" ]; then
+# recompress the image if we need to
+case $compression in
+.xz)
+xz --check=crc32 -f -9 "$vmz.$$"
+;;
+.gz)
 gzip -n -f -9 "$vmz.$$"
-fi
+;;
+*)
+# drop the compression suffix so the stripped vmlinux is used
+compression=
+   ;;
+esac
 
 if [ -n "$cacheit" ]; then
-   mv -f "$vmz.$$$gzip" "$vmz$gzip"
+   mv -f "$vmz.$$$compression" "$vmz$compression"
 else
vmz="$vmz.$$"
 fi
 else
-# Calculate the vmlinux.strip size
-${CROSS}objcopy $objflags "$kernel" "$vmz.$$"
-strip_size=$(stat -c %s $vmz.$$)
 rm -f $vmz.$$
 fi
 
+vmz="$vmz$compression"
+
 if [ "$make_space" = "y" ]; then
# Round the size to next higher MB limit
round_size=$(((strip_size + 0xf) & 0xfff0))
@@