[Bug 1701132] Re: Please use xz instead of pxz
This bug was fixed in the package livecd-rootfs - 2.408.25 --- livecd-rootfs (2.408.25) xenial; urgency=medium * Sort and list dependencies nicely * Depend on packages instead of installing them at run time * Use xz -T4 instead of pxz. Supported xz versions before 5.2 accept the -T4 option but ignore it. Also depend on xz-utils instead of downloading pxz. (LP: #1701132) -- Balint ReczeyWed, 06 Dec 2017 11:34:04 +0100 ** Changed in: livecd-rootfs (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701132 Title: Please use xz instead of pxz To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1701132/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701132] Re: Please use xz instead of pxz
Verified by building cloud images using the proposed package and ensuring the resulting artifacts were satisfactory. ** Tags removed: verification-needed verification-needed-xenial ** Tags added: verification-done verification-done-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701132 Title: Please use xz instead of pxz To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1701132/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701132] Re: Please use xz instead of pxz
(verified with 2.408.25) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701132 Title: Please use xz instead of pxz To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1701132/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701132] Re: Please use xz instead of pxz
Hello Balint, or anyone else affected, Accepted livecd-rootfs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd- rootfs/2.408.25 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Tags added: verification-needed verification-needed-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701132 Title: Please use xz instead of pxz To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1701132/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701132] Re: Please use xz instead of pxz
** Changed in: livecd-rootfs (Ubuntu Xenial) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701132 Title: Please use xz instead of pxz To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1701132/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701132] Re: Please use xz instead of pxz
** Description changed: [Impact] - * Livecd-rootfs switches from using pxz to xz - * Pxz is installed at run-time instead of being a package dependency and pxz is in universe despite livecd-rootfs being in main. Both of those are undesired and fixing them makes livecd-rootfs easier to maintain following best practice and policy in Ubuntu. + * Livecd-rootfs switches from using pxz to xz + * Pxz is installed at run-time instead of being a package dependency and pxz is in universe despite livecd-rootfs being in main. Both of those are undesired and fixing them makes livecd-rootfs easier to maintain following best practice and policy in Ubuntu. [Test Case] - * Livecd-rootfs in Xenial uses (p)xz only for compressing raspi2 images. -To test the change install updated livecd-rootfs on armhf and create images setting the following environment variables: ARCH=armhf SUBARCH=raspi2 PROJECT=ubuntu-cpc. -Notice that the xz-compressed rapi2 image is still created. + * Livecd-rootfs in Xenial uses (p)xz only for compressing raspi2 images. + To test the change install updated livecd-rootfs on armhf and create images setting the following environment variables: ARCH=armhf SUBARCH=raspi2 PROJECT=ubuntu-cpc. + Notice that the xz-compressed rapi2 image is still created. + * Test-build Xenial image and make sure they are not broken. A subset of those images are built in the autopkgtest suite. [Regression Potential] - * It is a known and accepted regression that raspi2 image generation takes more wall-clock time after switching to xz because while Xenial's xz version does accept the -T4 option it does not compress in a parallel manner. - * The original change in Artful set -0 compression for xz which resulted in ~35% bigger compressed images. This regression is fixed in LP: #1732681 which is scheduled to be back-ported together with switching to xz. - * In case of this change causes regression it would manifest itself as failing build for raspi2, failing build for ubuntu-cpc images or increased raspi2 image size. - + * It is a known and accepted regression that raspi2 image generation takes more wall-clock time after switching to xz because while Xenial's xz version does accept the -T4 option it does not compress in a parallel manner. + * The original change in Artful set -0 compression for xz which resulted in ~35% bigger compressed images. This regression is fixed in LP: #1732681 which is scheduled to be back-ported together with switching to xz. + * In case of this change causes regression it would manifest itself as failing build for raspi2, failing build for ubuntu-cpc images or increased raspi2 image size. + [Original bug report text] Livecd-rootfs (main) 2.447 uses "pxz -T4" for xz compression, while pxz is not in main. Xz 5.2 also supports multi-threaded compression thus xz could easily replace current pxz usage. A quick test on compressing livecd.ubuntu-cpc.img (created by ubuntu- image) shows that there is very little difference in compressed file size while xz's -T4 brings substantial speedup in wall clock time: $ cat test-xz #!/bin/sh TESTFILE=$1 echo "Compr.\tThreads\tWall.t\tUser.t\tSys.t\tRes.t" > times.txt echo "Wall.t\tUser.t\tSys.t\tRes.t" > decompression-times.txt echo "Size(kB)" > sizes.txt for threads in 1 4; do for compression in $(seq 0 9) $(seq -f '%0.fe' 0 9); do /usr/bin/time -a -o times.txt -f "-$compression\t -T$threads\t%e\t%U\t%S\t%M" -- xz -T$threads -$compression -k $TESTFILE /usr/bin/time -a -o decompression-times.txt -f "%e\t%U\t%S\t%M" -- xz -t $TESTFILE.xz ls -k -s $TESTFILE.xz | cut -d' ' -f1 >> sizes.txt rm $TESTFILE.xz done done paste times.txt decompression-times.txt sizes.txt > time-size.txt $ cat time-size.txt Compr. Threads Wall.t User.t Sys.t Res.t Wall.t User.t Sys.t Res.t Size(kB) -0 -T163.48 63.07 0.4049321.251.160.082432 416324 -1 -T1102.90 102.26 0.6311064 1.531.260.143252 414624 -2 -T1131.04 130.40 0.6218544 1.431.270.094224 414616 -3 -T1146.22 145.61 0.6033992 1.431.240.126364 414616 -4 -T1142.24 141.76 0.4750488 1.341.210.106424 414584 -5 -T1147.81 147.26 0.5397616 1.371.220.1010420 414564 -6 -T1148.61 148.06 0.5497608 1.351.230.1010536 414564 -7 -T1154.29 153.70 0.58191800 1.441.220.1418820 414564 -8 -T1155.82 155.16 0.65380336 1.531.270.1335208 414564 -9 -T1194.08 193.18 0.88691128 1.591.310.1367700 41 -0e -T176.81 76.31 0.4862961.271.210.052536 416256 -1e -T1116.49 115.96 0.53
[Bug 1701132] Re: Please use xz instead of pxz
** Description changed: + [Impact] + + * Livecd-rootfs switches from using pxz to xz + * Pxz is installed at run-time instead of being a package dependency and pxz is in universe despite livecd-rootfs being in main. Both of those are undesired and fixing them makes livecd-rootfs easier to maintain following best practice and policy in Ubuntu. + + [Test Case] + + * Livecd-rootfs in Xenial uses (p)xz only for compressing raspi2 images. +To test the change install updated livecd-rootfs on armhf and create images setting the following environment variables: ARCH=armhf SUBARCH=raspi2 PROJECT=ubuntu-cpc. +Notice that the xz-compressed rapi2 image is still created. + + [Regression Potential] + + * It is a known and accepted regression that raspi2 image generation takes more wall-clock time after switching to xz because while Xenial's xz version does accept the -T4 option it does not compress in a parallel manner. + * The original change in Artful set -0 compression for xz which resulted in ~35% bigger compressed images. This regression is fixed in LP: #1732681 which is scheduled to be back-ported together with switching to xz. + * In case of this change causes regression it would manifest itself as failing build for raspi2, failing build for ubuntu-cpc images or increased raspi2 image size. + + [Original bug report text] + Livecd-rootfs (main) 2.447 uses "pxz -T4" for xz compression, while pxz is not in main. Xz 5.2 also supports multi-threaded compression thus xz could easily replace current pxz usage. A quick test on compressing livecd.ubuntu-cpc.img (created by ubuntu- image) shows that there is very little difference in compressed file size while xz's -T4 brings substantial speedup in wall clock time: $ cat test-xz #!/bin/sh TESTFILE=$1 echo "Compr.\tThreads\tWall.t\tUser.t\tSys.t\tRes.t" > times.txt echo "Wall.t\tUser.t\tSys.t\tRes.t" > decompression-times.txt echo "Size(kB)" > sizes.txt for threads in 1 4; do - for compression in $(seq 0 9) $(seq -f '%0.fe' 0 9); do - /usr/bin/time -a -o times.txt -f "-$compression\t -T$threads\t%e\t%U\t%S\t%M" -- xz -T$threads -$compression -k $TESTFILE - /usr/bin/time -a -o decompression-times.txt -f "%e\t%U\t%S\t%M" -- xz -t $TESTFILE.xz - ls -k -s $TESTFILE.xz | cut -d' ' -f1 >> sizes.txt - rm $TESTFILE.xz - done + for compression in $(seq 0 9) $(seq -f '%0.fe' 0 9); do + /usr/bin/time -a -o times.txt -f "-$compression\t -T$threads\t%e\t%U\t%S\t%M" -- xz -T$threads -$compression -k $TESTFILE + /usr/bin/time -a -o decompression-times.txt -f "%e\t%U\t%S\t%M" -- xz -t $TESTFILE.xz + ls -k -s $TESTFILE.xz | cut -d' ' -f1 >> sizes.txt + rm $TESTFILE.xz + done done paste times.txt decompression-times.txt sizes.txt > time-size.txt - $ cat time-size.txt + $ cat time-size.txt Compr. Threads Wall.t User.t Sys.t Res.t Wall.t User.t Sys.t Res.t Size(kB) -0 -T163.48 63.07 0.4049321.251.160.082432 416324 -1 -T1102.90 102.26 0.6311064 1.531.260.143252 414624 -2 -T1131.04 130.40 0.6218544 1.431.270.094224 414616 -3 -T1146.22 145.61 0.6033992 1.431.240.126364 414616 -4 -T1142.24 141.76 0.4750488 1.341.210.106424 414584 -5 -T1147.81 147.26 0.5397616 1.371.220.1010420 414564 -6 -T1148.61 148.06 0.5497608 1.351.230.1010536 414564 -7 -T1154.29 153.70 0.58191800 1.441.220.1418820 414564 -8 -T1155.82 155.16 0.65380336 1.531.270.1335208 414564 -9 -T1194.08 193.18 0.88691128 1.591.310.1367700 41 -0e -T176.81 76.31 0.4862961.271.210.052536 416256 -1e -T1116.49 115.96 0.5315068 1.281.200.073232 414568 -2e -T1137.13 136.62 0.5026892 1.301.200.104292 414564 -3e -T1144.34 143.84 0.4950556 1.311.210.096340 414564 -4e -T1146.93 146.48 0.4450412 1.301.200.106340 414568 -5e -T1150.59 150.04 0.5397552 1.321.220.1010416 414564 -6e -T1153.40 152.77 0.6297564 1.321.220.0910404 414568 -7e -T1157.10 156.46 0.62191956 1.351.260.0818548 414568 -8e -T1160.36 159.66 0.69380248 1.461.250.1335040 414568 -9e -T1198.32 197.42 0.88691256 1.491.190.1467648 414448 -0 -T421.97 86.83 0.4024896 1.281.220.052448 416368 -1 -T435.73 140.12 0.4573456 1.291.220.063136 414640 -2
[Bug 1701132] Re: Please use xz instead of pxz
I opened LP: #1732681 to clarify if the regression is acceptable or xz should use a different compression than -0. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701132 Title: Please use xz instead of pxz To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1701132/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701132] Re: Please use xz instead of pxz
The original command's CPU usage is the following in xenial: rbalint@xenial-test:~/Downloads$ time pxz -T4 -c ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img > ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.p.xz real3m31.117s user10m12.812s sys 0m17.988s rbalint@xenial-test:~/Downloads$ ls -al ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.p.xz -rw-rw-r-- 1 rbalint rbalint 254621780 Nov 16 00:03 ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.p.xz On the same system I retested the matrix of xz options and it shows that -T is ignored and -3 has the closest wall clock time with 258680 kB result which is slightly larger than the original but even -8 consumes system time and creates smaller compressed image: rbalint@xenial-test:~/Downloads/xz-test$ cat time-size.txt Compr. Threads Wall.t User.t Sys.t Res.t Wall.t User.t Sys.t Res.t Size(kB) -0 -T182.59 74.28 1.51479225.38 25.16 0.102404 329984 -1 -T1109.10 96.64 1.6310712 23.63 22.55 0.083080 298056 -2 -T1136.69 127.53 1.7818392 22.36 21.71 0.084196 286276 -3 -T1164.91 158.57 1.6433948 21.46 20.93 0.096260 275036 -4 -T1320.00 314.05 1.9050320 21.18 20.86 0.136120 258680 -5 -T1375.79 370.80 1.8297480 20.46 20.23 0.1010340 247316 -6 -T1476.02 471.64 2.0697416 20.45 20.18 0.1010352 245768 -7 -T1497.44 492.16 2.35191624 21.24 20.06 0.0918408 240188 -8 -T1513.29 507.79 2.24380108 19.43 19.16 0.1034908 229736 -9 -T1873.02 579.58 7.55690952 22.61 19.30 0.1467700 224492 -0e -T1746.35 743.14 1.76622423.69 23.32 0.112420 301004 -1e -T1809.93 806.56 1.9814988 21.57 21.31 0.113180 272976 -2e -T1846.12 843.60 1.8226748 22.10 20.75 0.074140 262860 -3e -T1699.59 696.83 1.9750316 20.90 20.68 0.076260 252052 -4e -T1879.00 875.42 2.0450256 20.35 19.96 0.106292 252048 -5e -T1723.80 720.15 1.8197480 21.35 20.37 0.1110360 245172 -6e -T1911.69 908.11 2.0397464 20.37 19.83 0.0710348 245168 -7e -T1942.32 936.49 2.09191616 19.72 19.25 0.1218352 239552 -8e -T1967.20 962.26 2.42380156 18.85 18.07 0.1434904 228956 -9e -T11177.95 1036.56 6.36690784 19.78 18.08 0.1167680 223748 -0 -T481.56 73.64 1.46482029.27 25.82 0.092252 329984 -1 -T4103.68 95.14 1.6710800 24.41 22.86 0.123180 298060 -2 -T4134.05 126.22 1.7618480 22.06 21.65 0.134232 286276 -3 -T4170.57 162.72 1.7233812 21.62 20.94 0.126188 275036 -4 -T4322.15 317.83 2.1050284 21.55 21.05 0.076136 258684 -5 -T4407.15 399.93 1.9297656 21.85 20.50 0.1210412 247320 -6 -T4478.64 473.32 1.9597660 21.92 20.34 0.0910348 245772 -7 -T4493.97 485.52 1.99191692 20.66 20.01 0.0818548 240192 -8 -T4512.28 506.08 2.20380096 22.03 19.58 0.1134888 229740 -9 -T4652.71 559.20 4.78691076 20.57 19.15 0.0867700 224496 -0e -T4750.00 743.68 1.74617224.59 23.34 0.112444 301004 -1e -T4812.97 805.57 1.9415004 21.81 21.36 0.113280 272976 -2e -T4849.79 842.25 2.0526828 20.96 20.63 0.124312 262860 -3e -T4702.49 696.67 1.6950380 20.85 20.61 0.096360 252056 -4e -T4882.09 874.62 1.9750552 20.11 19.89 0.086292 252052 -5e -T4725.29 719.12 1.9097512 20.42 20.20 0.0610320 245176 -6e -T4917.31 908.11 1.8797632 20.26 19.78 0.1210312 245168 -7e -T4941.70 935.85 1.91191656 20.47 19.27 0.1018432 239552 -8e -T4966.09 961.14 2.21380140 18.52 18.10 0.1335028 228956 -9e -T41147.79 1040.24 5.46690956 19.88 18.09 0.0667612 223748 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701132 Title: Please use xz instead of pxz To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1701132/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701132] Re: Please use xz instead of pxz
Please disregard my previous result, the xz -0 does generate a bigger .xz , it was a test running in parallel which updated the .xz file. I'll attach the full test results when it is finished, but the actual file size was bigger with xz -0: 323Mubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.xz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701132 Title: Please use xz instead of pxz To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1701132/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701132] Re: Please use xz instead of pxz
I tested the change with the image we actually ship with .xz compression and to my greatest surprise xz -0 actually performed much better than pxz with default compression: $ du -sh * 1.2Gubuntu-16.04.3-preinstalled-server-armhf+raspi2.img 243Mubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.orig.xz 243Mubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.p.xz 168Mubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.xz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701132 Title: Please use xz instead of pxz To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1701132/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701132] Re: Please use xz instead of pxz
This bug was fixed in the package livecd-rootfs - 2.448 --- livecd-rootfs (2.448) artful; urgency=medium [ Balint Reczey ] * Depend on packages in main instead of installing them at runtime * Depend on vmdk-stream-converter instead of installing it at run time * Use xz -T4 -0 instead of pxz (LP: #1701132) Supported xz versions before 5.2 also accept the -T4 option but ignore it. * Depend on xz-utils instead of pxz [ Mathieu Trudel-Lapierre ] * auto/config: enable Debug::pkgDepCache::AutoInstall so tracking down dependency bugs in the livefs is a little clearer. [ Robert C Jennings ] * live-build/functions: Update path for VMDKstream.py (LP: #1706116) -- Brian MurrayMon, 24 Jul 2017 11:06:24 -0700 ** Changed in: livecd-rootfs (Ubuntu) Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701132 Title: Please use xz instead of pxz To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1701132/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701132] Re: Please use xz instead of pxz
** Branch linked: lp:~rbalint/livecd-rootfs/xz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701132 Title: Please use xz instead of pxz To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1701132/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701132] Re: Please use xz instead of pxz
** Attachment added: "Nicely indented test results" https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1701132/+attachment/4905008/+files/time-size.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701132 Title: Please use xz instead of pxz To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1701132/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs