Re: LiveCD optimisations
Hello Louis, Louis Simard [2010-11-01 17:10 -0400]: AdvanceCOMP is packaged in maverick universe as advancecomp. jpegoptim is packaged in maverick universe as jpegoptim. Could these programs be added to the build scripts, or would that be discouraged since they're in universe? Would these optimisations be a case for inclusion into main? Yes, of course. I added some work items for advancecomp and jpegoptim to the spec. Thanks! Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
Hello Louis, Louis Simard [2010-05-20 20:35 -0400]: Optimising the PNG images saves 5.5 MB on the filesystem.squashfs. For the record, I just uploaded a new pkgbinarymangler to natty which now calls optipng on PNG files, as part of https://blueprints.launchpad.net/ubuntu/+spec/performance-desktop-n-install-footprint Thanks a lot for bringing this up! Optimising the SVG files saves an additional 7 MB. This is next on my list. I'll package scour, and add it to cdbs gnome.mk with some test cases. Thanks, Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) signature.asc Description: Digital signature -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
Hi Martin, Thanks for the notification. While you're working on PNG optimisations in the build scripts, I have something to ask you. There has been discussion on the More LiveCD space optimisations thread [1] of using AdvanceCOMP to further reduce the size of PNG files (even after OptiPNG, PNG files can get recompressed further!). There has also been discussion of using jpegoptim to losslessly recompress JPEG files, and AdvanceCOMP for ZIP/JAR and gzip files. AdvanceCOMP is packaged in maverick universe as advancecomp. jpegoptim is packaged in maverick universe as jpegoptim. Could these programs be added to the build scripts, or would that be discouraged since they're in universe? Would these optimisations be a case for inclusion into main? Regarding this: I'll package scour, and add it to cdbs gnome.mk with some test cases. Thanks for this. Scour also has a fair amount of unit tests and other test cases that you could use. If you need to communicate with Scour for packaging adjustments, bugs or gaps in documentation, don't hesitate to file bugs and/or patches against Scour, or e-mail me. I'm the co-maintainer for Scour since June 2010, but even if I can't do releases, I can commit to the trunk. [1] https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2010-October/012181.html Regards, -- Louis Simard Conspicuous absence of digital signature here -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
Hi Martin, I intended to send the following to lyx-devel, but my dog ate it. Louis has mentioned AdvanceCOMP, but I thought you might find the summary below useful. I think that the figures may be out by about 30%, possibly do to squashfs doing some form of intra-file de-duplication, but I hope it gives a good feel as to where space can be saved. On Thu, Oct 14, 2010 at 5:45 PM, Scott Ritchie sc...@open-vote.org wrote: On 10/03/2010 11:50 PM, Martin Pitt wrote: * Optimize images in packages, as proposed by Louis Simard in May. We could do this in a very simple and automated fashion by just running pngcrush on every png image. It could even be a part of the build daemon or built into debhelper. Note that running advdef -z4 (after optipng) can further compress png images, saving another 3MB. Advdef could also compress the existing gz files by about 5MB (less if there are less gz files because we remove man pages etc.). Optimizing svg files with Scour.py could save a further 7MB on the LiveCD, but we should not run Scour on cards as card games make use of non-visual tags in those SVG files [1]. Other XML files can be optimized, but applications expect a particular format; for a list of XML files safe to optimize see [2]. If we want to go crazy with lossless recompression we could also run jpegrescan on jpegs saving 0.5MB, and replace gz files with lzma saving up to 10MB. Optimizing HTML files with webpack and html::clean could save 100k on the LiveCD (2MB when uncompressed) but doing this right is hard, and probably not worth it. By comparison, the changelogs take about 30MB, and manpages take about 15MB. If we keep the changelogs, they would be a good candidate for lzma recompression since they shrink by 4.4MB. For more information, see the thread: More LiveCD space optimizations http://osdir.com/ml/ubuntu-devel-discuss/2010-10/msg0.html [1] http://www.mail-archive.com/ubuntu-devel-discuss@lists.ubuntu.com/msg11350.html [2] http://www.mail-archive.com/ubuntu-devel-discuss@lists.ubuntu.com/msg11410.html -- John C. McCabe-Dansted -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
On Thu, May 20, 2010 at 6:35 PM, Louis Simard louis.sim...@gmail.com wrote: Optimising the PNG images saves 5.5 MB on the filesystem.squashfs. Optimising the SVG files saves an additional 7 MB. This is a total of 12.5 MB which could be used to pack more software or another language pack or two onto the LiveCD. Speaking of saving space on the LiveCD, I note on http://www.omgubuntu.co.uk/2010/05/see-ya-f-spot-shotwell-comes-to-ubuntu.html that F-Spot is supposed to be bumped (in favor of Shotwell) for Maverick. Does this mean that we can finally remove Mono now too? (Tomboy can be replaced with Gnote; gBrainy can be replaced with some other game... it's not like there aren't lots to pick from :) Anyone have an estimate of how much space would be saved by doing that? CK -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
On 24 May 2010 10:33, Conrad Knauer ath...@gmail.com wrote: On Thu, May 20, 2010 at 6:35 PM, Louis Simard louis.sim...@gmail.com wrote: Optimising the PNG images saves 5.5 MB on the filesystem.squashfs. Optimising the SVG files saves an additional 7 MB. This is a total of 12.5 MB which could be used to pack more software or another language pack or two onto the LiveCD. Speaking of saving space on the LiveCD, I note on http://www.omgubuntu.co.uk/2010/05/see-ya-f-spot-shotwell-comes-to-ubuntu.html that F-Spot is supposed to be bumped (in favor of Shotwell) for Maverick. Does this mean that we can finally remove Mono now too? No. (Tomboy can be replaced with Gnote; gBrainy can be replaced with some Gnote is abandoned by the author and has less functionality then Tomboy (less plugins, no ubuntuone integration etc.) gBrainy ROCKS =) other game... it's not like there aren't lots to pick from :) Anyone have an estimate of how much space would be saved by doing that? No clue =) and to late for Maverick this should have been proposed as a spec for UDS and discussed there. ps. I would really want for gnome-do to get into default install ;-) CK -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
On 21 May 2010 15:48, Phillip Susi ps...@cfl.rr.com wrote: snip Also could you explain a bit what you mean by optimizations? You can of course, use a higher lossy compression on the png images, but that lowers their quality, which I think is not a desirable tradeoff. png does not do lossy compression -- Matt Wheeler m...@funkyhat.org -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
On Mon, May 24, 2010 at 4:33 AM, Dmitrijs Ledkovs dmitrij.led...@ubuntu.com wrote: Tomboy can be replaced with Gnote Gnote is abandoned by the author On what basis do you claim this? Lucid uses 0.6.2 according to http://packages.ubuntu.com/search?keywords=gnote I note the following release dates according to the files in http://ftp.gnome.org/pub/GNOME/sources/gnote/ gnote-0.6.3 28-Nov-2009 gnote-0.6.4 22-Mar-2010 gnote-0.7.0 31-Dec-2009 gnote-0.7.1 04-Jan-2010 gnote-0.7.2 12-Mar-2010 Debian is up to 0.7.1 as per http://packages.debian.org/search?keywords=gnote and has less functionality then Tomboy (less plugins, no ubuntuone integration etc.) Please see http://www.stefanoforenza.com/getting-gnote-facts-straight/ CK -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
On 24 May 2010 17:57, Conrad Knauer ath...@gmail.com wrote: On Mon, May 24, 2010 at 4:33 AM, Dmitrijs Ledkovs dmitrij.led...@ubuntu.com wrote: Tomboy can be replaced with Gnote Gnote is abandoned by the author On what basis do you claim this? Last time I cared about Tomboy vs Gnote arguments it was this: http://mail.gnome.org/archives/gnote-list/2009-October/msg1.html Lucid uses 0.6.2 according to http://packages.ubuntu.com/search?keywords=gnote I note the following release dates according to the files in http://ftp.gnome.org/pub/GNOME/sources/gnote/ gnote-0.6.3 28-Nov-2009 gnote-0.6.4 22-Mar-2010 gnote-0.7.0 31-Dec-2009 gnote-0.7.1 04-Jan-2010 gnote-0.7.2 12-Mar-2010 Debian is up to 0.7.1 as per http://packages.debian.org/search?keywords=gnote Fair enough so taking ~6 months as cutoff date which puts at gnote 0.6.2 tomboy 1.1.0 Comparing: http://git.gnome.org/browse/gnote/tree/NEWS http://git.gnome.org/browse/tomboy/tree/NEWS and has less functionality then Tomboy (less plugins, no ubuntuone integration etc.) Please see http://www.stefanoforenza.com/getting-gnote-facts-straight/ So has the syncing been implemented yet? IMHO it's the killer feature to sync tomboy with linux, mac, win cloud. Also note that gnote vs tomboy in terms of disk space savings is really about gtkmm vs mono. As far as I remember (again could be out-of-date and less relevant with GObject-Interspcection) that gtkmm is big and currently not-included by default on Ubuntu CD's. ps. I use emacs-org mode and I don't have tomboy/gnote installed =) -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
At 2010-05-21 04:41 GMT, Martin Owens docto...@gmail.com wrote: Are there no more things that could be optimised? For instance does using xmllint with --noblanks on the 12496 xml files save any space? My testing with XML files is done now, and here are the results! (And the modified script, attached to this email) 'xmllint --noblanks --nsclean FILE' gives savings of 3 MB (pre-squashfs). It actually *enlarges* some files containing non-7-bit characters, such as gconf-tree in French (by a bit, due to accented chars), Greek and Japanese (by a lot, due to every single text-node character being entity'd). 'xmllint --noblanks --nsclean --encode utf-8 FILE' gives savings of 10 MB. It shrinks even the French, Greek and Japanese files. On the squashfs'd side, this gives modest savings of 0.79 MB. HOWEVER: The optimisations made card games (Klondike etc.) unplayable, as no cards appear, due to the change in /usr/share/gnome-games-common/cards/gnomangelo_bitmap.svg. Gbrainy started crashing when a new game of verbal analogies was started, due to xmllint's addition of an ?xml? tag in /usr/share/games/gbrainy/verbal_analogies.xml. Nautilus lost its toolbars, icons and right-click menu. The help viewer (System / Help and Support) complains that every file is not a well-formed XML document. So perhaps XML optimisations aren't so good? :( 1379 HTML files could be optimised too, but they might get hopelessly mangled by xmllint - is there a utility for that? 136 JPEG files... well, those are lossy :) 379 GIF files... some are in HTML docs and could be replaced with PNGs, but that can't be done automatically, and so the Ubuntu Doc team would have to get involved. (the images are so small it's probably not worth it, except to get away from the LZW patent...) There are spinner/throbber animations in .gif format in some packages (Gwibber, Rhythmbox), as well as animated clipart in OpenOffice, which probably can't get replaced. 1 TIFF image: /usr/share/app-install/icons/_usr_lib_GNUstep_Applications_GTAMSAnalyzer.app_Resources_largeApp.tif - This could become a PNG too. ubuntu-optimisations.sh Description: Bourne shell script -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
Le samedi 22 mai 2010 à 04:31 -0400, Louis Simard a écrit : At 2010-05-21 04:41 GMT, Martin Owens docto...@gmail.com wrote: Are there no more things that could be optimised? For instance does using xmllint with --noblanks on the 12496 xml files save any space? My testing with XML files is done now, and here are the results! (And the modified script, attached to this email) 'xmllint --noblanks --nsclean FILE' gives savings of 3 MB (pre-squashfs). It actually *enlarges* some files containing non-7-bit characters, such as gconf-tree in French (by a bit, due to accented chars), Greek and Japanese (by a lot, due to every single text-node character being entity'd). 'xmllint --noblanks --nsclean --encode utf-8 FILE' gives savings of 10 MB. It shrinks even the French, Greek and Japanese files. On the squashfs'd side, this gives modest savings of 0.79 MB. HOWEVER: The optimisations made card games (Klondike etc.) unplayable, as no cards appear, due to the change in /usr/share/gnome-games-common/cards/gnomangelo_bitmap.svg. Gbrainy started crashing when a new game of verbal analogies was started, due to xmllint's addition of an ?xml? tag in /usr/share/games/gbrainy/verbal_analogies.xml. Nautilus lost its toolbars, icons and right-click menu. The help viewer (System / Help and Support) complains that every file is not a well-formed XML document. So perhaps XML optimisations aren't so good? :( 1379 HTML files could be optimised too, but they might get hopelessly mangled by xmllint - is there a utility for that? 136 JPEG files... well, those are lossy :) 379 GIF files... some are in HTML docs and could be replaced with PNGs, but that can't be done automatically, and so the Ubuntu Doc team would have to get involved. (the images are so small it's probably not worth it, except to get away from the LZW patent...) There are spinner/throbber animations in .gif format in some packages (Gwibber, Rhythmbox), as well as animated clipart in OpenOffice, which probably can't get replaced. 1 TIFF image: /usr/share/app-install/icons/_usr_lib_GNUstep_Applications_GTAMSAnalyzer.app_Resources_largeApp.tif - This could become a PNG too. I think the only relevant data is how much size is gain on the iso itself (so once mksquashfs is runned). I'm afraid it won't be that much with the xml + images optimization (is 0.79 MB containing the whole optimization or just the xml one?) and as you have already triggered, those optimization are error prone in applications. Not sure it worths the risk if the real size gain in the iso is only 0.79 MB. Didier -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
At 2010-05-22 09:06 GMT, Didier Roche didro...@ubuntu.com wrote: (is 0.79 MB containing the whole optimization or just the xml one?) [snip] Not sure it worths the risk if the real size gain in the iso is only 0.79 MB. 0.79 MB (on the squashfs) is for XML files only, and is HIGHLY error-prone due to applications either not recognising whitespace properly, not recognising the ?xml? declaration, not recognising the encoding, parsing using a homegrown parser or regex that relies on indentation/pretty-printing, or just being finnicky. But the Ubuntu LiveCD still stands to gain an easy 11 MB, even without that 0.79 MB. :) 5 MB (on the squashfs) for PNG images: completely safe conversion, because OptiPNG doesn't have bad bugs. 6 MB (on the squashfs) for SVG images: completely safe conversion except for the card games, which need ID names to seaprate the cards from the one SVG file. All application icons and user interface button icons etc. are okay. Regards, - Louis -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
On 22 May 2010 09:31, Louis Simard louis.sim...@gmail.com wrote: HOWEVER: The optimisations made card games (Klondike etc.) unplayable, as no cards appear, due to the change in /usr/share/gnome-games-common/cards/gnomangelo_bitmap.svg. Gbrainy started crashing when a new game of verbal analogies was started, due to xmllint's addition of an ?xml? tag in /usr/share/games/gbrainy/verbal_analogies.xml. Nautilus lost its toolbars, icons and right-click menu. The help viewer (System / Help and Support) complains that every file is not a well-formed XML document. So perhaps XML optimisations aren't so good? :( Is it due to them using GMarkup instead of libxml to parse XML's? I yes it's a bug in glib then =) i would be cool to compress xml's as much as possible. Afterall people should be getting the source packages to edit those and apps should parse xml's just fine without spaces and with/without ?xml? tag. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
2010-05-21 02:35 keltezéssel, Louis Simard írta: Optimising the CD to put files at the end allows it to boot marginally faster (about 10 seconds on my benchmarks), start applications faster, and allows the CD drive on a user's computer to run quieter while using his/her applications, as reading near the end (edge of the disc) requires slower spinning. This sounds really-really good! Spare user's time, yeah! MG -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
At 2010-05-22 10:59 GMT, Dmitrijs Ledkovs dmitrij.led...@ubuntu.com wrote: Is it due to them using GMarkup instead of libxml to parse XML's? I yes it's a bug in glib then =) i would be cool to compress xml's as much as possible. Afterall people should be getting the source packages to edit those and apps should parse xml's just fine without spaces and with/without ?xml? tag. I think optimisations on the XMLs should be called off for now. All of the XMLs in /usr/share/gnome/help are getting their ENTITY declarations duplicated from /usr/share/gnome/help/libs/global.ent with xmllint... this is probably the cause of the help viewer not working. As for applications using GMarkup, that's entirely possible; the dependency list for Gbrainy, for instance, has libglib2.0-cil and libmono-system2.0-cil which probably implement Mono's System.Xml namespace. However, Yelp depends on docbook-xml and libxml2. I can confirm that my XML optimisations broke Gbrainy and Nautilus by excluding these files from the script: /usr/share/branding/gnome-games-common/cards/gnomangelo_bitmap.svg /usr/share/nautilus/ (recursive) *.xml /usr/share/games/gbrainy/verbal_analogies.xml The Scour optimisation should also be called off only for the cards. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
At 2010-05-21 04:41 GMT, Martin Owens docto...@gmail.com wrote: Hey Louis, Hey Martin, thanks for the reply! Sounds great and looks like a pretty good script, I have some comments: You may be able to make it a little faster by using the find results in one like like this: find / -type f -name *.svg -print0 | xargs -0 -I FILE sh -c '/tmp/scour/scour.py --enable-id-stripping --indent=none -i FILE -o FILE-opt test -s FILE-opt mv FILE-opt FILE || rm FILE-opt' I had considered using sh -c to execute the Scouring and renaming, yes, but didn't know how to go about detecting empty files except with another 'find'. Thanks for telling me about test -s :) Although if you can get all that into a script file, so much the better so it's not all on one line. But at least it's not doing a find 3 times for the same files. True. This is a case of optimising the optimiser, which I consider a micro-optimisation because the later invocations of 'find' are highly likely to have the needed disk blocks in RAM - but every little bit helps, just like with these image files. (Speaking of which, Scour.py imports the Psyco JIT if it's available, but it doesn't help that much. It makes the Python code itself run faster, yes, but at the cost of greater startup time for each Scour.py instance, and most files are optimised in 0.06 second anyway.) Do you need to chroot into the file system to perform these steps? considering that your downloading code to do it (with bzr which isn't installed ont he cd). Would it not be good to perform these steps outside of the squashfs and iso file system? For instance I got resolve issues when it tried to do the apt update. I probably don't. That was part of a script that allowed me to customise more things, such as updating packages (which I needed to chroot for), removing the desktop background, updating Linux and all that; I just trimmed it down for this email. I'll move the chroot processing to the host. Are there no more things that could be optimised? For instance does using xmllint with --noblanks on the 12496 xml files save any space? Will test this shortly. I hadn't thought of that yet, and I'm flabbergasted by the number of XML files! Seeing as SVG files are also XML files, and Scour.py seems to pretty-print XML even with --indent=none, that might save even more, actually. Finally... should some of these optimisations work their way upstream so all packages have optimised files, smaller downloads, smarter mirror storage etc? Of course! :) Working with upstreams would avoid keeping debdiffs around for the optimised files in Ubuntu repositories, and will help other distributions too. I'll attach a modified script to my next email with more testing results regarding XML. Regards, - Louis -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
On 5/20/2010 8:35 PM, Louis Simard wrote: Greetings ubuntu-devel-discuss :) I have a proposal for you, and I'll present it simply with the 5 W's. snip When attaching scripts please make sure they are attached with an inline disposition so they are readily reviewable while reading the email instead of having to save them and open them in another text editor. Also could you explain a bit what you mean by optimizations? You can of course, use a higher lossy compression on the png images, but that lowers their quality, which I think is not a desirable tradeoff. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations explained
On 5/21/2010 1:40 PM, Louis Simard wrote: Err... While I know what you want me to do (you want Content-Disposition: inline), I don't know how to do that in the Gmail web interface. Perhaps I'll set up Mozilla Thunderbird, if it can do that :-) Heh, yea, I've struggled with this on thunderbird too, which is why I usually end up submitting patches via something like mime-construct or some other command line mime editor where I can force it to use Content-Disposition: inline. - For PNG: the data used to store some images on the CD is not compressed to the highest level. OptiPNG takes those files and tries to recompress them to the highest level, while ensuring that every pixel's color value ends up being the same. I believe that PNG applies a lossey compression first, then gzips the result. It sounds like you are saying that the gzip is done with -3 instead of -9, so you ungzip it and recompress on -9. Is that more or less correct? If so that sounds pretty good, but like you mentioned before, should be done upstream rather than only for the livecd. - For SVG: the data used to store ALL images on the CD is not optimal for rendering purposes. Inkscape metadata, Sodipodi metadata, ID names for elements that end up unused, gradients defined dozens of times, etc., are bloating the files. Scour.py takes those files and removes this bloat, while ensuring that the new versions render identically to the original. However, since Inkscape's metadata ends up removed, it could be more difficult for users to open these new files in Inkscape. Sounds good, and also would be good to do upstream instead of just for the lived. - For XML, as described by Martin Owens: xmllint would remove everything superfluous from all files on the CD, while ensuring that the data is parsed identically. I haven't tested this yet except on one file from the CD (squashfs - /var/lib/gconf/defaults/%gconf-tree.xml), but that file went from 2,095,034 bytes to 1,779,376 (a savings of 315,658). There's more hope yet. I noticed the bloated gconf xml files a few years back myself and brought it up on the devel list. IIRC I saw even more wasted space than you mention here, due to 10, 20, even 30 characters of whitespace indenting each line. This does add a lot of bloat to the files I don't like to have on an installed system, but once compressed into the squashfs for the livecd, the whitespace drops out, so there wasn't much concern about it. At one point I tried just converting the whitespace into hard tabs and saved quite a bit of space, while preserving the indentation for human editing. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
On 21 May 2010 01:35, Louis Simard louis.sim...@gmail.com wrote: -- WHAT? -- Optimise the PNG images and SVG files on the Ubuntu LiveCD. Optimise the Ubuntu LiveCD by putting start-up files and programs near the end of the CD. -- Implementation -- 1) Should this go into deb-package mangler run by soyuz? 2) Or should this be implemented as debhelper addon / cdbs as no-op ubuntu-patch and then if successful (all the quirks are worked out) and pushed to Debian? -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: LiveCD optimisations
Hey Louis, Sounds great and looks like a pretty good script, I have some comments: You may be able to make it a little faster by using the find results in one like like this: find / -type f -name *.svg -print0 | xargs -0 -I FILE sh -c '/tmp/scour/scour.py --enable-id-stripping --indent=none -i FILE -o FILE-opt test -s FILE-opt mv FILE-opt FILE || rm FILE-opt' Although if you can get all that into a script file, so much the better so it's not all on one line. But at least it's not doing a find 3 times for the same files. Do you need to chroot into the file system to perform these steps? considering that your downloading code to do it (with bzr which isn't installed ont he cd). Would it not be good to perform these steps outside of the squashfs and iso file system? For instance I got resolve issues when it tried to do the apt update. Consider doing a check to see if the iso exists before reserving scratch, errors are likely if you don't. Are there no more things that could be optimised? For instance does using xmllint with --noblanks on the 12496 xml files save any space? Finally... should some of these optimisations work their way upstream so all packages have optimised files, smaller downloads, smarter mirror storage etc? Martin, On Thu, 2010-05-20 at 20:35 -0400, Louis Simard wrote: Greetings ubuntu-devel-discuss :) I have a proposal for you, and I'll present it simply with the 5 W's. -- WHAT? -- Optimise the PNG images and SVG files on the Ubuntu LiveCD. Optimise the Ubuntu LiveCD by putting start-up files and programs near the end of the CD. -- WHY? -- Optimising the PNG images saves 5.5 MB on the filesystem.squashfs. Optimising the SVG files saves an additional 7 MB. This is a total of 12.5 MB which could be used to pack more software or another language pack or two onto the LiveCD. Optimising the CD to put files at the end allows it to boot marginally faster (about 10 seconds on my benchmarks), start applications faster, and allows the CD drive on a user's computer to run quieter while using his/her applications, as reading near the end (edge of the disc) requires slower spinning. These changes will give prospective users a better view of Ubuntu right from the LiveCD. There might also be additional benefits to having smaller PNG and SVG images, such as saving space on a user's hard disk when installed. The uncompressed (pre-squashfs) savings for the SVG images is 18 MB. -- WHEN? -- Now! :) Just kidding. As soon as possible would be nice. Maybe even for the next Ubuntu version, codename Maverick Meerkat! -- WHO? -- Ubuntu developers. But don't go thinking that you'll do all the grunt work of testing these optimisations for yourself! (See HOW? below) -- HOW? -- Attached to this email is a bash script I've made to perform all of these optimisations on any Canonical-supported Ubuntu 10.04 LiveCD image, almost automatically. (After optimisations are done, you can check the state of the LiveCD in a bash shell from within it. The rest is fully automatic.) The real savings would come from optimising the PNG and SVG images right in the packages themselves, not just the LiveCD. Given a directory containing PNG and SVG images, the part of my script dealing with OptiPNG and Scour.py can automatically optimise these files. The best candidate for such a Scouring would be ubuntu-docs, as it has tons of PNG images. Most application packages have an SVG icon or two as well. Thanks for your time! - Louis -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
