Bug#678227: Bug#641967: Apt downloads all description translations
David Kalnischkies, le Thu 06 Sep 2012 16:49:55 +0200, a écrit : On Fri, Aug 31, 2012 at 12:07 AM, Samuel Thibault sthiba...@debian.org wrote: David Kalnischkies, le Thu 30 Aug 2012 19:43:21 +0200, a écrit : On Tue, Aug 28, 2012 at 1:14 PM, Julian Andres Klode j...@debian.org wrote: We could add a switch to apt-cdrom to copy all configured locales for d-i purposes, The question is how apt-cdrom gets to know which locales are configured. If I read the d-i syslog correctly apt-cdrom is run before 'locales' No, locales is part of the base system, and thus installed and configured before CD-ROM inspection by apt-cdrom. apt-cdrom-setup is not a step in d-i, it's apt-setup which is one, done after bootstrap, I take your No and re-raise it to No, yes, but twice. apt-cdrom is run twice, first as I described it. Ah, right, I didn't know about that one. Meaning that on the first run above you will get only Translation-en, while you get Translation-en and Translation-$LANG (and possibly some more friends defined by the various environment variables and configs) in the second. Confirmed with a daily image, thanks! Samuel -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#678227: Bug#641967: Apt downloads all description translations
On Fri, Aug 31, 2012 at 12:07 AM, Samuel Thibault sthiba...@debian.org wrote: David Kalnischkies, le Thu 30 Aug 2012 19:43:21 +0200, a écrit : On Tue, Aug 28, 2012 at 1:14 PM, Julian Andres Klode j...@debian.org wrote: We could add a switch to apt-cdrom to copy all configured locales for d-i purposes, The question is how apt-cdrom gets to know which locales are configured. If I read the d-i syslog correctly apt-cdrom is run before 'locales' No, locales is part of the base system, and thus installed and configured before CD-ROM inspection by apt-cdrom. apt-cdrom-setup is not a step in d-i, it's apt-setup which is one, done after bootstrap, I take your No and re-raise it to No, yes, but twice. apt-cdrom is run twice, first as I described it. Feel free to check it yourself, the first package installed (aka the first entry in /var/log/apt/history.log) is 'locales' because everything not part of the bootstrap is installed with apt-get which uses as source the cdrom (what else, as mirrors aren't configured yet). After some more packages are installed the installation reaches the apt- setup stage in which a mirror will be chosen - but as its first task it runs apt-cdrom again as you said it - in the locale chosen and installed earlier. This makes this problem actually solvable, at least for installers with that behavior … and invoking apt-cdrom-setup. apt-cdrom is actually run in the locale chosen during installation, so it can simply follow that (I wonder why it doesn't seem to be doing it). As said, apt-cdrom never did and nobody changed it until now - likely also because nobody of the few people who take care of APT use apt-cdrom on a daily basis currently - or at least in my case at all. (which explains a few more needed bugfixes in upcoming 0.9.7.5 …) I wonder how many people actually depend on that behavior (meaning especially other distro-installers and multi-language systems), but ignoring my fear to be once again be flamed to death for an easy and only logical behavior change I just changed apt-cdrom to drop all not configured Translation-* files from the copy list. Meaning that on the first run above you will get only Translation-en, while you get Translation-en and Translation-$LANG (and possibly some more friends defined by the various environment variables and configs) in the second. Installers allowing to create multiple users while installing (with different $LANG) need to re-run apt-cdrom in all of these $LANGs to get the optimal behavior. I tried it with a modified multi-arch beta1 cd [0], so at least for debian it seems to work, but I only run it until a mirror is chosen for the lack of network, so more testers are welcomed of course … (You find it only the bzr branch for now, as we haven't made our minds yet how to solve #686346 without too much work and breakage risk - life would be so easy if dpkg and APT^Wmyself would agree just once …) If someone feels the urge to kill me for this change now: Add APT::CDROM::DropTranslation false; to your configuration file/commandline invocation and you are back to your normal and beloved previous behavior. Best regards David Kalnischkies [0] some hints at http://wiki.debian.org/DebianInstaller/Modify/CD -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#678227: Bug#641967: Apt downloads all description translations
On Tue, Aug 28, 2012 at 1:14 PM, Julian Andres Klode j...@debian.org wrote: We could add a switch to apt-cdrom to copy all configured locales for d-i purposes, The question is how apt-cdrom gets to know which locales are configured. If I read the d-i syslog correctly apt-cdrom is run before 'locales' is installed and localechooser sets the chroot up, so to have apt-cdrom in the correct environment localechooser setup either needs to be part of the bootstrap or we make it so that apt-cdrom doesn't copy any Translation files and localechooser calls after setting up 'locales' apt-get update to copy the Translation files over [0]. Or we tell localechooser (or some later d-i stage) that it should remove 'excess' Translation files. The later would have the advantage of working better in multi-user setups (with different l10n environments), but I am not sure if you can actually do that with d-i currently - the first two don't, the second additional changes behavior of apt-cdrom in a way breaking older d-i versions and/or any other installer/user. and then once we have translation files in /var/lib/apt/lists only recognize those languages and the ones configured via APT configuration. We don't even need a switch, we could just look at this list if no languages are explicitely configured for APT or translations found in /var/lib/apt/lists. Mhh. Yeah, we really need to do this to 'preserve' configuration or otherwise we just get all Translation files again on each update with a cdrom source (but kinda orthogonal to the problem at hand). So something along the lines of maybe: if acquire::languages set OR files in /var/lib/apt/lists: copy as configured else copy everything acquire::languages set is a bit flaky through, so maybe just if LANG != C OR files in /var/lib/apt/lists (and C.UTF-8 and POSIX, right?) Best regards David Kalnischkies -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#678227: Bug#641967: Apt downloads all description translations
David Kalnischkies, le Thu 30 Aug 2012 19:43:21 +0200, a écrit : On Tue, Aug 28, 2012 at 1:14 PM, Julian Andres Klode j...@debian.org wrote: We could add a switch to apt-cdrom to copy all configured locales for d-i purposes, The question is how apt-cdrom gets to know which locales are configured. If I read the d-i syslog correctly apt-cdrom is run before 'locales' No, locales is part of the base system, and thus installed and configured before CD-ROM inspection by apt-cdrom. apt-cdrom-setup is not a step in d-i, it's apt-setup which is one, done after bootstrap, and invoking apt-cdrom-setup. apt-cdrom is actually run in the locale chosen during installation, so it can simply follow that (I wonder why it doesn't seem to be doing it). Samuel -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#678227: Bug#641967: Apt downloads all description translations
On Mon, Aug 27, 2012 at 7:43 PM, Samuel Thibault sthiba...@debian.org wrote: Control: tags 641967 + important Is closed for a long time now and unrelated in so far as nobody is setting this variable - otherwise the auto-detection would not play safe here. I am dropping the cc to it therefore. (And the other thing mentioned in it: That the download of 10 files is slower than the download of 5 files is not a bug, but common sense.) Control: tags 678227 + important (Happily went under my radar as his reporter manages every time he got out of my kill-file to rejoin it in a few days, if not hours as I am probably on his as he continues his tirade against InRelease or now Translation-* without giving a damn what I or even others say) If you could add a message to this bug telling a casual reader what you think is a (or the) bug here, please do so. I don't see one. I only see one which quotes the documentation explaining exactly why this behavior is present and yet vouches to break it without a reason beside that an update is now slower than before. (With the hidden message that data is download which isn't needed by him, but failing to understand that APT has no way of knowing what all users on the system might need and therefore does the most sensible thing: keeping all data up-to-date it currently uses) The issue also happens at debian-installer time, thus downloading way (I don't know why an also is in that sentence …) apt-cdrom copies every Translation-* file from the disk to /var/lib/apt/lists and libapt applications nowadays will proceed with updating them all rather than dropping all not in the current environment (which nowadays is usually LANG=C as run by cron or an APT front-end). This is NOT new. Translation files exist for a long time and apt-cdrom copies them for a long time. The difference is that we now have a) a Translation-en file containing the English long descriptions b) libapt supporting users who can understand more than one language c) libapt supporting systems with more than one admin So these files were already copied to /var/lib/apt/lists, it's just that apt-get and everything else using libapt will not drop Translation-* files in there on the first run as there is a good chance that this run will be done by cron or a front-end in LANG=C loosing all this valuable data. more than necessary, and taking a lot of disk space (seen 82M here) for no reason, thus moving the embedded barrier yet higher for Debian. No reason indeed if your only target audience is a crowd of people who perfectly speak English (en_US) and can choose packages just by looking at a short description. If you want to support more people - and the effort many people put into creating these Translations files suggests that at least some want to support them - we need to ship these files to them. Not to mention the poor souls who like to read English long descriptions … Embedded systems aren't in any way less supported than before by this. The very same config option used since ever to disable the download of Translation files is still supported. In a way, it is even better supported as you have the option of getting right of -en, too, saving even more MB. Best regards David Kalnischkies -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#678227: Bug#641967: Apt downloads all description translations
David Kalnischkies, le Tue 28 Aug 2012 10:31:23 +0200, a écrit : The issue also happens at debian-installer time, thus downloading way (I don't know why an also is in that sentence …) That's actually because 678227 is what made debian-boot consider the issue quoted above already reported (see #684954). apt-cdrom copies every Translation-* file from the disk to /var/lib/apt/lists Why? Why not just for the current locale? and libapt applications nowadays will proceed with updating them all rather than dropping all not in the current environment (which nowadays is usually LANG=C as run by cron or an APT front-end). This is NOT new. I haven't seen that with squeeze: when I run apt-get update, only the Translation file for the current locale gets downloaded. Translation files exist for a long time and apt-cdrom copies them for a long time. The difference is that we now have a) a Translation-en file containing the English long descriptions b) libapt supporting users who can understand more than one language c) libapt supporting systems with more than one admin I don't see how that should make apt-get download all languages. So these files were already copied to /var/lib/apt/lists, it's just that apt-get and everything else using libapt will not drop Translation-* files in there on the first run as there is a good chance that this run will be done by cron or a front-end in LANG=C loosing all this valuable data. Then don't drop it in the LANG=C case, but do drop it in the LANG!=C case. more than necessary, and taking a lot of disk space (seen 82M here) for no reason, thus moving the embedded barrier yet higher for Debian. No reason indeed if your only target audience is a crowd of people who perfectly speak English (en_US) and can choose packages just by looking at a short description. sarcastic tooThere is no reason to download all languages unless you target the very few people who can speak all languages the descriptions have been translated to./sarcastic too Embedded systems aren't in any way less supported than before by this. 82M clearly hurts. Really. That's like 15% of the installation size! Samuel -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#678227: Bug#641967: Apt downloads all description translations
On Tue, Aug 28, 2012 at 10:47:43AM +0200, Samuel Thibault wrote: David Kalnischkies, le Tue 28 Aug 2012 10:31:23 +0200, a écrit : The issue also happens at debian-installer time, thus downloading way (I don't know why an also is in that sentence …) That's actually because 678227 is what made debian-boot consider the issue quoted above already reported (see #684954). apt-cdrom copies every Translation-* file from the disk to /var/lib/apt/lists Why? Why not just for the current locale? How about copying files for all configured locales on the system? Unless someone installes locales-all, this would be the best solution in my opinion (and the user can still overwrite it with APT configuration options anyway). So these files were already copied to /var/lib/apt/lists, it's just that apt-get and everything else using libapt will not drop Translation-* files in there on the first run as there is a good chance that this run will be done by cron or a front-end in LANG=C loosing all this valuable data. Then don't drop it in the LANG=C case, but do drop it in the LANG!=C case. And let us not forget C.UTF-8 -- it seems that C.utf8 and similar is not recognized, contrary to normal languages, so we'd need to check for C and C.UTF-8. -- Julian Andres Klode - Debian Developer, Ubuntu Member See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#678227: Bug#641967: Apt downloads all description translations
On Tue, Aug 28, 2012 at 12:47:09 +0200, Julian Andres Klode wrote: On Tue, Aug 28, 2012 at 10:47:43AM +0200, Samuel Thibault wrote: David Kalnischkies, le Tue 28 Aug 2012 10:31:23 +0200, a écrit : The issue also happens at debian-installer time, thus downloading way (I don't know why an also is in that sentence …) That's actually because 678227 is what made debian-boot consider the issue quoted above already reported (see #684954). apt-cdrom copies every Translation-* file from the disk to /var/lib/apt/lists Why? Why not just for the current locale? How about copying files for all configured locales on the system? Unless someone installes locales-all, this would be the best solution in my opinion (and the user can still overwrite it with APT configuration options anyway). Just because I have locales-all installed doesn't mean I want every single translation downloaded. Getting just the english ones is painful enough thankyouverymuch. Cheers, Julien signature.asc Description: Digital signature
Bug#678227: Bug#641967: Apt downloads all description translations
On Tue, Aug 28, 2012 at 01:05:33PM +0200, Julien Cristau wrote: On Tue, Aug 28, 2012 at 12:47:09 +0200, Julian Andres Klode wrote: On Tue, Aug 28, 2012 at 10:47:43AM +0200, Samuel Thibault wrote: David Kalnischkies, le Tue 28 Aug 2012 10:31:23 +0200, a écrit : The issue also happens at debian-installer time, thus downloading way (I don't know why an also is in that sentence …) That's actually because 678227 is what made debian-boot consider the issue quoted above already reported (see #684954). apt-cdrom copies every Translation-* file from the disk to /var/lib/apt/lists Why? Why not just for the current locale? How about copying files for all configured locales on the system? Unless someone installes locales-all, this would be the best solution in my opinion (and the user can still overwrite it with APT configuration options anyway). Just because I have locales-all installed doesn't mean I want every single translation downloaded. Getting just the english ones is painful enough thankyouverymuch. Right, but the installer does not install locales-all (right?). So by default, only the files for the configured locales will be copied. We could add a switch to apt-cdrom to copy all configured locales for d-i purposes, and then once we have translation files in /var/lib/apt/lists only recognize those languages and the ones configured via APT configuration. We don't even need a switch, we could just look at this list if no languages are explicitely configured for APT or translations found in /var/lib/apt/lists. This solves the problem with initial setup copying all translation files and should work for everyone. -- Julian Andres Klode - Debian Developer, Ubuntu Member See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#678227: Bug#641967: Apt downloads all description translations
On Tue, 28 Aug 2012 10:47:43 +0200 Samuel Thibault sthiba...@debian.org wrote: more than necessary, and taking a lot of disk space (seen 82M here) for no reason, thus moving the embedded barrier yet higher for Debian. I would say that this is at best misdirection - there are numerous ways to sort out download size issues for embedded systems, none of which require changes to apt. There can be other problems with apt on embedded, but those have nothing to do with translated descriptions and I'm waiting until after the release before setting aside time for reporting/discussing those. No reason indeed if your only target audience is a crowd of people who perfectly speak English (en_US) and can choose packages just by looking at a short description. (Many embedded systems do not allow any part of the base OS to be visible, let alone translated. The only translations are in the top level UI and no users get to see the base system, ever, so there is no problem here. This is why manpages and translations can be trivially removed in Emdebian - the user is never allowed to login. Any translations are constrained only to the single UI.) Embedded devices do not typically expose a package manager via the UI - typically because the embedded device has no direct connectivity - so this is a complete red herring. Developer boards do not determine what is useful for embedded systems in general. sarcastic tooThere is no reason to download all languages unless you target the very few people who can speak all languages the descriptions have been translated to./sarcastic too Embedded systems aren't in any way less supported than before by this. 82M clearly hurts. Really. That's like 15% of the installation size! Embedded systems can just turn off the downloads entirely or use packages from Emdebian which do not provide the translations *and* give you a 40% smaller download in the first place, etc.etc. Whatever other reasons there may be for this bug, I would recommend that the embedded barrier argument is entirely false and should be ignored for this issue. If 82Mb is 15% of your installation size, you should be considering Emdebian which can get your entire installation below 100Mb or ~200Mb with an X server. That is not a problem with apt, it's a problem of selecting a different apt source and/or existing apt config options. -- Neil Williams = http://www.linux.codehelp.co.uk/ pgp0mX7ewSNQZ.pgp Description: PGP signature
Bug#678227: Bug#641967: Apt downloads all description translations
Note that I have put quotes around embedded. I don't mean embedded as in user-oriented devices, but anything that people use Debian to tinker with: virtual machines, small routers, etc. We have already crossed the 512M barrier, if we could avoid the 1G barrier as long as easily possible, that'd be good. If 82Mb is 15% of your installation size, you should be considering Emdebian which can get your entire installation below 100Mb or ~200Mb with an X server. It's not my installation size. It's the standard Debian base installation size. Having to use embedian just because Debian got a 15% increase for no apparent good reason looks wrong. Samuel -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#678227: Bug#641967: Apt downloads all description translations
On 2012-08-27 13:43, Samuel Thibault wrote: Control: tags 641967 + important Control: tags 678227 + important Hello, The issue also happens at debian-installer time, thus downloading way more than necessary, and taking a lot of disk space (seen 82M here) for no reason, thus moving the embedded barrier yet higher for Debian. Samuel Thanks Samuel, but I suppose you meant severity foo important. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#678227: Bug#641967: Apt downloads all description translations
Filipus Klutiero, le Mon 27 Aug 2012 17:42:05 -0400, a écrit : On 2012-08-27 13:43, Samuel Thibault wrote: Control: tags 641967 + important Control: tags 678227 + important Thanks Samuel, but I suppose you meant severity foo important. Right, thanks :) Samuel -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
