Re: [gentoo-dev] RFC: enabling ipc-sandbox network-sandbox by default
On Mon, 12 May 2014 19:39:09 +0200 Michał Górny mgo...@gentoo.org wrote: I don't know postgresql well enough but does the test db reside in temporary build directory? That is, can you guarantee that: 1) it will never ever collide with user's database, 2) it will be properly cleaned up even if the test suite terminates unexpectedly? The closest thing you could do would be to create a separate tablespace residing in the build directory. I wouldn't consider this a good idea however, as you'd need postgres superuser permissions, it would have some unintented side effects (like breaking on SELinux machines), you'd have to patch the test suites and it still wouldn't ensure an automatic cleanup -- on unexpected test suite terminations the dir in which the tablespace resides would vanish, but postgres would still expect one there, which might even create further problems (especially on re-emerge). I wouldn't recommend using this approach even when accessing the host postgres. On top of that, many postgres installations with reasonably secure configuration wouldn't grant portage access anyway. As it isn't hard at all to run a separate postgres instance (upstream is explicitly supporting it), I'd strongly recommend doing so even with network-sandbox being disabled. -- Regards, Luis Ressel signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: using .xz for doc/man/info compression
On Tue, 13 May 2014 07:55:56 +0200 Ulrich Mueller wrote: On Tue, 13 May 2014, Andrew Savchenko wrote: Please consider that by default du shows block size, not byte size. Than means that if file is actually 1234 bytes large, without -b it will be still accounted for 4096 bytes on 4K-block filesystem. This raises another question, namely if files with = 4096 bytes size should be compressed at all? Portage already has a fixed size limit of 128 bytes (see bug 169260), but maybe this could be made configurable. In no doubt this limit should be configurable, because defaults fine for one setup may harm another. If we are trying to consider all possible cases, some filesystems may benefit even from compression of very small files (e.g. from 140 to 100 bytes) due to packing of multiple small files in the same inode. ReiserFS is a good example, but more may be somewhere there. If we are trying to consider a majority of users (and thus to select reasonable defaults), from disk usage + decompression overhead point of view it will be the best to store compressed files if they are at least one filesystem block smaller than original file. FS block size may be extracted runtime for any man or doc, or alike directory used, so this is doable. But this approach may overcomplicate implementation. Best regards, Andrew Savchenko pgpD0q4Q5vY8_.pgp Description: PGP signature
Re: [gentoo-dev] RFC: using .xz for doc/man/info compression
On Tue, May 13, 2014 at 7:01 AM, Andrew Savchenko birc...@gmail.com wrote: If we are trying to consider all possible cases, some filesystems may benefit even from compression of very small files (e.g. from 140 to 100 bytes) due to packing of multiple small files in the same inode. ReiserFS is a good example, but more may be somewhere there. Btrfs also supports file inlining, so every byte saved on small files does actually help (I believe the data structure that stores the inlined data doesn't have a fixed record size). Then again, btrfs also supports lzo compression and I believe this is fairly widely used, so I'm not sure that the impact of not compressing small files will be felt. I don't think ext4 supports inlining, but I see some discussions of attempts to add it. For VERY small files I would think that overhead would become an issue. Unless we have a bunch of 30-byte man pages I'd think that both simplicity and some potential for utility would lead us to use the best algorithm possible. Rich
Re: [gentoo-dev] RFC: using .xz for doc/man/info compression
On Tue, 13 May 2014, Rich Freeman wrote: Btrfs also supports file inlining, so every byte saved on small files does actually help (I believe the data structure that stores the inlined data doesn't have a fixed record size). Then again, btrfs also supports lzo compression and I believe this is fairly widely used, so I'm not sure that the impact of not compressing small files will be felt. I don't think ext4 supports inlining, but I see some discussions of attempts to add it. For VERY small files I would think that overhead would become an issue. Unless we have a bunch of 30-byte man pages I'd think that both simplicity and some potential for utility would lead us to use the best algorithm possible. Compression for very small files was systematically studied by vapier in bug 169260, which led to the current threshold of 128 bytes. Files smaller than that usually don't compress at all. Ulrich pgp1gkikB2hcQ.pgp Description: PGP signature
Re: [gentoo-dev] RFC: using .xz for doc/man/info compression
On Tue, 13 May 2014 06:08:52 +0400 Andrew Savchenko birc...@gmail.com wrote: 1. How tools like man or info are supposed to work with such bundle? They are not expecting to have multiple man/info files into single xz bundle. Hmm, true; they would need to be adapted, which involves talking to upstream. Benchmarking and working on this would be a ML thread on its own; it would be nice to check it out, if someone is interested in it. 2. This will put a stress on decompression procedure: in order to extract one file whole xz will have to be docompressed. This would only work for compression algorithms that would allow you to seek to a specific file to extract; I don't know about xz, but from a quick look at the man page it indeed seems that that isn't possible. Some algorithms tend to work better for smaller files, whereas others work better for larger files; might this be the case for bzip2 vs. xz? It doesn't really matter because small files will still require one filesystem block for majority of users. For people with reiserfs or squashfs this may matter of course. Thank you for sharing this insight. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: enabling ipc-sandbox network-sandbox by default
Dnia 2014-05-13, o godz. 09:28:49 Andrew Savchenko birc...@gmail.com napisał(a): I tried network-sandbox — this is a disaster. It brokes distcc completely since distcc client can't connect to remote servers (and even to a local one if any). Calling something a disaster just because it breaks one thing is unpleasant at least. This is a corner case with no good solution at the moment. Though it is entirely doable to make FEATURES=distcc and FEATURES=network-sandbox conflict, or the former disable the latter implicitly. As Rich noted, we do not enable distcc by default so there's no reason why we can't enable conflicting options by default. As far as distcc is concerned, I don't currently have time to work on it. This would probably require some kind of bridging -- or allowing distcc processes to escape the sandbox. -- Best regards, Michał Górny signature.asc Description: PGP signature
[gentoo-dev] Re: RFC: using .xz for doc/man/info compression
Rich Freeman posted on Tue, 13 May 2014 08:18:25 -0400 as excerpted: Btrfs also supports file inlining, so every byte saved on small files does actually help (I believe the data structure that stores the inlined data doesn't have a fixed record size). There's an option for it, altho I've not screwed with it and don't know the default without looking it up. The overall metadata node size (set at mkfs.btrfs time) originally defaulted to the filesystem block size, which is the memory page size, thus 4096 bytes on x86/amd64 and I believe arm. However, the metadata node size default recently changed to 16KiB (or page size where that is larger than 16KiB), altho I'd guess there's still more 4KiB node size users due to all the legacy btrfs out there, but 16KiB will certainly be the majority at some point. Individual file inline size is certainly smaller than metadata node size, but again, I've not messed with that so don't know the actual default for it. Then again, btrfs also supports lzo compression and I believe this is fairly widely used, so I'm not sure that the impact of not compressing small files will be felt. Of course there's gzip as well, and it's the (now legacy) default if compression is specified but not type, altho lzo is recommended as faster with good enough compression. The other factor to consider is replication mode. On a single device filesystem data replication mode is single by default, with metadata dup (two copies), except on detected ssd, where the metadata default is (somewhat controversially) single due to some ssds doing internal deduplication. On multi-device filesystems the metadata default is (two- copy, regardless of the number of devices) raid1, while the data default remains single. So from a size perspective, assuming defaults of single data, dup or raid1 metadata, uncompressed, the cutover should be near 2048 bytes, since under that, duplicated metadata inlining will still be smaller than the 4096 byte data block size, while over that, sticking it in a single- mode data extent should be more efficient. Bottom line, there's enough btrfs variables including inlining size, data vs. metadata replication modes, metadata node sizes and compression and compression type, and the chances that gentoo btrfs users are likely to be tweaking at least one of those variables is high enough, that I'm not sure a generic ideal cutover makes a lot of sense, but to the extent that there is one, it's likely to be near 2048 bytes. FWIW I believe I'm still using portage bzip2 docs compression by default here, altho in the context of this thread I should really examine that since I use compress=lzo at the filesystem level. Both data and metadata are raid1 here, so inlining doesn't matter except that AFAIK inlining is NOT compressed while data extents can be, so portage level compression is likely to make even less difference if it's in the range that portage level bzip2 compression makes it small enough to be inlined, vs not portage level compressed but then big enough to not be inlined, thus btrfs-level transparent lzo compressed as a data extent. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
[gentoo-dev] Re: The gx86-multilib project needs your help! (+ roadmap reminder)
El dom, 11-05-2014 a las 20:56 +0200, Michał Górny escribió: [...] 4. whenever possible, depend on the specific subslot that is known to provide SONAME equal to the required by your package, e.g. for libgcrypt.so.20 you depend on libgcrypt:0/20, [...] Why is this needed? Thanks for the explanation :)
Re: [gentoo-dev] Re: The gx86-multilib project needs your help! (+ roadmap reminder)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 13/05/14 03:19 PM, Pacho Ramos wrote: El dom, 11-05-2014 a las 20:56 +0200, Michał Górny escribió: [...] 4. whenever possible, depend on the specific subslot that is known to provide SONAME equal to the required by your package, e.g. for libgcrypt.so.20 you depend on libgcrypt:0/20, [...] Why is this needed? Thanks for the explanation :) I believe that this relates to cases of, by example, a 32bit pre-rolled binary package on amd64. I expect one could use a specific version range (upper/lower) for the dependency instead, so long as it is just as accurate. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlNycTQACgkQ2ugaI38ACPBswQEAugV6KeFBCPz30cBLeG7Ebsnn Y0C8E4bG1WxSY5JNSfIA/2vEMZPdcbFMnfpHI0AEqvo7YkreXzvvo+oE6t0mlrI/ =W3gq -END PGP SIGNATURE-
[gentoo-dev] Re: [gentoo-commits] gentoo-x86 commit in dev-vcs/cvs: cvs-1.12.12-r6.ebuild cvs-1.12.12-r8.ebuild cvs-1.12.12-r10.ebuild cvs-1.11.23.ebuild ChangeLog cvs-1.12.12-r7.ebuild cvs-1.12.12-r9
On Tue, May 13, 2014 at 07:26:46PM +, Jeroen Roovers (jer) wrote: jer 14/05/13 19:26:46 Modified: cvs-1.12.12-r6.ebuild cvs-1.12.12-r8.ebuild cvs-1.12.12-r10.ebuild cvs-1.11.23.ebuild ChangeLog cvs-1.12.12-r7.ebuild cvs-1.12.12-r9.ebuild Log: There is no such thing as FEATURES=maketest. Showing my age as a dev, but there used to be. FEATURES=maketest runs 'make test' if the Makefiles would automatically do so, whereas FEATURES=test supports any custom code in src_test. Since early 2006, they are no longer different. To see the full history of why they both existed, you'll need to find the old Portage SVN repo, as the Git migration was done without the history prior to August 28 2005. Anyway, I'm ripping that out entirely to replace with RESTRICT=test. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
Re: [gentoo-dev] RFC: using .xz for doc/man/info compression
On Tue, 13 May 2014 08:18:25 -0400 Rich Freeman wrote: On Tue, May 13, 2014 at 7:01 AM, Andrew Savchenko birc...@gmail.com wrote: If we are trying to consider all possible cases, some filesystems may benefit even from compression of very small files (e.g. from 140 to 100 bytes) due to packing of multiple small files in the same inode. ReiserFS is a good example, but more may be somewhere there. Btrfs also supports file inlining, so every byte saved on small files does actually help (I believe the data structure that stores the inlined data doesn't have a fixed record size). Then again, btrfs also supports lzo compression and I believe this is fairly widely used, so I'm not sure that the impact of not compressing small files will be felt. I did not meant inlining. I was talking about block suballocation which allows to store small files in underused blocks of another files: http://en.wikipedia.org/wiki/Block_suballocation I don't think ext4 supports inlining, but I see some discussions of attempts to add it. Ext4 supports inlining for files up to 59 bytes: https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Inline_Data For VERY small files I would think that overhead would become an issue. Unless we have a bunch of 30-byte man pages I'd think that both simplicity and some potential for utility would lead us to use the best algorithm possible. Agreed, though performance should be considered still. I doubt paq8l -9 will be used for this task, though it is about 1.5 times more effective than xz -9e on text files, even on small ones like man pages; on large files it is at least 2 times better. Best regards, Andrew Savchenko pgpHZlB7ZRZ7U.pgp Description: PGP signature
Re: [gentoo-dev] RFC: using .xz for doc/man/info compression
On 12 May 2014 21:35, Tom Wijsman tom...@gentoo.org wrote: What about putting multiple doc / man / info files in a single .xz file for each package? How would one use them if they're installed as a single .xz file per package? Is there a trick that exists to allow this to even work for man man ? I'm guessing you *could* have an extension wrapper that handles a symlink to such a file to extract the desired content, but that seems messy. ( ie: less /path/to/bar.polyxz = polyxz /path/to/bar.polyxz = polyxz reads symlink target, decodes the .xz file and returns bar from it based on the symlink source name ) Though I'd imagine that would mitigate the marginal savings made by unifying them as a single file by needing extra file system allocations to house the symlinks. -- Kent
Re: [gentoo-dev] Re: [gentoo-commits] gentoo-x86 commit in dev-vcs/cvs: cvs-1.12.12-r6.ebuild cvs-1.12.12-r8.ebuild cvs-1.12.12-r10.ebuild cvs-1.11.23.ebuild ChangeLog cvs-1.12.12-r7.ebuild cvs-1.12.1
On Wed, 14 May 2014 00:11:21 + Robin H. Johnson robb...@gentoo.org wrote: Anyway, I'm ripping that out entirely to replace with RESTRICT=test. Great. jer
