Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On 03 Jun 2015 10:26, Alexis Ballier wrote: On Tue, 2 Jun 2015 10:13:54 -0400 Mike Frysinger wrote: On 01 Jun 2015 10:15, Alexis Ballier wrote: On Sun, 31 May 2015 11:17:50 -0400 Mike Frysinger wrote: (3) considering the glibc effort has been stalled for over a year, (1) is something we can help accomplish and make reasonable progress on (4) glibc isn't the only one that implements LFS in a transparent backwards compatible manner maybe the fact that some file operations are broken with glibc's default settings on a somewhat popular fs would be a good argument to un-stall it ? it isn't that black white. while for many packages the change would be harmless, glibc is more conservative than that. it can be shown that a not uncommon coding style is to use long's everywhere when dealing with the fs. when sizeof(off_t)==sizeof(long), this works out. when off_t is suddenly twice as large, then you can get truncation at best (which is kind of the status quo) and random corruption at worse (like an API that takes pointers to off_t's but the caller passed in a long). if you want to join the list upstream and contribute to the discussion, you're more than welcome to. the maintainers aren't clueless -- they fully understand the trade-offs we're discussing here and know that 32bit fs accesses lead to failures (including stat). since sitting on our hands and hoping for the best continues to accomplish nothing, getting the various upstream packages to opt-in to LFS themselves can make real progress now. agreed, thanks for the explanation i take it as the main goal of these qa warnings is to fix the above mentioned half broken packages and that pushing lfs flags everywhere is a slow but safe way to achieve it :) it'd be interesting to have data about how many packages can break; my belief is that more than half the c/c++ programs use one way or another fs ops but maybe less than one in a thousand will break; i can understand this is still too much for a libc (even though other 'breaking fixes' weren't so controversial in the past), and we'll see after gathering the data your qa warnings will provide let's ignore the open/read/write issue for now on the assumption it's more uncommon to incidentally have that break. instead, let's look at things that can hit inodes -- stat dir listing. a file simply being allocated a 64bit inode will break an app -- both directly stating and even trying to list the dir. oops! :) here's the command i'm using for just inode funcs: scanelf / -mqRygs \ '^((__)?[fl]?x?stat(v?fs)?(at)?|alphasort|getdirentries|readdir(_r)?|scandir(at)?|versionsort)$' you'd have to ignore glibc and sandbox as they provide both 32bit 64bit. i have an x86 chroot for random testing, but it's not as full as my desktop. it has a measily 245 packages installed. of those: - 25 are using 32bit inode functions - 34 are using some 32bit fs interface (this is a superset of the above) if someone had a fuller install of like 1000 pkgs, that'd be interesting. as i already said, changing the default in *glibc* doesn't help non-glibc systems. even with ten different libc's, the order of magnitude is still smaller for changing the default :) true. each fight though will be significantly higher i think, assuming you can even find someone to talk to (as not all are open source projects). and it will take longer for the fix to trickle out to deployments. -mike signature.asc Description: Digital signature
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On Tue, 2 Jun 2015 10:13:54 -0400 Mike Frysinger vap...@gentoo.org wrote: On 01 Jun 2015 10:15, Alexis Ballier wrote: On Sun, 31 May 2015 11:17:50 -0400 Mike Frysinger wrote: (3) considering the glibc effort has been stalled for over a year, (1) is something we can help accomplish and make reasonable progress on (4) glibc isn't the only one that implements LFS in a transparent backwards compatible manner maybe the fact that some file operations are broken with glibc's default settings on a somewhat popular fs would be a good argument to un-stall it ? it isn't that black white. while for many packages the change would be harmless, glibc is more conservative than that. it can be shown that a not uncommon coding style is to use long's everywhere when dealing with the fs. when sizeof(off_t)==sizeof(long), this works out. when off_t is suddenly twice as large, then you can get truncation at best (which is kind of the status quo) and random corruption at worse (like an API that takes pointers to off_t's but the caller passed in a long). if you want to join the list upstream and contribute to the discussion, you're more than welcome to. the maintainers aren't clueless -- they fully understand the trade-offs we're discussing here and know that 32bit fs accesses lead to failures (including stat). since sitting on our hands and hoping for the best continues to accomplish nothing, getting the various upstream packages to opt-in to LFS themselves can make real progress now. agreed, thanks for the explanation i take it as the main goal of these qa warnings is to fix the above mentioned half broken packages and that pushing lfs flags everywhere is a slow but safe way to achieve it :) it'd be interesting to have data about how many packages can break; my belief is that more than half the c/c++ programs use one way or another fs ops but maybe less than one in a thousand will break; i can understand this is still too much for a libc (even though other 'breaking fixes' weren't so controversial in the past), and we'll see after gathering the data your qa warnings will provide which leads me to the next part ... my first suggestion in the tracker is for autotool based projects to use AC_SYS_LARGEFILE because: (a) it supports a variety of systems (b) as new systems come up or bugs are found or whatever, the autoconf macro will improve and people eventually get those fixes for free (c) it does it all transparently for you -- add the macro and you're done (d) it fixes the package for all users, new old the reason i listed only the raw CPPFLAGS and autoconf macros are because those are the two i'm familiar with. i don't know how other build systems (e.g. cmake) support this (assuming they do at all). if people have other recipes on hand, then it would be great to collect more there. yes, but that is not what i am discussing: unless i missed something, they'll all end up one way or another adding the relevant defines; whether it is done with an m4 macro, append-lfs-flags, a cmake function or what else is an implementation detail of little interest to me trying to understand why we don't simply change the default :) as i already said, changing the default in *glibc* doesn't help non-glibc systems. even with ten different libc's, the order of magnitude is still smaller for changing the default :) Alexis.
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On Sun, 31 May 2015 11:17:50 -0400 Mike Frysinger vap...@gentoo.org wrote: well if we're going to do arbitrary lists ;) (1) your options aren't mutually exclusive (2) implementing both are desirable good to know your longterm plan :) however, even if both can be done, i still don't see the point of going through patching if we end up changing the default anyway. (3) considering the glibc effort has been stalled for over a year, (1) is something we can help accomplish and make reasonable progress on (4) glibc isn't the only one that implements LFS in a transparent backwards compatible manner maybe the fact that some file operations are broken with glibc's default settings on a somewhat popular fs would be a good argument to un-stall it ? which leads me to the next part ... my first suggestion in the tracker is for autotool based projects to use AC_SYS_LARGEFILE because: (a) it supports a variety of systems (b) as new systems come up or bugs are found or whatever, the autoconf macro will improve and people eventually get those fixes for free (c) it does it all transparently for you -- add the macro and you're done (d) it fixes the package for all users, new old the reason i listed only the raw CPPFLAGS and autoconf macros are because those are the two i'm familiar with. i don't know how other build systems (e.g. cmake) support this (assuming they do at all). if people have other recipes on hand, then it would be great to collect more there. -mike yes, but that is not what i am discussing: unless i missed something, they'll all end up one way or another adding the relevant defines; whether it is done with an m4 macro, append-lfs-flags, a cmake function or what else is an implementation detail of little interest to me trying to understand why we don't simply change the default :) Alexis.
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On May 31, 2015 7:33:28 AM PDT, Alexis Ballier aball...@gentoo.org wrote: I'm not sure what's best for every one: 1. Push hundreds of patches upstream to add lfs flags; 2. Apply your patch to our glibc ebuilds, fix the corner cases, and go back to glibc upstream with these data. If the changes are made to glibc, would these be under a new symbol version for ABI compatibility, or just be changes to headers to make _FILE_OFFSET_BITS=64 the default? If not, what about binary software? Not saying you haven’t considered the relevant issues; I just haven’t seen binary software brought up on this list yet. -- Christopher Head
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On 31 May 2015 11:58, Mike Gilbert wrote: On Sat, May 30, 2015 at 2:54 PM, Mike Frysinger wrote: we've got a new QA check that warns whenever a package is built using a 32bit filesystem interface. in practice, this applies to arm/mips/ppc/sh/x86 systems (not including multilib -- for now). this topic has come up in Gentoo a few times over the years but we've never really amassed the will power to fix it. instead we fix it in one-offs based on user reports (like can't download 4GiB file with ftp #101038). this was worked well enough because most users have moved on to 64bit systems and the interaction with 2GiB files tends to correlate with a few packages. however, recent winds have started blowing where file systems are utilizing 64bit inodes to handle large file counts. this means apps that do even basic things like stat() will now fail. the number of applications that this affects is significantly higher, although still relegated to systems that happen to use a file system with 64bit inodes. Does this issue affect software that only reads/writes small files and never calls stat? For example, pkg-config. it is possible to use some file system operations and never run into problems. the trouble is that it requires diligence and careful auditing, and you have to make sure your interactions with other packages are also OK. considering how bad/unaware the wider community is wrt LFS, ABI compatibility, and such critical but oft overlooked issues, i suspect most people also won't have the will power/capabilities to keep up here. conversely, flip the autoconf/cppflags magic, and most projects Just Work. It might still be nice to adjust such packages for consistency, but it might be harder to justify patches to upstream developers. pkg-config already merged it and it's already been pulled back into Gentoo ;) https://bugs.freedesktop.org/show_bug.cgi?id=90078 https://bugs.gentoo.org/550508 that said, i'm not entirely sure why there would be a lot of push back. when i gathered data on this years ago (which i no longer have, so i'd have to recreate), the increased foot print and runtime cost was not significant. especially when you consider that we're talking about I/O operations which inherently kill CPU performance due to stalls. -mike signature.asc Description: Digital signature
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On 31 May 2015 16:33, Alexis Ballier wrote: On Sun, 31 May 2015 10:17:02 -0400 Mike Frysinger wrote: On 31 May 2015 15:52, Alexis Ballier wrote: On Sun, 31 May 2015 13:50:49 +0200 Diego Elio Pettenò wrote: On 31 May 2015 at 12:59, Alexis Ballier wrote: nice, but can't we add the lfs flags to our default toolchain flags or even better patch glibc headers to always redefine these functions to the 64bits variants? No, because that can easily break ABI of programs that actually want the non-LFS one (for instance anything that wraps around function calls, including but not limited to padsp, aoss, and similar wrappers.) This seems easily fixed with an opt-out for lfs flags that such programs can use. They'll need to be touched to disable the QA warning anyway. this is a discussion for upstream toolchain packages (largely glibc) and in fact i started such a heretical thread over a year ago. it was not well received due to the implicit/silent ABI change that new builds would receive. glibc likes to be conservative as it is the foundation of everything. so unless glibc changes, updating our copy of glibc would only somewhat help our users. conversely, getting the changes pushed to the respective upstream would help everyone. I'm not sure what's best for every one: 1. Push hundreds of patches upstream to add lfs flags; 2. Apply your patch to our glibc ebuilds, fix the corner cases, and go back to glibc upstream with these data. Least to say is that I'm not a big fan of lfs flags as done in glibc and I certainly prefer 2., but you'd know better. well if we're going to do arbitrary lists ;) (1) your options aren't mutually exclusive (2) implementing both are desirable (3) considering the glibc effort has been stalled for over a year, (1) is something we can help accomplish and make reasonable progress on (4) glibc isn't the only one that implements LFS in a transparent backwards compatible manner which leads me to the next part ... my first suggestion in the tracker is for autotool based projects to use AC_SYS_LARGEFILE because: (a) it supports a variety of systems (b) as new systems come up or bugs are found or whatever, the autoconf macro will improve and people eventually get those fixes for free (c) it does it all transparently for you -- add the macro and you're done (d) it fixes the package for all users, new old the reason i listed only the raw CPPFLAGS and autoconf macros are because those are the two i'm familiar with. i don't know how other build systems (e.g. cmake) support this (assuming they do at all). if people have other recipes on hand, then it would be great to collect more there. -mike signature.asc Description: Digital signature
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On Sat, May 30, 2015 at 2:54 PM, Mike Frysinger vap...@gentoo.org wrote: we've got a new QA check that warns whenever a package is built using a 32bit filesystem interface. in practice, this applies to arm/mips/ppc/sh/x86 systems (not including multilib -- for now). this topic has come up in Gentoo a few times over the years but we've never really amassed the will power to fix it. instead we fix it in one-offs based on user reports (like can't download 4GiB file with ftp #101038). this was worked well enough because most users have moved on to 64bit systems and the interaction with 2GiB files tends to correlate with a few packages. however, recent winds have started blowing where file systems are utilizing 64bit inodes to handle large file counts. this means apps that do even basic things like stat() will now fail. the number of applications that this affects is significantly higher, although still relegated to systems that happen to use a file system with 64bit inodes. Does this issue affect software that only reads/writes small files and never calls stat? For example, pkg-config. It might still be nice to adjust such packages for consistency, but it might be harder to justify patches to upstream developers.
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On Sun, May 31, 2015 at 12:50 PM, Mike Frysinger vap...@gentoo.org wrote: On 31 May 2015 11:58, Mike Gilbert wrote: It might still be nice to adjust such packages for consistency, but it might be harder to justify patches to upstream developers. pkg-config already merged it and it's already been pulled back into Gentoo ;) https://bugs.freedesktop.org/show_bug.cgi?id=90078 https://bugs.gentoo.org/550508 that said, i'm not entirely sure why there would be a lot of push back. when i gathered data on this years ago (which i no longer have, so i'd have to recreate), the increased foot print and runtime cost was not significant. especially when you consider that we're talking about I/O operations which inherently kill CPU performance due to stalls. -mike Sure, I don't really expect much resistance. I just like to be prepared in the event some developer starts asking questions.
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On Sat, 30 May 2015 14:54:42 -0400 Mike Frysinger vap...@gentoo.org wrote: an example warning message: * QA Notice: The following files were not built with LFS support: * Please file a bug at http://bugs.gentoo.org/ and mark it as a blocker of 471102. * See that tracker bug (https://bugs.gentoo.org/471102) for more details. * fopen@@GLIBC_2.1 bin/route * fopen@@GLIBC_2.1 bin/ifconfig * fopen@@GLIBC_2.1 bin/hostname * __fxstat@@GLIBC_2.0,open@@GLIBC_2.0,fopen@@GLIBC_2.1,readdir@@GLIBC_2.0 bin/netstat * fopen@@GLIBC_2.1 sbin/nameif * fopen@@GLIBC_2.1 sbin/ipmaddr * fopen@@GLIBC_2.1 sbin/arp * fopen@@GLIBC_2.1 sbin/iptunnel * fopen@@GLIBC_2.1 sbin/rarp * open@@GLIBC_2.0,creat@@GLIBC_2.0,fopen@@GLIBC_2.1 sbin/slattach nice, but can't we add the lfs flags to our default toolchain flags or even better patch glibc headers to always redefine these functions to the 64bits variants? I don't understand why one should add append-lfs-flags to almost every single package out there. Freebsd has been doing this since version 7. And even dropped the 32bits symbols when going to libc.so.7 I think. Some packages might break with such a change, but they're already half-broken according to your e-mail, so those are the ones that should be fixed. Alexis.
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On 31 May 2015 at 12:59, Alexis Ballier aball...@gentoo.org wrote: nice, but can't we add the lfs flags to our default toolchain flags or even better patch glibc headers to always redefine these functions to the 64bits variants? No, because that can easily break ABI of programs that actually want the non-LFS one (for instance anything that wraps around function calls, including but not limited to padsp, aoss, and similar wrappers.) FreeBSD has removed the symbols on an ABI bump, which hopefully could happen for glibc in a far and remote future, too. But as long as the symbols are there, the defines shouldn't be forced. Mike, thanks for doing this, it has been a pain in my shoe since 2008 https://blog.flameeyes.eu/2008/11/who-wants-to-support-largefile. I'm looking forward to the protests that 64-bit inodes don't exist, though. Diego Elio Pettenò — Flameeyes https://blog.flameeyes.eu/
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
150531 Diego Elio Pettenò wrote: Mike, thanks for doing this, it has been a pain in my shoe since 2008 https://blog.flameeyes.eu/2008/11/who-wants-to-support-largefile. Do users with 64-bit systems have to pay attention to this ? -- ,, SUPPORT ___//___, Philip Webb ELECTRIC /] [] [] [] [] []| Cities Centre, University of Toronto TRANSIT`-O--O---' purslowatchassdotutorontodotca
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On 31 May 2015 at 14:33, Philip Webb purs...@ca.inter.net wrote: Do users with 64-bit systems have to pay attention to this ? Only as far as multilib is concerned. The 64-bit ABIs are (AFAIR) all LFS with no opt-out. x86-64 is definitely LFS. Diego Elio Pettenò — Flameeyes https://blog.flameeyes.eu/
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On 31 May 2015 12:59, Alexis Ballier wrote: I don't understand why one should add append-lfs-flags to almost every single package out there. no one is suggesting that route. append-lfs-flags is a kludge (albeit an effective one) in the face of no upstream support. if you look at the referenced tracker it includes pointers for fixing packages in ways that can be sent upstream. -mike signature.asc Description: Digital signature
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On Sun, 31 May 2015 09:46:30 -0400 Mike Frysinger vap...@gentoo.org wrote: On 31 May 2015 12:59, Alexis Ballier wrote: I don't understand why one should add append-lfs-flags to almost every single package out there. no one is suggesting that route. append-lfs-flags is a kludge (albeit an effective one) in the face of no upstream support. if you look at the referenced tracker it includes pointers for fixing packages in ways that can be sent upstream. well, other suggestions are similar, except of course that appending the flags is done upstream in some way or another however, reading more in details the links you provided, what happened to your patch setting _FILE_OFFSET_BITS default to 64 ? ( https://sourceware.org/ml/libc-alpha/2014-03/msg00290.html ) which seems to be a much better solution to what i initially proposed.
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On 31 May 2015 15:52, Alexis Ballier wrote: On Sun, 31 May 2015 13:50:49 +0200 Diego Elio Pettenò wrote: On 31 May 2015 at 12:59, Alexis Ballier wrote: nice, but can't we add the lfs flags to our default toolchain flags or even better patch glibc headers to always redefine these functions to the 64bits variants? No, because that can easily break ABI of programs that actually want the non-LFS one (for instance anything that wraps around function calls, including but not limited to padsp, aoss, and similar wrappers.) This seems easily fixed with an opt-out for lfs flags that such programs can use. They'll need to be touched to disable the QA warning anyway. this is a discussion for upstream toolchain packages (largely glibc) and in fact i started such a heretical thread over a year ago. it was not well received due to the implicit/silent ABI change that new builds would receive. glibc likes to be conservative as it is the foundation of everything. so unless glibc changes, updating our copy of glibc would only somewhat help our users. conversely, getting the changes pushed to the respective upstream would help everyone. -mike signature.asc Description: Digital signature
Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
On Sun, 31 May 2015 10:17:02 -0400 Mike Frysinger vap...@gentoo.org wrote: On 31 May 2015 15:52, Alexis Ballier wrote: On Sun, 31 May 2015 13:50:49 +0200 Diego Elio Pettenò wrote: On 31 May 2015 at 12:59, Alexis Ballier wrote: nice, but can't we add the lfs flags to our default toolchain flags or even better patch glibc headers to always redefine these functions to the 64bits variants? No, because that can easily break ABI of programs that actually want the non-LFS one (for instance anything that wraps around function calls, including but not limited to padsp, aoss, and similar wrappers.) This seems easily fixed with an opt-out for lfs flags that such programs can use. They'll need to be touched to disable the QA warning anyway. this is a discussion for upstream toolchain packages (largely glibc) and in fact i started such a heretical thread over a year ago. it was not well received due to the implicit/silent ABI change that new builds would receive. glibc likes to be conservative as it is the foundation of everything. so unless glibc changes, updating our copy of glibc would only somewhat help our users. conversely, getting the changes pushed to the respective upstream would help everyone. I'm not sure what's best for every one: 1. Push hundreds of patches upstream to add lfs flags; 2. Apply your patch to our glibc ebuilds, fix the corner cases, and go back to glibc upstream with these data. Least to say is that I'm not a big fan of lfs flags as done in glibc and I certainly prefer 2., but you'd know better. Alexis.
[gentoo-dev] LFS QA warnings coming soon to a build near you
we've got a new QA check that warns whenever a package is built using a 32bit filesystem interface. in practice, this applies to arm/mips/ppc/sh/x86 systems (not including multilib -- for now). this topic has come up in Gentoo a few times over the years but we've never really amassed the will power to fix it. instead we fix it in one-offs based on user reports (like can't download 4GiB file with ftp #101038). this was worked well enough because most users have moved on to 64bit systems and the interaction with 2GiB files tends to correlate with a few packages. however, recent winds have started blowing where file systems are utilizing 64bit inodes to handle large file counts. this means apps that do even basic things like stat() will now fail. the number of applications that this affects is significantly higher, although still relegated to systems that happen to use a file system with 64bit inodes. since our QA warnings in portage have historically proven themselves to (1) get users to generate bug reports and (2) get the warnings fixed (both here and in upstream), there is now a QA check for this issue. the plan is to let this bake in portage- for a while before it gets kicked into ~arch and the flood gates are unleashed. more background details (including recipes for fixing the issue in your package) can be found in the tracker: https://bugs.gentoo.org/471102 an example warning message: * QA Notice: The following files were not built with LFS support: * Please file a bug at http://bugs.gentoo.org/ and mark it as a blocker of 471102. * See that tracker bug (https://bugs.gentoo.org/471102) for more details. * fopen@@GLIBC_2.1 bin/route * fopen@@GLIBC_2.1 bin/ifconfig * fopen@@GLIBC_2.1 bin/hostname * __fxstat@@GLIBC_2.0,open@@GLIBC_2.0,fopen@@GLIBC_2.1,readdir@@GLIBC_2.0 bin/netstat * fopen@@GLIBC_2.1 sbin/nameif * fopen@@GLIBC_2.1 sbin/ipmaddr * fopen@@GLIBC_2.1 sbin/arp * fopen@@GLIBC_2.1 sbin/iptunnel * fopen@@GLIBC_2.1 sbin/rarp * open@@GLIBC_2.0,creat@@GLIBC_2.0,fopen@@GLIBC_2.1 sbin/slattach -mike signature.asc Description: Digital signature
