Bug#871956: lintian: false positive: binary-file-built-without-LFS-support on x32
On Fri, 2017-09-01 at 15:02:03 +0100, Chris Lamb wrote: > > I'd suggest just marking what we have an idea about. It's not > > like lintian warnings matter for something that's not in the archive... > > That's true. > > (Implementation braindump for anyone interested in jumping in: we > could simply blacklist x32 directly in checks/binaries.pm, but we > have data/common/architectures so we might need to add a field there > and update Lintian::Architecture. The testsuite will need some > cleverness so that we don't fail under x32.) I think this information is somewhat already provided by the dpkg arch tables. Any architecture with a 64-bit CPU and with a 32-bit ABI, as defined by the cputable and abitable files. So this would include x32, arm64ilp32 and the mipsn32* arches. AFAIR the OpenBSD port on 32-bit CPUs would be affected too as they did a scorched-earth migration to 64-bit types ignoring any backwards binary compatibility. But then I guess we can just ignore that one. :) Thanks, Guillem
Bug#871956: lintian: false positive: binary-file-built-without-LFS-support on x32
hi Guillem, all, On Sun, 3 Sep 2017, Guillem Jover wrote: When you define _FILE_OFFSET_BITS=64 the interfaces are transparently mapped to the 64-bit variants and no other code change is required. So you *must* not be switching from foo to foo64 in the call sites. This is IMO the preferred way to enable LFS. Do you mean that the preferred way is to always hardwire _FILE_OFFSET_BITS to 64? Or is using AC_SYS_LARGEFILE, which seems to me to *not* set this on amd64, also a long-term sustainable solution? (To me it seems that if some systems do not need to define it to 64, and if AC_SYS_LARGEFILE handles this correctly, then it's best to allow AC_SYS_LARGEFILE to do this.) Nope, that looks wrong. The __USE_* macros are internal implementation details, and applications should not be using those. In addition you should just be using pread() and pwrite(), which will call the correct interfaces. OK, thanks for the correction. [So presumably this means that the only bug in mpgrafic-0.3.15-1 in terms of LFS is the failure to #include config.h.] That patch needs to be reverted. I can provide a new patch explaining in more detail this though and giving more pointers perhaps. Please go ahead - the documentation bug is here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=874078 Cheers Boud
Bug#871956: lintian: false positive: binary-file-built-without-LFS-support on x32
On Sun, Sep 03, 2017 at 11:38:28PM +0200, Guillem Jover wrote: > On Fri, 2017-09-01 at 21:35:07 +0200, Adam Borowski wrote: > > The problem is in snowflake packages that do things their own way and enable > > LFS only when it's actually needed. Here's where the lintian false positive > > triggers. > > This is a common misconception. Pretty much all programs need to > be LFS enabled, because even if these programs do not handle large > files they will probably still fail when encountering files in the > filsystem with large inode numbers (or other large metadata) for > example. By "if needed", I meant "on archs where off_t != off64_t". There's neither any benefit nor any harm in enabling LFS on amd64 or x32, no matter if a program uses anything LFSey or not. I agree with you that there's no good reason with skipping LFS on i386. This bug log includes some confusion with an unrelated problem with mpgrafic on i386; my original report is about too smart packages that don't switch to *64 variants if sizeof(off_t) is already 8 -- this triggers a lintian warning on x32 (and, unconfirmed, on arm64ilp32). Not sure if you'd want to clone this bug for the other problem (true positive in mpgrafic and not helpful enough lintian message). Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢰⠒⠀⣿⡁ Vat kind uf sufficiently advanced technology iz dis!? ⢿⡄⠘⠷⠚⠋⠀ -- Genghis Ht'rok'din ⠈⠳⣄
Bug#871956: lintian: false positive: binary-file-built-without-LFS-support on x32
On Fri, 2017-09-01 at 21:35:07 +0200, Adam Borowski wrote: > And what AC_SYS_LARGEFILE does, at least on Linux, is to return a hardcoded > setting so programs switch from off_t to off64_t whether they need to or > not. This does the right thing on old 32-bit archs and is harmless on > 64-bit and new 32-bit. Just to make this extra clear, when setting _FILE_OFFSET_BITS=64, the types and functions are transparently redirected to the 64-bit variants if need be, so nothing needs to be changed in the code. > The problem is in snowflake packages that do things their own way and enable > LFS only when it's actually needed. Here's where the lintian false positive > triggers. This is a common misconception. Pretty much all programs need to be LFS enabled, because even if these programs do not handle large files they will probably still fail when encountering files in the filsystem with large inode numbers (or other large metadata) for example. Thanks, Guillem
Bug#871956: lintian: false positive: binary-file-built-without-LFS-support on x32
On Sat, 2017-09-02 at 21:47:15 +0200, Boud Roukema wrote: > On Sat, 2 Sep 2017, Adam Borowski wrote: > > > AC_SYS_LARGEFILE on i386 (linux) for mpgrafic-0.3.15-1 apparently does > > > *not* > > > always do the right thing > > Nope, you use pwrite() not pwrite64() -- on i386 you need the latter to get > > LFS; they're aliases only on amd64, x32 and so on. That's not correct, neither the new changes to the description of the tag. When you define _FILE_OFFSET_BITS=64 the interfaces are transparently mapped to the 64-bit variants and no other code change is required. So you *must* not be switching from foo to foo64 in the call sites. This is IMO the preferred way to enable LFS. The other way to enable LFS is to define _LARGEFILE64_SOURCE, which makes visible the 64-bit variants in parallel to the 32-bit default interfaces, so you get foo and foo64, and you can call either from the code. This might be useful in case you are for example writing a library and need to support entry points for both 32-bit and 64-bit functions for backwards compatibility. Otherwise I'd strongly discourage its use. > OK, so config.h + p(read|write)64 should hopefully solve the mpgrafic > bug. I initially tried hardwiring pwrite64 and pread64 (compiling on > amd64), but got "warning: implicit declaration of function ‘pwrite64’ > [-Wimplicit-function-declaration]". My understanding is that > _FILE_OFFSET_BITS and __USE_FILE_OFFSET64 are *not* set for amd64 by > AC_SYS_LARGEFILE, and instead only set for some systems, such as i386. > The following should correctly work together with AC_SYS_LARGEFILE, it > seems to me. > > Does this look OK? > > #ifndef __USE_FILE_OFFSET64 > i_stat=pwrite(fd,buffer,size,offset); > #else > i_stat=pwrite64(fd,buffer,size,offset); > #endif > > #ifndef __USE_FILE_OFFSET64 > i_stat = pread(fd,buffer,size,offset); > #else > i_stat = pread64(fd,buffer,size,offset); > #endif Nope, that looks wrong. The __USE_* macros are internal implementation details, and applications should not be using those. In addition you should just be using pread() and pwrite(), which will call the correct interfaces. > > But in any case, even if it had been indeed a false positive, it'd be a > > different bug. > > I think I should post a lintian bug for improving the info field for > the LFS unsafe detection - I spent quite a bit of time searching for how > to handle this, and having more detailed information could help other > debian contributors respond more easily to built-without-LFS-support > warnings. That patch needs to be reverted. I can provide a new patch explaining in more detail this though and giving more pointers perhaps. Thanks, Guillem
Bug#871956: lintian: false positive: binary-file-built-without-LFS-support on x32
hi Adam, all, On Sat, 2 Sep 2017, Adam Borowski wrote: AC_SYS_LARGEFILE on i386 (linux) for mpgrafic-0.3.15-1 apparently does *not* always do the right thing: You call it but don't actually use what it returns: mpicc -DHAVE_CONFIG_H -I. -I.. -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/<>=. -fstack-protector-strong -Wformat -Werror=format-security -c -o parallel_io.o parallel_io.c #define _XOPEN_SOURCE 500 #include #include #include #include #include #include #include #include You don't include "config.h" anywhere. You're right - thanks for spotting that :). Nope, you use pwrite() not pwrite64() -- on i386 you need the latter to get LFS; they're aliases only on amd64, x32 and so on. OK, so config.h + p(read|write)64 should hopefully solve the mpgrafic bug. I initially tried hardwiring pwrite64 and pread64 (compiling on amd64), but got "warning: implicit declaration of function ‘pwrite64’ [-Wimplicit-function-declaration]". My understanding is that _FILE_OFFSET_BITS and __USE_FILE_OFFSET64 are *not* set for amd64 by AC_SYS_LARGEFILE, and instead only set for some systems, such as i386. The following should correctly work together with AC_SYS_LARGEFILE, it seems to me. Does this look OK? #ifndef __USE_FILE_OFFSET64 i_stat=pwrite(fd,buffer,size,offset); #else i_stat=pwrite64(fd,buffer,size,offset); #endif #ifndef __USE_FILE_OFFSET64 i_stat = pread(fd,buffer,size,offset); #else i_stat = pread64(fd,buffer,size,offset); #endif But in any case, even if it had been indeed a false positive, it'd be a different bug. I think I should post a lintian bug for improving the info field for the LFS unsafe detection - I spent quite a bit of time searching for how to handle this, and having more detailed information could help other debian contributors respond more easily to built-without-LFS-support warnings. Cheers Boud
Bug#871956: lintian: false positive: binary-file-built-without-LFS-support on x32
On Sat, Sep 02, 2017 at 12:18:59AM +0200, Boud Roukema wrote: > On Fri, 1 Sep 2017, Adam Borowski wrote: > > And what AC_SYS_LARGEFILE does, at least on Linux, is to return a hardcoded > > setting so programs switch from off_t to off64_t whether they need to or > > not. This does the right thing on old 32-bit archs and is harmless on > > 64-bit and new 32-bit. > > AC_SYS_LARGEFILE on i386 (linux) for mpgrafic-0.3.15-1 apparently does *not* > always do the right thing: You call it but don't actually use what it returns: mpicc -DHAVE_CONFIG_H -I. -I.. -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/<>=. -fstack-protector-strong -Wformat -Werror=format-security -c -o parallel_io.o parallel_io.c #define _XOPEN_SOURCE 500 #include #include #include #include #include #include #include #include You don't include "config.h" anywhere. > > N: Processing source package mpgrafic (version 0.3.15-1, arch source) ... > > C: mpgrafic source (0.3.15-1) [source]: debian-build-system dh > > C: mpgrafic source (0.3.15-1) [source]: source-format 3.0 (quilt) > > P: mpgrafic source (0.3.15-1) [source]: debian-watch-may-check-gpg-signature > > N: > > N: Processing binary package mpgrafic (version 0.3.15-1, arch i386) ... > > X: mpgrafic binary (0.3.15-1) [i386]: binary-file-built-without-LFS-support > > usr/bin/mpgrafic > > C: mpgrafic binary (0.3.15-1) [i386]: no-ctrl-scripts > > N: > > N: Processing binary package mpgrafic (version 0.3.15-1, arch amd64) ... > > C: mpgrafic binary (0.3.15-1) [amd64]: no-ctrl-scripts > > N: > > N: Processing binary package mpgrafic-dbgsym (version 0.3.15-1, arch i386) > > ... > > C: mpgrafic-dbgsym binary (0.3.15-1) [i386]: no-ctrl-scripts > > N: > > N: Processing binary package mpgrafic-dbgsym (version 0.3.15-1, arch amd64) > > ... > > https://anonscm.debian.org/git/debian-astro/packages/mpgrafic.git/tree/configure.ac > > https://anonscm.debian.org/git/debian-astro/packages/mpgrafic.git/tree/configure > > This is a lintian false positive with AC_SYS_LARGEFILE. Nope, you use pwrite() not pwrite64() -- on i386 you need the latter to get LFS; they're aliases only on amd64, x32 and so on. But in any case, even if it had been indeed a false positive, it'd be a different bug. -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢰⠒⠀⣿⡁ Vat kind uf sufficiently advanced technology iz dis!? ⢿⡄⠘⠷⠚⠋⠀ -- Genghis Ht'rok'din ⠈⠳⣄
Bug#871956: lintian: false positive: binary-file-built-without-LFS-support on x32
On Fri, 1 Sep 2017, Adam Borowski wrote: And what AC_SYS_LARGEFILE does, at least on Linux, is to return a hardcoded setting so programs switch from off_t to off64_t whether they need to or not. This does the right thing on old 32-bit archs and is harmless on 64-bit and new 32-bit. AC_SYS_LARGEFILE on i386 (linux) for mpgrafic-0.3.15-1 apparently does *not* always do the right thing: https://lintian.debian.org/resources/e617c1e7ba35e94b411994e8320643594559faf6920ef535c71c911eb32657fd.gz N: Processing source package mpgrafic (version 0.3.15-1, arch source) ... C: mpgrafic source (0.3.15-1) [source]: debian-build-system dh C: mpgrafic source (0.3.15-1) [source]: source-format 3.0 (quilt) P: mpgrafic source (0.3.15-1) [source]: debian-watch-may-check-gpg-signature N: N: Processing binary package mpgrafic (version 0.3.15-1, arch i386) ... X: mpgrafic binary (0.3.15-1) [i386]: binary-file-built-without-LFS-support usr/bin/mpgrafic C: mpgrafic binary (0.3.15-1) [i386]: no-ctrl-scripts N: N: Processing binary package mpgrafic (version 0.3.15-1, arch amd64) ... C: mpgrafic binary (0.3.15-1) [amd64]: no-ctrl-scripts N: N: Processing binary package mpgrafic-dbgsym (version 0.3.15-1, arch i386) ... C: mpgrafic-dbgsym binary (0.3.15-1) [i386]: no-ctrl-scripts N: N: Processing binary package mpgrafic-dbgsym (version 0.3.15-1, arch amd64) ... https://anonscm.debian.org/git/debian-astro/packages/mpgrafic.git/tree/configure.ac https://anonscm.debian.org/git/debian-astro/packages/mpgrafic.git/tree/configure This is a lintian false positive with AC_SYS_LARGEFILE.
Bug#871956: lintian: false positive: binary-file-built-without-LFS-support on x32
On Fri, Sep 01, 2017 at 08:50:05PM +0200, Boud Roukema wrote: > On Fri, 1 Sep 2017, Adam Borowski wrote: > > Also, the vast majority of packages don't trigger this warning as they > > request LFS unconditionally instead of trying to autodetect it. > > The lintian warning here: > https://lintian.debian.org/tags/binary-file-built-without-LFS-support.html > says "This can be done by using the AC_SYS_LARGEFILE macro with autoconf, > or... " > > This is what I've done in mpgrafic, since (in principle) autoconf > makes things less hardwired. > > Isn't it cleaner to use the AC_SYS_LARGEFILE autoconf macro rather > than hardwire something such as -D_FILE_OFFSET_BITS=64 as a > non-optional compile option? That assumes your project uses autotools -- they seem to have rapidly regressed recently, and projects are fleeing them left and right. And what AC_SYS_LARGEFILE does, at least on Linux, is to return a hardcoded setting so programs switch from off_t to off64_t whether they need to or not. This does the right thing on old 32-bit archs and is harmless on 64-bit and new 32-bit. The problem is in snowflake packages that do things their own way and enable LFS only when it's actually needed. Here's where the lintian false positive triggers. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢰⠒⠀⣿⡁ Vat kind uf sufficiently advanced technology iz dis!? ⢿⡄⠘⠷⠚⠋⠀ -- Genghis Ht'rok'din ⠈⠳⣄
Bug#871956: lintian: false positive: binary-file-built-without-LFS-support on x32
On Fri, 1 Sep 2017, Adam Borowski wrote: Also, the vast majority of packages don't trigger this warning as they request LFS unconditionally instead of trying to autodetect it. The lintian warning here: https://lintian.debian.org/tags/binary-file-built-without-LFS-support.html says "This can be done by using the AC_SYS_LARGEFILE macro with autoconf, or... " This is what I've done in mpgrafic, since (in principle) autoconf makes things less hardwired. Isn't it cleaner to use the AC_SYS_LARGEFILE autoconf macro rather than hardwire something such as -D_FILE_OFFSET_BITS=64 as a non-optional compile option? Also, a rather important question seems to me to be: are there some architectures (e.g. especially 32-bit) where there is no point trying to ensure LFS support, since it won't work anyway? Or conversely: can gcc implement a software solution for LFS support no matter what the architecture is (among debian official + unofficial ports)?
Bug#871956: lintian: false positive: binary-file-built-without-LFS-support on x32
Hi Adam, > I'd suggest just marking what we have an idea about. It's not > like lintian warnings matter for something that's not in the archive... That's true. (Implementation braindump for anyone interested in jumping in: we could simply blacklist x32 directly in checks/binaries.pm, but we have data/common/architectures so we might need to add a field there and update Lintian::Architecture. The testsuite will need some cleverness so that we don't fail under x32.) Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#871956: lintian: false positive: binary-file-built-without-LFS-support on x32
On Fri, Sep 01, 2017 at 09:51:56AM +0100, Chris Lamb wrote:
> > false positive: binary-file-built-without-LFS-support on x32
>
> I think the next step here would be to identify which of these
> archs should be skipped for this check:
>
>
> https://anonscm.debian.org/git/lintian/lintian.git/tree/data/common/architectures
I'm afraid I don't know how to check this in a systematic way.
* x32 is a fully functional second-class Debian port, so all it takes is to
compile and run printf("%ld", sizeof(off_t));
* arm64ilp32 has Documentation/arm64/ilp32.txt say: "off_t is s64 type."
... but what about the rest?
Thus, unless someone knows where the authoritative table these definitions
come from is, I'd suggest just marking what we have an idea about. It's not
like lintian warnings matter for something that's not in the archive...
Also, the vast majority of packages don't trigger this warning as they
request LFS unconditionally instead of trying to autodetect it.
Meow!
--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢰⠒⠀⣿⡁ Vat kind uf sufficiently advanced technology iz dis!?
⢿⡄⠘⠷⠚⠋⠀ -- Genghis Ht'rok'din
⠈⠳⣄
Bug#871956: lintian: false positive: binary-file-built-without-LFS-support on x32
Hi all, > false positive: binary-file-built-without-LFS-support on x32 I think the next step here would be to identify which of these archs should be skipped for this check: https://anonscm.debian.org/git/lintian/lintian.git/tree/data/common/architectures Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#871956: lintian: false positive: binary-file-built-without-LFS-support on x32
Package: lintian Version: 2.5.52 Severity: normal Hi! For all 32-bit architectures lintian complains if functions using off_t instead of off64_t are used. On legacy architectures, this is indeed a good check, as for old ABI compatibility reasons sizeof(off_t) is only 4. However, new 32-bit architectures such as x32 have sizeof(off_t)==8, and functions like lseek() and lseek64() are the same. These include at least x32 and arm64ilp32. Thus, could you please drop these architectures from has_lfs checks? Reported by Thorsten Glaser, initially found in package cvs. Most packages don't trigger this warning as "#define _FILE_OFFSET_BITS 64" pointlessly but harmlessly aliases off_t to off64_t. Meow! -- System Information: Debian Release: buster/sid APT prefers unstable APT policy: (500, 'unstable') Architecture: x32 (x86_64) Kernel: Linux 4.13.0-rc4-debug-00052-gbdd287aecdf1 (SMP w/6 CPU cores) Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE=C.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: sysvinit (via /sbin/init) Versions of packages lintian depends on: ii binutils 2.29-4 ii bzip2 1.0.6-8.1 ii diffstat 1.61-1 ii dpkg 1.18.24 ii file 1:5.31-1 ii gettext 0.19.8.1-2+b3 ii intltool-debian 0.35.0+20060710.4 ii libapt-pkg-perl 0.1.32+b2 ii libarchive-zip-perl 1.59-1 ii libclass-accessor-perl0.34-1 ii libclone-perl 0.38-2+b2 ii libdpkg-perl 1.18.24 ii libemail-valid-perl 1.202-1 ii libfile-basedir-perl 0.07-1 ii libipc-run-perl 0.96-1 ii liblist-moreutils-perl0.416-1+b3 ii libparse-debianchangelog-perl 1.2.0-12 ii libperl5.26 [libdigest-sha-perl] 5.26.0-5 ii libtext-levenshtein-perl 0.13-1 ii libtimedate-perl 2.3000-2 ii liburi-perl 1.72-1 ii libxml-simple-perl2.24-1 ii libyaml-libyaml-perl 0.63-2+b2 ii man-db2.7.6.1-2 ii patchutils0.3.4-2 ii perl 5.26.0-5 ii t1utils 1.40-2 ii xz-utils 5.2.2-1.3 Versions of packages lintian recommends: pn libperlio-gzip-perl Versions of packages lintian suggests: pn binutils-multiarch ii dpkg-dev 1.18.24 ii libhtml-parser-perl3.72-3+b2 pn libtext-template-perl -- no debconf information
