Bug#1020909: Enable lfs (large file support) in apt-cacher-ng
On 9/29/22 00:15, Eduard Bloch wrote: [*] _LARGEFILE_SOURCE or _FILE_OFFSET_BITS=64 Both of them? I don't think so. _LARGEFILE_SOURCE will export some XYZo functions of the C API, but I don't use them. Ok. apt-cacher-ng has been adding those defines since the early days of its creation. Really? I don't find the CFLAGS _LARGEFILE_SOURCE or _FILE_OFFSET_BITS=64 in the sources. Strange, my grep command has no problems doing right that. $ grep FILE_OFFSET src/* CMakeLists.txt src/config.h:// added in Makefile... #define _FILE_OFFSET_BITS 64 src/meta.h:// _FILE_OFFSET_BITS mostly irrelevant. But if it's set, watch out for user's "experiments". src/meta.h:#if _FILE_OFFSET_BITS == 32 src/meta.h:#error Unsupported: _FILE_OFFSET_BITS == 32 with large long size src/meta.h:#if 64 == _FILE_OFFSET_BITS src/meta.h:#if 32 == _FILE_OFFSET_BITS CMakeLists.txt:SET(ACNG_COMPFLAGS "-D_FILE_OFFSET_BITS=64") You are absolutely right! I have no idea why I didn't find them too. Maybe I grepped the wrong directory, maybe only for _LARGEFILE_SOURCE? Anyway, I agree _FILE_OFFSET_BITS is set. I opened the bug because I faced this cronjob-error sometimes: /etc/cron.daily/apt-cacher-ng: Aborted run-parts: /etc/cron.daily/apt-cacher-ng exited with return code 134 I assumed it's because of lfs, but it's probably something else. Helge
Bug#1020909: Enable lfs (large file support) in apt-cacher-ng
Hallo, * Helge Deller [Wed, Sep 28 2022, 10:12:52PM]: > On 9/28/22 21:52, Eduard Bloch wrote: > If you call readdir() a few lines below, this readdir will not be able > to store the file info in the dirent struct (if the file is located in > high area of discs), instead returns NULL with errno set (e.g. E2BIG or like > that). > Finally the readdir() will not work as expected as it doesn't find some files. > > [*] _LARGEFILE_SOURCE or _FILE_OFFSET_BITS=64 Both of them? I don't think so. _LARGEFILE_SOURCE will export some XYZo functions of the C API, but I don't use them. And _FILE_OFFSET_BITS IS set, I have no idea what you are talking about. Check the logs, like https://buildd.debian.org/status/fetch.php?pkg=apt-cacher-ng=i386=3.7.4-1%2Bb1=1652551064=0 > I found a similiar issue with glibc right recently: > https://sourceware.org/bugzilla/show_bug.cgi?id=29583 > > > So do you have your cache on NFS with thousands of files in a single > > directory? > > No, but a disc of TB size on a 32-bit platform. Exactly, no. So where is the problem you are observing actually coming from? So far I saw a weak theory, observation of some error status, and pointing at "missing LFS". And the last part is not even true as far as I could see. > > > Sometimes I see a cron job warning: > > > /etc/cron.daily/apt-cacher-ng: > > > Aborted > > > run-parts: /etc/cron.daily/apt-cacher-ng exited with return code 134 > > > > > > Change is trivial, just add "future=+lfs" to DEB_BUILD_MAINT_OPTIONS: > > > > This does not make sense. That would only inject a couple of runtime > > influencing defines. > > Well, not just runtime influencing. More importantly: Build-time. Obviously. > They enlarge the dirent struct for 64-bit wide inode numbers and > thus allow readdir() [which actually calls the readdir64 syscall then] > to function properly. You don't need to teach me the basics. I have added LFS support over a decade ago. Please do at least some basic research to find things like https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=588048 . > > apt-cacher-ng has been adding those defines since > > the early days of its creation. > > Really? > I don't find the CFLAGS _LARGEFILE_SOURCE or _FILE_OFFSET_BITS=64 in the > sources. Strange, my grep command has no problems doing right that. $ grep FILE_OFFSET src/* CMakeLists.txt src/config.h:// added in Makefile... #define _FILE_OFFSET_BITS 64 src/meta.h:// _FILE_OFFSET_BITS mostly irrelevant. But if it's set, watch out for user's "experiments". src/meta.h:#if _FILE_OFFSET_BITS == 32 src/meta.h:#error Unsupported: _FILE_OFFSET_BITS == 32 with large long size src/meta.h:#if 64 == _FILE_OFFSET_BITS src/meta.h:#if 32 == _FILE_OFFSET_BITS CMakeLists.txt:SET(ACNG_COMPFLAGS "-D_FILE_OFFSET_BITS=64") MfG, Eduard.
Bug#1020909: Enable lfs (large file support) in apt-cacher-ng
On 9/28/22 22:12, Helge Deller wrote: The problem here is, that if those flags [*] aren't set, the "struct dirent" will only be able to hold file numbers which are max. 32bits width. "File numbers" is wrong... I meant "Inode number" (of the file). Helge
Bug#1020909: Enable lfs (large file support) in apt-cacher-ng
On 9/28/22 21:52, Eduard Bloch wrote:
Hallo,
Hallo Eduard,
* Helge Deller [Wed, Sep 28 2022, 12:46:53PM]:
Package: apt-cacher-ng
Severity: 3.7.4-1
Tags: lfs, hppa, patch
Please enable large file support.
apt-cacher-ng uses readdir() which can fail on 32-bit arches
running on large discs.
This does not make sense. readdir() does not care about large discs. It
might care about the file number in a directory. And while there are
some limitations with readdir for virtual filesystems (like NFS) there
should be no issue with local filesystem where kernel is maintaining the
internal handle with proper means.
In apt-cacher-ng-3.7.4/src/dirwalk.cc, line 100ff you find:
struct dirent *dp;
...
while ( nullptr != (dp = readdir(dir)) )
{
if (strcmp(dp->d_name, ".") && strcmp(dp->d_name, ".."))
{
The problem here is, that if those flags [*] aren't set, the "struct dirent"
will only be able to hold file numbers which are max. 32bits width.
If you call readdir() a few lines below, this readdir will not be able
to store the file info in the dirent struct (if the file is located in
high area of discs), instead returns NULL with errno set (e.g. E2BIG or like
that).
Finally the readdir() will not work as expected as it doesn't find some files.
[*] _LARGEFILE_SOURCE or _FILE_OFFSET_BITS=64
I found a similiar issue with glibc right recently:
https://sourceware.org/bugzilla/show_bug.cgi?id=29583
So do you have your cache on NFS with thousands of files in a single
directory?
No, but a disc of TB size on a 32-bit platform.
Sometimes I see a cron job warning:
/etc/cron.daily/apt-cacher-ng:
Aborted
run-parts: /etc/cron.daily/apt-cacher-ng exited with return code 134
Change is trivial, just add "future=+lfs" to DEB_BUILD_MAINT_OPTIONS:
This does not make sense. That would only inject a couple of runtime
influencing defines.
Well, not just runtime influencing. More importantly: Build-time.
They enlarge the dirent struct for 64-bit wide inode numbers and
thus allow readdir() [which actually calls the readdir64 syscall then]
to function properly.
apt-cacher-ng has been adding those defines since
the early days of its creation.
Really?
I don't find the CFLAGS _LARGEFILE_SOURCE or _FILE_OFFSET_BITS=64 in the
sources.
And how would that affect readdir related syscalls?
I hope I explained above.
Helge
Bug#1020909: Enable lfs (large file support) in apt-cacher-ng
Hallo, * Helge Deller [Wed, Sep 28 2022, 12:46:53PM]: > Package: apt-cacher-ng > Severity: 3.7.4-1 > Tags: lfs, hppa, patch > > Please enable large file support. > apt-cacher-ng uses readdir() which can fail on 32-bit arches > running on large discs. This does not make sense. readdir() does not care about large discs. It might care about the file number in a directory. And while there are some limitations with readdir for virtual filesystems (like NFS) there should be no issue with local filesystem where kernel is maintaining the internal handle with proper means. So do you have your cache on NFS with thousands of files in a single directory? > Sometimes I see a cron job warning: > /etc/cron.daily/apt-cacher-ng: > Aborted > run-parts: /etc/cron.daily/apt-cacher-ng exited with return code 134 > > Change is trivial, just add "future=+lfs" to DEB_BUILD_MAINT_OPTIONS: This does not make sense. That would only inject a couple of runtime influencing defines. apt-cacher-ng has been adding those defines since the early days of its creation. And how would that affect readdir related syscalls? Best regards, Eduard. -- Wenn wir bedenken, daß wir alle verrückt sind, ist das Leben erklärt. -- Mark Twain (eigl. Samuel Langhorne Clemens)
Bug#1020909: Enable lfs (large file support) in apt-cacher-ng
Package: apt-cacher-ng Severity: 3.7.4-1 Tags: lfs, hppa, patch Please enable large file support. apt-cacher-ng uses readdir() which can fail on 32-bit arches running on large discs. Sometimes I see a cron job warning: /etc/cron.daily/apt-cacher-ng: Aborted run-parts: /etc/cron.daily/apt-cacher-ng exited with return code 134 Change is trivial, just add "future=+lfs" to DEB_BUILD_MAINT_OPTIONS: +++ ./rules 2022-09-28 10:05:10.348649957 + @@ -3,7 +3,7 @@ -export DEB_BUILD_MAINT_OPTIONS = hardening=+all +export DEB_BUILD_MAINT_OPTIONS = hardening=+all future=+lfs
