Bug#826783: libapt-pkg5.0: aptitude aborts due to a fd leak

[Manuel A. Fernandez Montecelo]

Hi again,

2016-06-12 15:10 Manuel A. Fernandez Montecelo:

Control: tags -1 + moreinfo


Hi Jiri,

2016-06-09 02:42 Jiri Palecek:


Then I looked at the file descriptors that were open:


Did you actually see more file descriptors here, at the point where you
cut the text?

With apt 1.2.13 installed, fixing LZ4 and other leaks, I can see 2
"copies" of the file descriptors as above (probably because aptitude
creates threads for the UI, downloads, etc), but no matter what I do, I
cannot see more than 2 "copies" of the descriptors.

Depending on the operation (package lists updates, package downloads,
installations), I see a few more, including pipes, but not many more
than that.  So that means, far below 1083 -- something like 30.


If they were indeed leaks, I can see how having a long-running session
of aptitude and updating available packages many times, could have
caused this.  Were you using aptitude in that way?

However, I downgraded apt packages to 1.2.12 (before the fix for LZ4
leaks), set the options to keep indices compressed etc., repeated the
operations described above, and similar results -- only 20~30 file
descriptors.


Do you still experience this problem, and if you do, can you provide the
information requested above?


Cheers.
--
Manuel A. Fernandez Montecelo 

Bug#826783: libapt-pkg5.0: aptitude aborts due to a fd leak

[Manuel A. Fernandez Montecelo]

Control: tags -1 + moreinfo


Hi Jiri,

2016-06-09 02:42 Jiri Palecek:


Then I looked at the file descriptors that were open:
[...]
aptitude 2856 root0u   CHR  136,2  0t0   5 /dev/pts/2
aptitude 2856 root1u   CHR  136,2  0t0   5 /dev/pts/2
aptitude 2856 root2u   CHR  136,2  0t0   5 /dev/pts/2
aptitude 2856 root3uW  REG8,60 2474525 /var/lib/dpkg/lock
aptitude 2856 root4r   REG8,6  4413957 2475249 /var/lib/dpkg/status
aptitude 2856 root5r   REG8,648049 2064548 
/var/lib/apt/lists/_home_jirka_public%5fhtml_ltp_dists_sid_main_binary-i386_Packages
aptitude 2856 root6r   REG8,622244 2064707 
/var/lib/apt/lists/emacs.naquadah.org_unstable_Packages
aptitude 2856 root7r   REG8,6 7692 2065017 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_experimental_non-free_i18n_Translation-en.lz4
aptitude 2856 root8r   REG8,6 9603 2065060 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_experimental_non-free_binary-i386_Packages.lz4
aptitude 2856 root9r   REG8,6 4606 2065265 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_experimental_contrib_i18n_Translation-en.lz4
aptitude 2856 root   10r   REG8,6 6224 2065213 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_experimental_contrib_binary-i386_Packages.lz4
aptitude 2856 root   11r   REG8,6   371262 2065168 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_experimental_main_i18n_Translation-en.lz4
aptitude 2856 root   12r   REG8,6   755795 2065166 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_experimental_main_binary-i386_Packages.lz4
aptitude 2856 root   13r   REG8,6   134176 2065093 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_unstable_non-free_i18n_Translation-en.lz4
aptitude 2856 root   14r   REG8,6   127605 2064999 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_unstable_non-free_binary-i386_Packages.lz4
aptitude 2856 root   15r   REG8,681846 2065091 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_unstable_contrib_i18n_Translation-en.lz4
aptitude 2856 root   16r   REG8,693960 2065094 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_unstable_contrib_binary-i386_Packages.lz4
aptitude 2856 root   17r   REG8,6  9964496 2065162 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_unstable_main_i18n_Translation-en.lz4
aptitude 2856 root   18r   REG8,6   967240 2065167 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_unstable_main_i18n_Translation-cs.lz4
aptitude 2856 root   19r   REG8,6 15028352 2065059 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_unstable_main_binary-i386_Packages.lz4
aptitude 2856 root   20r   REG8,6   126208 2065195 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_testing_non-free_i18n_Translation-en.lz4
aptitude 2856 root   21r   REG8,6   139261 2065124 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_testing_non-free_binary-i386_Packages.lz4
aptitude 2856 root   22r   REG8,674845 2064698 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_testing_contrib_i18n_Translation-en.lz4
aptitude 2856 root   23r   REG8,699424 2065072 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_testing_contrib_binary-i386_Packages.lz4
aptitude 2856 root   24r   REG8,6  9510768 2065084 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_testing_main_i18n_Translation-en.lz4
aptitude 2856 root   25r   REG8,6   931870 2065147 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_testing_main_i18n_Translation-cs.lz4
aptitude 2856 root   26r   REG8,6 16558917 2065078 
/var/lib/apt/lists/ftp.cz.debian.org_debian_dists_testing_main_binary-i386_Packages.lz4
aptitude 2856 root   27r   REG8,6 9750 2065281 
/var/lib/apt/lists/debian-mirrors.sdinet.de_deb-multimedia_dists_experimental_main_binary-i386_Packages.lz4
aptitude 2856 root   28r   REG8,6 8969 2065243 
/var/lib/apt/lists/debian-mirrors.sdinet.de_deb-multimedia_dists_unstable_non-free_binary-i386_Packages.lz4
aptitude 2856 root   29r   REG8,6   337438 2065144 
/var/lib/apt/lists/debian-mirrors.sdinet.de_deb-multimedia_dists_unstable_main_binary-i386_Packages.lz4
aptitude 2856 root   30r   REG8,6 9351 2065222 
/var/lib/apt/lists/debian-mirrors.sdinet.de_deb-multimedia_dists_stable_non-free_binary-i386_Packages.lz4
aptitude 2856 root   31r   REG8,6   319256 2065136 
/var/lib/apt/lists/debian-mirrors.sdinet.de_deb-multimedia_dists_stable_main_binary-i386_Packages.lz4
aptitude 2856 root   32r   REG8,6 9346 2065218 
/var/lib/apt/lists/debian-mirrors.sdinet.de_deb-multimedia_dists_testing_non-free_binary-i386_Packages.lz4
aptitude 2856 root   33r   REG8,6   338152 2065080 
/var/lib/apt/lists/debian-mirrors.sdinet.de_deb-multimedia_dists_testing_main_binary-i386_Packages.lz4
aptitude 2856 root   34uW  REG   0,200   38732 /run/lock/aptitude
aptitude 2856 root   35r   REG8,6  4413957 2475249 /var/lib/dpkg/status
aptitude 2856 root   36r   REG

Bug#826783: libapt-pkg5.0: aptitude aborts due to a fd leak

[David Kalnischkies]

Control: reassign -1 aptitude

On Thu, Jun 09, 2016 at 03:42:30AM +0200, Jiri Palecek wrote:
> Then I looked at the file descriptors that were open:
[…]
> aptitude 2856 root7r   REG8,6 7692 2065017 
> /var/lib/apt/lists/ftp.cz.debian.org_debian_dists_experimental_non-free_i18n_Translation-en.lz4
> aptitude 2856 root8r   REG8,6 9603 2065060 
> /var/lib/apt/lists/ftp.cz.debian.org_debian_dists_experimental_non-free_binary-i386_Packages.lz4
> aptitude 2856 root9r   REG8,6 4606 2065265 
> /var/lib/apt/lists/ftp.cz.debian.org_debian_dists_experimental_contrib_i18n_Translation-en.lz4
> aptitude 2856 root   10r   REG8,6 6224 2065213 
> /var/lib/apt/lists/ftp.cz.debian.org_debian_dists_experimental_contrib_binary-i386_Packages.lz4
[… more lz4 files …]
> It seems something in aptitude or libapt-pkg is opening those files, but
> not closing them. These file descriptors are then missing for
> downloading new packages. Could you look into it?

So, I went hunting yesterday for FD leaks in libapt and found a few…

Three of them quite boring from a user point of view, but one of them is
a leak in the lz4 (de)compression code which explains many of the leaked
fds in your output as you have compressed indexes enabled & lz4 is the
currently lowest cost compression type – but that output also contains
a few uncompressed files (like the dpkg/status file) multiple times
which I can't explain & haven't seen with any apt tool so far and
I ended up running our testcases with strategically placed:

auto const pid = getpid();
std::string ls;
strprintf(ls, "ls -l /proc/%d/fd >&2", pid);
system(ls.c_str());

so those leaks (if they are the fault of libapt) are caused by how
aptitude is calling it which apt tools don't do, so I reassign this bug
for now over to them so they can investigate were these leaks happen in
the hope that we can then figure out if its misuse of libapt or "just"
uncommon use causing a leak as that is likely more efficient than
letting me wade through aptitude code on my own…

(as the lz4 leak is by far the biggest eater of fds in your case this
bug could be closed with it, but we gonna catch them all…)


Best regards

David Kalnischkies


signature.asc
Description: PGP signature

Bug#826783: libapt-pkg5.0: aptitude aborts due to a fd leak

[Jiri Palecek]

Package: libapt-pkg5.0
Version: 1.2.12
Severity: normal

Dear Maintainer,

after trying to upgrade several packages from the KDE suite in aptitude, I got a
strange SIGABRT when trying to download them. Via gdb, I obtained
backtrace:

Program received signal SIGABRT, Aborted.
[Switching to Thread 0xab40 (LWP 8710)]
0xb773cdad in __kernel_vsyscall ()
(gdb) bt
#0  0xb773cdad in __kernel_vsyscall ()
#1  0xb6d5de26 in __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:55
#2  0xb6d5f3f7 in __GI_abort () at abort.c:89
#3  0xb6d9b2a0 in __libc_message (do_abort=2, fmt=0xb6e90ff3 "*** %s ***: %s 
terminated\n") at ../sysdeps/posix/libc_fatal.c:175
#4  0xb6e2aeb7 in __GI___fortify_fail (msg=0xb6e90f8a "buffer overflow 
detected") at fortify_fail.c:31
#5  0xb6e290b8 in __GI___chk_fail () at chk_fail.c:28
#6  0xb6e2adda in __fdelt_chk (d=1083) at fdelt_chk.c:25
#7  0xb75eeafe in pkgAcquire::SetFds(int&, fd_set*, fd_set*) () from 
/usr/lib/i386-linux-gnu/libapt-pkg.so.5.0
#8  0xb75f418f in pkgAcquire::Run(int) () from 
/usr/lib/i386-linux-gnu/libapt-pkg.so.5.0
#9  0x8045f2a9 in download_manager::do_download (this=0x84f96cd4) at 
../../../../src/generic/apt/download_manager.cc:35
#10 0x8012b8db in download_thread::operator() (this=0x84ff0df8) at 
../../src/download_thread.cc:226
#11 0x801fee7f in 
cwidget::threads::noncopy_bootstrap::operator() 
(this=0xa318) at /usr/include/cwidget/generic/threads/threads.h:279
#12 0x801febfd in 
cwidget::threads::thread::bootstrap (p=0x84f771e0) at /usr/include/cwidget/generic/threads/threads.h:117
#13 0xb70cc2ce in start_thread (arg=0xab40) at pthread_create.c:334
#14 0xb6e1a23e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:122

After googling the __fdelt_chk function, it seems to check that the
passed fd is less than FD_SETSIZE and abort otherwise. Fair enough, 1083
seems pretty high.

Then I looked at the file descriptors that were open:

debian:/mnt/extras/src# lsof -p $(pidof aptitude)
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
  Output information may be incomplete.
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFFNODE NAME
aptitude 2856 root  cwdDIR8,628672 2113538 /home/jirka  


   
aptitude 2856 root  rtdDIR8,6 4096   2 /


   
aptitude 2856 root  txtREG8,6  6610336 1770506 /usr/bin/aptitude-curses 


   
aptitude 2856 root  memREG8,6   149869 4096209 
/usr/share/locale/cs/LC_MESSAGES/libc.mo


aptitude 2856 root  memREG8,6 31813113 2080784 
/var/cache/apt/pkgcache.bin 


aptitude 2856 root  memREG8,646736  230569 
/lib/i386-linux-gnu/libnss_files-2.22.so


aptitude 2856 root  memREG8,696128  230332 
/lib/i386-linux-gnu/libnsl-2.22.so  


aptitude 2856 root  memREG8,626992 4096400 
/usr/share/locale/cs/LC_MESSAGES/libapt-pkg5.0.mo   


aptitude 2856 root  memREG8,6   144168 4096084 
/usr/share/locale/cs/LC_MESSAGES/aptitude.mo


aptitude 2856 root  memREG8,6  2543120 1277970 
/usr/lib/locale/locale-archive  


aptitude 2856 root  memREG8,618092  231838