[Touch-packages] [Bug 1332440] Re: apt-get update very slow when ulimit -n is big

2015-04-26 Thread Ian Babrou 
Yep, it's not fixed:

~ λ docker run --rm ubuntu:14.04 bash -c 'ulimit -n 50  apt-get install 
strace  strace -c -f apt-get update'
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  strace
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 113 kB of archives.
After this operation, 504 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty/main strace amd64 4.8-1ubuntu5 
[113 kB]
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin:
Fetched 113 kB in 0s (179 kB/s)
Selecting previously unselected package strace.
(Reading database ... 11527 files and directories currently installed.)
Preparing to unpack .../strace_4.8-1ubuntu5_amd64.deb ...
Unpacking strace (4.8-1ubuntu5) ...
Setting up strace (4.8-1ubuntu5) ...
Process 47 attached
Process 48 attached
Process 49 attached
Ign http://archive.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
Ign http://archive.ubuntu.com trusty-security InRelease
Hit http://archive.ubuntu.com trusty Release.gpg
Get:1 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]
Get:2 http://archive.ubuntu.com trusty-security Release.gpg [933 B]
Process 50 attached
Process 51 attached
Hit http://archive.ubuntu.com trusty Release
Get:3 http://archive.ubuntu.com trusty-updates Release [63.5 kB]
Process 52 attached
Process 53 attached
Get:4 http://archive.ubuntu.com trusty-security Release [63.5 kB]
Process 54 attached
Get:5 http://archive.ubuntu.com trusty/main Sources [1335 kB]
Process 55 attached
Process 56 attached
^CGet:6 http://archive.ubuntu.com trusty/restricted Sources [5335 B]
Get:7 http://archive.ubuntu.com trusty/universe Sources [7926 kB]
Get:8 http://archive.ubuntu.com trusty/main amd64 Packages [1743 kB]
Get:9 http://archive.ubuntu.com trusty/restricted amd64 Packages [16.0 kB]
Get:10 http://archive.ubuntu.com trusty/universe amd64 Packages [7589 kB]
Get:11 http://archive.ubuntu.com trusty-updates/main Sources [248 kB]
Get:12 http://archive.ubuntu.com trusty-updates/restricted Sources [2310 B]
Get:13 http://archive.ubuntu.com trusty-updates/universe Sources [141 kB]
Get:14 http://archive.ubuntu.com trusty-updates/main amd64 Packages [636 kB]
Get:15 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [15.1 
kB]
Get:16 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [356 kB]
Get:17 http://archive.ubuntu.com trusty-security/main Sources [97.7 kB]
Get:18 http://archive.ubuntu.com trusty-security/restricted Sources [1874 B]
Get:19 http://archive.ubuntu.com trusty-security/universe Sources [23.3 kB]
Get:20 http://archive.ubuntu.com trusty-security/main amd64 Packages [329 kB]
Get:21 http://archive.ubuntu.com trusty-security/restricted amd64 Packages 
[14.8 kB]
Get:22 http://archive.ubuntu.com trusty-security/universe amd64 Packages [128 
kB]
FATA[0368] Error response from daemon: Cannot destroy container 
327825463e914c72c59f01a5aa1ab25b8ab13eeae55ac961159a729e0f4f3599: Failed to set 
container state to RemovalInProgress: Status is already RemovalInProgress
~ λ docker run --rm ubuntu:14.04 bash -c 'ulimit -n 500  apt-get install 
strace  time strace -c -f apt-get update'
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  strace
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 113 kB of archives.
After this operation, 504 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty/main strace amd64 4.8-1ubuntu5 
[113 kB]
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin:
Fetched 113 kB in 0s (170 kB/s)
Selecting previously unselected package strace.
(Reading database ... 11527 files and directories currently installed.)
Preparing to unpack .../strace_4.8-1ubuntu5_amd64.deb ...
Unpacking strace (4.8-1ubuntu5) ...
Setting up strace (4.8-1ubuntu5) ...
Process 48 attached
Process 49 attached
Process 50 attached
Ign http://archive.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
Ign http://archive.ubuntu.com trusty-security InRelease
Hit http://archive.ubuntu.com trusty Release.gpg
Get:1 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]
Get:2 http://archive.ubuntu.com trusty-security Release.gpg [933 B]
Process 51 attached

[Touch-packages] [Bug 1332440] Re: apt-get update very slow when ulimit -n is big

2015-01-03 Thread Ian Babrou 
Looks like it was fixed in some update. Can anyone confirm that?

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/1332440

Title:
  apt-get update very slow when ulimit -n is big

Status in apt package in Ubuntu:
  Confirmed

Bug description:
  I use strace to see why apt-get update is too slow,
  finally I found apt-get loop 3~n to '(? , F_SETFD, FD_CLOEXEC) = 0'

  ubuntu 14.04

  root@dev2:/tmp/apt-1.0.1ubuntu2.1# apt-cache show apt
  Package: apt
  Priority: important
  Section: admin
  Installed-Size: 3576
  Maintainer: Ubuntu Developers ubuntu-devel-disc...@lists.ubuntu.com
  Original-Maintainer: APT Development Team de...@lists.debian.org
  Architecture: amd64
  Version: 1.0.1ubuntu2.1
  Replaces: manpages-it ( 2.80-4~), manpages-pl ( 20060617-3~), 
openjdk-6-jdk ( 6b24-1.11-0ubuntu1~), sun-java5-jdk ( 0), sun-java6-jdk ( 
0)
  Depends: libapt-pkg4.12 (= 0.9.16), libc6 (= 2.15), libgcc1 (= 1:4.1.1), 
libstdc++6 (= 4.6), ubuntu-keyring, gnupg
  Suggests: aptitude | synaptic | wajig, dpkg-dev (= 1.17.2), apt-doc, 
python-apt
  Conflicts: python-apt ( 0.7.93.2~)
  Breaks: manpages-it ( 2.80-4~), manpages-pl ( 20060617-3~), openjdk-6-jdk 
( 6b24-1.11-0ubuntu1~), sun-java5-jdk ( 0), sun-java6-jdk ( 0)
  Filename: pool/main/a/apt/apt_1.0.1ubuntu2.1_amd64.deb
  Size: 952374

  
  [pid 20525] 
stat(/var/lib/apt/lists/mirrors.163.com_ubuntu_dists_trusty-updates_universe_i18n_Translation-en%5fUS,
 0x7fffaabe5b30) = -1 ENOENT (No such file or directory)
  [pid 20525] stat(/usr/lib/apt/methods/bzip2, {st_mode=S_IFREG|0755, 
st_size=23056, ...}) = 0
  [pid 20525] pipe([9, 11])   = 0
  [pid 20525] pipe([13, 14])  = 0
  [pid 20525] fcntl(9, F_SETFD, FD_CLOEXEC) = 0
  [pid 20525] fcntl(11, F_SETFD, FD_CLOEXEC) = 0
  [pid 20525] fcntl(13, F_SETFD, FD_CLOEXEC) = 0
  [pid 20525] fcntl(14, F_SETFD, FD_CLOEXEC) = 0
  [pid 20525] clone(child_stack=0, 
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, 
child_tidptr=0x7f99f1ea9a50) = 20538
  Process 20538 attached
  [pid 20525] fcntl(9, F_GETFL)   = 0 (flags O_RDONLY)
  [pid 20525] fcntl(9, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
  [pid 20538] rt_sigaction(SIGPIPE, {SIG_DFL, [PIPE], SA_RESTORER|SA_RESTART, 
0x7f99f0e72ff0},  unfinished ...
  [pid 20525] fcntl(14, F_GETFL unfinished ...
  [pid 20538] ... rt_sigaction resumed {SIG_IGN, [PIPE], 
SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, 8) = 0
  [pid 20525] ... fcntl resumed )   = 0x1 (flags O_WRONLY)
  [pid 20538] rt_sigaction(SIGQUIT, {SIG_DFL, [QUIT], SA_RESTORER|SA_RESTART, 
0x7f99f0e72ff0},  unfinished ...
  [pid 20525] fcntl(14, F_SETFL, O_WRONLY|O_NONBLOCK unfinished ...
  [pid 20538] ... rt_sigaction resumed {SIG_DFL, [], 0}, 8) = 0
  [pid 20525] ... fcntl resumed )   = 0
  [pid 20538] rt_sigaction(SIGINT, {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 
0x7f99f0e72ff0},  unfinished ...
  [pid 20525] close(11 unfinished ...
  [pid 20538] ... rt_sigaction resumed {SIG_DFL, [], 0}, 8) = 0
  [pid 20525] ... close resumed )   = 0
  [pid 20538] rt_sigaction(SIGWINCH, {SIG_DFL, [WINCH], SA_RESTORER|SA_RESTART, 
0x7f99f0e72ff0},  unfinished ...
  [pid 20525] close(13 unfinished ...
  [pid 20538] ... rt_sigaction resumed {0x409140, [WINCH], 
SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, 8) = 0
  [pid 20525] ... close resumed )   = 0
  [pid 20538] rt_sigaction(SIGCONT, {SIG_DFL, [CONT], SA_RESTORER|SA_RESTART, 
0x7f99f0e72ff0},  unfinished ...
  [pid 20525] select(10, [9], NULL, NULL, NULL unfinished ...
  [pid 20538] ... rt_sigaction resumed {SIG_DFL, [], 0}, 8) = 0
  [pid 20538] rt_sigaction(SIGTSTP, {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 
0x7f99f0e72ff0}, {SIG_DFL, [], 0}, 8) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(3, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(4, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(5, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(6, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(7, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(8, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(9, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(10, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(11, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(12, F_SETFD, FD_CLOEXEC) = 0

[Touch-packages] [Bug 1332440] Re: apt-get update very slow when ulimit -n is big

2014-07-10 Thread Ian Babrou 
Building docker image with ulimit -n 500k takes literally forever in
virtualbox. Iterating 500k fds that are not even open on every fork in
apt (and there are many of them) is not okay.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/1332440

Title:
  apt-get update very slow when ulimit -n is big

Status in “apt” package in Ubuntu:
  New

Bug description:
  I use strace to see why apt-get update is too slow,
  finally I found apt-get loop 3~n to '(? , F_SETFD, FD_CLOEXEC) = 0'

  ubuntu 14.04

  root@dev2:/tmp/apt-1.0.1ubuntu2.1# apt-cache show apt
  Package: apt
  Priority: important
  Section: admin
  Installed-Size: 3576
  Maintainer: Ubuntu Developers ubuntu-devel-disc...@lists.ubuntu.com
  Original-Maintainer: APT Development Team de...@lists.debian.org
  Architecture: amd64
  Version: 1.0.1ubuntu2.1
  Replaces: manpages-it ( 2.80-4~), manpages-pl ( 20060617-3~), 
openjdk-6-jdk ( 6b24-1.11-0ubuntu1~), sun-java5-jdk ( 0), sun-java6-jdk ( 
0)
  Depends: libapt-pkg4.12 (= 0.9.16), libc6 (= 2.15), libgcc1 (= 1:4.1.1), 
libstdc++6 (= 4.6), ubuntu-keyring, gnupg
  Suggests: aptitude | synaptic | wajig, dpkg-dev (= 1.17.2), apt-doc, 
python-apt
  Conflicts: python-apt ( 0.7.93.2~)
  Breaks: manpages-it ( 2.80-4~), manpages-pl ( 20060617-3~), openjdk-6-jdk 
( 6b24-1.11-0ubuntu1~), sun-java5-jdk ( 0), sun-java6-jdk ( 0)
  Filename: pool/main/a/apt/apt_1.0.1ubuntu2.1_amd64.deb
  Size: 952374

  
  [pid 20525] 
stat(/var/lib/apt/lists/mirrors.163.com_ubuntu_dists_trusty-updates_universe_i18n_Translation-en%5fUS,
 0x7fffaabe5b30) = -1 ENOENT (No such file or directory)
  [pid 20525] stat(/usr/lib/apt/methods/bzip2, {st_mode=S_IFREG|0755, 
st_size=23056, ...}) = 0
  [pid 20525] pipe([9, 11])   = 0
  [pid 20525] pipe([13, 14])  = 0
  [pid 20525] fcntl(9, F_SETFD, FD_CLOEXEC) = 0
  [pid 20525] fcntl(11, F_SETFD, FD_CLOEXEC) = 0
  [pid 20525] fcntl(13, F_SETFD, FD_CLOEXEC) = 0
  [pid 20525] fcntl(14, F_SETFD, FD_CLOEXEC) = 0
  [pid 20525] clone(child_stack=0, 
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, 
child_tidptr=0x7f99f1ea9a50) = 20538
  Process 20538 attached
  [pid 20525] fcntl(9, F_GETFL)   = 0 (flags O_RDONLY)
  [pid 20525] fcntl(9, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
  [pid 20538] rt_sigaction(SIGPIPE, {SIG_DFL, [PIPE], SA_RESTORER|SA_RESTART, 
0x7f99f0e72ff0},  unfinished ...
  [pid 20525] fcntl(14, F_GETFL unfinished ...
  [pid 20538] ... rt_sigaction resumed {SIG_IGN, [PIPE], 
SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, 8) = 0
  [pid 20525] ... fcntl resumed )   = 0x1 (flags O_WRONLY)
  [pid 20538] rt_sigaction(SIGQUIT, {SIG_DFL, [QUIT], SA_RESTORER|SA_RESTART, 
0x7f99f0e72ff0},  unfinished ...
  [pid 20525] fcntl(14, F_SETFL, O_WRONLY|O_NONBLOCK unfinished ...
  [pid 20538] ... rt_sigaction resumed {SIG_DFL, [], 0}, 8) = 0
  [pid 20525] ... fcntl resumed )   = 0
  [pid 20538] rt_sigaction(SIGINT, {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 
0x7f99f0e72ff0},  unfinished ...
  [pid 20525] close(11 unfinished ...
  [pid 20538] ... rt_sigaction resumed {SIG_DFL, [], 0}, 8) = 0
  [pid 20525] ... close resumed )   = 0
  [pid 20538] rt_sigaction(SIGWINCH, {SIG_DFL, [WINCH], SA_RESTORER|SA_RESTART, 
0x7f99f0e72ff0},  unfinished ...
  [pid 20525] close(13 unfinished ...
  [pid 20538] ... rt_sigaction resumed {0x409140, [WINCH], 
SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, 8) = 0
  [pid 20525] ... close resumed )   = 0
  [pid 20538] rt_sigaction(SIGCONT, {SIG_DFL, [CONT], SA_RESTORER|SA_RESTART, 
0x7f99f0e72ff0},  unfinished ...
  [pid 20525] select(10, [9], NULL, NULL, NULL unfinished ...
  [pid 20538] ... rt_sigaction resumed {SIG_DFL, [], 0}, 8) = 0
  [pid 20538] rt_sigaction(SIGTSTP, {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 
0x7f99f0e72ff0}, {SIG_DFL, [], 0}, 8) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(3, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(4, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(5, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(6, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(7, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(8, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(9, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(10, F_SETFD, FD_CLOEXEC) = 0
  [pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
  [pid 20538] fcntl(11, F_SETFD, FD_CLOEXEC) = 0