Thanks for the quick reply. On Mon, Jan 7, 2019 at 7:37 PM Julian Andres Klode <[email protected]> wrote:
> On Mon, Jan 07, 2019 at 06:40:02PM +0100, Michael Stapelberg wrote: > > Package: apt > > Version: 1.8.0~alpha3 > > Severity: normal > > > > For the past few weeks, I haven’t been able to update my Debian system. I > > finally took some time to look into what’s going wrong. > > We'd have to figure out when this started, I guess. > Sorry, didn’t keep track of that. > > > > > The symptom is that “apt update” just hung forever after fetching all > files from > > my configured mirrors. > > > > A closer look revealed 100% CPU usage of 1 core in the > > /usr/lib/apt/methods/store process, which was writing to files in > > /var/lib/apt/lists/partial. > > > > Attaching gdb revealed that store was spending its time in a zstd > function: > > > > #0 0x00007f376cc0bf78 in ?? () from > /usr/lib/x86_64-linux-gnu/libzstd.so.1 > > #1 0x00007f376cbcc275 in ?? () from > /usr/lib/x86_64-linux-gnu/libzstd.so.1 > > #2 0x00007f376cbcdedf in ?? () from > /usr/lib/x86_64-linux-gnu/libzstd.so.1 > > #3 0x00007f376cbce461 in ZSTD_compressContinue () from > /usr/lib/x86_64-linux-gnu/libzstd.so.1 > > #4 0x00007f376cbcfe9e in ?? () from > /usr/lib/x86_64-linux-gnu/libzstd.so.1 > > #5 0x00007f376d9874af in ?? () from > /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0 > > #6 0x00007f376d97ffd1 in FileFd::Write(void const*, unsigned long long) > () from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0 > > #7 0x000055afaedf1073 in StoreMethod::Fetch(pkgAcqMethod::FetchItem*) > () at include/apt-pkg/fileutl.h:168 > > #8 0x00007f376d939f83 in pkgAcqMethod::Run(bool) () from > /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0 > > #9 0x000055afaedf0759 in main () at > /usr/include/c++/8/bits/basic_string.h:252 > > #10 0x00007f376d32e09b in __libc_start_main (main=0x55afaedf06e0 <main>, > argc=1, argv=0x7ffe0782f238, init=<optimized out>, fini=<optimized out>, > rtld_fini=<optimized out>, stack_end=0x7ffe0782f228) at > ../csu/libc-start.c:308 > > #11 0x000055afaedf083a in _start () at ../methods/store.cc:147 > > > > Sure enough, raising the cost of the zstd compression in my apt config > fixed the > > issue: > > > > % cat /etc/apt/apt.conf.d/99disable-zstd > > APT::Compressor::zstd::Cost "999"; > > > > I have a few remarks: > > > > 1. Why are the lists re-compressed at all? > > (Also note that I set Acquire::GzipIndexes "yes", so I have already > signaled > > that I would prefer to accept the lists as they are.) > > No. You have indicated that you want them stored compressed. Which picks > the > cheapest compressor, that's usually lz4. > Ugh. Is there a way to tell apt to not bother re-compressing files ever? > > Yes, it used to mean to not uncompress files and take them directly, but > that > changed a few years ago, in 1.2 or so, as lz4 basically has not much > overhead, > so it's much faster than keeping xz indexes around. > > > > > 2. Why is store only using 1 of 16 CPU cores for compression? > > store is using 1 process per file (up to a limit that's a multiple of your > cores), parallel compression of single files is meh. > That’s not what I’m observing. For me, precisely 1 store process is running, and it seems to be handling all files sequentially. > > > > > 3. Should we disable zstd for the time being, given that it turns a > 1-minute apt > > update into a many-minute apt update, to the point where I had > assumed it was > > hanging? > > It's not used by default. It's unclear to me why this happens to you. Do > you have > another config file which sets a lower zstd cost perhaps? > No, it doesn’t look like it: % grep -ri zst /etc/apt /etc/apt/apt.conf.d/99disable-zstd:APT::Compressor::zstd::Cost "999"; % > > > > -- > debian developer - deb.li/jak | jak-linux.org - free software dev > ubuntu core developer i speak de, en > -- Best regards, Michael
