Bug#645999: (lbzip2: uses excessive amounts of memory when decompressing highly compressible data)
On Thu, 20 Oct 2011 20:46:22 +0200 Miko??aj Izdebski wrote: > There is a simple way of solving this issue. Changes that need to be made: [...] > If anyone is interested I can prepare an appropriate patch. Just let me know. Miko??aj, I am interested in a patch. I looked over lbzip2 code enough to realize I couldn't think of a simple patch for this issue. I was thinking it would be simple to limit the number of memory buffers between worker and muxer. Then looking at how blocks are split into input for the workers, started to worry that this would get into trouble with deadlocks on blocks which are split between two input blocks. So I filed a bug with no patch. ;) My guess of the easiest way to "fix" this is to change the behavior of the splitter. Instead of sending random blocks of input to the worker threads for them to split, the splitter actually reads and scans for the block markers at the same time and passes along single blocks into a queue of worker threads. Then a worker who is picking up a new block to decompress can check if he is "too far ahead" of the muxer and then block/wait for the muxer to catch up. I think this approach would mean that blocks always being decompression in correct order and it's very easy to control the size of worker to muxer queue. A downside is the splitter would be both I/O and CPU bound scanning for blocks ... which could be "fixed" by having two threads a "reader" and a different "splitter" (aka "scanner). On Thu, 20 Oct 2011 20:23:05 +0200 (CEST) "Ersek, Laszlo" wrote: > I'll tell you this: you might want to check out pbzip2; I believe > pbzip2 should have a working memory throttle staring with v1.1.2. Laszlo, Yes, thanks for the suggestion. I do fallback on pbzip2 when lbzip2 fails. I used to use pbzip2, back then decompression required it to read the input file twice, first pass was single threaded and scanned for block markers, the second read was feed to worker threads to decompress... for large input streams (anything larger than fit in ram) that implementation sucked as it was input I/O bound for many minutes before generating any output. This double read also prevented it from decompressing pipes. At that time, I think due to the double I/O, lbzip2 was wall clock faster for any given input stream. I see now that the pbzip2 I/O layer has changed and it appears to support pipes and scans input only once. I should re-compare them. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#645999: lbzip2: uses excessive amounts of memory when decompressing highly compressible data
Package: lbzip2 Version: 0.23-1 Severity: important lbzip2 uses excessive amounts of memory (and is killed) when decompressing highly compressible data and the writer (aka: "muxer") is slow. I recently tried to decompress some hard drive images, which have large sections of zero filled area, unfortunately due to runaway memory usage these files were uncompressable with lbzip2. What I think is happening, the splitter pulls some blocks of highly compressed input into ram, a worker gets one and enters scan mode, which causes it to very rapidly find and queue up many of the small compressed blocks for decompression. Nothing in the code directly throttles output block creation between decompressor and muxer, so if the muxer can't write out blocks quickly enough to keep up the process size explodes. The throttling between splitter and muxer does not help here because the just one splitter is around 16G of output, even if the splitter loads only one block I won't have enough ram to decompress into ram that single input block. The simple test case below, use 100G of compressed zeros. This file has a little more than 6 blocks of input from splitter's perspective. On a machine with 6G of ram and 4 processors, it takes about 10 seconds to be killed, at that time it had used 20G of virtual memory and 6G of resident memory. The kill is fast for me because I do not have swap enabled on this machine. If swap even a small (2G) swap was enabled the system will trash various things to disk for several minutes before the OOM-killer triggers... so if you are on linux and have swap enabled, I recommend monitoring top and hitting ^C on lbzip2 when you run the test case. The simple test case: === # generate input file for problem report dd if=/dev/zero bs=1024k count=$[1024*100] | ./lbzip2 -c9 > zero.bz2 102400+0 records in 102400+0 records out 107374182400 bytes (107 GB) copied, 648.662 s, 166 MB/s # check size of file ls -l zero.bz2 -rw-r--r-- 1 user user 5726783 2011-10-20 04:44 zero.bz2 # reproduce problem, the kill -9 /usr/bin/time ./lbzip2 -cd zero.bz2 | sleep 30 Command terminated by signal 9 22.40user 5.72system 0:10.48elapsed 268%CPU (0avgtext+0avgdata 24060336maxresident)k 12312inputs+0outputs (11major+1508100minor)pagefaults 0swaps # confirm OOM kill in dmesg dmesg | grep lbzip2 | tail -2 [622439.076387] Out of memory: Kill process 10299 (lbzip2) score 739 or sacrifice child [622439.076391] Killed process 10299 (lbzip2) total-vm:20247368kB, anon-rss:6015048kB, file-rss:4kB === With this example input the muxer writing to /dev/null can't keep up with just two workers... this is killed: lbzip2 -cd -n 2 zero.bz2 > /dev/null Even running in single worker mode is killed: lbzip2 -cd -n 1 zero.bz2 | sleep 60 -- System Information: Debian Release: squeeze/sid APT prefers natty-updates APT policy: (500, 'natty-updates'), (500, 'natty-security'), (500, 'natty') Architecture: amd64 (x86_64) Kernel: Linux 2.6.38-11-generic (SMP w/4 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Versions of packages lbzip2 depends on: ii libbz2-1.01.0.5-6ubuntu1 high-quality block-sorting file co ii libc6 2.13-0ubuntu13 Embedded GNU C Library: Shared lib lbzip2 recommends no packages. lbzip2 suggests no packages. -- no debconf information -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#644637: cdparanoia: needs option to limit retries
Package: cdparanoia
Version: 3.10.2+debian-10
Severity: normal
Tags: patch
Problem is that cdparanoia has hard coded retry count of 20.
On very badly damaged media cdparanoia retries bad sectors
much too long, I've seen it try a single track for over 12 hours.
If I pass the -Z option,
all retry/jitter fixup logic is disabled (the quality audio is bad).
If I pass the -z 3 option to limit retries,
cdparanoia aborts the rest of the disk at first unreadable sector.
I want something in between, retry a few times and then move along.
Included is a patch to add the --max-retries (-x) option,
which if given limits how many times cdparanoia will retry a bad block.
By setting this to a low value (like -x 3) I can get most audio
from badly damaged disks in much less time than default of 20 retries.
This issue is similar to bug:
#163320 Add timeout to skip when ripping bad tracks
however I include a patch and took a different approach than what
was recommended in that bug. I think a timer based setting as
requested there would also be useful but is much more complicated
than this approach.
-- System Information:
Debian Release: squeeze/sid
APT prefers natty-updates
APT policy: (500, 'natty-updates'), (500, 'natty-security'), (500, 'natty')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.38-8-generic (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages cdparanoia depends on:
ii libc6 2.13-0ubuntu13 Embedded GNU C Library: Shared lib
ii libcdparanoia0 3.10.2+debian-10 audio extraction tool for sampling
cdparanoia recommends no packages.
cdparanoia suggests no packages.
-- no debconf information
diff -ur cdparanoia-3.10.2+debian/cdparanoia.1 cdparanoia-3.10.2+debian.maxretries/cdparanoia.1
--- cdparanoia-3.10.2+debian/cdparanoia.1 2011-10-06 12:44:20.0 -0700
+++ cdparanoia-3.10.2+debian.maxretries/cdparanoia.1 2011-10-06 12:48:59.0 -0700
@@ -29,6 +29,10 @@
read behavior. Implies -vQL.
.TP
+.B \-x --max-retries
+Limit the number of times cdparanoia retries a read, defaults to 20.
+
+.TP
.B \-v --verbose
Be absurdly verbose about the auto-sensing and reading process. Good
for setup and debugging.
diff -ur cdparanoia-3.10.2+debian/configure.guess cdparanoia-3.10.2+debian.maxretries/configure.guess
--- cdparanoia-3.10.2+debian/configure.guess 2006-11-14 18:26:17.0 -0800
+++ cdparanoia-3.10.2+debian.maxretries/configure.guess 2011-10-06 12:49:00.0 -0700
@@ -1,10 +1,10 @@
#! /bin/sh
# Attempt to guess a canonical system name.
# Copyright (C) 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999,
-# 2000, 2001, 2002, 2003, 2004, 2005, 2006 Free Software Foundation,
-# Inc.
+# 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010
+# Free Software Foundation, Inc.
-timestamp='2006-07-02'
+timestamp='2009-12-30'
# This file is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
@@ -27,16 +27,16 @@
# the same distribution terms that you use for the rest of that program.
-# Originally written by Per Bothner .
-# Please send patches to . Submit a context
-# diff and a properly formatted ChangeLog entry.
+# Originally written by Per Bothner. Please send patches (context
+# diff format) to and include a ChangeLog
+# entry.
#
# This script attempts to guess a canonical system name similar to
# config.sub. If it succeeds, it prints the system name on stdout, and
# exits with 0. Otherwise, it exits with 1.
#
-# The plan is that this can be called by configure scripts if you
-# don't specify an explicit build system type.
+# You can get the latest version of this script from:
+# http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD
me=`echo "$0" | sed -e 's,.*/,,'`
@@ -56,8 +56,9 @@
GNU config.guess ($timestamp)
Originally written by Per Bothner.
-Copyright (C) 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005
-Free Software Foundation, Inc.
+Copyright (C) 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000,
+2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 Free
+Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
@@ -161,6 +162,7 @@
arm*) machine=arm-unknown ;;
sh3el) machine=shl-unknown ;;
sh3eb) machine=sh-unknown ;;
+ sh5el) machine=sh5le-unknown ;;
*) machine=${UNAME_MACHINE_ARCH}-unknown ;;
esac
# The Operating System including object format, if it has switched
@@ -169,7 +171,7 @@
arm*|i386|m68k|ns32k|sh3*|sparc|vax)
eval $set_cc_for_build
if echo __ELF__ | $CC_FOR_BUILD -E - 2>/dev/null \
- | grep __ELF__ >/dev/null
+ | grep -q __ELF__
then
# Once all utilities can be ECOFF (ne
Bug#526732: libpango1.0-0: pango forces use of bold font in gnome-terminal
Package: libpango1.0-0 Version: 1.24.0-3+b1 Severity: normal After a recent restart of gnome-terminal the wrong font was being used. The new font is some bold font. The gnome-terminal UI shows the correct font name and the font looks correct in the preview area but attempts to make configuration changes had no effect on open real terminals. I use a small fixed-width font[1] in gnome-terminal and had made no preference changes to the terminal or font config changes for months. I found another machine were the my font preference was working correctly. I created a new user, confirmed the font was working. Did a full update to testing and confirmed the problem appeared in the new user account. Then using snapshot.debian.org I backed out/reapplied packages one at a time until I found that just libpango1.0 introduced this failure. this package works: libpango1.0-0=1.22.4-2 this package fails: libpango1.0-0=1.24.0-3+b1 1: The font I use is "Neep Alt" size 10 from xfonts-jmk package Test case: It's best if you use a different user account to test this, to isolate this test from your real user preferences. - apt-get install libpango1.0-0=1.22.4-2 - apt-get install xfonts-jmk - enable bitmap fonts [2] - run a new gnome-terminal --disable-factory - select Neep Alt size 10 from profile preferences - run ls / man ... observe font ... take a screen shot ;) - close down all gnome-terminals - apt-get install libpango1.0-0=1.24.0-3+b1 - run another new gnome-terminal --disable-factory expected results : fonts are unchanged actual results : second window uses bold fonts for all text 2: to enable bitmap fonts... there are at least two options to enable bitmap fonts. - per user: cat ~/.fonts.conf === false === - system wide: === rm /etc/fonts/conf.d/70-no-bitmaps.conf ln -s /etc/fonts/conf.avail/70-yes-bitmaps.conf /etc/fonts/conf.d/ === -- System Information: Debian Release: squeeze/sid APT prefers testing APT policy: (990, 'testing'), (500, 'unstable') Architecture: i386 (i686) Kernel: Linux 2.6.22.10-blitz16 (PREEMPT) Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968) Shell: /bin/sh linked to /bin/bash Versions of packages libpango1.0-0 depends on: ii libc6 2.9-4 GNU C Library: Shared libraries ii libcairo2 1.8.6-2+b1The Cairo 2D vector graphics libra ii libdatrie0 0.1.4-1 Double-array trie library ii libfontconfig1 2.6.0-3 generic font configuration library ii libfreetype6 2.3.9-4 FreeType 2 font engine, shared lib ii libglib2.0-0 2.20.0-2 The GLib library of C routines ii libpango1.0-common 1.24.0-3 Modules and configuration files fo ii libthai0 0.1.11-3 Thai language support library ii libx11-6 2:1.2.1-1 X11 client-side library ii libxft22.1.13-3 FreeType-based font drawing librar ii libxrender11:0.9.4-2 X Rendering Extension client libra ii zlib1g 1:1.2.3.3.dfsg-13 compression library - runtime libpango1.0-0 recommends no packages. libpango1.0-0 suggests no packages. -- no debconf information -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#522417: Reproduction steps for broken bit device
I'm ran into the same issue, the bit device appears to be broken in 8.64~dfsg-1 version of ghostscript. When told to output to bit device no output is generated. Here is a simple reproducer, taken straight from the brother printer drivers scripts using a cups test page postscript: # First run with 8.64~dfsg-1, get no output /usr/bin/gs -r1200 -g10200x13200 -q -dNOPROMPT -dNOPAUSE -dSAFER -sDEVICE=bit -sOutputFile=- - -c quit < /usr/share/cups/data/testprint.ps | wc 0 0 0 # Then downgrade ghostscript packages apt-get install ghostscript=8.63.dfsg.1-2 libgs8=8.63.dfsg.1-2 ghostscript-x=8.63.dfsg.1-2 gs-common=8.63.dfsg.1-2 [...] # Then run same command 8.63.dfsg.1-2, get 16 megs of output /usr/bin/gs -r1200 -g10200x13200 -q -dNOPROMPT -dNOPAUSE -dSAFER -sDEVICE=bit -sOutputFile=- - -c quit < /usr/share/cups/data/testprint.ps | wc 0 15613 1683 apt-cache policy ghostscript ghostscript: Installed: 8.63.dfsg.1-2 Candidate: 8.64~dfsg-1 Version table: 8.64~dfsg-1 0 990 http://http.us.debian.org testing/main Packages 500 http://http.us.debian.org unstable/main Packages *** 8.63.dfsg.1-2 0 500 file: ./ Packages 100 /var/lib/dpkg/status -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#425097: /usr/bin/pfmon: pfmon fails with standard debian kernels (x86-32)
Package: pfmon Version: 3.2.060926-2 Severity: normal File: /usr/bin/pfmon pfmon fails when run on a standard debian kernel... === blitz# uname -a Linux blitz 2.6.20-1-686 #1 SMP Tue Apr 24 21:52:11 UTC 2007 i686 GNU/Linux blitz:~# pfmon -l host kernel does not have perfmon support === The package documentation says that the required module is provided by 2.6 Debian kernels. /usr/share/doc/pfmon/README.Debian === Kernel Modules Required --- Your kernel must have been built with CONFIG_PERFMON set so that there will be a perfmon kernel module (note the spelling) that can be loaded when needed. For kernels prior to 2.6, this was an optional parameter. For 2.6 Debian kernels, CONFIG_PERFMON is set by default. === ... either I'm doing something wrong or the documentation needs to be revised. It appears that the PERFMON source code for x86 is not part of the standard kernel tree or debian patches (I could be looking in the wrong place). I'm hoping to see a small section added to the README.Debian file explaining where to get the debian perfmon kernel module (or explain briefly how to build a kernel with the patches applied if no debian package is available). Thanks -- System Information: Debian Release: lenny/sid APT prefers testing APT policy: (990, 'testing'), (500, 'unstable') Architecture: i386 (i686) Kernel: Linux 2.6.20-1-686 (SMP w/1 CPU core) Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968) Shell: /bin/sh linked to /bin/bash Versions of packages pfmon depends on: ii libc6 2.3.6.ds1-13 GNU C Library: Shared libraries ii libelfg00.8.6-3 an ELF object file access library ii libpfm3-3.2 3.2.060926-2 Performance Monitor Unit (PMU) -- pfmon recommends no packages. -- no debconf information -- GPG key: http://simons-clan.com/~msimons/gpg/msimons.asc Fingerprint: 524D A726 77CB 62C9 4D56 8109 E10C 249F B7FA ACBE -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#415110: /usr/bin/gij-wrapper-4.1: SocketChannel.get(ByteBuffer) does not detect EOF on a non-blocking socket
Package: gij-4.1
Version: 4.1.1-20
Severity: normal
File: /usr/bin/gij-wrapper-4.1
This bug report is a variation of bug:
http://bugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=281602
I've confirmed that bug 281602 was fixed... the source code in that
example used a blocking socket, this example uses a non-blocking socket.
To run:
===
# compile
javac test.java
# start server
java test
# connect a client, send "foo" and close after 3 seconds
echo "foo" | nc -q 3 localhost 2003
===
If you strace the java test process you should see a bunch of this after
the socket is closed:
===
ioctl(6, FIONREAD, [0]) = 0
ioctl(6, FIONREAD, [0]) = 0
ioctl(6, FIONREAD, [0]) = 0
===
GIJ never actually calls read to determine if the socket was closed,
it just assumes that because no bytes are available to read now, that
it can return 0 to caller.
sample source code:
===
import java.net.*;
import java.nio.*;
import java.nio.channels.*;
import java.util.Iterator;
public class test {
public static void main(String[] args) {
try {
Selector serverSelector;
serverSelector = Selector.open();
ServerSocketChannel ssc = ServerSocketChannel.open();
ssc.configureBlocking(true);
ssc.socket().bind(new InetSocketAddress(2003));
SocketChannel sc = ssc.accept();
sc.configureBlocking(false);
while (true) {
ByteBuffer bb = ByteBuffer.allocate(100);
int i = sc.read(bb);
if (i > 0) {
System.out.println("Read : " + i + " bytes.");
for(int j = 0; j < i; j++) {
System.out.print((char)bb.get(j));
}
}
if (i < 0) {
System.out.println("Closing : " + i);
sc.close();
return;
}
}
} catch(Exception ex) {
}
}
}
===
-- System Information:
Debian Release: 4.0
APT prefers testing
APT policy: (990, 'testing'), (500, 'unstable')
Architecture: i386 (i686)
Shell: /bin/sh linked to /bin/bash
Kernel: Linux 2.6.18-blitz8
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Versions of packages gij-4.1 depends on:
ii gcj-4.1-base4.1.1-20 The GNU Compiler Collection (gcj b
ii libc6 2.3.6.ds1-13 GNU C Library: Shared libraries
ii libgcc1 1:4.1.1-21 GCC support library
ii libgcj7-0 4.1.1-20 Java runtime library for use with
ii zlib1g 1:1.2.3-13 compression library - runtime
gij-4.1 recommends no packages.
-- debconf-show failed
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#415109: /usr/bin/gij-wrapper-4.1: SelectableChannel.configureBlocking fails to throw IllegalBlockingModeException
Package: gij-4.1
Version: 4.1.1-20
Severity: normal
File: /usr/bin/gij-wrapper-4.1
The java API for SelectableChannel requires that attempts to toggle
blocking mode with configureBlocking fail when a channel is registered.
This test program demonstrates that gcj fails to throw the expected
exception...
===
# first use sun jvm
[EMAIL PROTECTED]:~/java/listen$ ~/java/jdk1.5.0_10/bin/java test2
java.nio.channels.IllegalBlockingModeException
at
java.nio.channels.spi.AbstractSelectableChannel.configureBlocking(AbstractSelectableChannel.java:257)
at test2.main(test2.java:19)
# now gcj
[EMAIL PROTECTED]:~/java/listen$ java test2
[EMAIL PROTECTED]:~/java/listen$ reportbug -o blocking.bug `which java`
===
http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/SelectableChannel.html
===
Blocking mode
A selectable channel is either in blocking mode or in non-blocking
mode. In blocking mode, every I/O operation invoked upon the channel
will block until it completes. In non-blocking mode an I/O operation
will never block and may transfer fewer bytes than were requested or
possibly no bytes at all. The blocking mode of a selectable channel may
be determined by invoking its isBlocking method.
Newly-created selectable channels are always in blocking
mode. Non-blocking mode is most useful in conjunction with selector-based
multiplexing. A channel must be placed into non-blocking mode before
being registered with a selector, and may not be returned to blocking
mode until it has been deregistered.
===
sample code:
===
import java.net.InetSocketAddress;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.ServerSocketChannel;
import java.nio.channels.spi.SelectorProvider;
public class test2 {
public static void main(String[] args) {
try {
Selector sel = null;
InetSocketAddress isa = new InetSocketAddress(2300);
ServerSocketChannel ssc = ServerSocketChannel.open();
ssc.configureBlocking(false);
ssc.socket().bind(isa);
sel = SelectorProvider.provider().openSelector();
ssc.register(sel, SelectionKey.OP_ACCEPT);
ssc.configureBlocking(true);
} catch(Exception e) {
e.printStackTrace();
}
}
}
===
-- System Information:
Debian Release: 4.0
APT prefers testing
APT policy: (990, 'testing'), (500, 'unstable')
Architecture: i386 (i686)
Shell: /bin/sh linked to /bin/bash
Kernel: Linux 2.6.18-blitz8
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Versions of packages gij-4.1 depends on:
ii gcj-4.1-base4.1.1-20 The GNU Compiler Collection (gcj b
ii libc6 2.3.6.ds1-13 GNU C Library: Shared libraries
ii libgcc1 1:4.1.1-21 GCC support library
ii libgcj7-0 4.1.1-20 Java runtime library for use with
ii zlib1g 1:1.2.3-13 compression library - runtime
gij-4.1 recommends no packages.
-- debconf-show failed
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
