Bug#1069989: bmusb: Add Appstream metainfo announcing HW support
On Sun, Apr 28, 2024 at 09:35:46AM +0200, Steinar H. Gunderson wrote: > It would probably be more useful I package v4l2proxy, which has been part > of bmusb for a while; it would allow “anything” to go use it, although > with some local setup. I believe Nageru is the only other software that > uses bmusb right now. This is now waiting in NEW (for the new binary package). /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1069989: bmusb: Add Appstream metainfo announcing HW support
On Sun, Apr 28, 2024 at 08:22:41AM +0200, Petter Reinholdtsen wrote: > Here is a patch for bmusb to add Appstream metainfo XML announcing the > hardware handled by this package. Thanks for the patch. I assume this is also suitable for upstream? > I was a bit unsure if it should be > attached to the library or -dev package, but concluded that the -dev > package seem more useful for endusers than the library package on its > own. It would probably be more useful I package v4l2proxy, which has been part of bmusb for a while; it would allow “anything” to go use it, although with some local setup. I believe Nageru is the only other software that uses bmusb right now. > +usb:v1EDBpBD3Bd* > +usb:v1EDBpBD4Fd* Can you say something about where these values come from? Is it base64 somehow? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1066136: NMU pending for #1066136
Control: tags 1066136 + pending Hi, I've prepared an NMU for python-xapian-haystack (versioned as 2.1.1-1+deb12u0.1) and uploaded it to DELAYED/14-day. (The unusually long delay is since this is a stable upload.) NMU diff is attached. /* Steinar */ -- Homepage: https://www.sesse.net/ diff -Nru python-xapian-haystack-2.1.1/debian/changelog python-xapian-haystack-2.1.1/debian/changelog --- python-xapian-haystack-2.1.1/debian/changelog 2021-10-19 22:10:20.0 +0200 +++ python-xapian-haystack-2.1.1/debian/changelog 2024-03-19 23:35:19.0 +0100 @@ -1,3 +1,12 @@ +python-xapian-haystack (2.1.1-1+deb12u0.1) bookworm; urgency=medium + + * Non-maintainer upload, to stable. + * Remove dependency on the django.utils.six module, which no longer exists, +causing the package to be completely broken. Patch by Ole Peder Brandtzæg, +based on reducing a patch from upstream. (Closes: #1066136) + + -- Steinar H. Gunderson Tue, 19 Mar 2024 23:35:19 +0100 + python-xapian-haystack (2.1.1-1) unstable; urgency=low [ Debian Janitor ] diff -Nru python-xapian-haystack-2.1.1/debian/patches/0002-Remove-dependency-on-six.patch python-xapian-haystack-2.1.1/debian/patches/0002-Remove-dependency-on-six.patch --- python-xapian-haystack-2.1.1/debian/patches/0002-Remove-dependency-on-six.patch 1970-01-01 01:00:00.0 +0100 +++ python-xapian-haystack-2.1.1/debian/patches/0002-Remove-dependency-on-six.patch 2024-03-19 23:35:19.0 +0100 @@ -0,0 +1,32 @@ +Index: python-xapian-haystack-2.1.1/xapian_backend.py +=== +--- python-xapian-haystack-2.1.1.orig/xapian_backend.py python-xapian-haystack-2.1.1/xapian_backend.py +@@ -7,7 +7,6 @@ import re + import shutil + import sys + +-from django.utils import six + from django.conf import settings + from django.core.exceptions import ImproperlyConfigured + from django.utils.encoding import force_text +@@ -346,7 +345,7 @@ class XapianSearchBackend(BaseSearchBack + def _get_ngram_lengths(value): + values = value.split() + for item in values: +-for ngram_length in six.moves.range(NGRAM_MIN_LENGTH, NGRAM_MAX_LENGTH + 1): ++for ngram_length in range(NGRAM_MIN_LENGTH, NGRAM_MAX_LENGTH + 1): + yield item, ngram_length + + for obj in iterable: +@@ -356,8 +355,8 @@ class XapianSearchBackend(BaseSearchBack + def ngram_terms(value): + for item, length in _get_ngram_lengths(value): + item_length = len(item) +-for start in six.moves.range(0, item_length - length + 1): +-for size in six.moves.range(length, length + 1): ++for start in range(0, item_length - length + 1): ++for size in range(length, length + 1): + end = start + size + if end > item_length: + continue diff -Nru python-xapian-haystack-2.1.1/debian/patches/series python-xapian-haystack-2.1.1/debian/patches/series --- python-xapian-haystack-2.1.1/debian/patches/series 2021-10-19 22:10:20.0 +0200 +++ python-xapian-haystack-2.1.1/debian/patches/series 2024-03-19 23:35:19.0 +0100 @@ -1 +1,2 @@ 0001-Use-io.open-to-read-UTF-8-encoded-README-file.patch +0002-Remove-dependency-on-six.patch
Bug#1065435: aptitude: FTBFS on armhf and armel (probably -Werror=implicit-function-declaration related)
On Mon, Mar 04, 2024 at 07:34:06PM +0100, Sven Joachim wrote:
> Not really, these arches now default to a 64-bit time_t and therefore
> you get the conflicting types (suseconds_t is a long int,
> __suseconds64_t a long long int). This has nothing to do with implicit
> function declarations.
It's a bit crazy that tv_usec changes type, it should be in [0,100)
no matter what tv_sec is:
struct timeval
{
#ifdef __USE_TIME_BITS64
__time64_t tv_sec;/* Seconds. */
__suseconds64_t tv_usec; /* Microseconds. */
#else
__time_t tv_sec; /* Seconds. */
__suseconds_t tv_usec;/* Microseconds. */
#endif
};
#endif
But the fix is straightforward, no? Just change the offending test to something
like
CPPUNIT_ASSERT_EQUAL(static_cast(99), c.tv_usec);
and it should work no matter what type c.tv_usec is. Or, if you prefer
brace-initialization (which would prevent you from ever writing in a too-big
value, like 999 or something):
CPPUNIT_ASSERT_EQUAL(decltype(c.tv_usec){99}, c.tv_usec);
/* Steinar */
--
Homepage: https://www.sesse.net/
Bug#1060256: Please enable the Rust parts
On Wed, Feb 14, 2024 at 12:32:13PM +0200, Jonathan Carter wrote: >> Fair enough, do you want to do the honor as the maintainer? Or should >> I change the upload's version number to 24+really1.4.2~git(etc).? > Since you made the change, I think you should own it. I don't have the resources to do this at the current time, sorry. I'll remove the upload from the DELAYED queue, then others can determine what they want to do. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1060256: Please enable the Rust parts
On Wed, Feb 14, 2024 at 11:53:23AM +0200, Jonathan Carter wrote: > Thanks for the work, although Debian policy requires consensus from > debian-devel before bumping an epoch: > > https://www.debian.org/doc/debian-policy/ch-controlfields.html#version Fair enough, do you want to do the honor as the maintainer? Or should I change the upload's version number to 24+really1.4.2~git(etc).? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1060256: Please enable the Rust parts
On Tue, Feb 06, 2024 at 11:55:55AM +0200, Faidon Liambotis wrote: >> Still required. > I uploaded that last week, currently sitting in NEW. Now that this is through NEW, I uploaded an NMU to DELAYED/7-day. I see that this package is in LowThresholdNmu, but given that it adds an epoch, I'm giving everyone a week to speak up if they feel this is wrong :-) The package can be found at https://storage.sesse.net/bcachefs-tools/ and the git repository I built from at https://git.sesse.net/?p=bcachefs-tools-debian I used (with upstream being github.com/koverstreet/bcachefs-tools) gbp buildpackage --git-upstream-tree=upstream/master and then built with pbuilder. I've tested that this works fine with bcachefs as root device on my workstation (which also has some patches for multi-device root; see #1060256). Thanks to Faidon for paving the way here :-) /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1061775: shlibs is stricter than it needs to be
On Mon, Jan 29, 2024 at 06:42:41PM +0200, Peter Pentchev wrote: > Hmmm, my understanding might be wrong, but when using dh_makeshlibs > directly without a symbols file, isn't it going to be a problem if > a program is built against libzstd 1.5.5 and it uses the new > ZSTD_CCtx_setFParams() function? If that function is new in 1.5.5, then indeed, yes. I didn't see that from the changelog, but perhaps I misunderstood something or it just flew under the radar. > Huh. That's actually pretty cool, so now I can start using a symbols > file for libzstd, too! Nice! > OK, so since I still believe that lowering the version given to > dh_makeshlibs would break programs that use the two new functions in > 1.5.5, what do you think about the idea to start using symbols files > instead? I fully agree this would be the best solution, yes, assuming manpower exists for it. :-) /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1061775: shlibs is stricter than it needs to be
Package: libzstd-dev Version: 1.5.5+dfsg2-2 Severity: normal Hi, debian/rules contains dh_makeshlibs -plibzstd1 -V'libzstd1 (>= 1.5.5)' --add-udeb=libzstd1-udeb Would it be possible to change it to 1.5.4? I don't see anything between 1.5.4 and 1.5.5 that would mean 1.5.5-built packages wouldn't work on 1.5.4, and this would allow trixie-built .debs to be installable on bookworm. (Of course, it would break if upstream truly introduces new APIs, but that's fine.) Ideally, of course, one would ship a .symbols file instead of .shlibs, but that requires more careful tracking. -- System Information: Debian Release: 12.4 APT prefers stable-security APT policy: (500, 'stable-security'), (500, 'stable-debug'), (500, 'proposed-updates'), (500, 'stable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 6.7.0 (SMP w/56 CPU threads; PREEMPT) Kernel taint flags: TAINT_WARN Locale: LANG=en_DK.UTF-8, LC_CTYPE=en_DK.UTF-8 (charmap=UTF-8), LANGUAGE=en_NO:en_US:en_GB:en Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) Versions of packages libzstd-dev depends on: ii libzstd1 1.5.5+dfsg2-2 libzstd-dev recommends no packages. libzstd-dev suggests no packages. -- debconf-show failed
Bug#1061525: does not boot from multi-device root filesystems
On Sun, Jan 28, 2024 at 10:06:48PM +0100, Steinar H. Gunderson wrote: >> - The initramfs scripts attempt to rewrite UUID= _back_ to a >>single /dev device through probing, and give that to mount. It needs >>to avoid doing so for (multi-device) bcachefs filesystems, or they >>will never mount since they contain too few devices. > Fix in #1060411. Also a patch for klibc's fstype, though I figured afterwards that blkid seems to work just fine, in #1061662. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1061525: does not boot from multi-device root filesystems
forward 1061525 https://savannah.gnu.org/bugs/index.php?65151 block 1061525 by 1060256 block 1061525 by 1060411 tags 1061525 + patch kthxbye On Thu, Jan 25, 2024 at 10:44:04PM +0100, Steinar H. Gunderson wrote: > - Likewise, root= on the kernel command line must contain the UUID >or the initramfs scripts will wait on some colon-separated device that does >not (and will never) exist. grub-mkconfig must be updated to write root= >_back_ from a colon-separated device list to a UUID; I don't know if there >is already some kind of support for UUID-roots, but I think so. >As of today, it fails with: > > /usr/sbin/grub-probe: error: failed to get canonical path of > `/dev/nvme1n1p1:/dev/dm-2'. Patch in https://savannah.gnu.org/bugs/index.php?65151 . > - initramfs must contain mount.bcachefs, since that is the only thing >that knows how to probe a UUID into multiple devices. This means that >the Rust parts needs to be built again, too (see #1060256). #1060256 now has a patch/unofficial package. > - The initramfs scripts attempt to rewrite UUID= _back_ to a >single /dev device through probing, and give that to mount. It needs >to avoid doing so for (multi-device) bcachefs filesystems, or they >will never mount since they contain too few devices. Fix in #1060411. I've verified that this makes my multi-device root filesystem (specified with UUID= in fstab, and without errors=remount-ro) configure GRUB and boot on its own, without any obvious issues. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1060411: initramfs: can't boot bcachefs storage with multi devices
On Sun, Jan 28, 2024 at 11:06:53AM +0100, Steinar H. Gunderson wrote:
> I'm a bit unsure how this would work; unless you manually set
> rootfstype=bcachefs on the GRUB command line, I think this is autodetected
> from fstype (in klibc-utils), which doesn't understand bcachefs right now?
Evidently, blkid does, so that part is fine.
With the attached patch, and bcachefs-tools upgraded to a version with
mount.bcachefs included, I can boot with root=UUID= using a multi-device
filesystem. grub-mkconfig still errors out, though.
/* Steinar */
--
Homepage: https://www.sesse.net/
diff -Nru initramfs-tools-0.142/scripts/local initramfs-tools-0.142/scripts/local
--- initramfs-tools-0.142/scripts/local 2022-04-10 21:59:31.0 +0200
+++ initramfs-tools-0.142/scripts/local 2022-07-12 23:51:34.0 +0200
@@ -148,15 +148,30 @@
panic "No root device specified. Boot arguments must include a root= parameter."
fi
local_device_setup "${ROOT}" "root file system"
- ROOT="${DEV}"
# Get the root filesystem type if not set
if [ -z "${ROOTFSTYPE}" ] || [ "${ROOTFSTYPE}" = auto ]; then
- FSTYPE=$(get_fstype "${ROOT}")
+ FSTYPE=$(get_fstype "${DEV}")
else
FSTYPE=${ROOTFSTYPE}
fi
+ # If we have an UUID bcachefs mount, we must not set $ROOT to the real device,
+ # since the filesystem may be a multi-device filesystem. If so, we must keep
+ # the UUID= argument; mount.bcachefs will figure out what devices to include.
+ if [ "${FSTYPE}" = "bcachefs" ]; then
+ case "$ROOT" in
+ UUID=*)
+ ;;
+ *)
+ ROOT="${DEV}"
+ ;;
+ esac
+ else
+ # For all other filesystems, we resolve to the actual device.
+ ROOT="${DEV}"
+ fi
+
local_premount
if [ "${readonly?}" = "y" ]; then
Bug#1061662: fstype support for bcachefs
Package: klibc-utils
Version: 2.0.12-1
Severity: normal
Tags: upstream patch
Hi,
fstype should autodetect bcachefs, not the least because bcachefs filesystems
may require other treatment of UUID mounts (#1060411, #1061525). I've attached
a simple patch.
-- System Information:
Debian Release: 12.4
APT prefers stable-security
APT policy: (500, 'stable-security'), (500, 'stable-debug'), (500,
'proposed-updates'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 6.7.0 (SMP w/56 CPU threads; PREEMPT)
Kernel taint flags: TAINT_WARN
Locale: LANG=en_DK.UTF-8, LC_CTYPE=en_DK.UTF-8 (charmap=UTF-8),
LANGUAGE=en_NO:en_US:en_GB:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages klibc-utils depends on:
ii libklibc 2.0.12-1
klibc-utils recommends no packages.
klibc-utils suggests no packages.
-- debconf-show failed
Description:
TODO: Put a short summary on the line above and replace this paragraph
with a longer explanation of this change. Complete the meta-information
with other relevant fields (see below for details). To make it easier, the
information below has been extracted from the changelog. Adjust it or drop
it.
.
klibc (2.0.13-3) experimental; urgency=medium
.
* Install klibc shared library in /usr/lib
Author: Ben Hutchings
---
The information above should follow the Patch Tagging Guidelines, please
checkout https://dep.debian.net/deps/dep3/ to learn about the format. Here
are templates for supplementary fields that you might want to add:
Origin: (upstream|backport|vendor|other), (|commit:)
Bug:
Bug-Debian: https://bugs.debian.org/
Bug-Ubuntu: https://launchpad.net/bugs/
Forwarded: (no|not-needed|)
Applied-Upstream: , (|commit:)
Reviewed-By:
Last-Update: 2024-01-28
Index: klibc-2.0.13/usr/kinit/fstype/bcachefs_fs.h
===
--- /dev/null
+++ klibc-2.0.13/usr/kinit/fstype/bcachefs_fs.h
@@ -0,0 +1,20 @@
+#ifndef __BCACHEFS_FS_H
+#define __BCACHEFS_FS_H
+
+#define BCACHEFS_MAGIC \
+ {0xc6, 0x85, 0x73, 0xf6, 0x66, 0xce, 0x90, 0xa9, \
+0xd9, 0x6a, 0x60, 0xcf, 0x80, 0x3d, 0xf7, 0xef};
+
+/*
+ * Structure of the superblock
+ */
+struct bch_sb_layout {
+ __u8 magic[16]; /* bcachefs superblock UUID */
+ __u8 layout_type;
+ __u8 sb_max_size_bits; /* base 2 of 512 byte sectors */
+ __u8 nr_superblocks;
+ __u8 pad[5];
+ __le64 sb_offset[61];
+} __attribute__((packed));
+
+#endif /* __BCACHEFS_FS_H */
Index: klibc-2.0.13/usr/kinit/fstype/fstype.c
===
--- klibc-2.0.13.orig/usr/kinit/fstype/fstype.c
+++ klibc-2.0.13/usr/kinit/fstype/fstype.c
@@ -23,6 +23,7 @@
#define cpu_to_be32(x) __cpu_to_be32(x)/* Needed by romfs_fs.h */
+#include "bcachefs_fs.h"
#include "btrfs.h"
#include "cramfs_fs.h"
#include "ext2_fs.h"
@@ -161,6 +162,24 @@ static int ext2_image(const void *buf, u
return 0;
}
+static int bcachefs_image(const void *buf, unsigned long long *bytes)
+{
+ static const __u8 BCACHEFS_MAGIC_STRING[16] = BCACHEFS_MAGIC;
+ /* layout superblock starts at sector 7 */
+ const struct bch_sb_layout *sb =
+ (const struct bch_sb_layout *)((const char *)buf + 512);
+
+ if (memcmp(sb->magic, BCACHEFS_MAGIC_STRING,
+ sizeof(BCACHEFS_MAGIC_STRING)) == 0) {
+ /* Knowing the size of the filesystem can potentially
+ entail opening multiple devices and looking into their
+ superblocks */
+ *bytes = 0;
+ return 1;
+ }
+ return 0;
+}
+
static int reiserfs_image(const void *buf, unsigned long long *bytes)
{
const struct reiserfs_super_block *sb =
@@ -389,6 +408,7 @@ static struct imagetype images[] = {
{1, "minix", minix_image},
{1, "nilfs2", nilfs2_image},
{2, "ocfs2", ocfs2_image},
+ {3, "bcachefs", bcachefs_image},
{8, "reiserfs", reiserfs_image},
{64, "reiserfs", reiserfs_image},
{64, "reiser4", reiser4_image},
Bug#1060411: initramfs: can't boot bcachefs storage with multi devices
On Wed, Jan 10, 2024 at 09:34:18PM +0100, antonio wrote:
> The problem is in the file "/usr/share/initramfs-tools/scripts/functions" and
> depends on the "get_fstype" and "resolve_device" functions that cannot locate
> or determine the file system (since bcachefs uses the form
> device:device:device).
See also #1061525.
> get_fstype ()
> {
> + if [ "$ROOTFSTYPE" == "bcachefs" ]; then
> + FSTYPE="$ROOTFSTYPE"
> + echo "$ROOTFSTYPE"
> + return 0
> + fi
I'm a bit unsure how this would work; unless you manually set
rootfstype=bcachefs on the GRUB command line, I think this is autodetected
from fstype (in klibc-utils), which doesn't understand bcachefs right now?
/* Steinar */
--
Homepage: https://www.sesse.net/
Bug#1060256: Please enable the Rust parts
On Mon, Jan 08, 2024 at 12:45:35PM +0200, Faidon Liambotis wrote: > The bcachefs-tools package builds the C portion of the tarball, and not > the parts written in Rust (under rust-src/). This results in a crippled > functionality, such as the missing "mount" binary (#1057295). I took a stab at enabling the Rust parts, now that upstream git no longer needs its own bindgen: https://storage.sesse.net/bcachefs-tools/ I have no experience in this, but the package builds in a pbuilder and seems to work OK (I get mount.bcachefs in the initramfs, which I can use for an UUID-based mount if I do it manually). I see that bcachefs-tools is under LowThresholdNmu, but adding a version epoch in an NMU seems a bit dubious :-) It also includes the missing udev crate, but someone else needs to figure out if that dependency is properly packaged or not (it comes straight from debcargo, modulo debian/copyright changes). /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1061525: does not boot from multi-device root filesystems
On Thu, Jan 25, 2024 at 10:44:04PM +0100, Steinar H. Gunderson wrote: > - The GRUB command line must be rw, not ro; mounting with -o remount,rw >gives: “bcachefs: bch2_parse_mount_opts() Invalid mount option errors: >invalid selection”. I don't know if this is an upstream issue or if >this is just considered a bug in our setup. This is evidently due to “errors=remount-ro” in fstab; removing that fixes it. But it's probably an upstream bug, since supposedly that's a valid option. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1061525: does not boot from multi-device root filesystems
Package: bcachefs-tools Version: 24+really1.3.4-2 Severity: normal I have / as a multi-device bcachefs filesystem (two different SSDs, with replicas=1). Booting from it was an, well, interesting endeavor :-) It seems the following must be done in Debian before this Just Works(TM): - /etc/fstab must contain the UUID, not a colon-separated list of devices, or systemd will wait forever (because it doesn't understand the syntax; I believe this is an upstream issue). - Likewise, root= on the kernel command line must contain the UUID or the initramfs scripts will wait on some colon-separated device that does not (and will never) exist. grub-mkconfig must be updated to write root= _back_ from a colon-separated device list to a UUID; I don't know if there is already some kind of support for UUID-roots, but I think so. As of today, it fails with: /usr/sbin/grub-probe: error: failed to get canonical path of `/dev/nvme1n1p1:/dev/dm-2'. - initramfs must contain mount.bcachefs, since that is the only thing that knows how to probe a UUID into multiple devices. This means that the Rust parts needs to be built again, too (see #1060256). - The initramfs scripts attempt to rewrite UUID= _back_ to a single /dev device through probing, and give that to mount. It needs to avoid doing so for (multi-device) bcachefs filesystems, or they will never mount since they contain too few devices. - The GRUB command line must be rw, not ro; mounting with -o remount,rw gives: “bcachefs: bch2_parse_mount_opts() Invalid mount option errors: invalid selection”. I don't know if this is an upstream issue or if this is just considered a bug in our setup. I don't have patches for any of this, unfortunately; I've just finagled it by hand in my local system. But it's at least a laundry list :-) /* Steinar */
Bug#1060217: plocate command blocks waiting io_uring operation
On Wed, Jan 10, 2024 at 11:51:22PM +0100, Manolis Stamatogiannakis wrote: > But I take this as a good opportunity to learn a bit about io_uring, so > I'll give it a shot myself. From my first experiments, it appears that the > code is deadlocking somewhere in IOUringEngine::finish(). Anything else would be surprising, given that this is the part that talks to io_uring. io_uring is fairly straightforward; you put out a request and then eventually it comes back with a result. The challenge for plocate is that you want to have a balance between throughput and latency; the optimal for throughput would probably be to just send every single request down to io_uring at once, but since results need to be printed in-order (anything else would be very hard to interpret for the user), that would risk showing nothing at screen before very late on, since there is no internal prioritization between the results. So plocate tries to keep only a given number of requests in flight to ensure that earlier requests are not starved by later ones, and then delay printing results until it's sure that it has the correct result up to a given point (i.e., all requests that could print something earlier have come back). This is the source of most of the complexity in there. (Also, things like path traversal that needs multiple semi-dependent requests...) > I have temporarily cloned the repo on GH [1] if you have time to check it > out. I cannot debug your debugging code, sorry. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1060217: plocate command blocks waiting io_uring operation
On Tue, Jan 09, 2024 at 07:34:56PM +0100, Manolis Stamatogiannakis wrote: > I also gave it a shot reproducing on a RPi Zero 2, but it's either too fast > (even with cpulimit), or the issue is architecture-specific and does not > manifest on ARMv7. Can I claim “this is obviously a kernel bug” and pass the buck? :-) I mean, if you would be willing to give out access on your machine, I could SSH in and debug there, but obviously this is a pretty narrow case no matter what the actual issue is. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1060217: plocate command blocks waiting io_uring operation
On Mon, Jan 08, 2024 at 08:44:25AM +0100, Steinar H. Gunderson wrote: > Doing so, thanks. I could probably see if Debian has a porterbox where I can > reproduce this, but I'm not too optimistic (and I don't have a lot of extra > time to dedicate to this right now, unfortunately). There's a single Marvell porterbox, but it runs armhf (not armel) and a 4.9.0 kernel, so no, there's no simple way for me to reproduce this. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1060217: plocate command blocks waiting io_uring operation
tag 1060217 + unreproducible thanks On Mon, Jan 08, 2024 at 12:01:07AM +0100, Manolis Stamatogiannakis wrote: > So, feel free to mark this as unreproducible for now. Doing so, thanks. I could probably see if Debian has a porterbox where I can reproduce this, but I'm not too optimistic (and I don't have a lot of extra time to dedicate to this right now, unfortunately). > Since you are also the author of plocate, it may make sense to expose the > -DWITHOUT_URING flag > through meson to make compiling without io_uring a bit more > straightforward. (Trival patch attached.) That doesn't really sound like a good long-term solution, though. :-) If there really is a bug here, and it is in plocate, it should “just” be fixed. Especially since most users are unlikely to go in and recompile their packages with such flags. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1060217: plocate command blocks waiting io_uring operation
On Sun, Jan 07, 2024 at 08:21:35PM +0200, Manolis Stamatogiannakis wrote: > I am trying to get plocate to run on an old ARM-based NAS running Debian > bookworm. Building the database with updatedb works fine, but plocate > command itself blocks forever without giving any results back. > > I attached gdb to the process, and the source of the problem appears to > be related to the use of io_uring. > > Trying to understand what is going on, I ran plocate under strace as > root. Surprisingly, it ran without blocking. This is interesting, but fundamentally pretty impossible to debug from my side. It sounds very much like a timing-related bug, which is probably why strace affects it (everything goes much slower); another common case is that one attaches strace to a running process, which causes the io_uring syscall to return EINTR, which gets things running again. Is this reproducible every time? >* reports an error for not being able to access the database when run > under strace as user (which sounds like the expected behaviour) Yes, this is expected. > Architecture: armel (armv5tel) > Kernel: Linux 6.1.0-17-marvell (UP) Is this even supported by Debian anymore? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1025099: plocate: autofs pruning doesn't seem to work
On Wed, Nov 30, 2022 at 12:09:56AM +0100, Steinar H. Gunderson wrote: > Most of this logic is inherited from mlocate's updatedb, though not all. > It may be fixable or it may not; I'd really need to check. But it really > sounds like one should be able to stat() something without bad things > happening :-) I guess as a workaround, you can ask systemd to sandbox away > the mount for updatedb? Hi, plocate 1.1.21 doesn't fully fix this, but it gives you a new possible workaround; if you add the path (/mnt/storage in your case) to PRUNEPATHS, updatedb won't be entering it. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1058933: cubemap: introduced new file in /lib
On Mon, Dec 18, 2023 at 04:33:26PM +0100, Chris Hofstaedtler wrote:
> I don't think there is anything you can "ask" about this.
>
> Generally the idea is that in trixie and later, --prefix=/usr really
> means that. Anything that excluded subdirs from ${prefix} should be
> a thing of the past. If the upstream build system ignores ${prefix}
> for a certain location, you'll need to override it :-(
Well, it's pretty straight-up autoconf (no automake). Makefile.in contains
PREFIX=@prefix@
SYSCONFDIR=@sysconfdir@
LOCALSTATEDIR=@localstatedir@
LIBDIR=@libdir@
and installs into $(LIBDIR).
dh_auto_configure runs configure with (among others)
--prefix=/usr --libdir=\${prefix}/lib/x86_64-linux-gnu
But nothing ever defines lowercase ${prefix}. It's possible that it expects
the upstream Makefile to do
prefix=$(PREFIX)
or something similar? But it's unclear to my why it doesn't just do
--libdir=/usr/lib/x86_64-linux-gnu
/* Steinar */
--
Homepage: https://www.sesse.net/
Bug#1058933: cubemap: introduced new file in /lib
On Mon, Dec 18, 2023 at 03:28:54PM +0100, Chris Hofstaedtler wrote:
> cubemap 1.5.1-1 introduced a new file into
> /lib/${DEB_HOST_MULTIARCH}. This is diametral to the ongoing
> UsrMerge effort [1].
Can you say something about where I should get libdir from?
Some dpkg invocation?
/* Steinar */
--
Homepage: https://www.sesse.net/
Bug#1057242: bmusb: will FTBFS when udev.pc changes udevdir
On Sat, Dec 02, 2023 at 02:32:03AM +0100, Chris Hofstaedtler wrote: > thank you for forwarding the Makefile changes from #1056997 to upstream. > The upstream change works as expected. > > However, udev.pc will change udevdir soon. When this happens, bmusb will > FTBFS. The upstream build system will install the udev rules into the > new path, but the Debian packaging expects them in the old path. Oops, sorry! I noticed that 0.26 upstream already had fixed the Makefile parts, and then noticed a bit too late that you also had debian/ changes in your patch. :-) I should have made a new upload immediately, of course. > For your convenience I'm attaching a patch to fix this. Thank you again! I'll apply it basically except for the author in the changelog. > Please apply at your earliest convenience. Per the wiki [1], it is useful > to first upload to experimental and wait a few days for the dumat tool > evaluating the change, and only then upload to unstable. OK. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1056997: libbmusb6: Let udev.pc decide where to install rules
On Mon, Nov 27, 2023 at 06:57:05PM +0100, Chris Hofstaedtler wrote: > your package ships a udev rules file in /lib/udev/rules.d, and currently > hard-codes this path. As part of the UsrMerge effort[1], the install > path for udev rules must and will change soon. To pick up this change > with a binNMU, your package can use pkg-config to determine the install > location. > > I'm attaching a patch to implement this. Hi. Your patch is modifying debian/patches/debian-multiarch.diff, but this has nothing to do with multiarch, right? It should just be a completely separate patch, which should go upstream? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1055450: Plocate's database easily becomes corrupted, resulting in locate finding nothing
On Mon, Nov 06, 2023 at 07:15:55PM +0100, Alain Knaff wrote: > But then, shouldn't it keep the shorter path (if both are the root of their > filesystem)? There's no heuristic that will work in all cases. What is a “shorter” path anyway; is /var/spool/tmp shorter or longer than /mnt/tmp-spool-mount? > And even if it used the longer path, shouldn't it find files which I locate > under that longer path? > > And during investigation, I added /run/schroot to PRUNEPATHS so updatedb > shouldn't have "seen" the longer path at all. The failure to locate files in > my home directory happened both before and after this addition. I assume that it pruned out /run for being a tmpfs (though I haven't checked the logs). I guess the interaction between the prune criteria could be improved, but it's not necessarily trivial, because we don't know the filesystem type until we've started indexing the path. (Anything else opens up a slew of time-of-check-to-time-of-use issues.) All of this logic comes straight from mlocate, FWIW. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1055450: Plocate's database easily becomes corrupted, resulting in locate finding nothing
On Mon, Nov 06, 2023 at 06:43:33PM +0100, Alain Knaff wrote: > Nov 06 18:33:15 hitchhiker updatedb.plocate[98659]: => adding `/home' > (duplicate of mount point > `/run/schroot/mount/buster-53c7e4fc-0416-4408-8421-959dc1fdaa1d/home') So your /home is mounted in two places, and updatedb picks one of them. There's no way of knowing which one you want, I guess? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1055450: Plocate's database easily becomes corrupted, resulting in locate finding nothing
On Mon, Nov 06, 2023 at 03:09:48PM +0100, Alain Knaff wrote: > On 2 separate Debian 12 machines, I'm observing the following issue: > > Search for a file that obviously exists returns nothing. > > Running updatedb and then locate doesn't fix this. > > Removing /var/lib/plocate/plocate.db, and then re-running updatedb does fix > it... until the issue reappears again several days later. I haven't had reports that this is a common occurrence, no. There were some issues with upgrades from mlocate, but nothing obvious. You could in theory dump out what's in plocate.db and see what it looks like? (E.g. through plocate '' as root.) What filesystem is this? Could there be something very wrong with timestamps? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1055450: Plocate's database easily becomes corrupted, resulting in locate finding nothing
On Mon, Nov 06, 2023 at 03:41:50PM +0100, Alain Knaff wrote: > On the box where plocate.db is currently corrupted, /home is a btrfs. It it by any chance a subvolume? (If so, known btrfs bug/design issue; see the updatedb.conf man page.) /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1055259: please include MPTCP patch
Source: openssh Version: 1:9.2p1-2+deb12u1 Severity: wishlist Hi, MPTCP support has been available in a pull request against upstream for a while: https://github.com/openssh/openssh-portable/pull/335 Unfortunately, upstream does not want it because OpenBSD doesn't support MPTCP. Would it be possible for Debian to pull it in? I believe network-manager and everything else should be MPTCP-ready now, giving you essentially mosh-like roaming straight out of the box. The patch is essentially only adding an option to change from IPPROTO_TCP to IPPROTO_MPTCP and then add some documentation. -- System Information: Debian Release: 12.2 APT prefers stable-security APT policy: (500, 'stable-security'), (500, 'stable-debug'), (500, 'proposed-updates'), (500, 'stable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 6.4.10 (SMP w/56 CPU threads; PREEMPT) Locale: LANG=en_DK.UTF-8, LC_CTYPE=en_DK.UTF-8 (charmap=UTF-8), LANGUAGE=en_NO:en_US:en_GB:en Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system)
Bug#1052934: plocate: Error running inside LXC container using systemd service (timer) with PrivateNetwork=true set
On Tue, Sep 26, 2023 at 05:57:03PM +0100, Alastair Sherringham wrote: > Is a re-assignment to LXC something you do, or I do? Anyone can do it; you probably know better than me what the package name is. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1052934: plocate: Error running inside LXC container using systemd service (timer) with PrivateNetwork=true set
On Tue, Sep 26, 2023 at 04:11:12PM +0100, Alastair Sherringham wrote: > Yes, probably something somewhere else. Maybe a library plocate uses breaks > with "PrivateNetwork" on. I do not know enough about the internals of > containers, namespaces or systemd to know. No, my point is; I don't see that this is plocate-specific at all. Every service or timer that uses PrivateNetwork=yes will be affected by this. So plocate is the wrong place to fix it, and also the wrong place to document it. I guess a good place to start would be to reassign the bug to LXC and see if they can work something out with systemd. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1052934: plocate: Error running inside LXC container using systemd service (timer) with PrivateNetwork=true set
On Tue, Sep 26, 2023 at 02:45:43PM +0100, Alastair Sherringham wrote: > So there seems to be a problem with the systemd "PrivateNetwork" and > plocate inside an LXC container - which might not surprise due to LXC > using namespace magic as well. Hi, Thanks for tracking this down. To me, this sounds like a bug in either systemd or LXC; plocate isn't involved at all? I mean, there's nothing I can change in plocate to fix this issue, short of just not using the not-working systemd option. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1050738: plocate: please add "9p" to PRUNEFS= (WSL2 support)
On Mon, Aug 28, 2023 at 10:10:18PM +0200, Alexandre Detiste wrote: > It was like 1h then we cancelled with ^C. But normally it runs through cron/systemd, and then it should be fast the next few times? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1050738: plocate: please add "9p" to PRUNEFS= (WSL2 support)
On Mon, Aug 28, 2023 at 09:37:01PM +0200, Alexandre Detiste wrote: > Microsoft's "Windows Subsystem for Linux 2" emulator > is reusing the old/experimental '9p' filesystem > to mount the Windows filesystems inside then Debian container. > > This makes plocate takes forever to index local (/remote ?) files. Hi, What is “forever”? One would imagine that if the filesystem is mounted, it contains files that would also be useful to index? All the others that are ignored are either special filesystems that don't actually contain any “files” in the usual sense (e.g. procfs, cgroup2), or things that genuinely must be reached over the network (e.g. nfs, lustre, cifs). I don't see that the WSL host filesystem falls into any of those categories. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1041721: nageru: ThemeMenu items all share the same checked state
On Sat, Jul 22, 2023 at 05:44:56PM +0200, Stefano Rivera wrote: > You don't have any kind of upstream bug tracker, do you? Unfortunately not :-) Feel free to report bugs here. I've fixed this one in upstream now, updated Debian package is on its way. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1041711: libmovit8: nageru fails to compile a shader on startup
On Sat, Jul 22, 2023 at 03:40:32PM +, stefa...@debian.org wrote: > Nope, same error with LC_ALL=C. > Nicolas next to me (also on an AMD system running Debian unstable) hits > exactly the same bug. I don't have any AMD machines anymore, unfortunately, only Intel and NVidia. (Back when I had one, the drivers used to be untenably buggy...) Is there any way I can perhaps SSH into a system or similar to try to figure out what's going on? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1041711: libmovit8: nageru fails to compile a shader on startup
On Sat, Jul 22, 2023 at 04:09:26PM +0200, Stefano Rivera wrote: > I get this crash on nageru startup (AMD box, VA-API doesn't work, but > that's another issue). > > Rolling back libmovit8 to 1.6.3-5 gets it working again. This is very weird. It seems there's a truncation somewhere: > /* 811 */ > /* 812 */ #undef LOAD_P#undef FUNCNAME > /* 813 */ #undef INPUT1 > /* 814 */ #undef INPUT2 This should read: #undef LOAD_PIXEL_BLOCK #undef DIFF #undef YADIF_ENABLE_SPATIAL_INTERLACING_CHECK #undef FUNCNAME #undef INPUT1 #undef INPUT2 I suppose there's nothing odd locale-wise or similar about your setup? It seems to be concatenated just fine here, and the string appears OK enough in the binary, but of course, the length could somehow be broken. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1040300: plocate: Transition from mlocate during upgrade to Bookworm introduced breaking changes
On Wed, Jul 05, 2023 at 12:41:51PM +0200, Larsen wrote: > ...and then output the results for all files at once, leading to the same > behaviour as mlocate while being much faster. Because you would have to worry about deduplication of the results, which is nontrivial to do without incurring unbounded memory costs and/or messing up the expected ordering of the output. In short, it's not worth it. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1040300: plocate: Transition from mlocate during upgrade to Bookworm introduced breaking changes
On Tue, Jul 04, 2023 at 11:22:15AM +0200, Steinar H. Gunderson wrote: >> Instead of using this: >> locate --existing dpkg-dist dpkg-new dpkg-old dpkg-bak ucf-dist ucf-new >> ucf-old | egrep -v >> "dpkg-distaddfile|dpkg_dateien_vor_update_|/var/backup/burp|/root/upgrade_auf_" > Yes. The “simplest” workaround is probably I thought of something simpler: locate --regex '(dpkg-dist|dpkg-new|dpkg-old|dpkg-bak|ucf-dist|ucf-new|ucf-old)' but it will be very slow; almost as slow as mlocate. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1040342: RM: mlocate -- NBS; transitional package
Package: ftp.debian.org Severity: normal User: ftp.debian@packages.debian.org Usertags: remove X-Debbugs-Cc: mloc...@packages.debian.org Control: affects -1 + src:plocate Hi, After plocate 1.1.19-2, the mlocate binary package is no longer built (it used to be a transitional package built by plocate), and can be removed. It was used in bookworm as a transitional package to move from mlocate to plocate.
Bug#1040300: plocate: Transition from mlocate during upgrade to Bookworm introduced breaking changes
tags 1040300 + wontfix thanks On Tue, Jul 04, 2023 at 10:34:34AM +0200, Larsen wrote: > with the upgrade from Bullseye to Bookworm, mlocate got replaced by > plocate. This introduced a breaking change as there is no OR-mode anymore > on which I rely on in scripts and commands. "apt-listchanges" didn't > mention this when I performed the upgrade. It is correct that there is no OR-mode, nor is any planned (it is a major pain to implement performantly and correctly, and fairly niche). If you wish to reintroduce mlocate, the transitional package for mlocate will be gone in bookworm+1 and you are free to maintain it, but note that there is no upstream development anymore, so you would also be de-facto upstream. > Instead of using this: > locate --existing dpkg-dist dpkg-new dpkg-old dpkg-bak ucf-dist ucf-new > ucf-old | egrep -v > "dpkg-distaddfile|dpkg_dateien_vor_update_|/var/backup/burp|/root/upgrade_auf_" > > I now have to run this, which (although certainly could be improved) is > pretty cumbersome compared to the one liner before. > locate --existing dpkg-dist | egrep -v ... > locate --existing dpkg-new | egrep -v ... > locate --existing dpkg-old | egrep -v ... > locate --existing dpkg-bak | egrep -v ... > locate --existing ucf-dist | egrep -v ... > locate --existing ucf-new | egrep -v ... > locate --existing ucf-old | egrep -v ... Yes. The “simplest” workaround is probably (locate --existing dpkg- ; locate --existing ucf-) | grep -E '(dpkg-(dist|new|old|bak)|(ucf-(dist|new|old)))' | egrep -v etc. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1039503: wrongly splits up extended grapheme clusters (like certain emoji)
On Tue, Jun 27, 2023 at 12:49:53AM +0200, Steinar H. Gunderson wrote:
> The attached patch seems to get us halfway there; screen now combines all of
> them correctly into one cluster. However, it's still split for whatever
> reason; only if I redraw (C-a l) the flag shows up, and the text breaking is
> wrong, so I assume there's at least something wrong with the width
> calculation, and possibly lots of other things I've not thought of. :-)
Version 2 of the patch; this seems to actually work. Only lightly tested,
though, and there are enough subtleties in the code that it probably needs
review from someone who knows screen from before.
/* Steinar */
--
Homepage: https://www.sesse.net/
diff -ur /home/sesse/nmu/orig/screen-4.9.0/ansi.c ../ansi.c
--- /home/sesse/nmu/orig/screen-4.9.0/ansi.c 2023-06-27 22:53:44.0 +0200
+++ ../ansi.c 2023-06-27 23:00:26.667216364 +0200
@@ -694,9 +694,9 @@
}
curr->w_rend.font = 0;
}
- if (curr->w_encoding == UTF8 && c >= 0x0300 && utf8_iscomb(c))
+ if (curr->w_encoding == UTF8 && c >= 0x0300)
{
- int ox, oy;
+ int ox, oy, c_prev;
struct mchar omc;
ox = curr->w_x - 1;
@@ -718,15 +718,35 @@
omc.mbcs = 0xff;
}
}
- if (ox >= 0)
- {
- utf8_handle_comb(c, );
- MFixLine(curr, oy, );
- copy_mchar2mline(, >w_mlines[oy], ox);
- LPutChar(>w_layer, , ox, oy);
- LGotoPos(>w_layer, curr->w_x, curr->w_y);
- }
- break;
+ c_prev = omc.image | (omc.font << 8) | omc.fontx << 16;
+ if (!grapheme_cluster_break(c_prev, c))
+{
+ if (ox >= 0)
+ {
+ utf8_handle_comb(c, );
+ MFixLine(curr, oy, );
+ copy_mchar2mline(, >w_mlines[oy], ox);
+ if (!utf8_isdouble(c_prev) &&
+ utf8_isdouble(omc.image | (omc.font << 8) | omc.fontx << 16))
+{
+ /* A combining character switched us from single-width to double-width.
+ Do what the w_mbcs = 0xff path would have done below if the character
+ was double-width to begin with. */
+ omc.mbcs = 0xff;
+ if (ox < cols - 1)
+{
+ curr->w_mlines[oy].image[ox + 1] = 0xff;
+ curr->w_mlines[oy].font[ox + 1] = 0xff;
+ curr->w_mlines[oy].fontx[ox + 1] = 0;
+ curr->w_x++;
+}
+ }
+ LPutChar(>w_layer, , ox, oy);
+ LGotoPos(>w_layer, curr->w_x, curr->w_y);
+ }
+ break;
+}
+ /* fall through */
}
# ifdef DW_CHARS
if (curr->w_encoding == UTF8 && utf8_isdouble(c))
diff -ur /home/sesse/nmu/orig/screen-4.9.0/encoding.c ../encoding.c
--- /home/sesse/nmu/orig/screen-4.9.0/encoding.c 2023-06-27 22:53:44.0 +0200
+++ ../encoding.c 2023-06-27 22:57:29.386767268 +0200
@@ -1215,6 +1233,398 @@
return bisearch(c, combining, sizeof(combining) / sizeof(struct interval) - 1);
}
+static bool
+is_grapheme_extend(c)
+int c;
+{
+ /* https://unicode.org/Public/15.0.0/ucd/DerivedCoreProperties.txt */
+ static const struct interval grapheme_extend[] = {
+{ 0x0300, 0x036F }, { 0x0483, 0x0487 }, { 0x0488, 0x0489 },
+{ 0x0591, 0x05BD }, { 0x05BF, 0x05BF }, { 0x05C1, 0x05C2 },
+{ 0x05C4, 0x05C5 }, { 0x05C7, 0x05C7 }, { 0x0610, 0x061A },
+{ 0x064B, 0x065F }, { 0x0670, 0x0670 }, { 0x06D6, 0x06DC },
+{ 0x06DF, 0x06E4 }, { 0x06E7, 0x06E8 }, { 0x06EA, 0x06ED },
+{ 0x0711, 0x0711 }, { 0x0730, 0x074A }, { 0x07A6, 0x07B0 },
+{ 0x07EB, 0x07F3 }, { 0x07FD, 0x07FD }, { 0x0816, 0x0819 },
+{ 0x081B, 0x0823 }, { 0x0825, 0x0827 }, { 0x0829, 0x082D },
+{ 0x0859, 0x085B }, { 0x0898, 0x089F }, { 0x08CA, 0x08E1 },
+{ 0x08E3, 0x0902 }, { 0x093A, 0x093A }, { 0x093C, 0x093C },
+{ 0x0941, 0x0948 }, { 0x094D, 0x094D }, { 0x0951, 0x0957 },
+{ 0x0962, 0x0963 }, { 0x0981, 0x0981 }, { 0x09BC, 0x09BC },
+{ 0x09BE, 0x09BE }, { 0x09C1, 0x09C4 }, { 0x09CD, 0x09CD },
+{ 0x09D7, 0x09D7 }, { 0x09E2, 0x09E3 }, { 0x09FE, 0x09FE },
+{ 0x0A01, 0x0A02 }, { 0x0A3C, 0x0A3C }, { 0x0A41, 0x0A42 },
+{ 0x0A47, 0x0A48 }, { 0x0A4B, 0x0A4D }, { 0x0A51, 0x0A51 },
+{ 0x0A70, 0x0A71 }, { 0x0A75, 0x0A75 }, { 0x0A81, 0x0A82 },
+{ 0x0ABC, 0x0ABC }, { 0x0AC1, 0x0AC5 }, { 0x0AC7, 0x0AC8 },
+{ 0x0ACD, 0x0ACD }, { 0x0AE2, 0x0AE3 }, { 0x0AFA, 0x0AFF },
+{ 0x0B01, 0x0B01 }, { 0x0B3C, 0x0B3C }, { 0x0B3E, 0x0B3E },
+{ 0x0B3F, 0x0B3F }, { 0x0B41, 0x0B44 }, { 0x0B4D, 0x0B4D },
+{ 0x0B55, 0x0B56 }, { 0x0B57,
Bug#1039503: wrongly splits up extended grapheme clusters (like certain emoji)
On Mon, Jun 26, 2023 at 08:04:28PM +0200, Steinar H. Gunderson wrote:
> Is it possible to retrofit these rules? This specific rule would seem to
> hit a lot of modern emoji sequences (the Unicode Consortium seems to prefer
> using such sequences instead of defining new code points where possible);
> I would assume including the full table of grapheme clusters would help
> e.g. Korean text, too, although I cannot read Korean and have no idea
> whether it's actually a problem.
The attached patch seems to get us halfway there; screen now combines all of
them correctly into one cluster. However, it's still split for whatever
reason; only if I redraw (C-a l) the flag shows up, and the text breaking is
wrong, so I assume there's at least something wrong with the width
calculation, and possibly lots of other things I've not thought of. :-)
/* Steinar */
--
Homepage: https://www.sesse.net/
Description: Better detection of grapheme clusters
Author: Steinar H. Gunderson
Bug-Debian: https://bugs.debian.org/1039503
Last-Update: 2023-06-25
---
--- screen-4.9.0.orig/ansi.c
+++ screen-4.9.0/ansi.c
@@ -694,9 +702,9 @@ register int len;
}
curr->w_rend.font = 0;
}
- if (curr->w_encoding == UTF8 && c >= 0x0300 && utf8_iscomb(c))
+ if (curr->w_encoding == UTF8 && c >= 0x0300)
{
- int ox, oy;
+ int ox, oy, c_prev;
struct mchar omc;
ox = curr->w_x - 1;
@@ -718,15 +726,20 @@ register int len;
omc.mbcs = 0xff;
}
}
- if (ox >= 0)
- {
- utf8_handle_comb(c, );
- MFixLine(curr, oy, );
- copy_mchar2mline(, >w_mlines[oy], ox);
- LPutChar(>w_layer, , ox, oy);
- LGotoPos(>w_layer, curr->w_x, curr->w_y);
- }
- break;
+ c_prev = omc.image | (omc.font << 8) | omc.fontx << 16;
+ if (!grapheme_cluster_break(c_prev, c))
+{
+ if (ox >= 0)
+ {
+ utf8_handle_comb(c, );
+ MFixLine(curr, oy, );
+ copy_mchar2mline(, >w_mlines[oy], ox);
+ LPutChar(>w_layer, , ox, oy);
+ LGotoPos(>w_layer, curr->w_x, curr->w_y);
+ }
+ break;
+}
+ /* fall through */
}
# ifdef DW_CHARS
if (curr->w_encoding == UTF8 && utf8_isdouble(c))
--- screen-4.9.0.orig/encoding.c
+++ screen-4.9.0/encoding.c
@@ -1215,6 +1221,398 @@ int c;
return bisearch(c, combining, sizeof(combining) / sizeof(struct interval) - 1);
}
+static bool
+is_grapheme_extend(c)
+int c;
+{
+ /* https://unicode.org/Public/15.0.0/ucd/DerivedCoreProperties.txt */
+ static const struct interval grapheme_extend[] = {
+{ 0x0300, 0x036F }, { 0x0483, 0x0487 }, { 0x0488, 0x0489 },
+{ 0x0591, 0x05BD }, { 0x05BF, 0x05BF }, { 0x05C1, 0x05C2 },
+{ 0x05C4, 0x05C5 }, { 0x05C7, 0x05C7 }, { 0x0610, 0x061A },
+{ 0x064B, 0x065F }, { 0x0670, 0x0670 }, { 0x06D6, 0x06DC },
+{ 0x06DF, 0x06E4 }, { 0x06E7, 0x06E8 }, { 0x06EA, 0x06ED },
+{ 0x0711, 0x0711 }, { 0x0730, 0x074A }, { 0x07A6, 0x07B0 },
+{ 0x07EB, 0x07F3 }, { 0x07FD, 0x07FD }, { 0x0816, 0x0819 },
+{ 0x081B, 0x0823 }, { 0x0825, 0x0827 }, { 0x0829, 0x082D },
+{ 0x0859, 0x085B }, { 0x0898, 0x089F }, { 0x08CA, 0x08E1 },
+{ 0x08E3, 0x0902 }, { 0x093A, 0x093A }, { 0x093C, 0x093C },
+{ 0x0941, 0x0948 }, { 0x094D, 0x094D }, { 0x0951, 0x0957 },
+{ 0x0962, 0x0963 }, { 0x0981, 0x0981 }, { 0x09BC, 0x09BC },
+{ 0x09BE, 0x09BE }, { 0x09C1, 0x09C4 }, { 0x09CD, 0x09CD },
+{ 0x09D7, 0x09D7 }, { 0x09E2, 0x09E3 }, { 0x09FE, 0x09FE },
+{ 0x0A01, 0x0A02 }, { 0x0A3C, 0x0A3C }, { 0x0A41, 0x0A42 },
+{ 0x0A47, 0x0A48 }, { 0x0A4B, 0x0A4D }, { 0x0A51, 0x0A51 },
+{ 0x0A70, 0x0A71 }, { 0x0A75, 0x0A75 }, { 0x0A81, 0x0A82 },
+{ 0x0ABC, 0x0ABC }, { 0x0AC1, 0x0AC5 }, { 0x0AC7, 0x0AC8 },
+{ 0x0ACD, 0x0ACD }, { 0x0AE2, 0x0AE3 }, { 0x0AFA, 0x0AFF },
+{ 0x0B01, 0x0B01 }, { 0x0B3C, 0x0B3C }, { 0x0B3E, 0x0B3E },
+{ 0x0B3F, 0x0B3F }, { 0x0B41, 0x0B44 }, { 0x0B4D, 0x0B4D },
+{ 0x0B55, 0x0B56 }, { 0x0B57, 0x0B57 }, { 0x0B62, 0x0B63 },
+{ 0x0B82, 0x0B82 }, { 0x0BBE, 0x0BBE }, { 0x0BC0, 0x0BC0 },
+{ 0x0BCD, 0x0BCD }, { 0x0BD7, 0x0BD7 }, { 0x0C00, 0x0C00 },
+{ 0x0C04, 0x0C04 }, { 0x0C3C, 0x0C3C }, { 0x0C3E, 0x0C40 },
+{ 0x0C46, 0x0C48 }, { 0x0C4A, 0x0C4D }, { 0x0C55, 0x0C56 },
+{ 0x0C62, 0x0C63 }, { 0x0C81, 0x0C81 }, { 0x0CBC, 0x0CBC },
+{ 0x0CBF, 0x0CBF }, { 0x0CC2, 0x0CC2 }, { 0x0CC6, 0x0CC6 },
+{ 0x0CCC, 0x0CCD }, { 0x0CD5, 0x0CD6 }, { 0x0CE2, 0x0CE3 },
+{ 0x0D00, 0x0D01 }, { 0x0D3B, 0x0D3C }, { 0x0D3E, 0x0D3E },
+{ 0x0D41, 0x0D44 }, { 0x0D4D, 0x0D4D }, { 0x0D57, 0x0D57 },
+{ 0x0D62, 0x0D63 }, { 0x0D81, 0x0D81 }, { 0x0DCA, 0x0DCA },
+{ 0x0DCF, 0x0DCF }, { 0x0DD2, 0x0DD4 }, { 0x0DD6, 0x0DD6 },
+{ 0x0DDF, 0x0DDF }, {
Bug#1039503: wrongly splits up extended grapheme clusters (like certain emoji)
Package: screen
Version: 4.9.0-4
Severity: normal
Tags: upstream
Hi,
I was trying to figure out why irssi sometimes garbles the display when certain
emoji are involved in the channel topic; after some debugging, it seems the
issue
is with screen, not irssi. To reproduce, start up screen and do (in a shell)
echo '\0360\0237\0217\0263\0357\0270\0217\0342\0200\0215\0360\0237\0214\0210'
(That is UTF-8 for U+1F3F3 U+FE0F U+200D U+1F308.)
Outside of screen, I correctly get ️, , RAINBOW FLAG. However, inside
screen, I get ️, space, (WHITE FLAG, space, RAINBOW). Curiously enough,
if I switch windows and get an irssi redraw, it actually gets drawn correctly,
but that's probably something more complex.
I've tried following the code, and this is my understanding of what's going on:
screen really wants one cell = one codepoint, yet it needs to support combining
characters. So internally, it seems to keep a sort of cache of combining
sequences,
using the otherwise-reserved-for-surrogates range U+D800..U+DFFF.
This seems to happen in two steps; utf8_handle_comb() (in encoding.c) adds more
code points to a given sequence, allocating points and keeping a linked list.
(There's some confusion in that the “font” member of struct mchar is used to
hold the upper 8 bits of the resulting surrogate, but I think that's just some
sort of hack because the “image” member is 8-bit only? And there's something
about double-wide characters that I don't fully understand.) Then, at display
time, ToUtf8_comb() follows this linked list back to output the entire sequence.
The screen debug log appears to go through this (I've kept only what I think
are relevant messages):
read UNICODE 1f3f3
read UNICODE fe0f
combinig char 1f3f3 fe0f -> d800
bring to front: 0
GotoPos (1,1) -> (0,1)
---LGotoPos 1 1
read UNICODE 200d
combinig char d800 200d -> d801
bring to front: 1
bring to front: 0
GotoPos (1,1) -> (0,1)
---LGotoPos 1 1
read UNICODE 1f308
---LGotoPos 0 1
Seemingly, it understands that the two first codepoints are to be combined,
allocates U+D800 for that, and then continues reading. Then it reads
the third one, combines it with U+D800 to create U+D801, but then mistakenly
does _not_ combine U+1F308 with it. This is why we end up with two different
things on screen.
It really seems to me that screen simply doesn't understand Unicode extended
grapheme clusters, only the legacy “legacy grapheme clusters”; from UAX#29
(https://unicode.org/reports/tr29/):
A legacy grapheme cluster is defined as a base (such as A or カ) followed by
zero or more continuing characters. One way to think of this is as a sequence
of characters that form a “stack”.
This seems to come from ansi.c line 705 (WriteString()):
if (curr->w_encoding == UTF8 && c >= 0x0300 && utf8_iscomb(c))
{
[…]
utf8_handle_comb(c, );
In other words, it thinks certain code points are inherently combining
and thus are to be attached to the previous character. However, that's
changed since Unicode 5.1.0, circa 2008; now all of these four together
should form an extended grapheme cluster.
The rules for extended grapheme clusters are locale-dependent, but I'd
guess screen would do just fine with the default grapheme clusters
(certainly much better than today). The rules are actually pretty simple,
if a tad verbose:
https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundary_Rules
In particular, I would believe what we need here is rule GB11,
“Do not break within emoji modifier sequences or emoji zwj sequences.”:
\p{Extended_Pictographic} Extend* ZWJ × \p{Extended_Pictographic}
where “×” means “don't break here”, i.e., continue to run
utf8_handle_comb(). Specifically, the rule matches because:
U+1F3F3 WAVING WHITE FLAG is indeed Extended_Pictographic
(according to
https://unicode.org/Public/15.0.0/ucd/emoji/emoji-data.txt)
U+FE0F VARIATION SELECTOR-16 is Extend, because it is Grapheme_Extend
(according to
https://unicode.org/Public/15.0.0/ucd/DerivedCoreProperties.txt)
U+200D ZERO-WIDTH JOINER is, well, ZWJ
U+1F308 RAINBOW is also Extended_Pictographic (same file)
Is it possible to retrofit these rules? This specific rule would seem to
hit a lot of modern emoji sequences (the Unicode Consortium seems to prefer
using such sequences instead of defining new code points where possible);
I would assume including the full table of grapheme clusters would help
e.g. Korean text, too, although I cannot read Korean and have no idea
whether it's actually a problem.
As a hack, it seems that anything that follows a ZWJ would be very likely
to keep being part of the same grapheme cluster, but I haven't tested this
out in practice.
FWIW, tmux seems to have no problems showing the flag, although I haven't
checked its implementation.
-- Package-specific info:
File Existence and Permissions
--
drwxr-xr-x 34 root root 1140 Jun 26 17:15 /run
lrwxrwxrwx 1 root root 4
Bug#1036960: plocate: coredump on any search
On Tue, May 30, 2023 at 05:09:07PM -0400, Nick Black (Public gmail account)
wrote:
> I've been using plocate for many months on all my machines without problems.
> Recently, I get a coredump on any search, on all the machines on which I've
> tested. I've got a stack trace that points at do_search_file():
>
> Thread 1 (Thread 0x7f426c687740 (LWP 914015) "locate"):
> #0 __pthread_kill_implementation (threadid=,
> signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
> #1 0x7f426c2a9d2f in __pthread_kill_internal (signo=6,
> threadid= out>) at ./nptl/pthread_kill.c:78
> #2 0x7f426c25aef2 in __GI_raise (sig=sig@entry=6) at
> ../sysdeps/posix/raise.c:26
> #3 0x7f426c245472 in __GI_abort () at ./stdlib/abort.c:79
> #4 0x7f426c29e2d0 in __libc_message (action=action@entry=do_abort,
> fmt=fmt@entry=0x7f426c3b8459 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
> #5 0x7f426c2b364a in malloc_printerr (str=str@entry=0x7f426c3b60b1
> "free(): invalid pointer") at ./malloc/malloc.c:5660
> #6 0x7f426c2b53d4 in _int_free (av=, p=,
> have_lock=have_lock@entry=0) at ./malloc/malloc.c:4435
> #7 0x7f426c2b7d2f in __GI___libc_free (mem=) at
> ./malloc/malloc.c:3385
> #8 0x5617abf86e1a in do_search_file (needles=std::vector of length 1,
> capacity 1 = {...}, filename="plocate.db") at ../plocate.cpp:491
> #9 0x5617abf804b8 in main (argc=, argv=) at
> ../plocate.cpp:995
The stack trace seems somewhat misleading, but I wonder perhaps if this is
https://git.sesse.net/?p=plocate;a=commitdiff;h=7f39444852308224240ebfb75e0bf8f39403afa0
Could you check if downgrading liburing helps? (If so, this shouldn't affect
bookworm as far as I understand)
/* Steinar */
--
Homepage: https://www.sesse.net/
Bug#1034527: unblock: nageru/2.2.1-1
Package: release.debian.org
Severity: normal
User: release.debian@packages.debian.org
Usertags: unblock
X-Debbugs-Cc: nag...@packages.debian.org
Control: affects -1 + src:nageru
Please unblock package nageru
Please consider unblocking Nageru 2.2.1-1. It contains a number
of focused fixes, mostly for crash bugs related to video stream
inputs (e.g. from video files, or from the network). As upstream,
I made 2.2.1 specifically as a set of targeted fixes over 2.2.0
to make this feature less broken for bookworm.
Nageru is a leaf package, so there should be fairly low risk.
I've tested both myself and had user reports verifying the fixes
(in upstream git).
There are no Debian package changes (beyond the changelog),
only upstream changes. debdiff is attached.
[ Checklist ]
[x] all changes are documented in the d/changelog
[x] I reviewed all changes and I approve them
[x] attach debdiff against the package in testing
unblock nageru/2.2.1-1
diff -Nru nageru-2.2.0/debian/changelog nageru-2.2.1/debian/changelog
--- nageru-2.2.0/debian/changelog 2022-11-23 09:26:33.0 +0100
+++ nageru-2.2.1/debian/changelog 2023-04-17 18:37:27.0 +0200
@@ -1,3 +1,10 @@
+nageru (2.2.1-1) unstable; urgency=medium
+
+ * New upstream release.
+* Fixes several crash bugs related to video inputs. (Closes: #1034471)
+
+ -- Steinar H. Gunderson Mon, 17 Apr 2023 18:37:27 +0200
+
nageru (2.2.0-2) unstable; urgency=medium
* Remove ppc64el from futatabi's architecture list.
diff -Nru nageru-2.2.0/meson.build nageru-2.2.1/meson.build
--- nageru-2.2.0/meson.build2022-11-15 00:25:44.0 +0100
+++ nageru-2.2.1/meson.build2023-04-17 18:35:47.0 +0200
@@ -1,4 +1,4 @@
-project('nageru', 'cpp', default_options: ['buildtype=debugoptimized'],
version: '2.2.0')
+project('nageru', 'cpp', default_options: ['buildtype=debugoptimized'],
version: '2.2.1')
cxx = meson.get_compiler('cpp')
qt5 = import('qt5')
diff -Nru nageru-2.2.0/nageru/cef_capture.cpp
nageru-2.2.1/nageru/cef_capture.cpp
--- nageru-2.2.0/nageru/cef_capture.cpp 2022-11-15 00:25:44.0 +0100
+++ nageru-2.2.1/nageru/cef_capture.cpp 2023-04-17 18:35:47.0 +0200
@@ -165,8 +165,6 @@
});
}
-#define FRAME_SIZE (8 << 20) // 8 MB.
-
void CEFCapture::configure_card()
{
if (video_frame_allocator == nullptr) {
diff -Nru nageru-2.2.0/nageru/decklink_capture.cpp
nageru-2.2.1/nageru/decklink_capture.cpp
--- nageru-2.2.0/nageru/decklink_capture.cpp2022-11-15 00:25:44.0
+0100
+++ nageru-2.2.1/nageru/decklink_capture.cpp2023-04-17 18:35:47.0
+0200
@@ -24,8 +24,6 @@
#include "shared/memcpy_interleaved.h"
#include "v210_converter.h"
-#define FRAME_SIZE (8 << 20) // 8 MB.
-
using namespace std;
using namespace std::chrono;
using namespace std::placeholders;
diff -Nru nageru-2.2.0/nageru/defs.h nageru-2.2.1/nageru/defs.h
--- nageru-2.2.0/nageru/defs.h 2022-11-15 00:25:44.0 +0100
+++ nageru-2.2.1/nageru/defs.h 2023-04-17 18:35:47.0 +0200
@@ -3,11 +3,12 @@
#include
-#define MAX_FPS 60
+#define TYPICAL_FPS 60
#define FAKE_FPS 25 // Must be an integer.
// #define MAX_VIDEO_CARDS 16 // defined in shared_defs.h.
#define MAX_ALSA_CARDS 16
#define MAX_BUSES 256 // Audio buses.
+#define FRAME_SIZE (8 << 20) // 8 MB. (FIXME: Not enough for a 2160p frame!)
// For deinterlacing. See also comments on InputState.
#define FRAME_HISTORY_LENGTH 5
diff -Nru nageru-2.2.0/nageru/ffmpeg_capture.cpp
nageru-2.2.1/nageru/ffmpeg_capture.cpp
--- nageru-2.2.0/nageru/ffmpeg_capture.cpp 2022-11-15 00:25:44.0
+0100
+++ nageru-2.2.1/nageru/ffmpeg_capture.cpp 2023-04-17 18:35:47.0
+0200
@@ -26,6 +26,7 @@
#include
#include
#include
+#include
#include
#include
@@ -43,8 +44,6 @@
#include
#endif
-#define FRAME_SIZE (8 << 20) // 8 MB.
-
using namespace std;
using namespace std::chrono;
using namespace bmusb;
@@ -454,24 +453,63 @@
}
}
-AVPixelFormat get_vaapi_hw_format(AVCodecContext *ctx, const AVPixelFormat
*fmt)
+template
+AVPixelFormat get_hw_format(AVCodecContext *ctx, const AVPixelFormat *fmt)
{
- for (const AVPixelFormat *fmt_ptr = fmt; *fmt_ptr != -1; ++fmt_ptr) {
- for (int i = 0;; ++i) { // Termination condition inside loop.
- const AVCodecHWConfig *config =
avcodec_get_hw_config(ctx->codec, i);
- if (config == nullptr) { // End of list.
- fprintf(stderr, "Decoder %s does not support
device.\n", ctx->codec->name);
- break;
- }
- if (config->methods &
AV_CODEC_HW_CONFIG_METHOD_HW_DEVICE_CTX &&
- config->device_type == AV_HWDEVICE_TYPE_VAAPI &&
- config->pix_fmt == *fmt_ptr) {
+
Bug#1034471: crashes when sending 60 fps video over SRT
Package: nageru Version: 2.2.0-2 Severity: important Tags: upstream How to reproduce: 1. Start Nageru. 2. Connect Larix Broadcaster (or a similar app) to Nageru over SRT. 3. Nageru crashes with a message that dts > pts. The underlying problem is that Larix defaults to 60 fps (on phones that support it), and its timestamps are a little bit off, so that it de-facto ends up sending 60.06 Hz. This violates an underlying assumption in Naguru's QSV encoder (MAX_FPS is set to 60, and DTS generation will go awry if this constraint is violated). Upstream git, which is going to be 2.2.1, has a fix for this, by changing the DTS algorithm so that it no longer needs MAX_FPS at all. (The new algorithm is also much simpler.) -- System Information: Debian Release: 12.0 APT prefers testing-security APT policy: (500, 'testing-security'), (500, 'testing-debug'), (500, 'stable-security'), (500, 'proposed-updates'), (500, 'oldoldstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 6.2.9 (SMP w/56 CPU threads; PREEMPT) Locale: LANG=en_DK.UTF-8, LC_CTYPE=en_DK.UTF-8 (charmap=UTF-8), LANGUAGE=en_NO:en_US:en_GB:en Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) Versions of packages nageru depends on: ii libasound2 1.2.8-1+b1 pn libavcodec57 | libavcodec-extra57 ii libavcodec587:4.3.5-0+deb11u1 ii libavcodec597:5.1.2-3 pn libavformat57 pn libavformat58 ii libavformat59 7:5.1.2-3 pn libavresample3 pn libavutil55 ii libavutil56 7:4.3.5-0+deb11u1 ii libavutil57 7:5.1.2-3 pn libbmusb2 pn libbmusb6 ii libc6 2.36-8 ii libepoxy0 1.5.10-1 ii libgcc-s1 [libgcc1] 12.2.0-14 ii libgcc1 1:8.3.0-6 ii libjpeg62-turbo 1:2.1.5-2 ii liblua5.2-0 5.2.4-3 pn libluajit-5.1-2 pn libluajit-5.1-2 | libluajit2-5.1-2 pn libmicrohttpd12 pn libmovit5 pn libmovit8 ii libpci3 1:3.9.0-4 pn libprotobuf10 pn libprotobuf23 ii libprotobuf32 3.21.12-3 pn libqcustomplot2.0 ii libqt5core5a5.15.8+dfsg-3 ii libqt5gui5 5.15.8+dfsg-3 ii libqt5opengl5 5.15.8+dfsg-3 ii libqt5widgets5 5.15.8+dfsg-3 pn libsrt1.4-gnutls ii libsrt1.5-gnutls1.5.1-1 ii libstdc++6 12.2.0-14 ii libswresample3 7:4.3.5-0+deb11u1 ii libswresample4 7:5.1.2-3 pn libswscale4 pn libswscale5 ii libswscale6 7:5.1.2-3 ii libusb-1.0-02:1.0.26-1 pn libva-drm1 ii libva-drm2 2.17.0-1 pn libva-x11-1 ii libva-x11-2 2.17.0-1 pn libva1 ii libva2 2.17.0-1 ii libx11-62:1.8.4-2 pn libx264-148 ii libx264-160 2:0.160.3011+gitcde9a93-2.1 ii libx264-164 2:0.164.3095+gitbaee400-2+b1 pn libzita-resampler1 nageru recommends no packages. nageru suggests no packages.
Bug#1029571: nageru: please build with sha1 to let lld link correctly
On Tue, Jan 24, 2023 at 06:25:57PM +0100, Gianfranco Costamagna wrote: > Hello, with newer toolchains the default changed to sha256 IIRC, and lld > fails to link objects Shouldn't this be fixed in the Debian lld package, if you can't build Debian packages with lld? It seems a bit weird to have every package that could be using lld deal with this. Or better yet, in debugedit :-) > BTW according to your build log, you are using bfd linker, not lld, this is > probably why its still working in Debian That's pretty odd; I'll need to see what's going on at some point. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1027702: plocate: "Try `ulimit -n 262144' or similar (current limit is 0)."
tags 1027702 + pending thanks On Mon, Jan 02, 2023 at 10:13:59AM +0100, Jakub Wilk wrote: > I saw this in an error message sent by cron: > > Hint: Try `ulimit -n 262144' or similar (current limit is 0). > > The number in parentheses in wrong of course. Fixed in upstream git, so this will be fixed whenever 1.1.18 is released. Thanks! /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1027702: plocate: "Try `ulimit -n 262144' or similar (current limit is 0)."
On Mon, Jan 02, 2023 at 10:13:59AM +0100, Jakub Wilk wrote:
> I saw this in an error message sent by cron:
>
> Hint: Try `ulimit -n 262144' or similar (current limit is 0).
>
> The number in parentheses in wrong of course.
>
> This is likely related to these compiler warnings:
>
> ../updatedb.cpp:710:81: warning: format ‘%lu’ expects argument of type
> ‘long unsigned int’, but argument 3 has type ‘rlim_t’ {aka ‘long long
> unsigned int’} [-Wformat=]
> ../updatedb.cpp:710:115: warning: format ‘%lu’ expects argument of type
> ‘long unsigned int’, but argument 4 has type ‘rlim_t’ {aka ‘long long
> unsigned int’} [-Wformat=]
Do you know if there are any compiler macros for rlim_t, like there is for
e.g. PRId64? Or should I simply cast to uint64_t and use an existing macro?
/* Steinar */
--
Homepage: https://www.sesse.net/
Bug#1027322: libapreq2: diff for NMU version 2.17-0.1
On Fri, Dec 30, 2022 at 12:33:34PM +0100, Tobias Frost wrote: > (as announced in #1018191) I've tried to packasge 2.17 to address > the CVE. I've rebuilt all reverse depencies successfully (except lua-apr, > which was alredy broken before, #935271) > > I've prepared an NMU for libapreq2 (versioned as 2.17-0.1) and > uploaded it to DELAYED/15. Please feel free to tell me if I > should delay it longer or if I should accelerate it. Just a few comments: - Please set maintainer to Debian QA Group, orphaning the package. - I don't think it's a good idea to modify old changelog entries. - Apart from that, there's no need to delay the NMU. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1018191: libapreq2: CVE-2022-22728: multipart form parse memory corruption
On Fri, Dec 30, 2022 at 12:04:29PM +0100, Tobias Frost wrote: > (I'm currently take a look at 2.17, to see if I can get it packages, if I'm > succeeding, > there will be an NMU announcement :)) If you are NMUing, could you orphan the package in the upload? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1018191: libapreq2: CVE-2022-22728: multipart form parse memory corruption
On Fri, Dec 30, 2022 at 11:04:46AM +0100, Tobias Frost wrote:
> I was trying to triage this CVE and *maybe* those revisions are related:
>
> r1894937 ("apreq_parse_headers: Discard CRLF of folded values.")
> r1894940 ("reindent (no functional change).")
> r1894977 ("Follow up to r1894937: Fix setting of empty value.")
> r1895054 ("Follow up to r1894937: Always eat CRLF at the end of header
> value.")
Perhaps it's best to remove libapreq2 entirely? I don't use nor maintain it
anymore, it's been out of testing for a while, and there's this CVE.
/* Steinar */
--
Homepage: https://www.sesse.net/
Bug#1025099: plocate: autofs pruning doesn't seem to work
On Tue, Nov 29, 2022 at 03:04:01PM -0800, Ross Vandegrift wrote: > Interesting - stat -f does the same, and it always returns nfs. Here's > a lame idea: /proc/mounts knows it's actually autofs... Most of this logic is inherited from mlocate's updatedb, though not all. It may be fixable or it may not; I'd really need to check. But it really sounds like one should be able to stat() something without bad things happening :-) I guess as a workaround, you can ask systemd to sandbox away the mount for updatedb? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1025099: plocate: autofs pruning doesn't seem to work
On Tue, Nov 29, 2022 at 11:31:39AM -0800, Ross Vandegrift wrote: >> Subject: Bug#1025099: plocate: autofs pruning doesn't seem to work > That subject wasn't too clear- updatedb.plocate does not index the > remote filesystem, it just triggeres the automount unnecessarily. I don't think that's possible to avoid; the directory must be stat()-ed to get to know what type of filesystem it is, and I guess that would trigger the mount? I guess it's an interesting question why it even knows how to enter that directory, if it's not mounted; are you running with ghost on? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1024725: RM: futatabi [missing libluajit-5.1-dev] -- ANAIS; ppc64el
Package: ftp.debian.org Severity: normal Hi, futatabi (nageru source package) no longer has ppc64el in its architecture list, since libluajit-5.1-dev does not exist there. This was already fixed for nageru itself, but futatabi has stuck around due to an error. Please rm the binary package so that nageru can transition to testing (fixes an RC bug).
Bug#1020529: 1:9.16.33-1~deb11u1 security upgrade breaks DNSSEC setups
On Fri, Sep 23, 2022 at 12:09:45AM +0200, Ondřej Surý wrote: > Nope, the plan to follow upstream releases was acked by both the security > and release teams, so I am not doing anything really surprising here. Well, it is really surprising to users :-) Other packages that have been doing the same thing (e.g. firefox and chromium) have been explicit about fixing-by-wholesale-upgrading in the advisory in the past. > So, let’s focus on what could be improved here. I don’t think anything like > this will happen ever again in 9.16, but I’ll try to be more vigilant about > possibly breaking changes next time. > > Also Steinar, while I understand this has caused you some distress, this is > not really a grave bug. It does not make: the package in question unusable > or mostly so, or causes data loss, or introduces a security hole allowing > access to the accounts of users who use the package. I'm fine with downgrading the severity (it's mostly semantics anyway); but do note that security updates are treated differently from nearly every other update. In particular, unattended-upgrades will install them by default, and administrators are urged to install them as quickly as possible on their production servers without testing in a staging environment first. This makes them much more sensitive than a normal stable update, which goes through proposed-stable-updates first. In our case, it happened to break the entire system; it uses 127.0.0.1 in resolv.conf, and login uses Kerberos, which is dependent on DNS... Note that if we add “inline-signing yes;” on all of the zones with dnssec-policy on, BIND crashes with an assertion failure on reloading the config: Sep 22 22:32:52 cirkus named[3064511]: rbtdb.c:6837: REQUIRE(((rbtnode->nsec == DNS_RBT_NSEC_NSEC3 && (rdataset->type == ((dns_rdatatype_t)dns_rdatatype_nsec3) || rdataset->covers == ((dns_rdatatype_t)dns_rdatatype_nsec3))) || (rbtnode->nsec != DNS_RBT_NSEC_NSEC3 && rdataset->type != ((dns_rdatatype_t)dns_rdatatype_nsec3) && rdataset->covers != ((dns_rdatatype_t)dns_rdatatype_nsec3 failed, back trace Sep 22 22:32:52 cirkus named[3064511]: #0 0x55dd86375f1d in ?? Sep 22 22:32:52 cirkus named[3064511]: #1 0x7fddb899090a in ?? Sep 22 22:32:52 cirkus named[3064511]: #2 0x7fddb8ac981c in ?? Sep 22 22:32:52 cirkus named[3064511]: #3 0x7fddb8b9a461 in ?? Sep 22 22:32:52 cirkus named[3064511]: #4 0x7fddb89c36bd in ?? Sep 22 22:32:52 cirkus named[3064511]: #5 0x7fddb89ac6a9 in ?? Sep 22 22:32:52 cirkus named[3064511]: #6 0x7fddb89ac825 in ?? Sep 22 22:32:52 cirkus named[3064511]: #7 0x7fddb89ad003 in ?? Sep 22 22:32:52 cirkus named[3064511]: #8 0x7fddb876ee7d in ?? Sep 22 22:32:52 cirkus named[3064511]: #9 0x7fddb8780a23 in ?? Sep 22 22:32:52 cirkus named[3064511]: #10 0x7fddb876f714 in ?? Sep 22 22:32:52 cirkus named[3064511]: #11 0x7fddb89ac8d7 in ?? Sep 22 22:32:52 cirkus named[3064511]: #12 0x7fddb89c5ec6 in ?? Sep 22 22:32:52 cirkus named[3064511]: #13 0x7fddb8745ea7 in ?? Sep 22 22:32:52 cirkus named[3064511]: #14 0x7fddb8327aef in ?? Sep 22 22:32:52 cirkus named[3064511]: exiting (due to assertion failure) This is the old version, though, so it might be better with the new one. > As I said in the other email, I’ll take a look if I can soften the > requirement for the configuration in the Debian package tomorrow. Perhaps > just downgrade it to a prominent warning instead of hard error. That would be most appreciated, yes. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1020529: 1:9.16.33-1~deb11u1 security upgrade breaks DNSSEC setups
On Thu, Sep 22, 2022 at 08:13:53PM +0200, Ondřej Surý wrote: > I am sorry this has caused inconvenience for you, but the original problem > here was that the implicit inline-signing with the dnssec-policy was also > problematic and causing other problems, see the upstream issue: > https://gitlab.isc.org/isc-projects/bind9/-/issues/3381 > > Especially this: > https://gitlab.isc.org/isc-projects/bind9/-/issues/3381#note_308893 Sure, but that's not appropriate to change in a stable-security update. You can ask the SRMs whether it is applicable for a stable update (I would say no; the point of stable is that you know what issues you have and don't get new ones), but you cannot just put it into a security update unless they are positively required for the security issues in question. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1020529: 1:9.16.33-1~deb11u1 security upgrade breaks DNSSEC setups
Package: bind9 Version: 1:9.16.33-1~deb11u1 Severity: grave Hi, After applying the security updates for DSA 5235-1, named completely breaks and refuses to start. (This caused downtime in production for us.) The reason seems to be that the patch includes a full minor version bump, including policy changes such as: 5941. [func] Zones with dnssec-policy now require dynamic DNS or inline-siging to be configured explicitly. [GL #3381] Since we have DNSSEC zones (as is recommended!), and use dnssec-policy to configure them (also recommended!) it dies on startup with Sep 22 16:17:59 cirkus named[3045282]: /etc/bind/named.conf.local:388: 'dnssec-policy;' requires dynamic DNS or inline-signing to be configured for the zone Sep 22 16:17:59 cirkus named[3045282]: loading configuration: failure Sep 22 16:17:59 cirkus named[3045282]: exiting (due to fatal error) It seems that one can change add “inline-signing yes;” manually to each zone to work around it, but this is not the kind of change that really should be done in a security update without a very good reason _and_ a very clear warning. :-) It seems the test suite even had to be changed for this change, which should have been a pretty clear red flag: https://salsa.debian.org/dns-team/bind9/-/commit/9036610e13ed037f776460d7806ea0a0e3841bdf#6f59e8ac2674d0c1120aa79f6f2ac2aa946d99e5 I haven't checked if there are other breaking changes. -- System Information: Debian Release: 11.5 APT prefers stable-security APT policy: (500, 'stable-security'), (500, 'stable-debug'), (500, 'proposed-updates'), (500, 'oldoldstable'), (500, 'stable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 5.19.10 (SMP w/56 CPU threads; PREEMPT) Locale: LANG=en_DK.UTF-8, LC_CTYPE=en_DK.UTF-8 (charmap=UTF-8), LANGUAGE=en_NO:en_US:en_GB:en Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages bind9 depends on: ii adduser 3.118 ii bind9-libs1:9.16.27-1~deb11u1 pn bind9-utils pn bind9utils ii debconf [debconf-2.0] 1.5.77 ii dns-root-data 2021011101 ii init-system-helpers 1.60 ii iproute2 5.10.0-4 pn libbind9-140 pn libbind9-161 ii libc6 2.31-13+deb11u4 ii libcap2 1:2.44-1 ii libcom-err2 [libcomerr2] 1.46.2-2 pn libdns1104 pn libdns162 ii libfstrm0 0.6.0-1+b1 ii libgeoip1 1.6.12-7 ii libgssapi-krb5-2 1.18.3-6+deb11u2 pn libirs141 pn libisc1100 pn libisc160 pn libisccc140 pn libisccc161 pn libisccfg140 pn libisccfg163 pn libjson-c3 ii libjson-c50.15-2 ii libk5crypto3 1.18.3-6+deb11u2 ii libkrb5-3 1.18.3-6+deb11u2 ii liblmdb0 0.9.24-1 pn liblwres141 pn liblwres161 ii libmaxminddb0 1.5.2-1 ii libnghttp2-14 1.43.0-1 ii libprotobuf-c11.3.3-1+b2 pn libssl1.0.2 ii libssl1.1 1.1.1n-0+deb11u3 ii libuv11.40.0-2 ii libxml2 2.9.10+dfsg-6.7+deb11u2 ii lsb-base 11.1.0 ii net-tools 1.60+git20181103.0eebece-1 ii netbase 6.3 ii zlib1g1:1.2.11.dfsg-2+deb11u2 bind9 recommends no packages. Versions of packages bind9 suggests: pn bind-doc ii bind9-dnsutils [dnsutils] 1:9.16.27-1~deb11u1 pn bind9-doc ii dnsutils 1:9.16.27-1~deb11u1 pn resolvconf pn ufw -- Configuration Files: /etc/bind/named.conf.local changed [not included] /etc/bind/named.conf.options changed [not included]
Bug#1018191: libapreq2: CVE-2022-22728: multipart form parse memory corruption
On Fri, Aug 26, 2022 at 09:07:06PM +0200, Salvatore Bonaccorso wrote: > The following vulnerability was published for libapreq2. > > CVE-2022-22728[0]: > | A flaw in Apache libapreq2 versions 2.16 and earlier could cause a > | buffer overflow while processing multipart form uploads. A remote > | attacker could send a request causing a process crash which could lead > | to a denial of service attack. Based on the description, I assume it is this one: http://svn.apache.org/viewvc?view=revision=1866760 I'm not sure if it counts as “buffer overflow”, but given that it only mentions DoS and not arbitrary code execution, NULL pointer dereference looks a lot like it. 2.13 appears vulnerable to me, given the description. I don't use libapreq2 anymore, so anyone wanting to pick up the package would be more than welcome. Upstream homepage is now seemingly at: https://httpd.apache.org/apreq/ /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1015731: plocate database gaps after upgrade from mlocate
On Wed, Jul 20, 2022 at 12:24:56PM -0700, Ross Boylan wrote: > I had read the man page, but none of my key partitions is mounted with > a bind option, and so the bind mount exclusion did not seem to apply. > > Could you add a warning, like "btrfs may accomplish a subvolume mount > with a bind mount even if you do not explicitly request it." This text exists in the man page: “Note that Btrfs subvolume mounts are handled internally in the kernel as bind mounts (see btrfs-subvolume(8)), and thus, may get skipped if you have also mounted the filesystem root itself. To counteract this, make your root directory a Btrfs subvolume, too.” > Also, it would be nice to have an explanation of the --debug-pruning output. That is for internal usage, so it's not really supported. :-) > Now update/locate seem to be operating as I wanted. And it's > incredibly fast: 28 seconds for my terabytes of disk space. Nice job! locate needs 28 seconds? That's really, really slow unless you have thousands of hits. It should be on the order of milliseconds. updatedb could very well be in that range. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1015731: plocate database gaps after upgrade from mlocate
On Tue, Jul 19, 2022 at 02:37:17PM -0700, Ross Boylan wrote: > I installed plocate; during installation it gave messages suggesting it was > doing a scan, although it seemed awfully quick. > > After I ran locate, and again it missed directories it should have found: > locate -r "/\.hg$" The conversion procedure from mlocate generally trusts that the database is correct. If you have bugs in the mlocate database, they will certainly carry forward through plocate. > 2. Delete /var/lib/plocate/plocate.db. But perhaps at least an empty file > with > the right permissions is required? Then rerun updatedb. Yes, you should just rm /var/lib/plocate/plocate.db. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1015304: new upstream version (0.11) available
Package: svt-av1 Version: 0.9.1+dfsg-1 Severity: wishlist Hi, SVT-AV1 0.11 has been available for a while, and 0.12 is now starting the release process. There are quality and speed improvements (as well as the occasional API extension); would it be possible to package the newer version? -- System Information: Debian Release: 11.4 APT prefers stable-security APT policy: (500, 'stable-security'), (500, 'stable-debug'), (500, 'proposed-updates'), (500, 'oldoldstable'), (500, 'stable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 5.17.5 (SMP w/56 CPU threads; PREEMPT) Locale: LANG=en_DK.UTF-8, LC_CTYPE=en_DK.UTF-8 (charmap=UTF-8), LANGUAGE=en_NO:en_US:en_GB:en Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system)
Bug#1015257: misleading package description
Package: libsvtav1-dev Version: 0.9.1+dfsg-1 Severity: minor Hi, libsvtav1-dev writes: This package provides the development files for libsvtav1dec and libsvtav1enc. However, it does not. It provides _some_ development files (the header files), but not _the_ development files; in fact, it is pretty useless on its own. It took me a long time before I figured out that the package that I wanted was libsvtav1enc-dev. Suggested rewording: This package provides the header files shared between libsvtav1enc-dev and libsvtav1dec-dev. -- System Information: Debian Release: 11.4 APT prefers stable-security APT policy: (500, 'stable-security'), (500, 'stable-debug'), (500, 'proposed-updates'), (500, 'oldoldstable'), (500, 'stable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 5.17.5 (SMP w/56 CPU threads; PREEMPT) Locale: LANG=en_DK.UTF-8, LC_CTYPE=en_DK.UTF-8 (charmap=UTF-8), LANGUAGE=en_NO:en_US:en_GB:en Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system)
Bug#1012619: want /etc/updatedb/updatedb.conf.d/
tags 1012619 + wontfix thanks On Sat, Jun 11, 2022 at 12:49:09PM +0100, Ian Jackson wrote: > I don't think "if you are bothered by this" is really right. It's > clearly a bug, and it can cause the backups to actually fail. If so, that's a bug on chiark-backup. I do not intend do to put resources into this at the moment. I would consider patches, assuming they do not cause significant extra review or long-term maintenance. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1012619: want /etc/updatedb/updatedb.conf.d/
On Sat, Jun 11, 2022 at 02:12:50AM +0100, Ian Jackson wrote: > How do you suggest I fix #1012617 ? I don't know how your system works (and there's no documentation that I can find), so I'm not in a position to say much about it. (If you use filesystem-level snapshots, PRUNE_BIND_MOUNTS would take a fair amount of such structures, but I really don't know what you're doing.) But there's nothing preventing you from simply documenting that if you are bothered by this, you should exclude the relevant directories in updatedb.conf (manually, or through something like Puppet). /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1012619: want /etc/updatedb/updatedb.conf.d/
On Fri, Jun 10, 2022 at 01:00:17PM +0100, Ian Jackson wrote: > The updatedb cron job reads /etc/updatedb.conf. > > I have a package (chiark-backup) which sometimes mounts snapshot volumes in a > predicteable location, and which should not be scanned by updatedb. > I don't want to edit /etc/updatedb.conf in my package's maintscripts. > > Rather, I would like to be able to droo a file into some .d directory, > containing > PRUNEPATHS+=' /var/lib/chiark-backup ' > or some such. Hi Ian, I understand the desire, but what you're asking for is basically making updatedb.conf into a multi-file language (there's no += at the moment, for instance), and I think this is too niche to spend resources and complexity on. I can tag this as wontfix or something, but I'm not sure if it would make much difference? > I'm filing this as "normal" rather than "wishlist" because although > it's a feature request here in mlocate, it's a *bug* in chiark-utils > (#1012617). I assume you mean plocate now, not mlocate, but no big change (they share config file parsers). /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1012556: nageru: no need to (wrongly) hardcode luajit architecture
On Thu, Jun 09, 2022 at 10:26:15AM +0200, Paul Gevers wrote: > We're in the process of add support for a second implemtentation of > luajit [1] that supposedly is properly supported on IBM architectures > (ppc64el and s390x). While I scheduled binNMU's for nageru, I noticed > it doesn't build on all architectures where luajit is available and > reading d/control suggests that the reason why your package has an > explicit architecture list is because of luajit. In general, it's > recommended to just use Architecture: any in such cases, as your > package will not build on architectecture where your build > dependencies aren't available. Yes and no. It also needs: - USB - A little-endian CPU - Working VA-API - Working OpenGL (not GLES) and especially the little-endian CPU demand is hard to express using Architecture: any. :-) You also tend to get FTBFS bugs filed pretty blindly by porters, for a software package that doesn't really make sense on such architectures. E.g. there's absolutely no way anyone would run Nageru on s390x, ever. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1011948: plocate: updatedb error with CIFS
On Fri, May 27, 2022 at 04:35:27PM +0200, rollopack wrote: > close(50) = 0 > newfstatat(51, "", 0x7ffd0628d900, AT_EMPTY_PATH) = -1 EACCES (Permesso > negato) So basically, there is a directory that we can open, but not stat once open? That's a bit weird, but I guess the server can deny pretty much anything it would like. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1004511: luajit SEGFAULTS on ppc64el
On Thu, May 19, 2022 at 08:57:35PM +0200, Paul Gevers wrote: > reassign -6 src:nageru 2.1.0-1 Hi, If you're doing this kind of filing, perhaps also send an email to maintainers? All I got was a message that Nageru was marked for autoremoval for testing (over a bug I had never heard of before), and I had to start figuring out why myself. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1010485: plocate PRUNE_BIND_MOUNTS incompatible with btrfs subvolumes
On Tue, May 03, 2022 at 07:59:48PM +0200, Julian Andres Klode wrote: > I fail to see how naming it @root instead of @, or @screwedup for that > matter makes a difference. Try it. :-) > This is a significant regression for users coming from mlocate. They had > working locate No, did they not. The mlocate updatedb code had broken bind mount detection after the switch from /etc/mtab to /proc/mountinfo, which specifically broke btrfs subvolumes: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=746943 > If you prefer to keep plocate broken with btrfs in Debian, Please stop misrepresenting the issue. plocate isn't broken with btrfs, but you can configure btrfs subvolumes in a way such that plocate (and basically any other tool that tries to skip bind mounts) will not work in the default configuration. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1010485: plocate PRUNE_BIND_MOUNTS incompatible with btrfs subvolumes
On Mon, May 02, 2022 at 06:47:46PM +0200, Julian Andres Klode wrote: > It says to make / as subvolume as well, which seems incomplete, as > mounted at / is the subvolume @ (that's the standard Ubuntu layout); That would be to make it at @root or similar. > updatedb needs to parse /proc/mounts, look for subvol= flag, and > for each subvol= traverse the shortest path (or first path it sees). There's no way you can meaningfully say that /a is shorter than /b. And the order Linux gives you a directory entry back is pretty much random, so “first it sees“ would be a nightmare for incremental updatedb. If you don't want bind mounts pruned (and that includes btrfs subvolumes on the inside of each other), you'll need to turn off the bind mount pruning option. Or you'd have to fix btrfs so that it doesn't implement subvolumes as bind mounts :-) /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1009143: plocate: Similar issue here
On Mon, Apr 18, 2022 at 04:50:58PM -0400, James Cloos wrote: > SG> This was fixed in plocate 1.1.12. ... Which version are you running? > as i mentioned the sbc has buster, so: > > ii plocate1.1.8-2~bpo10+1 armhfmuch faster locate And also, a very old kernel? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1009827: plocate updatedb is not being run
On Mon, Apr 18, 2022 at 03:46:21PM -0400, Ron Murray wrote: > root@khufu:~# systemctl status plocate-updatedb.timer > ○ plocate-updatedb.timer - Update the plocate database daily > Loaded: loaded (/lib/systemd/system/plocate-updatedb.timer; disabled; > vendor preset: enabled) The postinst script enables it (through dh_installsystemd), so this really indicates something or someone disabled it after installation. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1009827: plocate updatedb is not being run
On Mon, Apr 18, 2022 at 02:34:22PM -0400, Ron Murray wrote: >The daily update.db job for plocate is not being run. Last time it > ran on my system was December 30, 2021. Manual updates work fine. What does systemd say about the timer? (E.g., sudo systemctl status plocate-updatedb.timer.) /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1009143: plocate: Similar issue here
On Mon, Apr 18, 2022 at 12:32:20AM -0400, James Cloos wrote: > the strace output ends with: > > openat(AT_FDCWD, "/var/lib/plocate/", O_WRONLY|O_LARGEFILE|O_TMPFILE, 0640) > = -1 EISDIR (Is a directory) > > i've only seen it on arm32 buster, though. It shouldn't end there, though; if the kernel doesn't support O_TMPFILE (which suggests it's pretty old by now), it should fall back to mkstemp(). This was fixed in plocate 1.1.12. And in any case, it wouldn't then really be different from cron. Which version are you running? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1009143: plocate: Similar issue here
On Sun, Apr 17, 2022 at 04:39:30PM -0400, Ron Murray wrote: >Sorry, perhaps I wasn’t too clear. Running updatedb manually works fine. >I just had to do it again. I think the only issue is the auto update. I think that should be on a different bug, then. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1009143: plocate: Similar issue here
On Sun, Apr 17, 2022 at 04:07:21PM -0400, Ron Murray wrote: > I think the problem is that the cron.daily plocate job isn't being run. > I'd suspect the systemd timer doesn't work, but I'm not sure. I'll hack > the cron.daily/plocate script to save some diagnostic information, and > perhaps that'll help. If so, I don't understand why running updatedb manually doesn't do anything. Perhaps there are multiple issues in play on the same bug number. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1009143: plocate: Similar issue here
On Sat, Apr 09, 2022 at 07:50:09PM -0400, Ron Murray wrote: >Steinar, you may be right about problems with the upgrade. I started > looking into this earlier today because 'locate' couldn't find files > that I knew were present in the filesystem. Based on what you said, I > checked plocate.db and siscovered that it hadn't been updated since > December 30. The files I was looking for had, of course, been added > since then. That's strange; do you know if the updatedb.plocate command was ever actually run? Did it output anything to the error log? I've tried reproducing this by installing mlocate and then upgrading, but I can't get it to break. >I also note that there's still a 'locate' in /etc/cron.daily, which > runs '/usr/bin/updatedb.findutils'. Should these be still on the > sysytem? Could they be interfering with plocate? No, it's entirely independent. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1009143: plocate doesn't find any file
On Sat, Apr 09, 2022 at 11:28:58AM +0200, Christoph Berg wrote: > I had not touched plocate after it got automatically installed as > mlocate replacement. > > In fact I have this problem on three different bookworm machines (out > of 3 I use), and only on one of them I got it working after trying to > poke around with permissions on the database file, and probably > deleting it once during fiddling around. I think maybe there's something in the upgrade that's broken, when it tries to convert the mlocate database and then can't reuse it for the second update? But it's a bit strange, there is an autopkgtest. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1009143: plocate doesn't find any file
On Fri, Apr 08, 2022 at 10:26:46PM +0200, Christoph Berg wrote: > $ sudo plocate --debug myon2.jpg > Corpus init done after 0,1 ms. > Dictionary initialized after 0,3 ms. > Hashtable lookups done after 0,3 ms. > trigram 'myo' (6 bytes) decoded to 3 entries > trigram 'yon' (26 bytes) decoded to 20 entries (2 left) > trigram '.jp' (40 bytes) decoded to 30 entries (none left, search is done) > trigram 'jpg' (50 bytes) decoded to 37 entries > trigram '2.j' (33 bytes) decoded to 39 entries (5 left) > trigram 'n2.' (86 bytes) decoded to 70 entries (none left, search is done) > trigram 'on2' (94 bytes) decoded to 78 entries > Intersection done after 0,5 ms. Doing final verification and printing: > Done in 1,5 ms, found 0 matches. OK, so /home isn't pruned, but it still thinks there are zero .jp files anywhere on your filesystem. My guess then would be that it thinks that /home is unchanged since last updatedb _and_ that the existing database doesn't contain anything (for whatever buggy reason). Could you try simply rm-ing the .db file and rerun updatedb? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1009143: plocate doesn't find any file
On Thu, Apr 07, 2022 at 06:36:25PM +0200, Christoph Berg wrote: > [0] 18:29 cbe@lehmann:~ master $ sudo updatedb > > [0] 18:32 cbe@lehmann:~ master $ l /var/lib/plocate/plocate.db > -rw-r- 1 root plocate 6977518 7. Apr 18:32 /var/lib/plocate/plocate.db > > [0] 18:33 cbe@lehmann:~ master $ l myon2.jpg > -rw-rw-r-- 1 cbe cbe 2062 28. Jan 2011 myon2.jpg > > [0] 18:33 cbe@lehmann:~ master $ locate myon2.jpg > > [0] 18:33 cbe@lehmann:~ master $ sudo strace locate myon2.jpg Hi Myon, Could you please try sudo updatedb --debug-pruning sudo plocate --debug myon2.jpg Your plocate.db looks a little small, so it might be that it's not reading your /home for whatever reason. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1007260: performance regression due to linking against libnettle
Package: libunbound8 Version: 1.13.1-1 Severity: normal Hi, We were investigating a performance regression in production that crept in at some point (we noticed by accident that something had become very slow and started investigating). It turns out the culprit was libunbound; we do a series of DNS lookups against localhost using ub_resolve(), and each of them now takes a bit over 6 ms, which is huge for sending a UDP packet and getting an answer from cache. It turns out that some of this is because libunbound in Debian is now built against libnettle (it wasn't when we built the system). The libnettle code in util/random.c goes through a very slow reseeding phase; and worse, it does it for both creating a context (we create a new one for each call, because Reasons(TM)) and for each and every query (because ub_resolve() starts its own worker, which reseeds). This reseeding is responsible for 60% of the CPU usage or so, according to perf. According to pkg-config --libs libunbound, it seems one links to OpenSSL anyway, so perhaps the simplest solution is to stop linking against libnettle? Optionally, one can use getentropy() (which calls getrandom()) unconditionally on Linux, at least with modern kernels. Doing the latter, and also reusing contexts (which is a pain for us, and I don't think we had to do it before?) takes it down to a more reasonable 0.5 ms. -- System Information: Debian Release: 11.2 APT prefers stable-security APT policy: (500, 'stable-security'), (500, 'stable-debug'), (500, 'proposed-updates'), (500, 'oldoldstable'), (500, 'stable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 5.16.12 (SMP w/40 CPU threads; PREEMPT) Locale: LANG=en_DK.UTF-8, LC_CTYPE=en_DK.UTF-8 (charmap=UTF-8), LANGUAGE=en_NO:en_US:en_GB:en Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages libunbound8 depends on: ii libc6 2.31-13+deb11u2 ii libevent-2.1-7 2.1.12-stable-1 ii libgmp102:6.2.1+dfsg-1+deb11u1 ii libhogweed6 3.7.3-1 ii libnettle8 3.7.3-1 libunbound8 recommends no packages. libunbound8 suggests no packages. -- debconf-show failed
Bug#1006906: plocate: cifs file system not being indexed by plocate or mlocate
On Wed, Mar 09, 2022 at 04:03:01PM -0800, Edmund Biow wrote: > Thank you for the suggestion. I noticed that the output of updatedb > mentioned my server as cifs, which is how it is mounted, but also as "type > 'autofs'", which I didn't have installed, so I didn't consider it. I removed > autofs from the PRUNEFS= line, ran updatedb again, and plocate worked as > expected. Thanks again. That's odd, but it's not something updatedb should try to invent on its own. Does /proc/mounts say it is mounted as autofs? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1006906: plocate: cifs file system not being indexed by plocate or mlocate
On Mon, Mar 07, 2022 at 05:12:47PM -0800, Ed Biow wrote: > I always change the /etc/updatedb.conf file to remove cifs from PRUNEFS. The > command works as expected on my Ubuntu systems, mostly 20.04 using the > 0.26-3ubuntu3 of mlocate. In the past mlocate worked on Debian. > The locate command is pretty useless to me without indexing my cifs directory. There is a flag --debug-pruning to updatedb; could you add it and see whether it gives any useful information? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1004083: plocate: stops with misleading error on pruned filesystem
On Sun, Jan 30, 2022 at 01:42:54PM -0700, Kevin Locke wrote: > Sure thing. After re-creating the broken CIFS mount on /mnt, I ran: > > updatedb -o /tmp/plocate.db > locate -d /tmp/plocate.db debian_version Great! Thanks for testing; I'll probably push out a new version shortly. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1004083: plocate: stops with misleading error on pruned filesystem
On Sun, Jan 30, 2022 at 07:25:55PM +0100, Steinar H. Gunderson wrote: > Yes, I wonder if this is the best fix for this specific issue. > If there's an error, there's no performance issue, since the alternative > is total failure; we can check /proc/mounts before deciding to crash out > or nt. I've pushed a change for this to upstream git; are you in a position to try it out? /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1004083: plocate: stops with misleading error on pruned filesystem
On Sun, Jan 30, 2022 at 11:10:00AM -0700, Kevin Locke wrote: > I think the behavior is understandable in both of the above cases, > although it's unfortunate that a flakey mount on an excluded > filesystem at an otherwise always empty directory causes updatedb to > fail entirely. I'm not sure if it might be worth the effort to check > /proc/mounts or statfs(2) on stat(2) failure (for certain error > codes?), or to consider whether the directory is currently empty in > the database, or to skip it without discarding previous entries, or > something else. I'll defer to your judgement. Yes, I wonder if this is the best fix for this specific issue. If there's an error, there's no performance issue, since the alternative is total failure; we can check /proc/mounts before deciding to crash out or nt. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1004083: plocate: stops with misleading error on pruned filesystem
On Thu, Jan 20, 2022 at 08:05:14AM -0700, Kevin Locke wrote: > It turns out this the failure was due to a network issue which > prevented contacting the host for the CIFS mount on /mnt. However, > this presents several issues that might be worth addressing: > > 1. The error message contains the parent directory, rather than the >directory which could not be read, which makes the error harder to >diagnose. This is a real bug, and should be easy to fix. > 2. The error occurred on a CIFS filesystem, which is excluded by >PRUNEFS in /etc/updatedb.conf. Presumably it should be skipped >rather than causing an error. The problem is that updatedb doesn't really know for sure it entered a new filesystem until it's stat-ed the directory. This is logic that's inherited from mlocate, and I guess it would be possible to rewrite it in theory, but such changes tend to be at great peril :-) So what's happening is basically that it stats the directory, checks if is device number is different from the parent, and if so, checks the filesystem from /proc/mounts. You can't just scan it once at the start and be done with it, because someone could be mounting new paths while you're doing updatedb (remember, updatedb can take hours), so you'd need a way to deal with new mounts appearing. (I already have complaints it doesn't immediately skip out if someone _unmounts_ something it's in the process of scanning!) The bind mount code has some special checks for this, since it cannot rely on st_dev (there's a separate thread waiting for changes to mountinfo), but it's fundamentally racy. > 3. updatedb immediately aborts, rather than updating the database for >the rest of the filesystem. In this case, since the failing mount >was at the root, the entire tree is skipped. I don't honestly think this is a bug. If updatedb goes wrong for whatever reason, it's not good to just silently discard parts of the database. If nothing else, the next rescan could become very expensive. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1003089: man-db has become prohibitively slow
On Sat, Jan 22, 2022 at 12:41:56AM +, Colin Watson wrote: >> Technically, UTF-8 validation can be done at a few gigabytes per second >> per core: >> >> >> https://lemire.me/blog/2018/05/16/validating-utf-8-strings-using-as-little-as-0-7-cycles-per-byte/ >> >> but that is probably overkill. :-) > Quite :-) It struck me that it can probably be folded for free into the lexer. If you add symbols for all invalid UTF-8 sequences, I believe it should just go into the state machine. But I'm fine with those 20%; the perfect need not be the enemy of the good here. In general, I don't think I need to look at it again now, unless there are any special questions. Thanks for taking care of this! Looking forward to bookworm being faster (and of course sid before that), and then I'll happily live with this on bullseye, knowing that it's transient. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1003089: man-db has become prohibitively slow
On Fri, Jan 21, 2022 at 09:48:06PM +, Colin Watson wrote: > So the current behaviour isn't a bug as such, but there's definitely > room for optimization here: when operating in-process, and in the common > case where the target encoding is UTF-8, the UTF-8 to UTF-8 trial > decoding path could be changed to just do a read-only "is this UTF-8" > test rather than effectively copying everything to a new buffer via > iconv. I don't know how much faster that would be, though it seems > likely to be an improvement. Technically, UTF-8 validation can be done at a few gigabytes per second per core: https://lemire.me/blog/2018/05/16/validating-utf-8-strings-using-as-little-as-0-7-cycles-per-byte/ but that is probably overkill. :-) > I'll see if I can make time for this, though I think a reasonable > priority for me is to finish working on your existing MR comments first > and get this ready to land. Sure, I agree this is a good prioritization. I saw a new patch set landed, but I'm not sure if you wanted me to look at it again yet? (Fundamentally, though, almost everything I have is style nits; if the patch went into man-db as-is, I would still be happy about it.) /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1003089: man-db has become prohibitively slow
On Mon, Jan 17, 2022 at 10:46:29PM +, Colin Watson wrote:
> test_manfile (which despite the name is not a test function) calls
> find_name with file!="-" and encoding=NULL; that causes find_name to
> call get_page_encoding, which always returns something non-NULL
> ("ISO-8859-1" for English pages), and then call add_manconv from that to
> UTF-8.
I think there's a potential bug here; from attaching gdb and breaking on
iconv_open, it seems there's a lot of encoding from UTF-8 to UTF-8, which
should be no-ops (except that it might do some additional well-formedness
checking). Is that intentional?
Apart from that, I've given your patch a quick run, and it seems to cut out
nearly all of the unneeded overhead. So what we're potentially left with
without doing strange things like multithreading or simplifying the lexer,
is:
- Get rid of the unneeded conversions (~15–20% overhead, it seems).
- Launch in the background, potentially.
Does that make sense? In any case, what we have here is already a huge
improvement.
/* Steinar */
--
Homepage: https://www.sesse.net/
Bug#1003089: man-db has become prohibitively slow
On Mon, Jan 17, 2022 at 10:46:29PM +, Colin Watson wrote: > Mild preference for GitLab MR comments, but I'm not that fussy. I dropped some comments (I've never used the interface before, but it seemed to work OK). What I haven't done is to download the patch and test performance myself; I'll try to get that done soon. I haven't gone deeply into the buffer management code, and of course I haven't tried to figure out any style nits. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1003089: man-db has become prohibitively slow
On Mon, Jan 17, 2022 at 04:10:02AM +, Colin Watson wrote:
> Significant progress! See the end of this email.
Thanks for dealing with this. I'm deleting most of your text, but be sure
that I've read it :-) And I understand most of it (although I of course
disagree at some points, I think those are beside the discussion).
> We definitely do need to sort out encoding conversion, though. Although
> UTF-8 has been recommended for many years, policy still allows "the
> usual legacy encoding" and we've never got round to mandating UTF-8:
>
> $ w3m -dump https://lintian.debian.org/tags/national-encoding | grep
> --count usr/share/man
> 502
I tried looking at this, but TBH I don't think the man-db path _ever_
inserts a conversion. The parameter to the lexer path is simply always NULL
in all calls, except for from a test. Am I missing something?
> I've been experimenting with a few things, and I think the most elegant
> way to do this is in fact to add another layer of abstraction! This is
> a much cheaper one, though. If instead of returning a pipeline we have
> decompress_open return a new tagged union type, we can clone the simple
> pipeline_{read,peek}* functions on top of that, and convert everything
> over to use those rather than talking to libpipeline directly; the
> functions that specifically need pipeline support (mainly in man(1), but
> also things like the cat page case in lexgrog.l) can ask to drop down to
> a lower level.
This sounds fine to me. It's not something I would pick if I maintained the
code myself, but I'm obviously not doing that, and I don't see anything in it
that would preclude optimization. :-)
> Would you care to have a look at this?
>
> https://gitlab.com/cjwatson/man-db/-/merge_requests/2
Thanks! I'll have a go at a review, but it might need to wait until the end
of the week. I'll try to get it done earlier, though. How would you like any
review comments? Email or somehow in Gitlab?
> There's probably still room for improvement, but unlikely to be much
> more than a factor of two or so at this point, and I think this should
> get us comfortably back to the point where it's no longer annoying
> people during upgrades.
Someone else suggested an idea I thought of throwing around: In addition
to optimizing the code, perhaps the postinst trigger should simply launch the
man-db timer in the background? At least for systemd users, this should be
pretty straightforward, and give the control back to the installation
process. (Given that dpkg is very much single-threaded, we're not generally
throughput-bound, so using up a core shouldn't be a big problem.)
/* Steinar */
--
Homepage: https://www.sesse.net/
Bug#1003089: man-db has become prohibitively slow
On Tue, Jan 04, 2022 at 10:28:51PM +0100, Steinar H. Gunderson wrote:
> I made a straw man to test whether this was really true, and it turns it is.
> See the attached patch,
Now actually attached.
/* Steinar */
--
Homepage: https://www.sesse.net/
--- man-db-2.9.4.orig/lib/sandbox.c
+++ man-db-2.9.4/lib/sandbox.c
@@ -52,6 +52,7 @@
#include
#include
+#undef HAVE_LIBSECCOMP
#ifdef HAVE_LIBSECCOMP
# include
# include
--- man-db-2.9.4.orig/src/lexgrog.c
+++ man-db-2.9.4/src/lexgrog.c
@@ -2,6 +2,8 @@
# include "config.h"
#endif /* HAVE_CONFIG_H */
+#include "zlib.h"
+
#include "manconfig.h"
/* Flex emits several functions which might reasonably have various
@@ -13,7 +15,7 @@
#pragma GCC diagnostic ignored "-Wsuggest-attribute=pure"
#pragma GCC diagnostic ignored "-Wunused-macros"
-#line 17 "lexgrog.c"
+#line 19 "lexgrog.c"
#define YY_INT_ALIGNED short int
@@ -2646,7 +2648,7 @@ int yy_flex_debug = 0;
char *yytext;
#line 1 "lexgrog.l"
-#line 19 "lexgrog.l"
+#line 21 "lexgrog.l"
/*
* lexgrog.l: extract 'whatis' info from nroff man / formatted cat pages.
@@ -2861,22 +2863,25 @@ static char filters[MAX_FILTERS];
static int fill_mode;
static int waiting_for_quote;
-static pipeline *decomp;
+//static pipeline *decomp;
+const char *decomp, *decomp_end;
#define YY_INPUT(buf,result,max_size) { \
size_t size = max_size; \
- const char *block = pipeline_read (decomp, ); \
- if (block && size != 0) { \
- memcpy (buf, block, size); \
- buf[size] = '\0'; \
+ if (max_size > decomp_end - decomp) { \
+ size = decomp_end - decomp; \
+ } \
+ if (size > 0) { \
+ memcpy (buf, decomp, size); \
+ decomp += size; \
result = size; \
} else \
result = YY_NULL; \
}
#define YY_NO_INPUT
-#line 2878 "lexgrog.c"
+#line 2883 "lexgrog.c"
-#line 292 "lexgrog.l"
+#line 297 "lexgrog.l"
/* Please add to this list if you know how. */
/* Note that, since flex only supports UTF-8 by accident, character classes
* including non-ASCII characters must be written out as (a|b|c|d) rather
@@ -2884,7 +2889,7 @@ static pipeline *decomp;
*/
/* NOME also works for gl, pt */
/* eptgrv : eqn, pic, tbl, grap, refer, vgrind */
-#line 2888 "lexgrog.c"
+#line 2893 "lexgrog.c"
#define INITIAL 0
#define MAN_PRENAME 1
@@ -3116,11 +3121,11 @@ YY_DECL
}
{
-#line 340 "lexgrog.l"
+#line 345 "lexgrog.l"
/* begin NAME section processing */
-#line 3124 "lexgrog.c"
+#line 3129 "lexgrog.c"
while ( /*CONSTCOND*/1 ) /* loops until end-of-file is reached */
{
@@ -3176,42 +3181,42 @@ do_action: /* This label is used only to
case 1:
/* rule 1 can match eol */
YY_RULE_SETUP
-#line 343 "lexgrog.l"
+#line 348 "lexgrog.l"
BEGIN (MAN_PRENAME);
YY_BREAK
case 2:
/* rule 2 can match eol */
YY_RULE_SETUP
-#line 344 "lexgrog.l"
+#line 349 "lexgrog.l"
BEGIN (CAT_NAME);
YY_BREAK
/* general text matching */
case 3:
-#line 349 "lexgrog.l"
+#line 354 "lexgrog.l"
case 4:
-#line 350 "lexgrog.l"
+#line 355 "lexgrog.l"
case 5:
-#line 351 "lexgrog.l"
+#line 356 "lexgrog.l"
case 6:
/* rule 6 can match eol */
YY_RULE_SETUP
-#line 351 "lexgrog.l"
+#line 356 "lexgrog.l"
YY_BREAK
case 7:
-#line 356 "lexgrog.l"
+#line 361 "lexgrog.l"
case 8:
-#line 357 "lexgrog.l"
+#line 362 "lexgrog.l"
case 9:
/* rule 9 can match eol */
-#line 358 "lexgrog.l"
+#line 363 "lexgrog.l"
case 10:
/* rule 10 can match eol */
YY_RULE_SETUP
-#line 358 "lexgrog.l"
+#line 363 "lexgrog.l"
YY_BREAK
@@ -3219,45 +3224,45 @@ YY_RULE_SETUP
case 11:
/* rule 11 can match eol */
YY_RULE_SETUP
-#line 362 "lexgrog.l"
+#line 367 "lexgrog.l"
filters[TBL_FILTER] = 't';
YY_BREAK
case 12:
/* rule 12 can match eol */
YY_RULE_SETUP
-#line 363 "lexgrog.l"
+#line 368 "lexgrog.l"
filters[EQN_FILTER] = 'e';
YY_BREAK
case 13:
/* rule 13 can match eol */
YY_RULE_SETUP
-#line 364 "lexgrog.l"
+#line 369 "lexgrog.l"
filters[PIC_FILTER] = 'p';
YY_BREAK
case 14:
/* rule 14 can match eol */
YY_RULE_SETUP
-#line 365 "lexgrog.l"
+#line 370 "lexgrog.l"
filters[GRAP_FILTER] = 'g';
YY_BREAK
case 15:
/* rule 15 can match eol */
-#line 367 "lexgrog.l"
+#line 372 "lexgrog.l"
case 16:
/* rule 16 can match eol */
YY_RULE_SETUP
-#line 367 "lexgrog.l"
+#line 372 "lexgrog.l"
filters[REF_FILTER] = 'r';
YY_BREAK
case 17:
/* rule 17 can match eol */
YY_RULE_SETUP
-#line 368 "lexgrog.l"
+#line 373 "lexgrog.l"
filters[VGRIND_FILTER] =
Bug#1003089: man-db has become prohibitively slow
On Tue, Jan 04, 2022 at 10:21:58AM +0100, Steinar H. Gunderson wrote: > I took a look at mandb's profile in perf, and even after turning off > libseccomp, it appears that perhaps 10–11% of its time is spent doing real > work (decompression, lexer, malloc, character set conversion). The rest is > kernel overhead from launching and exiting pipelines, which is a fairly steep > price. I made a straw man to test whether this was really true, and it turns it is. See the attached patch, which rips out the pipelines from mandb and replaces them by simple one-shot decompression buffers; it's by no means something that should be applied in its current state, but it reliably finds and parses all 24216 man pages on my system, giving identical behavior (as determined by “apropos ''”) to the current man-db. But it finishes in 4.7 seconds instead of 51, so indeed, about 9% of the time, and at a performance point where it is unlikely to be too painful in a full-upgrade. Of course, this is an unfair comparison because it does not do encoding conversion, but most pages do not need that, and in any case, adding it would be unlikely to get up to more than 10–11%. It also does not do error handling, arbitrary-size man pages (it uses fixed buffers instead of streaming; again completely possible to fix, or one could set upper limits), anything resembling good coding style, or shelling out to col(1). (I believe the latter is only for parsing cat pages, though, which would seem less relevant today.) But it does set a bar that I think should be approachable by mandb, without any special microoptimization, multithreading, fancy parsing algorithms or the likes. :-) /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1003089: man-db has become prohibitively slow
On Mon, Jan 03, 2022 at 10:34:08PM +, Colin Watson wrote: > This part is already covered in https://bugs.debian.org/696503. I admit > that this was last updated ten years ago; I really need to get back to > that and get it to work one way or another. :-/ Ah, well, that wasn't so easy to understand from man-db's bug list. :-) > This is an interesting observation. I do think that seccomp is valuable > defence in general given the amount of rather ad-hoc parsing that's > going on, but I hadn't noticed that it made such a big difference to > mandb performance. (About 3ms per subprocess here purely for setup, I > think.) > > The seccomp sandbox is mainly so that man-db can more safely operate on > manual pages shipped by packaging systems that expect to run code under > confinement and thus can ship relatively untrusted code (such as snapd). > If we limited the mandb invocation in the postinst to only > /usr/share/man, we could reasonably assume that all pages there were > installed by dpkg, and those don't need the same level of care since any > package installed by dpkg can run code directly as root anyway. It > would then be reasonable to run it under MAN_DISABLE_SECCOMP=1. Let me see if I understand the threat model here, to see where you're coming from. The idea is that the user installs a snap (whose installation and running is nominally sandboxed), which contains a malicious man page containing an arbitrary execution exploit for man-db's apropos parser or similar? That sounds like something that is possible for the postinst to ignore, yes. OTOH, if you assume packages can carry malicious man files, is it also part of the threat model that it could be overriding other package's documentation with spam and/or really bad advice (like “run this vulnerable script as root”)? TBH man-db sounds fairly safe compared to many other things, though. I've only skimmed the code, but as I understand it, a typical mandb run sends the file through gzip and a flex parser? If you find a bug in zlib or in flex, I would assume there are _much_ juicier targets out there than dropping a malicious man page into a snap -- so we're really left with the surrounding code that takes the output of the flex parser. Which isn't really all that much code. I understand these are the famous last words, of course... :-) I understand that man(1) itself pushes things through groff, which is a different story -- but only for the case where you run man setuid. Of course, rewriting these into a safe language would be the best solution, but it's probably out-of-scope. > The difference between MAN_DISABLE_SECCOMP=1 and building without > libseccomp looks like a relatively simple fix, at least: > > > https://gitlab.com/cjwatson/man-db/-/commit/50200d151dfedb9d5064ec7008c09f6cf0f5ee24 OK, that's a good thing. > I really don't want to make apropos less useful. I'll have another go > at tackling the core problem here; but the fixes described above should > at least get us back to merely the situation we've had for years, rather > than the (as you say) much worse behaviour recently. Yes, it would be really nice to have speedups and not just no regressions. :-) We keep installing more packages and man pages, so software probably needs to be made for larger data sets eventually. It's a bit like dpkg -- a lot of the design decisions made sense with 100 packages and 10k files, but when you get to 10x or 100x that, it starts getting a real pain point even if you just freeze it and don't make it any slower. I took a look at mandb's profile in perf, and even after turning off libseccomp, it appears that perhaps 10–11% of its time is spent doing real work (decompression, lexer, malloc, character set conversion). The rest is kernel overhead from launching and exiting pipelines, which is a fairly steep price. On the surface of it, it's not immediately obvious to me what the pipeline abstraction is all about. Is there a reason why you can't just open the man page, decompress it in one shot and then use fmemopen()? (Or similarly, open_memstream().) If it's bigger than, say, 16 MB uncompressed, you can probably just kick it from apropos. > Given #696503, would you mind if I repurposed this bug to be just about > fixing the seccomp-related performance regression? I think the plan > above will let me deal with just that part of it relatively quickly, and > the fixes would be small enough to be viable candidates for stable. Sure, that would be fine. It would perhaps be useful to make 696503 a bit clearer to the casual observer, though; i.e., link it to man-db somehow and make the title more about performance. /* Steinar */ -- Homepage: https://www.sesse.net/
