Bug#1015731: plocate database gaps after upgrade from mlocate
On Wed, Jul 20, 2022 at 12:24:56PM -0700, Ross Boylan wrote: > I had read the man page, but none of my key partitions is mounted with > a bind option, and so the bind mount exclusion did not seem to apply. > > Could you add a warning, like "btrfs may accomplish a subvolume mount > with a bind mount even if you do not explicitly request it." This text exists in the man page: “Note that Btrfs subvolume mounts are handled internally in the kernel as bind mounts (see btrfs-subvolume(8)), and thus, may get skipped if you have also mounted the filesystem root itself. To counteract this, make your root directory a Btrfs subvolume, too.” > Also, it would be nice to have an explanation of the --debug-pruning output. That is for internal usage, so it's not really supported. :-) > Now update/locate seem to be operating as I wanted. And it's > incredibly fast: 28 seconds for my terabytes of disk space. Nice job! locate needs 28 seconds? That's really, really slow unless you have thousands of hits. It should be on the order of milliseconds. updatedb could very well be in that range. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1015731: plocate database gaps after upgrade from mlocate
I had read the man page, but none of my key partitions is mounted with a bind option, and so the bind mount exclusion did not seem to apply. Could you add a warning, like "btrfs may accomplish a subvolume mount with a bind mount even if you do not explicitly request it." Also, it would be nice to have an explanation of the --debug-pruning output. I did disable pruning of bind mounts, and had already added the duplicate paths to PRUNEPATHS. Now update/locate seem to be operating as I wanted. And it's incredibly fast: 28 seconds for my terabytes of disk space. Nice job!
Bug#1015731: plocate database gaps after upgrade from mlocate
Update on other failed solution: On Tue, Jul 19, 2022 at 5:46 PM Ross Boylan wrote: > > Maybe if I added some of these duplicates, e.g., /root/btr02/root, to > the prunepaths list it would let the remaining one to be indexed? No. I added "/root/btr02/root /root/btr02/usr /root/btr02/backup2 " to PRUNEPATHS, but it still ends with Skipping `/': bind mount root@barley:/etc# ls -l /var/lib/plocate/ total 8 -rw-r--r-- 1 root root183 Nov 17 2021 CACHEDIR.TAG -rw-r- 1 root plocate 952 Jul 19 17:57 plocate.db plocate.db is a little longer, perhaps to hold the additional PRUNEPATHS, but still basically empty. P.S. I also rm -rf /var/lib/mlocate earlier.
Bug#1015731: plocate database gaps after upgrade from mlocate
Thanks for your quick response. On Tue, Jul 19, 2022 at 2:46 PM Steinar H. Gunderson wrote: > . > > > 2. Delete /var/lib/plocate/plocate.db. But perhaps at least an empty file > > with > > the right permissions is required? Then rerun updatedb. > > Yes, you should just rm /var/lib/plocate/plocate.db. > > /* Steinar */ I tried that, but I seem to be going backwards. After updatedb the database is almost empty: root@barley:~# date; time updatedb Tue 19 Jul 2022 05:09:17 PM PDT real0m0.259s user0m0.206s sys0m0.052s root@barley:~# ls -l /var/lib/plocate/ total 8 -rw-r--r-- 1 root root183 Nov 17 2021 CACHEDIR.TAG -rw-r- 1 root plocate 899 Jul 19 17:09 plocate.db and even the files I could locate before have vanished from it. I initially tried systemctl start plocate-updatedb.service with the same result. Looking at the diagnostics below it seems that plocate has decided that / is a bind mount, and so updatedb skips it and everything below. The last line is Skipping `/': bind mount But I'm not sure of my interpretation. For example, just above that it says it is adding / and several other paths, but that perhaps means adding it to the list of bind mounts. Note that the final bind mount listed, for bacula.sql, is a single file. Maybe that caused trouble? BTW, the same directory is available in 2 different spots: / and /root/btr02/root. /root/btr02 is a mountpoint; /root/btr02/root is not, though it is a btrfs subvolume. Also note that some other "top level" systems are mounted, e.g., /root/ubuntu, and so there are several directories that play the role of '/' and several that are '/root' when the corresponding OS is active. Maybe if I added some of these duplicates, e.g., /root/btr02/root, to the prunepaths list it would let the remaining one to be indexed? That's a little brittle if I may be mounting and unmounting as I go; some rule like "use the first" (not sure if the order is available) or "use the one in fstab" might be more robust. Ross Diagnostics follow. # updatedb --debug-pruning -v Tue 19 Jul 2022 05:10:28 PM PDT conf_block: prune_bind_mounts\000 1\000 \000 prunefs\000 AFS\000 AUTOFS\000 BINFMT_MISC\000 CEPH\000 CGROUP\000 CGROUP2\000 CIFS\000 CODA\000 CONFIGFS\000 CURLFTPFS\000 DEBUGFS\000 DEVFS\000 DEVPTS\000 DEVTMPFS\000 ECRYPTFS\000 FTPFS\000 FUSE.CEPH\000 FUSE.GLUSTERFS\000 FUSE.GVFSD-FUSE\000 FUSE.MFS\000 FUSE.RCLONE\000 FUSE.ROZOFS\000 FUSE.SSHFS\000 FUSECTL\000 FUSESMB\000 HUGETLBFS\000 ISO9660\000 LUSTRE\000 LUSTRE_LITE\000 MFS\000 MQUEUE\000 NCPFS\000 NFS\000 NFS4\000 OCFS\000 OCFS2\000 PROC\000 PSTORE\000 RPC_PIPEFS\000 SECURITYFS\000 SHFS\000 SMBFS\000 SYSFS\000 TMPFS\000 TRACEFS\000 UDEV\000 UDF\000 USBFS\000 \000 prunenames\000 \000 prunepaths\000 /media\000 /tmp\000 /var/lib/ceph\000 /var/lib/os-prober\000 /var/spool\000 \000 --- Rebuilding bind_mount_paths: `/sys' (22 on 28) is `/' of `sysfs' (0:20), type `sysfs' `/proc' (23 on 28) is `/' of `proc' (0:21), type `proc' `/dev' (24 on 28) is `/' of `udev' (0:5), type `devtmpfs' `/dev/pts' (25 on 24) is `/' of `devpts' (0:22), type `devpts' `/run' (26 on 28) is `/' of `tmpfs' (0:23), type `tmpfs' `/' (28 on 1) is `/root' of `/dev/mapper/btr02_crypt' (0:24), type `btrfs' `/usr' (29 on 28) is `/usr' of `/dev/mapper/btr02_crypt' (0:24), type `btrfs' `/sys/kernel/security' (27 on 22) is `/' of `securityfs' (0:6), type `securityfs' `/dev/shm' (30 on 24) is `/' of `tmpfs' (0:28), type `tmpfs' `/run/lock' (31 on 26) is `/' of `tmpfs' (0:29), type `tmpfs' `/sys/fs/cgroup' (32 on 22) is `/' of `cgroup2' (0:30), type `cgroup2' `/sys/fs/pstore' (33 on 22) is `/' of `pstore' (0:31), type `pstore' `/sys/firmware/efi/efivars' (34 on 22) is `/' of `efivarfs' (0:32), type `efivarfs' `/sys/fs/bpf' (35 on 22) is `/' of `none' (0:33), type `bpf' `/proc/sys/fs/binfmt_misc' (36 on 23) is `/' of `systemd-1' (0:34), type `autofs' `/dev/mqueue' (37 on 24) is `/' of `mqueue' (0:19), type `mqueue' `/sys/kernel/debug' (38 on 22) is `/' of `debugfs' (0:7), type `debugfs' `/dev/hugepages' (39 on 24) is `/' of `hugetlbfs' (0:35), type `hugetlbfs' `/sys/kernel/tracing' (40 on 22) is `/' of `tracefs' (0:11), type `tracefs' `/sys/fs/fuse/connections' (41 on 22) is `/' of `fusectl' (0:36), type `fusectl' `/sys/kernel/config' (42 on 22) is `/' of `configfs' (0:37), type `configfs' `/root/btr02' (91 on 28) is `/' of `/dev/mapper/btr02_crypt' (0:24), type `btrfs' `/boot' (90 on 28) is `/' of `/dev/sdh3' (8:115), type `ext2' `/root/ubuntu' (96 on 28) is `/' of `/dev/nvme0n1p3' (259:3), type `ext4' `/root/ubuntu/boot/efi' (99 on 96) is `/' of `/dev/nvme0n1p2' (259:2), type `vfat' `/root/installer' (102 on 28) is `/' of `/dev/loop0' (7:0), type `iso9660' `/home' (105 on 28) is `/' of `/dev/mapper/vgb2-home' (253:14), type `ext4' `/var' (108 on 28) is `/' of `/dev/mapper/vgb2-var' (253:15), type `ext4' `/var/local/old-bacula' (112 on 108) is `/backup2' of `/dev/mapper/btr02_crypt' (0:2
Bug#1015731: plocate database gaps after upgrade from mlocate
On Tue, Jul 19, 2022 at 02:37:17PM -0700, Ross Boylan wrote: > I installed plocate; during installation it gave messages suggesting it was > doing a scan, although it seemed awfully quick. > > After I ran locate, and again it missed directories it should have found: > locate -r "/\.hg$" The conversion procedure from mlocate generally trusts that the database is correct. If you have bugs in the mlocate database, they will certainly carry forward through plocate. > 2. Delete /var/lib/plocate/plocate.db. But perhaps at least an empty file > with > the right permissions is required? Then rerun updatedb. Yes, you should just rm /var/lib/plocate/plocate.db. /* Steinar */ -- Homepage: https://www.sesse.net/
Bug#1015731: plocate database gaps after upgrade from mlocate
Package: plocate Version: 1.1.8-2+deb11u1 Severity: normal X-Debbugs-Cc: rossboy...@stanfordalumni.org -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 ORIGINAL PROBLEM I was using mlocate 0.26-5, but it was missing files on btrfs partitions. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=746943 describes the problem and ends with an apparent suggestion to try plocate. PLOCATE PROBLEM I installed plocate; during installation it gave messages suggesting it was doing a scan, although it seemed awfully quick. After I ran locate, and again it missed directories it should have found: locate -r "/\.hg$" DIAGNOSIS updatedb --debug-pruning -U /root -o test-plocate.db /root/btr02, a mount of the root of the btrfs file system, appears not to be excluded, which is good. locate -d test-plocate.db -r "/\.hg$" now correctly recovers lots of .hg directories under btr02. POSSIBLE RECOVERY STRATEGIES 1. Maybe if I wait for the next daily update things will be OK. 2. Delete /var/lib/plocate/plocate.db. But perhaps at least an empty file with the right permissions is required? Then rerun updatedb. 3. More thorough deletion of mlocate, e.g., /var/lib/mlocate still exists and has a database. 4. Upgrade to the later version of plocate available in backports, 1.1.13-1. But I'm not sure if the improved "upgrade from mlocate" logic will be triggered, since the initial plocate install removed the package. I would appreciate guidance on which, if any, of those approaches are advisable. COMMENTS https://sources.debian.org/src/plocate/1.1.16-1/debian/changelog/ indicates the 1.1.8-3, had significant changes in mlocate handling, including the deletion considered in option 3 above. 1.1.8-2 is the version in stable. I'm guessing that the problem is related to the prior existence of mlocate and that plocate is using mlocate's database, rather than the disk, to figure out where to scan. Certainly my diagnostic test indicates plocate can handle btrfs volumes. - -- System Information: Debian Release: 11.4 APT prefers stable-security APT policy: (500, 'stable-security'), (500, 'stable') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 5.10.0-16-amd64 (SMP w/8 CPU threads) Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages plocate depends on: ii libc6 2.31-13+deb11u3 ii libgcc-s1 10.2.1-6 ii libstdc++6 10.2.1-6 ii liburing1 0.7-3 ii libzstd11.4.8+dfsg-2.1 plocate recommends no packages. Versions of packages plocate suggests: ii nocache 1.1-1+b1 ii systemd-sysv 247.3-7 -BEGIN PGP SIGNATURE- iQFSBAEBCgA8FiEEreS674/HIyV9gBfdnAYPmOsbK2AFAmLXI+YeHHJvc3Nib3ls YW5Ac3RhbmZvcmRhbHVtbmkub3JnAAoJEJwGD5jrGytgwTkH/jV8znne/5p+RgRY TNwC5GlubDR576hCNv7Hi5/p/LaTwb4+vk0FwPAE8tt5ZkVZ+3U1aLdDt2DC5FB0 WwBjZvYyBQtpHttpery4ivAAl21t+TiD1RAOMACxXEupWZc9Wtgbbl3tvTgPY/8f KvR/f9XJEYMwEDpKp6dwJoPa41VlWoZX8blhulIDzL2VXCZ35KlqxScRM4s/iV3x qjNFt2XNIqSNPcd4M2K1Fttdbt8f6KMRcsC/tPAp1U1uYaCWVdouZdBiE2MTwwNU nvk/96P2g6ndIucucmD2QDDzHgM+rBMr7OfXemlFyYbb+uYbkmLCo/mgTO406B+G DLdPUHQ= =8QT6 -END PGP SIGNATURE-
