Re: Issues with manuals+amd(8) for a handful of ports
Hi, are there any other users who have /usr/local or /usr/local/man as a symlink or mounted via amd(8)? Note that i do NOT recommend such very unusual configurations at all, they have caused trouble in the past, i'm merely asking for testing purposes. If you use such a configuration, please update your mandoc such that you have OpenBSD: mandocdb.c,v 1.215 (or later) (by installing a snapshot dated at least a full day after 2020/01/26 21:24:58 UTC or by compiling from source and installing in /usr/src/usr.bin/mandoc/) and try whether after installing packages that contain symlinks in manual page directories, man(1) works as expected on such manual pages even *without* running makewhatis(8) manually and *without* running weekly(8). And of course, watch out whether you see any regressions. It turned out the filescan() function in mandocdb.c is well suited as a habitat for worms, and if i have disturbed any of them with the commit below, i'd prefer appeasing them again ASAP, well before we approach release. Here are a few examples of manual pages that are good for testing: package manual page vim rgvim # as opposed to vim(1) itself python-2.7python2# as opposed to python2.7(1) openldap-client ldap_init # as opposed to ldap_open(3) texlive_texmf-minimal pdflatex # as opposed to pdftex(1) The following command may give you more examples of pages for testing on your system: $ find /usr/local/man -type l Please make sure that *after* updating mandoc, you pkg_delete(1) the package(s) in question and then pkg_add(1) them anew, and only test man(1) after that. Please report any cases where pages aren't found as expected, report any messages similar to man: outdated mandoc.db lacks rgvim(1) entry, run makewhatis /usr/local/man you may encounter, and report any other regressions. Always include output from "man -w" for the pages you are trying to read, in addition to describing your findings. In case any of this screws up your system, running makewhatis(8) manually without any arguments will almost certainly fix it (but please report first). Do not attempt to downgrade again in the unlikely case that testing reveals issues; that would be more likely to make matters worse rather than help. Finally, note that nothing is wrong in pkg_add(1), the issues were in makewhatis(8) only. Thanks, Ingo CVSROOT:/cvs Module name:src Changes by: schwa...@cvs.openbsd.org2020/01/26 14:24:58 Modified files: usr.bin/mandoc : mandocdb.c Log message: Repair more of the issues that i found in filescan() while investigating the report from on ports@: For a symlink, use the first of the following names that is available: 1. In -t mode, the symlink itself (unchanged). 2. When the (unresolved) symlink already resides inside the manpath, just strip the manpath and use the rest (unchanged). 3. When prefix(es) of the unresolved symlink point to the manpath, strip the longest such prefix and use the rest (new); this fixes situations where the manpath or one of its parent directories is a symlink and at the same time contains symlinks to manual pages. 4. Fall back to the fully resolved symlink, with the manpath stripped (new); this may for example happen when the command line passes symlinks from outside the manpath that point to manual pages inside the manpath, or if manual page trees contain symlinks to symlinks and not all of them are given on the command line. The fallback (4) isn't perfect. You can construct symlink spaghetti in such a way that this algorithm will not enter all manual page names into the database that a human would be able to deduce. But i do not expect such spaghetti to actually occur in practice (not even in ports), and a full fix would require re-implementing realpath(3) in terms of step-by-step readlink(2) calls, repeating the complicated algorithm (3) after each step. While here, also stop using PATH_MAX as the size of a static buffer in filescan(); on some systems, it can be unreasonably large. Instead, allocate path strings dynamically.
Re: Issues with manuals+amd(8) for a handful of ports
On Sat, Jan 25, 2020 at 10:16:47AM +0100, Kusalananda Kähäri wrote: > On Sat, Jan 25, 2020 at 03:07:00AM +0100, Ingo Schwarze wrote: > > Hi Andreas, > > Good morning Ingo, > > See below. > > > > > Andreas Kusalananda wrote on Wed, Jan 22, 2020 at 08:58:26PM +0100: > > > > > I mount my /usr/local directory from a remote server over NFS > > > using amd(8). This means that I have the directories directly [cut] > Since running makewhatis on /usr/local/man (and actually also without > a directory argument at all) corrects the database, I expect that the > /etc/weekly script will actually correct the issue on my other machines, > rather than corrupt the database with bogus entries. Forgot to say that running # sh /etc/weekly fixes the issue on a machine where the manual database has these boges entries. I've moved the weekly job to another day, so I'll wait and see if the automatic run of this fixes the issue on other machines (but I would expect it to fix it). -- Andreas (Kusalananda) Kähäri SciLifeLab, NBIS, ICM Uppsala University, Sweden .
Re: Issues with manuals+amd(8) for a handful of ports
On Sat, Jan 25, 2020 at 03:07:00AM +0100, Ingo Schwarze wrote:
> Hi Andreas,
Good morning Ingo,
See below.
>
> Andreas Kusalananda wrote on Wed, Jan 22, 2020 at 08:58:26PM +0100:
>
> > I mount my /usr/local directory from a remote server over NFS
> > using amd(8). This means that I have the directories directly
> > beneath /usr/local appear on demand as symbolic links into
> > /tmp_mnt/eeyore/vol/local/pooh where the /usr/local for the local
> > machine "pooh" is mounted over NFS from "eeyore" (and mounted/unmounted
> > on demand).
> >
> > This works very well, but there is a handful of ports that misbehave
> > with regards to manuals. Of the 70 or so pre-built packages that I've
> > installed on my amd64-current system, the only ones misbehaving are
> >
> > databases/pkglocatedb (pkglocatedb-1.5)
> > devel/git,-main (git-2.25.0)
> > sysutils/p5-File-Rename-1.10(p5-File-Rename-1.10)
> >
> > When I do "man pkglocate", for example, I get
> >
> > man: /tmp_mnt/eeyore/vol/local/pooh/man//usr/local/man/man1/pkg_locate.1:
> > SYSERR: open: No such file or directory
> >
> > ... and likewise for any other utility etc. from those ports. Commands
> > like "git clone --help" gives similar diagnostics.
> >
> > The manuals are actually visible where they are supposed to be:
> >
> > $ ls -l /usr/local/man/man1/{git{,-clone},pkg_locate}.1
> > -rw-r--r-- 1 root bin 20005 Jan 20 22:20 /usr/local/man/man1/git-clone.1
> > -rw-r--r-- 1 root bin 50901 Jan 13 19:49 /usr/local/man/man1/git.1
> > -rw-r--r-- 1 root bin 2845 Jan 19 08:14 /usr/local/man/man1/pkg_locate.1
> >
> > ... so it looks as if man(1) is, for whatever reason, looking in the
> > wrong location for the file.
>
> The man(1) manual page says:
>
> The mandoc.db(5) database is used for looking up manual page entries.
> In cases where the database is absent, outdated, or corrupt, man falls
> back to looking for files called name.section.
>
> However, when the database is present and in the correct format but
> references non-existent files like
> /tmp_mnt/eeyore/vol/local/pooh/man/usr/local/man/man1/pkg_locate.1
> there is no fallback. The reason is that by the time it turns out
> the file name found in the database is bogus, man(1) has progressed
> beyond the stage where it can repeat the search using a different
> method. Fallback to file system search only happens when no match
> is found, not when the database contains a match but the path in
> the database happens to be wrong.
>
> > Manuals for other utilities, like rsync, pv, pwgen, openvpn, etc., do
> > work as they should.
>
> This sounds as if pkg_add(1) sometimes worked well but sometimes got
> confused when running makewhatis(8) and consequently your database
> is now corrupt in a weird way that i have never seen before.
>
> Please show the output of
>
>$ man -w pkglocate rsync
$ man -w pkglocate rsync
/tmp_mnt/eeyore/vol/local/pooh/man//usr/local/man/man1/pkg_locate.1
/tmp_mnt/eeyore/vol/local/pooh/man/man1/pkg_locate.1
/tmp_mnt/eeyore/vol/local/pooh/man/man1/rsync.1
>
> The expected, correct result is
>
>$ man -w pkglocate rsync
> /usr/local/man/man1/pkg_locate.1
> /usr/local/man/man1/rsync.1
>
> I expect you may see
>
> /tmp_mnt/eeyore/vol/local/pooh/man//usr/local/man/man1/pkg_locate.1
> /usr/local/man/man1/rsync.1
Yes, but with the *correct* entry for pkg_locate.1 also present after
the first bogus one.
>
> I suspect if you run
>
># makewhatis /tmp_mnt/eeyore/vol/local/pooh/man
Yes, this corrects the database:
$ man -w pkglocate rsync
/tmp_mnt/eeyore/vol/local/pooh/man/man1/pkg_locate.1
/tmp_mnt/eeyore/vol/local/pooh/man/man1/rsync.1
The command
# makewhatis /usr/local/man
also corrects the database. As does running makewhatis with no
arguments.
>
> the problem may go away, but after that, you may no longer be able
> to reproduce the problem. Also, the problem might re-emerge when
> you install more packages because pkg_add(1) will run makewhatis(8)
> on /usr/local/man, not on /tmp_mnt/... - and so will weekly(8).
Since running makewhatis on /usr/local/man (and actually also without
a directory argument at all) corrects the database, I expect that the
/etc/weekly script will actually correct the issue on my other machines,
rather than corrupt the database with bogus entries.
The issue seems to arise from the pkg_* tools.
Here I've made sure that rsync is not installed and that the manual
database is correct:
$ man -w rsync
man: No entry for rsync in the manual.
$ doas pkg_add -D snap rsync
doas (k...@pooh.prefix.duckdns.org) password:
quirks-3.216 signed on 2020-01-24T11:25:24Z
Ambiguous: choose package for rsync
a 0:
1: rsync-3.1.3
2: rsync-3.1.3-iconv
Your choice: 1
rsync-3.1.3: ok
The following new rcscripts were installed: /etc/rc.d/rsyncd
See rcctl(8) for details.
$ man -w rsync
/tmp_mnt/eeyore/vol/local/pooh/man//usr/local/man/man1/rsync.1
$
Re: Issues with manuals+amd(8) for a handful of ports
Hi Andreas,
Andreas Kusalananda wrote on Wed, Jan 22, 2020 at 08:58:26PM +0100:
> I mount my /usr/local directory from a remote server over NFS
> using amd(8). This means that I have the directories directly
> beneath /usr/local appear on demand as symbolic links into
> /tmp_mnt/eeyore/vol/local/pooh where the /usr/local for the local
> machine "pooh" is mounted over NFS from "eeyore" (and mounted/unmounted
> on demand).
>
> This works very well, but there is a handful of ports that misbehave
> with regards to manuals. Of the 70 or so pre-built packages that I've
> installed on my amd64-current system, the only ones misbehaving are
>
> databases/pkglocatedb (pkglocatedb-1.5)
> devel/git,-main (git-2.25.0)
> sysutils/p5-File-Rename-1.10(p5-File-Rename-1.10)
>
> When I do "man pkglocate", for example, I get
>
> man: /tmp_mnt/eeyore/vol/local/pooh/man//usr/local/man/man1/pkg_locate.1:
> SYSERR: open: No such file or directory
>
> ... and likewise for any other utility etc. from those ports. Commands
> like "git clone --help" gives similar diagnostics.
>
> The manuals are actually visible where they are supposed to be:
>
> $ ls -l /usr/local/man/man1/{git{,-clone},pkg_locate}.1
> -rw-r--r-- 1 root bin 20005 Jan 20 22:20 /usr/local/man/man1/git-clone.1
> -rw-r--r-- 1 root bin 50901 Jan 13 19:49 /usr/local/man/man1/git.1
> -rw-r--r-- 1 root bin 2845 Jan 19 08:14 /usr/local/man/man1/pkg_locate.1
>
> ... so it looks as if man(1) is, for whatever reason, looking in the
> wrong location for the file.
The man(1) manual page says:
The mandoc.db(5) database is used for looking up manual page entries.
In cases where the database is absent, outdated, or corrupt, man falls
back to looking for files called name.section.
However, when the database is present and in the correct format but
references non-existent files like
/tmp_mnt/eeyore/vol/local/pooh/man/usr/local/man/man1/pkg_locate.1
there is no fallback. The reason is that by the time it turns out
the file name found in the database is bogus, man(1) has progressed
beyond the stage where it can repeat the search using a different
method. Fallback to file system search only happens when no match
is found, not when the database contains a match but the path in
the database happens to be wrong.
> Manuals for other utilities, like rsync, pv, pwgen, openvpn, etc., do
> work as they should.
This sounds as if pkg_add(1) sometimes worked well but sometimes got
confused when running makewhatis(8) and consequently your database
is now corrupt in a weird way that i have never seen before.
Please show the output of
$ man -w pkglocate rsync
The expected, correct result is
$ man -w pkglocate rsync
/usr/local/man/man1/pkg_locate.1
/usr/local/man/man1/rsync.1
I expect you may see
/tmp_mnt/eeyore/vol/local/pooh/man//usr/local/man/man1/pkg_locate.1
/usr/local/man/man1/rsync.1
I suspect if you run
# makewhatis /tmp_mnt/eeyore/vol/local/pooh/man
the problem may go away, but after that, you may no longer be able
to reproduce the problem. Also, the problem might re-emerge when
you install more packages because pkg_add(1) will run makewhatis(8)
on /usr/local/man, not on /tmp_mnt/... - and so will weekly(8).
In general, makewhatis(8) dislikes symbolic links pointing outside
the respective manpath and uses realpath(3) to normalize them,
then rejects the files if they aren't in a valid location.
What's a bit weird is that here it apparently accepted the files
but then somehow constructed wrong filenames anyway.
I might have to do some experimentation with symlinks to figure
out what exactly is going on.
You could also go to
$ cd /usr/src/regress/usr.bin/mandoc/db/dbm_dump
$ man mandoc.db
$ man -l dbm_dump.1
$ rm -f obj
$ make cleandir
$ make obj
$ make cleandir
$ make
$ ./obj/dbm_dump /usr/local/man/mandoc.db | less
to look at the (probably) broken database in detail.
> I haven't created or changed any MAN* environment variables nor do I
> have a /etc/man.conf file. I haven't been able to figure out what's
> special about the manuals for these ports.
I don't expect anything to be special about these ports; i suspect
that makewhatis(8) was run in a slightly different way or under
slightly different circumstances. I'm not surprised that your
stunt of symnlinking /usr/local/man somewhere else turns out to be
fragile.
Yours,
Ingo
Issues with manuals+amd(8) for a handful of ports
Hi,
I mount my /usr/local directory from a remote server over NFS
using amd(8). This means that I have the directories directly
beneath /usr/local appear on demand as symbolic links into
/tmp_mnt/eeyore/vol/local/pooh where the /usr/local for the local
machine "pooh" is mounted over NFS from "eeyore" (and mounted/unmounted
on demand).
This works very well, but there is a handful of ports that misbehave
with regards to manuals. Of the 70 or so pre-built packages that I've
installed on my amd64-current system, the only ones misbehaving are
databases/pkglocatedb (pkglocatedb-1.5)
devel/git,-main (git-2.25.0)
sysutils/p5-File-Rename-1.10(p5-File-Rename-1.10)
When I do "man pkglocate", for example, I get
man: /tmp_mnt/eeyore/vol/local/pooh/man//usr/local/man/man1/pkg_locate.1:
SYSERR: open: No such file or directory
... and likewise for any other utility etc. from those ports. Commands
like "git clone --help" gives similar diagnostics.
The manuals are actually visible where they are supposed to be:
$ ls -l /usr/local/man/man1/{git{,-clone},pkg_locate}.1
-rw-r--r-- 1 root bin 20005 Jan 20 22:20 /usr/local/man/man1/git-clone.1
-rw-r--r-- 1 root bin 50901 Jan 13 19:49 /usr/local/man/man1/git.1
-rw-r--r-- 1 root bin 2845 Jan 19 08:14 /usr/local/man/man1/pkg_locate.1
... so it looks as if man(1) is, for whatever reason, looking in the
wrong location for the file.
Manuals for other utilities, like rsync, pv, pwgen, openvpn, etc., do
work as they should.
I haven't created or changed any MAN* environment variables nor do I
have a /etc/man.conf file. I haven't been able to figure out what's
special about the manuals for these ports.
--
Andreas (Kusalananda) Kähäri
SciLifeLab, NBIS, ICM
Uppsala University, Sweden
.
