Users of indent(1) (was: Re: Does anyone use checknr(1)?)
Am 24.05.2024 um 22:05 schrieb B Harder: > > >> On May 24, 2024, at 13:01, Roland Illig wrote: >> >> Hi, >> >> I found /usr/bin/checknr and tried it: >> >> $ checknr /usr/share/man/man1/*.1 >> >> It spit out an enormous amount of error messages and crashed on 68 of >> the files. Its source code hasn't essentially been modified for 18 >> years, and the code still contains several out-of-bounds memory >> accesses, after some half-hearted attempts to fix them. >> >> Does anyone use this tool? Is it worth keeping? >> >> If it's worth keeping it, I could invest some time in making it >> bullet-proof, but I don't want to waste time like I did for indent(1). > > Am I the only person who uses indent(1)? I occasionally use it in "supervised" mode, that is, I manually check all its changes because it still makes lots of mistakes when it comes to distinguishing type names from other identifiers, and for indenting function definitions. Compared to 2019, it has gotten a lot better, but it's still below my expectations. Roland
Does anyone use checknr(1)?
Hi, I found /usr/bin/checknr and tried it: $ checknr /usr/share/man/man1/*.1 It spit out an enormous amount of error messages and crashed on 68 of the files. Its source code hasn't essentially been modified for 18 years, and the code still contains several out-of-bounds memory accesses, after some half-hearted attempts to fix them. Does anyone use this tool? Is it worth keeping? If it's worth keeping it, I could invest some time in making it bullet-proof, but I don't want to waste time like I did for indent(1). Roland
Re: ccd error with two large components
Am 03.03.2024 um 19:55 schrieb Paul Goyette: > I have two 2TB nvme devices, configured with ``ccdconfig ccd0 64 > none /dev/dk1 /dev/dk0'' > then i mount the ccd on /mnt > and then ccdconfig -g goes boom!! > > prevented access to 0x7f7fff9e7fbc (SMAP) > ccd_info_sysctl+77 > > The instruction decode at that point is > > movl 0(%r8), %esi That's this line: > unit = *(const int *)newp; I don't know at which point newp is validated; maybe that validation is missing in this case, although I'd expect it to be in the common sysctl infrastructure code.
Re: dtracing unlink
Am 20.10.2023 um 22:38 schrieb Thomas Klausner: > Hi! > > I'm trying to find out what a program does, and found it does a lot of > unlink syscalls, so I wanted to see what it unlinks. Did you try 'ktruss | grep NAMI' before diving deep into dtrace? Roland
Re: tar: Pathname can't be converted from UTF-8 to current locale
Am 24.02.2023 um 22:55 schrieb Greywolf:
worked.
Glad that it worked out. Your first try was:
TAR=/usr/local/bin/gtar make -e TAR=/usr/local/bin/gtar extract
This approach would have worked for some of the commands that are listed
in pkgtools/bootstrap-mk-files/files/sys.mk.
In pkgsrc, the situation is a bit more complicated. There, each platform
defines the location where each of the common tools is found, in
mk/tools/tools.NetBSD.mk, in this case TOOLS_PLATFORM.tar.
Additionally, mk/tools/defaults.mk defines _TOOLS_VARNAME.tar=TAR, and
at the bottom of that file, the variable TAR is set.
When you overrode TAR=/usr/local/bin/tar, you only modified a result
variable but not the source of truth.
Before the 'extract' phase, the 'tools' phase runs, as documented at the
top of mk/bsd.pkg.mk. There, a wrapper for 'tar' is installed in $PATH,
based on TOOLS_PLATFORM.tar, not on the TAR you overrode.
make -e EXTRACT_CMD='gtar -zxf $${extract_file}' returned
'{extract_file}: No such file or directory'.
Perhaps I needed to double-quote that string.
Now that part is tricky and goes deep into the internals of make. I had
to look it up myself how that works, as I know that '$' characters often
behave unintuitively, especially when they are passed as command line
arguments. It could be regarded a bug in make, but '$' on the command
line is seldom used. If you're interested in further details, just ask. :)
EXTRACT_CMD='gtar -zxf $${extract_file}'
export EXTRACT_CMD; #I'm old school -- this is habit.
make -e extract
Instead, when you set EXTRACT_CMD via an environment variable, make just
passes the environment unaltered to its child processes, not touching
any '$' that is embedded there. That's why this variant worked.
Roland
Re: More POSIX Issue 8 (next version) changes
Am 17.01.2023 um 17:41 schrieb Robert Elz:
https://austingroupbugs.net/view.php?id=1436#c5496
Adds the -j option to make. I doubt there's anything in this which
is different than what we do, but someone familiar with make should
take a look and make sure.
I've read through all the comments, and they look good, with a single
exception.
In '-n' mode, NetBSD make runs command lines starting with '+'. The
suggestion for line 98476 is to also run commands that contain the magic
string '${MAKE}' or '$(MAKE)'. I'd like to preserve the current
behavior, as I don't see a need to have two separate mechanisms to
achieve the same goal. The leading '+' is reliable, the proposed magic
strings aren't.
https://austingroupbugs.net/view.php?id=1437#c5489
More make ... I think this just adds mention of .NOTPARALLEL and .WAIT
in the Rationale section, but once again ...
NetBSD make allows '.NOTPARALLEL: target' lines, while comment 0005489
suggests "without prerequisites". Generating an error for them would be
trivial to implement, I don't know anything about real-world usage of
this directive though.
https://austingroupbugs.net/view.php?id=1471#c5440
More make ... adds :::= and +:= (or something like that).
I don't like them. NetBSD make only knows a single kind of variables,
while GNU make distinguishes immediate variables from delayed variables.
Adding these would make NetBSD make unnecessarily complicated.
https://austingroupbugs.net/view.php?id=1479
More make - should be just a clarification ...
Nothing to do from our side, the issue is about interpreting the default
rules, in our case from .
https://austingroupbugs.net/view.php?id=1513
More make, things missed in earlier changes.
Again the ':::=' assignment operator.
https://austingroupbugs.net/view.php?id=1520#c5556
Even more make ... adds $^
Easy to implement, there's already a comment for '$^' in
usr.bin/make/var.c but no corresponding unit test.
https://austingroupbugs.net/view.php?id=1505#c5616
And yet more make (will this never end?) -- deals with expanding
unset macros.
That's fine, no change to the current behavior.
https://austingroupbugs.net/view.php?id=1549#c5754
And we continue, more make, this one relating to escaped newlines
in macro expansions. I really don't want to know!
Interesting edge case, I didn't have a unit test for a backslash-newline
sequence _after_ macro expansion. The suggestion from 0005754 also
doesn't seem to expect that the backslash-newline sequence can be
generated without occurring literally in the makefile.
Thank you for collecting all these proposed changes to POSIX regarding
make. Even going through this curated list has already taken some time. :)
Roland
Re: Troubles makeing distribution
Am 27.11.2022 um 14:28 schrieb Lizbeth Mutterhunt, Ph.D: No matter I try, if I use ./build./sh I get a compile/install error at @ “liblua”” Hello Lizbeth, Whenever the build fails for you, please post the _exact_ output from the build. In many cases, the last few lines already give enough hints, but in any case, save the complete build log just in case there are further questions. Roland
Finding memory bugs automatically
Hi, Do any of the releng builds and tests run with MALLOC_CONF=junk:true? Background story: I'm tired of finding memory allocation bugs myself when there is a really simple way of doing the same thing automatically. How much time would it cost to run the NetBSD build and the tests with memory allocation debugging, and would it make sense to at least run _some_ of the tests with this option? Is jemalloc capable of running in "evil bug-finding" mode, in which it would return a separate memory page for each allocation request, protecting against out-of-bounds access, and immediately freeing the page as soon as possible and never reusing pages? If not, why isn't there such a mode, and is there an alternative memory allocator that can find these kinds of bugs? Roland
Broken references in manual pages
Hi,
Out of curiosity, I ran the following commands on NetBSD-current/amd64:
cd /usr/share/man
printf '%s\n' */*.*
| xargs grep -n '^\.Xr ' \
| while read src topic section rest; do
test -f "man$section/$topic.$section" \
|| echo "${src%:*}: missing $topic $section"; \
done
This simple command returned 1958 broken references between the manual
pages. This number is a rough estimage. Some broken references are
missed, as some '.Xr' lines contain more than one reference. Some
references point to section "3.", which does not exist.
I checked a few instances of the reported messages, which all resulted
in "man: no entry for %s in the manual".
Shouldn't there be an automatic check for these? Running this check
after a "make distribution" should be fairly easy.
What about other obvious mistakes in manual pages? I know there is
wizd, but if this task can be automated easily, what is preventing us
from doing that?
Roland
Re: Archive of .iso images for -current
Am 15.06.2022 um 12:45 schrieb Valery Ushakov: On Wed, Jun 15, 2022 at 00:05:43 +, Chavdar Ivanov wrote: On 14 June 2022 23:06:56 (+01:00), Roland Illig wrote: NetBSD-current fails for me in a VirtualBox with more than 1 CPU. http://virtualbox.org/manual/ch08.html#vboxmanage-modifyvm and search for cpuid. Thanks for this information and the link to the manual, that's good to know for further investigation. Using VBoxManage, I switched off ACPI, and things got worse: I got kernel panics from process 203.203 (sh) when it called cnopen, as well as from process 0.2, in the idle loop. These kernel panics looked so strange to me that I stopped my experiments there. Do they make sense to someone more involved in kernel development? Roland
Archive of .iso images for -current
Hi, NetBSD-current fails for me in a VirtualBox with more than 1 CPU. https://gnats.netbsd.org/56883 https://gnats.netbsd.org/56884 https://gnats.netbsd.org/56885 NetBSD 9.2 from 2020-02-14 works fine. NetBSD 9.99.97 from 2022-06-14 doesn't work. There's more than 2 years of active development in between. To narrow this down further, I'd like to bisect this without having to build the whole release myself each time. Is there a monthly archive of NetBSD-current somewhere? I looked on https://cdn.netbsd.org/pub/ and https://archive.netbsd.org/ but couldn't find anything. In particular, I'm interested in amd64/boot.iso. Any other ideas? Roland
Random boot failures with multiple CPUs in VirtualBox
Hi, I installed https://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/202204091610Z/amd64/installation/cdrom/boot.iso on a freshly set up VM with the default settings, using VirtualBox 6.1.32 on Windows 11. The installation went smooth, and NetBSD booted quickly when the VM had only 1 CPU configured. I shut down the VM, increased the CPUs to 4 and started the VM. Booting did not succeed, the last message is: [1.0369161] apcicpu3 at cpu3: ACPI CPU After that, the VM burns CPU cycles but does not proceed. After I stopped the VM forcefully and restarted it, booting continued to this message: [14.5656984] kern.module.path=/stand/amd64/9.99.96/modules [77.6403715] WARNING: system needs entropy for security; see entropy(7) The long delay between these messages looks unhealthy. At this point, ACPI shutdown worked though. On the next try, booting succeeded in almost no time. After "shutdown -r now", the next boot hangs after this message: [1.0080102] acpicpu3 at cpu3: ACPI CPU After configuring the VM to have 2 CPUs, booting succeeded quickly 5 times in a row. After configuring the VM to have 3 CPUs, booting stopped after "acpicpu2 at cpu2: ACPI CPU". Are these known problems? Roland
Re: test for interrupted stdio
Am 07.09.2021 um 22:40 schrieb RVP: > On Tue, 7 Sep 2021, Roland Illig wrote: > >> I am running tests/lib/libc/stdio/t_intr on NetBSD 9.99.88/amd64 inside >> VirtualBox. The test fails spuriously. >> >> I find this surprising since cat uses raw I/O. Or is it maybe h_intr >> that exposes the bug? >> > > It's a stdio fwrite()/fflush() bug. Try running h_intr with `-w'. Oops, I completely misread the code of maxwrite. I had thought that it would not handle short reads, but it does exactly that, and its name even says so. No idea what I was thinking. Now I agree that h_intr looks fine and the bug is probably in fwrite/fflush.
test for interrupted stdio
Hi, I am running tests/lib/libc/stdio/t_intr on NetBSD 9.99.88/amd64 inside VirtualBox. The test fails spuriously. Since I didn't want to dive into the internals of bin/dd, I modified t_intr by replacing dd with cat. The test still fails spuriously. I find this surprising since cat uses raw I/O. Or is it maybe h_intr that exposes the bug? It took me some time to get the test to fail, but then ktrace shows: 9535 9535 cat GIO fd 0 read 32 bytes "12769\n12770\n12771\n12772\n12773\n12" 9535 9535 cat RET read 16384/0x4000 ... 9535 9535 cat CALL read(0,0x7126f1f1a480,0x4000) 9535 9535 cat GIO fd 0 read 4088 bytes "87185\n87186\n..." Between these calls to read(0), there are no other reads. This looks to me as if h_intr would contain a bug. If wr_fn(ofp, buf, n) is ever interrupted, it does not try to resubmit the data but simply skips the block. Due to that, cat receives only the partial input. I think h_intr should be more robust when it comes to interrupted system calls. Thoughts? Roland
Re: Blank screen when build packages
Am 07.06.2021 um 16:12 schrieb Dmitrii Postolov: > Hi! Sorry for my bad English... > > NetBSD 9.99.83 GENERIC Sun Jun 6 2021 > > After install NetBSD 9.99.83 and download and unpack pkgsrc-current, I try to > build some apps > from pkgsrc, for example 'editors/nano'. Afrer 'make install clean' the app > begin to build but > after some time the screen is blank and build stop. There is no this problem > in > NetBSD 9.2_STABLE and pkgsrc-current. > > How can I resolve this problem in NetBSD 9.99.x? > > Video: https://disk.yandex.ru/i/ruKLubdmfKYpHQ Hi Dimitrii, that video indeed looks interesting. The most interesting detail is at 00:20. There, the screen does not become black at once, and the lines do not scroll up with constant speed. My guess is that some program changes the foreground and background colors of the terminal, so that the remaining build commands are still shown, but the colors are "black on black", which of course looks invisible. To verify this assumption, please try this command: make install clean 2>&1 | tr -d '\033' This removes any ANSI escape sequences from the output. Now all build commands should be visible. Roland
Re: sys.mk broken for single-suffix rules since 1.144 (2021/11/09)
On 22.03.2021 01:56, bch wrote:
First thought is to alert rillig@, who’s been working a lot on make/lint
It's true that I've been working a lot there, but in this case I'm
innocent. :)
This is really about the wrong command line in sys.mk. Here are the
relevant parts:
OBJECT_TARGET ?= -o ${.TARGET}${defined(CTFCONVERT):?.o:}
.c:
${LINK.c} ${OBJECT_TARGET} ${.IMPSRC} ${LDLIBS}
In this condensed form, it's easy to see that when a .c file is directly
compiled to an executable, the -o option is added nevertheless.
Hmmm, code coverage for makefiles. Sounds obvious since make has its
built-in turing-complete programming language, even allowing for
self-modifying code. ;)
It is definitely possible to record the code coverage for each value of
a variable, for each .if and .for directive, and for each line of shell
commands for the targets. Maybe even for each ':D' and ':U' modifier,
to make sure that both cases are covered. This would be a nice project
to hack into make. Running this coverage analysis will probably find
several edge cases that have previously not been tested thoroughly, at
least that's my experience from working on pkglint (99.91% coverage),
make (85%) and lint1 (75%).
Roland
Re: Automated report: NetBSD-current/i386 test failure
On 07.02.2021 01:33, NetBSD Test Fixture wrote: This is an automatically generated notice of new failures of the NetBSD test suite. The newly failing test cases are: usr.bin/make/t_make:archive usr.bin/make/t_make:cmdline I will take care of these. Roland
Re: bmake inefficiencies
01.02.2021 07:58:26 Simon J. Gerraty : > Martin Husemann wrote: > >> [External Email. Be cautious of content] >> >> >> On Sun, Jan 31, 2021 at 12:41:47PM -0800, Simon J. Gerraty wrote: >>> David Holland wrote: "static volatile sig_atomic_t reap_children;" >>> >>> I chose int (which is what sig_atomic_t is) for portability. >> >> No, it is not int - several NetBSD architectures use long. > > Sorry, the one I checked as int, but I don't see that it should really > matter? int should be the type that any architecture deals with most > naturally. According to https://en.cppreference.com/w/c/program/sig_atomic_t, it has been available since C90, so don't worry about portability. There might be architectures where atomic memory access must happen in units of 64 bit. If you use a 32-bit int, this might result in load64 + mix + store64 instead of a single store32 (at whatever implementation level of the memory), which is not atomic anymore.
Re: bmake inefficiencies
On 31.01.2021 19:28, David Holland wrote: (I'm not sure how much of the jobs/non-jobs copypasting you've cleaned up lately, if any, since I'm way behind on reading the diffs.) That's exactly my point here. I've already cleaned up the part where the shell programs are written in jobs mode, and I wrote a single unit test to demonstrate that the job markers "--- job-a ---" do not always appear at the beginning of a line. Other than that, it's not as simple as waiting for a single child process, as there are the multiple communication pipes, plus updating of the jobs table and probably a few more subtleties. That's why I consider myself still unfamiliar with that part. Roland
Re: bmake inefficiencies
On 29.01.2021 23:22, Roland Illig wrote: On 26.01.2021 15:41, Mateusz Guzik wrote: 3. avoidable temp files JobOpenTmpFile opens a file and later checks if anything was written to it. Not an actual patch, but a hack to demonstrate the fix can be found at the end. Looks like checking can be done upfront in O(1) time (not done in the patch below). i.e. about 44% of all temp files can be avoided at least in this workload. Fixed in usr.bin/make/job.c 1.399, explained in usr.bin/make/unit-tests/jobs-empty-commands.mk. Thanks for the suggestion. Roland
Re: bmake inefficiencies
On 26.01.2021 15:41, Mateusz Guzik wrote:
I had a look with truss at what bmake is doing and I think a lot of
the syscalls can be easily avoided. Examples come from FreeBSD with
bmake-20210110 and I don't know to what extent they are representative
for other systems.
1. spurious waitpid
when doing buildkernel I counted the following:
7010 cases of error code 0 (== nothing to do)
371 cases of ECHILD
1526 cases of reaping
I'm not familiar enough with waitpid, so I will not touch this topic.
Simon?
2. FD_CLOEXEC
for example mkstemp is used and FD_CLOEXEC performed on the result.
Instead mkostemp can be used and mkTempFile can accept O_CLOEXEC as an
argument to save explicit fcntl.
On current NetBSD and FreeBSD, this may be fine, for other platforms
there would need to be autoconf support. Simon?
3. avoidable temp files
JobOpenTmpFile opens a file and later checks if anything was written
to it. Not an actual patch, but a hack to demonstrate the fix can be
found at the end. Looks like checking can be done upfront in O(1) time
(not done in the patch below).
With this in place I get the following result for buildworld:
87930 JobOpenTmpFile: created!
70564 JobOpenTmpFile: skipped!
i.e. about 44% of all temp files can be avoided at least in this workload.
This is something I understood and will tackle. Thank you for doing the
analysis. I think the main cause for this are chains of targets such as:
all: build
build: do-build
do-build: do-build-c
do-build-c: ${SRCS:M*.c:.c=.o}
Thanks for the patch, that looks both simple and correct.
Roland
Re: cross-build error on Linux host
On 26.01.2021 13:45, Christian Groessler wrote: Hi, dependall ===> tools/lint1 # compile lint1/tyname.lo 86 | intern(const char *name) Thanks for the notice, I had added the __noinline accidentally. I removed it again in tyname.c 1.26. Roland
Re: Automated report: NetBSD-current/i386 build failure
On 23.01.2021 19:34, NetBSD Test Fixture wrote: This is an automatically generated notice of a NetBSD-current/i386 build failure. The failure occurred on babylon5.netbsd.org, a NetBSD/amd64 host, using sources from CVS date 2021.01.23.17.58.03. An extract from the build.sh output follows: *** Failed target: dependall-lint1 Just as a heads-up: the build still fails, although in another area now: --- virtio_pci.o --- /tmp/build/2021.01.24.11.34.01-i386/src/sys/dev/pci/virtio_pci.c:743:1: error: static declaration of 'bus_space_write_8' follows non-static declaration
Re: Automated report: NetBSD-current/i386 build failure
On 03.01.2021 21:00, NetBSD Test Fixture wrote: This is an automatically generated notice of a NetBSD-current/i386 build failure. The failure occurred on babylon5.netbsd.org, a NetBSD/amd64 host, using sources from CVS date 2021.01.03.19.15.36. An extract from the build.sh output follows: dependall ===> tools/lint1 sh: cannot open err.c: no such file Sorry, fixed in Makefile.err-msgs-h 1.2. Roland
Re: Automated report: NetBSD-current/i386 build failure
On 07.12.2020 07:42, Andreas Gustafsson wrote: The build is now failing differently: === 2 extra files in DESTDIR = Files in DESTDIR but missing from flist. File is obsolete or flist is out of date ? -- ./usr/tests/usr.bin/make/unit-tests/opt-keep-going-multiple.exp ./usr/tests/usr.bin/make/unit-tests/opt-keep-going-multiple.mk = end of 2 extra files === Fixed in distrib/sets/lists/tests/mi 1.984.
Re: Automated report: NetBSD-current/i386 build failure
On 19.07.2020 19:34, Andreas Gustafsson wrote: > The build is still broken as of source date 2020.07.19.16.22.44: > > > http://releng.netbsd.org/b5reports/i386/commits-2020.07.html#2020.07.19.16.22.44 I'm sorry, I should have had a build running in parallel to my changes all the time. I just committed usr.bin/make/var.c r1.270 which seems to fix everything. (It was a double-free.) I'll try to add a corresponding test to the make(1) unit tests. Roland
Re: pkgsrc current dbus build failure
On 09.05.2020 12:53, Roland Illig wrote: On 08.05.2020 18:36, Ron Georgia wrote: Thank you for pointing that out. I am updating now. I downloaded the NetBSD-9.99.60-amd64-install.img (date stamped May 07, 2020) this morning and installed on my Lenovo X200. I did select the option to download pkgsrc. I changed the path from stable to current and that is what was pulled in. I didn't find NetBSD-9.99.60-amd64-install.img anywhere, therefore I used another ISO image. I installed NetBSD 8 in a VM and, like you, changed the pkgsrc path from stable to current. This way, pkgsrc was downloaded from https://ftp.netbsd.org/pub/pkgsrc/current/, where all files have been updated today in the morning. I would expect that these files are updated daily. There are exactly 7 days between 2020-05-02 and 2020-05-09 though, therefore it could also be that the files are updated weekly. I don't think that's the case, we'll see tomorrow. Indeed the archives in /pub/pkgsrc/current/ are only updated weekly. The pkgsrc tree with the individual files is updated daily though. In such a case you can always "cvs update" locally since the archives include the CVS metadata.
Re: pkgsrc current dbus build failure
On 08.05.2020 18:36, Ron Georgia wrote: Thank you for pointing that out. I am updating now. I downloaded the NetBSD-9.99.60-amd64-install.img (date stamped May 07, 2020) this morning and installed on my Lenovo X200. I did select the option to download pkgsrc. I changed the path from stable to current and that is what was pulled in. I didn't find NetBSD-9.99.60-amd64-install.img anywhere, therefore I used another ISO image. I installed NetBSD 8 in a VM and, like you, changed the pkgsrc path from stable to current. This way, pkgsrc was downloaded from https://ftp.netbsd.org/pub/pkgsrc/current/, where all files have been updated today in the morning. I would expect that these files are updated daily. There are exactly 7 days between 2020-05-02 and 2020-05-09 though, therefore it could also be that the files are updated weekly. I don't think that's the case, we'll see tomorrow. Having a pkgsrc installation from 2020-05-02 would explain everything though.
Re: pkgsrc current dbus build failure
On 08.05.2020 16:44, Ron Georgia wrote: Installed current NetBSD 9.99.60 (GENERIC) #0 and pkgsrc current on 05/08/2020 at about 0900 EST. I tried to build dbus but got an error when it tried to build perl5. sh: 1: Syntax error: Word "/d"p" unexpected (expecting ")") That was a bug in mk/subst.mk r1.91. It was fixed in r1.92, which is from 2020-05-02. This contradicts that you are using pkgsrc-current from 2020-05-08, which is 6 days later.
