Re: main [and, likely, stable/14]: do not set vfs.zfs.bclone_enabled=1 with that zpool feature enabled because it still leads to panics
On 09.09.2023 12:32, Mark Millard wrote: On Sep 8, 2023, at 21:54, Mark Millard wrote: On Sep 8, 2023, at 18:19, Mark Millard wrote: On Sep 8, 2023, at 17:03, Mark Millard wrote: On Sep 8, 2023, at 15:30, Martin Matuska wrote: On 9. 9. 2023 0:09, Alexander Motin wrote: Thank you, Martin. I was able to reproduce the issue with your script and found the cause. I first though the issue is triggered by the `cp`, but it appeared to be triggered by `cat`. It also got copy_file_range() support, but later than `cp`. That is probably why it slipped through testing. This patch fixes it for me: https://github.com/openzfs/zfs/pull/15251 . Mark, could you please try the patch? I finally stopped it at 7473 built (a little over 13 hrs elapsed): ^C[13:08:30] Error: Signal SIGINT caught, cleaning up and exiting [main-amd64-bulk_a-default] [2023-09-08_19h51m52s] [sigint:] Queued: 34588 Built: 7473 Failed: 23Skipped: 798 Ignored: 335 Fetched: 0 Tobuild: 25959 Time: 13:08:26 [13:08:30] Logs: /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-09-08_19h51m52s [13:08:31] Cleaning up [13:17:10] Unmounting file systems Exiting with status 1 In part that was more evidence for deadlocks at least being fairly rare as well. None of the failed ones looked odd. (A fair portion are because the bulk -a was mostly doing WITH_DEBUG= builds. Many upstreams change library names, some other file names, or paths used for debug builds and ports generally do not cover well building the debug builds for such. I've used these runs to extend my list of exceptions that avoid using WITH_DEBUG .) So no evidence of corruptions. Thank you, Mark. The patch was accepted upstream and merged to both master and zfs-2.2-release branches. -- Alexander Motin
Re: Looks like the kyua zfs tests likely are not used on aarch64 or other contexts with unsigned char
On Sep 10, 2023, at 11:21, Warner Losh wrote:
> On Sun, Sep 10, 2023, 11:10 AM Mark Millard wrote:
>> On Sep 10, 2023, at 00:31, Mark Millard wrote:
>>
>> > kyua tests that use the:
>> >
>> > /usr/tests/sys/cddl/zfs/bin/mkfile
>> >
>> > program like so (for example):
>> >
>> > mkfile 500M /testpool.1861/bigfile.0
>> >
>> > (which should be valid) end up with mkfile
>> > instead reporting:
>> >
>> > Standard error:
>> > Usage: mkfile [-nv] [e|p|t|g|m|k|b] ...
>> >
>> > which prevent the kyua test involved from working.
>> >
>> > Turns out this is from expecting char to be always
>> > signed (so a -1 vs. 255 distinction, here in an
>> > aarch64 context):
>> >
>> > . . .
>> > (gdb) list
>> > 179 /* Options. */
>> > 180 while ((ch = getopt(argc, argv, "nv")) != -1) {
>> > 181 switch (ch) {
>> > 182 case 'n':
>> > 183 nofill = 1;
>> > 184 break;
>> > 185 case 'v':
>> > (gdb) print ch
>> > $16 = 255 '\377'
>> > (gdb) print/x -1
>> > $17 = 0x
>> > (gdb) print/x ch
>> > $18 = 0xff
>> > . . .
>> >
>> > With the mix of unsigned and signed it ends up
>> > being a 0xffu != 0xu test, which is
>> > always true.
>> >
>> > So the switch is reached as if a "-" prefix was
>> > present (that is not). Then the "option" is classified
>> > as invalid and the usage message is produced.
>> >
>> > Apparently no one had noticed. That, in turn, suggests a
>> > lack of inspected testing on aarch64, powerpc64,
>> > powerpc64le, armv7, powerpc, and powerpcspe. That, in
>> > turn, suggests that kyua test inspection for the likes
>> > of aarch64 is not historically a part of the release
>> > process for openzfs or for operating systems that include
>> > openzfs.
>> >
>>
>> Looks like the mkfile.c traces back to a former port
>> sysutils/mkfile that was unfetchable as of 2019. And,
>> looking around, it seems the kyua zfs tests may be a
>> FreeBSD only thing, not adopted in openzfs.
>>
>> So various implicit assumptions when I wrote the note
>> do not actually hold.
>>
>> FreeBSD would have to do additional testing via kyua,
>> beyond what openzfs does for testing, to discover the
>> unsigned char related mis-behavior in the mkfile that
>> FreeBSD's kyua tests use. Only FreeBSD variants are
>> likely to have a similar status, not general openzfs
>> including operating systems.
>
> I wonder how hard ot would be to look for the char = getopt() pattern with
> coccinelle
>
Unsure.
But to be sure that the implication that I was also trying
to point out is not lost: kyua testing of zfs (and more?)
for aarch64 (tier 1) is apparently not being done (or at
least the results are not being inspected). Similarly for
armv7 and all the powerpc*'s (not tier 1's, however, so
not as surprising).
Side note:
Via other exchanges that have been going on I learned to
look in the likes of:
https://ci.freebsd.org/job/FreeBSD-main-amd64-testvm/*/consoleText
for what to "pkg install" for kyua test runs to use for
normal runs (at least the subset compatible with
architecture being tested). I'd only figured out a (large)
subset previously for aarch64 and armv7.
I'm not aware of there being other documentation for what
is appropriate for setting up such for kyua runs.
===
Mark Millard
marklmi at yahoo.com
Re: Looks like the kyua zfs tests likely are not used on aarch64 or other contexts with unsigned char
On Sun, Sep 10, 2023, 11:10 AM Mark Millard wrote:
> On Sep 10, 2023, at 00:31, Mark Millard wrote:
>
> > kyua tests that use the:
> >
> > /usr/tests/sys/cddl/zfs/bin/mkfile
> >
> > program like so (for example):
> >
> > mkfile 500M /testpool.1861/bigfile.0
> >
> > (which should be valid) end up with mkfile
> > instead reporting:
> >
> > Standard error:
> > Usage: mkfile [-nv] [e|p|t|g|m|k|b] ...
> >
> > which prevent the kyua test involved from working.
> >
> > Turns out this is from expecting char to be always
> > signed (so a -1 vs. 255 distinction, here in an
> > aarch64 context):
> >
> > . . .
> > (gdb) list
> > 179 /* Options. */
> > 180 while ((ch = getopt(argc, argv, "nv")) != -1) {
> > 181 switch (ch) {
> > 182 case 'n':
> > 183 nofill = 1;
> > 184 break;
> > 185 case 'v':
> > (gdb) print ch
> > $16 = 255 '\377'
> > (gdb) print/x -1
> > $17 = 0x
> > (gdb) print/x ch
> > $18 = 0xff
> > . . .
> >
> > With the mix of unsigned and signed it ends up
> > being a 0xffu != 0xu test, which is
> > always true.
> >
> > So the switch is reached as if a "-" prefix was
> > present (that is not). Then the "option" is classified
> > as invalid and the usage message is produced.
> >
> > Apparently no one had noticed. That, in turn, suggests a
> > lack of inspected testing on aarch64, powerpc64,
> > powerpc64le, armv7, powerpc, and powerpcspe. That, in
> > turn, suggests that kyua test inspection for the likes
> > of aarch64 is not historically a part of the release
> > process for openzfs or for operating systems that include
> > openzfs.
> >
>
> Looks like the mkfile.c traces back to a former port
> sysutils/mkfile that was unfetchable as of 2019. And,
> looking around, it seems the kyua zfs tests may be a
> FreeBSD only thing, not adopted in openzfs.
>
> So various implicit assumptions when I wrote the note
> do not actually hold.
>
> FreeBSD would have to do additional testing via kyua,
> beyond what openzfs does for testing, to discover the
> unsigned char related mis-behavior in the mkfile that
> FreeBSD's kyua tests use. Only FreeBSD variants are
> likely to have a similar status, not general openzfs
> including operating systems.
>
I wonder how hard ot would be to look for the char = getopt() pattern with
coccinelle
Warner
===
> Mark Millard
> marklmi at yahoo.com
>
>
>
Re: Looks like the kyua zfs tests likely are not used on aarch64 or other contexts with unsigned char
On Sep 10, 2023, at 00:31, Mark Millard wrote:
> kyua tests that use the:
>
> /usr/tests/sys/cddl/zfs/bin/mkfile
>
> program like so (for example):
>
> mkfile 500M /testpool.1861/bigfile.0
>
> (which should be valid) end up with mkfile
> instead reporting:
>
> Standard error:
> Usage: mkfile [-nv] [e|p|t|g|m|k|b] ...
>
> which prevent the kyua test involved from working.
>
> Turns out this is from expecting char to be always
> signed (so a -1 vs. 255 distinction, here in an
> aarch64 context):
>
> . . .
> (gdb) list
> 179 /* Options. */
> 180 while ((ch = getopt(argc, argv, "nv")) != -1) {
> 181 switch (ch) {
> 182 case 'n':
> 183 nofill = 1;
> 184 break;
> 185 case 'v':
> (gdb) print ch
> $16 = 255 '\377'
> (gdb) print/x -1
> $17 = 0x
> (gdb) print/x ch
> $18 = 0xff
> . . .
>
> With the mix of unsigned and signed it ends up
> being a 0xffu != 0xu test, which is
> always true.
>
> So the switch is reached as if a "-" prefix was
> present (that is not). Then the "option" is classified
> as invalid and the usage message is produced.
>
> Apparently no one had noticed. That, in turn, suggests a
> lack of inspected testing on aarch64, powerpc64,
> powerpc64le, armv7, powerpc, and powerpcspe. That, in
> turn, suggests that kyua test inspection for the likes
> of aarch64 is not historically a part of the release
> process for openzfs or for operating systems that include
> openzfs.
>
Looks like the mkfile.c traces back to a former port
sysutils/mkfile that was unfetchable as of 2019. And,
looking around, it seems the kyua zfs tests may be a
FreeBSD only thing, not adopted in openzfs.
So various implicit assumptions when I wrote the note
do not actually hold.
FreeBSD would have to do additional testing via kyua,
beyond what openzfs does for testing, to discover the
unsigned char related mis-behavior in the mkfile that
FreeBSD's kyua tests use. Only FreeBSD variants are
likely to have a similar status, not general openzfs
including operating systems.
===
Mark Millard
marklmi at yahoo.com
Re: sed in CURRENT fails in textproc/jq
Hi Warner, Thank you for your response. Am Sun, Sep 10, 2023 at 09:53:03AM -0600 schrieb Warner Losh: > On Sun, Sep 10, 2023, 7:36 AM Robert Clausecker wrote: > > > Hi Warner, > > > > I have pushed a fix. It should hopefully address those failing tests. > > The same issue should also affect memcmp(), but unlike for memchr(), it is > > illegal to pass a length to memcmp() that extends past the actual end of > > the buffer as memcmp() is permitted to examine the whole buffer regardless > > of where the first mismatch is. > > > > I am considering a change to improve the behaviour of memcmp() on such > > errorneous inputs. There are two options: (a) I could change memcmp() the > > same way I fixed memchr() and have implausible buffer lengths behave as if > > the buffer goes to the end of the address space or (b) I could change > > memcmp() to crash loudly if it detects such a case. I could also > > (c) leave memcmp() as is. Which of these three choices is preferable? > > > > What does the standard say? I'm highly skeptical that these corner cases are > UB behavior. > > I'd like actual support for this statement, rather than your conjecture > that it's > illegal. Even if you can come up with that, preserving the old behavior is > my > first choice. Especially since many of these functions aren't well defined > by > a standard, but are extensions. > > As for memchr, > https://pubs.opengroup.org/onlinepubs/009696799/functions/memchr.html > has no such permission to examine 'the entire buffer at once' nor any > restirction > as to the length extending beyond the address space. I'm skeptical of your > reading > that it allows one to examine all of [b, b + len), so please explain where > the standard > supports reading past the first occurance. memchr() in particular is specified to only examine the input until the matching character is found (ISO/IEC 9899:2011 § 7.24.5.1): *** The memchr function locates the first occurrence of c (converted to an unsigned char) in the initial n characters (each interpreted as unsigned char) of the object pointed to by s. The implementation shall behave as if it reads the characters sequentially and stops as soon as a matching character is found. *** Therefore, it appears reasonable that calls with fake buffer lengths (e.g. SIZE_MAX, to read until a mismatch occurs) must be supported. However, memcmp() has no such language and the text explicitly states that the whole buffer is compared (ISO/IEC 9899:2011 § 7.24.4.1): *** The memcmp function compares the first n characters of the object pointed to by s1 to the first n characters of the object pointed to by s2. *** By omission, this seems to give license to e.g. implement memcmp() like timingsafe_memcmp() where it inspects all n characters of both buffers and only then gives a result. So if n is longer than the actual buffer (e.g. n == SIZE_MAX), behaviour may not be defined (e.g. there could be a crash due to crossing into an unmapped page). Thus I have patched memchr() to behave correctly when length SIZE_MAX is given (commit b2618b65). My memcmp() suffers from similarly flawed logic and may need to be patched. However, as the language I cited above does not indicate that such usage needs to be supported for memcmp() (whereas it must be for memchr(), contrary to my assumptions), I was asking you for how to proceed with memcmp (hence choices (a)--(c)). > Also, all the tests that started failing with your commit, available here, > > https://ci.freebsd.org/job/FreeBSD-main-amd64-test/24136/ > > should be fixed at a bare minimum. Yes, there's a lot of other tests > that are failing: those should be fixed or disabled also. Have these > tests been fixed? They are showing up as failed still in the latest run > > https://ci.freebsd.org/job/FreeBSD-main-amd64-test/24141/ > > shows them still failing by my reading. This run crashed kyua due to an issue that seems to be an instance of bug #273481 and no final report was generated. However, it seems like the unit tests in question do not fail anymore. Let's wait for build 24142. > > Also, we're several weeks in, and we're still fixing basic, fundamental > functions that show breakage. This indicates to me that proper care > hasn't been taken to replace the critical functions in the tree. It also > indicates to me that a stronger level of rigor is needed given that the > problems are trickling in every few days to weeks. This suggests that > the unusual step of backing all this work out of stable/14 and releng/14.0 > would be indicated until we can go through this process to the point > that the new bugs have stopped (which given the decay rate seen so > far suggests we won't be done before 14.0 is released). i know this is > an enormous pain, but I think the track record so far supports backing > these out of the release we're about to do. They just arrived too late > for the normal 'cooking' process of FreeBSD to sufficiently cook them > in time for 14.0. I
Re: sed in CURRENT fails in textproc/jq
Warner Losh writes: > As for memchr, > https://pubs.opengroup.org/onlinepubs/009696799/functions/memchr.html > has no such permission to examine 'the entire buffer at once' nor any > restirction > as to the length extending beyond the address space. I'm skeptical of your > reading > that it allows one to examine all of [b, b + len), so please explain where > the standard > supports reading past the first occurance. C11 says: (7.24.4.1 §2) The memcmp function compares the first n characters of the object pointed to by s1 to the first n characters of the object pointed to by s2. and (7.24.5.1 §2) The memchr function locates the first occurrence of c (converted to an unsigned char) in the initial n characters (each interpreted as unsigned char) of the object pointed to by s. The implementation shall behave as if it reads the characters sequentially and stops as soon as a matching character is found. The second sentence in the description of memchr() means a bogus length is allowed as long as we get a match before we read past the end of the object. There is no such provision for memcmp(), so calling memcmp() with a bogus length invokes UB. DES -- Dag-Erling Smørgrav - d...@freebsd.org
Re: sed in CURRENT fails in textproc/jq
On Sun, Sep 10, 2023, 7:36 AM Robert Clausecker wrote:
> Hi Warner,
>
> I have pushed a fix. It should hopefully address those failing tests.
> The same issue should also affect memcmp(), but unlike for memchr(), it is
> illegal to pass a length to memcmp() that extends past the actual end of
> the buffer as memcmp() is permitted to examine the whole buffer regardless
> of where the first mismatch is.
>
> I am considering a change to improve the behaviour of memcmp() on such
> errorneous inputs. There are two options: (a) I could change memcmp() the
> same way I fixed memchr() and have implausible buffer lengths behave as if
> the buffer goes to the end of the address space or (b) I could change
> memcmp() to crash loudly if it detects such a case. I could also
> (c) leave memcmp() as is. Which of these three choices is preferable?
>
What does the standard say? I'm highly skeptical that these corner cases are
UB behavior.
I'd like actual support for this statement, rather than your conjecture
that it's
illegal. Even if you can come up with that, preserving the old behavior is
my
first choice. Especially since many of these functions aren't well defined
by
a standard, but are extensions.
As for memchr,
https://pubs.opengroup.org/onlinepubs/009696799/functions/memchr.html
has no such permission to examine 'the entire buffer at once' nor any
restirction
as to the length extending beyond the address space. I'm skeptical of your
reading
that it allows one to examine all of [b, b + len), so please explain where
the standard
supports reading past the first occurance.
Also, all the tests that started failing with your commit, available here,
https://ci.freebsd.org/job/FreeBSD-main-amd64-test/24136/
should be fixed at a bare minimum. Yes, there's a lot of other tests
that are failing: those should be fixed or disabled also. Have these
tests been fixed? They are showing up as failed still in the latest run
https://ci.freebsd.org/job/FreeBSD-main-amd64-test/24141/
shows them still failing by my reading.
Also, we're several weeks in, and we're still fixing basic, fundamental
functions that show breakage. This indicates to me that proper care
hasn't been taken to replace the critical functions in the tree. It also
indicates to me that a stronger level of rigor is needed given that the
problems are trickling in every few days to weeks. This suggests that
the unusual step of backing all this work out of stable/14 and releng/14.0
would be indicated until we can go through this process to the point
that the new bugs have stopped (which given the decay rate seen so
far suggests we won't be done before 14.0 is released). i know this is
an enormous pain, but I think the track record so far supports backing
these out of the release we're about to do. They just arrived too late
for the normal 'cooking' process of FreeBSD to sufficiently cook them
in time for 14.0.
Warner
Yours,
> Robert Clausecker
>
> Am Sun, Sep 10, 2023 at 05:51:43AM -0600 schrieb Warner Losh:
> > On Sat, Sep 9, 2023, 10:51 PM Robert Clausecker wrote:
> >
> > > Greetings,
> > >
> > > I apologise for the inconvenience. The issue seems to boil down to
> > > various places calling
> > >
> > > memchr(buf, c, SIZE_MAX);
> > >
> > > which causes an overflow when my newly written memchr() computes buf +
> > > len to find the end of the buffer. A patch to alleviate this issue can
> > > be found here:
> > >
> > >
> > >
> http://fuz.su/~fuz/freebsd/0001-lib-libc-amd64-string-memchr.S-fix-behaviour-with-ov.patch
> > >
> > > Please check if it does the trick for you. If yes, I'll go ahead and
> > > push it tomorrow-ish.
> > >
> >
> >
> > There are half a dozen or do kyua tests that are likely failing because
> of
> > this or other reasons related to strings. When you push this fix you'll
> > get the list. They are hidden among about 80 or so networking tests that
> > fail. I plan on disabling those tests soon If no one fixes them.
> >
> > Warner
> >
> > >
> > > Yours,
> > > Robert Clausecker
> > >
> > > Am Sat, Sep 09, 2023 at 07:12:29PM +0200 schrieb Dag-Erling Smørgrav:
> > > > Antoine Brodin writes:
> > > > > Yuri writes:
> > > > > > Either something has changed in sed(1) in CURRENT, or sed just
> fails
> > > > > > during the configure stage of textproc/jq:
> > > > > >
> > > > > > sed: No error: 0
> > > > > > checking for sys/cygwin.h... eval: ${+...}: Bad substitution
> > > > > This seems to be a recent issue (less than 5 days).
> > > > > Hundreds of configure scripts now fail to run on 15-current due to
> > > > > this sed failure: [...]
> > > >
> > > > Try adding ARCHLEVEL=scalar to CONFIGURE_ENV on one of these. If
> that
> > > > helps, yell at fuz@ :)
> > > >
> > > > DES
> > > > --
> > > > Dag-Erling Smørgrav - d...@freebsd.org
> > > >
> > >
> > > --
> > > () ascii ribbon campaign - for an 8-bit clean world
> > > /\ - against html email - against proprietary attachments
> > >
> > >
>
> --
> () ascii ribbon campaign - for an 8-bit
Re: Looks like the kyua zfs tests likely are not used on aarch64 or other contexts with unsigned char
On Sep 10, 2023, at 05:58, Mike Karels wrote:
> On 10 Sep 2023, at 2:31, Mark Millard wrote:
>
>> kyua tests that use the:
>>
>> /usr/tests/sys/cddl/zfs/bin/mkfile
>>
>> program like so (for example):
>>
>> mkfile 500M /testpool.1861/bigfile.0
>>
>> (which should be valid) end up with mkfile
>> instead reporting:
>>
>> Standard error:
>> Usage: mkfile [-nv] [e|p|t|g|m|k|b] ...
>>
>> which prevent the kyua test involved from working.
>>
>> Turns out this is from expecting char to be always
>> signed (so a -1 vs. 255 distinction, here in an
>> aarch64 context):
>>
>> . . .
>> (gdb) list
>> 179 /* Options. */
>> 180 while ((ch = getopt(argc, argv, "nv")) != -1) {
>> 181 switch (ch) {
>> 182 case 'n':
>> 183 nofill = 1;
>> 184 break;
>> 185 case 'v':
>> (gdb) print ch
>> $16 = 255 '\377'
>> (gdb) print/x -1
>> $17 = 0x
>> (gdb) print/x ch
>> $18 = 0xff
>> . . .
>>
>> With the mix of unsigned and signed it ends up
>> being a 0xffu != 0xu test, which is
>> always true.
>
> mkfile is broken. getopt returns an int, and -1 on end.
> It never returns 0xff. But mkfile declares ch as char,
> which truncates the return value -1. ch is a bad (misleading)
> variable name, although getopt(3) uses it as well (but declared
> as int).
Yep: for char being signed, the code is still wrong
via the char ch use. But the observed behavior is
very different than for char being used but being
unsigned. In this context, consequences of the
unsigned char behavioral results are observable in
the kyua run results but went unnoticed.
I used to run into examples of the use of unsigned
char for holding the getopt result back in my powerpc
days as well and dealt with upstreams for a port or 2
for getting it fixed after finding such was the source
of odd behavior I'd observed. If I remember right,
this is the first example of running into the specific
issue in my aarch64 and armv7 time frame.
> Mike
>
>> So the switch is reached as if a "-" prefix was
>> present (that is not). Then the "option" is classified
>> as invalid and the usage message is produced.
>>
>> Apparently no one had noticed. That, in turn, suggests a
>> lack of inspected testing on aarch64, powerpc64,
>> powerpc64le, armv7, powerpc, and powerpcspe. That, in
>> turn, suggests that kyua test inspection for the likes
>> of aarch64 is not historically a part of the release
>> process for openzfs or for operating systems that include
>> openzfs.
>
===
Mark Millard
marklmi at yahoo.com
Re: 14-ALPHA2 panic on cold boot
For those who interested or affected by this issue. Looks like this regression was introduced by https://reviews.freebsd.org/D34117 Currently active PR is https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268393 It contains a patch proposed by one of users. But the PR can't move forward without committers attention. --- Original Message --- On Tuesday, August 29th, 2023 at 11:09 AM, cglogic wrote: > Hello dear FreeBSD community, > > Recently I tried to boot from FreeBSD 14-ALPHA2 image and got panic on first > boot after power on. > My hardware is AMD Ryzen 9 5900X on ASUS ROG STRIX B550-E GAMING. > It's related to audio chip ALC1220 initialization. > > If I remember correctly, when I tried FreeBSD 13-STABLE, around 2 years ago, > no panic was observed. > But at that time my GPU was unsupported by drm-kmod. So I installed Linux and > used it for 2 years. > On Linux, no such a panic observed. Now drm-kmod supports my GPU and I want > move to FreeBSD. > > Also tried 13.2, got the same panic. > Can't try earlier FreeBSD 13 releases, as they are not available for download > anymore. > > I found two related bug reports: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264305 > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268393 > > My backtrace is the same as in reported issues. So I did not write it from a > screen to a paper. > > As I understand, this bug affects all AMD Zen2, Zen3 and Zen4 computers and > laptops. > > We are near FreeBSD 14.0 release now, maybe it will be possible to fix it > before release. > > Thanks.
Re: sed in CURRENT fails in textproc/jq
Hi Warner,
I have pushed a fix. It should hopefully address those failing tests.
The same issue should also affect memcmp(), but unlike for memchr(), it is
illegal to pass a length to memcmp() that extends past the actual end of
the buffer as memcmp() is permitted to examine the whole buffer regardless
of where the first mismatch is.
I am considering a change to improve the behaviour of memcmp() on such
errorneous inputs. There are two options: (a) I could change memcmp() the
same way I fixed memchr() and have implausible buffer lengths behave as if
the buffer goes to the end of the address space or (b) I could change
memcmp() to crash loudly if it detects such a case. I could also
(c) leave memcmp() as is. Which of these three choices is preferable?
Yours,
Robert Clausecker
Am Sun, Sep 10, 2023 at 05:51:43AM -0600 schrieb Warner Losh:
> On Sat, Sep 9, 2023, 10:51 PM Robert Clausecker wrote:
>
> > Greetings,
> >
> > I apologise for the inconvenience. The issue seems to boil down to
> > various places calling
> >
> > memchr(buf, c, SIZE_MAX);
> >
> > which causes an overflow when my newly written memchr() computes buf +
> > len to find the end of the buffer. A patch to alleviate this issue can
> > be found here:
> >
> >
> > http://fuz.su/~fuz/freebsd/0001-lib-libc-amd64-string-memchr.S-fix-behaviour-with-ov.patch
> >
> > Please check if it does the trick for you. If yes, I'll go ahead and
> > push it tomorrow-ish.
> >
>
>
> There are half a dozen or do kyua tests that are likely failing because of
> this or other reasons related to strings. When you push this fix you'll
> get the list. They are hidden among about 80 or so networking tests that
> fail. I plan on disabling those tests soon If no one fixes them.
>
> Warner
>
> >
> > Yours,
> > Robert Clausecker
> >
> > Am Sat, Sep 09, 2023 at 07:12:29PM +0200 schrieb Dag-Erling Smørgrav:
> > > Antoine Brodin writes:
> > > > Yuri writes:
> > > > > Either something has changed in sed(1) in CURRENT, or sed just fails
> > > > > during the configure stage of textproc/jq:
> > > > >
> > > > > sed: No error: 0
> > > > > checking for sys/cygwin.h... eval: ${+...}: Bad substitution
> > > > This seems to be a recent issue (less than 5 days).
> > > > Hundreds of configure scripts now fail to run on 15-current due to
> > > > this sed failure: [...]
> > >
> > > Try adding ARCHLEVEL=scalar to CONFIGURE_ENV on one of these. If that
> > > helps, yell at fuz@ :)
> > >
> > > DES
> > > --
> > > Dag-Erling Smørgrav - d...@freebsd.org
> > >
> >
> > --
> > () ascii ribbon campaign - for an 8-bit clean world
> > /\ - against html email - against proprietary attachments
> >
> >
--
() ascii ribbon campaign - for an 8-bit clean world
/\ - against html email - against proprietary attachments
Re: sed in CURRENT fails in textproc/jq
--- Original Message ---
On Sunday, September 10th, 2023 at 13:51, Warner Losh wrote:
>
>
> On Sat, Sep 9, 2023, 10:51 PM Robert Clausecker wrote:
>
> > Greetings,
> >
> > I apologise for the inconvenience. The issue seems to boil down to
> > various places calling
> >
> > memchr(buf, c, SIZE_MAX);
> >
> > which causes an overflow when my newly written memchr() computes buf +
> > len to find the end of the buffer. A patch to alleviate this issue can
> > be found here:
> >
> > http://fuz.su/~fuz/freebsd/0001-lib-libc-amd64-string-memchr.S-fix-behaviour-with-ov.patch
> >
> > Please check if it does the trick for you. If yes, I'll go ahead and
> > push it tomorrow-ish.
>
>
>
> There are half a dozen or do kyua tests that are likely failing because of
> this or other reasons related to strings. When you push this fix you'll get
> the list. They are hidden among about 80 or so networking tests that fail. I
> plan on disabling those tests soon If no one fixes them.
>
> Warner
>
> >
> > Yours,
> > Robert Clausecker
> >
> > Am Sat, Sep 09, 2023 at 07:12:29PM +0200 schrieb Dag-Erling Smørgrav:
> > > Antoine Brodin writes:
> > > > Yuri writes:
> > > > > Either something has changed in sed(1) in CURRENT, or sed just fails
> > > > > during the configure stage of textproc/jq:
> > > > >
> > > > > sed: No error: 0
> > > > > checking for sys/cygwin.h... eval: ${+...}: Bad substitution
> > > > This seems to be a recent issue (less than 5 days).
> > > > Hundreds of configure scripts now fail to run on 15-current due to
> > > > this sed failure: [...]
> > >
> > > Try adding ARCHLEVEL=scalar to CONFIGURE_ENV on one of these. If that
> > > helps, yell at fuz@ :)
> > >
> > > DES
> > > --
> > > Dag-Erling Smørgrav - d...@freebsd.org
> > >
> >
> > --
> > () ascii ribbon campaign - for an 8-bit clean world
> > /\ - against html email - against proprietary attachments
> >
The issue seems to also affect lang/gcc14-devel:
https://pkg-status.freebsd.org/beefy18/data/main-amd64-default/pe88937b01dd4_s3482f57f0f/logs/gcc14-devel-14.0.0.s20230903.log
Cheers,
Lorenzo Salvadore
Re: Looks like the kyua zfs tests likely are not used on aarch64 or other contexts with unsigned char
On 10 Sep 2023, at 2:31, Mark Millard wrote:
> kyua tests that use the:
>
> /usr/tests/sys/cddl/zfs/bin/mkfile
>
> program like so (for example):
>
> mkfile 500M /testpool.1861/bigfile.0
>
> (which should be valid) end up with mkfile
> instead reporting:
>
> Standard error:
> Usage: mkfile [-nv] [e|p|t|g|m|k|b] ...
>
> which prevent the kyua test involved from working.
>
> Turns out this is from expecting char to be always
> signed (so a -1 vs. 255 distinction, here in an
> aarch64 context):
>
> . . .
> (gdb) list
> 179 /* Options. */
> 180 while ((ch = getopt(argc, argv, "nv")) != -1) {
> 181 switch (ch) {
> 182 case 'n':
> 183 nofill = 1;
> 184 break;
> 185 case 'v':
> (gdb) print ch
> $16 = 255 '\377'
> (gdb) print/x -1
> $17 = 0x
> (gdb) print/x ch
> $18 = 0xff
> . . .
>
> With the mix of unsigned and signed it ends up
> being a 0xffu != 0xu test, which is
> always true.
mkfile is broken. getopt returns an int, and -1 on end.
It never returns 0xff. But mkfile declares ch as char,
which truncates the return value -1. ch is a bad (misleading)
variable name, although getopt(3) uses it as well (but declared
as int).
Mike
> So the switch is reached as if a "-" prefix was
> present (that is not). Then the "option" is classified
> as invalid and the usage message is produced.
>
> Apparently no one had noticed. That, in turn, suggests a
> lack of inspected testing on aarch64, powerpc64,
> powerpc64le, armv7, powerpc, and powerpcspe. That, in
> turn, suggests that kyua test inspection for the likes
> of aarch64 is not historically a part of the release
> process for openzfs or for operating systems that include
> openzfs.
>
>
> ===
> Mark Millard
> marklmi at yahoo.com
Re: sed in CURRENT fails in textproc/jq
On Sat, Sep 9, 2023, 10:51 PM Robert Clausecker wrote:
> Greetings,
>
> I apologise for the inconvenience. The issue seems to boil down to
> various places calling
>
> memchr(buf, c, SIZE_MAX);
>
> which causes an overflow when my newly written memchr() computes buf +
> len to find the end of the buffer. A patch to alleviate this issue can
> be found here:
>
>
> http://fuz.su/~fuz/freebsd/0001-lib-libc-amd64-string-memchr.S-fix-behaviour-with-ov.patch
>
> Please check if it does the trick for you. If yes, I'll go ahead and
> push it tomorrow-ish.
>
There are half a dozen or do kyua tests that are likely failing because of
this or other reasons related to strings. When you push this fix you'll
get the list. They are hidden among about 80 or so networking tests that
fail. I plan on disabling those tests soon If no one fixes them.
Warner
>
> Yours,
> Robert Clausecker
>
> Am Sat, Sep 09, 2023 at 07:12:29PM +0200 schrieb Dag-Erling Smørgrav:
> > Antoine Brodin writes:
> > > Yuri writes:
> > > > Either something has changed in sed(1) in CURRENT, or sed just fails
> > > > during the configure stage of textproc/jq:
> > > >
> > > > sed: No error: 0
> > > > checking for sys/cygwin.h... eval: ${+...}: Bad substitution
> > > This seems to be a recent issue (less than 5 days).
> > > Hundreds of configure scripts now fail to run on 15-current due to
> > > this sed failure: [...]
> >
> > Try adding ARCHLEVEL=scalar to CONFIGURE_ENV on one of these. If that
> > helps, yell at fuz@ :)
> >
> > DES
> > --
> > Dag-Erling Smørgrav - d...@freebsd.org
> >
>
> --
> () ascii ribbon campaign - for an 8-bit clean world
> /\ - against html email - against proprietary attachments
>
>
/var/db/etcupdate/ in hier(7) (was: user problems when upgrading to v15)
On 09/09/2023 17:02, John Baldwin wrote: On 9/2/23 7:11 AM, Dimitry Andric wrote: … … /var/db/etcupdate/conflicts, … Shall I make a simple (non-contentious) pull request for hier(7) to include /var/db/etcupdate/ and the conflicts/ subdirectory? /var/db/etcupdate/ is present by default with a clean installation of 15.0-CURRENT. I guess that conflicts/ appears (and remains after resolution) only after the first conflict is found. True?
Looks like the kyua zfs tests likely are not used on aarch64 or other contexts with unsigned char
kyua tests that use the:
/usr/tests/sys/cddl/zfs/bin/mkfile
program like so (for example):
mkfile 500M /testpool.1861/bigfile.0
(which should be valid) end up with mkfile
instead reporting:
Standard error:
Usage: mkfile [-nv] [e|p|t|g|m|k|b] ...
which prevent the kyua test involved from working.
Turns out this is from expecting char to be always
signed (so a -1 vs. 255 distinction, here in an
aarch64 context):
. . .
(gdb) list
179 /* Options. */
180 while ((ch = getopt(argc, argv, "nv")) != -1) {
181 switch (ch) {
182 case 'n':
183 nofill = 1;
184 break;
185 case 'v':
(gdb) print ch
$16 = 255 '\377'
(gdb) print/x -1
$17 = 0x
(gdb) print/x ch
$18 = 0xff
. . .
With the mix of unsigned and signed it ends up
being a 0xffu != 0xu test, which is
always true.
So the switch is reached as if a "-" prefix was
present (that is not). Then the "option" is classified
as invalid and the usage message is produced.
Apparently no one had noticed. That, in turn, suggests a
lack of inspected testing on aarch64, powerpc64,
powerpc64le, armv7, powerpc, and powerpcspe. That, in
turn, suggests that kyua test inspection for the likes
of aarch64 is not historically a part of the release
process for openzfs or for operating systems that include
openzfs.
===
Mark Millard
marklmi at yahoo.com
