Re: CVS commit: src

2020-06-04 Thread Robert Elz
Date:Fri, 5 Jun 2020 04:19:09 +0200
From:Kamil Rytarowski 
Message-ID:  <99440f2e-c0fc-5e47-4f8b-137bdf5a3...@netbsd.org>


  | I can see the problem now. It's a fault in ksh(1).

Whether this actually amounts to being called a "fault" or not is
not so clear ... most systems don't name the RT signals, that appears
to be a BSD affectation.

Even on NetBSD, when using bosh ()available in pkgsrc) the results are ...

bosh $ kill -l
HUP INT QUITILL TRAPABRTEMT FPE KILLBUS
SEGVSYS PIPEALRMTERMURG STOPTSTPCONTCHLD
TTINTTOUIO  XCPUXFSZVTALRM  PROFWINCH   INFOUSR1
USR2PWR RTMIN   RTMIN+1 RTMIN+2 RTMIN+3 RTMIN+4 RTMIN+5 RTMIN+6 RTMIN+7
RTMIN+8 RTMIN+9 RTMIN+10RTMIN+11RTMIN+12RTMIN+13   
RTMIN+14 RTMIN+15RTMAX-14RTMAX-13
RTMAX-12RTMAX-11RTMAX-10RTMAX-9 RTMAX-8 RTMAX-7 
RTMAX-6RTMAX-5  RTMAX-4 RTMAX-3
RTMAX-2 RTMAX-1 RTMAX
bosh $ 

(ignoring the funky formatting, made worse by my cut & paste).

In POSIX, just SIGRTMIN and SIGRTMAX are defined (and "kill" is
required to drop the SIG part) so bosh just invents "names" based
upon those two.

kre

ps: in modern NetBSD, you might want to try /bin/sh as your interactive
shell, it does most of what ksh can do (which is useful interactively, there's
no "select" command, but I can't imagine that being of interactive use ..
or actually of any real use).   If there's some particular impediment,
something lacking that you feel is required (something ksh can do which sh
cannot) let me know, and if it seems reasonable, it could be added.




Re: CVS commit: src

2020-06-04 Thread Kamil Rytarowski
On 05.06.2020 04:06, Robert Elz wrote:
> Date:Fri, 5 Jun 2020 01:50:47 +0200
> From:Kamil Rytarowski 
> Message-ID:  
> 
>   | What happened to RT signal names?
>   |
>   | I'm not sure what's wrong as this code works under a debugger.
> 
> No idea right now, but I will take a look.
> 
> Which shell are you using for that?   And which version.   That's
> relevant as kill is almost always a built in, so when you just
> run "kill -l" you're getting whatever the shell knows, but when you
> run it under the debugger, you're getting /bin/kill (you could also
> try running that explicitly, see if it makes a difference).
> 
> kre
> 

I can see the problem now. It's a fault in ksh(1).

Thanks for pointing it out!



signature.asc
Description: OpenPGP digital signature


Re: CVS commit: src

2020-06-04 Thread Robert Elz
Date:Fri, 5 Jun 2020 01:50:47 +0200
From:Kamil Rytarowski 
Message-ID:  

  | What happened to RT signal names?
  |
  | I'm not sure what's wrong as this code works under a debugger.

No idea right now, but I will take a look.

Which shell are you using for that?   And which version.   That's
relevant as kill is almost always a built in, so when you just
run "kill -l" you're getting whatever the shell knows, but when you
run it under the debugger, you're getting /bin/kill (you could also
try running that explicitly, see if it makes a difference).

kre



Re: CVS commit: src

2020-06-04 Thread Kamil Rytarowski
On 09.05.2017 13:14, Robert Elz wrote:
> Module Name:  src
> Committed By: kre
> Date: Tue May  9 11:14:16 UTC 2017
>
> Modified Files:
>   src/distrib/sets/lists/base: shl.mi
>   src/distrib/sets/lists/comp: mi
>   src/distrib/sets/lists/debug: shl.mi
>   src/include: signal.h
>   src/lib/libc: shlib_version
>   src/lib/libc/gen: Makefile.inc
> Added Files:
>   src/lib/libc/gen: signalname.3 signalname.c signalnext.c signalnumber.c
>
> Log Message:
> Add the new signalname/signalnext/signalnumber interface to libc.
>
> This as discussed on current-users in the thread
> entitled:
>   Proposal: new libc/libutil functions to map SIG <-> ""
> that can be found (starting at):
>   http://mail-index.netbsd.org/current-users/2017/04/28/msg031600.html
>
> These functions provide the mechanism to enable applications
> to divorce themselves from internal details of the signal
> implementation.
>
> Libc minor bumped, prototypes in , sets lists updated (and sorted).
>
> One and all: feel free to improve the sources & man page (etc), but
> please do not change the function signatures without discussion.
>

I have got a strange behavior of kill(1):

$ kill -l
HUP INT QUIT ILL TRAP ABRT EMT FPE KILL BUS SEGV SYS PIPE ALRM TERM URG
STOP TSTP CONT CHLD TTIN TTOU IO XCPU XFSZ VTALRM PROF WINCH INFO USR1
USR2 PWR ERR

$ ktruss -i -o /tmp/1.x kill -l
HUP INT QUITILL TRAPABRTEMT FPE KILL
BUS SEGVSYS PIPEALRMTERMURG STOPTSTP
CONTCHLDTTINTTOUIO  XCPUXFSZVTALRM  PROF
WINCH   INFOUSR1USR2PWR RT0 RT1 RT2 RT3
RT4 RT5 RT6 RT7 RT8 RT9 RT10RT11RT12
RT13RT14RT15RT16RT17RT18RT19RT20RT21
RT22RT23RT24RT25RT26RT27RT28RT29RT30

(gdb) r
Starting program: /bin/kill -l
HUP INT QUITILL TRAPABRTEMT FPE KILL
BUS SEGVSYS PIPEALRMTERMURG STOPTSTP
CONTCHLDTTINTTOUIO  XCPUXFSZVTALRM  PROF
WINCH   INFOUSR1USR2PWR RT0 RT1 RT2 RT3
RT4 RT5 RT6 RT7 RT8 RT9 RT10RT11RT12
RT13RT14RT15RT16RT17RT18RT19RT20RT21
RT22RT23RT24RT25RT26RT27RT28RT29RT30
[Inferior 1 (process 6257) exited normally]
(gdb)

What happened to RT signal names?

I'm not sure what's wrong as this code works under a debugger.

Sanitizers don't catch anything (at least when sanitizing kill.c).


Re: [stos, again] Re: CVS commit: src/sys/arch/amd64

2020-06-04 Thread Kamil Rytarowski
On 04.06.2020 23:41, Andrew Doran wrote:
> On Thu, Jun 04, 2020 at 02:35:17AM +0200, Kamil Rytarowski wrote:
> 
>> On 04.06.2020 00:42, Andrew Doran wrote:
>>> On Wed, Jun 03, 2020 at 02:03:22AM +0200, Kamil Rytarowski wrote:
>>>
 On 03.06.2020 01:49, Andrew Doran wrote:
> On the assembly thing recall that recently you expressed a desire to 
> remove
> all of the amd64 assembly string functions from libc because of 
> sanitizers -
> I invested my time to do up a little demo to try and show you why that's 
> not
> a good idea:
>
>   http://mail-index.netbsd.org/port-amd64/2020/04/19/msg003226.html

 Please note that interceptors for string functions are not just some
 extra burden, but also very useful approach to feedback a fuzzer through
 additional coverage.

 At least libFuzzer and honggfuzz wrap many kinds of string functions and
 use it for fuzzing. We should add a special mode in KCOV to feedback
 userland (syzkaller) with traces from string functions.

 https://github.com/google/honggfuzz/blob/bbb476eec95ad927d6d7d3d367d2b3e38eed3569/libhfuzz/memorycmp.c#L24
>>>
>>> No argument from me there at all.  I think that's a great idea and was
>>> looking at the interceptors in TSAN recently to see how they work.
>>>
>>> Andrew
>>>
>>
>> My note was not about switching away from ASM functions for certain
>> functions, but rather giving an option to disable them under a sanitizer
>> and adding an interceptor that can be useful for feedbacking a fuzzer.
>> It's still not clear whether we will create such interface in KCOV as it
>> has to be coordinated with Google+Linux as we would like to have a
>> compatible interface for syzkaller.
>>
>> TSAN - do you mean the userland ones?
> 
> Right, the userland one.
>  

Great.

We used to pass around 95% of upstream LLVM TSan tests. The remaining
ones were hard and I had no time to look into them.

>> BTW. There is a work-in-progress MKSANITIZER support for TSan, but it
>> used to create unkillable processes (kernel bug). Basically when using a
>> TSanitized userland applications, you will quickly end up with such
>> processes (from AFAIR calling ls(1) and other simple applications are
>> enough).
>>
>> If you are interested, I can share a reproducer.
> 
> Yes please.  Is the setup difficult?  I plan to look at some of the
> remaining issues noted on syzbot over the next while too.
> 

 ./build.sh -j8 -N0 -U -u -V MAKECONF=/dev/null -V MKCOMPAT=no -V
MKDEBUGLIB=yes -V MKDEBUG=yes -V MKSANITIZER=yes -V USE_SANITIZER=thread
-V MKLLVM=yes -V MKGCC=no -V HAVE_LLVM=yes -O /public/netbsd.fuzzer
distribution

Null mount /dev /dev/pts /tmp and chroot into it.

Try to run some basic applications and see creation unkillable
processes. Unless that was fixed by an accident/indirectly, you will
quickly reproduce it.

> Cheers,
> Andrew
> 



Re: [stos, again] Re: CVS commit: src/sys/arch/amd64

2020-06-04 Thread Andrew Doran
On Thu, Jun 04, 2020 at 02:35:17AM +0200, Kamil Rytarowski wrote:

> On 04.06.2020 00:42, Andrew Doran wrote:
> > On Wed, Jun 03, 2020 at 02:03:22AM +0200, Kamil Rytarowski wrote:
> > 
> >> On 03.06.2020 01:49, Andrew Doran wrote:
> >>> On the assembly thing recall that recently you expressed a desire to 
> >>> remove
> >>> all of the amd64 assembly string functions from libc because of 
> >>> sanitizers -
> >>> I invested my time to do up a little demo to try and show you why that's 
> >>> not
> >>> a good idea:
> >>>
> >>>   http://mail-index.netbsd.org/port-amd64/2020/04/19/msg003226.html
> >>
> >> Please note that interceptors for string functions are not just some
> >> extra burden, but also very useful approach to feedback a fuzzer through
> >> additional coverage.
> >>
> >> At least libFuzzer and honggfuzz wrap many kinds of string functions and
> >> use it for fuzzing. We should add a special mode in KCOV to feedback
> >> userland (syzkaller) with traces from string functions.
> >>
> >> https://github.com/google/honggfuzz/blob/bbb476eec95ad927d6d7d3d367d2b3e38eed3569/libhfuzz/memorycmp.c#L24
> > 
> > No argument from me there at all.  I think that's a great idea and was
> > looking at the interceptors in TSAN recently to see how they work.
> > 
> > Andrew
> > 
> 
> My note was not about switching away from ASM functions for certain
> functions, but rather giving an option to disable them under a sanitizer
> and adding an interceptor that can be useful for feedbacking a fuzzer.
> It's still not clear whether we will create such interface in KCOV as it
> has to be coordinated with Google+Linux as we would like to have a
> compatible interface for syzkaller.
> 
> TSAN - do you mean the userland ones?

Right, the userland one.
 
> BTW. There is a work-in-progress MKSANITIZER support for TSan, but it
> used to create unkillable processes (kernel bug). Basically when using a
> TSanitized userland applications, you will quickly end up with such
> processes (from AFAIR calling ls(1) and other simple applications are
> enough).
> 
> If you are interested, I can share a reproducer.

Yes please.  Is the setup difficult?  I plan to look at some of the
remaining issues noted on syzbot over the next while too.

Cheers,
Andrew


Re: CVS commit: src/lib/libpthread

2020-06-04 Thread Andrew Doran
Hi,

On Thu, Jun 04, 2020 at 06:26:07PM +, Martin Husemann wrote:

> On Wed, Jun 03, 2020 at 10:10:24PM +, Andrew Doran wrote:
> > Module Name:src
> > Committed By:   ad
> > Date:   Wed Jun  3 22:10:24 UTC 2020
> > 
> > Modified Files:
> > src/lib/libpthread: pthread.c pthread_cond.c pthread_mutex.c
> > 
> > Log Message:
> > Deal with a couple of problems with threads being awoken early due to
> > timeouts or cancellation where:
> 
> Not sure if it is caused by this commit or joergs TSD/malloc, but today
> most of the libpthread tests time out on aarch64, while everything but
> a few minor nits was fine on May 30.

I'll try to reproduce it over the weekend.

Andrew
 
> Martin
> 
> (this is on a hummingboard pulse 4-core board)
> lib/libpthread/t_barrier (398/848): 1 test cases
> barrier: [300.034098s] Failed: Test case timed out after 300 seconds
> [300.034568s]
> 
> lib/libpthread/t_cond (399/848): 9 test cases
> bogus_timedwaits: [300.016677s] Failed: Test case timed out after 300 
> seconds
> broadcast: [300.023040s] Failed: Test case timed out after 300 seconds
> cond_timedwait_race: [300.034567s] Failed: Test case timed out after 300 
> seconds
> condattr: [0.013756s] Passed.
> destroy_after_cancel: [300.025649s] Failed: Test case timed out after 300 
> seconds
> signal_before_unlock: [300.023630s] Failed: Test case timed out after 300 
> seconds
> signal_before_unlock_static_init: [300.022909s] Failed: Test case timed 
> out after 300 seconds
> signal_delay_wait: [300.031310s] Failed: Test case timed out after 300 
> seconds
> signal_wait_race: [300.025552s] Failed: Test case timed out after 300 
> seconds
> [2400.221956s]
> 
> lib/libpthread/t_condwait (400/848): 2 test cases
> cond_wait_mono: [2.023083s] Passed.
> cond_wait_real: [2.017972s] Passed.
> [4.042078s]
> 
> lib/libpthread/t_detach (401/848): 1 test cases
> pthread_detach: [300.021242s] Failed: Test case timed out after 300 
> seconds
> [300.021708s]
> 
> lib/libpthread/t_equal (402/848): 1 test cases
> pthread_equal: [300.027646s] Failed: Test case timed out after 300 seconds
> [300.028295s]
> 
> lib/libpthread/t_fork (403/848): 1 test cases
> fork: [300.018077s] Failed: Test case timed out after 300 seconds
> [300.018975s]
> 
> lib/libpthread/t_fpu (404/848): 1 test cases
> fpu: [0.020578s] Passed.
> [0.021314s]
> 
> lib/libpthread/t_join (405/848): 1 test cases
> pthread_join: [0.021632s] Passed.
> [0.022411s]
> 
> lib/libpthread/t_kill (406/848): 1 test cases
> simple: [300.033759s] Failed: Test case timed out after 300 seconds
> [300.034528s]
> 
> lib/libpthread/t_mutex (407/848): 7 test cases
> mutex1: [300.017542s] Failed: Test case timed out after 300 seconds
> mutex2: [300.023064s] Failed: Test case timed out after 300 seconds
> mutex3: [300.023259s] Failed: Test case timed out after 300 seconds
> mutex4: [300.023015s] Failed: Test case timed out after 300 seconds
> mutex5: [300.023495s] Failed: Test case timed out after 300 seconds
> mutexattr1: [0.013769s] Passed.
> mutexattr2: [0.025483s] Passed.
> [1500.154160s]
> 
> lib/libpthread/t_name (408/848): 1 test cases
> name: [0.014262s] Passed.
> [0.014951s]
> 
> ...


Re: CVS commit: src/lib/libpthread

2020-06-04 Thread Martin Husemann
On Wed, Jun 03, 2020 at 10:10:24PM +, Andrew Doran wrote:
> Module Name:  src
> Committed By: ad
> Date: Wed Jun  3 22:10:24 UTC 2020
> 
> Modified Files:
>   src/lib/libpthread: pthread.c pthread_cond.c pthread_mutex.c
> 
> Log Message:
> Deal with a couple of problems with threads being awoken early due to
> timeouts or cancellation where:

Not sure if it is caused by this commit or joergs TSD/malloc, but today
most of the libpthread tests time out on aarch64, while everything but
a few minor nits was fine on May 30.

Martin

(this is on a hummingboard pulse 4-core board)
lib/libpthread/t_barrier (398/848): 1 test cases
barrier: [300.034098s] Failed: Test case timed out after 300 seconds
[300.034568s]

lib/libpthread/t_cond (399/848): 9 test cases
bogus_timedwaits: [300.016677s] Failed: Test case timed out after 300 
seconds
broadcast: [300.023040s] Failed: Test case timed out after 300 seconds
cond_timedwait_race: [300.034567s] Failed: Test case timed out after 300 
seconds
condattr: [0.013756s] Passed.
destroy_after_cancel: [300.025649s] Failed: Test case timed out after 300 
seconds
signal_before_unlock: [300.023630s] Failed: Test case timed out after 300 
seconds
signal_before_unlock_static_init: [300.022909s] Failed: Test case timed out 
after 300 seconds
signal_delay_wait: [300.031310s] Failed: Test case timed out after 300 
seconds
signal_wait_race: [300.025552s] Failed: Test case timed out after 300 
seconds
[2400.221956s]

lib/libpthread/t_condwait (400/848): 2 test cases
cond_wait_mono: [2.023083s] Passed.
cond_wait_real: [2.017972s] Passed.
[4.042078s]

lib/libpthread/t_detach (401/848): 1 test cases
pthread_detach: [300.021242s] Failed: Test case timed out after 300 seconds
[300.021708s]

lib/libpthread/t_equal (402/848): 1 test cases
pthread_equal: [300.027646s] Failed: Test case timed out after 300 seconds
[300.028295s]

lib/libpthread/t_fork (403/848): 1 test cases
fork: [300.018077s] Failed: Test case timed out after 300 seconds
[300.018975s]

lib/libpthread/t_fpu (404/848): 1 test cases
fpu: [0.020578s] Passed.
[0.021314s]

lib/libpthread/t_join (405/848): 1 test cases
pthread_join: [0.021632s] Passed.
[0.022411s]

lib/libpthread/t_kill (406/848): 1 test cases
simple: [300.033759s] Failed: Test case timed out after 300 seconds
[300.034528s]

lib/libpthread/t_mutex (407/848): 7 test cases
mutex1: [300.017542s] Failed: Test case timed out after 300 seconds
mutex2: [300.023064s] Failed: Test case timed out after 300 seconds
mutex3: [300.023259s] Failed: Test case timed out after 300 seconds
mutex4: [300.023015s] Failed: Test case timed out after 300 seconds
mutex5: [300.023495s] Failed: Test case timed out after 300 seconds
mutexattr1: [0.013769s] Passed.
mutexattr2: [0.025483s] Passed.
[1500.154160s]

lib/libpthread/t_name (408/848): 1 test cases
name: [0.014262s] Passed.
[0.014951s]

...