Re: 10-STABLE hangups frequently

2016-02-04 Thread Peter Jeremy 
On 2016-Feb-04 11:45:56 +1030, Shane Ambler  wrote:
>Going by figures shown in top, ARC is usually in the 1500M to 2000M
>range but when wired gets over 6GB I often see ARC drop to 500MB which
>I now realise matches arc_min.

That's definitely abnormal.  You might like to run "vmstat -mz" when the
system is running normally and as the non-ARC wired memory increases to
identify where the RAM is going.

-- 
Peter Jeremy


signature.asc
Description: PGP signature

Re: 10-STABLE hangups frequently

2016-02-04 Thread Peter Jeremy 
On 2016-Feb-02 14:52:37 +0200, Konstantin Belousov  wrote:
>Please gather the information listed at
>https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html

Interestingly, as soon as I enable INVARIANTS, my kernel will no longer boot.
All the other options listed there are OK.

The console output ends:
TSC timecounter discards lower 1 bit(s)
Timecounter "TSC-low" frequency 1150026690 Hz quality 800
hwpc_core: unknown PMC architecture: 0
hwpmc: SOFT/16/64/0x67
init_KVMCLOCK_tc: 0x000b
KVM-style paravirtualized clock detected.
Timecounter "KVMCLOCK" frequency 10 Hz quality 1000
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Trying to mount root from zfs:zroot []...
<>

Unfortunately, I don't have write access to the console so I can't
do anything other than reboot at this point.

(This is 10-stable/amd64 r295088).

If I have some spare time, I'll try reproducing this in a local VBox over
the next few days.

-- 
Peter Jeremy


signature.asc
Description: PGP signature

Re: 82576 + NETMAP + VLAN

2016-02-04 Thread Slawa Olhovchenkov 
On Tue, Feb 02, 2016 at 11:44:47PM +0300, Slawa Olhovchenkov wrote:

> On Thu, Oct 22, 2015 at 11:24:53AM -0700, Luigi Rizzo wrote:
> 
> > On Thu, Oct 22, 2015 at 11:12 AM, Adrian Chadd  
> > wrote:
> > > On 22 October 2015 at 09:35, Slawa Olhovchenkov  wrote:
> > >> On Sun, Oct 18, 2015 at 07:45:52PM -0700, Adrian Chadd wrote:
> > >>
> > >>> Heh, file a bug with luigi; it should be defined better inside netmap 
> > >>> itself.
> > >>
> > >> I am CC: luigi.
> > >>
> > >> Next question: do kevent RX/TX sync?
> > >> In my setup I am need to manual NIOCTXSYNC/NIOCRXSYNC.
> > >
> > > Hi,
> > >
> > > Nope. kqueue() doesn't do the implicit sync like poll() does; it's
> > > just the notification path.
> > 
> > actually not. When the file descriptor is registered there
> > is an implicit sync, and there is another one when an event
> > is posted for the file descriptor.
> > 
> > unless there are bugs, of course.
> 
> I found strange behaivor:
> 
> 1. open netmap and register in main thread
> 2. kevent register in different thread
> 3. result: got event by kevent but no ring sinc (all head,tail,cur
> still 0).
> 
> Is this normal? Or is this bug?
> 
> open and registering netmap in same thread as kevent resolve this.

Also, kevent+netmap deadlocked for me:

  PIDTID COMM TDNAME   KSTACK   
 1095 100207 addos-mi_switch+0xe1 
sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 
kern_nanosleep+0x10e sys_nanosleep+0x51 amd64_syscall+0x40f Xfast_syscall+0xfb 
 1095 100208 addosworker#0 mi_switch+0xe1 
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_kevent+0x401 
sys_kevent+0x12a amd64_syscall+0x40f Xfast_syscall+0xfb 
 1095 100209 addosworker#1 mi_switch+0xe1 
turnstile_wait+0x42a __mtx_lock_sleep+0x26b knote+0x38 freebsd_selwakeup+0x8b 
netmap_notify+0x55 netmap_pipe_txsync+0x156 netmap_poll+0x400 netmap_knrw+0x6e 
kqueue_register+0x799 kern_kevent+0x158 sys_kevent+0x12a amd64_syscall+0x40f 
Xfast_syscall+0xfb 
 1095 100210 addosworker#2 mi_switch+0xe1 
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_kevent+0x401 
sys_kevent+0x12a amd64_syscall+0x40f Xfast_syscall+0xfb 
 1095 100211 addosworker#NOIP  mi_switch+0xe1 
sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_kevent+0x401 
sys_kevent+0x12a amd64_syscall+0x40f Xfast_syscall+0xfb 
 1095 100212 addosbalancer mi_switch+0xe1 
turnstile_wait+0x42a __mtx_lock_sleep+0x26b knote+0x38 freebsd_selwakeup+0x8b 
netmap_notify+0x2a netmap_pipe_rxsync+0x54 netmap_poll+0x774 netmap_knrw+0x6e 
kern_kevent+0x5cc sys_kevent+0x12a amd64_syscall+0x40f Xfast_syscall+0xfb 
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 10.2-RELEASE-p12 pf+GRE crashing

2016-02-04 Thread Matthew Grooms 

On 2/3/2016 6:47 PM, Matthew Grooms wrote:
This turned out to be another issue that was patched in head but not 
back ported to stable. I can't explain why it didn't get tripped when 
GRE tunnels were disabled. With the patch applied, I can reload my 
rule sets again without crashing ...


https://svnweb.freebsd.org/base?view=revision=264689



I wanted to clarify in case another user runs into this issue and 
searches the mailing list history for a solution: The patch I applied to 
fix this particular kernel crash wasn't 264689, it was ...


https://svnweb.freebsd.org/base?view=revision=264915

Sorry for the misinformation. I cut and pasted the wrong link.

-Matthew


(kgdb) bt
#0  doadump (textdump=) at pcpu.h:219
#1  0x807c81f2 in kern_reboot (howto=260) at 
../../../kern/kern_shutdown.c:451
#2  0x807c85d5 in vpanic (fmt=, ap=optimized out>)

at ../../../kern/kern_shutdown.c:758
#3  0x807c8463 in panic (fmt=0x0) at 
../../../kern/kern_shutdown.c:687

#4  0x80bdc10b in trap_fatal (frame=,
eva=) at ../../../amd64/amd64/trap.c:851
#5  0x80bdc40d in trap_pfault (frame=0xfe233a80,
usermode=) at ../../../amd64/amd64/trap.c:674
#6  0x80bdbaaa in trap (frame=0xfe233a80)
at ../../../amd64/amd64/trap.c:440
#7  0x80bc1fa2 in calltrap () at 
../../../amd64/amd64/exception.S:236
#8  0x809c07f4 in pfr_detach_table (kt=0x0) at 
../../../netpfil/pf/pf_table.c:2047

#9  0x809a91f4 in pf_empty_pool (poola=0x813c3d68)
at ../../../netpfil/pf/pf_ioctl.c:354
#10 0x809ab3e5 in pfioctl (dev=, 
cmd=,
addr=0xf8005eaf6800 "", flags=, td=optimized out>)

at ../../../netpfil/pf/pf_ioctl.c:2189
#11 0x806b5659 in devfs_ioctl_f (fp=0xf8000a2927d0, 
com=3295691827,
data=0xf8005eaf6800, cred=, 
td=0xf8000a25f000)

at ../../../fs/devfs/devfs_vnops.c:785
#12 0x8081b805 in kern_ioctl (td=0xf8000a25f000, fd=optimized out>,

com=2) at file.h:320
#13 0x8081b500 in sys_ioctl (td=0xf8000a25f000, 
uap=0xfe234b40)

at ../../../kern/sys_generic.c:718
#14 0x80bdca27 in amd64_syscall (td=0xf8000a25f000, traced=0)
at subr_syscall.c:134
#15 0x80bc228b in Xfast_syscall () at 
../../../amd64/amd64/exception.S:396

#16 0x000800dd9fda in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal

-Matthew
___
freebsd-...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Recent -STABLE crashes out of swap at 3am every day

2016-02-04 Thread Pete French 
I recently updates a system to -STABLE, which ahdnt been updates in
9 months or so. It started locking up at 3am every day. I have
updated to -STABLE form this mornign and verified that this still happens.

The porobel  is the 'periodic daily' process. I have veriied this
by running it by hand. What happens is that Swap space
starts to fill up. Wathcing in top, however, there are no large
processes running.

I also start to see these messages during the run...

Feb  4 14:57:07 toybox kernel: sonewconn: pcb 0xf8000eee6000: Listen queue 
overflow: 31 already in queue awaiting acceptance (1 occurrences)
Feb  4 14:58:07 toybox kernel: sonewconn: pcb 0xf8000eee6000: Listen queue 
overflow: 31 already in queue awaiting acceptance (26 occurrences)
Feb  4 14:59:07 toybox kernel: sonewconn: pcb 0xf8000eee6000: Listen queue 
overflow: 31 already in queue awaiting acceptance (193 occurrences)

The systems is mainly ZFS, booting from a small UFS partition for /.
It has 2GB of RAM only, but the ARC is limited to 512 meg. It
primarttily runs exim/clamav/spamd/mailman - these are failmy
memory hungry I realise, and the machine usually runs with
a few hundred meg swapped out, but iit has been very reliable until
the most recent upgrade.

I dont knwo preciusely what the periodc scripts do, but I underrstand
the use a substantial amount of ;find' and it is this pricess which
shows up when runing, usually in the state zio->io_ which is what I
would expect. The find process is not using a lot of memory, however.

I dont see this on any other achines runnign the most recemt -STABLE, but
then again this is my only machine with so little memory in it.

What can I do to tracve anmd fix this ? My obvious intiail move
is to disable the daily periopdic run, but thats hardly a solution!

help?

-pete.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 82576 + NETMAP + VLAN

2016-02-04 Thread Adrian Chadd 
I've no time to help with this, I'm sorry :(


-a


On 4 February 2016 at 05:00, Slawa Olhovchenkov  wrote:
> On Tue, Feb 02, 2016 at 11:44:47PM +0300, Slawa Olhovchenkov wrote:
>
>> On Thu, Oct 22, 2015 at 11:24:53AM -0700, Luigi Rizzo wrote:
>>
>> > On Thu, Oct 22, 2015 at 11:12 AM, Adrian Chadd  
>> > wrote:
>> > > On 22 October 2015 at 09:35, Slawa Olhovchenkov  wrote:
>> > >> On Sun, Oct 18, 2015 at 07:45:52PM -0700, Adrian Chadd wrote:
>> > >>
>> > >>> Heh, file a bug with luigi; it should be defined better inside netmap 
>> > >>> itself.
>> > >>
>> > >> I am CC: luigi.
>> > >>
>> > >> Next question: do kevent RX/TX sync?
>> > >> In my setup I am need to manual NIOCTXSYNC/NIOCRXSYNC.
>> > >
>> > > Hi,
>> > >
>> > > Nope. kqueue() doesn't do the implicit sync like poll() does; it's
>> > > just the notification path.
>> >
>> > actually not. When the file descriptor is registered there
>> > is an implicit sync, and there is another one when an event
>> > is posted for the file descriptor.
>> >
>> > unless there are bugs, of course.
>>
>> I found strange behaivor:
>>
>> 1. open netmap and register in main thread
>> 2. kevent register in different thread
>> 3. result: got event by kevent but no ring sinc (all head,tail,cur
>> still 0).
>>
>> Is this normal? Or is this bug?
>>
>> open and registering netmap in same thread as kevent resolve this.
>
> Also, kevent+netmap deadlocked for me:
>
>   PIDTID COMM TDNAME   KSTACK
>  1095 100207 addos-mi_switch+0xe1 
> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 
> kern_nanosleep+0x10e sys_nanosleep+0x51 amd64_syscall+0x40f Xfast_syscall+0xfb
>  1095 100208 addosworker#0 mi_switch+0xe1 
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_kevent+0x401 
> sys_kevent+0x12a amd64_syscall+0x40f Xfast_syscall+0xfb
>  1095 100209 addosworker#1 mi_switch+0xe1 
> turnstile_wait+0x42a __mtx_lock_sleep+0x26b knote+0x38 freebsd_selwakeup+0x8b 
> netmap_notify+0x55 netmap_pipe_txsync+0x156 netmap_poll+0x400 
> netmap_knrw+0x6e kqueue_register+0x799 kern_kevent+0x158 sys_kevent+0x12a 
> amd64_syscall+0x40f Xfast_syscall+0xfb
>  1095 100210 addosworker#2 mi_switch+0xe1 
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_kevent+0x401 
> sys_kevent+0x12a amd64_syscall+0x40f Xfast_syscall+0xfb
>  1095 100211 addosworker#NOIP  mi_switch+0xe1 
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_kevent+0x401 
> sys_kevent+0x12a amd64_syscall+0x40f Xfast_syscall+0xfb
>  1095 100212 addosbalancer mi_switch+0xe1 
> turnstile_wait+0x42a __mtx_lock_sleep+0x26b knote+0x38 freebsd_selwakeup+0x8b 
> netmap_notify+0x2a netmap_pipe_rxsync+0x54 netmap_poll+0x774 netmap_knrw+0x6e 
> kern_kevent+0x5cc sys_kevent+0x12a amd64_syscall+0x40f Xfast_syscall+0xfb
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 82576 + NETMAP + VLAN

2016-02-04 Thread Adrian Chadd 
.. but if it does, can you enable witness and see what it reports as
lock order violations?


-a


On 4 February 2016 at 10:47, Adrian Chadd  wrote:
> I've no time to help with this, I'm sorry :(
>
>
> -a
>
>
> On 4 February 2016 at 05:00, Slawa Olhovchenkov  wrote:
>> On Tue, Feb 02, 2016 at 11:44:47PM +0300, Slawa Olhovchenkov wrote:
>>
>>> On Thu, Oct 22, 2015 at 11:24:53AM -0700, Luigi Rizzo wrote:
>>>
>>> > On Thu, Oct 22, 2015 at 11:12 AM, Adrian Chadd  
>>> > wrote:
>>> > > On 22 October 2015 at 09:35, Slawa Olhovchenkov  wrote:
>>> > >> On Sun, Oct 18, 2015 at 07:45:52PM -0700, Adrian Chadd wrote:
>>> > >>
>>> > >>> Heh, file a bug with luigi; it should be defined better inside netmap 
>>> > >>> itself.
>>> > >>
>>> > >> I am CC: luigi.
>>> > >>
>>> > >> Next question: do kevent RX/TX sync?
>>> > >> In my setup I am need to manual NIOCTXSYNC/NIOCRXSYNC.
>>> > >
>>> > > Hi,
>>> > >
>>> > > Nope. kqueue() doesn't do the implicit sync like poll() does; it's
>>> > > just the notification path.
>>> >
>>> > actually not. When the file descriptor is registered there
>>> > is an implicit sync, and there is another one when an event
>>> > is posted for the file descriptor.
>>> >
>>> > unless there are bugs, of course.
>>>
>>> I found strange behaivor:
>>>
>>> 1. open netmap and register in main thread
>>> 2. kevent register in different thread
>>> 3. result: got event by kevent but no ring sinc (all head,tail,cur
>>> still 0).
>>>
>>> Is this normal? Or is this bug?
>>>
>>> open and registering netmap in same thread as kevent resolve this.
>>
>> Also, kevent+netmap deadlocked for me:
>>
>>   PIDTID COMM TDNAME   KSTACK
>>  1095 100207 addos-mi_switch+0xe1 
>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 
>> kern_nanosleep+0x10e sys_nanosleep+0x51 amd64_syscall+0x40f 
>> Xfast_syscall+0xfb
>>  1095 100208 addosworker#0 mi_switch+0xe1 
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_kevent+0x401 
>> sys_kevent+0x12a amd64_syscall+0x40f Xfast_syscall+0xfb
>>  1095 100209 addosworker#1 mi_switch+0xe1 
>> turnstile_wait+0x42a __mtx_lock_sleep+0x26b knote+0x38 
>> freebsd_selwakeup+0x8b netmap_notify+0x55 netmap_pipe_txsync+0x156 
>> netmap_poll+0x400 netmap_knrw+0x6e kqueue_register+0x799 kern_kevent+0x158 
>> sys_kevent+0x12a amd64_syscall+0x40f Xfast_syscall+0xfb
>>  1095 100210 addosworker#2 mi_switch+0xe1 
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_kevent+0x401 
>> sys_kevent+0x12a amd64_syscall+0x40f Xfast_syscall+0xfb
>>  1095 100211 addosworker#NOIP  mi_switch+0xe1 
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_kevent+0x401 
>> sys_kevent+0x12a amd64_syscall+0x40f Xfast_syscall+0xfb
>>  1095 100212 addosbalancer mi_switch+0xe1 
>> turnstile_wait+0x42a __mtx_lock_sleep+0x26b knote+0x38 
>> freebsd_selwakeup+0x8b netmap_notify+0x2a netmap_pipe_rxsync+0x54 
>> netmap_poll+0x774 netmap_knrw+0x6e kern_kevent+0x5cc sys_kevent+0x12a 
>> amd64_syscall+0x40f Xfast_syscall+0xfb
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 10-STABLE hangups frequently

2016-02-04 Thread Thierry Thomas 
Le mer  3 fév 16 à 17:55:03 +0100, Karl Denninger 
 écrivait :

> (Whistling.)
> 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594
> 
> (see if that helps you)

Thanks, I'll check it!
-- 
Th. Thomas.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"