Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-09-12 Thread Stefan Unterweger
With a combination of tight RAM (to provoke swapping), a heavy Synth job and an obscenely large file passing through a pipeline with almost a dozen ‚xz‘ processes in it I sometimes manage to trigger the flash reboots. This time I caught something in the log, though: | Sep 12 22:59:00 sumi

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-09-12 Thread Stefan Unterweger
Hi! I haven‘t seen your post until now. I have finally managed to set up a test machine and am currently throwing as many heavy jobs at it that I can think of. In general it feels more stable, but I still get crashes. The crashes are different though. In the last run, the machine just

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-09-05 Thread Matthew Dillon
A fix for at least one indefinite wait buffer bug has gone into master.(commit 10c39de26c1356d0). It has not been put into the release branch yet as it needs testing. It's possible that this is related to the bug you reported because the bug can occur when the system is paging to/from swap while

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-08-08 Thread Matthew Dillon
Its likely another virtio bug. We just haven't been able to test it very well. I'll take a pass on it and see if I can find anything obvious. -Matt On Mon, Aug 8, 2016 at 12:02 PM, Stefan Unterweger < 232.20...@chiffre.aleturo.com> wrote: > > * Matthew Dillon on Mon, Aug 08, 2016 at

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-08-08 Thread Matthew Dillon
Indefinite wait buffer typically means the disk driver locked up while the system was trying to swap to/from it. Two questions: (1) Is the swap encrypted ? And (2) What is the device attachment for the swap drive (in the dmesg output). -Matt On Mon, Aug 8, 2016 at 5:18 AM, Stefan Unterweger <

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-08-06 Thread Stefan Unterweger
I stalled the update a bit because I wanted to make sure that the boot problems were resolved first (our customers are already edgy; another two-hour downtime just for booting would have been too much). Given that 4.6 is now out, I’ve decided to take the plunge and do it. Booting went without a

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-07-26 Thread Matthew Dillon
On this issue, we've made a bunch of fixes in just the last few days, you will definitely want to update to the latest master. Things are looking a lot more stable now. We are continuing to life test three server configurations this week to locate any remaining low-hanging fruit. -Matt On Mon,

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-07-18 Thread Stefan Unterweger
* Matthew Dillon on Sun, Jul 17, 2016 at 10:21:04AM -0700: > The smp_invltlb() issue should hopefully be fixed in the latest master but > the code has not been well-tested yet. > > The cause of the original issue you reported is not entirely known (its an > issue with one of the other cpu cores,

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-07-17 Thread Matthew Dillon
The smp_invltlb() issue should hopefully be fixed in the latest master but the code has not been well-tested yet. The cause of the original issue you reported is not entirely known (its an issue with one of the other cpu cores, not the one that generated the message and backtrace), but it is

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-07-15 Thread Stefan Unterweger
* Stefan Unterweger on Thu, Jul 07, 2016 at 09:31:47AM +0200: > > >> Virtio (for block storage devices) could be the cause. There are known > > >> bugs in the DragonFly driver for virtio which haven't been tracked down > > >> yet > > >> (not enough of the devs are using virtual hosting to be

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-07-07 Thread Stefan Unterweger
* Sepherosa Ziehau on Wed, Jul 06, 2016 at 09:22:56AM +0800: > > * Matthew Dillon on Fri, May 27, 2016 at 10:08:06AM -0700: > >> Virtio (for block storage devices) could be the cause. There are known > >> bugs in the DragonFly driver for virtio which haven't been tracked down yet > >> (not enough

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-07-05 Thread Stefan Unterweger
Sorry, I haven’t been paying attention, this was in response to the thread with the virtio problem. Yeah, I have been updating precisely because of this; during the update, it seemed rather stable, even though it had a lot do do, with a few ports compiling, hammer cleanup running and something

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-07-05 Thread Stefan Unterweger
I’ve compiled it two days ago; was the fix in between there? I’ll try to hunt down another real machine so that I can somehow at least inject the kernel onto the server; right now, I can’t get it to boot up no matter what I do, im past attempt number twenty and feel like a lunatic for keeping at

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-07-05 Thread Sepherosa Ziehau
On Wed, Jul 6, 2016 at 6:17 AM, Stefan Unterweger <232.20...@chiffre.aleturo.com> wrote: > Sorry for the long delay. Other, more urgent projects kept me so busy > that I didn’t manage to catch up with this thread. > > > * Matthew Dillon on Fri, May 27, 2016 at 10:08:06AM -0700: >> Virtio (for

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-07-05 Thread Stefan Unterweger
Sorry for the long delay. Other, more urgent projects kept me so busy that I didn’t manage to catch up with this thread. * Matthew Dillon on Fri, May 27, 2016 at 10:08:06AM -0700: > Virtio (for block storage devices) could be the cause. There are known > bugs in the DragonFly driver for virtio

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-05-28 Thread Michael Neumann
lybsd.org <mailto:users@dragonflybsd.org>> Sent: Friday, May 27, 2016 3:38 AM Subject: Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent) * Matthew Dillon on Thu, May 26, 2016 at 11:00:18AM -0700:

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-05-27 Thread Matthew Dillon
>> >> Steve >> >> - Original Message - From: "Stefan Unterweger" < >> 232.20...@chiffre.aleturo.com> >> To: "Matthew Dillon" <dil...@backplane.com> >> Cc: <users@dragonflybsd.org> >> Sent: Friday, May 27, 2016 3:38 AM >

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-05-27 Thread Matthew Dillon
ewbie. > > Please see my commens inline below. > > Steve > > - Original Message - From: "Stefan Unterweger" < > 232.20...@chiffre.aleturo.com> > To: "Matthew Dillon" <dil...@backplane.com> > Cc: <users@dragonflybsd.org> >

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-05-27 Thread Steve Petrie, P.Eng.
- From: "Stefan Unterweger" <232.20...@chiffre.aleturo.com> To: "Matthew Dillon" <dil...@backplane.com> Cc: <users@dragonflybsd.org> Sent: Friday, May 27, 2016 3:38 AM Subject: Re: Random server crashes every few weeks (smp_invltlb: endless loop […] ret

Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

2016-05-27 Thread Stefan Unterweger
* Matthew Dillon on Thu, May 26, 2016 at 11:00:18AM -0700: > It's really hard to say from something which is virtually hosted. It kinda > sounds like the virtual host isn't assigning enough of its own cpus to the > virtual host. The fact that DragonFly is complaining about smp_invltlb() >