Re: [pfSense] pfSense 2.4 consistently crashes daily

[Liwei]

Ugh, I did not see the other patch that was in the FreeBSD tracker!

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220611

It appears that pfSense used a similar patch
https://github.com/pfsense/FreeBSD-src/commit/4204c9f01d2ab439f6e0b9454ab22d4ffcca8cc4

But it is not included in 2.4!

https://github.com/pfsense/FreeBSD-src/blob/RELENG_2_4/sys/netpfil/pf/pf_norm.c#L791

Was this removed intentionally? If not, seems like a good reason to open a bug

On 3 December 2017 at 02:27, Liwei  wrote:
> tl;dr: How do I figure out what the state of the ip6_forward (or other
> associated functions) are after a crash? It is really a pain trying to
> figure this out based on traffic and trying to replicate the correct
> set of pfSense configurations.
>
>
> Just an update on my adventures. It's quite complicated so I shall put
> it as a list of points...
>
> Part 1
> 1. Spent a weekend copying the pfSense VM to a physical machine and
> running it in place of the VM.
>
> 2. Eventually the crashes do occur, so I'm entirely convinced this is
> not a VM issue
>
> 3. Over the last week, I've been disabling various interfaces and
> services to narrow down the cause, finally it has been narrowed down
> to our VPN bridge interface 
>
> 4. If I bring any of the VPN interfaces down/take any of the VPN
> interfaces out of the bridge, or bring the bridge interface down, the
> crashes stop
>
>
> Part 2
> 1. I managed to dig up an old (closed) bug with the exact backtrace I
> was getting https://redmine.pfsense.org/issues/5428
>
> 2. There are two more bugs on the FreeBSD bugtracker with similar
> crashes, the latest was in July this year
>
> 3. However, they've all been fixed with the same patch. Checked
> RELENG_2_4/sys/netpfil/pf/pf.c , and the patch should be in the 2.4.X
> release
>
> a. The network is running with jumbo packets (9k), is it possible
> the patch does not cover such a case?
>
> 4. This similarity led me to believe that I could be facing a similar
> issue, apparently with IPv6 multicast traffic
>
> 5. Set up port remote tcpdump so I could capture traffic right before the 
> crash
>
> 6. Isolated the traffic cause! Two conditions happening causes the crash:
>
> a. There is at least one VPN client connected
> b. There is a macbook running Sierra/High Sierra on the main network
>
> 7. Each time the macbook joins the network/sleeps/wakes, the V6
> traffic, specifically a certain MDNS query, causes the crash
>
> 8. Now the somewhat random but consistent timing makes sense! We have
> someone using a macbook come in at around 8pm every day
>
> 9. Isolated 2 packet specimens that causes the crash, and 2 of the
> same type that does not
>
> a. It does contain names of our users' computers (which on a mac
> contains real names), so I'm not inclined to share them on the list,
> furthermore, I don't have steps to reproduce the crash with the
> packets from a vanilla install, so they're of limited use
> b. If anyone is interested to take a look at what the differences
> between these two sets of packets are, I can email them to you
> directly
>
> Part 3
> 1. Since I could cause the crash at will, I tried creating
> reproducible steps so I can properly report this as a bug
>
> 2. Set up a new pfSense install, replicated the interfaces, set up
> oVPN, made a single client connection
>
> 3. Unable to reproduce crash with clean install
>
> 4. Tried reproducing crash on the actual pfSense install... crashes
> now not happening??!!?
>
> 5. It was very late into the night, no devices except my dummy oVPN
> connection and test machine were online, maybe the captured MDNS
> packets were not the direct cause, but the response from one of the
> devices is?
>
>
> So I'm at a loss right now. I have things narrowed down really tight
> on the traffic end, but still have no way to reproduce it from a
> vanilla install, nor do I know where to even begin looking for the
> cause in the kernel code.
>
> I'll try again tomorrow to see if there is a response from some device
> that is the actual cause of the crash. But some suggestions are
> welcome!
>
>
>
> Liwei
>
>
>
> On 23 November 2017 at 01:17, WebDawg  wrote:
>> The bridging may need tested and filed as a bug.
>>
>> On Wed, Nov 22, 2017 at 11:15 AM, Liwei  wrote:
>>> On Thu, 23 Nov 2017 at 00:38 WebDawg  wrote:
>>>
 I am glad that you seemed to have resolved it, does the serial port
 get the standard kernel messages...

>>>
>>> It isn't really solved though as I have to take our bridged VPNs offline.
>>>
>>> Yes it does, but nothing relevant gets spewed out of the serial port before
>>> the panic comes up. The first sign I can see on the serial port of things
>>> going wrong is the kernel panic itself.
>>>
>>>

 usually you log in and tail some log files

>>>
>>> Got it
>>>
>>>

 (bridging our oVPN tap interfaces to the main and private LANs)

 This was bridging done in pfSense right?

>>>
>>> That's right.
>>>
>>>

 

Re: [pfSense] pfSense 2.4 consistently crashes daily

[Liwei]

tl;dr: How do I figure out what the state of the ip6_forward (or other
associated functions) are after a crash? It is really a pain trying to
figure this out based on traffic and trying to replicate the correct
set of pfSense configurations.


Just an update on my adventures. It's quite complicated so I shall put
it as a list of points...

Part 1
1. Spent a weekend copying the pfSense VM to a physical machine and
running it in place of the VM.

2. Eventually the crashes do occur, so I'm entirely convinced this is
not a VM issue

3. Over the last week, I've been disabling various interfaces and
services to narrow down the cause, finally it has been narrowed down
to our VPN bridge interface 

4. If I bring any of the VPN interfaces down/take any of the VPN
interfaces out of the bridge, or bring the bridge interface down, the
crashes stop


Part 2
1. I managed to dig up an old (closed) bug with the exact backtrace I
was getting https://redmine.pfsense.org/issues/5428

2. There are two more bugs on the FreeBSD bugtracker with similar
crashes, the latest was in July this year

3. However, they've all been fixed with the same patch. Checked
RELENG_2_4/sys/netpfil/pf/pf.c , and the patch should be in the 2.4.X
release

a. The network is running with jumbo packets (9k), is it possible
the patch does not cover such a case?

4. This similarity led me to believe that I could be facing a similar
issue, apparently with IPv6 multicast traffic

5. Set up port remote tcpdump so I could capture traffic right before the crash

6. Isolated the traffic cause! Two conditions happening causes the crash:

a. There is at least one VPN client connected
b. There is a macbook running Sierra/High Sierra on the main network

7. Each time the macbook joins the network/sleeps/wakes, the V6
traffic, specifically a certain MDNS query, causes the crash

8. Now the somewhat random but consistent timing makes sense! We have
someone using a macbook come in at around 8pm every day

9. Isolated 2 packet specimens that causes the crash, and 2 of the
same type that does not

a. It does contain names of our users' computers (which on a mac
contains real names), so I'm not inclined to share them on the list,
furthermore, I don't have steps to reproduce the crash with the
packets from a vanilla install, so they're of limited use
b. If anyone is interested to take a look at what the differences
between these two sets of packets are, I can email them to you
directly

Part 3
1. Since I could cause the crash at will, I tried creating
reproducible steps so I can properly report this as a bug

2. Set up a new pfSense install, replicated the interfaces, set up
oVPN, made a single client connection

3. Unable to reproduce crash with clean install

4. Tried reproducing crash on the actual pfSense install... crashes
now not happening??!!?

5. It was very late into the night, no devices except my dummy oVPN
connection and test machine were online, maybe the captured MDNS
packets were not the direct cause, but the response from one of the
devices is?


So I'm at a loss right now. I have things narrowed down really tight
on the traffic end, but still have no way to reproduce it from a
vanilla install, nor do I know where to even begin looking for the
cause in the kernel code.

I'll try again tomorrow to see if there is a response from some device
that is the actual cause of the crash. But some suggestions are
welcome!



Liwei



On 23 November 2017 at 01:17, WebDawg  wrote:
> The bridging may need tested and filed as a bug.
>
> On Wed, Nov 22, 2017 at 11:15 AM, Liwei  wrote:
>> On Thu, 23 Nov 2017 at 00:38 WebDawg  wrote:
>>
>>> I am glad that you seemed to have resolved it, does the serial port
>>> get the standard kernel messages...
>>>
>>
>> It isn't really solved though as I have to take our bridged VPNs offline.
>>
>> Yes it does, but nothing relevant gets spewed out of the serial port before
>> the panic comes up. The first sign I can see on the serial port of things
>> going wrong is the kernel panic itself.
>>
>>
>>>
>>> usually you log in and tail some log files
>>>
>>
>> Got it
>>
>>
>>>
>>> (bridging our oVPN tap interfaces to the main and private LANs)
>>>
>>> This was bridging done in pfSense right?
>>>
>>
>> That's right.
>>
>>
>>>
>>> On Wed, Nov 22, 2017 at 8:07 AM, Liwei  wrote:
>>> > On Tue, 21 Nov 2017 at 01:08 WebDawg  wrote:
>>> >
>>> >> It should work though.  A great many people virtualize pfSense:
>>> >>
>>> >> https://doc.pfsense.org/index.php/PfSense_on_VMware_vSphere_/_ESXi
>>> >>
>>> >> Here is some more information:
>>> >>
>>> >> https://doc.pfsense.org/index.php/VirtIO_Driver_Support
>>> >> https://doc.pfsense.org/index.php/Lost_Traffic_/_Packets_Disappear
>>> >> https://doc.pfsense.org/index.php/Virtualizing_pfSense_on_Proxmox
>>> >>
>>> >> I know what it is like to ask for support and see people stop helping
>>> >> because something is virtualized.  I have seen bad code fail in
>>> >> virtualization situations

Re: [pfSense] pfSense 2.4 consistently crashes daily

[WebDawg]

The bridging may need tested and filed as a bug.

On Wed, Nov 22, 2017 at 11:15 AM, Liwei  wrote:
> On Thu, 23 Nov 2017 at 00:38 WebDawg  wrote:
>
>> I am glad that you seemed to have resolved it, does the serial port
>> get the standard kernel messages...
>>
>
> It isn't really solved though as I have to take our bridged VPNs offline.
>
> Yes it does, but nothing relevant gets spewed out of the serial port before
> the panic comes up. The first sign I can see on the serial port of things
> going wrong is the kernel panic itself.
>
>
>>
>> usually you log in and tail some log files
>>
>
> Got it
>
>
>>
>> (bridging our oVPN tap interfaces to the main and private LANs)
>>
>> This was bridging done in pfSense right?
>>
>
> That's right.
>
>
>>
>> On Wed, Nov 22, 2017 at 8:07 AM, Liwei  wrote:
>> > On Tue, 21 Nov 2017 at 01:08 WebDawg  wrote:
>> >
>> >> It should work though.  A great many people virtualize pfSense:
>> >>
>> >> https://doc.pfsense.org/index.php/PfSense_on_VMware_vSphere_/_ESXi
>> >>
>> >> Here is some more information:
>> >>
>> >> https://doc.pfsense.org/index.php/VirtIO_Driver_Support
>> >> https://doc.pfsense.org/index.php/Lost_Traffic_/_Packets_Disappear
>> >> https://doc.pfsense.org/index.php/Virtualizing_pfSense_on_Proxmox
>> >>
>> >> I know what it is like to ask for support and see people stop helping
>> >> because something is virtualized.  I have seen bad code fail in
>> >> virtualization situations only to here 'do not virtualize'.
>> >>
>> >> From what I know, BSD has trouble with NIC interfaces and such.  Do
>> >> you have any limiters or QOS installed?  I would take a look at the
>> >> nic interfaces first.  Can you actively monitor the log to look for
>> >> errors once the VM is booted?
>> >>
>> >> I virtualized pfSense on proxmox about a year ago and BSD hated the
>> >> cpu timers and such.  I would get so many issues from it until I
>> >> figured it out but everything was plain as day in the kernel messages
>> >> that were outputted.
>> >>
>> >> There is an ova file available via the gold subscription:
>> >>
>> >> https://doc.pfsense.org/index.php/VMware_Appliance
>> >>
>> >> You need to get more information for me to help further.  It would be
>> >> great to get a copy of some logs.
>> >>
>> >> Here is a XenServer thread:
>> >> https://forum.pfsense.org/index.php?topic=88467
>> >>
>> >> Last time I virtualized the big deal was hvm nic vs pvhvm NIC.  You
>> >> could do limiters on one (I think hvm) but the NIC's become CPU bound
>> >> because of how HVM works.  I could only push like 10-30 mbits out of
>> >> an i3 processor.
>> >>
>> >> I do not know if this has been solved, or if it is solvable.  pfSense
>> >> follows FreeBSD so most of the fixes for this come from FreeBSD,
>> >> though pfSense had/has some of its own kernel hacks.
>> >>
>> >>
>> >>
>> > Hi Vick, thanks for the assistance, nonetheless!
>> >
>> > Hi WebDawg,
>> > Yeah, I guessed as much that the problem should be on my side,
>> because
>> > something this fatal should already be widely reported.
>> >
>> > I don't have any limiters or QoS set. I've set up logging of the
>> serial
>> > port so at least I know what are the events leading up to the crash.
>> > Nothing interesting though, it just... happens. How do I set up log
>> > monitoring? My guess is I'll probably have to turn on remote syslog and
>> log
>> > over. Will set up when I get the chance.
>> >
>> > The odd thing is this is a 7+ years old setup (but we did do a fresh
>> > install of 2.3 when we upgraded hardware 1+ years ago), and we never had
>> > any serious issues. In fact it was purring along nicely on 2.3 since it
>> was
>> > first installed, until we upgraded to 2.4.
>> >
>> > I'm pretty confident of the hardware since it is only a year old, the
>> > other VMs are not having any issues, and reverting to 2.3 works fine.
>> Thus
>> > based on a hunch I decided to remove a couple of bridge interfaces
>> > (bridging our oVPN tap interfaces to the main and private LANs) when I
>> sent
>> > my first email to the list.
>> >
>> > The crashes haven't occurred since then for 2 days. I'm not sure if
>> it
>> > is a coincidence or not, but it does seem like my configuration may be
>> > triggering some bug. Or I may have mis-configured something.
>> >
>> > I'll continue to iterate things around to narrow down the problem,
>> but
>> > given that I have to wait a few days after each change to be sure on
>> > whether it crashes or not, any suggestion is very welcome!
>> >
>> > Warm regards,
>> > Liwei
>> > --
>> > Clear Skies,LiweiCo-Founder, CTO
>> >
>> > TinyMOS
>> >
>> >
>> >  
>> >  
>> >
>> > 21 Heng Mui Keng Terrace, Level 1 The Hangar, Singapore 119613
>> > ___
>> > pfSense mailing list
>> > https://lists.pfsense.org/mailman/listinfo/list
>> > Support the pr

Re: [pfSense] pfSense 2.4 consistently crashes daily

[Liwei]

On Thu, 23 Nov 2017 at 00:38 WebDawg  wrote:

> I am glad that you seemed to have resolved it, does the serial port
> get the standard kernel messages...
>

It isn't really solved though as I have to take our bridged VPNs offline.

Yes it does, but nothing relevant gets spewed out of the serial port before
the panic comes up. The first sign I can see on the serial port of things
going wrong is the kernel panic itself.


>
> usually you log in and tail some log files
>

Got it


>
> (bridging our oVPN tap interfaces to the main and private LANs)
>
> This was bridging done in pfSense right?
>

That's right.


>
> On Wed, Nov 22, 2017 at 8:07 AM, Liwei  wrote:
> > On Tue, 21 Nov 2017 at 01:08 WebDawg  wrote:
> >
> >> It should work though.  A great many people virtualize pfSense:
> >>
> >> https://doc.pfsense.org/index.php/PfSense_on_VMware_vSphere_/_ESXi
> >>
> >> Here is some more information:
> >>
> >> https://doc.pfsense.org/index.php/VirtIO_Driver_Support
> >> https://doc.pfsense.org/index.php/Lost_Traffic_/_Packets_Disappear
> >> https://doc.pfsense.org/index.php/Virtualizing_pfSense_on_Proxmox
> >>
> >> I know what it is like to ask for support and see people stop helping
> >> because something is virtualized.  I have seen bad code fail in
> >> virtualization situations only to here 'do not virtualize'.
> >>
> >> From what I know, BSD has trouble with NIC interfaces and such.  Do
> >> you have any limiters or QOS installed?  I would take a look at the
> >> nic interfaces first.  Can you actively monitor the log to look for
> >> errors once the VM is booted?
> >>
> >> I virtualized pfSense on proxmox about a year ago and BSD hated the
> >> cpu timers and such.  I would get so many issues from it until I
> >> figured it out but everything was plain as day in the kernel messages
> >> that were outputted.
> >>
> >> There is an ova file available via the gold subscription:
> >>
> >> https://doc.pfsense.org/index.php/VMware_Appliance
> >>
> >> You need to get more information for me to help further.  It would be
> >> great to get a copy of some logs.
> >>
> >> Here is a XenServer thread:
> >> https://forum.pfsense.org/index.php?topic=88467
> >>
> >> Last time I virtualized the big deal was hvm nic vs pvhvm NIC.  You
> >> could do limiters on one (I think hvm) but the NIC's become CPU bound
> >> because of how HVM works.  I could only push like 10-30 mbits out of
> >> an i3 processor.
> >>
> >> I do not know if this has been solved, or if it is solvable.  pfSense
> >> follows FreeBSD so most of the fixes for this come from FreeBSD,
> >> though pfSense had/has some of its own kernel hacks.
> >>
> >>
> >>
> > Hi Vick, thanks for the assistance, nonetheless!
> >
> > Hi WebDawg,
> > Yeah, I guessed as much that the problem should be on my side,
> because
> > something this fatal should already be widely reported.
> >
> > I don't have any limiters or QoS set. I've set up logging of the
> serial
> > port so at least I know what are the events leading up to the crash.
> > Nothing interesting though, it just... happens. How do I set up log
> > monitoring? My guess is I'll probably have to turn on remote syslog and
> log
> > over. Will set up when I get the chance.
> >
> > The odd thing is this is a 7+ years old setup (but we did do a fresh
> > install of 2.3 when we upgraded hardware 1+ years ago), and we never had
> > any serious issues. In fact it was purring along nicely on 2.3 since it
> was
> > first installed, until we upgraded to 2.4.
> >
> > I'm pretty confident of the hardware since it is only a year old, the
> > other VMs are not having any issues, and reverting to 2.3 works fine.
> Thus
> > based on a hunch I decided to remove a couple of bridge interfaces
> > (bridging our oVPN tap interfaces to the main and private LANs) when I
> sent
> > my first email to the list.
> >
> > The crashes haven't occurred since then for 2 days. I'm not sure if
> it
> > is a coincidence or not, but it does seem like my configuration may be
> > triggering some bug. Or I may have mis-configured something.
> >
> > I'll continue to iterate things around to narrow down the problem,
> but
> > given that I have to wait a few days after each change to be sure on
> > whether it crashes or not, any suggestion is very welcome!
> >
> > Warm regards,
> > Liwei
> > --
> > Clear Skies,LiweiCo-Founder, CTO
> >
> > TinyMOS
> >
> >
> >  
> >  
> >
> > 21 Heng Mui Keng Terrace, Level 1 The Hangar, Singapore 119613
> > ___
> > pfSense mailing list
> > https://lists.pfsense.org/mailman/listinfo/list
> > Support the project with Gold! https://pfsense.org/gold
> ___
> pfSense mailing list
> https://lists.pfsense.org/mailman/listinfo/list
> Support the project with Gold! https://pfsense.org/gold
>
-- 
Clear Ski

Re: [pfSense] pfSense 2.4 consistently crashes daily

[Liwei]

On Thu, 23 Nov 2017 at 00:05 Steve Yates  wrote:

> Any chance it had a 32 bit install and you manually upgraded to 64?  I
> believe pfSense recommends a wipe and reinstall in that case.
>
> --
>
> Steve Yates
> ITS, Inc.
>

Nope, it had always been on 64-bit. We also did a wipe and reinstall last
year on 2.3.


>
> -Original Message-
> From: List [mailto:list-boun...@lists.pfsense.org] On Behalf Of Liwei
> Sent: Wednesday, November 22, 2017 8:08 AM
> To: pfSense Support and Discussion Mailing List 
> Subject: Re: [pfSense] pfSense 2.4 consistently crashes daily
>
> On Tue, 21 Nov 2017 at 01:08 WebDawg  wrote:
>
> > It should work though.  A great many people virtualize pfSense:
> >
> > https://doc.pfsense.org/index.php/PfSense_on_VMware_vSphere_/_ESXi
> >
> > Here is some more information:
> >
> > https://doc.pfsense.org/index.php/VirtIO_Driver_Support
> > https://doc.pfsense.org/index.php/Lost_Traffic_/_Packets_Disappear
> > https://doc.pfsense.org/index.php/Virtualizing_pfSense_on_Proxmox
> >
> > I know what it is like to ask for support and see people stop helping
> > because something is virtualized.  I have seen bad code fail in
> > virtualization situations only to here 'do not virtualize'.
> >
> > From what I know, BSD has trouble with NIC interfaces and such.  Do
> > you have any limiters or QOS installed?  I would take a look at the
> > nic interfaces first.  Can you actively monitor the log to look for
> > errors once the VM is booted?
> >
> > I virtualized pfSense on proxmox about a year ago and BSD hated the
> > cpu timers and such.  I would get so many issues from it until I
> > figured it out but everything was plain as day in the kernel messages
> > that were outputted.
> >
> > There is an ova file available via the gold subscription:
> >
> > https://doc.pfsense.org/index.php/VMware_Appliance
> >
> > You need to get more information for me to help further.  It would be
> > great to get a copy of some logs.
> >
> > Here is a XenServer thread:
> > https://forum.pfsense.org/index.php?topic=88467
> >
> > Last time I virtualized the big deal was hvm nic vs pvhvm NIC.  You
> > could do limiters on one (I think hvm) but the NIC's become CPU bound
> > because of how HVM works.  I could only push like 10-30 mbits out of
> > an i3 processor.
> >
> > I do not know if this has been solved, or if it is solvable.  pfSense
> > follows FreeBSD so most of the fixes for this come from FreeBSD,
> > though pfSense had/has some of its own kernel hacks.
> >
> >
> >
> Hi Vick, thanks for the assistance, nonetheless!
>
> Hi WebDawg,
> Yeah, I guessed as much that the problem should be on my side, because
> something this fatal should already be widely reported.
>
> I don't have any limiters or QoS set. I've set up logging of the serial
> port so at least I know what are the events leading up to the crash.
> Nothing interesting though, it just... happens. How do I set up log
> monitoring? My guess is I'll probably have to turn on remote syslog and log
> over. Will set up when I get the chance.
>
> The odd thing is this is a 7+ years old setup (but we did do a fresh
> install of 2.3 when we upgraded hardware 1+ years ago), and we never had
> any serious issues. In fact it was purring along nicely on 2.3 since it was
> first installed, until we upgraded to 2.4.
>
> I'm pretty confident of the hardware since it is only a year old, the
> other VMs are not having any issues, and reverting to 2.3 works fine. Thus
> based on a hunch I decided to remove a couple of bridge interfaces
> (bridging our oVPN tap interfaces to the main and private LANs) when I sent
> my first email to the list.
>
> The crashes haven't occurred since then for 2 days. I'm not sure if it
> is a coincidence or not, but it does seem like my configuration may be
> triggering some bug. Or I may have mis-configured something.
>
> I'll continue to iterate things around to narrow down the problem, but
> given that I have to wait a few days after each change to be sure on
> whether it crashes or not, any suggestion is very welcome!
>
> Warm regards,
> Liwei
> --
> Clear Skies,LiweiCo-Founder, CTO
>
> TinyMOS
>
>
> <http://tinymos.com/> <https://www.facebook.com/thetinymos/>
> <https://www.instagram.com/thetinymos/> <https://twitter.com/thetinymos>
>
> 21 Heng Mui Keng Terrace, Level 1 The Hangar, Singapore 119613
> ___
> pfSense mailing list
> h

Re: [pfSense] pfSense 2.4 consistently crashes daily

[WebDawg]

I am glad that you seemed to have resolved it, does the serial port
get the standard kernel messages...

usually you log in and tail some log files

(bridging our oVPN tap interfaces to the main and private LANs)

This was bridging done in pfSense right?

On Wed, Nov 22, 2017 at 8:07 AM, Liwei  wrote:
> On Tue, 21 Nov 2017 at 01:08 WebDawg  wrote:
>
>> It should work though.  A great many people virtualize pfSense:
>>
>> https://doc.pfsense.org/index.php/PfSense_on_VMware_vSphere_/_ESXi
>>
>> Here is some more information:
>>
>> https://doc.pfsense.org/index.php/VirtIO_Driver_Support
>> https://doc.pfsense.org/index.php/Lost_Traffic_/_Packets_Disappear
>> https://doc.pfsense.org/index.php/Virtualizing_pfSense_on_Proxmox
>>
>> I know what it is like to ask for support and see people stop helping
>> because something is virtualized.  I have seen bad code fail in
>> virtualization situations only to here 'do not virtualize'.
>>
>> From what I know, BSD has trouble with NIC interfaces and such.  Do
>> you have any limiters or QOS installed?  I would take a look at the
>> nic interfaces first.  Can you actively monitor the log to look for
>> errors once the VM is booted?
>>
>> I virtualized pfSense on proxmox about a year ago and BSD hated the
>> cpu timers and such.  I would get so many issues from it until I
>> figured it out but everything was plain as day in the kernel messages
>> that were outputted.
>>
>> There is an ova file available via the gold subscription:
>>
>> https://doc.pfsense.org/index.php/VMware_Appliance
>>
>> You need to get more information for me to help further.  It would be
>> great to get a copy of some logs.
>>
>> Here is a XenServer thread:
>> https://forum.pfsense.org/index.php?topic=88467
>>
>> Last time I virtualized the big deal was hvm nic vs pvhvm NIC.  You
>> could do limiters on one (I think hvm) but the NIC's become CPU bound
>> because of how HVM works.  I could only push like 10-30 mbits out of
>> an i3 processor.
>>
>> I do not know if this has been solved, or if it is solvable.  pfSense
>> follows FreeBSD so most of the fixes for this come from FreeBSD,
>> though pfSense had/has some of its own kernel hacks.
>>
>>
>>
> Hi Vick, thanks for the assistance, nonetheless!
>
> Hi WebDawg,
> Yeah, I guessed as much that the problem should be on my side, because
> something this fatal should already be widely reported.
>
> I don't have any limiters or QoS set. I've set up logging of the serial
> port so at least I know what are the events leading up to the crash.
> Nothing interesting though, it just... happens. How do I set up log
> monitoring? My guess is I'll probably have to turn on remote syslog and log
> over. Will set up when I get the chance.
>
> The odd thing is this is a 7+ years old setup (but we did do a fresh
> install of 2.3 when we upgraded hardware 1+ years ago), and we never had
> any serious issues. In fact it was purring along nicely on 2.3 since it was
> first installed, until we upgraded to 2.4.
>
> I'm pretty confident of the hardware since it is only a year old, the
> other VMs are not having any issues, and reverting to 2.3 works fine. Thus
> based on a hunch I decided to remove a couple of bridge interfaces
> (bridging our oVPN tap interfaces to the main and private LANs) when I sent
> my first email to the list.
>
> The crashes haven't occurred since then for 2 days. I'm not sure if it
> is a coincidence or not, but it does seem like my configuration may be
> triggering some bug. Or I may have mis-configured something.
>
> I'll continue to iterate things around to narrow down the problem, but
> given that I have to wait a few days after each change to be sure on
> whether it crashes or not, any suggestion is very welcome!
>
> Warm regards,
> Liwei
> --
> Clear Skies,LiweiCo-Founder, CTO
>
> TinyMOS
>
>
>  
>  
>
> 21 Heng Mui Keng Terrace, Level 1 The Hangar, Singapore 119613
> ___
> pfSense mailing list
> https://lists.pfsense.org/mailman/listinfo/list
> Support the project with Gold! https://pfsense.org/gold
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] pfSense 2.4 consistently crashes daily

[Steve Yates]

Any chance it had a 32 bit install and you manually upgraded to 64?  I believe 
pfSense recommends a wipe and reinstall in that case.

--

Steve Yates
ITS, Inc.

-Original Message-
From: List [mailto:list-boun...@lists.pfsense.org] On Behalf Of Liwei
Sent: Wednesday, November 22, 2017 8:08 AM
To: pfSense Support and Discussion Mailing List 
Subject: Re: [pfSense] pfSense 2.4 consistently crashes daily

On Tue, 21 Nov 2017 at 01:08 WebDawg  wrote:

> It should work though.  A great many people virtualize pfSense:
>
> https://doc.pfsense.org/index.php/PfSense_on_VMware_vSphere_/_ESXi
>
> Here is some more information:
>
> https://doc.pfsense.org/index.php/VirtIO_Driver_Support
> https://doc.pfsense.org/index.php/Lost_Traffic_/_Packets_Disappear
> https://doc.pfsense.org/index.php/Virtualizing_pfSense_on_Proxmox
>
> I know what it is like to ask for support and see people stop helping
> because something is virtualized.  I have seen bad code fail in
> virtualization situations only to here 'do not virtualize'.
>
> From what I know, BSD has trouble with NIC interfaces and such.  Do
> you have any limiters or QOS installed?  I would take a look at the
> nic interfaces first.  Can you actively monitor the log to look for
> errors once the VM is booted?
>
> I virtualized pfSense on proxmox about a year ago and BSD hated the
> cpu timers and such.  I would get so many issues from it until I
> figured it out but everything was plain as day in the kernel messages
> that were outputted.
>
> There is an ova file available via the gold subscription:
>
> https://doc.pfsense.org/index.php/VMware_Appliance
>
> You need to get more information for me to help further.  It would be
> great to get a copy of some logs.
>
> Here is a XenServer thread:
> https://forum.pfsense.org/index.php?topic=88467
>
> Last time I virtualized the big deal was hvm nic vs pvhvm NIC.  You
> could do limiters on one (I think hvm) but the NIC's become CPU bound
> because of how HVM works.  I could only push like 10-30 mbits out of
> an i3 processor.
>
> I do not know if this has been solved, or if it is solvable.  pfSense
> follows FreeBSD so most of the fixes for this come from FreeBSD,
> though pfSense had/has some of its own kernel hacks.
>
>
>
Hi Vick, thanks for the assistance, nonetheless!

Hi WebDawg,
Yeah, I guessed as much that the problem should be on my side, because
something this fatal should already be widely reported.

I don't have any limiters or QoS set. I've set up logging of the serial
port so at least I know what are the events leading up to the crash.
Nothing interesting though, it just... happens. How do I set up log
monitoring? My guess is I'll probably have to turn on remote syslog and log
over. Will set up when I get the chance.

The odd thing is this is a 7+ years old setup (but we did do a fresh
install of 2.3 when we upgraded hardware 1+ years ago), and we never had
any serious issues. In fact it was purring along nicely on 2.3 since it was
first installed, until we upgraded to 2.4.

I'm pretty confident of the hardware since it is only a year old, the
other VMs are not having any issues, and reverting to 2.3 works fine. Thus
based on a hunch I decided to remove a couple of bridge interfaces
(bridging our oVPN tap interfaces to the main and private LANs) when I sent
my first email to the list.

The crashes haven't occurred since then for 2 days. I'm not sure if it
is a coincidence or not, but it does seem like my configuration may be
triggering some bug. Or I may have mis-configured something.

I'll continue to iterate things around to narrow down the problem, but
given that I have to wait a few days after each change to be sure on
whether it crashes or not, any suggestion is very welcome!

Warm regards,
Liwei
-- 
Clear Skies,LiweiCo-Founder, CTO

TinyMOS


<http://tinymos.com/> <https://www.facebook.com/thetinymos/>
<https://www.instagram.com/thetinymos/> <https://twitter.com/thetinymos>

21 Heng Mui Keng Terrace, Level 1 The Hangar, Singapore 119613
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] pfSense 2.4 consistently crashes daily

[Liwei]

On Tue, 21 Nov 2017 at 01:08 WebDawg  wrote:

> It should work though.  A great many people virtualize pfSense:
>
> https://doc.pfsense.org/index.php/PfSense_on_VMware_vSphere_/_ESXi
>
> Here is some more information:
>
> https://doc.pfsense.org/index.php/VirtIO_Driver_Support
> https://doc.pfsense.org/index.php/Lost_Traffic_/_Packets_Disappear
> https://doc.pfsense.org/index.php/Virtualizing_pfSense_on_Proxmox
>
> I know what it is like to ask for support and see people stop helping
> because something is virtualized.  I have seen bad code fail in
> virtualization situations only to here 'do not virtualize'.
>
> From what I know, BSD has trouble with NIC interfaces and such.  Do
> you have any limiters or QOS installed?  I would take a look at the
> nic interfaces first.  Can you actively monitor the log to look for
> errors once the VM is booted?
>
> I virtualized pfSense on proxmox about a year ago and BSD hated the
> cpu timers and such.  I would get so many issues from it until I
> figured it out but everything was plain as day in the kernel messages
> that were outputted.
>
> There is an ova file available via the gold subscription:
>
> https://doc.pfsense.org/index.php/VMware_Appliance
>
> You need to get more information for me to help further.  It would be
> great to get a copy of some logs.
>
> Here is a XenServer thread:
> https://forum.pfsense.org/index.php?topic=88467
>
> Last time I virtualized the big deal was hvm nic vs pvhvm NIC.  You
> could do limiters on one (I think hvm) but the NIC's become CPU bound
> because of how HVM works.  I could only push like 10-30 mbits out of
> an i3 processor.
>
> I do not know if this has been solved, or if it is solvable.  pfSense
> follows FreeBSD so most of the fixes for this come from FreeBSD,
> though pfSense had/has some of its own kernel hacks.
>
>
>
Hi Vick, thanks for the assistance, nonetheless!

Hi WebDawg,
Yeah, I guessed as much that the problem should be on my side, because
something this fatal should already be widely reported.

I don't have any limiters or QoS set. I've set up logging of the serial
port so at least I know what are the events leading up to the crash.
Nothing interesting though, it just... happens. How do I set up log
monitoring? My guess is I'll probably have to turn on remote syslog and log
over. Will set up when I get the chance.

The odd thing is this is a 7+ years old setup (but we did do a fresh
install of 2.3 when we upgraded hardware 1+ years ago), and we never had
any serious issues. In fact it was purring along nicely on 2.3 since it was
first installed, until we upgraded to 2.4.

I'm pretty confident of the hardware since it is only a year old, the
other VMs are not having any issues, and reverting to 2.3 works fine. Thus
based on a hunch I decided to remove a couple of bridge interfaces
(bridging our oVPN tap interfaces to the main and private LANs) when I sent
my first email to the list.

The crashes haven't occurred since then for 2 days. I'm not sure if it
is a coincidence or not, but it does seem like my configuration may be
triggering some bug. Or I may have mis-configured something.

I'll continue to iterate things around to narrow down the problem, but
given that I have to wait a few days after each change to be sure on
whether it crashes or not, any suggestion is very welcome!

Warm regards,
Liwei
-- 
Clear Skies,LiweiCo-Founder, CTO

TinyMOS


 
 

21 Heng Mui Keng Terrace, Level 1 The Hangar, Singapore 119613
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] pfSense 2.4 consistently crashes daily

[WebDawg]

It should work though.  A great many people virtualize pfSense:

https://doc.pfsense.org/index.php/PfSense_on_VMware_vSphere_/_ESXi

Here is some more information:

https://doc.pfsense.org/index.php/VirtIO_Driver_Support
https://doc.pfsense.org/index.php/Lost_Traffic_/_Packets_Disappear
https://doc.pfsense.org/index.php/Virtualizing_pfSense_on_Proxmox

I know what it is like to ask for support and see people stop helping
because something is virtualized.  I have seen bad code fail in
virtualization situations only to here 'do not virtualize'.

>From what I know, BSD has trouble with NIC interfaces and such.  Do
you have any limiters or QOS installed?  I would take a look at the
nic interfaces first.  Can you actively monitor the log to look for
errors once the VM is booted?

I virtualized pfSense on proxmox about a year ago and BSD hated the
cpu timers and such.  I would get so many issues from it until I
figured it out but everything was plain as day in the kernel messages
that were outputted.

There is an ova file available via the gold subscription:

https://doc.pfsense.org/index.php/VMware_Appliance

You need to get more information for me to help further.  It would be
great to get a copy of some logs.

Here is a XenServer thread:  https://forum.pfsense.org/index.php?topic=88467

Last time I virtualized the big deal was hvm nic vs pvhvm NIC.  You
could do limiters on one (I think hvm) but the NIC's become CPU bound
because of how HVM works.  I could only push like 10-30 mbits out of
an i3 processor.

I do not know if this has been solved, or if it is solvable.  pfSense
follows FreeBSD so most of the fixes for this come from FreeBSD,
though pfSense had/has some of its own kernel hacks.



On Mon, Nov 20, 2017 at 10:58 AM, Vick Khera  wrote:
> Oh, so you're not running it on hardware, but inside ESXi? Then I have no
> more ideas for you. You should mention these things when asking for help,
> by the way.
>
>
> On Mon, Nov 20, 2017 at 8:12 AM, Liwei  wrote:
>
>> Thanks for the quick reply. It is a Supermicro 5018A-FTN4 based on
>> the A1SRi-2758F which contains an Atom C2758. RAM tests are fine. This
>> machine also contains a few other VMs which are running fine.
>>
>> By the way, I missed out reporting the crash itself:
>>
>> Fatal trap 12: page fault while in kernel mode
>> cpuid = 2; apic id = 02
>> fault virtual address = 0x60
>> fault code = supervisor read data, page not present
>> instruction pointer = 0x20:0x80cbcb0f
>> stack pointer = 0x28:0xfe02390bf070
>> frame pointer = 0x28:0xfe02390bf070
>> code segment = base 0x0, limit 0xf, type 0x1b
>> = DPL 0, pres 1, long 1, def32 0, gran 1
>> processor eflags = interrupt enabled, resume, IOPL = 0
>> current process = 12 (irq267: vmx0)
>>
>> On Mon, 20 Nov 2017 at 20:55 Vick Khera  wrote:
>>
>> > On Mon, Nov 20, 2017 at 7:36 AM, Liwei  wrote:
>> >
>> > >
>> > > Anyone has any idea what's going on? Restoring to pfSense 2.3 seems
>> > to
>> > > solve this problem, so it is more likely a software than hardware
>> issue.
>> > >
>> > >
>> > What's your hardware? Have you tested your RAM using memtest86?
>> > ___
>> > pfSense mailing list
>> > https://lists.pfsense.org/mailman/listinfo/list
>> > Support the project with Gold! https://pfsense.org/gold
>> >
>> --
>> Clear Skies,LiweiCo-Founder, CTO
>>
>> TinyMOS
>>
>>
>>  
>>  
>>
>> 21 Heng Mui Keng Terrace, Level 1 The Hangar, Singapore 119613
>> ___
>> pfSense mailing list
>> https://lists.pfsense.org/mailman/listinfo/list
>> Support the project with Gold! https://pfsense.org/gold
>>
> ___
> pfSense mailing list
> https://lists.pfsense.org/mailman/listinfo/list
> Support the project with Gold! https://pfsense.org/gold
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] pfSense 2.4 consistently crashes daily

[Vick Khera]

Oh, so you're not running it on hardware, but inside ESXi? Then I have no
more ideas for you. You should mention these things when asking for help,
by the way.


On Mon, Nov 20, 2017 at 8:12 AM, Liwei  wrote:

> Thanks for the quick reply. It is a Supermicro 5018A-FTN4 based on
> the A1SRi-2758F which contains an Atom C2758. RAM tests are fine. This
> machine also contains a few other VMs which are running fine.
>
> By the way, I missed out reporting the crash itself:
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 2; apic id = 02
> fault virtual address = 0x60
> fault code = supervisor read data, page not present
> instruction pointer = 0x20:0x80cbcb0f
> stack pointer = 0x28:0xfe02390bf070
> frame pointer = 0x28:0xfe02390bf070
> code segment = base 0x0, limit 0xf, type 0x1b
> = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags = interrupt enabled, resume, IOPL = 0
> current process = 12 (irq267: vmx0)
>
> On Mon, 20 Nov 2017 at 20:55 Vick Khera  wrote:
>
> > On Mon, Nov 20, 2017 at 7:36 AM, Liwei  wrote:
> >
> > >
> > > Anyone has any idea what's going on? Restoring to pfSense 2.3 seems
> > to
> > > solve this problem, so it is more likely a software than hardware
> issue.
> > >
> > >
> > What's your hardware? Have you tested your RAM using memtest86?
> > ___
> > pfSense mailing list
> > https://lists.pfsense.org/mailman/listinfo/list
> > Support the project with Gold! https://pfsense.org/gold
> >
> --
> Clear Skies,LiweiCo-Founder, CTO
>
> TinyMOS
>
>
>  
>  
>
> 21 Heng Mui Keng Terrace, Level 1 The Hangar, Singapore 119613
> ___
> pfSense mailing list
> https://lists.pfsense.org/mailman/listinfo/list
> Support the project with Gold! https://pfsense.org/gold
>
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] pfSense 2.4 consistently crashes daily

[Liwei]

ESXi 6.5
On Mon, 20 Nov 2017 at 22:04 WebDawg  wrote:

> What virt software?
>
> On Mon, Nov 20, 2017 at 7:12 AM, Liwei  wrote:
> > Thanks for the quick reply. It is a Supermicro 5018A-FTN4 based on
> > the A1SRi-2758F which contains an Atom C2758. RAM tests are fine. This
> > machine also contains a few other VMs which are running fine.
> >
> > By the way, I missed out reporting the crash itself:
> >
> > Fatal trap 12: page fault while in kernel mode
> > cpuid = 2; apic id = 02
> > fault virtual address = 0x60
> > fault code = supervisor read data, page not present
> > instruction pointer = 0x20:0x80cbcb0f
> > stack pointer = 0x28:0xfe02390bf070
> > frame pointer = 0x28:0xfe02390bf070
> > code segment = base 0x0, limit 0xf, type 0x1b
> > = DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags = interrupt enabled, resume, IOPL = 0
> > current process = 12 (irq267: vmx0)
> >
> > On Mon, 20 Nov 2017 at 20:55 Vick Khera  wrote:
> >
> >> On Mon, Nov 20, 2017 at 7:36 AM, Liwei  wrote:
> >>
> >> >
> >> > Anyone has any idea what's going on? Restoring to pfSense 2.3
> seems
> >> to
> >> > solve this problem, so it is more likely a software than hardware
> issue.
> >> >
> >> >
> >> What's your hardware? Have you tested your RAM using memtest86?
> >> ___
> >> pfSense mailing list
> >> https://lists.pfsense.org/mailman/listinfo/list
> >> Support the project with Gold! https://pfsense.org/gold
> >>
> > --
> > Clear Skies,LiweiCo-Founder, CTO
> >
> > TinyMOS
> >
> >
> >  
> >  
> >
> > 21 Heng Mui Keng Terrace, Level 1 The Hangar, Singapore 119613
> > ___
> > pfSense mailing list
> > https://lists.pfsense.org/mailman/listinfo/list
> > Support the project with Gold! https://pfsense.org/gold
> ___
> pfSense mailing list
> https://lists.pfsense.org/mailman/listinfo/list
> Support the project with Gold! https://pfsense.org/gold
>
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] pfSense 2.4 consistently crashes daily

[WebDawg]

What virt software?

On Mon, Nov 20, 2017 at 7:12 AM, Liwei  wrote:
> Thanks for the quick reply. It is a Supermicro 5018A-FTN4 based on
> the A1SRi-2758F which contains an Atom C2758. RAM tests are fine. This
> machine also contains a few other VMs which are running fine.
>
> By the way, I missed out reporting the crash itself:
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 2; apic id = 02
> fault virtual address = 0x60
> fault code = supervisor read data, page not present
> instruction pointer = 0x20:0x80cbcb0f
> stack pointer = 0x28:0xfe02390bf070
> frame pointer = 0x28:0xfe02390bf070
> code segment = base 0x0, limit 0xf, type 0x1b
> = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags = interrupt enabled, resume, IOPL = 0
> current process = 12 (irq267: vmx0)
>
> On Mon, 20 Nov 2017 at 20:55 Vick Khera  wrote:
>
>> On Mon, Nov 20, 2017 at 7:36 AM, Liwei  wrote:
>>
>> >
>> > Anyone has any idea what's going on? Restoring to pfSense 2.3 seems
>> to
>> > solve this problem, so it is more likely a software than hardware issue.
>> >
>> >
>> What's your hardware? Have you tested your RAM using memtest86?
>> ___
>> pfSense mailing list
>> https://lists.pfsense.org/mailman/listinfo/list
>> Support the project with Gold! https://pfsense.org/gold
>>
> --
> Clear Skies,LiweiCo-Founder, CTO
>
> TinyMOS
>
>
>  
>  
>
> 21 Heng Mui Keng Terrace, Level 1 The Hangar, Singapore 119613
> ___
> pfSense mailing list
> https://lists.pfsense.org/mailman/listinfo/list
> Support the project with Gold! https://pfsense.org/gold
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] pfSense 2.4 consistently crashes daily

[Liwei]

Thanks for the quick reply. It is a Supermicro 5018A-FTN4 based on
the A1SRi-2758F which contains an Atom C2758. RAM tests are fine. This
machine also contains a few other VMs which are running fine.

By the way, I missed out reporting the crash itself:

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address = 0x60
fault code = supervisor read data, page not present
instruction pointer = 0x20:0x80cbcb0f
stack pointer = 0x28:0xfe02390bf070
frame pointer = 0x28:0xfe02390bf070
code segment = base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (irq267: vmx0)

On Mon, 20 Nov 2017 at 20:55 Vick Khera  wrote:

> On Mon, Nov 20, 2017 at 7:36 AM, Liwei  wrote:
>
> >
> > Anyone has any idea what's going on? Restoring to pfSense 2.3 seems
> to
> > solve this problem, so it is more likely a software than hardware issue.
> >
> >
> What's your hardware? Have you tested your RAM using memtest86?
> ___
> pfSense mailing list
> https://lists.pfsense.org/mailman/listinfo/list
> Support the project with Gold! https://pfsense.org/gold
>
-- 
Clear Skies,LiweiCo-Founder, CTO

TinyMOS


 
 

21 Heng Mui Keng Terrace, Level 1 The Hangar, Singapore 119613
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] pfSense 2.4 consistently crashes daily

[Vick Khera]

On Mon, Nov 20, 2017 at 7:36 AM, Liwei  wrote:

>
> Anyone has any idea what's going on? Restoring to pfSense 2.3 seems to
> solve this problem, so it is more likely a software than hardware issue.
>
>
What's your hardware? Have you tested your RAM using memtest86?
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

[pfSense] pfSense 2.4 consistently crashes daily

[Liwei]

Hi list,
I upgraded to pfSense 2.4.1 (from 2.3) recently, and ever since the
upgrade, the machine has been crashing multiple times daily. Here's a list
of Dumptimes extracted from the bug reporting module:

Line 20:   Dumptime: Thu Nov 16 19:59:59 2017
Line 36:   Dumptime: Thu Nov 16 20:02:04 2017
Line 52:   Dumptime: Sat Nov 18 19:11:50 2017
Line 68:   Dumptime: Sat Nov 18 20:06:03 2017
Line 84:   Dumptime: Sat Nov 18 20:08:33 2017
Line 100:   Dumptime: Sat Nov 18 20:10:41 2017
Line 116:   Dumptime: Sat Nov 18 20:12:57 2017
Line 132:   Dumptime: Sat Nov 18 20:18:13 2017
Line 148:   Dumptime: Sat Nov 18 20:20:51 2017
Line 164:   Dumptime: Sat Nov 18 22:13:42 2017
Line 180:   Dumptime: Sun Nov 19 00:01:08 2017
Line 196:   Dumptime: Sun Nov 19 09:14:27 2017
Line 212:   Dumptime: Thu Nov 16 20:04:19 2017
Line 228:   Dumptime: Sun Nov 19 12:24:57 2017
Line 244:   Dumptime: Sun Nov 19 19:00:57 2017
Line 260:   Dumptime: Sun Nov 19 19:16:59 2017
Line 276:   Dumptime: Sun Nov 19 19:19:04 2017
Line 292:   Dumptime: Sun Nov 19 19:21:34 2017
Line 308:   Dumptime: Sun Nov 19 19:31:04 2017
Line 324:   Dumptime: Sun Nov 19 20:19:43 2017
Line 340:   Dumptime: Sun Nov 19 21:19:33 2017
Line 356:   Dumptime: Sun Nov 19 21:55:31 2017
Line 372:   Dumptime: Mon Nov 20 18:52:17 2017
Line 388:   Dumptime: Thu Nov 16 20:08:31 2017
Line 404:   Dumptime: Mon Nov 20 18:54:32 2017
Line 420:   Dumptime: Mon Nov 20 18:57:50 2017

As you can see, it seems to occur around 7 to 8pm (GMT+8) daily and
occurs in rapid succession. There are exceptions though (e.g. the crashes
that happened at midnight, 9am and 1pm above). Eventually it does stabilise
and things appear well for another 24 hours or so.

Looking through the dumps, the backtrace is always this:

db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100085 td 0xf80006613560
strlen() at strlen+0x1f/frame 0xfe02390bf070
kvprintf() at kvprintf+0x93c/frame 0xfe02390bf160
vlog() at vlog+0x9b/frame 0xfe02390bf240
log() at log+0x3f/frame 0xfe02390bf2a0
ip6_forward() at ip6_forward+0xc5/frame 0xfe02390bf3f0
pf_refragment6() at pf_refragment6+0x17a/frame 0xfe02390bf4b0
pf_test6() at pf_test6+0x2176/frame 0xfe02390bf710
pf_check6_out() at pf_check6_out+0x1d/frame 0xfe02390bf730
pfil_run_hooks() at pfil_run_hooks+0x7b/frame 0xfe02390bf7c0
bridge_pfil() at bridge_pfil+0x61b/frame 0xfe02390bf850
bridge_broadcast() at bridge_broadcast+0x249/frame 0xfe02390bf8c0
bridge_forward() at bridge_forward+0x23b/frame 0xfe02390bf920
bridge_input() at bridge_input+0x2bd/frame 0xfe02390bf990
ether_nh_input() at ether_nh_input+0x292/frame 0xfe02390bf9f0
netisr_dispatch_src() at netisr_dispatch_src+0xa0/frame 0xfe02390bfa40
ether_input() at ether_input+0x26/frame 0xfe02390bfa60
vmxnet3_rxq_eof() at vmxnet3_rxq_eof+0x811/frame 0xfe02390bfae0
vmxnet3_legacy_intr() at vmxnet3_legacy_intr+0xf1/frame 0xfe02390bfb20
intr_event_execute_handlers() at intr_event_execute_handlers+0xec/frame
0xfe02390bfb60
ithread_loop() at ithread_loop+0xd6/frame 0xfe02390bfbb0
fork_exit() at fork_exit+0x85/frame 0xfe02390bfbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfe02390bfbf0

While it takes less than a minute for pfSense to reboot, the frequency
of it happening is starting to become a big nuisance.

Anyone has any idea what's going on? Restoring to pfSense 2.3 seems to
solve this problem, so it is more likely a software than hardware issue.

Warm regards,
Liwei
-- 
Clear Skies,LiweiCo-Founder, CTO

TinyMOS


 
 

21 Heng Mui Keng Terrace, Level 1 The Hangar, Singapore 119613
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold