Re: [pfSense] pfSense IP stack crashing.
do you have configurate limiters? https://redmine.pfsense.org/issues/4310 - Mensaje original - De: "Bryant Zimmerman" Para: "pfSense Support and Discussion Mailing List" Enviados: Jueves, 15 de Octubre 2015 20:04:51 Asunto: Re: [pfSense] pfSense IP stack crashing. WebDawg I am using an industrial 4GB USB DOM. It is less than 3 years old so I am surprised that it could be the issue, but hey it's technology so I should not be surprised. Thanks Bryant From: "WebDawg" Sent: Thursday, October 15, 2015 1:55 PM To: "pfSense Support and Discussion Mailing List" Subject: Re: [pfSense] pfSense IP stack crashing. On Thu, Oct 15, 2015 at 7:45 AM, wrote: > Hmh, > > 3 things you could try come up to my mind. > > 1. I'd try another SD-Card (if you are using nanoBSD, my guess is, that > you use an SD-Card?). Put the Master in permanent maintenance mode and shut > it down, take out the SD-Card and check for errors. Even if there are none, > copy the card and use the new one. > 2. Freezing normally could mean bad memory. Did you try a live CD and a > mem stress test for at least 24 hours? If not, do that too. > 3. Unusual and also very unlikely but maybe your box got hacked somehow? > Turn on the remote logging feature and log your messages to another > syslogd-Server and see what you get when the system gets unresponsive. > > HTH, > > Jens Simmoleit > Senior Linux Systems Administrator > > infoscore Profile Tracking GmbH > part of arvato Financial Solutions > Kaistrasse 7 > 40211 Düsseldorf > > Phone: +49 211 50 66 51- 88 > Fax: +49 211 50 66 51- 93 > Mobile: +49 160 97 80 46 94 > > Better yet, can you post the specs/detailed hardware of the system? Is it SD or CF media? ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] pfSense IP stack crashing.
WebDawg I am using an industrial 4GB USB DOM. It is less than 3 years old so I am surprised that it could be the issue, but hey it's technology so I should not be surprised. Thanks Bryant From: "WebDawg" Sent: Thursday, October 15, 2015 1:55 PM To: "pfSense Support and Discussion Mailing List" Subject: Re: [pfSense] pfSense IP stack crashing. On Thu, Oct 15, 2015 at 7:45 AM, wrote: > Hmh, > > 3 things you could try come up to my mind. > > 1. I'd try another SD-Card (if you are using nanoBSD, my guess is, that > you use an SD-Card?). Put the Master in permanent maintenance mode and shut > it down, take out the SD-Card and check for errors. Even if there are none, > copy the card and use the new one. > 2. Freezing normally could mean bad memory. Did you try a live CD and a > mem stress test for at least 24 hours? If not, do that too. > 3. Unusual and also very unlikely but maybe your box got hacked somehow? > Turn on the remote logging feature and log your messages to another > syslogd-Server and see what you get when the system gets unresponsive. > > HTH, > > Jens Simmoleit > Senior Linux Systems Administrator > > infoscore Profile Tracking GmbH > part of arvato Financial Solutions > Kaistrasse 7 > 40211 Düsseldorf > > Phone: +49 211 50 66 51- 88 > Fax: +49 211 50 66 51- 93 > Mobile: +49 160 97 80 46 94 > > Better yet, can you post the specs/detailed hardware of the system? Is it SD or CF media? ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] pfSense IP stack crashing.
Yeah, that sounds like the right path. Original post mentioned DOM, which I don’t understand. I don’t know that spin-rite has any value on SSD. I would be inclined to do a fresh OS install and import the configuration to eliminate data bit rot and hacking of the OS as possible problems. I had a mangled config that caused instability for me 6 months ago, and after new hardware and OS I finally just rebuilt the config from scratch and everything was fine. This was after a year of service and no significant changes on nano-BSD on compact-flash. Although a failing NIC can throw OS crashing nonsense and wouldn’t show up unless under load. It doesn’t sound like you have the option to swap out network cards but that is as likely the problem as anything else. ED. > On 2015, Oct 15, at 7:45 AM, jens.simmol...@bertelsmann.de wrote: > > Hmh, > > 3 things you could try come up to my mind. > > 1. I'd try another SD-Card (if you are using nanoBSD, my guess is, that you > use an SD-Card?). Put the Master in permanent maintenance mode and shut it > down, take out the SD-Card and check for errors. Even if there are none, copy > the card and use the new one. > 2. Freezing normally could mean bad memory. Did you try a live CD and a mem > stress test for at least 24 hours? If not, do that too. > 3. Unusual and also very unlikely but maybe your box got hacked somehow? Turn > on the remote logging feature and log your messages to another syslogd-Server > and see what you get when the system gets unresponsive. > > HTH, > > Jens Simmoleit > Senior Linux Systems Administrator > > infoscore Profile Tracking GmbH > part of arvato Financial Solutions > Kaistrasse 7 > 40211 Düsseldorf > > Phone:+49 211 50 66 51- 88 > Fax: +49 211 50 66 51- 93 > Mobile: +49 160 97 80 46 94 > > E-Mail: jens.simmol...@bertelsmann.de > > finance.arvato.com > > -- > infoscore Profile Tracking GmbH | Sitz: Düsseldorf I Amtsgericht HRB > Gütersloh 9368 | USt-IDNr.: DE 287843415 I > Geschäftsführer: Kai Kalchthaler, Matthias Schweizer > -- > Diese E-Mail und eventuelle Anlagen können vertrauliche und/oder rechtlich > geschützte Informationen enthalten. Wenn Sie nicht der > richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, > informieren Sie bitte sofort den Absender und vernichten Sie diese > E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail > sind nicht gestattet. > -- > This e-mail and any attachments may contain confidential and/or privileged > information. If you are not the intended recipient (or have > received this e-mail in error) please notify the sender immediately and > destroy this e-mail. Any unauthorized copying, disclosure or > distribution of the material in this e-mail is forbidden. > -- > Bitte denken Sie über Ihre Verantwortung gegenüber der Umwelt nach, bevor Sie > diese E-Mail ausdrucken! > > -Ursprüngliche Nachricht- > Von: List [mailto:list-boun...@lists.pfsense.org] Im Auftrag von WebDawg > Gesendet: Donnerstag, 15. Oktober 2015 01:10 > An: pfSense Support and Discussion Mailing List; brya...@zktech.com > Betreff: Re: [pfSense] pfSense IP stack crashing. > > On Oct 12, 2015 2:27 PM, "Bryant Zimmerman" wrote: >> >> I have two routers in an CARP stack. >> The primary yesterday started crashing it's IP stack. >> Things run for a bit of time and then all IP's become unresponsive. The >> secondary pfSense box takes over the VIP's and things keep running. Now > for >> some reason the 1st box will reboot some times. Other times it just hangs >> there. If I connect in via IPMI I can see the box is still up I can get >> console access. I can reboot the box and it comes back up until it hangs >> again. >> >> I ran a spin rite on the DOM running the OS and ran a memory and mother >> board test, and am not coming up with any errors there. >> I am on 2.2.4-Release (nano bsd) >> >> Any ideas
Re: [pfSense] pfSense IP stack crashing.
Jens Thank you for the ideas. I am leaning towards the DOM (storage) The lock up seems to occur after about 15 min. The odd thing is serial console does not lock up. I just don't see anything in the logs to explain this. Bryant From: jens.simmol...@bertelsmann.de Sent: Thursday, October 15, 2015 10:52 AM To: list@lists.pfsense.org Subject: Re: [pfSense] pfSense IP stack crashing. Hmh, 3 things you could try come up to my mind. 1. I'd try another SD-Card (if you are using nanoBSD, my guess is, that you use an SD-Card?). Put the Master in permanent maintenance mode and shut it down, take out the SD-Card and check for errors. Even if there are none, copy the card and use the new one. 2. Freezing normally could mean bad memory. Did you try a live CD and a mem stress test for at least 24 hours? If not, do that too. 3. Unusual and also very unlikely but maybe your box got hacked somehow? Turn on the remote logging feature and log your messages to another syslogd-Server and see what you get when the system gets unresponsive. HTH, Jens Simmoleit Senior Linux Systems Administrator infoscore Profile Tracking GmbH part of arvato Financial Solutions Kaistrasse 7 40211 Düsseldorf Phone: +49 211 50 66 51- 88 Fax: +49 211 50 66 51- 93 Mobile: +49 160 97 80 46 94 E-Mail: jens.simmol...@bertelsmann.de finance.arvato.com -- infoscore Profile Tracking GmbH | Sitz: Düsseldorf I Amtsgericht HRB Gütersloh 9368 | USt-IDNr.: DE 287843415 I Geschäftsführer: Kai Kalchthaler, Matthias Schweizer -- Diese E-Mail und eventuelle Anlagen können vertrauliche und/oder rechtlich geschützte Informationen enthalten. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail sind nicht gestattet. -- This e-mail and any attachments may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is forbidden. -- Bitte denken Sie über Ihre Verantwortung gegenüber der Umwelt nach, bevor Sie diese E-Mail ausdrucken! -Ursprüngliche Nachricht- Von: List [mailto:list-boun...@lists.pfsense.org] Im Auftrag von WebDawg Gesendet: Donnerstag, 15. Oktober 2015 01:10 An: pfSense Support and Discussion Mailing List; brya...@zktech.com Betreff: Re: [pfSense] pfSense IP stack crashing. On Oct 12, 2015 2:27 PM, "Bryant Zimmerman" wrote: > > I have two routers in an CARP stack. > The primary yesterday started crashing it's IP stack. > Things run for a bit of time and then all IP's become unresponsive. The > secondary pfSense box takes over the VIP's and things keep running. Now for > some reason the 1st box will reboot some times. Other times it just hangs > there. If I connect in via IPMI I can see the box is still up I can get > console access. I can reboot the box and it comes back up until it hangs > again. > > I ran a spin rite on the DOM running the OS and ran a memory and mother > board test, and am not coming up with any errors there. > I am on 2.2.4-Release (nano bsd) > > Any ideas would be appreciated. This units has been stable for 3 years > only rebooted when upgrades occur. This is so out of character for this box > and I need to figure this out ASAP. > > Thanks for any ideas you can offer. > > Bryant When you ipmi in...and get shell access. No messages in the logs? ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] pfSense IP stack crashing.
On Thu, Oct 15, 2015 at 7:45 AM, wrote: > Hmh, > > 3 things you could try come up to my mind. > > 1. I'd try another SD-Card (if you are using nanoBSD, my guess is, that > you use an SD-Card?). Put the Master in permanent maintenance mode and shut > it down, take out the SD-Card and check for errors. Even if there are none, > copy the card and use the new one. > 2. Freezing normally could mean bad memory. Did you try a live CD and a > mem stress test for at least 24 hours? If not, do that too. > 3. Unusual and also very unlikely but maybe your box got hacked somehow? > Turn on the remote logging feature and log your messages to another > syslogd-Server and see what you get when the system gets unresponsive. > > HTH, > > Jens Simmoleit > Senior Linux Systems Administrator > > infoscore Profile Tracking GmbH > part of arvato Financial Solutions > Kaistrasse 7 > 40211 Düsseldorf > > Phone: +49 211 50 66 51- 88 > Fax:+49 211 50 66 51- 93 > Mobile: +49 160 97 80 46 94 > > Better yet, can you post the specs/detailed hardware of the system? Is it SD or CF media? ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] pfSense IP stack crashing.
Hmh, 3 things you could try come up to my mind. 1. I'd try another SD-Card (if you are using nanoBSD, my guess is, that you use an SD-Card?). Put the Master in permanent maintenance mode and shut it down, take out the SD-Card and check for errors. Even if there are none, copy the card and use the new one. 2. Freezing normally could mean bad memory. Did you try a live CD and a mem stress test for at least 24 hours? If not, do that too. 3. Unusual and also very unlikely but maybe your box got hacked somehow? Turn on the remote logging feature and log your messages to another syslogd-Server and see what you get when the system gets unresponsive. HTH, Jens Simmoleit Senior Linux Systems Administrator infoscore Profile Tracking GmbH part of arvato Financial Solutions Kaistrasse 7 40211 Düsseldorf Phone: +49 211 50 66 51- 88 Fax:+49 211 50 66 51- 93 Mobile: +49 160 97 80 46 94 E-Mail: jens.simmol...@bertelsmann.de finance.arvato.com -- infoscore Profile Tracking GmbH | Sitz: Düsseldorf I Amtsgericht HRB Gütersloh 9368 | USt-IDNr.: DE 287843415 I Geschäftsführer: Kai Kalchthaler, Matthias Schweizer -- Diese E-Mail und eventuelle Anlagen können vertrauliche und/oder rechtlich geschützte Informationen enthalten. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail sind nicht gestattet. -- This e-mail and any attachments may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is forbidden. -- Bitte denken Sie über Ihre Verantwortung gegenüber der Umwelt nach, bevor Sie diese E-Mail ausdrucken! -Ursprüngliche Nachricht- Von: List [mailto:list-boun...@lists.pfsense.org] Im Auftrag von WebDawg Gesendet: Donnerstag, 15. Oktober 2015 01:10 An: pfSense Support and Discussion Mailing List; brya...@zktech.com Betreff: Re: [pfSense] pfSense IP stack crashing. On Oct 12, 2015 2:27 PM, "Bryant Zimmerman" wrote: > > I have two routers in an CARP stack. > The primary yesterday started crashing it's IP stack. > Things run for a bit of time and then all IP's become unresponsive. The > secondary pfSense box takes over the VIP's and things keep running. Now for > some reason the 1st box will reboot some times. Other times it just hangs > there. If I connect in via IPMI I can see the box is still up I can get > console access. I can reboot the box and it comes back up until it hangs > again. > > I ran a spin rite on the DOM running the OS and ran a memory and mother > board test, and am not coming up with any errors there. > I am on 2.2.4-Release (nano bsd) > > Any ideas would be appreciated. This units has been stable for 3 years > only rebooted when upgrades occur. This is so out of character for this box > and I need to figure this out ASAP. > > Thanks for any ideas you can offer. > > Bryant When you ipmi in...and get shell access. No messages in the logs? ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] pfSense IP stack crashing.
On Oct 12, 2015 2:27 PM, "Bryant Zimmerman" wrote: > > I have two routers in an CARP stack. > The primary yesterday started crashing it's IP stack. > Things run for a bit of time and then all IP's become unresponsive. The > secondary pfSense box takes over the VIP's and things keep running. Now for > some reason the 1st box will reboot some times. Other times it just hangs > there. If I connect in via IPMI I can see the box is still up I can get > console access. I can reboot the box and it comes back up until it hangs > again. > > I ran a spin rite on the DOM running the OS and ran a memory and mother > board test, and am not coming up with any errors there. > I am on 2.2.4-Release (nano bsd) > > Any ideas would be appreciated. This units has been stable for 3 years > only rebooted when upgrades occur. This is so out of character for this box > and I need to figure this out ASAP. > > Thanks for any ideas you can offer. > > Bryant When you ipmi in...and get shell access. No messages in the logs? ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] pfSense IP stack crashing.
On Wed, Oct 7, 2015 at 8:20 AM, Bryant Zimmerman wrote: > Any ideas would be appreciated. This units has been stable for 3 years > only rebooted when upgrades occur. This is so out of character for this box > and I need to figure this out ASAP. > I will vote hardware failure, possibly intermittent. Diagnostics don't always pick up everything. Many times it is the power supply, but it could be the NIC itself. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
[pfSense] pfSense IP stack crashing.
I have two routers in an CARP stack. The primary yesterday started crashing it's IP stack. Things run for a bit of time and then all IP's become unresponsive. The secondary pfSense box takes over the VIP's and things keep running. Now for some reason the 1st box will reboot some times. Other times it just hangs there. If I connect in via IPMI I can see the box is still up I can get console access. I can reboot the box and it comes back up until it hangs again. I ran a spin rite on the DOM running the OS and ran a memory and mother board test, and am not coming up with any errors there. I am on 2.2.4-Release (nano bsd) Any ideas would be appreciated. This units has been stable for 3 years only rebooted when upgrades occur. This is so out of character for this box and I need to figure this out ASAP. Thanks for any ideas you can offer. Bryant ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold