Re: [pfSense] Restoring at remote location before deployment

2017-05-19 Thread Steven Spencer
Mark,

Thanks very much for the reply. We should be able to accomplish this
using this method as well.

Steve

On 05/18/2017 08:26 AM, Mark Wiater wrote:
>
>
> On 5/17/2017 3:44 PM, Steven Spencer wrote:
>> All,
>>
>> When restoring a configuration for a site, we often do so from the home
>> office and then deploy after we are sure hardware is working as
>> expected. That means that we are restoring a backup and then on reboot,
>> their is no active WAN (because the WAN IP is set for the remote
>> location on restore)
>
> I do the same, build up the system in the home office for testing.
>
> I've created a separate environment within the home office that allows
> for the unit under test to hit the internet after getting natted to
> the home office's ip addr.
>
>
>
>
>
>
>
>
> esses.
> ___
> pfSense mailing list
> https://lists.pfsense.org/mailman/listinfo/list
> Support the project with Gold! https://pfsense.org/gold
>

-- 
-- 
Steven G. Spencer, Network Administrator
KSC Corporate - The Kelly Supply Family of Companies
Office 308-382-8764 Ext. 1131
Mobile 402-765-8010 

___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold


Re: [pfSense] Network interruption on pfSense Firewall

2017-05-19 Thread Steve Yates
>> Can you try to set disable hardware offloading in pfsense advanced.
>What would that do?

In Xen, at least, it's basically necessary to get usable throughput 
from VirtIO adapters.  It also solved a slew of false positives in our Suricata 
setup on our previous virtualization, and I want to say on at least one 
physical server also since we have it disabled there.  Worst case it uses a bit 
more CPU and you can turn it back on pretty easily.

https://doc.pfsense.org/index.php/Lost_Traffic_/_Packets_Disappear
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#TSO.2FLRO

(
https://doc.pfsense.org/index.php/VirtIO_Driver_Support
Xen/KVM networking will not work using default hypervisor settings!: 
https://forum.pfsense.org/index.php?topic=88467.0
)

--

Steve Yates
ITS, Inc.
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold


Re: [pfSense] Network interruption on pfSense Firewall

2017-05-19 Thread Luca De Andreis
Il 19 maggio 2017 16:47:15 CEST, Ugo Bellavance  ha scritto:
>On 2017-05-19 10:22 AM, WebDawg wrote:
>
>>
>> If you have your router virtualized, there are CPU requirements for
>the
>> virtual NICs that I do not think you can see from 'inside'.
>>
>> You have to look from the hypervisor in.  It depends on how you have
>> everything configured and what virtualisation you are using.  Are you
>using
>> PCI passthrough to have a true nic?
>
>No, the NIC is shared with all the other VMS. 2 10G NICs in the
>physical 
>server.
>
>The hypervisor has a max of 20.62% CPU usage, average of 8% over one
>day.
>
>The total throughput of the hypervisor in a day is max 1.6 GBps, and it
>
>doesn't correspond to the dpigner logs. Max network traffic is between 
>11PM and 3 AM.
>
>___
>pfSense mailing list
>https://lists.pfsense.org/mailman/listinfo/list
>Support the project with Gold! https://pfsense.org/gold

Can you try to set disable hardware offloading in pfsense advanced.

Luca
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold


Re: [pfSense] Network interruption on pfSense Firewall

2017-05-19 Thread Ugo Bellavance

On 2017-05-19 10:22 AM, WebDawg wrote:



If you have your router virtualized, there are CPU requirements for the
virtual NICs that I do not think you can see from 'inside'.

You have to look from the hypervisor in.  It depends on how you have
everything configured and what virtualisation you are using.  Are you using
PCI passthrough to have a true nic?


No, the NIC is shared with all the other VMS. 2 10G NICs in the physical 
server.


The hypervisor has a max of 20.62% CPU usage, average of 8% over one day.

The total throughput of the hypervisor in a day is max 1.6 GBps, and it 
doesn't correspond to the dpigner logs. Max network traffic is between 
11PM and 3 AM.


___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold


Re: [pfSense] Network interruption on pfSense Firewall

2017-05-19 Thread Ugo Bellavance

On 2017-05-19 08:53 AM, J. Hellenthal wrote:

Interesting. I see this same thing on a SG2440 at one of our smaller 
installation sites with a dual gateway setup it experiences very similar 
likeness to the packet loss and high latency.

All firmware is up-to-date... netgate boot & pfsense.

Have not had the chance to look deeper into this as I believed it may be a 
problem on the remote end and the frequency of events were very quick and 
disappeared for greater than 24 hours at a time.



Are you using all your bandwidth when it happens?


___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold


Re: [pfSense] Network interruption on pfSense Firewall

2017-05-19 Thread Ugo Bellavance

On 2017-05-19 08:33 AM, Angel Rengifo Cancino wrote:

On Fri, May 19, 2017 at 6:55 AM, Ugo Bellavance  wrote:


Hi,

We sometimes experience what looks like service interruptions on our
pfSense firewall.  The first symptom was that we came in the office in the
morning and found that all the ssh sessions that were opened and going
through the firewall would be disconnected.

I searched the pfsense logs and I found that:

May 19 04:35:48 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2231us
stddev 1209us loss 21%
May 19 04:36:01 fw1 dpinger: ISP 206.55.90.97: Clear latency 2253us
stddev 1266us loss 15%
May 19 04:54:24 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2021us
stddev 1042us loss 22%
May 19 04:54:39 fw1 dpinger: ISP 206.55.90.97: Clear latency 2564us
stddev 6028us loss 19%
May 19 05:13:05 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2203us
stddev 1345us loss 21%
May 19 05:13:17 fw1 dpinger: ISP 206.55.90.97: Clear latency 2044us
stddev 870us loss 17%

I opened a ticket with mi ISP, but I don't think that they'll find
anything. I must say they they're not the most knowledgeable.

I've experienced such packet loss before and it was always ISP's fault. If

your bandwidth usage is not full then there should not be a reason for
lossing so many packets.


Our bandwidth usage is quite high when it happens.


According to the logs, everytime that happens, pfSense tries to do a few
things:

- Update dyndns
- Restart VPN tunnels
- Reload filters

I'll keep on searching but I really wonder wether the post-clear-latency
actions cause the SSH disconnects (and possibly other network cuts) or if
it's the firewall that is too busy to receive the ICMP packets.

Once I had the same problem with 2 ISPs configured in my pfSense box and

disabling this option helped me to avoid such disconnection behavior:

System -> Advanced -> Miscellaneous -> State Killing on gateway failure


Interesting. Why would it be a good idea to kill the states on a gateway 
failure?




You can try it.



The firewall runs on a VMWare VM,

Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz
3 CPUs: 1 package(s) x 3 core(s)
1 GB RAM

The host is not cpu-bound.



Make sure VMware is not part of the problem. If possible, use a physical
server to start a basic monitoring using continuous ping to see if packet
loss also occurs on this host. If it doesn't happen the same loss of
connectivity then maybe your VMware infrastructure might be part of the
problem.


That's not really feasible, unfortunately, but it's good advice.

Thanks,

___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold


Re: [pfSense] Network interruption on pfSense Firewall

2017-05-19 Thread WebDawg
On Fri, May 19, 2017 at 10:18 AM, Ugo Bellavance  wrote:

> On 2017-05-19 10:09 AM, WebDawg wrote:
>
>> On Fri, May 19, 2017 at 9:46 AM, Ugo Bellavance  wrote:
>>
>> On 2017-05-19 08:24 AM, WebDawg wrote:
>>>
>>> Thanks for your quick answer.
>>>
>>> I mean.  Your net connection is dripping packets...is your gateway going
>>>
 down?


>>> My external Nagios system saw nothing up to now (it always sees my
>>> gateway
>>> as up from the outside). But it only checks once every minute and the
>>> packet losses that I experience last about 15 seconds.  1/4 chance of
>>> seeing it when pooling every minute.
>>>
>>> Your ISP should do something...your WAN connection is going down...unless
>>>
 you have a bad VM config.


>>> The firewall has been up for 187 days and we've been using this VM since
>>> 2012. However, there is more and more traffic going through the VM as
>>> time
>>> goes by. This problem happened about 6 times in the past year, but 3 of
>>> them were in the past 2 weeks.
>>>
>>> pfSense does do SOMETHING when a gateway goes down...do you have failover
>>>
 internet setup?  When pfSense marks a connection as down and then back
 up,
 some of the things your are describing, I think, are supposed to happen.


>>> Only one WAN.
>>>
>>> You can adjust latency settings in the advanced settings of the gateway.
>>>
 You can adjust loss settings too.  Some ISP QoS configs I think are
 known
 to drop ICMP in favor of higher priority things.  In that case it is
 usually better to do your own QoS.


>>> That is interesting. I'll look into that.
>>>
>>> For some reason every T1 I have ever used had latent ICMP when loaded.  I
>>>
 tried so many different QoS configs but I could only get it so good.


>>> In our case it's an ethernet link provided on a gigabit GPON. 50 mbps.
>>> But
>>> I can see that the problem occurs when traffic is at 50 mbps (backups
>>> replication) so I lowered the maximum bandwidth for the replication to 43
>>> mbps.
>>>
>>> If the IPS's equipement ignores your QoS (and I think that's what they
>>> do), if they decide to drop some ICMP messages, what will your own QoS
>>> do?
>>>
>>>
>>>
>>> There are specific types of QoS that are designed to stop the ISP's QoS
>> from coming into play.  CODELQ was part of that.
>>
>> https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_
>> do_about_Bufferbloat/
>>
>> The general concept is to lower your max QoS speed to less then what the
>> max of your connection is for, but I always wondered how this would effect
>> things down the line, lets say if an ISP sells you 50mbits but then then
>> over provisions there back hauls.
>>
>
> That is approximately what I did. When we saturate the link, it is
> outboud, to a remote location where we have replicas of our backups. I have
> a limiter over there but it was either not working or not low enough. I
> lowered it more to avoid maxing out the pipe.
>
> There is also things that other ISP's have been caught doing in the past
>> like resetting torrent connections and such.
>>
>> I also would wonder about links that have, no QoS and what the default is
>> for things like that.  But that can be tested with iperf and ping over a
>> standard ethernet link I would guess.
>>
>> You should run iperf tests on your virtualized install while pinging and
>> watch your CPU load externally via your hypervisor.  I took a trip down
>> the
>> virtualized router path and I paid attention to 3 things.  Traffic shaping
>> support with PV type drivers, performance out of HVM drivers, and CPU
>> queues for virtual NICs when applicable.  I think the max I could get out
>> of the best VM choice with pfSense and a i3 processor was 100-300 mbits
>> and
>> some configurations would provide so little mbits it was laughable.
>>
>
> The thing is that this outbound traffic is going through a VPN tunnel so
> there is a CPU requirement for the encryption.
>
> pfSense graphs shows an average of all CPUs, but since we have only one
> VPN tunnel, I think that it cannot saturate all 3 vCPUs.
>
>
> __
>

If you have your router virtualized, there are CPU requirements for the
virtual NICs that I do not think you can see from 'inside'.

You have to look from the hypervisor in.  It depends on how you have
everything configured and what virtualisation you are using.  Are you using
PCI passthrough to have a true nic?
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold


Re: [pfSense] Network interruption on pfSense Firewall

2017-05-19 Thread Ugo Bellavance

On 2017-05-19 10:09 AM, WebDawg wrote:

On Fri, May 19, 2017 at 9:46 AM, Ugo Bellavance  wrote:


On 2017-05-19 08:24 AM, WebDawg wrote:

Thanks for your quick answer.

I mean.  Your net connection is dripping packets...is your gateway going

down?



My external Nagios system saw nothing up to now (it always sees my gateway
as up from the outside). But it only checks once every minute and the
packet losses that I experience last about 15 seconds.  1/4 chance of
seeing it when pooling every minute.

Your ISP should do something...your WAN connection is going down...unless

you have a bad VM config.



The firewall has been up for 187 days and we've been using this VM since
2012. However, there is more and more traffic going through the VM as time
goes by. This problem happened about 6 times in the past year, but 3 of
them were in the past 2 weeks.

pfSense does do SOMETHING when a gateway goes down...do you have failover

internet setup?  When pfSense marks a connection as down and then back up,
some of the things your are describing, I think, are supposed to happen.



Only one WAN.

You can adjust latency settings in the advanced settings of the gateway.

You can adjust loss settings too.  Some ISP QoS configs I think are known
to drop ICMP in favor of higher priority things.  In that case it is
usually better to do your own QoS.



That is interesting. I'll look into that.

For some reason every T1 I have ever used had latent ICMP when loaded.  I

tried so many different QoS configs but I could only get it so good.



In our case it's an ethernet link provided on a gigabit GPON. 50 mbps. But
I can see that the problem occurs when traffic is at 50 mbps (backups
replication) so I lowered the maximum bandwidth for the replication to 43
mbps.

If the IPS's equipement ignores your QoS (and I think that's what they
do), if they decide to drop some ICMP messages, what will your own QoS do?




There are specific types of QoS that are designed to stop the ISP's QoS
from coming into play.  CODELQ was part of that.

https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Bufferbloat/

The general concept is to lower your max QoS speed to less then what the
max of your connection is for, but I always wondered how this would effect
things down the line, lets say if an ISP sells you 50mbits but then then
over provisions there back hauls.


That is approximately what I did. When we saturate the link, it is 
outboud, to a remote location where we have replicas of our backups. I 
have a limiter over there but it was either not working or not low 
enough. I lowered it more to avoid maxing out the pipe.



There is also things that other ISP's have been caught doing in the past
like resetting torrent connections and such.

I also would wonder about links that have, no QoS and what the default is
for things like that.  But that can be tested with iperf and ping over a
standard ethernet link I would guess.

You should run iperf tests on your virtualized install while pinging and
watch your CPU load externally via your hypervisor.  I took a trip down the
virtualized router path and I paid attention to 3 things.  Traffic shaping
support with PV type drivers, performance out of HVM drivers, and CPU
queues for virtual NICs when applicable.  I think the max I could get out
of the best VM choice with pfSense and a i3 processor was 100-300 mbits and
some configurations would provide so little mbits it was laughable.


The thing is that this outbound traffic is going through a VPN tunnel so 
there is a CPU requirement for the encryption.


pfSense graphs shows an average of all CPUs, but since we have only one 
VPN tunnel, I think that it cannot saturate all 3 vCPUs.


___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold


Re: [pfSense] Network interruption on pfSense Firewall

2017-05-19 Thread WebDawg
On Fri, May 19, 2017 at 9:46 AM, Ugo Bellavance  wrote:

> On 2017-05-19 08:24 AM, WebDawg wrote:
>
> Thanks for your quick answer.
>
> I mean.  Your net connection is dripping packets...is your gateway going
>> down?
>>
>
> My external Nagios system saw nothing up to now (it always sees my gateway
> as up from the outside). But it only checks once every minute and the
> packet losses that I experience last about 15 seconds.  1/4 chance of
> seeing it when pooling every minute.
>
> Your ISP should do something...your WAN connection is going down...unless
>> you have a bad VM config.
>>
>
> The firewall has been up for 187 days and we've been using this VM since
> 2012. However, there is more and more traffic going through the VM as time
> goes by. This problem happened about 6 times in the past year, but 3 of
> them were in the past 2 weeks.
>
> pfSense does do SOMETHING when a gateway goes down...do you have failover
>> internet setup?  When pfSense marks a connection as down and then back up,
>> some of the things your are describing, I think, are supposed to happen.
>>
>
> Only one WAN.
>
> You can adjust latency settings in the advanced settings of the gateway.
>> You can adjust loss settings too.  Some ISP QoS configs I think are known
>> to drop ICMP in favor of higher priority things.  In that case it is
>> usually better to do your own QoS.
>>
>
> That is interesting. I'll look into that.
>
> For some reason every T1 I have ever used had latent ICMP when loaded.  I
>> tried so many different QoS configs but I could only get it so good.
>>
>
> In our case it's an ethernet link provided on a gigabit GPON. 50 mbps. But
> I can see that the problem occurs when traffic is at 50 mbps (backups
> replication) so I lowered the maximum bandwidth for the replication to 43
> mbps.
>
> If the IPS's equipement ignores your QoS (and I think that's what they
> do), if they decide to drop some ICMP messages, what will your own QoS do?
>
>
>
There are specific types of QoS that are designed to stop the ISP's QoS
from coming into play.  CODELQ was part of that.

https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Bufferbloat/

The general concept is to lower your max QoS speed to less then what the
max of your connection is for, but I always wondered how this would effect
things down the line, lets say if an ISP sells you 50mbits but then then
over provisions there back hauls.

There is also things that other ISP's have been caught doing in the past
like resetting torrent connections and such.

I also would wonder about links that have, no QoS and what the default is
for things like that.  But that can be tested with iperf and ping over a
standard ethernet link I would guess.

You should run iperf tests on your virtualized install while pinging and
watch your CPU load externally via your hypervisor.  I took a trip down the
virtualized router path and I paid attention to 3 things.  Traffic shaping
support with PV type drivers, performance out of HVM drivers, and CPU
queues for virtual NICs when applicable.  I think the max I could get out
of the best VM choice with pfSense and a i3 processor was 100-300 mbits and
some configurations would provide so little mbits it was laughable.
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold


Re: [pfSense] Network interruption on pfSense Firewall

2017-05-19 Thread Ugo Bellavance

On 2017-05-19 08:24 AM, WebDawg wrote:

Thanks for your quick answer.


I mean.  Your net connection is dripping packets...is your gateway going
down?


My external Nagios system saw nothing up to now (it always sees my 
gateway as up from the outside). But it only checks once every minute 
and the packet losses that I experience last about 15 seconds.  1/4 
chance of seeing it when pooling every minute.



Your ISP should do something...your WAN connection is going down...unless
you have a bad VM config.


The firewall has been up for 187 days and we've been using this VM since 
2012. However, there is more and more traffic going through the VM as 
time goes by. This problem happened about 6 times in the past year, but 
3 of them were in the past 2 weeks.



pfSense does do SOMETHING when a gateway goes down...do you have failover
internet setup?  When pfSense marks a connection as down and then back up,
some of the things your are describing, I think, are supposed to happen.


Only one WAN.


You can adjust latency settings in the advanced settings of the gateway.
You can adjust loss settings too.  Some ISP QoS configs I think are known
to drop ICMP in favor of higher priority things.  In that case it is
usually better to do your own QoS.


That is interesting. I'll look into that.


For some reason every T1 I have ever used had latent ICMP when loaded.  I
tried so many different QoS configs but I could only get it so good.


In our case it's an ethernet link provided on a gigabit GPON. 50 mbps. 
But I can see that the problem occurs when traffic is at 50 mbps 
(backups replication) so I lowered the maximum bandwidth for the 
replication to 43 mbps.


If the IPS's equipement ignores your QoS (and I think that's what they 
do), if they decide to drop some ICMP messages, what will your own QoS do?



___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold


Re: [pfSense] Network interruption on pfSense Firewall

2017-05-19 Thread WebDawg
Did you try a different gateway?

On Fri, May 19, 2017 at 8:53 AM, J. Hellenthal 
wrote:

> Interesting. I see this same thing on a SG2440 at one of our smaller
> installation sites with a dual gateway setup it experiences very similar
> likeness to the packet loss and high latency.
>
> All firmware is up-to-date... netgate boot & pfsense.
>
> Have not had the chance to look deeper into this as I believed it may be a
> problem on the remote end and the frequency of events were very quick and
> disappeared for greater than 24 hours at a time.
>
> --
>  Onward!,
>  Jason Hellenthal,
>  Systems & Network Admin,
>  Mobile: 0x9CA0BD58,
>  JJH48-ARIN
>
> On May 19, 2017, at 07:33, Angel Rengifo Cancino 
> wrote:
>
> On Fri, May 19, 2017 at 6:55 AM, Ugo Bellavance  wrote:
>
> > Hi,
> >
> > We sometimes experience what looks like service interruptions on our
> > pfSense firewall.  The first symptom was that we came in the office in
> the
> > morning and found that all the ssh sessions that were opened and going
> > through the firewall would be disconnected.
> >
> > I searched the pfsense logs and I found that:
> >
> > May 19 04:35:48 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2231us
> > stddev 1209us loss 21%
> > May 19 04:36:01 fw1 dpinger: ISP 206.55.90.97: Clear latency 2253us
> > stddev 1266us loss 15%
> > May 19 04:54:24 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2021us
> > stddev 1042us loss 22%
> > May 19 04:54:39 fw1 dpinger: ISP 206.55.90.97: Clear latency 2564us
> > stddev 6028us loss 19%
> > May 19 05:13:05 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2203us
> > stddev 1345us loss 21%
> > May 19 05:13:17 fw1 dpinger: ISP 206.55.90.97: Clear latency 2044us
> > stddev 870us loss 17%
> >
> > I opened a ticket with mi ISP, but I don't think that they'll find
> > anything. I must say they they're not the most knowledgeable.
> >
> > I've experienced such packet loss before and it was always ISP's fault.
> If
> your bandwidth usage is not full then there should not be a reason for
> lossing so many packets.
>
>
> >
> > According to the logs, everytime that happens, pfSense tries to do a few
> > things:
> >
> > - Update dyndns
> > - Restart VPN tunnels
> > - Reload filters
> >
> > I'll keep on searching but I really wonder wether the post-clear-latency
> > actions cause the SSH disconnects (and possibly other network cuts) or if
> > it's the firewall that is too busy to receive the ICMP packets.
> >
> > Once I had the same problem with 2 ISPs configured in my pfSense box and
> disabling this option helped me to avoid such disconnection behavior:
>
> System -> Advanced -> Miscellaneous -> State Killing on gateway failure
>
> You can try it.
>
>
> > The firewall runs on a VMWare VM,
> >
> > Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz
> > 3 CPUs: 1 package(s) x 3 core(s)
> > 1 GB RAM
> >
> > The host is not cpu-bound.
> >
> >
> Make sure VMware is not part of the problem. If possible, use a physical
> server to start a basic monitoring using continuous ping to see if packet
> loss also occurs on this host. If it doesn't happen the same loss of
> connectivity then maybe your VMware infrastructure might be part of the
> problem.
>
>
> *Angel Rengifo*
> *CEO*
> (51) 946-521-913
> (511) 6429706
> areng...@sfinetworks.com
> Visitanos en http:// www.sfinetworks.com
> ¿Buscas soporte? http://soporte.sfinetworks.com
> ___
> pfSense mailing list
> https://lists.pfsense.org/mailman/listinfo/list
> Support the project with Gold! https://pfsense.org/gold
> ___
> pfSense mailing list
> https://lists.pfsense.org/mailman/listinfo/list
> Support the project with Gold! https://pfsense.org/gold
>
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] Network interruption on pfSense Firewall

2017-05-19 Thread J. Hellenthal
Interesting. I see this same thing on a SG2440 at one of our smaller 
installation sites with a dual gateway setup it experiences very similar 
likeness to the packet loss and high latency.

All firmware is up-to-date... netgate boot & pfsense.

Have not had the chance to look deeper into this as I believed it may be a 
problem on the remote end and the frequency of events were very quick and 
disappeared for greater than 24 hours at a time.

-- 
 Onward!, 
 Jason Hellenthal, 
 Systems & Network Admin, 
 Mobile: 0x9CA0BD58, 
 JJH48-ARIN

On May 19, 2017, at 07:33, Angel Rengifo Cancino  
wrote:

On Fri, May 19, 2017 at 6:55 AM, Ugo Bellavance  wrote:

> Hi,
> 
> We sometimes experience what looks like service interruptions on our
> pfSense firewall.  The first symptom was that we came in the office in the
> morning and found that all the ssh sessions that were opened and going
> through the firewall would be disconnected.
> 
> I searched the pfsense logs and I found that:
> 
> May 19 04:35:48 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2231us
> stddev 1209us loss 21%
> May 19 04:36:01 fw1 dpinger: ISP 206.55.90.97: Clear latency 2253us
> stddev 1266us loss 15%
> May 19 04:54:24 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2021us
> stddev 1042us loss 22%
> May 19 04:54:39 fw1 dpinger: ISP 206.55.90.97: Clear latency 2564us
> stddev 6028us loss 19%
> May 19 05:13:05 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2203us
> stddev 1345us loss 21%
> May 19 05:13:17 fw1 dpinger: ISP 206.55.90.97: Clear latency 2044us
> stddev 870us loss 17%
> 
> I opened a ticket with mi ISP, but I don't think that they'll find
> anything. I must say they they're not the most knowledgeable.
> 
> I've experienced such packet loss before and it was always ISP's fault. If
your bandwidth usage is not full then there should not be a reason for
lossing so many packets.


> 
> According to the logs, everytime that happens, pfSense tries to do a few
> things:
> 
> - Update dyndns
> - Restart VPN tunnels
> - Reload filters
> 
> I'll keep on searching but I really wonder wether the post-clear-latency
> actions cause the SSH disconnects (and possibly other network cuts) or if
> it's the firewall that is too busy to receive the ICMP packets.
> 
> Once I had the same problem with 2 ISPs configured in my pfSense box and
disabling this option helped me to avoid such disconnection behavior:

System -> Advanced -> Miscellaneous -> State Killing on gateway failure

You can try it.


> The firewall runs on a VMWare VM,
> 
> Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz
> 3 CPUs: 1 package(s) x 3 core(s)
> 1 GB RAM
> 
> The host is not cpu-bound.
> 
> 
Make sure VMware is not part of the problem. If possible, use a physical
server to start a basic monitoring using continuous ping to see if packet
loss also occurs on this host. If it doesn't happen the same loss of
connectivity then maybe your VMware infrastructure might be part of the
problem.


*Angel Rengifo*
*CEO*
(51) 946-521-913
(511) 6429706
areng...@sfinetworks.com
Visitanos en http:// www.sfinetworks.com
¿Buscas soporte? http://soporte.sfinetworks.com
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] Network interruption on pfSense Firewall

2017-05-19 Thread Angel Rengifo Cancino
On Fri, May 19, 2017 at 6:55 AM, Ugo Bellavance  wrote:

> Hi,
>
> We sometimes experience what looks like service interruptions on our
> pfSense firewall.  The first symptom was that we came in the office in the
> morning and found that all the ssh sessions that were opened and going
> through the firewall would be disconnected.
>
> I searched the pfsense logs and I found that:
>
> May 19 04:35:48 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2231us
> stddev 1209us loss 21%
> May 19 04:36:01 fw1 dpinger: ISP 206.55.90.97: Clear latency 2253us
> stddev 1266us loss 15%
> May 19 04:54:24 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2021us
> stddev 1042us loss 22%
> May 19 04:54:39 fw1 dpinger: ISP 206.55.90.97: Clear latency 2564us
> stddev 6028us loss 19%
> May 19 05:13:05 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2203us
> stddev 1345us loss 21%
> May 19 05:13:17 fw1 dpinger: ISP 206.55.90.97: Clear latency 2044us
> stddev 870us loss 17%
>
> I opened a ticket with mi ISP, but I don't think that they'll find
> anything. I must say they they're not the most knowledgeable.
>
> I've experienced such packet loss before and it was always ISP's fault. If
your bandwidth usage is not full then there should not be a reason for
lossing so many packets.


>
> According to the logs, everytime that happens, pfSense tries to do a few
> things:
>
> - Update dyndns
> - Restart VPN tunnels
> - Reload filters
>
> I'll keep on searching but I really wonder wether the post-clear-latency
> actions cause the SSH disconnects (and possibly other network cuts) or if
> it's the firewall that is too busy to receive the ICMP packets.
>
> Once I had the same problem with 2 ISPs configured in my pfSense box and
disabling this option helped me to avoid such disconnection behavior:

System -> Advanced -> Miscellaneous -> State Killing on gateway failure

You can try it.


> The firewall runs on a VMWare VM,
>
> Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz
> 3 CPUs: 1 package(s) x 3 core(s)
> 1 GB RAM
>
> The host is not cpu-bound.
>
>
Make sure VMware is not part of the problem. If possible, use a physical
server to start a basic monitoring using continuous ping to see if packet
loss also occurs on this host. If it doesn't happen the same loss of
connectivity then maybe your VMware infrastructure might be part of the
problem.


*Angel Rengifo*
*CEO*
(51) 946-521-913
(511) 6429706
areng...@sfinetworks.com
Visitanos en http:// www.sfinetworks.com
¿Buscas soporte? http://soporte.sfinetworks.com
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] Network interruption on pfSense Firewall

2017-05-19 Thread WebDawg
I mean.  Your net connection is dripping packets...is your gateway going
down?

Your ISP should do something...your WAN connection is going down...unless
you have a bad VM config.

pfSense does do SOMETHING when a gateway goes down...do you have failover
internet setup?  When pfSense marks a connection as down and then back up,
some of the things your are describing, I think, are supposed to happen.

You can adjust latency settings in the advanced settings of the gateway.
You can adjust loss settings too.  Some ISP QoS configs I think are known
to drop ICMP in favor of higher priority things.  In that case it is
usually better to do your own QoS.

For some reason every T1 I have ever used had latent ICMP when loaded.  I
tried so many different QoS configs but I could only get it so good.



On May 19, 2017 7:56 AM, "Ugo Bellavance"  wrote:

Hi,

We sometimes experience what looks like service interruptions on our
pfSense firewall.  The first symptom was that we came in the office in the
morning and found that all the ssh sessions that were opened and going
through the firewall would be disconnected.

I searched the pfsense logs and I found that:

May 19 04:35:48 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2231us stddev
1209us loss 21%
May 19 04:36:01 fw1 dpinger: ISP 206.55.90.97: Clear latency 2253us stddev
1266us loss 15%
May 19 04:54:24 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2021us stddev
1042us loss 22%
May 19 04:54:39 fw1 dpinger: ISP 206.55.90.97: Clear latency 2564us stddev
6028us loss 19%
May 19 05:13:05 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2203us stddev
1345us loss 21%
May 19 05:13:17 fw1 dpinger: ISP 206.55.90.97: Clear latency 2044us stddev
870us loss 17%

I opened a ticket with mi ISP, but I don't think that they'll find
anything. I must say they they're not the most knowledgeable.


According to the logs, everytime that happens, pfSense tries to do a few
things:

- Update dyndns
- Restart VPN tunnels
- Reload filters

I'll keep on searching but I really wonder wether the post-clear-latency
actions cause the SSH disconnects (and possibly other network cuts) or if
it's the firewall that is too busy to receive the ICMP packets.

The firewall runs on a VMWare VM,

Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz
3 CPUs: 1 package(s) x 3 core(s)
1 GB RAM

The host is not cpu-bound.

Any advice would be appreciated.

Thanks,

Ugo

___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold


[pfSense] Network interruption on pfSense Firewall

2017-05-19 Thread Ugo Bellavance

Hi,

We sometimes experience what looks like service interruptions on our 
pfSense firewall.  The first symptom was that we came in the office in 
the morning and found that all the ssh sessions that were opened and 
going through the firewall would be disconnected.


I searched the pfsense logs and I found that:

May 19 04:35:48 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2231us 
stddev 1209us loss 21%
May 19 04:36:01 fw1 dpinger: ISP 206.55.90.97: Clear latency 2253us 
stddev 1266us loss 15%
May 19 04:54:24 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2021us 
stddev 1042us loss 22%
May 19 04:54:39 fw1 dpinger: ISP 206.55.90.97: Clear latency 2564us 
stddev 6028us loss 19%
May 19 05:13:05 fw1 dpinger: ISP 206.55.90.97: Alarm latency 2203us 
stddev 1345us loss 21%
May 19 05:13:17 fw1 dpinger: ISP 206.55.90.97: Clear latency 2044us 
stddev 870us loss 17%


I opened a ticket with mi ISP, but I don't think that they'll find 
anything. I must say they they're not the most knowledgeable.



According to the logs, everytime that happens, pfSense tries to do a few 
things:


- Update dyndns
- Restart VPN tunnels
- Reload filters

I'll keep on searching but I really wonder wether the post-clear-latency 
actions cause the SSH disconnects (and possibly other network cuts) or 
if it's the firewall that is too busy to receive the ICMP packets.


The firewall runs on a VMWare VM,

Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz
3 CPUs: 1 package(s) x 3 core(s)
1 GB RAM

The host is not cpu-bound.

Any advice would be appreciated.

Thanks,

Ugo

___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold