> On 20 Apr 2017, at 13:00, Sjöholm Per-Olov <p...@incedo.org> wrote:
>
>
>> On 20 Apr 2017, at 01:18, Sjöholm Per-Olov <p...@incedo.org> wrote:
>>
>>
>>> On 20 Apr 2017, at 00:39, Fred <open...@crowsons.com> wrote:
>>>
>>> On 04/19/17 23:30, Sjöholm Per-Olov wrote:
>>>> Anyone with a clue would be _very_ much appreciated….
>>>> I upgraded from 6.0 to 6.1 two days ago and **did not change anything to
>>>> the network** stuff at all. After that clients have random problems
>>>> reaching my dmz web server (centos + nginx). I have checked the release
>>>> notes, but could not see any clue there. Se logs below
>>>> # Relevant rules from PF
>>>> LAN_INT="vlan2"
>>>> DMZ1_INT="vlan3"
>>>> DMZ2_INT="vlan4"
>>>> GUEST_INT="vlan1003"
>>>> INTERNET_INT="em3
>>>> ALL_INTERFACES="{" $LAN_INT $GUEST_INT $DMZ1_INT $DMZ2_INT $INTERNET_INT
>>>> "}"
>>>> pass out on $ALL_INTERFACES inet proto {tcp gre esp udp icmp ipv6} all
>>>> keep state
>>>> pass out on $ALL_INTERFACES inet6 proto {tcp gre esp udp icmp6} all keep
>>>> state
>>>> pass out on $IPV6_TUNNEL_INT inet6 all keep state
>>>> pass in log quick on $INTERNET_INT inet proto tcp from any to
>>>> $DMZ1_DAEDALUS port { 80 443 } label "webstats:$dstport" flags S/SAFR
>>>> keep state (max-src-nodes 90, max-src-states 150, max-src-conn 150,
>>>> max-src-conn-rate 250/30, overload <bad_hosts> flush global)
>>>> # Log that after upgrade shows problems in the logs related to this
>>>> directly after the upgrade
>>>> root@xanadu:/var/log#tcpdump -e -n -ttt -r /var/log/pflog.6|grep
>>>> block|grep 155.4|grep out |grep ': R'
>>>> tcpdump: WARNING: snaplen raised from 116 to 160
>>>> root@xanadu:/var/log#tcpdump -e -n -ttt -r /var/log/pflog.5|grep
>>>> block|grep 155.4|grep out |grep ': R'
>>>> tcpdump: WARNING: snaplen raised from 116 to 160
>>>> root@xanadu:/var/log#tcpdump -e -n -ttt -r /var/log/pflog.4|grep
>>>> block|grep 155.4|grep out |grep ': R'
>>>> tcpdump: WARNING: snaplen raised from 116 to 160
>>>> root@xanadu:/var/log#tcpdump -e -n -ttt -r /var/log/pflog.3|grep
>>>> block|grep 155.4|grep out |grep ': R'
>>>> tcpdump: WARNING: snaplen raised from 116 to 160
>>>> root@xanadu:/var/log#tcpdump -e -n -ttt -r /var/log/pflog.2|grep
>>>> block|grep 155.4|grep out |grep ': R'
>>>> tcpdump: WARNING: snaplen raised from 116 to 160
>>>> Apr 17 05:43:36.359067 rule 63/(match) block out on em3: 155.4.8.28.80 >
>>>> 164.132.161.92.46942: R 0:0(0) ack 2697518940 win 0 (DF)
>>>> Apr 17 05:43:37.358688 rule 63/(match) block out on em3: 155.4.8.28.80 >
>>>> 164.132.161.92.46942: R 0:0(0) ack 1 win 0 (DF)
>>>> Apr 17 05:43:39.362671 rule 63/(match) block out on em3: 155.4.8.28.80 >
>>>> 164.132.161.92.46942: R 0:0(0) ack 1 win 0 (DF)
>>>> Apr 17 06:10:24.490412 rule 63/(match) block out on em3: 155.4.8.28.80 >
>>>> 139.162.111.147.33930: R 0:0(0) ack 1409896759 win 0 (DF)
>>>> Apr 17 06:32:45.198754 rule 63/(match) block out on em3: 155.4.8.28.80 >
>>>> 180.76.15.26.42835: R 0:0(0) ack 3718886589 win 0 (DF)
>>>> Apr 17 06:32:46.198338 rule 63/(match) block out on em3: 155.4.8.28.80 >
>>>> 180.76.15.26.42835: R 0:0(0) ack 1 win 0 (DF)
>>>> Apr 17 06:41:29.366359 rule 63/(match) block out on em3: 155.4.8.28.80 >
>>>> 51.255.65.91.42819: R 0:0(0) ack 4294673273 win 0 (DF)
>>>> Apr 17 06:41:30.365396 rule 63/(match) block out on em3: 155.4.8.28.80 >
>>>> 51.255.65.91.42819: R 0:0(0) ack 1 win 0 (DF)
>>>> Apr 17 06:41:32.369399 rule 63/(match) block out on em3: 155.4.8.28.80 >
>>>> 51.255.65.91.42819: R 0:0(0) ack 1 win 0 (DF)
>>>> — cut the rest —
>>>> What have I missed?
>>>> Tnx in advance
>>>> Peo
>>>> Thanks
>>>> Peo
>>>
>>> You might get some clues from:
>>>
>>> pfctl -sr -R 63
>>>
>>> Cheers
>>>
>>> Fred
>>>
>>
>> I know that that rule is my default block…
>>
>> root@xanadu:/etc#pfctl -g -sr|grep "@63"
>> @63 block drop log all
>> root@xanadu:/etc#
>>
>> But why is this happening after upgrade. I have netiher touched pf.conf,
>> sysctl.conf or /etc/hostname* nor found any changes in release notes
>> related to this. So I see no reason for the packet to get stuck on that
>> rule. But I am probably missing something obvious :)
>>
>>
>> Peo
>>
>
>
> This is a tricky one…. I would very much appreciate “pro” help :)
>
> It seems a reload with pfctl -f /etc/pf.conf make these blocked packages go
> away for two hours or so (and everything is working). After that the problem
> comes back and I again see frequent intermittent block out on the internet
> interface.
>
> root@xanadu:/etc#tcpdump -e -n -ttt -r /var/log/pflog|grep block|grep
> 155.4|grep out |grep ': R'|tail -20
> tcpdump: WARNING: snaplen raised from 116 to 160
> Apr 20 12:34:18.052265 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 194.71.64.14.5910: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:34:21.249292 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 194.71.64.14.5910: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:34:24.449235 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 194.71.64.14.5910: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:34:27.649446 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 194.71.64.14.5910: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:34:30.849380 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 194.71.64.14.5910: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:34:37.049703 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 194.71.64.14.5910: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:34:49.253624 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 194.71.64.14.5910: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:35:13.449340 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 194.71.64.14.5910: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:35:19.053693 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 41.78.84.231.50310: R 0:0(0) ack 3874760854 win 0 (DF)
> Apr 20 12:35:22.048103 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 41.78.84.231.50310: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:35:28.045309 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 41.78.84.231.50310: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:37:15.284115 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 112.215.201.32.20948: R 0:0(0) ack 533408652 win 0 (DF)
> Apr 20 12:37:18.302353 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 112.215.201.32.20948: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:37:24.294055 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 112.215.201.32.20948: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:38:24.998973 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 209.85.218.51.33714: R 0:0(0) ack 3754971547 win 0 (DF)
> Apr 20 12:38:25.999057 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 209.85.218.51.33714: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:38:27.999024 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 209.85.218.51.33714: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:38:31.998695 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 209.85.218.51.33714: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:38:39.998591 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 209.85.218.51.33714: R 0:0(0) ack 1 win 0 (DF)
> Apr 20 12:38:55.998538 rule 63/(match) block out on em3: 155.4.8.28.25 >
> 209.85.218.51.33714: R 0:0(0) ack 1 win 0 (DF)
> root@xanadu:/etc#
>
>
> Could it be any buffers that is causing this in 6.1 but not in 6.0 ?
>
>
>
> Peo
>
>
Now it has worked from 12:38, but started to cause problems again at 16:15.
Have not done anything during this time frame. Here is a snippet from the
command:
tcpdump -e -n -ttt -r /var/log/pflog|grep block|grep 155.4|grep out |grep ': R'
where we can see that I had approximately 3 hours and 36 minutes without a
problem between 12:38 and 16:15. This drives me crazy!
Why did these errors come after several hours? This happens to both smtp as
well as webb traffic.
Any suggestions what to look for would be very much appreciated...
—snip--
Apr 20 12:34:37.049703 rule 63/(match) block out on em3: 155.4.8.28.25 >
194.71.64.14.5910: R 0:0(0) ack 1 win 0 (DF)
Apr 20 12:34:49.253624 rule 63/(match) block out on em3: 155.4.8.28.25 >
194.71.64.14.5910: R 0:0(0) ack 1 win 0 (DF)
Apr 20 12:35:13.449340 rule 63/(match) block out on em3: 155.4.8.28.25 >
194.71.64.14.5910: R 0:0(0) ack 1 win 0 (DF)
Apr 20 12:35:19.053693 rule 63/(match) block out on em3: 155.4.8.28.25 >
41.78.84.231.50310: R 0:0(0) ack 3874760854 win 0 (DF)
Apr 20 12:35:22.048103 rule 63/(match) block out on em3: 155.4.8.28.25 >
41.78.84.231.50310: R 0:0(0) ack 1 win 0 (DF)
Apr 20 12:35:28.045309 rule 63/(match) block out on em3: 155.4.8.28.25 >
41.78.84.231.50310: R 0:0(0) ack 1 win 0 (DF)
Apr 20 12:37:15.284115 rule 63/(match) block out on em3: 155.4.8.28.25 >
112.215.201.32.20948: R 0:0(0) ack 533408652 win 0 (DF)
Apr 20 12:37:18.302353 rule 63/(match) block out on em3: 155.4.8.28.25 >
112.215.201.32.20948: R 0:0(0) ack 1 win 0 (DF)
Apr 20 12:37:24.294055 rule 63/(match) block out on em3: 155.4.8.28.25 >
112.215.201.32.20948: R 0:0(0) ack 1 win 0 (DF)
Apr 20 12:38:24.998973 rule 63/(match) block out on em3: 155.4.8.28.25 >
209.85.218.51.33714: R 0:0(0) ack 3754971547 win 0 (DF)
Apr 20 12:38:25.999057 rule 63/(match) block out on em3: 155.4.8.28.25 >
209.85.218.51.33714: R 0:0(0) ack 1 win 0 (DF)
Apr 20 12:38:27.999024 rule 63/(match) block out on em3: 155.4.8.28.25 >
209.85.218.51.33714: R 0:0(0) ack 1 win 0 (DF)
Apr 20 12:38:31.998695 rule 63/(match) block out on em3: 155.4.8.28.25 >
209.85.218.51.33714: R 0:0(0) ack 1 win 0 (DF)
Apr 20 12:38:39.998591 rule 63/(match) block out on em3: 155.4.8.28.25 >
209.85.218.51.33714: R 0:0(0) ack 1 win 0 (DF)
Apr 20 12:38:55.998538 rule 63/(match) block out on em3: 155.4.8.28.25 >
209.85.218.51.33714: R 0:0(0) ack 1 win 0 (DF)
Apr 20 16:15:09.279725 rule 63/(match) block out on em3: 155.4.8.28.25 >
181.234.142.153.1954: R 0:0(0) ack 2524568547 win 0 (DF)
Apr 20 16:15:10.278863 rule 63/(match) block out on em3: 155.4.8.28.25 >
190.43.107.62.50622: R 0:0(0) ack 892069347 win 0 (DF)
Apr 20 16:15:12.330771 rule 63/(match) block out on em3: 155.4.8.28.25 >
181.234.142.153.1954: R 0:0(0) ack 1 win 0 (DF)
Apr 20 16:15:13.276805 rule 63/(match) block out on em3: 155.4.8.28.25 >
190.43.107.62.50622: R 0:0(0) ack 1 win 0 (DF)
Apr 20 16:15:18.235835 rule 63/(match) block out on em3: 155.4.8.28.25 >
181.234.142.153.1954: R 0:0(0) ack 1 win 0 (DF)
Apr 20 16:15:19.284027 rule 63/(match) block out on em3: 155.4.8.28.25 >
190.43.107.62.50622: R 0:0(0) ack 1 win 0 (DF)
Apr 20 16:17:54.217582 rule 63/(match) block out on em3: 155.4.8.28.25 >
119.30.45.76.2544: R 0:0(0) ack 3955486675 win 0 (DF)
Apr 20 16:17:57.197196 rule 63/(match) block out on em3: 155.4.8.28.25 >
119.30.45.76.2544: R 0:0(0) ack 1 win 0 (DF)
Apr 20 16:18:03.227666 rule 63/(match) block out on em3: 155.4.8.28.25 >
119.30.45.76.2545: R 0:0(0) ack 3955486675 win 0 (DF)
Apr 20 16:24:07.843355 rule 63/(match) block out on em3: 155.4.8.28.25 >
14.229.78.164.52657: R 0:0(0) ack 1524071302 win 0 (DF)
Apr 20 16:24:10.843120 rule 63/(match) block out on em3: 155.4.8.28.25 >
14.229.78.164.52657: R 0:0(0) ack 1 win 0 (DF)
Apr 20 16:24:16.843130 rule 63/(match) block out on em3: 155.4.8.28.25 >
14.229.78.164.52657: R 0:0(0) ack 1 win 0 (DF)
—snip--
