Re: pppoe encapsulated udp packets which appear on ethernet disappear between pppoe and ppp0 after pppoe hangup; continues to work after reboot

[Thomas Glanzmann]

Hello Michael,

> I'll be curious to know what product is at the other end.

After a few months we found the problem. The culprit for the wrong
session ID is not the Providers DSLAM or PPP, but the Zyxel ZyXEL
VMG1312-B30A or more specific:

> * https://support.aa.net.uk/VMG1312-B10A:_Bugs

> PPPoE Session-ID caching bug (In Bridge mode)
> =

> Issue Description
> -

> Last year we had an problem with Huawei FTTC modems, the standard ones that
> Openreach supply The bug appears to be that the modem manages to "blacklist"
> some UDP packets after a PPP restart. Typically this affects VPN tunnels. The
> short term fix is to unplugged and plugged back in!  We now have what looks to
> be the same fault on the ZyXELs - both on ADSL and VDSL.  When a PPPoE session
> finishes and a new one starts, ethernet frames containing IP packets with the
> same source and destination IP and port combination that were used in the
> previous session are received with the PPPoE Session-ID from the earlier
> session.  This affects long running sessions using protocols which use the 
> same
> source port for all communications. This includes IPsec and (in some
> circumstances) SIP.  Our understanding of this, having talked to Huawei last
> year to get a very similar bug fixed is that the problem is with the packet
> accelerator feature in the Broadcom chipset. It is caching frame headers
> including the PPPoE Session-ID, but not checking if the Session-ID is the same
> when searching for the entry in the cache for subsequent packets. Unplugging
> the ethernet cable from the VMG1312 momentarily resolves the problem - that
> action must trigger a cache flush in the Broadcom chipset.  Possible fixes
> would be to either not store the Session-ID in the packet accelerator cache at
> all, or to check the Session-ID in addition to the IP and ports when searching
> the cache. A workaround would be to disable the packet accelerator.  (Side 
> note
> for other ISPs looking at this: This does not affect lines that have dynamic
> WAN addresses, which none of our service do.)

> Date Reported
> -
> 2015-05-06

> Updates
> ---
> 2015-05-06 - Escalated with ZyXEL/Broadcom
> 2015-05-15 - ZyXEL staff came to AAISP offices and we demonstrated and 
> discussed the problem
> 2015-06-02 - Still in hand with ZyXEL HQ reproducing this in their lab
> 2016-10-01 - ZyXEL still unable to reproduce this, even though we have had 
> customers recently seeing the issue with their VPN sessions

> Resolution

> None yet.

We identified the ZyXEL VMG1312-B30A as culprit by doing:

Telekom DSLAM <-DSL-> Older Zyxel without Vectoring support <-Ethernet Bridge 
Sniffer-> Allnet DSLAN <-DSL-> Zyxel VMG1312-B30A <-> Ethernet

We sniffed on the Ethernet Bridge and found out that the PPPOE Session ID from
German Telekom are correct, but the PPPOE Session ID from the Zyxel was corrupt.

Thank you again for helping me identifying the issue.

Cheers,
Thomas
--
To unsubscribe from this list: send the line "unsubscribe linux-ppp" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: pppoe encapsulated udp packets which appear on ethernet disappear between pppoe and ppp0 after pppoe hangup; continues to work after reboot

[Thomas Glanzmann]

Hello everyone,

* Thomas Glanzmann  [2017-04-11 18:01]:
> another small follow-up in pppd 2.4.7 is [1]. When I manually hangup
> with this patch, all pending pppoe sessions are terminated as well.

> [1] 
> https://github.com/paulusmack/ppp/commit/cd2c14f998c57bbe6a01dc5854f2763c0d7f31fb

another small follow up. This does not work always. Tonight at 03:00 I
hangup and the VPN did not come backup up. But this works reliable;

ifconfig eth1 down; sleep 10; ifconfig eth1 up; pon dsl-provider

Cheers,
Thomas
--
To unsubscribe from this list: send the line "unsubscribe linux-ppp" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: pppoe encapsulated udp packets which appear on ethernet disappear between pppoe and ppp0 after pppoe hangup; continues to work after reboot

[Thomas Glanzmann]

Hello Michael,
another small follow-up in pppd 2.4.7 is [1]. When I manually hangup
with this patch, all pending pppoe sessions are terminated as well.

[1] 
https://github.com/paulusmack/ppp/commit/cd2c14f998c57bbe6a01dc5854f2763c0d7f31fb

Cheers,
Thomas
--
To unsubscribe from this list: send the line "unsubscribe linux-ppp" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: pppoe encapsulated udp packets which appear on ethernet disappear between pppoe and ppp0 after pppoe hangup; continues to work after reboot

[Thomas Glanzmann]

Hello Michael,
small follow up. Taking the link down on my site seems to reset the
pppoe session on the remote site. This symptom was probably triggered by
the reboot of the server.

--
(generate-03) [~] ping -c 5 172.17.0.254
PING 172.17.0.254 (172.17.0.254) 56(84) bytes of data.

--- 172.17.0.254 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4092ms

(generate-03) [~] poff; ifconfig eth1 down; sleep 10; ifconfig eth1 up; pon 
dsl-provider
Plugin rp-pppoe.so loaded.
(generate-03) [~] ping -c 5 172.17.0.254
PING 172.17.0.254 (172.17.0.254) 56(84) bytes of data.
64 bytes from 172.17.0.254: icmp_seq=1 ttl=64 time=41.5 ms
64 bytes from 172.17.0.254: icmp_seq=2 ttl=64 time=41.1 ms
64 bytes from 172.17.0.254: icmp_seq=3 ttl=64 time=41.4 ms
64 bytes from 172.17.0.254: icmp_seq=4 ttl=64 time=41.2 ms
64 bytes from 172.17.0.254: icmp_seq=5 ttl=64 time=41.4 ms

--- 172.17.0.254 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 41.194/41.377/41.579/0.148 ms
--

Also my ssh session survives, which it previously did not. Thank you again.

Cheers,
Thomas
--
To unsubscribe from this list: send the line "unsubscribe linux-ppp" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: pppoe encapsulated udp packets which appear on ethernet disappear between pppoe and ppp0 after pppoe hangup; continues to work after reboot

[Thomas Glanzmann]

Hello Michael,

> It sounds like they aren't part of the correct PPPoE session to me.
> As if the other end is not killing it's PPP session when the new one forms.

wow. You nailed it. Thank you for identifying the root cause. I'll open a call
with German Telekom to fix there ppp endpoint.

(x1) [~/pcaps] tshark -nr eth1.pcap -T fields -e 'pppoe.session_id' | sort -u

0xb53b
0xb5e0
0xb7b3

These are exactly the packages I was missing:

(x1) [~/pcaps] tshark -nr eth1.pcap -Y 'pppoe.session_id == 0xb5e0'
 2938  12.176318 88.198.215.20 → 217.92.232.50 UDP 118 5000 → 5000 Len=68
 5247  21.951571 88.198.215.20 → 217.92.232.50 UDP 118 5000 → 5000 Len=68
 5248  21.952077 88.198.215.20 → 217.92.232.50 UDP 118 5000 → 5000 Len=68
 5802  31.641323 88.198.215.20 → 217.92.232.50 UDP 118 5000 → 5000 Len=68
 5803  31.641807 88.198.215.20 → 217.92.232.50 UDP 118 5000 → 5000 Len=68

Cheers,
Thomas
--
To unsubscribe from this list: send the line "unsubscribe linux-ppp" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: pppoe encapsulated udp packets which appear on ethernet disappear between pppoe and ppp0 after pppoe hangup; continues to work after reboot

[Michael Richardson]

Thomas Glanzmann  wrote:
> The real interesting question is, why do perfectly healthy ip/udp
> packets from within the pppoe session drop before reaching ppp0? Why
> does it only happen after one hangup?

It sounds like they aren't part of the correct PPPoE session to me.
As if the other end is not killing it's PPP session when the new one forms.

--
]   Never tell me the odds! | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works| network architect  [
] m...@sandelman.ca  http://www.sandelman.ca/|   ruby on rails[

--
To unsubscribe from this list: send the line "unsubscribe linux-ppp" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: pppoe encapsulated udp packets which appear on ethernet disappear between pppoe and ppp0 after pppoe hangup; continues to work after reboot

[Thomas Glanzmann]

Hello Michael,

* Michael Richardson  [2017-04-10 22:02]:

> Are you using openvpn with a 0.0.0.0/0 route?

no, I don't.

> Can you post your routing table when it is working, and when it is
> not?

The routing table is the same:

(generate-03) [~] ip r s
default dev ppp0  scope link
172.17.0.0/24 via 172.17.0.254 dev falkenstein
172.17.0.254 dev falkenstein  proto kernel  scope link  src 192.168.168.1
172.19.0.0/24 via 172.19.0.254 dev eclogicnew
172.19.0.254 dev eclogicnew  proto kernel  scope link  src 192.168.168.1
172.20.0.0/24 via 172.19.0.254 dev eclogicnew
192.168.168.0/24 dev eth0  proto kernel  scope link  src 192.168.168.1
217.5.98.12 dev ppp0  proto kernel  scope link  src 217.92.232.50

The real interesting question is, why do perfectly healthy ip/udp packets from
within the pppoe session drop before reaching ppp0? Why does it only happen
after one hangup?

Cheers,
Thomas
--
To unsubscribe from this list: send the line "unsubscribe linux-ppp" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html