Re: NAT64 return traffic vanishes after successful de-alias
On 2019/12/15 12:54, Andrey V. Elsukov wrote: > On 15.12.2019 19:15, John W. O'Brien wrote: >> Yes, this is exactly the problem. Thank you very much! >> >> The reason it was working in the EC2 case is because the FreeBSD AMIs >> set ipv6_activate_all_interfaces="YES". >> >> It helps me quite a lot to learn the concept of "reschedules a packet >> again on the same interface". That fills in a gap that I am sure will >> come in handy when trying to reason about behavior in the future. >> >> Incidentally, where are those drops counted? I did start looking at >> "netstat -i" and "netstat -s" for clues, and even now that I know what >> to look for, I'm not sure I know what I'm seeing. Is it "ip6: output >> packets discarded due to no route"? > > I think you can see such drops in the `netstat -isp ip6` output for each > specific interface in the `input datagram discarded` row. > Ah, yes, that looks right. If I had waded further through the dizzying array of netstat modes, I might have noticed upon that. Thank you again for your help. -- John W. O'Brien OpenPGP keys: 0x33C4D64B895DBF3B signature.asc Description: OpenPGP digital signature
Re: NAT64 return traffic vanishes after successful de-alias
On 15.12.2019 19:15, John W. O'Brien wrote: > Yes, this is exactly the problem. Thank you very much! > > The reason it was working in the EC2 case is because the FreeBSD AMIs > set ipv6_activate_all_interfaces="YES". > > It helps me quite a lot to learn the concept of "reschedules a packet > again on the same interface". That fills in a gap that I am sure will > come in handy when trying to reason about behavior in the future. > > Incidentally, where are those drops counted? I did start looking at > "netstat -i" and "netstat -s" for clues, and even now that I know what > to look for, I'm not sure I know what I'm seeing. Is it "ip6: output > packets discarded due to no route"? I think you can see such drops in the `netstat -isp ip6` output for each specific interface in the `input datagram discarded` row. -- WBR, Andrey V. Elsukov signature.asc Description: OpenPGP digital signature
Re: NAT64 return traffic vanishes after successful de-alias
On 2019/12/15 05:44, Andrey V. Elsukov wrote: > On 14.12.2019 22:54, John W. O'Brien wrote: >> Hello FreeBSD Networking, >> >> As the subject summarizes, I have a mostly-working NAT64 rig, but return >> traffic is disappearing, and I haven't been able to figure out why. I >> observe the post-translation (4-to-6) packets via ipfwlog0, but a simple >> ipfw counter rule ipfw matches nothing. > > I suspect you have disabled IPv6 on the interface, where IPv4 address is > configured. Check that IFDISABLED flag is not set on the IPv4 side > interface. > > When NAT64 does translation, by default it reschedules a packet again on > the same interface, but from another address family, so if you have > disabled IPv6, a packet will be just dropped by ip6_input. > You can enable IPv6 by the following command: > > # ifconfig igb0 inet6 -ifdisabled Yes, this is exactly the problem. Thank you very much! The reason it was working in the EC2 case is because the FreeBSD AMIs set ipv6_activate_all_interfaces="YES". It helps me quite a lot to learn the concept of "reschedules a packet again on the same interface". That fills in a gap that I am sure will come in handy when trying to reason about behavior in the future. Incidentally, where are those drops counted? I did start looking at "netstat -i" and "netstat -s" for clues, and even now that I know what to look for, I'm not sure I know what I'm seeing. Is it "ip6: output packets discarded due to no route"? -- John W. O'Brien OpenPGP keys: 0x33C4D64B895DBF3B signature.asc Description: OpenPGP digital signature
Re: NAT64 return traffic vanishes after successful de-alias
On 14.12.2019 22:54, John W. O'Brien wrote: > Hello FreeBSD Networking, > > As the subject summarizes, I have a mostly-working NAT64 rig, but return > traffic is disappearing, and I haven't been able to figure out why. I > observe the post-translation (4-to-6) packets via ipfwlog0, but a simple > ipfw counter rule ipfw matches nothing. I suspect you have disabled IPv6 on the interface, where IPv4 address is configured. Check that IFDISABLED flag is not set on the IPv4 side interface. When NAT64 does translation, by default it reschedules a packet again on the same interface, but from another address family, so if you have disabled IPv6, a packet will be just dropped by ip6_input. You can enable IPv6 by the following command: # ifconfig igb0 inet6 -ifdisabled -- WBR, Andrey V. Elsukov signature.asc Description: OpenPGP digital signature
Re: NAT64 return traffic vanishes after successful de-alias
On 2019/12/14 17:36, Eugene Grosbein wrote: > 15.12.2019 2:54, John W. O'Brien пишет: >> Hello FreeBSD Networking, >> >> As the subject summarizes, I have a mostly-working NAT64 rig, but return >> traffic is disappearing, and I haven't been able to figure out why. I >> observe the post-translation (4-to-6) packets via ipfwlog0, but a simple >> ipfw counter rule ipfw matches nothing. > > Have you read NETWORK ADDRESS TRANSLATION (NAT) section of ipfw(8) manual > page carefully? > It tells: > >> To let the packet continue after being (de)aliased, set the sysctl >> variable net.inet.ip.fw.one_pass to 0. > > Did you set sysctl net.inet.ip.fw.one_pass=0 ? > Hi Eugene, Yes, I am familiar with the one_pass flag. It is disabled. However, I don't believe it applies to the nat64lsn module. The IPv6/IPv4 NETWORK ADDRESS AND PROTOCOL TRANSLATION section, Stateful translation subsection says: > After translation NAT64 translator by default sends packets through > corresponding netisr queue. I find no mention of an interaction between nat64lsn and one_pass. Furthermore, the outbound path (6-to-4) is working, and aliased packets are successfully matching ipfw rules. This is what the rule counters look like in the working case after sending a single ping6 from the v6 jail to the v4 jail via the host that performs NAT64: root@freebsd:~ # ipfw show 00100 2 72 setfib 1 ip4 from 198.51.100.4/30 to any 00200 2 72 allow ip4 from 198.51.100.4/30 to any 00300 2 112 setfib 1 ip6 from 2001:db8:64:64::/96 to 2001:db8::/64,2001:db8:1000::/64 00400 2 112 allow ip6 from 2001:db8:64:64::/96 to 2001:db8::/64,2001:db8:1000::/64 00500 1 56 nat64lsn magic ip6 from 2001:db8::/64,2001:db8:1000::/64 to 2001:db8:64:64::/96 // Alias 6-to-4 00600 1 36 nat64lsn magic ip4 from any to 198.51.100.4/30 // De-alias 4-to-6 00700 71 7780 allow ip from any to any 65535 26 2752 deny ip from any to any root@freebsd:~ # ipfw nat64lsn magic show nat64lsn magic prefix4 198.51.100.4/30 prefix6 2001:db8:64:64::/96 log The equivalent counters in the non-working case would be 0 for rules 300 and 400, but 100 and 200 would be non-zero. -- John W. O'Brien OpenPGP keys: 0x33C4D64B895DBF3B signature.asc Description: OpenPGP digital signature
Re: NAT64 return traffic vanishes after successful de-alias
On 2019/12/14 17:15, Chris wrote: > On Sat, 14 Dec 2019 14:54:26 -0500 John W. OBrien j...@saltant.com said > >> Hello FreeBSD Networking, >> >> As the subject summarizes, I have a mostly-working NAT64 rig, but return >> traffic is disappearing, and I haven't been able to figure out why. I >> observe the post-translation (4-to-6) packets via ipfwlog0, but a simple >> ipfw counter rule ipfw matches nothing. >> >> My attempt to develop a minimum reproducible example failed in the sense >> that I did not reproduce the problem. Of course, this implies that one >> of the many differences between the simplified test (EC2 instance, two >> jails) and the problem rig (physical server, lagg, vlans, other things >> going on) is the cause. >> >> What I am hoping this list can help me with is being smart about what I >> try next. Otherwise, I would probably just try to brute force a solution >> by thinking of ways to permute the config that would rule each possible >> difference in or out. >> >> So far my main troubleshooting tools have been ipfw for its rule >> counters and nat64lsn stats output, netstat to look at fibs, and tcpdump >> pointed at real and diagnostic interfaces. What debugging tools and >> techniques should I employ to do better than brute force? >> >> If it would help, I would gladly share the working, EC2/jail demo >> configs on the list. Sharing the non-working configs I would prefer to >> do privately or not at all. >> >> This is on 12.1-RELEASE. >> >> Thank you, > > pf(4) is pretty close to metal, and would probably be a good candidate for > acquiring the type of statistics your hoping to find; pfctl(8), pfctl -s, > and pfctl -T are a few examples. Hi Chris, Thank you for the suggestion. I think I need a little help understanding how I would put it into practice though. The nat64lsn module is part of the ipfw firewall, and pf in FreeBSD hasn't yet picked up a NAT64 capability, so I cannot abandon ipfw in this case. Is the idea to run ipfw and pf at the same time? -- John W. O'Brien OpenPGP keys: 0x33C4D64B895DBF3B signature.asc Description: OpenPGP digital signature
Re: NAT64 return traffic vanishes after successful de-alias
15.12.2019 2:54, John W. O'Brien пишет: > Hello FreeBSD Networking, > > As the subject summarizes, I have a mostly-working NAT64 rig, but return > traffic is disappearing, and I haven't been able to figure out why. I > observe the post-translation (4-to-6) packets via ipfwlog0, but a simple > ipfw counter rule ipfw matches nothing. Have you read NETWORK ADDRESS TRANSLATION (NAT) section of ipfw(8) manual page carefully? It tells: > To let the packet continue after being (de)aliased, set the sysctl > variable net.inet.ip.fw.one_pass to 0. Did you set sysctl net.inet.ip.fw.one_pass=0 ? ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: NAT64 return traffic vanishes after successful de-alias
On Sat, 14 Dec 2019 14:54:26 -0500 John W. OBrien j...@saltant.com said Hello FreeBSD Networking, As the subject summarizes, I have a mostly-working NAT64 rig, but return traffic is disappearing, and I haven't been able to figure out why. I observe the post-translation (4-to-6) packets via ipfwlog0, but a simple ipfw counter rule ipfw matches nothing. My attempt to develop a minimum reproducible example failed in the sense that I did not reproduce the problem. Of course, this implies that one of the many differences between the simplified test (EC2 instance, two jails) and the problem rig (physical server, lagg, vlans, other things going on) is the cause. What I am hoping this list can help me with is being smart about what I try next. Otherwise, I would probably just try to brute force a solution by thinking of ways to permute the config that would rule each possible difference in or out. So far my main troubleshooting tools have been ipfw for its rule counters and nat64lsn stats output, netstat to look at fibs, and tcpdump pointed at real and diagnostic interfaces. What debugging tools and techniques should I employ to do better than brute force? If it would help, I would gladly share the working, EC2/jail demo configs on the list. Sharing the non-working configs I would prefer to do privately or not at all. This is on 12.1-RELEASE. Thank you, pf(4) is pretty close to metal, and would probably be a good candidate for acquiring the type of statistics your hoping to find; pfctl(8), pfctl -s, and pfctl -T are a few examples. HTH --Chris -- John W. O'Brien OpenPGP keys: 0x33C4D64B895DBF3B ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
NAT64 return traffic vanishes after successful de-alias
Hello FreeBSD Networking, As the subject summarizes, I have a mostly-working NAT64 rig, but return traffic is disappearing, and I haven't been able to figure out why. I observe the post-translation (4-to-6) packets via ipfwlog0, but a simple ipfw counter rule ipfw matches nothing. My attempt to develop a minimum reproducible example failed in the sense that I did not reproduce the problem. Of course, this implies that one of the many differences between the simplified test (EC2 instance, two jails) and the problem rig (physical server, lagg, vlans, other things going on) is the cause. What I am hoping this list can help me with is being smart about what I try next. Otherwise, I would probably just try to brute force a solution by thinking of ways to permute the config that would rule each possible difference in or out. So far my main troubleshooting tools have been ipfw for its rule counters and nat64lsn stats output, netstat to look at fibs, and tcpdump pointed at real and diagnostic interfaces. What debugging tools and techniques should I employ to do better than brute force? If it would help, I would gladly share the working, EC2/jail demo configs on the list. Sharing the non-working configs I would prefer to do privately or not at all. This is on 12.1-RELEASE. Thank you, -- John W. O'Brien OpenPGP keys: 0x33C4D64B895DBF3B signature.asc Description: OpenPGP digital signature
