Hello, First thanks for trying to help me, I totally appreciate that. I don't seem to be able to figure it out on my own this time.
On Tue, May 29, 2018 at 1:47 PM, Antonio Quartulli <a...@unstable.cc> wrote: > > > On 29/05/18 18:41, Yuri Kanivetsky wrote: >> On Tue, May 29, 2018 at 11:09 AM, Antonio Quartulli <a...@unstable.cc> wrote: >>> Hi, >>> >>> On 29/05/18 13:44, Yuri Kanivetsky wrote: >>> >>> [cut] >>> >>>> >>>> These two commands make it work: >>>> >>>> $ ip rule add from srv.eth0.ip table 1 >>>> $ ip route add table 1 default via srv.eth0.gw >>>> >>>> More details can be found here: >>>> https://lists.debian.org/debian-user/2018/05/msg00780.html >>>> >>>> Can you explain what's going on here? Thanks in advance. >>>> >>> >>> Once you connect to the VPN and set a default route to the VPN server >>> IP, packets flowing back over the SSH connection do not go over eth0 >>> anymore, but they follow the new default route. >> >> From my experience, SSH packets from my local computer reach server's >> eth0, but disappear when kernel makes a routing decision: >> https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg > > Yes, they disappear on the way back to the SSH client, when the server > decides how to route them back. >From my experience they disappear on the server, when the kernel makes the routing decision for an incoming packet. Let's see once again. I hope ICMP will suffice. If not, I'll try to reproduce with HTTP, or SSH. Locally I run: # tcpdump icmp and host srv.eth0.ip Ssh to the server, ensure that iptables has the rules that log packets on every possible step: # iptables-save # Generated by iptables-save v1.4.21 on Sat Jun 2 14:24:23 2018 *nat :PREROUTING ACCEPT [16283:893922] :INPUT ACCEPT [16264:892774] :OUTPUT ACCEPT [5689:416371] :POSTROUTING ACCEPT [5689:416371] -A PREROUTING -p icmp -j LOG --log-prefix "iptables: nat: PREROUTING: " -A INPUT -p icmp -j LOG --log-prefix "iptables: nat: INPUT: " -A OUTPUT -p icmp -j LOG --log-prefix "iptables: nat: OUTPUT: " -A POSTROUTING -p icmp -j LOG --log-prefix "iptables: nat: POSTROUTING: " COMMIT # Completed on Sat Jun 2 14:24:23 2018 # Generated by iptables-save v1.4.21 on Sat Jun 2 14:24:23 2018 *mangle :PREROUTING ACCEPT [86523:9198352] :INPUT ACCEPT [86487:9195636] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [88104:18188588] :POSTROUTING ACCEPT [88104:18188588] -A PREROUTING -p icmp -j LOG --log-prefix "iptables: mangle: PREROUTING: " -A INPUT -p icmp -j LOG --log-prefix "iptables: mangle: INPUT: " -A FORWARD -p icmp -j LOG --log-prefix "iptables: mangle: FORWARD: " -A OUTPUT -p icmp -j LOG --log-prefix "iptables: mangle: OUTPUT: " -A POSTROUTING -p icmp -j LOG --log-prefix "iptables: mangle: POSTROUTING: " COMMIT # Completed on Sat Jun 2 14:24:23 2018 # Generated by iptables-save v1.4.21 on Sat Jun 2 14:24:23 2018 *security :INPUT ACCEPT [86487:9195636] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [88104:18188588] -A INPUT -p icmp -j LOG --log-prefix "iptables: security: INPUT: " -A FORWARD -p icmp -j LOG --log-prefix "iptables: security: FORWARD: " -A OUTPUT -p icmp -j LOG --log-prefix "iptables: security: OUTPUT: " COMMIT # Completed on Sat Jun 2 14:24:23 2018 # Generated by iptables-save v1.4.21 on Sat Jun 2 14:24:23 2018 *raw :PREROUTING ACCEPT [86523:9198352] :OUTPUT ACCEPT [88104:18188588] -A PREROUTING -p icmp -j LOG --log-prefix "iptables: raw: PREROUTING: " -A OUTPUT -p icmp -j LOG --log-prefix "iptables: raw: OUTPUT: " COMMIT # Completed on Sat Jun 2 14:24:23 2018 # Generated by iptables-save v1.4.21 on Sat Jun 2 14:24:23 2018 *filter :INPUT ACCEPT [86487:9195636] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [88104:18188588] -A INPUT -p icmp -j LOG --log-prefix "iptables: filter: INPUT: " -A FORWARD -p icmp -j LOG --log-prefix "iptables: filter: FORWARD: " -A OUTPUT -p icmp -j LOG --log-prefix "iptables: filter: OUTPUT: " COMMIT # Completed on Sat Jun 2 14:24:23 2018 Ensure that no extra routing table exists. Then run in tmux: # tcpdump icmp and host my.ip Ping from my local machine. Locally I see: 14:27:17.189373 IP my.ip > srv.eth0.ip: ICMP echo request, id 24713, seq 1, length 64 14:27:17.192064 IP srv.eth0.ip > my.ip: ICMP echo reply, id 24713, seq 1, length 64 On the server: 14:27:18.186103 IP my.ip > srv.eth0.ip: ICMP echo request, id 57147, seq 1, length 64 14:27:18.186423 IP srv.eth0.ip > my.ip ICMP echo reply, id 57147, seq 1, length 64 In journalctl: Jun 02 14:27:18 hostname kernel: iptables: raw: PREROUTING: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=16919 DF PROTO=ICMP TYPE=8 CODE=0 ID=57147 SEQ=1 Jun 02 14:27:18 hostname kernel: iptables: mangle: PREROUTING: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=16919 DF PROTO=ICMP TYPE=8 CODE=0 ID=57147 SEQ=1 Jun 02 14:27:18 hostname kernel: iptables: nat: PREROUTING: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=16919 DF PROTO=ICMP TYPE=8 CODE=0 ID=57147 SEQ=1 Jun 02 14:27:18 hostname kernel: iptables: mangle: INPUT: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=16919 DF PROTO=ICMP TYPE=8 CODE=0 ID=57147 SEQ=1 Jun 02 14:27:18 hostname kernel: iptables: filter: INPUT: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=16919 DF PROTO=ICMP TYPE=8 CODE=0 ID=57147 SEQ=1 Jun 02 14:27:18 hostname kernel: iptables: security: INPUT: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=16919 DF PROTO=ICMP TYPE=8 CODE=0 ID=57147 SEQ=1 Jun 02 14:27:18 hostname kernel: iptables: nat: INPUT: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=16919 DF PROTO=ICMP TYPE=8 CODE=0 ID=57147 SEQ=1 Jun 02 14:27:18 hostname kernel: iptables: raw: OUTPUT: IN= OUT=eth0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20462 PROTO=ICMP TYPE=0 CODE=0 ID=57147 SEQ=1 Jun 02 14:27:18 hostname kernel: iptables: mangle: OUTPUT: IN= OUT=eth0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20462 PROTO=ICMP TYPE=0 CODE=0 ID=57147 SEQ=1 Jun 02 14:27:18 hostname kernel: iptables: filter: OUTPUT: IN= OUT=eth0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20462 PROTO=ICMP TYPE=0 CODE=0 ID=57147 SEQ=1 Jun 02 14:27:18 hostname kernel: iptables: security: OUTPUT: IN= OUT=eth0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20462 PROTO=ICMP TYPE=0 CODE=0 ID=57147 SEQ=1 Jun 02 14:27:18 hostname kernel: iptables: mangle: POSTROUTING: IN= OUT=eth0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20462 PROTO=ICMP TYPE=0 CODE=0 ID=57147 SEQ=1 Then run in another tmux pane on the server: sleep 60; pkill openvpn And in another one: openvpn config.ovpn SSH connection freezes. When it resumes, locally I see: 14:29:53.740217 IP my.ip > srv.eth0.ip: ICMP echo request, id 26830, seq 1, length 64 On the server: 14:29:54.738241 IP my.ip > srv.eth0.ip: ICMP echo request, id 24343, seq 1, length 64 In journalctl: Jun 02 14:29:54 hostname kernel: iptables: raw: PREROUTING: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=21067 DF PROTO=ICMP TYPE=8 CODE=0 ID=24343 SEQ=1 Jun 02 14:29:54 hostname kernel: iptables: mangle: PREROUTING: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=21067 DF PROTO=ICMP TYPE=8 CODE=0 ID=24343 SEQ=1 Jun 02 14:29:54 hostname kernel: iptables: nat: PREROUTING: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=21067 DF PROTO=ICMP TYPE=8 CODE=0 ID=24343 SEQ=1 So, request packets disappears after net/PREROUTING, but before mangle/INPUT. That's when the kernel makes the routing decision. Am I missing something? > > > [cut] > > >> >> And why does this not work then? >> >> $ iptables -t mangle -A PREROUTING -i eth0 -m conntrack --ctstate NEW >> -j CONNMARK --set-mark 1 >> $ iptables -t mangle -A OUTPUT -m connmark --mark 1 -j MARK --set-mark 2 >> $ ip rule add fwmark 2 table 3 >> $ ip route add table 3 default via srv.eth0.gw > > well, this is entirely another topic, but I guess this is not working as > you expected and only *some* packets end up being marked with 2. Let's see, I add the following rules to the server's mangle table: -A PREROUTING -p icmp -m conntrack --ctstate NEW -j CONNMARK --set-mark 1 -A OUTPUT -m connmark --mark 1 -j MARK --set-mark 2 Then make VPN work: # ip rule add from srv.eth0.ip table 1 # ip route add table 1 default via srv.eth0.gw Start OpenVPN, then ping from my local machine and see: Jun 02 15:04:56 hostname kernel: iptables: raw: PREROUTING: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=50307 DF PROTO=ICMP TYPE=8 CODE=0 ID=13194 SEQ=1 Jun 02 15:04:56 hostname kernel: iptables: mangle: PREROUTING:IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=50307 DF PROTO=ICMP TYPE=8 CODE=0 ID=13194 SEQ=1 Jun 02 15:04:56 hostname kernel: iptables: nat: PREROUTING: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=50307 DF PROTO=ICMP TYPE=8 CODE=0 ID=13194 SEQ=1 Jun 02 15:04:56 hostname kernel: iptables: mangle: INPUT: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=50307 DF PROTO=ICMP TYPE=8 CODE=0 ID=13194 SEQ=1 Jun 02 15:04:56 hostname kernel: iptables: filter: INPUT: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=50307 DF PROTO=ICMP TYPE=8 CODE=0 ID=13194 SEQ=1 Jun 02 15:04:56 hostname kernel: iptables: security: INPUT: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=50307 DF PROTO=ICMP TYPE=8 CODE=0 ID=13194 SEQ=1 Jun 02 15:04:56 hostname kernel: iptables: nat: INPUT: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=50307 DF PROTO=ICMP TYPE=8 CODE=0 ID=13194 SEQ=1 Jun 02 15:04:56 hostname kernel: iptables: raw: OUTPUT: IN= OUT=eth0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20467 PROTO=ICMP TYPE=0 CODE=0 ID=13194 SEQ=1 Jun 02 15:04:56 hostname kernel: iptables: mangle: OUTPUT: IN= OUT=eth0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20467 PROTO=ICMP TYPE=0 CODE=0 ID=13194 SEQ=1 Jun 02 15:04:56 hostname kernel: iptables: filter: OUTPUT: IN= OUT=eth0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20467 PROTO=ICMP TYPE=0 CODE=0 ID=13194 SEQ=1 MARK=0x2 Jun 02 15:04:56 hostname kernel: iptables: security: OUTPUT: IN= OUT=eth0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20467 PROTO=ICMP TYPE=0 CODE=0 ID=13194 SEQ=1 MARK=0x2 Jun 02 15:04:56 hostname kernel: iptables: mangle: POSTROUTINGIN= OUT=eth0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20467 PROTO=ICMP TYPE=0 CODE=0 ID=13194 SEQ=1 MARK=0x2 Then run on the server: # ip rule add fwmark 2 table 3 # ip route add table 3 default via srv.tun0.gw dev tun0 And see: Jun 02 15:06:13 hostname kernel: iptables: raw: PREROUTING: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=14721 DF PROTO=ICMP TYPE=8 CODE=0 ID=61755 SEQ=1 Jun 02 15:06:13 hostname kernel: iptables: mangle: PREROUTING:IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=14721 DF PROTO=ICMP TYPE=8 CODE=0 ID=61755 SEQ=1 Jun 02 15:06:13 hostname kernel: iptables: nat: PREROUTING: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=14721 DF PROTO=ICMP TYPE=8 CODE=0 ID=61755 SEQ=1 Jun 02 15:06:13 hostname kernel: iptables: mangle: INPUT: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=14721 DF PROTO=ICMP TYPE=8 CODE=0 ID=61755 SEQ=1 Jun 02 15:06:13 hostname kernel: iptables: filter: INPUT: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=14721 DF PROTO=ICMP TYPE=8 CODE=0 ID=61755 SEQ=1 Jun 02 15:06:13 hostname kernel: iptables: security: INPUT: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=14721 DF PROTO=ICMP TYPE=8 CODE=0 ID=61755 SEQ=1 Jun 02 15:06:13 hostname kernel: iptables: nat: INPUT: IN=eth0 OUT= MAC=... SRC=my.ip DST=srv.eth0.ip LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=14721 DF PROTO=ICMP TYPE=8 CODE=0 ID=61755 SEQ=1 Jun 02 15:06:13 hostname kernel: iptables: raw: OUTPUT: IN= OUT=eth0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20468 PROTO=ICMP TYPE=0 CODE=0 ID=61755 SEQ=1 Jun 02 15:06:13 hostname kernel: iptables: mangle: OUTPUT: IN= OUT=eth0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20468 PROTO=ICMP TYPE=0 CODE=0 ID=61755 SEQ=1 Jun 02 15:06:13 hostname kernel: iptables: filter: OUTPUT: IN= OUT=eth0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20468 PROTO=ICMP TYPE=0 CODE=0 ID=61755 SEQ=1 MARK=0x2 Jun 02 15:06:13 hostname kernel: iptables: security: OUTPUT: IN= OUT=eth0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20468 PROTO=ICMP TYPE=0 CODE=0 ID=61755 SEQ=1 MARK=0x2 Jun 02 15:06:13 hostname kernel: iptables: mangle: POSTROUTINGIN= OUT=tun0 SRC=srv.eth0.ip DST=my.ip LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=20468 PROTO=ICMP TYPE=0 CODE=0 ID=61755 SEQ=1 MARK=0x2 So, the mark is being set after mangle/OUTPUT, and interface changes before mangle/POSTROUTING. But the issue is with the incoming packets. To remind you, I don't understand what happens with the incoming packets. And how exactly these commands make it work: # ip rule add from srv.eth0.ip table 1 # ip route add table 1 default via srv.eth0.gw Regards, Yuri Kanivetsky > >> >> Are there any other ways I can investigate the issue? >> > > what issue are you talking about? > > If you want to see the "lost" packets, you can easily use tcpdump on the > VPN tunnel while SSH is active and you will see the SSH packets going > out that way. > > > Cheers, > > -- > Antonio Quartulli > ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Openvpn-users mailing list Openvpn-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openvpn-users
