Hi, I joined the mailing list tonight and I've been struggling with getting Kubernetes to work with libreswan IPSec host to host encryption I spent many hours on this but I'm not an IPSec expert and maybe I'm missing something, such an iptables command or rule. I disabled the firewall.
I think I set up everything properly and would to know if anyone sees a mistake or something missing I have 3 servers (not visible on the internet) hackrhnode121.rtp.raleigh.ibm.com hackrhnode122.rtp.raleigh.ibm.com hackrhnode123.rtp.raleigh.ibm.com Configured as hackrhnode121 = node 1 - Kubernetes master node (host 1 or node 1) hackrhnode122 = node 2 - Kubernetes worker node 1 (host 2 or node 2) hackrhnode123 = node 3 - Kubernetes worker node 3 (host 3 or node 3) I can set up encryption between the master node (host 1 or node 1) and the 1st worker node (host 2 or node 2) and things work fine. Meaning our application still works and Kubernetes is working fine While leaving host 1 to host 2 encryption enabled, when I set up encryption between nodes 2 & 3 our application breaks. When I disable the encryption between hosts 2 & 3 and reboot things are fine. Similarly if I leave encryption on between hosts 1 & 2 and enable it between 1 & 3, Kubernetes breaks again. When I say Kubernetes breaks, there is a command that lists all the Kubernetes pods (We have 4 pods) and the pods simply do not start up. In our case we have Red Hat 7.4 Docker version 17.12.0-ce, build c97c6d6 Kubernetes version 1.7.11 (we will move to 1.9.3) I mostly followed https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/sec-securing_virtual_private_networks#sec-Host-To-Host_VPN_Using_Libreswan I've done this a few times but things break after I encrypt nodes 2 & 3 (I rolled back this encryption between these 2 nodes) Here is my detailed documentation and verification (I hope this .txt file gets through). (See attached file: commands-used-for-ipsec-rh-linux-kubernetes.txt) Thanks for any help and I'll take any suggestions ******** James Stroud Financial Crimes Insight Team Lead [email protected] - cell # = (703) 965 4516
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/sec-securing_virtual_private_networks#sec-Host-To-Host_VPN_Using_Libreswan https://libreswan.org/wiki/HOWTO:_Using_NSS_with_libreswan https://www.greglangford.co.uk/host-host-ipsec-libreswan-centos/ https://libreswan.org/wiki/Host_to_host_VPN http://www.freeswan.org/freeswan_trees/freeswan-1.97/doc/firewall.html for iptable rules - have not tried yet I created ipsec tunnels the kubernetes private network (network tunl0) but Anthony believes this does not perform encryption node to node. Here are the ips for kubernetes Before I started verified both https://hackrhnode121.rtp.raleigh.ibm.com:6883 https://hackrhnode121.rtp.raleigh.ibm.com:9443/console did steps on both as root user hackrhnode121.rtp.raleigh.ibm.com hackrhnode122.rtp.raleigh.ibm.com hackrhnode123.rtp.raleigh.ibm.com root is hackAdmin123 OS = Red Hat 7.4 Docker version 17.12.0-ce, build c97c6d6 Kubernetes version 1.7.11 (we will move to 1.9.3) hackrhnode121 = node 1 - kubernetes master node (host 1 or node 1) hackrhnode122 = node 2 - kubernetes worker node 1 (host 2 or node 2) hackrhnode123 = node 3 - kubernetes worker node 3 (host 3 or node 3) 1) Ensure Firewall is turned off service firewalld stop systemctl disable firewalld to get status of firewall enter this command firewall-cmd --state output is not running 2) Ensure Selinux is disabled Edit /etc/selinux/config Change the line below to: o SELINUX=disabled o Then save the file and reboot. Tip: You can use the setenforce 0 Run this command sestatus output SELinux status: disabled 3) on each host yum -y install libreswan tcpdump iptables-services Installation of libreswan for encryption, iptable services (helps in saving changes to iptables, tcpdump (for checking the network - testing purpose) 4) run on each host systemctl status ipsec systemctl stop ipsec rm /etc/ipsec.d/*db ls /etc/ipsec.d should only show the policies directory policies ipsec initnss output will be Initializing NSS database ls /etc/ipsec.d should now show the following cert9.db key4.db pkcs11.txt policies now start ipsec and enable the service systemctl start ipsec systemctl enable ipsec output Created symlink from /etc/systemd/system/multi-user.target.wants/ipsec.service to /usr/lib/systemd/system/ipsec.service. check status of ipsec on all servers systemctl status ipsec output will look like â ipsec.service - Internet Key Exchange (IKE) Protocol Daemon for IPsec Loaded: loaded (/usr/lib/systemd/system/ipsec.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-04-11 09:36:15 EDT; 14s ago Docs: man:ipsec(8) man:pluto(8) man:ipsec.conf(5) Main PID: 3834 (pluto) Status: "Startup completed." CGroup: /system.slice/ipsec.service ââ3834 /usr/libexec/ipsec/pluto --leak-detective --config /etc/ipsec.conf --nofork Apr 11 09:36:16 hackrhnode123.rtp.raleigh.ibm.com pluto[3834]: | setup callback for interface lo:4500 fd 22 Apr 11 09:36:16 hackrhnode123.rtp.raleigh.ibm.com pluto[3834]: | setup callback for interface lo:500 fd 21 Apr 11 09:36:16 hackrhnode123.rtp.raleigh.ibm.com pluto[3834]: | setup callback for interface ens192:4500 fd 20 Apr 11 09:36:16 hackrhnode123.rtp.raleigh.ibm.com pluto[3834]: | setup callback for interface ens192:500 fd 19 Apr 11 09:36:16 hackrhnode123.rtp.raleigh.ibm.com pluto[3834]: | setup callback for interface docker0:4500 fd 18 Apr 11 09:36:16 hackrhnode123.rtp.raleigh.ibm.com pluto[3834]: | setup callback for interface docker0:500 fd 17 Apr 11 09:36:16 hackrhnode123.rtp.raleigh.ibm.com pluto[3834]: | setup callback for interface tunl0:4500 fd 16 Apr 11 09:36:16 hackrhnode123.rtp.raleigh.ibm.com pluto[3834]: | setup callback for interface tunl0:500 fd 15 Apr 11 09:36:16 hackrhnode123.rtp.raleigh.ibm.com pluto[3834]: loading secrets from "/etc/ipsec.secrets" Apr 11 09:36:16 hackrhnode123.rtp.raleigh.ibm.com pluto[3834]: no secrets filename matched "/etc/ipsec.d/*.secrets" Now generate and show keys ipsec showhostkey --list output will show nothing until we do ipsec newhostkey command Now create host key, do this on each server ipsec newhostkey --nssdir /etc/ipsec.d --output /etc/ipsec.d/fci-ipsec.secrets output similar to (need the CKAID string from each output) 1st server Generated RSA key pair with CKAID dbc7fed50dd8101f03a0ae3b0cb55a4b38ce733a was stored in the NSS database 2nd server my output was Generated RSA key pair with CKAID 8c8c568a7cc89bcb530eb6c947c274b5216caff9 was stored in the NSS database 3rd server my output was Generated RSA key pair with CKAID 8c2ac590a263a671cd75df5723b1ce7cd293b75f was stored in the NSS database On 1st server (master node in kubernetes cluster) ipsec showhostkey --list < 1> RSA keyid: AwEAAboUv ckaid: dbc7fed50dd8101f03a0ae3b0cb55a4b38ce733a ipsec showhostkey --left --ckaid dbc7fed50dd8101f03a0ae3b0cb55a4b38ce733a output is # rsakey AwEAAboUv leftrsasigkey=0sAwEAAboUvX8sEi9QUGdASPh7ZN5ffIgwFWSkBZfs+TC4Ysjl2jGZ0xiDq745kkGzuR5+grCCGtGLBsLttCsoPWFPAX9UEDv12WkZQLlofg44iaI5LfT1syHkM6MyYk7PsUboQTxdHd4wN9fhUjoIxpcS51A7Jwuu83CxU9KuB0FQLnE1QgbjVKihL3zeSbyOTGrCHIwrkGJR0mcPjuZV4h3iZMVsnERR5YwSYBuKeVJKVsLVkglDhTsnzKZdD8QKt84ut/l0m9HelS8NC0LxhGqCbDIPl2P68MdxXRKTjHF4i+wmc7IL3oaGM4Jd8RqbbIKcrf/Rl/MUxsSoMatodDxcjOg7QYagZJDt/TYgxx5RNke7ar/oGyDHQZa+TO75fAHFbtGuFjWtSg1gQToqCkgt/Eu5NdzpQASAqQbvpWz1UgdFbM3tdilSfTh35EftibvAu2B60e5jg/y1Cf6ZSKTRQfG5aiz+fzkHUL4zBsaigcHnKcV1afU51k9YONfTKBI/vHn9SDwcirNzMea2c5RWCtQmJ/zAk+I74AdQjpjtrWF8pXSr8wgXI188iaFywaB8KQ== [root@hackrhnode121 ~] Now on 2nd server ipsec showhostkey --list output is < 1> RSA keyid: AwEAAbvKX ckaid: 8c8c568a7cc89bcb530eb6c947c274b5216caff9 ipsec showhostkey --right --ckaid 8c8c568a7cc89bcb530eb6c947c274b5216caff9 output is # rsakey AwEAAbvKX rightrsasigkey=0sAwEAAbvKXeJ/R6MLUFs5dF0ZELbTA5CU8Ns3sidbMxFTWWqiDmFjAnZw2w1umDW4el4StkKCCAkPB57S59/c6hgnNW850X0A7UUbw0km0v7y9Vm43xV54AmiPo4IzjCCDtNxqsgJL+vODj0z8j+wCENTbarv+EScZam3DEXi047qLpJWZr8MEyD7pIWNCLnYpeFNQw3hL9y+UfWDVfKzoanh37NIItQfVdlppI4oF3ZsAQeEfhQTtv05k3Ukpg1QlSWIxD5vLCPBJSEAPPVM13dQ2WT6Os8QlBPrKlZRGBYyTpZb7AchVruKrfNrj4fei/JrPBi+8Nkc2tJrvsY3HsofwOZLCIhx3U/bTLWAvzUHAf9YqVm7pL1nBu5b/9y+JBcsXDp31pfw6j4Jci9Y/6/gHisK6ltyZ5w4shVMW09wGqhfyDgGAs4sOvqjimxU6JhIeF9JUzGdxMjse7RZIeE55mC4Z5CcF5N8rsiMNnf6PNtUQCd6cVZBDPg3lPo7czefWhnVZg3oVzE37vtlXFvem2wc5+8FZW3J5kBsLue3tvv3rCbzh7T8y0EXAAG9r95TciNAd8f78IZPuBnIyOPSQT7/2b2NeAxnVl76eE6/ew== Now create config file and do 1st host vi /etc/ipsec.d/host-to-host.conf conn mytunnel-host1-to-host2 [email protected] left=9.37.132.121 leftrsasigkey=0sAwEAAboUvX8sEi9QUGdASPh7ZN5ffIgwFWSkBZfs+TC4Ysjl2jGZ0xiDq745kkGzuR5+grCCGtGLBsLttCsoPWFPAX9UEDv12WkZQLlofg44iaI5LfT1syHkM6MyYk7PsUboQTxdHd4wN9fhUjoIxpcS51A7Jwuu83CxU9KuB0FQLnE1QgbjVKihL3zeSbyOTGrCHIwrkGJR0mcPjuZV4h3iZMVsnERR5YwSYBuKeVJKVsLVkglDhTsnzKZdD8QKt84ut/l0m9HelS8NC0LxhGqCbDIPl2P68MdxXRKTjHF4i+wmc7IL3oaGM4Jd8RqbbIKcrf/Rl/MUxsSoMatodDxcjOg7QYagZJDt/TYgxx5RNke7ar/oGyDHQZa+TO75fAHFbtGuFjWtSg1gQToqCkgt/Eu5NdzpQASAqQbvpWz1UgdFbM3tdilSfTh35EftibvAu2B60e5jg/y1Cf6ZSKTRQfG5aiz+fzkHUL4zBsaigcHnKcV1afU51k9YONfTKBI/vHn9SDwcirNzMea2c5RWCtQmJ/zAk+I74AdQjpjtrWF8pXSr8wgXI188iaFywaB8KQ== [email protected] right=9.37.132.122 rightrsasigkey=0sAwEAAbvKXeJ/R6MLUFs5dF0ZELbTA5CU8Ns3sidbMxFTWWqiDmFjAnZw2w1umDW4el4StkKCCAkPB57S59/c6hgnNW850X0A7UUbw0km0v7y9Vm43xV54AmiPo4IzjCCDtNxqsgJL+vODj0z8j+wCENTbarv+EScZam3DEXi047qLpJWZr8MEyD7pIWNCLnYpeFNQw3hL9y+UfWDVfKzoanh37NIItQfVdlppI4oF3ZsAQeEfhQTtv05k3Ukpg1QlSWIxD5vLCPBJSEAPPVM13dQ2WT6Os8QlBPrKlZRGBYyTpZb7AchVruKrfNrj4fei/JrPBi+8Nkc2tJrvsY3HsofwOZLCIhx3U/bTLWAvzUHAf9YqVm7pL1nBu5b/9y+JBcsXDp31pfw6j4Jci9Y/6/gHisK6ltyZ5w4shVMW09wGqhfyDgGAs4sOvqjimxU6JhIeF9JUzGdxMjse7RZIeE55mC4Z5CcF5N8rsiMNnf6PNtUQCd6cVZBDPg3lPo7czefWhnVZg3oVzE37vtlXFvem2wc5+8FZW3J5kBsLue3tvv3rCbzh7T8y0EXAAG9r95TciNAd8f78IZPuBnIyOPSQT7/2b2NeAxnVl76eE6/ew== authby=rsasig # load and initiate automatically auto=start copy this file over to the 2nd host on 2nd host do the following cd /etc/ipsec.d scp root@hackrhnode121:/etc/ipsec.d/host-to-host.conf . One hosts 1 & 2 enter this command systemctl restart ipsec Then do status to make it started okay [root@hackrhnode121 ipsec.d]# systemctl status ipsec â ipsec.service - Internet Key Exchange (IKE) Protocol Daemon for IPsec Loaded: loaded (/usr/lib/systemd/system/ipsec.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-04-11 12:46:45 EDT; 6s ago Docs: man:ipsec(8) man:pluto(8) man:ipsec.conf(5) Process: 9041 ExecStopPost=/usr/sbin/ipsec --stopnflog (code=exited, status=0/SUCCESS) Process: 9037 ExecStopPost=/sbin/ip xfrm state flush (code=exited, status=0/SUCCESS) Process: 9035 ExecStopPost=/sbin/ip xfrm policy flush (code=exited, status=0/SUCCESS) Process: 9032 ExecStop=/usr/libexec/ipsec/whack --shutdown (code=exited, status=0/SUCCESS) Process: 9318 ExecStartPre=/usr/sbin/ipsec --checknflog (code=exited, status=0/SUCCESS) Process: 9313 ExecStartPre=/usr/sbin/ipsec --checknss (code=exited, status=0/SUCCESS) Process: 9051 ExecStartPre=/usr/libexec/ipsec/_stackmanager start (code=exited, status=0/SUCCESS) Process: 9049 ExecStartPre=/usr/libexec/ipsec/addconn --config /etc/ipsec.conf --checkconfig (code=exited, status=0/SUCCESS) Main PID: 9330 (pluto) Status: "Startup completed." Tasks: 4 Memory: 2.6M CGroup: /system.slice/ipsec.service ââ9330 /usr/libexec/ipsec/pluto --leak-detective --config /etc/ipsec.conf --nofork Apr 11 12:46:46 hackrhnode121.rtp.raleigh.ibm.com pluto[9330]: loaded private key for keyid: PKK_RSA:AwEAAboUv Apr 11 12:46:46 hackrhnode121.rtp.raleigh.ibm.com pluto[9330]: "mytunnel-host1-to-host2" #1: initiating Main Mode Apr 11 12:46:46 hackrhnode121.rtp.raleigh.ibm.com pluto[9330]: "mytunnel-host1-to-host2" #1: STATE_MAIN_I2: sent MI2, expecting MR2 Apr 11 12:46:46 hackrhnode121.rtp.raleigh.ibm.com pluto[9330]: "mytunnel-host1-to-host2" #1: STATE_MAIN_I3: sent MI3, expecting MR3 Apr 11 12:46:46 hackrhnode121.rtp.raleigh.ibm.com pluto[9330]: "mytunnel-host1-to-host2" #1: Peer ID is ID_FQDN: '@hackrhnode...com' Apr 11 12:46:46 hackrhnode121.rtp.raleigh.ibm.com pluto[9330]: "mytunnel-host1-to-host2" #1: STATE_MAIN_I4: ISAKMP SA establi...048} Apr 11 12:46:46 hackrhnode121.rtp.raleigh.ibm.com pluto[9330]: "mytunnel-host1-to-host2" #2: initiating Quick Mode RSASIG+ENC...048} Apr 11 12:46:47 hackrhnode121.rtp.raleigh.ibm.com pluto[9330]: "mytunnel-host1-to-host2" #2: STATE_QUICK_I1: retransmission; ...onse Apr 11 12:46:47 hackrhnode121.rtp.raleigh.ibm.com pluto[9330]: "mytunnel-host1-to-host2" #2: "mytunnel-host1-to-host2" #2: di...K_I1 Apr 11 12:46:47 hackrhnode121.rtp.raleigh.ibm.com pluto[9330]: "mytunnel-host1-to-host2" #2: STATE_QUICK_I2: sent QI2, IPsec ...ive} Hint: Some lines were ellipsized, use -l to show in full. Verify traffic is encrypted (do on hosts 1 & 2) tcpdump -n -i ens192 esp or udp port 500 or udp port 4500 you should see something like [root@hackrhnode121 ipsec.d]# tcpdump -n -i ens192 esp or udp port 500 or udp port 4500 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes 12:47:30.763996 IP 9.37.132.122 > 9.37.132.121: ESP(spi=0x36fb0ff3,seq=0x68c), length 292 12:47:30.764093 IP 9.37.132.122 > 9.37.132.121: ESP(spi=0x36fb0ff3,seq=0x68d), length 292 12:47:30.764396 IP 9.37.132.121 > 9.37.132.122: ESP(spi=0xe182e1fd,seq=0x7b5), length 340 12:47:30.764543 IP 9.37.132.121 > 9.37.132.122: ESP(spi=0xe182e1fd,seq=0x7b6), length 340 12:47:30.764686 IP 9.37.132.122 > 9.37.132.121: ESP(spi=0x36fb0ff3,seq=0x68e), length 100 on master node [root@hackrhnode121 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE fci-analytics-1559898221-8q7pd 1/1 Running 0 1d fci-messaging-2598678195-xc4f2 1/1 Running 0 1d fci-primaryds-710065177-xghwl 1/1 Running 0 1d fci-solution-1756280438-wc79p 1/1 Running 0 1d [root@hackrhnode121 ~]# hostname and I get get to url still works https://hackrhnode121.rtp.raleigh.ibm.com:9443/console But when I encrypt traffic between nodes 2 & 3 everything breaks. now encrypt 9.x.x.x from host 2 to 3 host 2 2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:50:56:b4:8c:63 brd ff:ff:ff:ff:ff:ff inet 9.37.132.122/24 brd 9.37.132.255 scope global ens192 host 3 2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:50:56:b4:cd:2e brd ff:ff:ff:ff:ff:ff inet 9.37.132.123/24 brd 9.37.132.255 scope global ens192 on host 2 [root@hackrhnode122 ipsec.d]# ipsec showhostkey --list < 1> RSA keyid: AwEAAbvKX ckaid: 8c8c568a7cc89bcb530eb6c947c274b5216caff9 ipsec showhostkey --left --ckaid 8c8c568a7cc89bcb530eb6c947c274b5216caff9 # rsakey AwEAAbvKX leftrsasigkey=0sAwEAAbvKXeJ/R6MLUFs5dF0ZELbTA5CU8Ns3sidbMxFTWWqiDmFjAnZw2w1umDW4el4StkKCCAkPB57S59/c6hgnNW850X0A7UUbw0km0v7y9Vm43xV54AmiPo4IzjCCDtNxqsgJL+vODj0z8j+wCENTbarv+EScZam3DEXi047qLpJWZr8MEyD7pIWNCLnYpeFNQw3hL9y+UfWDVfKzoanh37NIItQfVdlppI4oF3ZsAQeEfhQTtv05k3Ukpg1QlSWIxD5vLCPBJSEAPPVM13dQ2WT6Os8QlBPrKlZRGBYyTpZb7AchVruKrfNrj4fei/JrPBi+8Nkc2tJrvsY3HsofwOZLCIhx3U/bTLWAvzUHAf9YqVm7pL1nBu5b/9y+JBcsXDp31pfw6j4Jci9Y/6/gHisK6ltyZ5w4shVMW09wGqhfyDgGAs4sOvqjimxU6JhIeF9JUzGdxMjse7RZIeE55mC4Z5CcF5N8rsiMNnf6PNtUQCd6cVZBDPg3lPo7czefWhnVZg3oVzE37vtlXFvem2wc5+8FZW3J5kBsLue3tvv3rCbzh7T8y0EXAAG9r95TciNAd8f78IZPuBnIyOPSQT7/2b2NeAxnVl76eE6/ew== on host 3 [root@hackrhnode123 ~]# ipsec showhostkey --list < 1> RSA keyid: AwEAAacCi ckaid: 8c2ac590a263a671cd75df5723b1ce7cd293b75f [root@hackrhnode123 ~]# ipsec showhostkey --right --ckaid 8c2ac590a263a671cd75df5723b1ce7cd293b75f # rsakey AwEAAacCi rightrsasigkey=0sAwEAAacCi1geW4xqCCgIL54JcPUmAWlJQC5o601/BhzWSCwP/fX6TQnkmovA9s4A2np8GiKB3svMM8L4bEjijhX7h/S7PaOBvjYkWKTQmWX2jGA4bx+Y0AlxrLTezpnBy3Cn//jOYqKAyYy7CQ6nVaXsZd/+SyCPr+FPlzwmcS/D07F/4q10d5bwy6Tn3UkYWG0uMYLW5/0Ao7Qxdj0YXKsrPrMgMzJEu556QoqYFV93MF66Xflthf0sdC6UQuL6a+cnzHJSROehOJ1WxwHoQLYIOcSGvIfl7EDYZ8bEyxW5pFOJ+za5JZkLmrsNZQ7Nx29TRDWkrjl+oS/74V2GIrTfHiSbfpy7Kn04zX5f6+hr0+DXFQgKibvUR7eTx0g1o+QsFFlVL8HI8pOf08MFUWcr9GKYffTLmDLP3PemETmBmPCtnbGcPKL2+WDu/iX6TQnpdGvKGjOWB7T0FvdFNoA5zEJVUGUB9zxt+0OPsMMc6zc9imXpxF42Y3LGU1EDW0H4k5A2zZ57DQ== on host 2 append the following to the /etc/ipsec.d/host-to-host.conf conn mytunnel-host2-to-host3 [email protected] left=9.37.132.122 leftrsasigkey=0sAwEAAbvKXeJ/R6MLUFs5dF0ZELbTA5CU8Ns3sidbMxFTWWqiDmFjAnZw2w1umDW4el4StkKCCAkPB57S59/c6hgnNW850X0A7UUbw0km0v7y9Vm43xV54AmiPo4IzjCCDtNxqsgJL+vODj0z8j+wCENTbarv+EScZam3DEXi047qLpJWZr8MEyD7pIWNCLnYpeFNQw3hL9y+UfWDVfKzoanh37NIItQfVdlppI4oF3ZsAQeEfhQTtv05k3Ukpg1QlSWIxD5vLCPBJSEAPPVM13dQ2WT6Os8QlBPrKlZRGBYyTpZb7AchVruKrfNrj4fei/JrPBi+8Nkc2tJrvsY3HsofwOZLCIhx3U/bTLWAvzUHAf9YqVm7pL1nBu5b/9y+JBcsXDp31pfw6j4Jci9Y/6/gHisK6ltyZ5w4shVMW09wGqhfyDgGAs4sOvqjimxU6JhIeF9JUzGdxMjse7RZIeE55mC4Z5CcF5N8rsiMNnf6PNtUQCd6cVZBDPg3lPo7czefWhnVZg3oVzE37vtlXFvem2wc5+8FZW3J5kBsLue3tvv3rCbzh7T8y0EXAAG9r95TciNAd8f78IZPuBnIyOPSQT7/2b2NeAxnVl76eE6/ew== [email protected] right=9.37.132.123 rightrsasigkey=0sAwEAAacCi1geW4xqCCgIL54JcPUmAWlJQC5o601/BhzWSCwP/fX6TQnkmovA9s4A2np8GiKB3svMM8L4bEjijhX7h/S7PaOBvjYkWKTQmWX2jGA4bx+Y0AlxrLTezpnBy3Cn//jOYqKAyYy7CQ6nVaXsZd/+SyCPr+FPlzwmcS/D07F/4q10d5bwy6Tn3UkYWG0uMYLW5/0Ao7Qxdj0YXKsrPrMgMzJEu556QoqYFV93MF66Xflthf0sdC6UQuL6a+cnzHJSROehOJ1WxwHoQLYIOcSGvIfl7EDYZ8bEyxW5pFOJ+za5JZkLmrsNZQ7Nx29TRDWkrjl+oS/74V2GIrTfHiSbfpy7Kn04zX5f6+hr0+DXFQgKibvUR7eTx0g1o+QsFFlVL8HI8pOf08MFUWcr9GKYffTLmDLP3PemETmBmPCtnbGcPKL2+WDu/iX6TQnpdGvKGjOWB7T0FvdFNoA5zEJVUGUB9zxt+0OPsMMc6zc9imXpxF42Y3LGU1EDW0H4k5A2zZ57DQ== # load and initiate automatically auto=start [root@hackrhnode122 ipsec.d]# cat host-to-host.conf on host 3 copy host-to-host.conf file over cd /etc/ipsec.d/ [root@hackrhnode123 ipsec.d]# scp root@hackrhnode122:/etc/ipsec.d/host-to-host.conf . delete the 1st 10 lines of the file so it appears as conn mytunnel-host2-to-host3 [email protected] left=9.37.132.122 leftrsasigkey=0sAwEAAbvKXeJ/R6MLUFs5dF0ZELbTA5CU8Ns3sidbMxFTWWqiDmFjAnZw2w1umDW4el4StkKCCAkPB57S59/c6hgnNW850X0A7UUbw0km0v7y9Vm43xV54AmiPo4IzjCCDtNxqsgJL+vODj0z8j+wCENTbarv+EScZam3DEXi047qLpJWZr8MEyD7pIWNCLnYpeFNQw3hL9y+UfWDVfKzoanh37NIItQfVdlppI4oF3ZsAQeEfhQTtv05k3Ukpg1QlSWIxD5vLCPBJSEAPPVM13dQ2WT6Os8QlBPrKlZRGBYyTpZb7AchVruKrfNrj4fei/JrPBi+8Nkc2tJrvsY3HsofwOZLCIhx3U/bTLWAvzUHAf9YqVm7pL1nBu5b/9y+JBcsXDp31pfw6j4Jci9Y/6/gHisK6ltyZ5w4shVMW09wGqhfyDgGAs4sOvqjimxU6JhIeF9JUzGdxMjse7RZIeE55mC4Z5CcF5N8rsiMNnf6PNtUQCd6cVZBDPg3lPo7czefWhnVZg3oVzE37vtlXFvem2wc5+8FZW3J5kBsLue3tvv3rCbzh7T8y0EXAAG9r95TciNAd8f78IZPuBnIyOPSQT7/2b2NeAxnVl76eE6/ew== [email protected] right=9.37.132.123 rightrsasigkey=0sAwEAAacCi1geW4xqCCgIL54JcPUmAWlJQC5o601/BhzWSCwP/fX6TQnkmovA9s4A2np8GiKB3svMM8L4bEjijhX7h/S7PaOBvjYkWKTQmWX2jGA4bx+Y0AlxrLTezpnBy3Cn//jOYqKAyYy7CQ6nVaXsZd/+SyCPr+FPlzwmcS/D07F/4q10d5bwy6Tn3UkYWG0uMYLW5/0Ao7Qxdj0YXKsrPrMgMzJEu556QoqYFV93MF66Xflthf0sdC6UQuL6a+cnzHJSROehOJ1WxwHoQLYIOcSGvIfl7EDYZ8bEyxW5pFOJ+za5JZkLmrsNZQ7Nx29TRDWkrjl+oS/74V2GIrTfHiSbfpy7Kn04zX5f6+hr0+DXFQgKibvUR7eTx0g1o+QsFFlVL8HI8pOf08MFUWcr9GKYffTLmDLP3PemETmBmPCtnbGcPKL2+WDu/iX6TQnpdGvKGjOWB7T0FvdFNoA5zEJVUGUB9zxt+0OPsMMc6zc9imXpxF42Y3LGU1EDW0H4k5A2zZ57DQ== # load and initiate automatically auto=start One hosts 2 & 3 enter this command systemctl restart ipsec then do status to ensure no errors systemctl restart ipsec on master node [root@hackrhnode121 ipsec.d]# kubectl get pods NAME READY STATUS RESTARTS AGE fci-analytics-1559898221-8q7pd 1/1 Running 1 2d fci-messaging-2598678195-xc4f2 1/1 Running 1 2d fci-primaryds-710065177-xghwl 1/1 Running 1 2d fci-solution-1756280438-wc79p 1/1 Running 1 2d cannot get login page https://hackrhnode121.rtp.raleigh.ibm.com:9443/console rebooted all 3 nodes Now the kubernetes nodes did not start. ============================================= Another attempt ------------------------- april 16 will try to encrypt keep encryption from master to worker node 1 try to encrypt from master to worker node 2 I verified I could log in to hackrhnode121.rtp.raleigh.ibm.com via https://hackrhnode121.rtp.raleigh.ibm.com:9443/console noticed i we had everything running on node 3 (host3) [root@hackrhnode121 ipsec.d]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE fci-analytics-1559898221-gbzj1 1/1 Running 1 4d 10.244.139.79 hackrhnode123.rtp.raleigh.ibm.com fci-messaging-2598678195-rr5g3 1/1 Running 1 4d 10.244.139.78 hackrhnode123.rtp.raleigh.ibm.com fci-primaryds-710065177-xghwl 1/1 Running 3 7d 10.244.139.75 hackrhnode123.rtp.raleigh.ibm.com fci-solution-1756280438-pcq95 1/1 Running 1 4d 10.244.139.76 hackrhnode123.rtp.raleigh.ibm.com master and worker node 1 both have ipsec enabled while worker node 2 does not systemctl status ipsec master â ipsec.service - Internet Key Exchange (IKE) Protocol Daemon for IPsec Loaded: loaded (/usr/lib/systemd/system/ipsec.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-04-11 17:22:44 EDT; 4 days ago Docs: man:ipsec(8) man:pluto(8) man:ipsec.conf(5) Process: 1206 ExecStopPost=/usr/sbin/ipsec --stopnflog (code=exited, status=0/SUCCESS) Process: 1203 ExecStopPost=/sbin/ip xfrm state flush (code=exited, status=0/SUCCESS) Process: 1200 ExecStopPost=/sbin/ip xfrm policy flush (code=exited, status=0/SUCCESS) Process: 1188 ExecStop=/usr/libexec/ipsec/whack --shutdown (code=exited, status=0/SUCCESS) Process: 1508 ExecStartPre=/usr/sbin/ipsec --checknflog (code=exited, status=0/SUCCESS) Process: 1505 ExecStartPre=/usr/sbin/ipsec --checknss (code=exited, status=0/SUCCESS) Process: 1238 ExecStartPre=/usr/libexec/ipsec/_stackmanager start (code=exited, status=0/SUCCESS) Process: 1236 ExecStartPre=/usr/libexec/ipsec/addconn --config /etc/ipsec.conf --checkconfig (code=exited, status=0/SUCCESS) Main PID: 1521 (pluto) Status: "Startup completed." Tasks: 4 Memory: 2.9M CGroup: /system.slice/ipsec.service ââ1521 /usr/libexec/ipsec/pluto --leak-detective --config /etc/ipsec.conf --nofork Apr 16 12:51:48 hackrhnode121.rtp.raleigh.ibm.com pluto[1521]: "mytunnel-host1-to-host2" #173: STATE_MAIN_R2: sent MR2, expecting MI3 Apr 16 12:51:48 hackrhnode121.rtp.raleigh.ibm.com pluto[1521]: "mytunnel-host1-to-host2" #173: Peer ID is ID_FQDN: '@hackrhnode122.rtp.raleigh.ibm.com' Apr 16 12:51:48 hackrhnode121.rtp.raleigh.ibm.com pluto[1521]: "mytunnel-host1-to-host2" #173: STATE_MAIN_R3: sent MR3, ISAKMP SA established {auth=RSA_SIG c...ODP2048} Apr 16 13:02:13 hackrhnode121.rtp.raleigh.ibm.com pluto[1521]: "mytunnel-host1-to-host2" #172: deleting state (STATE_MAIN_R3) and sending notification Apr 16 13:33:49 hackrhnode121.rtp.raleigh.ibm.com pluto[1521]: "mytunnel-host1-to-host2" #174: responding to Main Mode Apr 16 13:33:49 hackrhnode121.rtp.raleigh.ibm.com pluto[1521]: "mytunnel-host1-to-host2" #174: STATE_MAIN_R1: sent MR1, expecting MI2 Apr 16 13:33:49 hackrhnode121.rtp.raleigh.ibm.com pluto[1521]: "mytunnel-host1-to-host2" #174: STATE_MAIN_R2: sent MR2, expecting MI3 Apr 16 13:33:49 hackrhnode121.rtp.raleigh.ibm.com pluto[1521]: "mytunnel-host1-to-host2" #174: Peer ID is ID_FQDN: '@hackrhnode122.rtp.raleigh.ibm.com' Apr 16 13:33:49 hackrhnode121.rtp.raleigh.ibm.com pluto[1521]: "mytunnel-host1-to-host2" #174: STATE_MAIN_R3: sent MR3, ISAKMP SA established {auth=RSA_SIG c...ODP2048} Apr 16 13:51:48 hackrhnode121.rtp.raleigh.ibm.com pluto[1521]: "mytunnel-host1-to-host2" #173: deleting state (STATE_MAIN_R3) and sending notification Hint: Some lines were ellipsized, use -l to show in full. worker node 1 - below is a problem STATE_MAIN_I2: sent MI2, expecting MR2 [root@hackrhnode122 ~]# systemctl status ipsec â ipsec.service - Internet Key Exchange (IKE) Protocol Daemon for IPsec Loaded: loaded (/usr/lib/systemd/system/ipsec.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-04-11 17:54:12 EDT; 4 days ago Docs: man:ipsec(8) man:pluto(8) man:ipsec.conf(5) Process: 1486 ExecStartPre=/usr/sbin/ipsec --checknflog (code=exited, status=0/SUCCESS) Process: 1480 ExecStartPre=/usr/sbin/ipsec --checknss (code=exited, status=0/SUCCESS) Process: 1040 ExecStartPre=/usr/libexec/ipsec/_stackmanager start (code=exited, status=0/SUCCESS) Process: 1013 ExecStartPre=/usr/libexec/ipsec/addconn --config /etc/ipsec.conf --checkconfig (code=exited, status=0/SUCCESS) Main PID: 1500 (pluto) Status: "Startup completed." Tasks: 4 Memory: 9.5M CGroup: /system.slice/ipsec.service ââ1500 /usr/libexec/ipsec/pluto --leak-detective --config /etc/ipsec.conf --nofork Apr 16 13:02:13 hackrhnode122.rtp.raleigh.ibm.com pluto[1500]: "mytunnel-host1-to-host2" #164: deleting state (STATE_MAIN_I4) and sending notification Apr 16 13:02:13 hackrhnode122.rtp.raleigh.ibm.com pluto[1500]: packet from 9.37.132.121:500: received and ignored empty informational notification payload Apr 16 13:33:49 hackrhnode122.rtp.raleigh.ibm.com pluto[1500]: "mytunnel-host1-to-host2" #166: initiating Main Mode to replace #165 Apr 16 13:33:49 hackrhnode122.rtp.raleigh.ibm.com pluto[1500]: "mytunnel-host1-to-host2" #166: STATE_MAIN_I2: sent MI2, expecting MR2 Apr 16 13:33:49 hackrhnode122.rtp.raleigh.ibm.com pluto[1500]: "mytunnel-host1-to-host2" #166: STATE_MAIN_I3: sent MI3, expecting MR3 Apr 16 13:33:49 hackrhnode122.rtp.raleigh.ibm.com pluto[1500]: "mytunnel-host1-to-host2" #166: Peer ID is ID_FQDN: '@hackrhnode121.rtp.raleigh.ibm.com' Apr 16 13:33:50 hackrhnode122.rtp.raleigh.ibm.com pluto[1500]: "mytunnel-host1-to-host2" #166: STATE_MAIN_I4: ISAKMP SA established {auth=RSA_SIG cipher=aes_...ODP2048} Apr 16 13:51:48 hackrhnode122.rtp.raleigh.ibm.com pluto[1500]: "mytunnel-host1-to-host2" #165: received Delete SA payload: self-deleting ISAKMP State #165 Apr 16 13:51:48 hackrhnode122.rtp.raleigh.ibm.com pluto[1500]: "mytunnel-host1-to-host2" #165: deleting state (STATE_MAIN_I4) and sending notification Apr 16 13:51:48 hackrhnode122.rtp.raleigh.ibm.com pluto[1500]: packet from 9.37.132.121:500: received and ignored empty informational notification payload worker node 2 Last login: Wed Apr 11 17:12:23 2018 from 9.80.216.80 IBM's internal systems must only be used for conducting IBM's business or for purposes authorized by IBM management [root@hackrhnode123 ~]# systemctl status ipsec â ipsec.service - Internet Key Exchange (IKE) Protocol Daemon for IPsec Loaded: loaded (/usr/lib/systemd/system/ipsec.service; disabled; vendor preset: disabled) Active: inactive (dead) Docs: man:ipsec(8) man:pluto(8) man:ipsec.conf(5) on host 1 (master) modified host-to-host.conf appended this to the file conn mytunnel-host1-to-host3 [email protected] left=9.37.132.121 leftrsasigkey=0sAwEAAboUvX8sEi9QUGdASPh7ZN5ffIgwFWSkBZfs+TC4Ysjl2jGZ0xiDq745kkGzuR5+grCCGtGLBsLttCsoPWFPAX9UEDv12WkZQLlofg44iaI5LfT1syHkM6MyYk7PsUboQTxdHd4wN9fhUjoIxpcS51A7Jwuu83CxU9KuB0FQLnE1QgbjVKihL3zeSbyOTGrCHIwrkGJR0mcPjuZV4h3iZMVsnERR5YwSYBuKeVJKVsLVkglDhTsnzKZdD8QKt84ut/l0m9HelS8NC0LxhGqCbDIPl2P68MdxXRKTjHF4i+wmc7IL3oaGM4Jd8RqbbIKcrf/Rl/MUxsSoMatodDxcjOg7QYagZJDt/TYgxx5RNke7ar/oGyDHQZa+TO75fAHFbtGuFjWtSg1gQToqCkgt/Eu5NdzpQASAqQbvpWz1UgdFbM3tdilSfTh35EftibvAu2B60e5jg/y1Cf6ZSKTRQfG5aiz+fzkHUL4zBsaigcHnKcV1afU51k9YONfTKBI/vHn9SDwcirNzMea2c5RWCtQmJ/zAk+I74AdQjpjtrWF8pXSr8wgXI188iaFywaB8KQ== [email protected] right=9.37.132.123 rightrsasigkey=0sAwEAAacCi1geW4xqCCgIL54JcPUmAWlJQC5o601/BhzWSCwP/fX6TQnkmovA9s4A2np8GiKB3svMM8L4bEjijhX7h/S7PaOBvjYkWKTQmWX2jGA4bx+Y0AlxrLTezpnBy3Cn//jOYqKAyYy7CQ6nVaXsZd/+SyCPr+FPlzwmcS/D07F/4q10d5bwy6Tn3UkYWG0uMYLW5/0Ao7Qxdj0YXKsrPrMgMzJEu556QoqYFV93MF66Xflthf0sdC6UQuL6a+cnzHJSROehOJ1WxwHoQLYIOcSGvIfl7EDYZ8bEyxW5pFOJ+za5JZkLmrsNZQ7Nx29TRDWkrjl+oS/74V2GIrTfHiSbfpy7Kn04zX5f6+hr0+DXFQgKibvUR7eTx0g1o+QsFFlVL8HI8pOf08MFUWcr9GKYffTLmDLP3PemETmBmPCtnbGcPKL2+WDu/iX6TQnpdGvKGjOWB7T0FvdFNoA5zEJVUGUB9zxt+0OPsMMc6zc9imXpxF42Y3LGU1EDW0H4k5A2zZ57DQ== authby=rsasig # load and initiate automatically auto=start on host 3 created this host-to-host.conf file conn mytunnel-host1-to-host3 [email protected] left=9.37.132.121 leftrsasigkey=0sAwEAAboUvX8sEi9QUGdASPh7ZN5ffIgwFWSkBZfs+TC4Ysjl2jGZ0xiDq745kkGzuR5+grCCGtGLBsLttCsoPWFPAX9UEDv12WkZQLlofg44iaI5LfT1syHkM6MyYk7PsUboQTxdHd4wN9fhUjoIxpcS51A7Jwuu83CxU9KuB0FQLnE1QgbjVKihL3zeSbyOTGrCHIwrkGJR0mcPjuZV4h3iZMVsnERR5YwSYBuKeVJKVsLVkglDhTsnzKZdD8QKt84ut/l0m9HelS8NC0LxhGqCbDIPl2P68MdxXRKTjHF4i+wmc7IL3oaGM4Jd8RqbbIKcrf/Rl/MUxsSoMatodDxcjOg7QYagZJDt/TYgxx5RNke7ar/oGyDHQZa+TO75fAHFbtGuFjWtSg1gQToqCkgt/Eu5NdzpQASAqQbvpWz1UgdFbM3tdilSfTh35EftibvAu2B60e5jg/y1Cf6ZSKTRQfG5aiz+fzkHUL4zBsaigcHnKcV1afU51k9YONfTKBI/vHn9SDwcirNzMea2c5RWCtQmJ/zAk+I74AdQjpjtrWF8pXSr8wgXI188iaFywaB8KQ== [email protected] right=9.37.132.123 rightrsasigkey=0sAwEAAacCi1geW4xqCCgIL54JcPUmAWlJQC5o601/BhzWSCwP/fX6TQnkmovA9s4A2np8GiKB3svMM8L4bEjijhX7h/S7PaOBvjYkWKTQmWX2jGA4bx+Y0AlxrLTezpnBy3Cn//jOYqKAyYy7CQ6nVaXsZd/+SyCPr+FPlzwmcS/D07F/4q10d5bwy6Tn3UkYWG0uMYLW5/0Ao7Qxdj0YXKsrPrMgMzJEu556QoqYFV93MF66Xflthf0sdC6UQuL6a+cnzHJSROehOJ1WxwHoQLYIOcSGvIfl7EDYZ8bEyxW5pFOJ+za5JZkLmrsNZQ7Nx29TRDWkrjl+oS/74V2GIrTfHiSbfpy7Kn04zX5f6+hr0+DXFQgKibvUR7eTx0g1o+QsFFlVL8HI8pOf08MFUWcr9GKYffTLmDLP3PemETmBmPCtnbGcPKL2+WDu/iX6TQnpdGvKGjOWB7T0FvdFNoA5zEJVUGUB9zxt+0OPsMMc6zc9imXpxF42Y3LGU1EDW0H4k5A2zZ57DQ== authby=rsasig # load and initiate automatically auto=start on host 1 enter this command systemctl stop ipsec systemctl status ipsec shows the following Apr 16 14:24:15 hackrhnode121.rtp.raleigh.ibm.com systemd[1]: Stopped Internet Key Exchange (IKE) Protocol Daemon for IPsec. on host 3 enter this command systemctl start ipsec systemctl stop ipsec shows error until you start ipsec on host 1 Apr 16 14:25:40 hackrhnode123.rtp.raleigh.ibm.com pluto[11133]: "mytunnel-host1-to-host3" #1: ERROR: asynchronous network error report on ens192 (sport=500) ...icated)] Hint: Some lines were ellipsized, use -l to show in full then after starting ipsec on host 1 i see this in the status systemctl status ipsec } Apr 16 14:25:55 hackrhnode123.rtp.raleigh.ibm.com pluto[11133]: "mytunnel-host1-to-host3" #3: STATE_QUICK_R2: IPsec SA established tunnel mode {ESP=>0x15475c...passive} Hint: Some lines were ellipsized, use -l to show in full. on host 1 enter this command systemctl start ipsec systemctl status ipsec shows running and okay} Apr 16 14:25:55 hackrhnode121.rtp.raleigh.ibm.com pluto[7366]: "mytunnel-host1-to-host2" #4: initiating Quick Mode RSASIG+ENCRYPT+TUNNEL+PFS+UP+IKEV1_ALLOW+I...ODP2048} Apr 16 14:25:55 hackrhnode121.rtp.raleigh.ibm.com pluto[7366]: "mytunnel-host1-to-host2" #4: STATE_QUICK_I2: sent QI2, IPsec SA established tunnel mode {ESP=...passive} Hint: Some lines were ellipsized, use -l to show in full. host 1 shows things are working [root@hackrhnode121 ipsec.d]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE fci-analytics-1559898221-gbzj1 1/1 Running 1 4d 10.244.139.79 hackrhnode123.rtp.raleigh.ibm.com fci-messaging-2598678195-rr5g3 1/1 Running 1 4d 10.244.139.78 hackrhnode123.rtp.raleigh.ibm.com fci-primaryds-710065177-xghwl 1/1 Running 3 7d 10.244.139.75 hackrhnode123.rtp.raleigh.ibm.com fci-solution-1756280438-pcq95 1/1 Running 1 4d 10.244.139.76 hackrhnode123.rtp.raleigh.ibm.com I logged in from via new browser unexpected got automatic updates not occurring message on host 3 to ipsec start up on boot systemctl enable ipsec output is Created symlink from /etc/systemd/system/multi-user.target.wants/ipsec.service to /usr/lib/systemd/system/ipsec.service. host 1 kubectl get pods -o wide fci-analytics-1559898221-gbzj1 1/1 Running 1 4d 10.244.139.79 hackrhnode123.rtp.raleigh.ibm.com fci-messaging-2598678195-rr5g3 1/1 Running 1 4d 10.244.139.78 hackrhnode123.rtp.raleigh.ibm.com fci-primaryds-710065177-xghwl 1/1 Running 3 7d 10.244.139.75 hackrhnode123.rtp.raleigh.ibm.com fci-solution-1756280438-pcq95 1/1 Running 1 4d 10.244.139.76 hackrhnode123.rtp.raleigh.ibm.com Things seem to be working. I will reboot host 3 host 3 entered following command reboot from master node after host 3 booted up NAME READY STATUS RESTARTS AGE IP NODE fci-analytics-1559898221-0x9qq 0/1 Init:0/1 0 57m 10.244.58.89 hackrhnode122.rtp.raleigh.ibm.com fci-messaging-2598678195-mnv85 1/1 Running 0 57m <none> hackrhnode122.rtp.raleigh.ibm.com fci-primaryds-710065177-dl2p0 1/1 Running 0 57m 10.244.58.87 hackrhnode122.rtp.raleigh.ibm.com fci-solution-1756280438-pn6tt 0/1 Init:0/1 0 57m <none> hackrhnode122.rtp.raleigh.ibm.com [root@hackrhnode121 ipsec.d]# things are not working I rebooted host 2 still not working, I get this NAME READY STATUS RESTARTS AGE IP NODE fci-analytics-1559898221-g6k5n 0/1 Init:0/1 0 2h 10.244.139.80 hackrhnode123.rtp.raleigh.ibm.com fci-messaging-2598678195-v2s2v 0/1 Error 6 2h <none> hackrhnode123.rtp.raleigh.ibm.com fci-primaryds-710065177-19v0m 1/1 Running 0 2h 10.244.139.82 hackrhnode123.rtp.raleigh.ibm.com fci-solution-1756280438-fc8dx 0/1 Init:0/1 0 2h <none> hackrhnode123.rtp.raleigh.ibm.com I will reboot host 3 still not working from the db2 container I can ping host 1, 2, 3 [root@fci-primaryds-710065177-1192k /]# ping hackrhnode123 PING hackrhnode123.rtp.raleigh.ibm.com (9.37.132.123) 56(84) bytes of data. 64 bytes from hackrhnode123.rtp.raleigh.ibm.com (9.37.132.123): icmp_seq=1 ttl=63 time=0.375 ms 64 bytes from hackrhnode123.rtp.raleigh.ibm.com (9.37.132.123): icmp_seq=2 ttl=63 time=0.623 ms 64 bytes from hackrhnode123.rtp.raleigh.ibm.com (9.37.132.123): icmp_seq=3 ttl=63 time=0.394 ms ^C --- hackrhnode123.rtp.raleigh.ibm.com ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.375/0.464/0.623/0.112 ms [root@fci-primaryds-710065177-1192k /]# ping hackrhnode122 PING hackrhnode122.rtp.raleigh.ibm.com (9.37.132.122) 56(84) bytes of data. 64 bytes from hackrhnode122.rtp.raleigh.ibm.com (9.37.132.122): icmp_seq=1 ttl=64 time=0.087 ms 64 bytes from hackrhnode122.rtp.raleigh.ibm.com (9.37.132.122): icmp_seq=2 ttl=64 time=0.114 ms ^C --- hackrhnode122.rtp.raleigh.ibm.com ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 0.087/0.100/0.114/0.016 ms [root@fci-primaryds-710065177-1192k /]# ping hackrhnode121 PING hackrhnode121.rtp.raleigh.ibm.com (9.37.132.121) 56(84) bytes of data. 64 bytes from hackrhnode121.rtp.raleigh.ibm.com (9.37.132.121): icmp_seq=1 ttl=63 time=0.494 ms 64 bytes from hackrhnode121.rtp.raleigh.ibm.com (9.37.132.121): icmp_seq=2 ttl=63 time=0.673 ms 64 bytes from hackrhnode121.rtp.raleigh.ibm.com (9.37.132.121): icmp_seq=3 ttl=63 time=0.595 ms ^C ------------------Misc ---------------- hackrhnode121.rtp.raleigh.ibm.com 7: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN qlen 1 link/ipip 0.0.0.0 brd 0.0.0.0 inet 10.244.20.192/32 scope global tunl0 hackrhnode122.rtp.raleigh.ibm.com 10: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN qlen 1 link/ipip 0.0.0.0 brd 0.0.0.0 inet 10.244.58.64/32 scope global tunl0 hackrhnode123.rtp.raleigh.ibm.com 4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN qlen 1 link/ipip 0.0.0.0 brd 0.0.0.0 inet 10.244.139.64/32 scope global tunl0 Below is if I did encryption on the 10.x private calico network #tcpdump -n -i tunl0 esp or udp port 500 or udp port 4500 output from host 1 is 11:14:36.671350 IP 10.244.20.192.isakmp > 10.244.58.64.isakmp: isakmp: phase 1 ? ident 11:14:36.671428 IP 10.244.20.192.isakmp > 10.244.58.64.isakmp: isakmp: phase 1 ? ident 11:14:36.674261 IP 10.244.58.64.isakmp > 10.244.20.192.isakmp: isakmp: phase 1 ? ident 11:14:36.674305 IP 10.244.58.64.isakmp > 10.244.20.192.isakmp: isakmp: phase 2/others ? inf[E] ---------------------------- IBM Cloud private https://www.ibm.com/support/knowledgecenter/en/SSBS6K_2.1.0.2/installing/ipsec_mesh.html IBM Cloud Private uses strongswan and not libreswan to get strongwan need to install epel yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm then install strongswan yum install -y strongswan default for ipip is false ibm cloud private requires it enabled calico_ipip_enabled https://www.ibm.com/support/knowledgecenter/en/SS8TQM_1.1.0/installing/config_yaml.html ------------- may need to use https://github.com/hwdsl2/setup-ipsec-vpn https://support.symantec.com/en_US/article.TECH83130.html command below gives status if ipsec tunnels ipsec whack --trafficstatus example output[root@hackrhnode121 ipsec.d]# ipsec whack --trafficstatus 006 #2: "mytunnel-host1-to-host2", type=ESP, add_time=1523935194, inBytes=2065, outBytes=2374, id='@hackrhnode122.rtp.raleigh.ibm.com' 006 #4: "mytunnel-host1-to-host2", type=ESP, add_time=1523935195, inBytes=3641852, outBytes=9231834, id='@hackrhnode122.rtp.raleigh.ibm.com' ipsec status
_______________________________________________ Swan mailing list [email protected] https://lists.libreswan.org/mailman/listinfo/swan
