Re: [lxc-users] Networking issues with LXC containers in EC2

[Fajar A. Nugraha]

On Wed, Jan 13, 2016 at 10:03 AM, Fajar A. Nugraha  wrote:
> On Tue, Jan 12, 2016 at 9:29 PM, Peter Steele  wrote:
>> On 01/12/2016 05:59 AM, Fajar A. Nugraha wrote:
>>>
>>> On Tue, Jan 12, 2016 at 8:40 PM, Peter Steele  wrote:

 I should have added that I have no issue running our software on a single
 EC2 instance with containers running on that instance. We can assign
 multiple IPs to the instance itself, as well as to the containers running
 under the instance, and the containers can all communicate with each
 other
 as well as with the host.
>>>
>>>
>>> can the containers in that setup communicate with systems outside the
>>> host (e.g. access the internet)?
>>>
>>> if "no", then you might hit the multiple mac problem
>>
>> Sadly the answer is no. They cannot even ping another host in the same
>> VPC...
>
> Looks like multiple mac problem. As in, EC2 only allows one mac from
> your interface.

>
> (3) See 
> https://www.mail-archive.com/lxc-users@lists.linuxcontainers.org/msg02380.html


Actually my reply on your past thread should be simpler:
https://lists.linuxcontainers.org/pipermail/lxc-users/2015-September/010069.html

-- 
Fajar
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Networking issues with LXC containers in EC2

[Fajar A. Nugraha]

On Tue, Jan 12, 2016 at 9:29 PM, Peter Steele  wrote:
> On 01/12/2016 05:59 AM, Fajar A. Nugraha wrote:
>>
>> On Tue, Jan 12, 2016 at 8:40 PM, Peter Steele  wrote:
>>>
>>> I should have added that I have no issue running our software on a single
>>> EC2 instance with containers running on that instance. We can assign
>>> multiple IPs to the instance itself, as well as to the containers running
>>> under the instance, and the containers can all communicate with each
>>> other
>>> as well as with the host.
>>
>>
>> can the containers in that setup communicate with systems outside the
>> host (e.g. access the internet)?
>>
>> if "no", then you might hit the multiple mac problem
>
> Sadly the answer is no. They cannot even ping another host in the same
> VPC...

Looks like multiple mac problem. As in, EC2 only allows one mac from
your interface.

Proxyarp should work:

(1) Make SURE your EC2 instances (I'd call them "host" from now on)
supports multiple IPs (private or elastic/public IPs, depending on
your needs). The easiest way is to add those IPs to your host
interface, make sure that that new IP can be accessed (e.g. ping that
IP from another host), and then remove it.

(2) Enable proxy arp on the host

echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp

of course, adjust to your environment (e.g. change interface name if
needed). You can also add entries in /etc/sysctl.conf or /etc/sysctl.d
so that this setting will persist on reboot.

(3) See 
https://www.mail-archive.com/lxc-users@lists.linuxcontainers.org/msg02380.html

This should make all outgoing packets use eth0's MAC, and the host
will effectively function as a router.

-- 
Fajar
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] compile lxd

[mattias jonsson]

How to do it?
There is no configure
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Networking issues with LXC containers in EC2

[Fajar A. Nugraha]

On Tue, Jan 12, 2016 at 8:40 PM, Peter Steele  wrote:
> I should have added that I have no issue running our software on a single
> EC2 instance with containers running on that instance. We can assign
> multiple IPs to the instance itself, as well as to the containers running
> under the instance, and the containers can all communicate with each other
> as well as with the host.


can the containers in that setup communicate with systems outside the
host (e.g. access the internet)?

if "no", then you might hit the multiple mac problem
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Networking issues with LXC containers in EC2

[Peter Steele]

I should have added that I have no issue running our software on a 
single EC2 instance with containers running on that instance. We can 
assign multiple IPs to the instance itself, as well as to the containers 
running under the instance, and the containers can all communicate with 
each other as well as with the host. The problem occurs when we have 
more than one EC2 instance and need to have the containers in separate 
instances to communicate with each other. You're right though: If no one 
on this list has actually dealt with this issue themselves, the quickest 
answer is probably to talk to AWS directly.


Thanks.

Peter

On 01/11/2016 06:55 PM, Fajar A. Nugraha wrote:

On Tue, Jan 12, 2016 at 6:31 AM, Peter Steele  wrote:

 From what I've read, I understand that Amazon has implemented some
special/restricted behavior for the networking stack of EC2 instances. The
question I have is whether I can accomplish what I've attempted here,
specifically, can I access a LXC container hosted on one EC2 instance
directly from another EC2 instance or from another LXC container hosted on
another EC2 instance?

You might want to ask them first. Looks like it's only available for
VPC setup: 
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

If they do allow multiple IP address, then the next step is to check
whether they allow multiple MACs (which is what you get when you use
bridge). There's a workaround for this if the ONLY limitation is the
MAC, using proxyarp.






___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] compile lxd

[Tycho Andersen]

On Tue, Jan 12, 2016 at 01:25:27PM +, mattias jonsson wrote:
> How to do it?
> There is no configure

It's a go program, so there is no autoconf. See the readme:

https://github.com/lxc/lxd#building-from-source

> lxc-users mailing list
> lxc-users@lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] re Networking issues with LXC containers in EC2

[Peter Steele]

On 01/12/2016 08:43 AM, Peter Steele wrote:


On 01/12/2016 06:35 AM, brian mullan wrote:


Peter

On AWS unless you are using VPC I don't think
you can use secondary addresses because AWS won't
route any of that traffic.  Also with your
addresses routing would be affected by the
split-horizon problem with the same network on 2
sides.

...

I install PeerVPN (www.peervpn.net )
on my AWS servers to support inter-server LXC
communications.   Its dead simple to setup, its
auto-learning & full mesh vpn and you can then
interconnect not only LXC on multiple AWS servers but on
different Clouds like AWS & Digital Ocean and/or your
own servers at home/office.

It also doesn't require firewall changes beyond what you
already use.

Flockport did a nice
writeup on how to install/use PeerVPN 
also. 


Maybe that will help.


Brian, thanks for this pointer. It looks like this is what
we'll need to solve our networking issues. However, I followed
through Flockport's L2 example and configured
peervpn on my two EC2 hosts but I still cannot ping containers
across hosts. If I could indulge on your expertise I'd appreciate
some pointers.
...
The output from peervpn on host 2 also shows that one peer
has connected, so it appears that my configuration is correct. Is
there something else that's needed on the containers? The
container IPs are all statically assigned and I don't need DHCP/dnsmasq
support.

Any advice would be appreciated.


I believe I solved the issue. I executed the command

ip link set dev peervpn0 master br0

on each host and I can now ping across my containers.

Thanks very much for pointing me to this tool. It never came up in any 
of the searches I did about networking between containers in EC2.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] re Networking issues with LXC containers in EC2

[brian mullan]

All I did was install/configure PeerVPN on say server1 and server2 and make
sure they
connected.

While logged into each of your servers you should then be able to ping
10.x.x.x IP address of the other PeerVPN member server(s) ... assuming you
are using PeerVPN as an L2 VPN and not a L3 VPN.

The next step I did was to connect the TEP (tunnel end-point) to the LXCBR0
or in your case I guess the BR0 bridge to enable containers attached to
that bridge to pass data over the VPN tunnel.

Since the PeerVPN TEP interface (“peervpn0” in the Tutorial example) is
just like any other Linux ethernet interface we can use the “ip link”
command to connect the peervpn0 interface to the LXC lxcbr0 (or BR0)
bridge.You need to do that on both of your server/instances.

*$ sudo ip link set dev peervpn0 master lxcbr0*
or

*   $ sudo ip link set dev peervpn0 master br0*

now the 10.x.x.x network (being an L2 VPN) is like one big ethernet from
the LXC container perspective on either host and you should be able to ping
from say cn1 on server1 to cn2 on server2.

I wrote up some of what I did a long time ago but I'd never gone back and
updated the info to reflect using a common dnsmasq for all containers on
all host/servers.   At the time I was just trying to see if it worked.

I don't know if my writeup

will help.

Brian
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] re Networking issues with LXC containers in EC2

[brian mullan]

Peter

On AWS unless you are using VPC I don't think you can use secondary
addresses because AWS won't route any of that traffic.  Also with your
addresses routing would be affected by the split-horizon problem with the
same network on 2 sides.

You probably know this ... but on AWS each instance has a Public & a
Private IP address.

If ec2-server1  pings  ec2-server2's Public IP address...

Inside EC2 AWS redirects your server1 to server2 pings to the Private IP of
server2 not the Public IP even though that's what you input to the ping
command.  EC2 inter-instance traffic is always to/from the Private IP
addresses.

This can be useful & save $$ because traffic internal to EC2 doesn't get
charged to you like outbound  or inbound traffic from the Internet does.

I install PeerVPN (www.peervpn.net) on my AWS servers to support
inter-server LXC communications.   Its dead simple to setup, its
auto-learning & full mesh vpn and you can then interconnect not only LXC on
multiple AWS servers but on different Clouds like AWS & Digital Ocean
and/or your own servers at home/office.

It also doesn't require firewall changes beyond what you already use.

Flockport did a nice writeup on how to install/use PeerVPN

also.   Maybe that will help.
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] re Networking issues with LXC containers in EC2

[Peter Steele]

On 01/12/2016 01:34 PM, brian mullan wrote:
All I did was install/configure PeerVPN on say server1 and server2 and 
make sure they

connected.

While logged into each of your servers you should then be able to ping 
10.x.x.x IP address of the other PeerVPN member server(s) ... assuming 
you are using PeerVPN as an L2 VPN and not a L3 VPN.


The next step I did was to connect the TEP (tunnel end-point) to the 
LXCBR0 or in your case I guess the BR0 bridge to enable


containers attached to that bridge to pass data over the VPN tunnel.

Since the PeerVPN TEP interface (“peervpn0” in the Tutorial example)

is just like any other Linux ethernet interface we can use the “ip link”
command to connect the peervpn0 interface to the LXC lxcbr0 (or BR0) 
bridge.You need to do that on both of your server/instances.


/*$ sudo ip link set dev peervpn0 master lxcbr0*/

or

/*$ sudo ip link set dev peervpn0 master br0*/

now the 10.x.x.x network (being an L2 VPN) is like one big ethernet 
from the LXC container perspective on either host and you should be 
able to ping from say cn1 on server1 to cn2 on server2.


I wrote up some of what I did a long time ago but I'd never gone back 
and updated the info to reflect using a common dnsmasq for all 
containers on all host/servers.   At the time I was just trying to see 
if it worked.


I don't know if my writeup 
will 
help.


Brian

I've already found your write-up and that pointed me to the missing ip 
link command. I've got it to work, although I'm having somewhat mixed 
results. For one thing, when I do get communication to work for 
containers, I am still unable to ping from a container on host 1 to a 
second host. I *can* ping a container in host 2 but not host 2 itself. I 
can also ping the host 2 host from host 1, just not from a container in 
the host 1. Is this expected? Our containers also need to be able to 
communicate with other hosts in our framework, not just with other 
containers.


For some reason, once I have it set up and working, it suddenly stops 
working, and I've even had my instance completely hang and needed to 
stop/start it. I'm seeing multiple warnings on this sort appearing on 
the screen in both of my peervpn sessions:


[44] warning: recursive packet filtered!

Is this indicative of some kind of issue or is this expected?

Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] re Networking issues with LXC containers in EC2

[brian mullan]

re: I *can* ping a container in host 2 but not host 2 itself

welcome to networking... its a layer 2 network and each host itself is the
tunnel end point.
I had kept something that explained some of it and if I can find it send it
to you tomorrow.

Your br0 interfaces on the 2 servers you assign an IP and are they
different IP addresses?

No its not expected to see that error etc. When I have it up its pretty
solid.

We can take this offline from the mailer list... just send email directly
to each other so we don't bug the other list members.

Brian


On Tue, Jan 12, 2016 at 6:39 PM, Peter Steele  wrote:

> On 01/12/2016 01:34 PM, brian mullan wrote:
>
> All I did was install/configure PeerVPN on say server1 and server2 and
> make sure they
> connected.
>
> While logged into each of your servers you should then be able to ping
> 10.x.x.x IP address of the other PeerVPN member server(s) ... assuming you
> are using PeerVPN as an L2 VPN and not a L3 VPN.
> The next step I did was to connect the TEP (tunnel end-point) to the
> LXCBR0 or in your case I guess the BR0 bridge to enable
>
> containers attached to that bridge to pass data over the VPN tunnel.
> Since the PeerVPN TEP interface (“peervpn0” in the Tutorial example)
>
> is just like any other Linux ethernet interface we can use the “ip link”
> command to connect the peervpn0 interface to the LXC lxcbr0 (or BR0)
> bridge.You need to do that on both of your server/instances.
>
> *$ sudo ip link set dev peervpn0 master lxcbr0*
> or
>
> *$ sudo ip link set dev peervpn0 master br0*
>
> now the 10.x.x.x network (being an L2 VPN) is like one big ethernet from
> the LXC container perspective on either host and you should be able to ping
> from say cn1 on server1 to cn2 on server2.
>
> I wrote up some of what I did a long time ago but I'd never gone back and
> updated the info to reflect using a common dnsmasq for all containers on
> all host/servers.   At the time I was just trying to see if it worked.
>
> I don't know if my writeup
> 
> will help.
>
> Brian
>
> I've already found your write-up and that pointed me to the missing ip
> link command. I've got it to work, although I'm having somewhat mixed
> results. For one thing, when I do get communication to work for containers,
> I am still unable to ping from a container on host 1 to a second host. I
> *can* ping a container in host 2 but not host 2 itself. I can also ping the
> host 2 host from host 1, just not from a container in the host 1. Is this
> expected? Our containers also need to be able to communicate with other
> hosts in our framework, not just with other containers.
>
> For some reason, once I have it set up and working, it suddenly stops
> working, and I've even had my instance completely hang and needed to
> stop/start it. I'm seeing multiple warnings on this sort appearing on the
> screen in both of my peervpn sessions:
>
> [44] warning: recursive packet filtered!
>
> Is this indicative of some kind of issue or is this expected?
>
> Peter
>
>
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] re Networking issues with LXC containers in EC2

[Mike Wright]

On 01/12/2016 04:24 PM, brian mullan wrote:

re: I *can* ping a container in host 2 but not host 2 itself

welcome to networking... its a layer 2 network and each host itself is
the tunnel end point.
I had kept something that explained some of it and if I can find it send
it to you tomorrow.

Your br0 interfaces on the 2 servers you assign an IP and are they
different IP addresses?

No its not expected to see that error etc. When I have it up its pretty
solid.

We can take this offline from the mailer list... just send email
directly to each other so we don't bug the other list members.


Please keep it on list.  I'd like to see the solution unfold.  I've had 
a bit of trouble following various Flockport write-ups and every 
additional piece of info helps me better understand the vagaries of 
advanced networking.


Thanx,
Mike Wright

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] is starting unprivileged containers as root as secure as running them as any other user?

[Serge Hallyn]

Quoting david.an...@bli.uzh.ch (david.an...@bli.uzh.ch):
> So  if I understood correctly, this means that lxd could potentially suffer 
> from a weakness in 'lxc monitor' meaning that it is more secure to run 
> unprivileged containers using the low level lxc-... functions?

I mentioned the lxc-monitor case for completeness, but as there should be no
sockets etc from the monitor avaiable to the container, it'll be a very
interesting case if/when such a vuln is found.
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] re Networking issues with LXC containers in EC2

[Peter Steele]

On 01/12/2016 06:35 AM, brian mullan wrote:

Peter

On AWS unless you are using VPC I don't think you can use secondary 
addresses because AWS won't route any of that traffic.  Also with your 
addresses routing would be affected by the split-horizon problem with 
the same network on 2 sides.


You probably know this ... but on AWS each instance has a Public & a 
Private IP address.


If ec2-server1  pings  ec2-server2's Public IP address...

Inside EC2 AWS redirects your server1 to server2 pings to the Private 
IP of server2 not the Public IP even though that's what you input to 
the ping command.  EC2 inter-instance traffic is always to/from the 
Private IP addresses.


This can be useful & save $$ because traffic internal to EC2 doesn't 
get charged to you like outbound  or inbound traffic from the Internet 
does.


I install PeerVPN (www.peervpn.net ) on my AWS 
servers to support inter-server LXC communications.   Its dead simple 
to setup, its auto-learning & full mesh vpn and you can then 
interconnect not only LXC on multiple AWS servers but on different 
Clouds like AWS & Digital Ocean and/or your own servers at home/office.


It also doesn't require firewall changes beyond what you already use.

Flockport did a nice writeup on how to install/use PeerVPN 
also. 
Maybe that will help.




Brian, thanks for this pointer. It looks like this is what we'll need to 
solve our networking issues. However, I followed through Flockport's L2 
example and configured peervpn on my two EC2 hosts but I still cannot 
ping containers across hosts. If I could indulge on your expertise I'd 
appreciate some pointers.


The peervpn config file for host 1's config file looks like this:

networkname PEERVPN
psk mypasswd
enabletunneling yes
interface peervpn0
ifconfig4 10.0.0.11/24
port 7000
initpeers 10.0.0.108 7000

and host 2's config looks like this:

networkname PEERVPN
psk mypasswd
enabletunneling yes
interface peervpn0
ifconfig4 10.0.0.12/24
port 7000

The IP address 10.0.0.108 is the private IP that is assigned to eth0 of 
host 2. Should I use this address the host's public IP instead? I did 
try this but I was still not able to ping containers across my two hosts.


From what I can tell peervpn is working correctly. A peervnp0 interface 
has been created on host 1 and assigned address 10.0.0.11, and a 
peervpn0 interface has been created on host 2 and assigned address 
10.0.0.12. The output from peervpn on host 1 shows that it has 
successfully connected to host 2:


opening sockets...
   IPv4/UDP: ok.
   IPv6/UDP: ok.
opening TAP device...
   device "peervpn0": ok.
   done.
preparing P2P engine...
   done.

Client ID: 
7701546D9169C314678310CC92AE22AFDBD0A66F2A39417644C0FC99DF498CB8
Network ID: 
EDE14D183315365CF171564C2BDEDFE3BA51310352EBDBE1CDF49FAF662CF5BB


entering main loop...

[0] 0 peers connected.
[0] resolving 10.0.0.108:7000...
 done.
[12] 1 peers connected.

The output from peervpn on host 2 also shows that one peer has 
connected, so it appears that my configuration is correct. Is there 
something else that's needed on the containers? The container IPs are 
all statically assigned and I don't need DHCP/dnsmasq support.


Any advice would be appreciated.

Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users