Re: [lxc-users] Networking issues with LXC containers in EC2

2016-04-27 Thread Peter Steele 

On 01/12/2016 07:03 PM, Fajar A. Nugraha wrote:

On Tue, Jan 12, 2016 at 9:29 PM, Peter Steele  wrote:

On 01/12/2016 05:59 AM, Fajar A. Nugraha wrote:

On Tue, Jan 12, 2016 at 8:40 PM, Peter Steele  wrote:

I should have added that I have no issue running our software on a single
EC2 instance with containers running on that instance. We can assign
multiple IPs to the instance itself, as well as to the containers running
under the instance, and the containers can all communicate with each
other
as well as with the host.


can the containers in that setup communicate with systems outside the
host (e.g. access the internet)?

if "no", then you might hit the multiple mac problem

Sadly the answer is no. They cannot even ping another host in the same
VPC...

Looks like multiple mac problem. As in, EC2 only allows one mac from
your interface.

Proxyarp should work:

(1) Make SURE your EC2 instances (I'd call them "host" from now on)
supports multiple IPs (private or elastic/public IPs, depending on
your needs). The easiest way is to add those IPs to your host
interface, make sure that that new IP can be accessed (e.g. ping that
IP from another host), and then remove it.

(2) Enable proxy arp on the host

echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp

It turned out that proxy arp was indeed the solution here, but a few 
other parameters had to be set as well. I just need to run the following 
commands on each EC2 instance:


echo 1 > /proc/sys/net/ipv4/conf/br0/forwarding
echo 1 > /proc/sys/net/ipv4/conf/br0/proxy_arp_pvlan
echo 1 > /proc/sys/net/ipv4/conf/br0/proxy_arp
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/br0/send_redirects

With these settings, my containers and hosts can all talk to each other 
just like they were all residing on the same subnet. An easy solution in 
the end.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Networking issues with LXC containers in EC2

2016-01-12 Thread Fajar A. Nugraha 
On Wed, Jan 13, 2016 at 10:03 AM, Fajar A. Nugraha  wrote:
> On Tue, Jan 12, 2016 at 9:29 PM, Peter Steele  wrote:
>> On 01/12/2016 05:59 AM, Fajar A. Nugraha wrote:
>>>
>>> On Tue, Jan 12, 2016 at 8:40 PM, Peter Steele  wrote:

 I should have added that I have no issue running our software on a single
 EC2 instance with containers running on that instance. We can assign
 multiple IPs to the instance itself, as well as to the containers running
 under the instance, and the containers can all communicate with each
 other
 as well as with the host.
>>>
>>>
>>> can the containers in that setup communicate with systems outside the
>>> host (e.g. access the internet)?
>>>
>>> if "no", then you might hit the multiple mac problem
>>
>> Sadly the answer is no. They cannot even ping another host in the same
>> VPC...
>
> Looks like multiple mac problem. As in, EC2 only allows one mac from
> your interface.

>
> (3) See 
> https://www.mail-archive.com/lxc-users@lists.linuxcontainers.org/msg02380.html


Actually my reply on your past thread should be simpler:
https://lists.linuxcontainers.org/pipermail/lxc-users/2015-September/010069.html

-- 
Fajar
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Networking issues with LXC containers in EC2

2016-01-12 Thread Fajar A. Nugraha 
On Tue, Jan 12, 2016 at 9:29 PM, Peter Steele  wrote:
> On 01/12/2016 05:59 AM, Fajar A. Nugraha wrote:
>>
>> On Tue, Jan 12, 2016 at 8:40 PM, Peter Steele  wrote:
>>>
>>> I should have added that I have no issue running our software on a single
>>> EC2 instance with containers running on that instance. We can assign
>>> multiple IPs to the instance itself, as well as to the containers running
>>> under the instance, and the containers can all communicate with each
>>> other
>>> as well as with the host.
>>
>>
>> can the containers in that setup communicate with systems outside the
>> host (e.g. access the internet)?
>>
>> if "no", then you might hit the multiple mac problem
>
> Sadly the answer is no. They cannot even ping another host in the same
> VPC...

Looks like multiple mac problem. As in, EC2 only allows one mac from
your interface.

Proxyarp should work:

(1) Make SURE your EC2 instances (I'd call them "host" from now on)
supports multiple IPs (private or elastic/public IPs, depending on
your needs). The easiest way is to add those IPs to your host
interface, make sure that that new IP can be accessed (e.g. ping that
IP from another host), and then remove it.

(2) Enable proxy arp on the host

echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp

of course, adjust to your environment (e.g. change interface name if
needed). You can also add entries in /etc/sysctl.conf or /etc/sysctl.d
so that this setting will persist on reboot.

(3) See 
https://www.mail-archive.com/lxc-users@lists.linuxcontainers.org/msg02380.html

This should make all outgoing packets use eth0's MAC, and the host
will effectively function as a router.

-- 
Fajar
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Networking issues with LXC containers in EC2

2016-01-12 Thread Fajar A. Nugraha 
On Tue, Jan 12, 2016 at 8:40 PM, Peter Steele  wrote:
> I should have added that I have no issue running our software on a single
> EC2 instance with containers running on that instance. We can assign
> multiple IPs to the instance itself, as well as to the containers running
> under the instance, and the containers can all communicate with each other
> as well as with the host.


can the containers in that setup communicate with systems outside the
host (e.g. access the internet)?

if "no", then you might hit the multiple mac problem
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Networking issues with LXC containers in EC2

2016-01-12 Thread Peter Steele 
I should have added that I have no issue running our software on a 
single EC2 instance with containers running on that instance. We can 
assign multiple IPs to the instance itself, as well as to the containers 
running under the instance, and the containers can all communicate with 
each other as well as with the host. The problem occurs when we have 
more than one EC2 instance and need to have the containers in separate 
instances to communicate with each other. You're right though: If no one 
on this list has actually dealt with this issue themselves, the quickest 
answer is probably to talk to AWS directly.


Thanks.

Peter

On 01/11/2016 06:55 PM, Fajar A. Nugraha wrote:

On Tue, Jan 12, 2016 at 6:31 AM, Peter Steele  wrote:

 From what I've read, I understand that Amazon has implemented some
special/restricted behavior for the networking stack of EC2 instances. The
question I have is whether I can accomplish what I've attempted here,
specifically, can I access a LXC container hosted on one EC2 instance
directly from another EC2 instance or from another LXC container hosted on
another EC2 instance?

You might want to ask them first. Looks like it's only available for
VPC setup: 
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

If they do allow multiple IP address, then the next step is to check
whether they allow multiple MACs (which is what you get when you use
bridge). There's a workaround for this if the ONLY limitation is the
MAC, using proxyarp.






___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] Networking issues with LXC containers in EC2

2016-01-11 Thread Peter Steele 
I first brought this issue up several weeks ago and have just got back 
to the work where I originally ran into this problem. The scenario is 
simple enough:


- Create two EC2 instances running CentOS 7.1
- Configure these instances to used bridged networking
- Create a LXC container running under each instance, using the command

   lxc-create -t download -n test1 --dir=/hf/test1/rootfs -- -d centos 
-r 7 -a amd64


Each container ends up with a config that looks something like this:

lxc.include = /usr/share/lxc/config/centos.common.conf
lxc.arch = x86_64
lxc.rootfs = /hf/test1/rootfs
lxc.utsname = test1
lxc.network.type = veth
lxc.network.link = br0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx

- Assign each EC2 instance and container a static IP address in the same 
subnet. In my test I assigned the two EC2 instances 10.0.0.102 and 
10.0.0.108 and the two containers 10.0.0.103 and 10.0.0.109.


In this scenario, each EC2 host can ping their own LXC container, and 
they can ping each other. Likewise, the containers can ping their host. 
However, Instance 102 cannot ping the container hosted on instance 108, 
and similarly instance 108 cannot ping the container hosted on instance 
102. If I configure this exact same scenario on real hardware or on KVM 
based virtual machines, this "crosstalk" problem does not 
occur--instance 102 can ping container 109 for example, even though it 
is hosted on a difference instance.


From what I've read, I understand that Amazon has implemented some 
special/restricted behavior for the networking stack of EC2 instances. 
The question I have is whether I can accomplish what I've attempted 
here, specifically, can I access a LXC container hosted on one EC2 
instance directly from another EC2 instance or from another LXC 
container hosted on another EC2 instance?



Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Networking issues with LXC containers in EC2

2016-01-11 Thread Fajar A. Nugraha 
On Tue, Jan 12, 2016 at 6:31 AM, Peter Steele  wrote:
> From what I've read, I understand that Amazon has implemented some
> special/restricted behavior for the networking stack of EC2 instances. The
> question I have is whether I can accomplish what I've attempted here,
> specifically, can I access a LXC container hosted on one EC2 instance
> directly from another EC2 instance or from another LXC container hosted on
> another EC2 instance?

You might want to ask them first. Looks like it's only available for
VPC setup: 
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

If they do allow multiple IP address, then the next step is to check
whether they allow multiple MACs (which is what you get when you use
bridge). There's a workaround for this if the ONLY limitation is the
MAC, using proxyarp.

-- 
Fajar
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users