Re: [lxc-users] Access /dev/mem in lxc

[Peter Steele]

On 04/27/2017 12:49 AM, Ganesh Sathyanarayanan wrote:

Hi All,

This is similar to a post by a John sometime in Aug-2010. He was 
trying to run Xorg in an lxc which required access to /dev/mem. Am 
trying to run a custom/proprietary application that needs the same 
(access to /dev/mem).


I have a privileged container - as in I've created the container as 
root on my device and start it as root. (root is the usually only user 
on embedded devices, unlike PC)
I have been trying to ‘expose’ the /dev/mem device to my container 
because the application I run there needs it.
However, am unable to do so - I always end up with a “Operation not 
permitted” error when I try to open /dev/mem. The following are the 
different things I tried
1) lxc-cgroup.devices.allow = c 1 1 in the conf file (and doing a 
"mknod /dev/mem c 1 1" on the container)
2) lxc-device -n  -- add /dev/mem to a running container (this 
causes /dev/mem to appear in the container without having to run any 
extra commands such as mknod. But opening it still fails)

3) lxc.aa_profile = unconfined (along with steps 1 & 2)

Please advise what I can do to make /dev/mem accessible in lxc. A 
simple test am doing prior to running my actual application, is 
something like "head /dev/mem" in the container and check that it 
displays anything (other than Operation not permitted error).


This is something we're interested in as well. We also are developing an 
embedded device with root as the only user--everything runs in a 
privileged mode. Can containers in an LXC based environment access /dev/mem?


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Enabling real time support in containers

[Peter Steele]

On 04/05/2017 06:45 AM, Serge E. Hallyn wrote:

  I correct in assuming LXC *does* provide a means to enable RT
The kernel has hardcoded checks (which are not namespaced) that
if you are not (global) root, you cannot set or change the rt
policy.  I suspect there is a way that could be safely relaxed
(i.e. if a container has exclusie use of a cpu), but we'd have
to talk to the scheduling experts about what would make sense.
(see
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/sched/core.c?id=refs/tags/v4.11-rc5#n4164
)

Otherwise, as a workaround (assuming this is the only problem you
hit) you could simply make sure that the RT policy is correct ahead
of time and the priority is high enough that the application is only
lowering it, then the kernel wouldn't stop it.  Certainly that's
more fragile.  Or you could get fancier and LD_PRELOAD to catch
sys_setscheduler and redirect to an api over a socket to a tiny
deamon on the host kernel which sets it up for you...  But certainly
it would be best for everyone if this was supported in the kernel the
right way.

Most of our containers do not require real time support. There are a 
couple of cases though where our software does use real time threads 
though. We originally were running under libvirt based VMs and real time 
support was not an issue in this kind of environment (it was fully 
supported). We ported our software to libvirt lxc based containers and 
with the appropriate configuration was able to get real time support 
working under this environment as well. We want to make one more 
transition now to LXC (mainly due to lack of active support for 
libvirt-lxc in CentOS). I had assumed this was simply a container 
configuration issue but your response makes me think that it's not as 
simple as that. Not sure where that leaves us.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Enabling real time support in containers

[Peter Steele]

On 03/31/2017 10:16 AM, Peter Steele wrote:
As you can see, the sched_setscheduler() call fails with an EPERM 
error. This same app runs fine on the host.


Ultimately I expect this app to fail when run under my container since 
I have not given the container any real time bandwidth. I had hoped 
the option


lxc.cgroup.cpu.rt_runtime_us = 475000

would do the trick but this option is rejected with anything other 
than "0". So presumably this isn't the correct way to give a container 
real time bandwidth.


I have more experience with the libvirt-lxc framework and I have been 
able to enable real time support for containers under libvirt. The 
approach used in this case involves explicitly setting cgroup 
parameters, specifically


/sys/fs/cgroup/cpu/machine.slice/cpu.rt_runtime_us

under the host and

/sys/fs/cgroup/cpu/cpu.rt_runtime_us

under the container. For example, I might do something like this:

echo 50 >/sys/fs/cgroup/cpu/machine.slice/cpu.rt_runtime_us  
--> on the host

echo 25000 >/sys/fs/cgroup/cpu/cpu.rt_runtime_us  --> on a container

These do not work for LXC based containers though.



The test code I'm running can be simplified to just this simple sequence:

#include 
#include 

int main() {
struct sched_param param;
param.sched_priority = 50;
const int myself  =  0; // 0 is the PID of ourself
if (0 != sched_setscheduler(myself, SCHED_FIFO, )) {
printf("Failure\n");
return -1;
}

printf("Success\n");
return 0;
}

On a container with RT support enabled, this should print "Success".

Am I correct in assuming LXC *does* provide a means to enable RT 
support? If not, we will need to another approach to this problem.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Enabling real time support in containers

[Peter Steele]

We have a need to create real time threads in some of our processes 
and I've been unable to configure an LXC container to support this. 
One reference I came across was to set a container's real time 
bandwidth via the lxc.cgroup.cpu.rt_runtime_us parameter in its config 
file:


lxc.utsname = test01
lxc.include = /var/lib/lxc/centos.conf
lxc.network.veth.pair = test01
lxc.network.hwaddr = fe:d6:e8:e2:fa:db
lxc.rootfs = /var/lib/lxc/test01/rootfs
lxc.rootfs.backend = dir
lxc.cgroup.cpuset.cpus = 0,1
lxc.cgroup.memory.limit_in_bytes = 2097152000
lxc.cgroup.memory.memsw.limit_in_bytes = 3170893824
lxc.cgroup.cpu.rt_runtime_us = 475000

This container starts up fine if lxc.cgroup.cpu.rt_runtime_us is 0 
(zero). Anything other than 0 is rejected, which means real time 
threads cannot be created in this container.


What am I missing to get this to work? I am using lxc version 2.0.6 
under CentOS 7.2. The container is being created using a custom CentOS 
7.2 image.


No takers on this? I assume this is possible? I'm looking for the 
configuration options that would allow code such as this to run :


pthread_attr_setinheritsched(, PTHREAD_EXPLICIT_SCHED);
pthread_attr_setschedpolicy(, SCHED_FIFO);
tsparam.sched_priority = sched_get_priority_max(SCHED_FIFO) - 7;
pthread_attr_setschedparam(, );
rc = pthread_create(_thread, , test, NULL);

While this code runs fine in my host, it returns a non-zero code on my 
containers, indicating I am not allowed to create real time threads. I'm 
assuming this is a configuration issue in my LXC containers since I've 
seen this same code run under other container frameworks. There doesn't 
seem to be a lot of hits on this topic though when it comes to LXC.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] Enabling real time support in containers

[Peter Steele]

We have a need to create real time threads in some of our processes and 
I've been unable to configure an LXC container to support this. One 
reference I came across was to set a container's real time bandwidth via 
the lxc.cgroup.cpu.rt_runtime_us parameter in its config file:


lxc.utsname = test01
lxc.include = /var/lib/lxc/centos.conf
lxc.network.veth.pair = test01
lxc.network.hwaddr = fe:d6:e8:e2:fa:db
xc.rootfs = /var/lib/lxc/test01/rootfs
lxc.rootfs.backend = dir
lxc.cgroup.cpuset.cpus = 0,1
lxc.cgroup.memory.limit_in_bytes = 2097152000
lxc.cgroup.memory.memsw.limit_in_bytes = 3170893824
lxc.cgroup.cpu.rt_runtime_us = 475000

This container starts up fine if lxc.cgroup.cpu.rt_runtime_us is 0 
(zero). Anything other than 0 is rejected, which means real time threads 
cannot be created in this container.


What am I missing to get this to work? I am using lxc version 2.0.6 
under CentOS 7.2. The container is being created using a custom CentOS 
7.2 image.


Thanks for the help.

Peter Steele

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Networking issues with LXC containers in EC2

[Peter Steele]

On 01/12/2016 07:03 PM, Fajar A. Nugraha wrote:

On Tue, Jan 12, 2016 at 9:29 PM, Peter Steele <pwste...@gmail.com> wrote:

On 01/12/2016 05:59 AM, Fajar A. Nugraha wrote:

On Tue, Jan 12, 2016 at 8:40 PM, Peter Steele <pwste...@gmail.com> wrote:

I should have added that I have no issue running our software on a single
EC2 instance with containers running on that instance. We can assign
multiple IPs to the instance itself, as well as to the containers running
under the instance, and the containers can all communicate with each
other
as well as with the host.


can the containers in that setup communicate with systems outside the
host (e.g. access the internet)?

if "no", then you might hit the multiple mac problem

Sadly the answer is no. They cannot even ping another host in the same
VPC...

Looks like multiple mac problem. As in, EC2 only allows one mac from
your interface.

Proxyarp should work:

(1) Make SURE your EC2 instances (I'd call them "host" from now on)
supports multiple IPs (private or elastic/public IPs, depending on
your needs). The easiest way is to add those IPs to your host
interface, make sure that that new IP can be accessed (e.g. ping that
IP from another host), and then remove it.

(2) Enable proxy arp on the host

echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp

It turned out that proxy arp was indeed the solution here, but a few 
other parameters had to be set as well. I just need to run the following 
commands on each EC2 instance:


echo 1 > /proc/sys/net/ipv4/conf/br0/forwarding
echo 1 > /proc/sys/net/ipv4/conf/br0/proxy_arp_pvlan
echo 1 > /proc/sys/net/ipv4/conf/br0/proxy_arp
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/br0/send_redirects

With these settings, my containers and hosts can all talk to each other 
just like they were all residing on the same subnet. An easy solution in 
the end.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LXC 2.0 and Centos 7

[Peter Steele]

The best alternative is probably libvirt lxc. That's what we use. We had 
started trying lxc/lxd under CentOS 7 and hit various issues. The 
project has been tabled for now and we're sticking with libvirt lxc...


On 04/20/2016 10:05 AM, Saint Michael wrote:

That is what I am afraid of.

On Wed, Apr 20, 2016 at 12:46 PM, jjs - mainphrame > wrote:


From everything I've seen, it would be an uphill battle.

IMHO, just as Ubuntu is the optimal distro for lxc/lxd, Centos 7 goes
with openvz, and works really well with it.

Jake

On Wed, Apr 20, 2016 at 8:35 AM, Saint Michael > wrote:
> I have been scratching my head on how to install LXC 2.0 on top
of Centos 7.
> has anybody accomplished this task?
>
>
>
> ___
> lxc-users mailing list
> lxc-users@lists.linuxcontainers.org

> http://lists.linuxcontainers.org/listinfo/lxc-users
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org

http://lists.linuxcontainers.org/listinfo/lxc-users




___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Console issues

[Peter Steele]

On 01/28/2016 03:25 PM, Peter Steele wrote:
As I've explained in this mailing list before, I create my own custom 
CentOS template that has some history, being initially used as a 
template for KVM based virtual machines, then OpenVZ based containers, 
then libvirt-lxc containers, and now finally we're tackling LXC. One 
issue I've noted is that when I create a container using my custom 
template lxc-console does not work. When I connect I get something 
like this:


# lxc-console -n vm-00
Connected to tty 1
Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a 
itself


and then at this point it basically hangs, just consuming whatever I 
type but not attaching to the console and giving me feedback. If I use 
same config file for the container but use the CentOS download 
template instead, the console works as expected.


In comparing the two cases, I've noticed that when I'm running with 
the downloaded template, I get a set of five getty processes:


# ps aux|grep agetty
root37  0.0  0.0   6452   800 pts/0Ss+  22:45   0:00 
/sbin/agetty --noclear --keep-baud pts/0 115200 38400 9600 vt220
root38  0.0  0.0   6452   796 pts/1Ss+  22:45   0:00 
/sbin/agetty --noclear --keep-baud pts/1 115200 38400 9600 vt220
root40  0.0  0.0   6452   804 pts/2Ss+  22:45   0:00 
/sbin/agetty --noclear --keep-baud pts/2 115200 38400 9600 vt220
root41  0.0  0.0   6452   808 pts/3Ss+  22:45   0:00 
/sbin/agetty --noclear --keep-baud pts/3 115200 38400 9600 vt220
root42  0.0  0.0   6452   796 lxc/console Ss+ 22:45   0:00 
/sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220


Using my own template, I only get a single console process:

# ps aux|grep agetty
root   279  0.0  0.0   6424   792 lxc/console Ss+ 13:57   0:00 
/sbin/agetty --noclear --keep-baud console 115200 38400 9600


If I go ahead and start the processes manually, then lxc-console 
appears to work as expected. My question, what am I missing in my 
template that causes this behavior, or more specifically, results in 
the required agetty processes failing to start when the container is 
started?


For what it's worth, this same container template works with libvirt_lxc 
as far as console access is concerned. Out of curiosity I also tried the 
OpenVZ container image for CentOS 7 (available online) and the console 
did not work with this image either. It *did* work with libvirt_lxc. So 
what's missing from these templates that LXC needs for enabling console 
access?


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] Console issues

[Peter Steele]

As I've explained in this mailing list before, I create my own custom 
CentOS template that has some history, being initially used as a 
template for KVM based virtual machines, then OpenVZ based containers, 
then libvirt-lxc containers, and now finally we're tackling LXC. One 
issue I've noted is that when I create a container using my custom 
template lxc-console does not work. When I connect I get something like 
this:


# lxc-console -n vm-00
Connected to tty 1
Type  to exit the console,  to enter Ctrl+a itself

and then at this point it basically hangs, just consuming whatever I 
type but not attaching to the console and giving me feedback. If I use 
same config file for the container but use the CentOS download template 
instead, the console works as expected.


In comparing the two cases, I've noticed that when I'm running with the 
downloaded template, I get a set of five getty processes:


# ps aux|grep agetty
root37  0.0  0.0   6452   800 pts/0Ss+  22:45   0:00 
/sbin/agetty --noclear --keep-baud pts/0 115200 38400 9600 vt220
root38  0.0  0.0   6452   796 pts/1Ss+  22:45   0:00 
/sbin/agetty --noclear --keep-baud pts/1 115200 38400 9600 vt220
root40  0.0  0.0   6452   804 pts/2Ss+  22:45   0:00 
/sbin/agetty --noclear --keep-baud pts/2 115200 38400 9600 vt220
root41  0.0  0.0   6452   808 pts/3Ss+  22:45   0:00 
/sbin/agetty --noclear --keep-baud pts/3 115200 38400 9600 vt220
root42  0.0  0.0   6452   796 lxc/console Ss+ 22:45   0:00 
/sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220


Using my own template, I only get a single console process:

# ps aux|grep agetty
root   279  0.0  0.0   6424   792 lxc/console Ss+ 13:57   0:00 
/sbin/agetty --noclear --keep-baud console 115200 38400 9600


If I go ahead and start the processes manually, then lxc-console appears 
to work as expected. My question, what am I missing in my template that 
causes this behavior, or more specifically, results in the required 
agetty processes failing to start when the container is started?


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Networking issues with LXC containers in EC2

[Peter Steele]

I should have added that I have no issue running our software on a 
single EC2 instance with containers running on that instance. We can 
assign multiple IPs to the instance itself, as well as to the containers 
running under the instance, and the containers can all communicate with 
each other as well as with the host. The problem occurs when we have 
more than one EC2 instance and need to have the containers in separate 
instances to communicate with each other. You're right though: If no one 
on this list has actually dealt with this issue themselves, the quickest 
answer is probably to talk to AWS directly.


Thanks.

Peter

On 01/11/2016 06:55 PM, Fajar A. Nugraha wrote:

On Tue, Jan 12, 2016 at 6:31 AM, Peter Steele <pwste...@gmail.com> wrote:

 From what I've read, I understand that Amazon has implemented some
special/restricted behavior for the networking stack of EC2 instances. The
question I have is whether I can accomplish what I've attempted here,
specifically, can I access a LXC container hosted on one EC2 instance
directly from another EC2 instance or from another LXC container hosted on
another EC2 instance?

You might want to ask them first. Looks like it's only available for
VPC setup: 
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

If they do allow multiple IP address, then the next step is to check
whether they allow multiple MACs (which is what you get when you use
bridge). There's a workaround for this if the ONLY limitation is the
MAC, using proxyarp.






___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] re Networking issues with LXC containers in EC2

[Peter Steele]

On 01/12/2016 08:43 AM, Peter Steele wrote:


On 01/12/2016 06:35 AM, brian mullan wrote:


Peter

On AWS unless you are using VPC I don't think
you can use secondary addresses because AWS won't
route any of that traffic.  Also with your
addresses routing would be affected by the
split-horizon problem with the same network on 2
sides.

...

I install PeerVPN (www.peervpn.net <http://www.peervpn.net>)
on my AWS servers to support inter-server LXC
communications.   Its dead simple to setup, its
auto-learning & full mesh vpn and you can then
interconnect not only LXC on multiple AWS servers but on
different Clouds like AWS & Digital Ocean and/or your
own servers at home/office.

It also doesn't require firewall changes beyond what you
already use.

Flockport did a nice
writeup on how to install/use PeerVPN 
<https://www.flockport.com/build-layer2-and-layer-3-overlay-networks-with-peervpn/>also. 


Maybe that will help.


Brian, thanks for this pointer. It looks like this is what
we'll need to solve our networking issues. However, I followed
through Flockport's L2 example and configured
peervpn on my two EC2 hosts but I still cannot ping containers
across hosts. If I could indulge on your expertise I'd appreciate
some pointers.
...
The output from peervpn on host 2 also shows that one peer
has connected, so it appears that my configuration is correct. Is
there something else that's needed on the containers? The
container IPs are all statically assigned and I don't need DHCP/dnsmasq
support.

Any advice would be appreciated.


I believe I solved the issue. I executed the command

ip link set dev peervpn0 master br0

on each host and I can now ping across my containers.

Thanks very much for pointing me to this tool. It never came up in any 
of the searches I did about networking between containers in EC2.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] re Networking issues with LXC containers in EC2

[Peter Steele]

On 01/12/2016 01:34 PM, brian mullan wrote:
All I did was install/configure PeerVPN on say server1 and server2 and 
make sure they

connected.

While logged into each of your servers you should then be able to ping 
10.x.x.x IP address of the other PeerVPN member server(s) ... assuming 
you are using PeerVPN as an L2 VPN and not a L3 VPN.


The next step I did was to connect the TEP (tunnel end-point) to the 
LXCBR0 or in your case I guess the BR0 bridge to enable


containers attached to that bridge to pass data over the VPN tunnel.

Since the PeerVPN TEP interface (“peervpn0” in the Tutorial example)

is just like any other Linux ethernet interface we can use the “ip link”
command to connect the peervpn0 interface to the LXC lxcbr0 (or BR0) 
bridge.You need to do that on both of your server/instances.


/*$ sudo ip link set dev peervpn0 master lxcbr0*/

or

/*$ sudo ip link set dev peervpn0 master br0*/

now the 10.x.x.x network (being an L2 VPN) is like one big ethernet 
from the LXC container perspective on either host and you should be 
able to ping from say cn1 on server1 to cn2 on server2.


I wrote up some of what I did a long time ago but I'd never gone back 
and updated the info to reflect using a common dnsmasq for all 
containers on all host/servers.   At the time I was just trying to see 
if it worked.


I don't know if my writeup 
will 
help.


Brian

I've already found your write-up and that pointed me to the missing ip 
link command. I've got it to work, although I'm having somewhat mixed 
results. For one thing, when I do get communication to work for 
containers, I am still unable to ping from a container on host 1 to a 
second host. I *can* ping a container in host 2 but not host 2 itself. I 
can also ping the host 2 host from host 1, just not from a container in 
the host 1. Is this expected? Our containers also need to be able to 
communicate with other hosts in our framework, not just with other 
containers.


For some reason, once I have it set up and working, it suddenly stops 
working, and I've even had my instance completely hang and needed to 
stop/start it. I'm seeing multiple warnings on this sort appearing on 
the screen in both of my peervpn sessions:


[44] warning: recursive packet filtered!

Is this indicative of some kind of issue or is this expected?

Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] re Networking issues with LXC containers in EC2

[Peter Steele]

On 01/12/2016 06:35 AM, brian mullan wrote:

Peter

On AWS unless you are using VPC I don't think you can use secondary 
addresses because AWS won't route any of that traffic.  Also with your 
addresses routing would be affected by the split-horizon problem with 
the same network on 2 sides.


You probably know this ... but on AWS each instance has a Public & a 
Private IP address.


If ec2-server1  pings  ec2-server2's Public IP address...

Inside EC2 AWS redirects your server1 to server2 pings to the Private 
IP of server2 not the Public IP even though that's what you input to 
the ping command.  EC2 inter-instance traffic is always to/from the 
Private IP addresses.


This can be useful & save $$ because traffic internal to EC2 doesn't 
get charged to you like outbound  or inbound traffic from the Internet 
does.


I install PeerVPN (www.peervpn.net ) on my AWS 
servers to support inter-server LXC communications.   Its dead simple 
to setup, its auto-learning & full mesh vpn and you can then 
interconnect not only LXC on multiple AWS servers but on different 
Clouds like AWS & Digital Ocean and/or your own servers at home/office.


It also doesn't require firewall changes beyond what you already use.

Flockport did a nice writeup on how to install/use PeerVPN 
also. 
Maybe that will help.




Brian, thanks for this pointer. It looks like this is what we'll need to 
solve our networking issues. However, I followed through Flockport's L2 
example and configured peervpn on my two EC2 hosts but I still cannot 
ping containers across hosts. If I could indulge on your expertise I'd 
appreciate some pointers.


The peervpn config file for host 1's config file looks like this:

networkname PEERVPN
psk mypasswd
enabletunneling yes
interface peervpn0
ifconfig4 10.0.0.11/24
port 7000
initpeers 10.0.0.108 7000

and host 2's config looks like this:

networkname PEERVPN
psk mypasswd
enabletunneling yes
interface peervpn0
ifconfig4 10.0.0.12/24
port 7000

The IP address 10.0.0.108 is the private IP that is assigned to eth0 of 
host 2. Should I use this address the host's public IP instead? I did 
try this but I was still not able to ping containers across my two hosts.


From what I can tell peervpn is working correctly. A peervnp0 interface 
has been created on host 1 and assigned address 10.0.0.11, and a 
peervpn0 interface has been created on host 2 and assigned address 
10.0.0.12. The output from peervpn on host 1 shows that it has 
successfully connected to host 2:


opening sockets...
   IPv4/UDP: ok.
   IPv6/UDP: ok.
opening TAP device...
   device "peervpn0": ok.
   done.
preparing P2P engine...
   done.

Client ID: 
7701546D9169C314678310CC92AE22AFDBD0A66F2A39417644C0FC99DF498CB8
Network ID: 
EDE14D183315365CF171564C2BDEDFE3BA51310352EBDBE1CDF49FAF662CF5BB


entering main loop...

[0] 0 peers connected.
[0] resolving 10.0.0.108:7000...
 done.
[12] 1 peers connected.

The output from peervpn on host 2 also shows that one peer has 
connected, so it appears that my configuration is correct. Is there 
something else that's needed on the containers? The container IPs are 
all statically assigned and I don't need DHCP/dnsmasq support.


Any advice would be appreciated.

Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] Networking issues with LXC containers in EC2

[Peter Steele]

I first brought this issue up several weeks ago and have just got back 
to the work where I originally ran into this problem. The scenario is 
simple enough:


- Create two EC2 instances running CentOS 7.1
- Configure these instances to used bridged networking
- Create a LXC container running under each instance, using the command

   lxc-create -t download -n test1 --dir=/hf/test1/rootfs -- -d centos 
-r 7 -a amd64


Each container ends up with a config that looks something like this:

lxc.include = /usr/share/lxc/config/centos.common.conf
lxc.arch = x86_64
lxc.rootfs = /hf/test1/rootfs
lxc.utsname = test1
lxc.network.type = veth
lxc.network.link = br0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx

- Assign each EC2 instance and container a static IP address in the same 
subnet. In my test I assigned the two EC2 instances 10.0.0.102 and 
10.0.0.108 and the two containers 10.0.0.103 and 10.0.0.109.


In this scenario, each EC2 host can ping their own LXC container, and 
they can ping each other. Likewise, the containers can ping their host. 
However, Instance 102 cannot ping the container hosted on instance 108, 
and similarly instance 108 cannot ping the container hosted on instance 
102. If I configure this exact same scenario on real hardware or on KVM 
based virtual machines, this "crosstalk" problem does not 
occur--instance 102 can ping container 109 for example, even though it 
is hosted on a difference instance.


From what I've read, I understand that Amazon has implemented some 
special/restricted behavior for the networking stack of EC2 instances. 
The question I have is whether I can accomplish what I've attempted 
here, specifically, can I access a LXC container hosted on one EC2 
instance directly from another EC2 instance or from another LXC 
container hosted on another EC2 instance?



Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] sshd-keygen fails during container boot

[Peter Steele]

On 12/10/2015 06:13 AM, Peter Steele wrote:

On 12/09/2015 06:43 PM, Serge Hallyn wrote:


Ok, systemd does behave differently if it shouldn't be able

to create devices.  If you add

lxc.cap.drop = mknod sys_rawio

to your configs does that help?


This did not help. I took it a step further and did an install with the
lxc capabilities configured to be as similar as possible to my libvirt
containers and even with this I saw the systemd errors. The only
difference between the cap sets of the two was cap_audit_control; the
lxc containers would not start without this capability but libvirt
containers didn't seem to need it.



I don't know if this is relevant, but we are running the 4.0.5 release 
of the kernel-ml package set from elrepo. The stock CentOS 7.1 kernel 
(3.10) has a bug that impacts bond modes 5 and 6 in containers, so we 
had to find an alternative kernel. Other than a problem with RAID 1 
mdadm volumes, the 4.0.5 kernel has been solid for us with libvirt based 
containers.


I did another test this morning, installing six containers based on the 
downloaded CentOS template. When these containers are started 
simultaneously there are no errors reported with systemd. I then went 
into each container and updated the set of CentOS packages making up the 
template to include the additional rpms that we use in our containers. 
The default template has something like 157 rpms. After installing the 
additional rpms, the containers had 354 installed packages. I then did 
another test of shutting down all the containers and restarting them 
simultaneously using


for vm in `lxc-ls`; do lxc-start -n $vm; done

I hit the systemd errors on the very first try. This would seem to imply 
the problem may be related to one of the additional CentOS rpms that we 
use, although it certainly isn't clear which one (ones?) this might be. 
I'm going to iteratively reduce the set of packages we use to try to 
narrow down the cultprit.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] sshd-keygen fails during container boot

[Peter Steele]

On 12/09/2015 06:43 PM, Serge Hallyn wrote:

Ok, systemd does behave differently if it shouldn't be able
to create devices.  If you add
lxc.cap.drop = mknod sys_rawio
to your configs does that help?
This did not help. I took it a step further and did an install with the 
lxc capabilities configured to be as similar as possible to my libvirt 
containers and even with this I saw the systemd errors. The only 
difference between the cap sets of the two was cap_audit_control; the 
lxc containers would not start without this capability but libvirt 
containers didn't seem to need it.



___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] sshd-keygen fails during container boot

[Peter Steele]

On 12/09/2015 09:43 AM, Serge Hallyn wrote:

And "the systemd errors" is the ssh-keygen ones only?  Or is there
more?
Various services are being impacted, for example, I saw these errors in 
a run yesterday:


Dec  7 13:52:00 pws-vm-00 systemd: Failed at step CGROUP spawning 
/usr/bin/kmod: No such file or directory

Dec  7 13:52:00 pws-vm-00 systemd: Mounted Huge Pages File System.
Dec  7 13:52:00 pws-vm-00 systemd: kmod-static-nodes.service: main 
process exited, code=exited, status=219/CGROUP
Dec  7 13:52:00 pws-vm-00 systemd: Failed to start Create list of 
required static device nodes for the current kernel.
Dec  7 13:52:00 pws-vm-00 systemd: Unit kmod-static-nodes.service 
entered failed state.


Dec  7 13:52:01 pws-vm-00 systemd: Failed at step CGROUP spawning 
/etc/rc.d/init.d/jexec: No such file or directory
Dec  7 13:52:01 pws-vm-00 systemd: jexec.service: control process 
exited, code=exited status=219
Dec  7 13:52:01 pws-vm-00 systemd: Failed to start LSB: Supports the 
direct execution of binary formats..

Dec  7 13:52:01 pws-vm-00 systemd: Unit jexec.service entered failed state.

At least a half dozen different services have failed in the various 
tests I've done, and the set is always different from run to run.

And you do, or do not, also get these with containers created
through the download template?

Most of my tests have been with my custom containers of course since we 
need the additional tools and files that make up our management 
software. I did a test though where I blew away the containers that were 
created by my install framework and replaced them all with the generic 
CentOS download template. I was unable to reproduce the systemd errors 
with this simple container. I then installed the additional OS modules 
and other third party packages that we use in our software on top of 
this basic container and the systemd errors returned. I'm going to break 
this process down a bit more to see if I can identify what additions to 
the base container cause systemd to fail.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] sshd-keygen fails during container boot

[Peter Steele]

On 12/09/2015 01:56 PM, Peter Steele wrote:

On 12/09/2015 11:46 AM, Peter Steele wrote:

On 12/09/2015 10:18 AM, Serge Hallyn wrote:


I suppose just looking at the 'capsh --print' output difference for the
bounding set between the custom containers spawned by lxc and 
libvirt-lxc could

be enlightening.

Here's the diff:

# sdiff lxc libvirt
My apologies here. The output I had pasted in was nicely column 
aligned, with spaces. Something got lost along the way...


Peter


Actually, some tabs got mixed in. Hopefully this will look better:

cap_chown cap_chown
cap_dac_override cap_dac_override
cap_dac_read_search cap_dac_read_search
cap_fowner cap_fowner
cap_fsetid cap_fsetid
cap_kill cap_kill
cap_setgid cap_setgid
cap_setuid cap_setuid
cap_setpcap cap_setpcap
cap_linux_immutable cap_linux_immutable
cap_net_bind_service cap_net_bind_service
cap_net_broadcast cap_net_broadcast
cap_net_admin cap_net_admin
cap_net_raw cap_net_raw
cap_ipc_lock cap_ipc_lock
cap_ipc_owner cap_ipc_owner
> cap_sys_rawio
cap_sys_chroot cap_sys_chroot
cap_sys_ptrace cap_sys_ptrace
> cap_sys_pacct
cap_sys_admin cap_sys_admin
cap_sys_boot cap_sys_boot
> cap_sys_nice
cap_sys_resource cap_sys_resource
cap_sys_tty_config cap_sys_tty_config
cap_mknod <
cap_lease cap_lease
cap_audit_write cap_audit_write
cap_audit_control | cap_setfcap
cap_setfcap,cap_syslog | cap_mac_override
> cap_syslog

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] sshd-keygen fails during container boot

[Peter Steele]

On 12/09/2015 10:18 AM, Serge Hallyn wrote:
This is the kind of thing I'd expect when using cgmanager or lxcfs, 
but not with straight lxc+cgfs. Can you show what /sys/fs/cgroup tree 
and /proc/1/cgroup looks like in a working container? 

As requested:

# ll /sys/fs/cgroup(top level only)
total 0
drwxr-xr-x 3 root root 60 Dec  9 10:12 blkio
lrwxrwxrwx 1 root root 11 Dec  9 10:12 cpu -> cpu,cpuacct
drwxr-xr-x 3 root root 60 Dec  9 10:12 cpu,cpuacct
lrwxrwxrwx 1 root root 11 Dec  9 10:12 cpuacct -> cpu,cpuacct
drwxr-xr-x 3 root root 60 Dec  9 10:12 cpuset
drwxr-xr-x 3 root root 60 Dec  9 10:12 devices
drwxr-xr-x 3 root root 60 Dec  9 10:12 freezer
drwxr-xr-x 3 root root 60 Dec  9 10:12 hugetlb
drwxr-xr-x 3 root root 60 Dec  9 10:12 memory
lrwxrwxrwx 1 root root 16 Dec  9 10:12 net_cls -> net_cls,net_prio
drwxr-xr-x 3 root root 60 Dec  9 10:12 net_cls,net_prio
lrwxrwxrwx 1 root root 16 Dec  9 10:12 net_prio -> net_cls,net_prio
drwxr-xr-x 3 root root 60 Dec  9 10:12 perf_event
dr-xr-xr-x 4 root root  0 Dec  9 10:28 systemd

# cat /proc/1/cgroup
10:hugetlb:/lxc/vm-00
9:perf_event:/lxc/vm-00
8:net_cls,net_prio:/lxc/vm-00
7:freezer:/lxc/vm-00
6:devices:/lxc/vm-00
5:memory:/lxc/vm-00
4:blkio:/lxc/vm-00
3:cpu,cpuacct:/lxc/vm-00
2:cpuset:/lxc/vm-00
1:name=systemd:/system.slice/supervisord.service

And for a bonus:

# mount
/dev/md1 on / type ext4 (rw,relatime,stripe=256,data=ordered)
none on /dev type tmpfs (rw,relatime,size=100k,mode=755)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/devices/virtual/net type sysfs (rw,relatime)
sysfs on /sys/devices/virtual/net type sysfs 
(rw,nosuid,nodev,noexec,relatime)
sysfs on /sys/fs/fuse/connections type sysfs 
(rw,nosuid,nodev,noexec,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs 
(rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
cgroup_root on /sys/fs/cgroup/hugetlb type tmpfs 
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/hugetlb/lxc/vm-00 type cgroup 
(rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup_root on /sys/fs/cgroup/perf_event type tmpfs 
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/perf_event/lxc/vm-00 type cgroup 
(rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup_root on /sys/fs/cgroup/net_cls,net_prio type tmpfs 
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/net_cls,net_prio/lxc/vm-00 type cgroup 
(rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup_root on /sys/fs/cgroup/freezer type tmpfs 
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/freezer/lxc/vm-00 type cgroup 
(rw,nosuid,nodev,noexec,relatime,freezer)
cgroup_root on /sys/fs/cgroup/devices type tmpfs 
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/devices/lxc/vm-00 type cgroup 
(rw,nosuid,nodev,noexec,relatime,devices)
cgroup_root on /sys/fs/cgroup/memory type tmpfs 
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/memory/lxc/vm-00 type cgroup 
(rw,nosuid,nodev,noexec,relatime,memory)
cgroup_root on /sys/fs/cgroup/blkio type tmpfs 
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/blkio/lxc/vm-00 type cgroup 
(rw,nosuid,nodev,noexec,relatime,blkio)
cgroup_root on /sys/fs/cgroup/cpu,cpuacct type tmpfs 
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/cpu,cpuacct/lxc/vm-00 type cgroup 
(rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup_root on /sys/fs/cgroup/cpuset type tmpfs 
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/cpuset/lxc/vm-00 type cgroup 
(rw,nosuid,nodev,noexec,relatime,cpuset,clone_children)
devpts on /dev/lxc/console type devpts 
(rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/lxc/tty1 type devpts 
(rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/lxc/tty2 type devpts 
(rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/lxc/tty3 type devpts 
(rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/lxc/tty4 type devpts 
(rw,relatime,gid=5,mode=620,ptmxmode=666)

tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup 
(rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)

debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)


Interesting.

I suppose just looking at the 'capsh --print' output difference for the
bounding set between the custom containers spawned by lxc and libvirt-lxc could
be enlightening.

Here's 

Re: [lxc-users] sshd-keygen fails during container boot

[Peter Steele]

On 12/08/2015 08:36 PM, Serge Hallyn wrote:

What do you mean by "when the server comes up"?  If you bring up the
server, let it set for 5 mins, then start them, they still fail?
What I meant here was that when my server boots, it launches our 
management software, which in turns launches the containers that are 
defined on that server. The systemd errors occur as the containers are 
started. Delaying when the containers are started doesn't have any 
effect. I've found though that if I put a five second delay between 
starting each container, the systemd errors don't occur (at least not in 
the tests I've run so far). I haven't had this issue with libvirt-lxc, 
and I hope there is a better solution than this arbitrary five second delay.
What lxc version are you using again? 

1.1.5.

Ok, so this shows that in the container 'sys/fs' existed,
but fuse did not.  This suggests that the fuse kernel module
was not done loading yet.

Could you add to /lib/systemd/system/lxc.service the line

ExecStartPre=modprobe fuse

and see if that helps?  (I'm not sure if you'd also need to sleep
a short time to give syfs time to catch up, or if the modprobe
would wait...  you could just use a script that waits until
/sys/fs/fuse exists on the host)

I added this line to lxc.service and that cleared up the fuse issues. 
This did not have any effect on the systemd errors though.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] sshd-keygen fails during container boot

[Peter Steele]

On 12/08/2015 08:00 AM, Serge Hallyn wrote:

Ok, can you change the launch command in the scripts to

lxc-start -n $containername -L /tmp/$containername.cout -l trace -o 
/tmp/$containername.dout -- /sbin/init log_target=console log_level=debug

The console output will go into the .cout file and lxc debug output into .dout.

I've actually made some progress in reproducing this outside of my 
framework. I originally thought the problem only occurred during the 
first boot of the containers. I've discovered that it can happen any 
time the server is rebooted and the containers are started when the 
server comes up. I've only seen this problem when multiple containers 
are starting at the same time.


I incorporated your modified start command into a test as follows:

# for vm in `lxc-ls`; do lxc-start -n $vm -L /tmp/$vm.cout -l trace -o 
/tmp/$vm.dout -- /sbin/init log_target=console log_level=debug; done


This starts all of my previously created containers at roughly the same 
time, and when I do this some of the containers encounter the systemd 
errors I've been seeing. Which containers hit these errors vary from 
test to test. In looking at the .dout logs, I noticed the following:


  lxc-start 1449591253.647 DEBUGlxc_conf - 
conf.c:setup_rootfs:1295 - mounted '/hf/cs/vm-00/rootfs' on 
'/usr/lib64/lxc/rootfs'
  lxc-start 1449591253.647 INFO lxc_conf - 
conf.c:setup_utsname:928 - 'pws-vm-00' hostname has been setup
  lxc-start 1449591253.660 DEBUGlxc_conf - 
conf.c:setup_hw_addr:2368 - mac address 'fe:d6:e8:96:7e:2d' on 'eth0' 
has been setup
  lxc-start 1449591253.660 DEBUGlxc_conf - 
conf.c:setup_netdev:2595 - 'eth0' has been setup
  lxc-start 1449591253.660 INFO lxc_conf - 
conf.c:setup_network:2616 - network has been setup
  lxc-start 1449591253.660 INFO lxc_conf - 
conf.c:mount_autodev:1157 - Mounting container /dev
  lxc-start 1449591253.661 INFO lxc_conf - 
conf.c:mount_autodev:1179 - Mounted tmpfs onto /usr/lib64/lxc/rootfs/dev
  lxc-start 1449591253.661 INFO lxc_conf - 
conf.c:mount_autodev:1197 - Mounted container /dev
  lxc-start 1449591253.661 ERRORlxc_utils - 
utils.c:open_without_symlink:1626 - No such file or directory - Error 
examining fuse in /usr/lib64/lxc/rootfs/sys/fs/fuse/connections
  lxc-start 1449591253.661 INFO lxc_conf - 
conf.c:mount_entry:1727 - failed to mount '/sys/fs/fuse/connections' on 
'/usr/lib64/lxc/rootfs/sys/fs/fuse/connections' (optional): No such file 
or directory


All of the containers report this error, but what caught my eye is the 
mount point referenced in this error. Is this same mount point used for 
all containers that are being started? I assume this error is misleading 
but I tried changing my for loop to add a 1 second delay in starting 
each container, and after doing this there were no systemd errors. 
Unfortunately, adding a delay in my install framework had no effect, so 
I suspect the apparent positive results in adding a delay to the for 
loop was just luck.


This same container reported the following errors in its /var/log/messages:

Dec  8 08:06:39 pws-vm-00 systemd: Starting Dump dmesg to /var/log/dmesg...
Dec  8 08:06:39 pws-vm-00 systemd: Failed at step CGROUP spawning 
/etc/rc.d/init.d/jexec: No such file or directory

Dec  8 08:06:39 pws-vm-00 systemd: Starting Permit User Sessions...
Dec  8 08:06:39 pws-vm-00 systemd: Starting LSB: Bring up/down networking...
Dec  8 08:06:39 pws-vm-00 systemd: jexec.service: control process 
exited, code=exited status=219
Dec  8 08:06:39 pws-vm-00 systemd: Failed to start LSB: Supports the 
direct execution of binary formats..

Dec  8 08:06:39 pws-vm-00 systemd: Unit jexec.service entered failed state.

The .cout file for this same container didn't really have any thing of 
note, other than it also reported some of these errors:


Starting LSB: Bring up/down networking...
Starting D-Bus System Message Bus...
OK  Started D-Bus System Message Bus.
FAILED Failed to start LSB: Supports the direct execution of binary 
formats..

See 'systemctl status jexec.service' for details.

There was nothing related to this error in the .dout file. The start of 
this file does have some warnings:


  lxc-start 1449590798.820 INFO lxc_start_ui - 
lxc_start.c:main:264 - using rcfile /var/lib/lxc/vm-00/config
  lxc-start 1449590798.822 WARN lxc_confile - 
confile.c:config_pivotdir:1801 - lxc.pivotdir is ignored.  It will soon 
become an error.
  lxc-start 1449590798.823 WARN lxc_cgfs - 
cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup 
cpuset unknown to /var/lib/lxc vm-00
  lxc-start 1449590798.823 WARN lxc_cgfs - 
cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup cpu 
unknown to /var/lib/lxc vm-00
  lxc-start 1449590798.823 WARN lxc_cgfs - 
cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup 
blkio unknown to /var/lib/lxc vm-00
  lxc-start 

Re: [lxc-users] sshd-keygen fails during container boot

[Peter Steele]

On 12/08/2015 02:21 PM, Peter Steele wrote:

In this case of course the containers are using the stock downloaded
CentOS 7 image instead of my custom image. I was unable to reproduce
the systemd error through multiple start/stop tests of my
containers. They always started up without any complaints. Granted,
these stock images are simpler than my custom images, but this seems
to point to something in my image that's causing this issue. At
least that gives me a bit to go on, although it's hard to understand
what additional rpm modules would cause systemd to behave this way
during bootup.

Are there specific lxc related configuration files needed in a CentOS 
template that my custom image might not have? I'm asking because in 
looking at /usr/share/lxc/templates/lxc-centos I see there are several 
configuration steps that are performed against the installed CentOS 
rootfs, such as disabling selinix, creating a minimal fstab, etc. Most 
of these are already done as part of my image creation process, but I 
see there are other steps I do not include, such as creating 
/etc/init/lxc-sysinit.conf. This is done as part of a specific check for 
CentOS release 6, so I wonder if there is something equivalent needed 
for CentOS 7?


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] sshd-keygen fails during container boot

[Peter Steele]

On 12/08/2015 11:10 AM, Peter Steele wrote:

On 12/08/2015 08:00 AM, Serge Hallyn wrote:

Ok, can you change the launch command in the scripts to

lxc-start -n $containername -L /tmp/$containername.cout -l trace -o 
/tmp/$containername.dout -- /sbin/init log_target=console 
log_level=debug


The console output will go into the .cout file and lxc debug output 
into .dout.




I did another test and hit the original sshd-keygen error on one of my 
containers. The /var/log/messages file for that container reported the 
following:


Dec  8 11:58:13 pws-vm-04 systemd: Starting OpenSSH Server Key Generation...
Dec  8 11:58:13 pws-vm-04 systemd: Failed at step CGROUP spawning 
/usr/sbin/sshd-keygen: No such file or directory
Dec  8 11:58:13 pws-vm-04 systemd: sshd-keygen.service: main process 
exited, code=exited, status=219/CGROUP
Dec  8 11:58:13 pws-vm-04 systemd: Failed to start OpenSSH Server Key 
Generation.
Dec  8 11:58:13 pws-vm-04 systemd: Unit sshd-keygen.service entered 
failed state.

Dec  8 11:58:16 pws-vm-04 systemd: Starting OpenSSH server daemon...
Dec  8 11:58:16 pws-vm-04 systemd: Started OpenSSH server daemon.
Dec  8 11:58:16 pws-vm-04 sshd: Could not load host key: 
/etc/ssh/ssh_host_rsa_key
Dec  8 11:58:16 pws-vm-04 sshd: Could not load host key: 
/etc/ssh/ssh_host_ecdsa_key
Dec  8 11:58:16 pws-vm-04 sshd: Could not load host key: 
/etc/ssh/ssh_host_ed25519_key
Dec  8 11:58:58 pws-vm-04 sshd[722]: error: Could not load host key: 
/etc/ssh/ssh_host_rsa_key
Dec  8 11:58:58 pws-vm-04 sshd[722]: error: Could not load host key: 
/etc/ssh/ssh_host_ecdsa_key
Dec  8 11:58:58 pws-vm-04 sshd[722]: error: Could not load host key: 
/etc/ssh/ssh_host_ed25519_key
Dec  8 11:59:00 pws-vm-04 sshd[724]: error: Could not load host key: 
/etc/ssh/ssh_host_rsa_key
Dec  8 11:59:00 pws-vm-04 sshd[724]: error: Could not load host key: 
/etc/ssh/ssh_host_ecdsa_key
Dec  8 11:59:00 pws-vm-04 sshd[724]: error: Could not load host key: 
/etc/ssh/ssh_host_ed25519_key
Dec  8 11:59:00 pws-vm-04 sshd[726]: error: Could not load host key: 
/etc/ssh/ssh_host_rsa_key
Dec  8 11:59:00 pws-vm-04 sshd[726]: error: Could not load host key: 
/etc/ssh/ssh_host_ecdsa_key
Dec  8 11:59:00 pws-vm-04 sshd[726]: error: Could not load host key: 
/etc/ssh/ssh_host_ed25519_key
Dec  8 11:59:00 pws-vm-04 sshd[728]: error: Could not load host key: 
/etc/ssh/ssh_host_rsa_key
Dec  8 11:59:00 pws-vm-04 sshd[728]: error: Could not load host key: 
/etc/ssh/ssh_host_ecdsa_key
Dec  8 11:59:00 pws-vm-04 sshd[728]: error: Could not load host key: 
/etc/ssh/ssh_host_ed25519_key
Dec  8 11:59:00 pws-vm-04 sshd[730]: error: Could not load host key: 
/etc/ssh/ssh_host_rsa_key
Dec  8 11:59:00 pws-vm-04 sshd[730]: error: Could not load host key: 
/etc/ssh/ssh_host_ecdsa_key
Dec  8 11:59:00 pws-vm-04 sshd[730]: error: Could not load host key: 
/etc/ssh/ssh_host_ed25519_key
Dec  8 11:59:02 pws-vm-04 sshd[831]: error: Could not load host key: 
/etc/ssh/ssh_host_rsa_key
Dec  8 11:59:02 pws-vm-04 sshd[831]: error: Could not load host key: 
/etc/ssh/ssh_host_ecdsa_key
Dec  8 11:59:02 pws-vm-04 sshd[831]: error: Could not load host key: 
/etc/ssh/ssh_host_ed25519_key
Dec  8 11:59:02 pws-vm-04 sshd[833]: error: Could not load host key: 
/etc/ssh/ssh_host_rsa_key
Dec  8 11:59:02 pws-vm-04 sshd[833]: error: Could not load host key: 
/etc/ssh/ssh_host_ecdsa_key
Dec  8 11:59:02 pws-vm-04 sshd[833]: error: Could not load host key: 
/etc/ssh/ssh_host_ed25519_key


The .cout file for this container looked normal except, of course, for 
some ssh related messages:


Starting OpenSSH Server Key Generation...
FAILED Failed to start OpenSSH Server Key Generation.
See 'systemctl status sshd-keygen.service' for details.
Starting OpenSSH server daemon...

The .dout file had no errors related to sshd in this particular test.

In another test, I tried creating several containers with the command

lxc-create -n testN -t download -- -d centos -r 7 -a amd64

and then started them all in the same manner as my other tests using:

for vm in `lxc-ls`; do lxc-start -n $vm -L /tmp/$vm.cout -l trace -o 
/tmp/$vm.dout -- /sbin/init log_target=console log_level=debug; done


In this case of course the containers are using the stock downloaded 
CentOS 7 image instead of my custom image. I was unable to reproduce the 
systemd error through multiple start/stop tests of my containers. They 
always started up without any complaints. Granted, these stock images 
are simpler than my custom images, but this seems to point to something 
in my image that's causing this issue. At least that gives me a bit to 
go on, although it's hard to understand what additional rpm modules 
would cause systemd to behave this way during bootup.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] sshd-keygen fails during container boot

[Peter Steele]

On 12/07/2015 07:49 AM, Serge Hallyn wrote:

Quoting Peter Steele (pwste...@gmail.com):

I'm actually not (yet) running lxcfs. My understanding was that it
isn't absolutely required but it does offer several benefits. I'd
planned to tackle lxcfs after getting things running without it
first, and my containers appear to be running reasonably okay
(although I am seeing some odd behavior). If it's *is* needed, then
I guess it's time to tackle lxcfs.

Oh right, since you're running your containers unconfined, you should
be fine without it.

Still would be useful to see systemd's output :)

I'm having some trouble reproducing this behavior when I launch a 
container with


lxc-start -n containername -F -o /dev/stdout -- /sbin/init log_target=console 
log_level=debug


I'm seeing these errors when the containers are launched as part of my 
automated installation framework. The installs that are done in this 
manner are hands-free and the containers, once installed, are launched 
with a simple


lxc-start -n containername

Multiple containers are created and launched in parallel as part of this 
installation system. This is all controlled through automation (Python), 
and the framework expects the launched container processes to become 
daemons. I don't get the detailed startup output that happens when 
lxc-start is run in the foreground. If I create a container 
interactively and launch it with -F option manually, there are no errors 
in /var/log/messages.


The errors I'm seeing are not limited to the sshd-keygen errors I 
originally reported. In further tests since then I see similar errors 
being reported for a variety of things. For example:


Dec  7 13:52:00 pws-vm-00 systemd: Failed at step CGROUP spawning 
/usr/bin/kmod: No such file or directory

Dec  7 13:52:00 pws-vm-00 systemd: Mounted Huge Pages File System.
Dec  7 13:52:00 pws-vm-00 systemd: kmod-static-nodes.service: main 
process exited, code=exited, status=219/CGROUP
Dec  7 13:52:00 pws-vm-00 systemd: Failed to start Create list of 
required static device nodes for the current kernel.
Dec  7 13:52:00 pws-vm-00 systemd: Unit kmod-static-nodes.service 
entered failed state.


Dec  7 13:52:01 pws-vm-00 systemd: Failed at step CGROUP spawning 
/etc/rc.d/init.d/jexec: No such file or directory
Dec  7 13:52:01 pws-vm-00 systemd: jexec.service: control process 
exited, code=exited status=219
Dec  7 13:52:01 pws-vm-00 systemd: Failed to start LSB: Supports the 
direct execution of binary formats..

Dec  7 13:52:01 pws-vm-00 systemd: Unit jexec.service entered failed state.

I've also seen errors reported for network.service and others. In every 
case, if I reboot my server forcing the hosted containers to be 
restarted, everything comes up fine. If I run the same installation 
framework using the same container images but instead use libvirt-lxc 
create/start commands, I do not see these systemd errors. To the best of 
my knowledge, the LXC config I'm using more or less matches the config 
I'm using with my libvirt containers, for example


# cat /var/lib/lxc/vm-00/config
lxc.utsname = pws-vm-00
lxc.include = /var/lib/hf/lxc.conf
lxc.network.veth.pair = vm-00
lxc.network.hwaddr = fe:d6:e8:96:7e:2d
lxc.rootfs = /hf/cs/vm-00/rootfs
lxc.cgroup.memory.limit_in_bytes = 1073741824
lxc.cgroup.memory.memsw.limit_in_bytes = 2147483648
lxc.hook.autodev = /var/lib/lxc/vm-00/autodev
lxc.cgroup.devices.allow = b 8:3 rwm

# cat /var/lib/hf/lxc.conf
lxc.include = /usr/share/lxc/config/centos.common.conf
lxc.arch = x86_64

# Networking defaults
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0

Am I missing anything in my config? I am running LXC version 1.1.5 under 
a CentOS host. All containers are also CentOS and are privileged.


Peter


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Converting from libvirt lxc

[Peter Steele]

On 12/03/2015 08:42 PM, Fajar A. Nugraha wrote:

lxc.autodev = 1

That is not common.conf (though I'm not sure whether it matters)
I included this early on when I was encountering the funky udev issue. 
it didn't help but I kept it in place, admittedly for no good reason.

lxc.kmsg = 0

Neither is that. Though it should be the default value
In my original tests with LXC 1.0.7 I hit an issue where systemd on my 
containers was running at 100%. I did some research and found the 
problem described with the solution suggested being to add this lxc.kmsg 
line. This did in fact solve the problem. I just did a test without this 
though and the CPU issue did not occur, so presumably LXC 1.1.5 has 
fixed this problem.


# Remove capabilities we don't want in containers
lxc.cap.drop = mac_admin mac_override sys_time sys_module

centos.common.conf also has  lxc.cap.drop = sys_nice sys_pacct 
sys_rawio. You don't have that.


I excluded this line because we need sys_nice enabled in our containers. 
I wasn't sure about sys_pacct and sys_rawio and was going to do more 
investigation on these later.



lxc.cgroup.devices.allow = c 5:0 rwm


lxc.cgroup.devices.allow = c 136:* rwm
## /dev/ptmx pty master
lxc.cgroup.devices.allow = c 5:2 rwm


you' re missing 5:1 (console), 10:229 (fuse). Both are in common.conf.

There was in fact no common.conf in the 1.0.7 release I originally was 
using, and the centos.common.conf did not have the console and fuse 
entries. When I switched to 1.1.5 common.conf was introduced and these 
device definitions were moved there. I took a quick look at these 
definitions and added the fuse entry but didn't notice console had been 
added as well. Thanks for noticing this.


Is there a reason why you didn't test simply using the same config, 
which also does the "includes" instead of copying SOME of them? Is 
there a reason wht you don't copy ALL of them? It should be easier to 
start with a known good setup, then do incremental changes.
Well, as I said we need sys_nice and so that was one reason why I didn't 
want to use the config files directly. I also noticed that proc was 
mounted in mixed mode and we need at least some rw access to a portion 
of /proc/sys, and I thought I'd probably need to change this mixed 
entry. Since all of our work is based on centos, I also didn't see the 
need to include the lxc-templates rpm in my package set. Our server is 
based on a minimal centos config and I try to avoid adding additional 
rpms if I can avoid it.


That said, I did change my install framework this morning to include 
lxc-templates and to use centos.common.conf and common.conf directly 
rather rely on than my manually crafted version. This causes sys_nice to 
be dropped, as I just mentioned above, and I need to solve that problem. 
So, if I have this:


lxc.include = /usr/share/lxc/config/centos.common.conf

can I then add the entry

lxc.cap.keep = sys_nice

after this? Based on the description in the man page I assume this will 
not just add this one capability but will instead remove everything 
except this. So, what's the correct way to use common.conf and to re-add 
dropped capabilities?


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] sshd-keygen fails during container boot

[Peter Steele]

I'm seeing these messages on some of my containers during their initial 
start-up:


systemd: Failed at step CGROUP spawning /usr/sbin/sshd-keygen: No such 
file or directory
systemd: sshd-keygen.service: main process exited, code=exited, 
status=219/CGROUP

systemd: Failed to start OpenSSH Server Key Generation.
systemd: Unit sshd-keygen.service entered failed state.

The net effect of this of course is that I cannot ssh into the 
containers that encounter this problem. The odd thing is that 
/usr/sbin/sshd-keygen *does* exist, and restarting the containers 
corrects the problem (the keygen works on the reboot). It's not clear to 
me why the containers think this file is missing and why some containers 
running pretty much the identical image do not hit this problem. Has 
anyone seen this?


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] sshd-keygen fails during container boot

[Peter Steele]

On 12/04/2015 01:38 PM, Serge Hallyn wrote:

My guess is that the no such file or directory is talking about a
cgroup dir.  what does /proc/1/cgroup in the container show? Make sure
to run the latest lxcfs on the host, as that's needed because
systemd moves itself to name=systemd:/init.scope cgroup.

When systemd is having trouble, you can usually find out more info by running

lxc-start -n containername -F -o /dev/stdout -- /sbin/init log_target=console 
log_level=debug

I'm actually not (yet) running lxcfs. My understanding was that it isn't 
absolutely required but it does offer several benefits. I'd planned to 
tackle lxcfs after getting things running without it first, and my 
containers appear to be running reasonably okay (although I am seeing 
some odd behavior). If it's *is* needed, then I guess it's time to 
tackle lxcfs.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] setproctitle error doing an lxc-start

[Peter Steele]

On 12/02/2015 03:01 PM, Stéphane Graber wrote:

On Wed, Dec 02, 2015 at 02:56:51PM -0800, Peter Steele wrote:

 From the searches I've done this seems to be a known issue but I'm
not clear what the solution is. I'm using LXC 1.1.5 under CentOS 7.1
and created a container using

# lxc-create -t centos -n test1

This completed without issues. I followed this up with a start command:

# lxc-start -n test1
lxc-start: utils.c: setproctitle: 1461 Invalid argument - setting
cmdline failed
lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 346 To get more details, run the
container in foreground mode.
lxc-start: lxc_start.c: main: 348 Additional information can be
obtained by setting the --logfile and --logpriority options.

I had earlier been using LXC 1.0.7 and did not get this error when I
ran lxc-start. Do I need to role back or is there a workaround?

Peter

The setproctitle error can be ignored, we've silenced it later. It
doesn't actually affect container startup at all, so your failure is
coming from something else.

Please follow the recommendations in the error message and re-run your
lxc-start with the -F flag to get the actual error message.
I realized the issue at 3:00 am (couldn't sleep). The default centos 
template is configured to use lxcbr0 whereas my servers have br0. I just 
changed the container's config to use br0 instead and the container came 
up without issues (other than this setproctitle error that apparently 
can be ignored).


The important thing is that udev is not running, unlike my earlier 
tests, so I'm making progress.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Converting from libvirt lxc

[Peter Steele]

On 12/02/2015 08:47 PM, Fajar A. Nugraha wrote:
On Thu, Dec 3, 2015 at 1:14 AM, Peter Steele <pwste...@gmail.com 
<mailto:pwste...@gmail.com>> wrote:



On 12/02/2015 07:23 AM, Fajar A. Nugraha wrote:

On Wed, Dec 2, 2015 at 9:49 PM, Peter Steele <pwste...@gmail.com
<mailto:pwste...@gmail.com>>wrote:

On 12/01/2015 08:25 PM, Fajar A. Nugraha wrote:

Is there a reason why you can't install a centos7 container
using the download template? It would've been MUCH easier,
and some of the things you asked wouldn't even be an issue.




lxc-create -t centos -n test1

to create a container using the centos default settings. The
resulting config file doesn't look a whole lot different than my
manually crafted version.



You DID notice that repeatedly say "DOWNLOAD template"? as in someting 
like


# lxc-create -t download -n c7 -- -d centos -r 7 -a amd64

The template was downloaded automatically when I ran the lxc-create 
command the first time. Is there a difference in how the download is 
done using the command you've listed above?




Short version: if you use 
http://copr.fedoraproject.org/coprs/thm/lxc1.1/ , you need to do some 
things first:

- edit /etc/sysconfig/lxc, USE_LXC_BRIDGE="true"
- systemctl enable lxc-net
- systemctl enable lxc
- systemctl start lxc-net
- brctl show
- ip ad li lxcbr0
If you HAVE lxcbr0 with the default ip 10.0.3.1 (you can change this 
later), you're all set. If not, doublecheck your setup.
If you're asking "where's the docs that mention this", as the package 
manager :)


The alternative is to configure your own bridge and configure your 
containers to use that. After you get the bridge working, you can 
start and monitor its boot progress with something like this:


That's exactly what I did. I realized this later that the default centos 
container assumes you have a lxcbr0 defined (I had hit this issue 
before). My servers use br0 so I just changed my test container's config 
and it came up fine. Most importantly, the udev service was not running. 
So I tweaked the lxc config I had in my custom install process to more 
closely match what was used in my standalone test and my containers are 
now coming up fine, or at least udev is no longer running. The /dev 
directory still has more entries than my libvirt containers (for 
example, /dev/snd is still present), but at least there are no udev 
errors in /var/log/messages.


There *are* other issues (our software isn't running properly), but I 
think the major container issues have been resolved. I changed a few 
things, including the version of LXC that I'm using, so it's hard to say 
what the culprit was with regards to this udev issue.



# lxc-start -n c7;lxc-console -n c7 -t 0

The benefit of using this approach instead of "lxc-start -F" is that 
you can detach the console session later using "ctrl-a q". Note that 
you can NOT login on this console yet, as by default the root password 
is not set. From another shell session, you need to do


# lxc-attach -n c7 -- passwd

Then you can login from the console session. You'll then see on the 
container (I tested this just now on up-to-date centos7)


[root@c7 ~]# ls /dev
console  core  fd  full  hugepages  initctl  log  lxc  mqueue  null 
 ptmx  pts  random  shm  stderr  stdin  stdout  tty  tty1  tty2  tty3 
 tty4  urandom  zero


Apparently this works even without lxfs.

If you DO manage to get lxcfs installed and working later (disclaimer: 
I've only use it on ubuntu and debian), you'll be able to get some 
additional benefits like the container only seeing its allocated 
resources (set using "lxc.cgroup" settings on lxc config file). For 
example, if "lxc.cgroup.cpuset.cpus = 0", then the container will only 
use cpu0, and "htop" or "cat /proc/cpuinfo" will only show 1 cpu even 
when your host has multiple cpus.


That would definitely be nice. Libvirt does a reasonably good job in 
this area but it is far from complete, with /proc/cpuinfo being one of 
the weak points. I'll definitely have to check out lxcfs.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Converting from libvirt lxc

[Peter Steele]

On 12/03/2015 07:25 AM, Fajar A. Nugraha wrote:
On Thu, Dec 3, 2015 at 9:27 PM, Peter Steele <pwste...@gmail.com 
<mailto:pwste...@gmail.com>>wrote:


On 12/02/2015 08:47 PM, Fajar A. Nugraha wrote:


centos template -> download lost of packages (i.e. RPM) one by one 
using yum, and then install it


download template -> download one big tar.xz file (plus several small 
config files), and then extract it. MUCH faster, and works for unpriv 
containers as well (not sure what the current state of unpriv 
containers on centos though)


However I was actually more concerned about the fact that the 
templates are maintained separately, so there could be some difference 
in the resulting container/config. The download template works (I've 
tested it), while (based on your previous output) the centos template 
doesn't provide the desired /dev entries.


I just did a test using the download approach and it worked nicely, 
obviously much cleaner than downloading the rpms individually. The 
containers created from the two approaches seem to be identical, as far 
as a cursory glance is concerned, with identical config files.


Which is why I suggested the download template.

I also tested using the resulting config with rootfs replaced by a 
"native" centos7 install (to be exact, a disk clone of minimal centos7 
install on virtualbox), still result in the desired /dev entries (i.e. 
minimal /dev entries, no /dev/snd).


I can't really use the downloaded template for our rootfs, as I 
explained earlier. We already have a process that generates a custom 
centos tar ball with the specific set of packages that we need in our 
containers. Our tarball includes other third party packages as well, 
such as supervisord and ctdb. I've used the downloaded template's config 
file to create a custom config for our containers. The container 
specific portion of the config looks something like this:


lxc.utsname = pws-vm-03
lxc.rootfs = /hf/cs/vm-03/rootfs
lxc.network.veth.pair = vm-03
lxc.network.hwaddr = fe:d6:e8:dc:c8:db
lxc.rootfs = /hf/cs/vm-03/rootfs
lxc.cgroup.memory.limit_in_bytes = 1073741824
lxc.cgroup.memory.memsw.limit_in_bytes = 2147483648
lxc.include = /var/lib/hf/lxc.conf

and the settings that are common to all containers (lxc.conf) include 
the following:


lxc.autodev = 1
lxc.devttydir = lxc
lxc.tty = 4
lxc.pts = 1024
lxc.kmsg = 0
lxc.arch = x86_64

# Networking defaults
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0

# Remove capabilities we don't want in containers
lxc.cap.drop = mac_admin mac_override sys_time sys_module

# Set the pivot directory
lxc.pivotdir = lxc_putold

# Control Group devices: all denied except those white-listed
lxc.cgroup.devices.deny = a
## Allow any mknod (but not reading/writing the node)
lxc.cgroup.devices.allow = c *:* m
lxc.cgroup.devices.allow = b *:* m
## /dev/null
lxc.cgroup.devices.allow = c 1:3 rwm
## /dev/zero
lxc.cgroup.devices.allow = c 1:5 rwm
## /dev/full
lxc.cgroup.devices.allow = c 1:7 rwm
## /dev/tty
lxc.cgroup.devices.allow = c 5:0 rwm
## /dev/random
lxc.cgroup.devices.allow = c 1:8 rwm
## /dev/urandom
lxc.cgroup.devices.allow = c 1:9 rwm
## /dev/tty[1-4] ptys and lxc console
lxc.cgroup.devices.allow = c 136:* rwm
## /dev/ptmx pty master
lxc.cgroup.devices.allow = c 5:2 rwm

# Setup the default mounts
lxc.mount.auto = cgroup:mixed proc:mixed sys:mixed
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none 
bind,optional 0 0


As you can see this was largely pulled from centos.common.conf and 
common.conf.I assume something isn't quite right since I see more 
entries under /dev than I do when I'm running under libvirt, using the 
same custom tarball. I'll be satisfied with this for now though as long 
as the extra entries aren't causing issues.


There *are* other issues (our software isn't running properly),
but I think the major container issues have been resolved.


Which is?

Well, mainly the udev issue, plus the fact that the containers booted 
*really* slowly.


I changed a few things, including the version of LXC that I'm
using, so it's hard to say what the culprit was with regards to
this udev issue.



IIRC systemd containers are only supported on lxc-1.1.x, so upgrading 
lxc probably has a big part in that.

Yeah, things definitely started working better after I upgraded.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Converting from libvirt lxc

[Peter Steele]

On 12/03/2015 11:27 AM, Neil Greenwood wrote:

On 3 December 2015 17:10:29 GMT+00:00, Peter Steele <pwste...@gmail.com> wrote:

I can't really use the downloaded template for our rootfs, as I
explained earlier. We already have a process that generates a custom
centos tar ball with the specific set of packages that we need in our
containers. Our tarball includes other third party packages as well,
such as supervisord and ctdb. I've used the downloaded template's
config

I am not an expert with LXC, but I think you can get your tarball working using 
'-t download' a.k.a. the download template. You would use petercentos rather 
centos as the template to download, and provide a petercentos configuration 
that points to your tarball. Obviously you will use your company name in the 
production version :-)

Neil
That's where I'd like to get to eventually. That said, since this is all 
part of an automated process and no one actually runs lxc-create 
interactively (it's all done in Python scripts), it doesn't really 
matter if our custom tar ball gets installed as an official formal 
template. The lxc-create command works quite well using "none" for the 
template, as long as the rootfs is put in place through some other 
means. Since we're transitioning from libvirt to LXC and need to keep 
both frameworks in play for a while, it's easier in our code to install 
rootfs explicitly rather than having lxc-create do it.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Converting from libvirt lxc

[Peter Steele]

On 12/01/2015 08:25 PM, Fajar A. Nugraha wrote:
Is there a reason why you can't install a centos7 container using the 
download template? It would've been MUCH easier, and some of the 
things you asked wouldn't even be an issue.
Well, there's a bit of history involved. Originally we were building 
systems using KVM based virtual machines. As part of this, we developed 
a custom installation process where the user burns our boot image onto a 
USB stick and boots a server with this image. There is a corresponding 
UI a user installs on his workstation that lets the user communicate 
with the server being installed and customize the installation. After 
the user hits "start install" in the UI the process is mostly hands-free 
where the server gets installed with our CentOS based hypervisor and 
then VMs being automatically created on top of that. The OS image that's 
used for the VMs gets created on the fly as part of this process. 
Everything is self-contained and we cannot make the assumption that the 
server has access to the external internet. Everything that's needed is 
on the boot stick.


This past year we moved to libvirt based containers instead of VMs, and 
we were able to use the same image we build for the VMs for the libvirt 
containers, with only a few minor changes, and our CentOS hypervisor now 
manages containers instead of VMs. The installation process is largely 
unchanged from the user's perspective and the fact that containers are 
being used is completely hidden.


Unfortunately, libvirt-lxc is being deprecated by Redhat, and that means 
it will eventually disappear in CentOS. That's why we're moving to LXC, 
but it is a transitional process. With the work I'm doing, when I create 
a boot image it can be flagged as either using libvirt containers or LXC 
containers. I need to keep the overall process as similar as possible 
and eventually, when the port to LXC is complete, we can make the 
official switch to LXC. We will need to handle upgrades in the field as 
well for our existing customers, converting their libvirt based systems 
to LXC based systems in-place. That means we'll need to create LXC 
flavored config files that match the XML definitions of the libvirt 
containers. We ultimately had to do the same thing when we transitioned 
from VMs to libvirt containers, so we know the process well. We just 
have to learn the particulars of LXC containers.


So, that long winded answer is why we can't just use the LXC template 
for CentOS directly. I was assuming (hoping) that the libvirt container 
image we build would be largely LXC friendly. Apparently it's not going 
to be quite as straightforward as I'd hoped. I'm going to have to 
dissect the steps used for creating a CentOS LXC template and make sure 
our container image provides what is needed/expected by LXC.


My containers are coming up but things are running really slowly,
although CPU usage is low. I'm not entirely sure what's going on
and need to do some more digging.


Works for me. However I seem to recall an issue with centos' version 
of systemd sometime ago.


Yes, I hit that systemd issue early on and found the fix for it. The 
slowness I'm seeing now is something else.



whereas under LXC /dev has the following:


I know how to trim the /dev/ttyN entries to match libvirt, but I'm
not sure what's needed for the others. For example, how do I get
rid of /dev/snd?


Here's mine. Centos 7 container, ubuntu 14.04 host, lxc-1.1.4 and 
lxcfs-0.10 from ubuntu ppa:


c7 / # ls /dev
console  core  fd  full  hugepages  initctl  log  lxc  mqueue  null 
 ptmx  pts  random  shm  stderr  stdin  stdout  tty  tty1  tty2  tty3 
 tty4  urandom  zero



That ultimately is very similar to my libvirt dev list. I clearly need 
to dig into the CentOS template to see what's being done differently 
compared to my libvirt image.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Converting from libvirt lxc

[Peter Steele]

On 12/02/2015 10:29 AM, Thomas Moschny wrote:

2015-12-02 19:14 GMT+01:00 Peter Steele <pwste...@gmail.com>:

I am using the version 1.0.7 RPMs that are available on EPEL. I assume there
are no RPMs available for 1.1? We tend to use binary versions of the third
party packages we've included in our system but I will check out 1.1 and
investigate lxcfs. The set of LXC RPMs I installed from EPEL are:

lua-lxc.1.0.7-4.el7.x86_64
lxc.1.0.7-4.el7.x86_64
lxc-libs.1.0.7-4.el7.x86_64
lxc-templates.1.0.7-4.el7.x86_64

LXC 1.0.8 RPMs are currently in testing. I also maintain a copr
repository for LXC 1.1.x here:
http://copr.fedoraproject.org/coprs/thm/lxc1.1/


Perfect! Thanks.

Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Converting from libvirt lxc

[Peter Steele]

On 12/02/2015 08:54 AM, Peter Steele wrote:

On 12/02/2015 08:09 AM, Saint Michael wrote:
I could not find on Google any mention of Red Hat killing LXC on 
Libvirt. Care to elaborate?


Here's the first reference I came across a few months ago: 
https://access.redhat.com/articles/1365153. There's no date indicated 
here so I really don't know what this means, but I just did another 
search to see if I could find some more information. I came across 
this thread:


https://www.redhat.com/archives/libvirt-users/2015-August/msg00026.html

This was a fairly recent thread and I'd not found this before. If you 
read through the follow-ups apparently libvirt-lxc is *not* being 
deprecated:


https://www.redhat.com/archives/libvirt-users/2015-August/msg00030.html

So, it appears I was mistaken. I'm not sure where that leaves me 
though. One issue we've had with libvirt-lxc is that although it's a 
great product there seems to be very little activity with the project. 
My own posts to the libvirt-lxc mailing list often go unanswered, 
whereas this mailing list (by comparison) is great. Posts to more 
generic forums are more often than not to find information related to 
LXC and not libvirt-lxc. The community for libvirt-lxc just doesn't 
seem that large.


Looks like I have some thinking to do...
Actually, I guess what this means is just that Redhat is deprecating it, 
but the libvirt-lxc project as a whole is still moving forward for 
whatever distros want to include it, so we probably could still use it 
as a non-Redhat supported package. But, I think we'll continue with our 
move to LXC since it does seem to have a lot of momentum going for it. I 
think it's the right direction to move for the long term.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Converting from libvirt lxc

[Peter Steele]

On 12/02/2015 10:38 AM, Saint Michael wrote:
In my unauthorized opinion, Ubuntu has a much sold LXC that the Red 
Hat derivatives. That is why I run my apps in Fedora containers and my 
LXC servers in Ubuntu, The Fedora management does not quite understand 
that LXC is the only possible game, not Docker,


Our product is based around a CentOS environment and switching to 
Ubuntu/Fedora would unfortunately not be a trivial process. Even moving 
from CentOS 6.5 to CentOS 7.1 was a big project for us. Once you have 
customers, you sort of get locked in...



___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] setproctitle error doing an lxc-start

[Peter Steele]

From the searches I've done this seems to be a known issue but I'm not 
clear what the solution is. I'm using LXC 1.1.5 under CentOS 7.1 and 
created a container using


# lxc-create -t centos -n test1

This completed without issues. I followed this up with a start command:

# lxc-start -n test1
lxc-start: utils.c: setproctitle: 1461 Invalid argument - setting 
cmdline failed

lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 346 To get more details, run the container 
in foreground mode.
lxc-start: lxc_start.c: main: 348 Additional information can be obtained 
by setting the --logfile and --logpriority options.


I had earlier been using LXC 1.0.7 and did not get this error when I ran 
lxc-start. Do I need to role back or is there a workaround?


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Converting from libvirt lxc

[Peter Steele]

On 12/02/2015 11:39 AM, Saint Michael wrote:

I don't explain myself.
You need an Ubuntu 14.04 server with nothing else running, but LXC. 
100% of the real work gets done via Centos containers.It works 
perfectly and it is rock solid.
The only thing on top is the latest available kernel 
3.19.0-33-generic. Yo never have to login or otherwise touch Ubuntu, 
it becomes a simple container host. I have literally hundreds of 
containers with this architecture. For some reason, the fact that 
Ubuntu does not use systemd, makes it stable and almost perfect. I 
cannot explain it, but it becomes like the engine of a Mercedes, you 
know it is there, but you don't need to see it, it becomes invisible. 
I could never use Fedora as a good container host, for you end up 
having to compile your own RPMs and it fails often. They just don't 
take LXC seriously, or they would be at the same level of Ubuntu.


Our software runs in CentOS containers which in turn run under CentOS 
based hypervisors, working together in a cluster. Even switching out our 
servers to run Ubuntu instead of CentOS would be a non-trivial process. 
We'd need to support an upgrade path for example where we would upgrade 
our customers' servers, swapping out CentOS in-place in favor or Ubuntu. 
Doable but not something we really have the bandwidth to take on and 
keep with our release schedule.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Converting from libvirt lxc

[Peter Steele]

On 12/02/2015 10:42 AM, Peter Steele wrote:

On 12/02/2015 10:29 AM, Thomas Moschny wrote:

2015-12-02 19:14 GMT+01:00 Peter Steele <pwste...@gmail.com>:
I am using the version 1.0.7 RPMs that are available on EPEL. I 
assume there
are no RPMs available for 1.1? We tend to use binary versions of the 
third
party packages we've included in our system but I will check out 1.1 
and

investigate lxcfs. The set of LXC RPMs I installed from EPEL are:

lua-lxc.1.0.7-4.el7.x86_64
lxc.1.0.7-4.el7.x86_64
lxc-libs.1.0.7-4.el7.x86_64
lxc-templates.1.0.7-4.el7.x86_64

LXC 1.0.8 RPMs are currently in testing. I also maintain a copr
repository for LXC 1.1.x here:
http://copr.fedoraproject.org/coprs/thm/lxc1.1/


Perfect! Thanks.

I've downloaded  1.1.5-1 rpms for the set of LXC packages that I'm 
using--thanks very much for pointing me to this site. It's been 
suggested that I should check out lxcfs but I don't see an rpm for this 
in your copr repository. Is there an rpm available for this somewhere or 
do I need to build it from source?


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] Converting from libvirt lxc

[Peter Steele]

This message is a bit long and I apologize for that, although the bulk 
is cut-and-paste output. I'm migrating our container project from 
libvirt-lxc under CentOS 7.1 to LXC and I'm seeing some errors in 
/var/log/messages that I don't see in libvirt-lxc. The LXC containers I 
am creating are based on the same custom CentOS image that I've been 
using with libvirt-lxc. My assumption is that this image should be able 
to be used without any significant changes as long as I have the 
appropriate config file defined for this image when an LXC container is 
installed.


The lxc-create command I'm using looks generally like this:

# lxc-create -f /hf/cs/vm-03/config -t /bin/true -n vm-03 
--dir=/hf/cs/vm-03/rootfs


where the config file has the following options defined:

lxc.tty = 4
lxc.pts = 1024
lxc.kmsg = 0
lxc.utsname = vm-03
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.veth.pair = vm-03
lxc.network.hwaddr = fe:d6:e8:f2:aa:e6
lxc.rootfs = /hf/cs/vm-03/rootfs

When a container boots, I'm seeing the set of errors below:

Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sdb, 
10) failed: No such file or directory
Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, 
/dev/sdb1, 10) failed: No such file or directory
Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sda, 
10) failed: No such file or directory
Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, 
/dev/sdb2, 10) failed: No such file or directory
Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, 
/dev/sdb4, 10) failed: No such file or directory
Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, 
/dev/sdb3, 10) failed: No such file or directory
Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, 
/dev/sda4, 10) failed: No such file or directory
Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, 
/dev/sda3, 10) failed: No such file or directory
Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, 
/dev/sda2, 10) failed: No such file or directory
Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, 
/dev/sda1, 10) failed: No such file or directory
Nov 30 09:28:49 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sdc, 
10) failed: No such file or directory
Nov 30 09:28:49 vm-03 systemd-udevd: inotify_add_watch(7, 
/dev/sdc2, 10) failed: No such file or directory
Nov 30 09:28:49 vm-03 systemd-udevd: inotify_add_watch(7, 
/dev/sdc1, 10) failed: No such file or directory

...
Nov 30 09:28:56 vm-03 systemd-udevd: Failed to apply ACL on 
/dev/snd/hwC0D0: No such file or directory
Nov 30 09:28:56 vm-03 systemd-udevd: Failed to apply ACL on 
/dev/snd/controlC0: No such file or directory
Nov 30 09:28:56 vm-03 systemd-udevd: Failed to apply ACL on 
/dev/snd/pcmC0D0c: No such file or directory
Nov 30 09:28:56 vm-03 systemd-udevd: Failed to apply ACL on 
/dev/snd/pcmC0D0p: No such file or directory
Nov 30 09:28:56 vm-03 systemd-udevd: Failed to apply ACL on 
/dev/dri/card0: No such file or directory


The host's drives have not been made available in the containers, and 
that's intentional. These errors are all being created by the udev 
service of course, and that's the ultimate cause. When I create a 
container under libvirt-lxc though, the udev service is not enabled and 
I therefore do not see these errors. Containers created with LXC using 
the same CentOS image have the udev suite of services enabled, and even 
if I explicitly disable them using


# systemctl disable systemd-udevd-kernel.socket
# systemctl disable systemd-udevd-control.socket
# systemctl disable systemd-udevd.service
# systemctl disable systemd-udev-trigger.service

when I restart the container the services are enabled and I still see 
these errors. My guess is I'm missing something in the config file for 
my LXC containers but I'm not sure what's needed. This appears to be 
further indicated by the set of sys services that are running in my 
libvirt-lxc containers:


# systemctl|grep sys-
sys-fs-fuse-connections.mount  loaded active mounted FUSE Control 
File System
sys-kernel-config.mountloaded active mounted Configuration 
File System

sys-kernel-debug.mount loaded active mounted Debug File System

compared to what I see in my equivalent LXC container:

# systemctl|grep sys-
sys-devices-pci:00-:00:04.0-sound-card0.device loaded active 
plugged   QEMU Virtual Machine
sys-devices-pci:00-:00:05.7-usb1-1\x2d1-1\x2d1:1.0-host8-target8:0:0-8:0:0:0-block-sdc-sdc1.device 
loaded active plugged   QEMU_HARDDISK
sys-devices-pci:00-:00:05.7-usb1-1\x2d1-1\x2d1:1.0-host8-target8:0:0-8:0:0:0-block-sdc-sdc2.device 
loaded active plugged   QEMU_HARDDISK
sys-devices-pci:00-:00:05.7-usb1-1\x2d1-1\x2d1:1.0-host8-target8:0:0-8:0:0:0-block-sdc.device 
loaded active plugged   QEMU_HARDDISK

Re: [lxc-users] Containers have network issues when their host uses a bonded interface

[Peter Steele]

Here's an interesting data point: We noticed that our bond interface was 
reporting a large number of dropped packets, particularly compared to 
when we were using CentOS 6.5. As a test I decided to try out the new 
"team" alternative to bonding that's offered by CentOS 7 and the dropped 
packets virtually disappeared. Unfortunately, the original arp table 
problem I reported in this thread reappeared, even though we're now 
using the mainline 4.2 kernel. Apparently they fixed the bug for bonds 
but the newer team feature is still susceptible.


Peter


On 09/11/2015 12:02 PM, Peter Steele wrote:

On 09/10/2015 11:14 PM, Guido Jäkel wrote:
* Is even LXC not needed to reproduce the issue but just a bridge on 
a bound and some other devices?
I have not been able to reproduce the problem except between 
containers running on different hosts. Behavior is the same for lxc 
and libvirt-lxc.
* Did this issue appear if you use a higher MAC prefix than 64:... 
for the veths?
I did try this, using 54:C0:xx:xx:xx:xx for my mac addresses. It did 
not help.
May you dump the arp traffic (maybe together with icmp) on the host? 
I'm interested in what happens in/around moment the gateway is pinged!


On the bridge, did you set forward delay to 0, a low hello intervall 
(e.g. 1s) and turn on STP?


I tried stp on and off, as well as setting a delay to 0 and sleep to 
1. No luck.


If I pre-populate everyone's arp tables, this solves the problem, but 
that's not a practical solution.




___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Eth0 not present on boot

[Peter Steele]

Up to know, we've only seen it happen on specific servers, running one 
of our test suites. The same test suite on my personal cluster does not 
exhibit this issue, whereas the test engineer that encountered this 
problem sees it regularly. My servers have SAS drives whereas his have 
SATA drives. Other than that the servers are not especially different 
(1U four drive systems). I'm going to have to get some quality time on 
the misbehaving servers and try to get a better understanding of what's 
going on using a simpler test to reproduce the problem.


Peter

On 09/23/2015 01:38 PM, Guido Jäkel wrote:

Dear Peter,

I also rename the veth to the name of the container and without this, as the 
default a random name will be used. You'll see why with the later you don't 
notice an issue:

If a container goes down, the veth did not vanish as long as there is a tcp 
connection using it. And in case of a shutdown scenario, there's a good chance 
to interrupt communication processes in the middle of conversation, e.g. during 
delivery of content. Then, an outerside client will wait for output until an 
application level timeout or because of TCP handshake timeouts. This typically 
will take about 5min.

In my experience, sometimes you even cant "delete the link" as described by Fayar, but i 
was able to rename the interface "away" (ip link set dev $DEV name $DEV._away). It will 
vanish later but the restarting container may use the name again for a new veth.


greetings

Guido

On 23.09.2015 03:24, Peter Steele wrote:

On 09/22/2015 08:08 AM, Guido Jäkel wrote:

* Do you use  lxc.network.veth.pair  to name the hosts side of the veth?

Yes. I rename the veth interfaces to match the names of the containers.

* Was the Container up and running "just before" and you (re)start it within 
less than 5min?


Yes. When the problem occurs, a reboot has just been issued on the container (using the 
"reboot" command). When the container restarts, its eth0 is missing. Another 
reboot and the eth0 interface reappears. Curiously, it appears to happen much more 
frequently on some hardware than it does on others.




___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Eth0 not present on boot

[Peter Steele]

On 09/22/2015 08:08 AM, Guido Jäkel wrote:

* Do you use  lxc.network.veth.pair  to name the hosts side of the veth?

Yes. I rename the veth interfaces to match the names of the containers.

* Was the Container up and running "just before" and you (re)start it within 
less than 5min?

Yes. When the problem occurs, a reboot has just been issued on the 
container (using the "reboot" command). When the container restarts, its 
eth0 is missing. Another reboot and the eth0 interface reappears. 
Curiously, it appears to happen much more frequently on some hardware 
than it does on others.



___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Eth0 not present on boot

[Peter Steele]

On 09/21/2015 03:27 PM, Fajar A. Nugraha wrote:
I remembered something similar a while ago, in ubuntu precise host and 
containers, with both lxc 1.0.x and lxc-1.x from ppa. At that time a 
container's interface would mysteriously dissapear, including it's 
host side veth pair. Only on one container though, others were fine.


Upgrading to lxc-1.x only didn't help. I ended up upgrading the host 
to trusty and the kernel (currently running wily's 4.2). Haven't seen 
the problem recently, but still not sure what caused the issue.


@Peter, are you still running centos7's stock kernel?



No, we're running 4.2 now from elrepo. I saw this issue as well though 
when we were running with the 3.1 CentOS kernel. One curious thing is 
that it seems to only happen on certain servers here, so there may be a 
hardware aspect to the problem.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Eth0 not present on boot

[Peter Steele]

On 09/21/2015 01:20 PM, Peter Steele wrote:

On 09/21/2015 08:32 AM, Serge Hallyn wrote:

In these cases does /sys/class/net/eth0 exist?


I'll try to reproduce the condition and check this...

I just checked this. This directory does not exist. There is only an 
entry for lo.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Eth0 not present on boot

[Peter Steele]

On 09/21/2015 08:32 AM, Serge Hallyn wrote:

In these cases does /sys/class/net/eth0 exist?


I'll try to reproduce the condition and check this...

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] Eth0 not present on boot

[Peter Steele]

We sometimes hit an error where eth0 in a container does not come up, 
leaving the container with only the "lo" device. The system messages in 
the container lists the error


Sep 17 15:58:50 vm-00 network: Bringing up interface eth0:  ERROR : 
[/etc/sysconfig/network-scripts/ifup-eth] Device eth0 does not seem to 
be present, delaying initialization


The only solution we've found is to reboot the container. To date, that 
has always restored the missing interface. The question we have though 
is what causes this in the first place? This is a CentOS 7.1 container 
running on a CentOS 7.1 host. The containers use bridged networking.


Has anyone seen this behavior?

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Containers have network issues when their host uses a bonded interface

[Peter Steele]

On 09/14/2015 11:02 PM, Fajar A. Nugraha wrote:
Assuming your problem is caused by bridging the veth interface, 
there's an alternate networking setup with proxyarp + route that might 
work. It doesn't use bridge, and only works for privileged containers.


I'll investigate how this could be setup with CentOS. However, it's not 
solely related to bridging since bond mode 1 (active/backup) works fine. 
It's only when we use the balance-rr modes that this issue occurs.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Containers have network issues when their host uses a bonded interface

[Peter Steele]

On 09/15/2015 01:29 AM, Andrey Repin wrote:

2b. If you absolutely want to communicate with containers from host via
network, you will need a similarly set up interface on the host.
This is a little complicated without a helper script, but still doable:


We do need to be able to communicate with the containers from the hosts, 
both on the same server as well as across servers. I'll investigate how 
this can be setup in CentOS.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Containers have network issues when their host uses a bonded interface

[Peter Steele]

On 09/13/2015 06:19 AM, Fajar A. Nugraha wrote:

Had you use Ubuntu you could probably say something like "kernel 4.2
should be released as linux-generic-lts-wily for ubuntu 14.04 in about
two months, and we can switch to distro-supported package then"

Had you use Oracle Linux with support subscription, you could probably
nag them about when they would relase unbreakable enterprise kernel
release 4, which should be newer kernel, with some features added by
oracle, tested and supported to run on oracle linux (which is mostly a
clone of RHEL, with some additions)

Had you use RHEL with support subscription, you could probably ask
Redhat to backport kernel fix for this particular issue to RHEL 7's
kernel.

Since you use Centos, your best bet would probably:
- Hope someone else nags Redhat, so the fix would eventually trickle
down to centos. OR
- Use 4.2 kernel, with its consequences (e.g. not tested and supported
by redhat). OR
- Try kernel 4.1. That version should be an LTS kernel, and elrepo
would probably release kernel-lt 4.1 for rhel/centos7 (like they did
with rhel/centos 5 and 6) sometime in the future. Being longterm
kernel, you should be able to stick with kernel 4.1.x for a few years,
without having to upgrade to mainline/stable kernel.

The decision to go with CentOS was made some time ago. We did a network 
throughput comparison between the Ubuntu and CentOS releases available 
at that time and CentOS performed better. Since then CentOS has been 
very solid for us, and this is the first real issue we've faced. If we 
had gone with RHEL of course, we'd be hitting the same problem right 
now, and of course we'd be reporting this bug. Alas, getting a fix from 
Redhat in a timely manner, even with a support contract, would not 
necessarily be an expedient process.


We will have to do some thorough testing with the 4.2 (or possibly 4.1) 
kernel over the next few weeks to make sure this kernel doesn't 
introduce new issues. Our only other option would be to fall back to KVM 
instead of containers and that's not something we really want to do.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Containers have network issues when their host uses a bonded interface

[Peter Steele]

On 09/10/2015 11:14 PM, Guido Jäkel wrote:

* Is even LXC not needed to reproduce the issue but just a bridge on a bound 
and some other devices?
I have not been able to reproduce the problem except between containers 
running on different hosts. Behavior is the same for lxc and libvirt-lxc.

* Did this issue appear if you use a higher MAC prefix than 64:... for the 
veths?
I did try this, using 54:C0:xx:xx:xx:xx for my mac addresses. It did not 
help.

May you dump the arp traffic (maybe together with icmp) on the host? I'm 
interested in what happens in/around moment the gateway is pinged!

On the bridge, did you set forward delay to 0, a low hello intervall (e.g. 1s) 
and turn on STP?

I tried stp on and off, as well as setting a delay to 0 and sleep to 1. 
No luck.


If I pre-populate everyone's arp tables, this solves the problem, but 
that's not a practical solution.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Containers have network issues when their host uses a bonded interface

[Peter Steele]

On 09/11/2015 12:08 PM, Andrey Repin wrote:
So, have you tried getting rid of the bridge in first place? 


The problem isn't the bridge per se, it's the bond mode. If I use 
active-backup the veth->bridge->bond path from container to container 
works as expected. Bond modes using load balancing on the bond though 
cause problems.


Would removing the bridge from this scenario solve the problem. How can 
a container be configured without bridging and be able to get to a 
container on another host?



___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Containers have network issues when their host uses a bonded interface

[Peter Steele]

On 09/10/2015 07:57 PM, Fajar A. Nugraha wrote:
If I read the bug report correctly, it's not moved to lxc. Rather, KVM 
is not required to reproduce it, using lxc is enough to trigger the 
bug. Using KVM will of course still trigger the bug as well. 
Sorry, I didn't mean the bug was moved to lxc, I just meant the 
discussion moved to lxc. Since we've been using KVM for years though and 
have never seen this issue, my guess is that it may have been fixed for KVM.

Probably because not many people use balance-rr. Mission-critical
setups would use 802.3ad (with suitable switches), while beginners
would mostly stick with active-backup.
We use balance-alb (mode 5) of course so we can get more bandwidth in 
our transmit pipe. We install our software in generic environments and 
cannot rely on the customer having special switches. This means mode 5 
is the best option for us, although we support mode 6 (balance-alb) 
installs as well.

Since the original bug report is quite old, and you're not using
ubuntu, I suggest you:
- try latest kernel, to make sure the bug still exists there (e.g.
using 4.2 from http://elrepo.org/tiki/kernel-ml)
- file a bug report to centos and/or linux kernel maintainers
I tried the 4.2 kernel and the problem appears to be fixed in this 
kernel for my LXC tests--the arp tables update properly with this 
kernel. CentOS 7.1 is running kernel 3.10. Not sure what the QA folks 
here will think about using the 4.2 kernel to fix this issue...


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Containers have network issues when their host uses a bonded interface

[Peter Steele]

I believe this link describes the exact problem I've been experiencing:

https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/785668

and although the original post targets KVM, later in the thread it moves 
to LXC. This is an old bug report and I'm surprised that this has not 
been addressed in recent kernels.


Peter

On 09/10/2015 02:52 PM, Peter Steele wrote:
I've configured a standard CentOS bridge/bond, the exact same setup 
that I use for creating VMs. VMs on different hosts communicate 
through the bridge without issues. Containers that use the identical 
bridge however cannot reliably connect to containers on different 
hosts. We've determined that it is some kind of ARP table issue, or at 
least appears to be. I have reproduced a scenario where container A on 
host X is unable to ping container B on host Y. If I connect to B from 
host Y and ping say the gateway of our subnet, suddenly container A on 
host X can ping container B. Here's a specific case. Container A on 
host X has the IP 172.16.110.106. Its arp table looks like this:


# arp -v -n
Address  HWtype  HWaddress   Flags 
MaskIface
172.16.110.112   ether   00:16:3e:21:c6:c5 
C eth0
172.16.110.111   ether   00:16:3e:31:2a:f0 
C eth0
172.16.110.110   ether   00:16:3e:5c:62:a9 
C eth0

Entries: 4  Skipped: 0  Found: 4

Container B on host Y has the IP 172.16.110.113. Its arp table looks 
like this:


# arp -v -n
Address  HWtype  HWaddress   Flags 
MaskIface
172.16.110.112   ether   00:16:3e:21:c6:c5 
C eth0
172.16.110.106   ether   00:16:3e:20:df:87 
C eth0

Entries: 2  Skipped: 0  Found: 2

If I try to ping 172.16.110.113 (container B) from 172.16.110.106 
(container A), I get a "Host Unreachable":


# ping 172.16.110.113
PING 172.16.110.113 (172.16.110.113) 56(84) bytes of data.
From 172.16.110.106 icmp_seq=1 Destination Host Unreachable
From 172.16.110.106 icmp_seq=2 Destination Host Unreachable
From 172.16.110.106 icmp_seq=3 Destination Host Unreachable
From 172.16.110.106 icmp_seq=4 Destination Host Unreachable
From 172.16.110.106 icmp_seq=5 Destination Host Unreachable
...

If while this ping is running I connect to container B and ping the 
gateway (172.16.0.1), the ping running on container A suddenly starts 
working:


...
From 172.16.110.106 icmp_seq=6 Destination Host Unreachable
From 172.16.110.106 icmp_seq=7 Destination Host Unreachable
From 172.16.110.106 icmp_seq=8 Destination Host Unreachable
64 bytes from 172.16.110.113: icmp_seq=57 ttl=64 time=993 ms
64 bytes from 172.16.110.113: icmp_seq=58 ttl=64 time=0.283 ms
64 bytes from 172.16.110.113: icmp_seq=59 ttl=64 time=0.274 ms
...

The arp table on container A now has an entry for container B:

# arp -v -n
Address  HWtype  HWaddress   Flags 
MaskIface
172.16.110.112   ether   00:16:3e:21:c6:c5 
C eth0
172.16.110.111   ether   00:16:3e:31:2a:f0 
C eth0
172.16.110.113   ether   00:16:3e:65:2a:c5 
C eth0
172.16.110.110   ether   00:16:3e:5c:62:a9 
C eth0

Entries: 5  Skipped: 0  Found: 5

The arp table on container B of course now has an entry for the gateway:

# arp -v -n
Address  HWtype  HWaddress   Flags 
MaskIface
172.16.110.112   ether   00:16:3e:21:c6:c5 
C eth0
172.16.110.106   ether   00:16:3e:20:df:87 
C eth0
172.16.0.1   ether   64:12:25:e3:d4:4c 
C eth0

Entries: 3  Skipped: 0  Found: 3

We've been running VMs (KVM) with this identical bridged/bonded 
configuration for years and have never had an issue with a VM on one 
host being unable to communicate with a VM on another host.We've never 
had to force an arp table update of any kind. Why do containers behave 
in this manner?


Peter

On 09/10/2015 01:01 PM, Bostjan Skufca wrote:

Hi Peter,

since you mentioned you are using bridged interfaces, is my assumption
correct that your containers's network connection is joined directly
to this bridge and containers talk to the world direcly (L2) and not
via routed (L3) network over host OS?

Did you try using routed setup (using bond0 directly and creating
dedicated br1just  for containers) and taking bridging functionality
of the linux kernel out of the picture?

I am very interested in your findings, as I have similar setup planned
for deployment in the coming weeks (vlan trunk -> bond -> bridge ->
containers).

(I had similar issues on bonded interfaces on VMware, where tap-based
OpenVPN server would not work at all. It had to do something with how
vmware handles bonds and this not being compatible with L2 stuff
coming out of VM. The seco

Re: [lxc-users] Containers have network issues when their host uses a bonded interface

[Peter Steele]

I've configured a standard CentOS bridge/bond, the exact same setup that 
I use for creating VMs. VMs on different hosts communicate through the 
bridge without issues. Containers that use the identical bridge however 
cannot reliably connect to containers on different hosts. We've 
determined that it is some kind of ARP table issue, or at least appears 
to be. I have reproduced a scenario where container A on host X is 
unable to ping container B on host Y. If I connect to B from host Y and 
ping say the gateway of our subnet, suddenly container A on host X can 
ping container B. Here's a specific case. Container A on host X has the 
IP 172.16.110.106. Its arp table looks like this:


# arp -v -n
Address  HWtype  HWaddress   Flags 
MaskIface
172.16.110.112   ether   00:16:3e:21:c6:c5 C 
eth0
172.16.110.111   ether   00:16:3e:31:2a:f0 C 
eth0
172.16.110.110   ether   00:16:3e:5c:62:a9 C 
eth0

Entries: 4  Skipped: 0  Found: 4

Container B on host Y has the IP 172.16.110.113. Its arp table looks 
like this:


# arp -v -n
Address  HWtype  HWaddress   Flags 
MaskIface
172.16.110.112   ether   00:16:3e:21:c6:c5 C 
eth0
172.16.110.106   ether   00:16:3e:20:df:87 C 
eth0

Entries: 2  Skipped: 0  Found: 2

If I try to ping 172.16.110.113 (container B) from 172.16.110.106 
(container A), I get a "Host Unreachable":


# ping 172.16.110.113
PING 172.16.110.113 (172.16.110.113) 56(84) bytes of data.
From 172.16.110.106 icmp_seq=1 Destination Host Unreachable
From 172.16.110.106 icmp_seq=2 Destination Host Unreachable
From 172.16.110.106 icmp_seq=3 Destination Host Unreachable
From 172.16.110.106 icmp_seq=4 Destination Host Unreachable
From 172.16.110.106 icmp_seq=5 Destination Host Unreachable
...

If while this ping is running I connect to container B and ping the 
gateway (172.16.0.1), the ping running on container A suddenly starts 
working:


...
From 172.16.110.106 icmp_seq=6 Destination Host Unreachable
From 172.16.110.106 icmp_seq=7 Destination Host Unreachable
From 172.16.110.106 icmp_seq=8 Destination Host Unreachable
64 bytes from 172.16.110.113: icmp_seq=57 ttl=64 time=993 ms
64 bytes from 172.16.110.113: icmp_seq=58 ttl=64 time=0.283 ms
64 bytes from 172.16.110.113: icmp_seq=59 ttl=64 time=0.274 ms
...

The arp table on container A now has an entry for container B:

# arp -v -n
Address  HWtype  HWaddress   Flags 
MaskIface
172.16.110.112   ether   00:16:3e:21:c6:c5 C 
eth0
172.16.110.111   ether   00:16:3e:31:2a:f0 C 
eth0
172.16.110.113   ether   00:16:3e:65:2a:c5 C 
eth0
172.16.110.110   ether   00:16:3e:5c:62:a9 C 
eth0

Entries: 5  Skipped: 0  Found: 5

The arp table on container B of course now has an entry for the gateway:

# arp -v -n
Address  HWtype  HWaddress   Flags 
MaskIface
172.16.110.112   ether   00:16:3e:21:c6:c5 C 
eth0
172.16.110.106   ether   00:16:3e:20:df:87 C 
eth0
172.16.0.1   ether   64:12:25:e3:d4:4c C 
eth0

Entries: 3  Skipped: 0  Found: 3

We've been running VMs (KVM) with this identical bridged/bonded 
configuration for years and have never had an issue with a VM on one 
host being unable to communicate with a VM on another host.We've never 
had to force an arp table update of any kind. Why do containers behave 
in this manner?


Peter

On 09/10/2015 01:01 PM, Bostjan Skufca wrote:

Hi Peter,

since you mentioned you are using bridged interfaces, is my assumption
correct that your containers's network connection is joined directly
to this bridge and containers talk to the world direcly (L2) and not
via routed (L3) network over host OS?

Did you try using routed setup (using bond0 directly and creating
dedicated br1just  for containers) and taking bridging functionality
of the linux kernel out of the picture?

I am very interested in your findings, as I have similar setup planned
for deployment in the coming weeks (vlan trunk -> bond -> bridge ->
containers).

(I had similar issues on bonded interfaces on VMware, where tap-based
OpenVPN server would not work at all. It had to do something with how
vmware handles bonds and this not being compatible with L2 stuff
coming out of VM. The second thing I remember very vaguely is that I
tried using bridged interface inside container too, and it did not
cooperate well with host's bridge, so I stopped using bridges inside
containers altogether.)

b.




___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Containers have network issues when their host uses a bonded interface

[Peter Steele]

I don't think I'm up to the challenge of an ASCII art representation of 
our cluster. How's this instead?


We run on a cluster of 1U and 2U servers. Each server is connected to 
the same network by either multiple 1GigE or 10GigE links and is 
configured with a bonded/bridged interface. Every server runs some 
number of containers, based on resources available and the configuration 
parameters we select when installing the cluster. Almost all network 
traffic is between the various containers, both within the same host as 
well as between containers running on different hosts. There is minimal 
communication in our software between containers and the servers 
themselves. We have some protocols based on UDP and a few based on TCP. 
Pretty standard stuff ultimately.


We run this identical configuration with VMs (KVM) without issues. We're 
moving to containers since our benchmarks indicate we'd gain some in 
performance.


On 09/07/2015 10:35 PM, Guido Jäkel wrote:

Dear Peter,

may you paint a small ASCII art of your desired network setup and the network 
configuration? I see no reason that "LXC", i.e. the veth connecting two 
differently namespaced IP stacks is responsible for any problems.

BTW: I'm using a complex network setup with no problems, too. A real NIC is 
connected to a selected trunk of VLANs; they are decoded and connected to a 
couple of bridges and the containers are attached to one or more of it as 
needed for the business case.

greetings

Guido


On 07.09.2015 20:49, Peter Steele wrote:

We're having issues with networking connections in our containers when the host 
is configured with bonded interfaces. When we configure these same servers to 
run with VMs, everything works fine, but when we swap out the VMs for 
equivalently configured containers, we get all kinds of network connection 
failures between containers. The network issues go away when we reduce the bond 
to have a single member. More specifically, we configure our server with a host 
bridge (br0), where the host bridge links to the host's bonded interface 
(bond0). We've been using this model for years with VMs. Something isn't 
working quite right with containers though.

Is there something we need to set in our containers' config files when the host 
is setup with bonding?

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Containers have network issues when their host uses a bonded interface

[Peter Steele]

On 09/07/2015 11:49 AM, Peter Steele wrote:
We're having issues with networking connections in our containers when 
the host is configured with bonded interfaces. When we configure these 
same servers to run with VMs, everything works fine, but when we swap 
out the VMs for equivalently configured containers, we get all kinds 
of network connection failures between containers. The network issues 
go away when we reduce the bond to have a single member. More 
specifically, we configure our server with a host bridge (br0), where 
the host bridge links to the host's bonded interface (bond0). We've 
been using this model for years with VMs. Something isn't working 
quite right with containers though.


Is there something we need to set in our containers' config files when 
the host is setup with bonding?



Incidentally, our hosts and containers are all running CentOS 7.1.

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] Containers have network issues when their host uses a bonded interface

[Peter Steele]

We're having issues with networking connections in our containers when 
the host is configured with bonded interfaces. When we configure these 
same servers to run with VMs, everything works fine, but when we swap 
out the VMs for equivalently configured containers, we get all kinds of 
network connection failures between containers. The network issues go 
away when we reduce the bond to have a single member. More specifically, 
we configure our server with a host bridge (br0), where the host bridge 
links to the host's bonded interface (bond0). We've been using this 
model for years with VMs. Something isn't working quite right with 
containers though.


Is there something we need to set in our containers' config files when 
the host is setup with bonding?


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Enabling sys_nice in a privileged container

[Peter Steele]

On 09/05/2015 10:35 AM, Peter Steele wrote:
I have a privileged container that runs ctdb and needs to have real 
time scheduling enabled. The error reported by ctdb is:


Sep 05 10:27:05 pws-01-vm-05 systemd[1]: Starting CTDB...
Sep 05 10:27:06 pws-01-vm-05 ctdbd[1598]: CTDB starting on node
Sep 05 10:27:06 pws-01-vm-05 ctdbd[1599]: Starting CTDBD (Version 
2.5.4) as PID: 1599
Sep 05 10:27:06 pws-01-vm-05 ctdbd[1599]: Created PID file 
/run/ctdb/ctdbd.pid
Sep 05 10:27:06 pws-01-vm-05 ctdbd[1599]: Unable to set scheduler to 
SCHED_FIFO (Operation not permitted)

Sep 05 10:27:06 pws-01-vm-05 ctdbd[1599]: CTDB daemon shutting down

Apparently. my container is dropping the sys_nice capability which is 
needed for real time scheduling. I thought I could just add the line


lxc.cap.keep = sys_nice

but this has the side effect of dropping all capabilities except this 
one so that just made things worse. What is the correct way to enable 
a specific capability for a container?


I'm running CentOS 7 and am using a custom template. By config is 
pretty basic with just the following parameters defined:


lxc.tty = 4
lxc.pts = 1024
lxc.utsname = test
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.veth.pair = veth-test
lxc.network.hwaddr = 00:16:3e:16:ef:32
lxc.rootfs = /lxc/test

Peter



For what it's worth, I hit this very same problem when we were using 
libvirt-lxc containers. In that case though the containers did have 
sys_nice enabled but were still unable to schedule real time jobs 
because by default libvirt doesn't give any real time bandwidth to 
containers. This is defined by the cgroup parameter


/sys/fs/cgroup/cpu/cpu.rt_runtime_us

This is zero by default for libvirt containers, and setting this solved 
the CTDB issue. LXC containers *do* set this parameter though, so the 
error in this case is because sys_nice has been dropped for the 
containers. Or at least that's my assumption. The underlying call being 
made is pthread_create, and our own real-time threads are failing as 
well with the same "operation not permitted" error. So the containers 
are clearly lacking the required permissions.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Determining a container's MAC address

[Peter Steele]

On 09/06/2015 09:52 AM, Guido Jäkel wrote:

Dear Peter,

don't use a MAC prefix that is lower than that of the upstream device of the 
bridge the containers are attached: The Linux software bridge will use the 
lowest MAC of it's attached devices as the MAC of the outgoing packets. 
Therefore, you will risk short traffic interruptions to others, if you shutdown 
a Container and this one was the one with the lowest MAC on it's bridge. 
Because then, all network affected components outside have to learn a new MAC 
to route the packets for the remaining Containers to.

I'm currently using a "quite high" prefix and a tail that is derived from the assigned 
IP. The same "formula" is used to setup a auxiliary DHCP server because use DHCP for the 
container network setup.

HWADDR=`IP=${IP#*.}; printf "00:50:C2:%02X:%02X:%02X" ${IP//./ }`# 
a.b.c.d -> 00:50:C2:bb:cc:dd (hex)
I was not aware of this behavior. Although I don't believe we've noticed 
any suspicious behavior in our network traffic, I can easily change our 
automation logic to use a high prefix for the containers that we create.


Thanks for the tip.

Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Determining a container's MAC address

[Peter Steele]

On 09/06/2015 09:52 AM, Guido Jäkel wrote:

Dear Peter,

don't use a MAC prefix that is lower than that of the upstream device of the 
bridge the containers are attached: The Linux software bridge will use the 
lowest MAC of it's attached devices as the MAC of the outgoing packets. 
Therefore, you will risk short traffic interruptions to others, if you shutdown 
a Container and this one was the one with the lowest MAC on it's bridge. 
Because then, all network affected components outside have to learn a new MAC 
to route the packets for the remaining Containers to.

I'm currently using a "quite high" prefix and a tail that is derived from the assigned 
IP. The same "formula" is used to setup a auxiliary DHCP server because use DHCP for the 
container network setup.

HWADDR=`IP=${IP#*.}; printf "00:50:C2:%02X:%02X:%02X" ${IP//./ }`# 
a.b.c.d -> 00:50:C2:bb:cc:dd (hex)
I was not aware of this behavior. Although I don't believe we've noticed 
any suspicious behavior in our network traffic, I can easily change our 
automation logic to use a high prefix for the containers that we create.


Thanks for the tip.

Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Determining a container's MAC address

[Peter Steele]

On 09/06/2015 09:52 AM, Guido Jäkel wrote:

Dear Peter,

don't use a MAC prefix that is lower than that of the upstream device of the 
bridge the containers are attached: The Linux software bridge will use the 
lowest MAC of it's attached devices as the MAC of the outgoing packets. 
Therefore, you will risk short traffic interruptions to others, if you shutdown 
a Container and this one was the one with the lowest MAC on it's bridge. 
Because then, all network affected components outside have to learn a new MAC 
to route the packets for the remaining Containers to.

I'm currently using a "quite high" prefix and a tail that is derived from the assigned 
IP. The same "formula" is used to setup a auxiliary DHCP server because use DHCP for the 
container network setup.

HWADDR=`IP=${IP#*.}; printf "00:50:C2:%02X:%02X:%02X" ${IP//./ }`# 
a.b.c.d -> 00:50:C2:bb:cc:dd (hex)
I was not aware of this behavior. Although I don't believe we've noticed 
any suspicious behavior in our network traffic, I can easily change our 
automation logic to use a high prefix for the containers that we create.


Thanks for the tip.

Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] Enabling sys_nice in a privileged container

[Peter Steele]

I have a privileged container that runs ctdb and needs to have real time 
scheduling enabled. The error reported by ctdb is:


Sep 05 10:27:05 pws-01-vm-05 systemd[1]: Starting CTDB...
Sep 05 10:27:06 pws-01-vm-05 ctdbd[1598]: CTDB starting on node
Sep 05 10:27:06 pws-01-vm-05 ctdbd[1599]: Starting CTDBD (Version 2.5.4) 
as PID: 1599
Sep 05 10:27:06 pws-01-vm-05 ctdbd[1599]: Created PID file 
/run/ctdb/ctdbd.pid
Sep 05 10:27:06 pws-01-vm-05 ctdbd[1599]: Unable to set scheduler to 
SCHED_FIFO (Operation not permitted)

Sep 05 10:27:06 pws-01-vm-05 ctdbd[1599]: CTDB daemon shutting down

Apparently. my container is dropping the sys_nice capability which is 
needed for real time scheduling. I thought I could just add the line


lxc.cap.keep = sys_nice

but this has the side effect of dropping all capabilities except this 
one so that just made things worse. What is the correct way to enable a 
specific capability for a container?


I'm running CentOS 7 and am using a custom template. By config is pretty 
basic with just the following parameters defined:


lxc.tty = 4
lxc.pts = 1024
lxc.utsname = test
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.veth.pair = veth-test
lxc.network.hwaddr = 00:16:3e:16:ef:32
lxc.rootfs = /lxc/test

Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] Limiting number of cores in a container

[Peter Steele]

Our application needs to limit the number of cores a container can use.  
With libvirt-lxc I use the command "virsh setvpus" to set the number of 
cores a container can use. With this command you only have to specify 
the number of cores assigned to the container, not a specific core 
number. I can't seem to find an equivalent for this with LXC. I've found 
the parameter lxc.cgroup.cpuset.cpus that can be set to bind a container 
to use a specific CPU (core?), as well as the parameter 
lxc.cgroup.cpu.shares that can be used to designate a number of CPU 
"shares" to be assigned to a container, but I'm not exactly sure how 
this works, especially in the case of over-committing CPU resources.


Let's assume we have a system with 16 cores that will be hosting seven 
containers. Six of these will be limited to two cores each and one will 
be assigned four cores. With libvirt-lxc I can simply assign the desired 
CPU count to each container and let the system would decide how the CPUs 
are scheduled. In fact, if I had a less powerful server, say with 8 
cores instead of 16, libvirt would let me over-commit the CPUs assigned 
to my containers, in exactly the same way one can over-commit CPUs to 
VMs. This is very useful in our test environment where engineers may not 
all have high end systems. The CentOS lscpu command accurately reflects 
the virtual CPU count of the container, despite how many physical CPUs 
are actually present on the host:


# lscpu
Architecture:  x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:Little Endian
CPU(s):4
On-line CPU(s) list:   0-3
Thread(s) per core:1
Core(s) per socket:1
Socket(s): 4
NUMA node(s):  1

This shows a container with four virtual CPUs.

We have an automation system that creates and manages our containers. 
Due to its pending demise, we're migrating from libvirt-lxc to "stock" 
LXC and I'm trying to map the various operations used in creating 
libvirt containers to equivalent operations for LXC containers. It's not 
entirely clear to me how to deal with this CPU count issue. Can anyone 
give me some insight on how to setup something at least approximating 
what we're doing with libvirt-lxc?


Thanks.

Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Determining a container's MAC address

[Peter Steele]

On 09/01/2015 02:06 AM, Andrey Repin wrote:

Greetings, Peter Steele!


lxc.network.hwaddr = 00:xx:xx:xx:xx:xx

Do NOT do this.
If you want completely random private MAC's, start with 02:...

Ref: http://www.iana.org/assignments/ethernet-numbers/ethernet-numbers.xhtml


in my default.conf as well and that generated a random mac address
starting with b6:38, then 78:47, and so on. Is there a recommended
prefix for the auto-generated lxc mac addresses?

The recommended prefix is the one that is set by default.
You MAY use private prefix, but make sure it is properly indicated.

I know I don't want arbitrarily random mac addresses, I was asking what 
is the recommended lxc prefix? A previous poster suggested


00:16:3e:xx:xx:xx

I decided to try

00:xx:xx:xx:xx:xx

just to see if lxc would pick its own hard coded prefix but it seemed to 
generate something different each time. You recommended that I should use the 
one that is set by default, but what is that? My default.conf had no entry 
lxc.network.hwaddr until I put one there.

Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Adding a host block device to a container

[Peter Steele]

On 09/01/2015 09:15 AM, Serge Hallyn wrote:

In that case that's exactly what templates were meant to do.  So while
I'd still like to see lxc-device updated to support persistence, you
could do what you want by

1. creating a lxc.hook.autodev hook which creates the device you want
using mknod,
2. adding lxc.hook.autodev = /usr/local/bin/my-dev-hook to the container
configuration
3. adding the lxc.cgroup rule you need to the container configuration

You could add the two config additions to your global
/etc/lxc/default.conf, add them to the config you pass to
lxc-create -f, or set them in a new template you create and use
instead of /bin/true.

Using /bin/true was the quickest way to get off to the races, using a 
template I had already been using for libvirt-lxc. It definitely makes 
sense though to take it to the next step and turn my template into a 
proper lxc template. I'll get there eventually.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Determining a container's MAC address

[Peter Steele]

On 08/31/2015 06:32 AM, Serge Hallyn wrote:


Should show up in a line like

lxc.network.hwaddr = 00:16:3e:83:d1:8f

in the created container config.
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users


This is what I'd use to create a container with my own custom mac 
address, but this field does not appear to get populated when a 
container is created. I don't need my own custom address-- I'm happy to 
let LXC pick its own mac address. I'd just like to be able to determine 
what mac it has assigned a container without having to boot that 
container first.



___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] Adding a host block device to a container

[Peter Steele]

This is likely a newbie question but here goes...

I have some privileged containers that need access to certain block 
devices available on their host. For example, I'd like /dev/sda3 to be 
accessible from my container test. The major/minor values for this 
device is:


# ll /dev/sda3
brw-rw 1 root disk 8, 3 Aug 31 12:49 /dev/sda3

According to the documentation to import this device into a container I 
need to define the the parameter lxc.cgroup.devices.allow, so I added 
the following line to /var/lib/lxc/test/config:


lxc.cgroup.devices.allow = b 8:3 rwm

and restarted my container. This didn't seem to do the trick through. 
Running this command on the host however did do the trick:


# mknod -m 777 /var/lib/lxc/test/rootfs.dev/sda3 b 8 3

I assume the parameter lxc.cgroup.devices.allow is intended to do 
something like this in an automated fashion, and I'd rather use this 
parameter than running mknod explicitly. What am I missing?



___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Determining a container's MAC address

[Peter Steele]

On 08/31/2015 10:03 PM, Serge Hallyn wrote:

Right - if you use lxc-create to create the config file, and your
initial lxc.conf (i.e. /etc/lxc/default.conf or whatever you pass
as CONF to lxc-create -f CONF) contains something like

lxc.network.hwaddr = 00:16:3e:xx:xx:xx

then your container config will have an expanded hwaddr entry.
That did the trick. Adding this entry to my default.conf resulted in a 
lxc.network.hwaddr entry to be added to my container's config file. I 
also tried using


lxc.network.hwaddr = 00:xx:xx:xx:xx:xx

in my default.conf as well and that generated a random mac address 
starting with b6:38, then 78:47, and so on. Is there a recommended 
prefix for the auto-generated lxc mac addresses?


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Adding a host block device to a container

[Peter Steele]

On 08/31/2015 08:41 PM, Fajar A. Nugraha wrote:
Nope. lxc.cgroup allows you to create and access the block device, but 
you still need to create the device node yourself.

Fair enough. Then I guess I'll use mknod...

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Configuring LXC containers to use a host bridge under CentOS 7

[Peter Steele]

On 08/30/2015 11:10 AM, Peter Steele wrote:
Clearly there is user error here on my part and I am not correctly 
specifying how to configure LXC containers to use host bridging under 
CentOS. I'll have to do some more digging.




I figured it out. I've been using commands similar to

lxc-create -t /bin/true -n test1 --dir=/lxc/test1/rootfs

I mistakenly assumed that by specifying the directory for the rootfs to 
be /lxc/test1/rootfs the config file for the container would be 
/lxc/test1/config. Stupid mistake. LXC still puts the config file in the 
default location /var/lib/lxc/test1/config, and this file was where 
virbr0 was being specified. My own config file was correct such as it 
was, except that it was being ignored. I changed the config file 
/var/lib/lxc/test1/config to use br0 instead of virbr0 and that solved 
my problem. My container can now see the full subnet.


A stupid beginner's mistake. What a waste of time... :-(

Thanks for all the feedback.

Peter



___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] Setting lxc.network.veth.pair

[Peter Steele]

I want to pick my own naming convention for the veth interfaces created 
by LXC rather than using the auto-generated names. I tried adding the entry


lxc.network.veth.pair = veth0

in one of my privileged containers but it still get a random name of 
'veth6HIE0A' instead of eth0. Am I mistaken as to how this is supposed 
to work. Ideally I would like to see something like 
'veth0-container-name' but 'vethN' when N either is assigned 
automatically when containers are started or alternatively where I set N 
myself in the container's config. What's possible and what's not?


Thanks.

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Configuring LXC containers to use a host bridge under CentOS 7

[Peter Steele]

On 08/29/2015 03:26 PM, Fajar A. Nugraha wrote:


It should be tcpdump -n -i bond0 host 172.16.0.1 and tcpdump -n -i 
veth5BJDXU host 172.16.0.1




Okay, I ran this test, plus a few others. This specific test generated 
no icmp traffic on either bond0 or the veth interface. After starting 
these tcpdump commands, I connected to the container and ran a ping to 
172.16.0.1. I got a host unreachable error so I'm not surprised 
nothing showed up in the tcpdump commands. I did the identical test with 
a libvirt container and got the expected icpm request and reply records:


10:44:05.379736 IP 172.16.0.1  172.16.110.204: ICMP echo reply, id 
2656, seq 3, length 64
10:44:06.390229 IP 172.16.110.204  172.16.0.1: ICMP echo request, id 
2656, seq 4, length 64
10:44:06.390689 IP 172.16.0.1  172.16.110.204: ICMP echo reply, id 
2656, seq 4, length 64
10:44:07.400236 IP 172.16.110.204  172.16.0.1: ICMP echo request, id 
2656, seq 5, length 64


It's pretty clear the LXC containers are not talking to the bridge. Once 
started, I can't even ping a container's IP address from the host, and 
likewise the container cannot ping its host. LXC containers can only 
ping each other, behaving exactly like I'd expect NAT to behave. The 
config I am using must not be correct. I'm using this config:


lxc.utsname = test1
lxc.network.type = veth
lxc.network.name = eth0
lxc.network.link = br0
lxc.network.flags = up

You'd think this would tell the container to link to the br0 bridge, but 
this isn't doing what I intend. The brctl command shows what's really 
going on:


# brctl show
bridge name bridge id   STP enabled interfaces
br0 8000.52540007b444   no bond0
vnet0
vnet1
virbr0  8000.525400d0df7b   yes veth5BJDXU
vethU3VLKX
virbr0-nic

The two vnet entries associated with the br0 bridge interface are the 
ones that get created when I start my libvirt-LXC containers. The two 
veth entries associated with virbr0 sre created when I start my LXC 
containers. The virbr0 bridge is created by libvirt to support 
containers (and VMs) that are configured to use NAT addressing. We've 
always used host bridging and so have never used this virbr0 interface. 
For whatever reason, the LXC containers want to link to virbr0 despite 
the fact br0 is specified in their config.


Clearly there is user error here on my part and I am not correctly 
specifying how to configure LXC containers to use host bridging under 
CentOS. I'll have to do some more digging.


Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Setting lxc.network.veth.pair

[Peter Steele]

On 08/30/2015 03:10 PM, Michael H. Warfield wrote:
I played with this a while back and found that you are severely 
limited for the name length. Using the container name for that is, 
sooner or later, going to overflow that limit and possibly generated 
an error on container startup. I think it's a kernel limit and not 
configurable or negotiable, but that was a couple of years ago and may 
have changed. YMMV
Our container names are fixed length of the form vm-NN, so I don't think 
we'd run into a length issue. In fact, what I'd like to use would be 
shorter than what is generated by lxc. I cleared out a setting in 
default.conf and then my custom setting for veth.pair worked as I 
wanted. I decided to go with vm-NN to match the container name.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Setting lxc.network.veth.pair

[Peter Steele]

On 08/30/2015 04:37 PM, Andrey Repin wrote:

Please start from the beginning.
What network topology you want for your containers, and what specific features
you need from VETH interface, that other networking modes do not offer?


I'm using host bridging with lxc.network.type = veth. Everything is 
working fine. I would just prefer a cleaner naming convention for the 
auto-generated veth interfaces, more in keeping with what one might 
expect for typical Linux interface names.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Configuring LXC containers to use a host bridge under CentOS 7

[Peter Steele]

On 08/29/2015 01:09 AM, Neil Greenwood wrote:

Hi Peter,

On 28 August 2015 23:11:51 BST, Peter Steele pwste...@gmail.com wrote:
Do you have a ifcfg-br0 in your LXC configuration? If the VMs can see each 
other, I think most of the settings are correct apart from the bridge not being 
connected to the host's eth0. I'm not that familiar with Centos networking 
though, so I don't know which bit you need to change.


Neil



Yes, I have a ifcfg-br0 file in my LXC configuration. It's identical to 
what I use with my libvirt-LXC setup. In my particular case, I have the 
bridge connected to a bonded interface, but it works the same as if I 
was to connect it to a hosts eth0. This lets me bond say four nics 
eth0-3 into one bond0 interface. The is connected to the bond0 interface 
but from its perspective it acts the same as an eth0 interface. I need 
to of course maintain this configuration in moving from libvirt-LXC to LXC.


Ultimately, in the long term, I will want to remove the entire set of 
libvirt packages from my setup and only install the LXC rpms. I know 
that LXC was developed by Canonical but my impression was that it would 
work under CentOS/Fedora as well. Unfortunately I'm not that familiar 
with Ubuntu networking and a lot of the examples appear to be specific 
to Ubuntu.


For example, I see references to the file /etc/network/interfaces. Is 
this an LXC thing or is this a standard file in Ubuntu networking?


Mark Constable asked a related question stemming from my original post 
and commented on the file /etc/default/lxc-net. I assume this file is 
*not* specific to Ubuntu. Do I need to create this file in my CentOS setup?


It might be useful to configure a CentOS system without libvirt 
installed and do some LXC tests without libvirt-LXC getting in the way. 
I was hoping making the move to LXC wouldn't be too painful but it looks 
like I'm going to have to work for it a bit.


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Configuring LXC containers to use a host bridge under CentOS 7

[Peter Steele]

On 08/29/2015 07:29 AM, Mark Constable wrote:

On 29/08/15 23:54, Peter Steele wrote:
For example, I see references to the file /etc/network/interfaces. Is 
this an

 LXC thing or is this a standard file in Ubuntu networking?


It's a standard pre-systemd debian/ubuntu network config file.


That's what I was beginning to suspect since creating this in my CentOS 
environment seemed to have no effect on LXC at all. Knowing this will 
help me filter out examples that talk about creating these files.


Do you suppose it's possible that Canonical LXC isn't entirely 
compatible with CentOS?


___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Configuring LXC containers to use a host bridge under CentOS 7

[Peter Steele]

On 08/28/2015 02:08 PM, Serge Hallyn wrote:

Can you show the host and container network details and container
xml for your libvirt-lxc setup?  If machines A and B are on the
same LAN, with containers on A, are you saying that B can ping
the containers on A?


Yes, in our libvirt-LXC setup, containers on machine A can ping 
containers on machine B. They all have static IPs taken from the same 
subnet. This was easy to setup in libvirt-LXC. In fact, I just used the 
default behavior provided by libvirt.


Each server has a br0 bridge interface with a static IP assigned to it. 
This is independent of anything to do with libvirt per se, the bridge is 
setup using a standard CentOS 7 configuration file. For example, one of 
my servers has a ifcfg-br0 file that looks like this:


# cat /etc/sysconfig/network-scripts/ifcfg-br0:
DEVICE=br0
NAME=br0
BOOTPROTO=none
ONBOOT=yes
TYPE=Bridge
USERCTL=no
NM_CONTROLLED=no
IPADDR=172.16.110.202
NETMASK=255.255.0.0
GATEWAY=172.16.0.1
DOMAIN=local.localdomain
DEFROUTE=yes

The containers themselves are created using a command similar to this:

virt-install --connect=lxc:///  \
--os-variant=rhel7 \
--network bridge=br0,mac=RANDOM \
--name=test1 \
--vcpus=2 \
--ram=4096 \
--container \
--nographics \
--noreboot \
--noautoconsole \
--wait=60  \
--filesystem /lxc/test1/rootfs/,/

The xml that this generates for the containers is pretty basic:

interface type='bridge'
  mac address='00:16:3e:e1:54:36'/
  source bridge='br0'/
/interface

The container ends up with an eth0 interface with the specified mac 
address, bridged through br0. The br0 interface itself is not visible in 
the container, only lo and eth0.


I did not have to configure anything specifically on the server beyond 
the ifcfg-br0 file. I relied on the default behavior and configuration 
provided by libvirt-LXC. There *is* a network related configuration for 
libvirt, but it's only used if a container uses NAT instead of bridging:


# virsh net-dumpxml default
network
  namedefault/name
  uuid43852829-3a0e-4b27-a365-72e48037020f/uuid
  forward mode='nat'
nat
  port start='1024' end='65535'/
/nat
  /forward
  bridge name='virbr0' stp='on' delay='0'/
  mac address='52:54:00:f9:cd:a3'/
  ip address='192.168.122.1' netmask='255.255.255.0'
dhcp
  range start='192.168.122.2' end='192.168.122.254'/
/dhcp
  /ip
/network

I don't think the info in this xml plays any role in containers 
configured with bridged networking.


The command I use to create my LXC containers looks like this:

# lxc-create -t /bin/true -n test1 --dir=/lxc/test1/rootfs

I populate the rootfs manually using the same template that I use with 
libvirt-LXC, and subsequently customize the container with its own 
ifcfg-eth0 file, /etc/hosts, etc.


I'm clearly missing a configuration step that's needed to setup LXC 
containers with bridged networking like I have with libvirt -LXC...


Peter



___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] Configuring LXC containers to use a host bridge under CentOS 7

[Peter Steele]

We're currently using the CentOS libvirt-LXC tool set for creating and 
managing containers under CentOS 7.1. This tool set is being deprecated 
though so we plan to change our containers to run under the 
linuxcontainers.org framework instead. For simplicity I'll refer to this 
as simply LXC instead of libvirt-LXC.


Under libvirt-LXC, we have our containers configured to use host 
bridging and so are connected to the host network. Each container has 
their own static IP and appear as physical machines on the network. They 
can see each other as well as other systems running on the same network.


I've been unable so far to get host bridging to work with LXC. There is 
a fair amount of information available on networking for LXC but there 
seems to be a lot of different flavors--everyone has their own unique 
solution. I've tried various configurations and I am able to get the 
containers to see each other but have not been able to get them to see 
the host network nor the external internet. The config I am using for my 
containers looks like this:


lxc.utsname = test1
lxc.network.type = veth
lxc.network.link = br0
lxc.network.flags = up

The br0 interface referenced here is the same bridge interface that I 
have configured for use with my libvirt-LXC containers. Some of the 
sites I've come across that discuss setting up host bridging for LXC say 
to configure rules in iptables. However, we do not need any such rules 
with libvirt-LXC, and in fact iptables (or more accurately, firewalld 
under CentOS 7) isn't even enabled on our servers.


In addition to this LXC config I'm using, I have also created 
/etc/sysconfig/network-scripts/ifcfg-eth0 with the following entries:


DEVICE=eth0
NM_CONTROLLED=no
ONBOOT=yes
BOOTPROTO=none
IPADDR=172.16.110.222
NETMASK=255.255.0.0
GATEWAY=172.16.0.1

This is a pretty standard configuration for specifying static IPs. This 
is the exact same file that I use for my libvirt-LXC based containers. 
As I stated, the LXC containers I've created can see each other, but 
they cannot access the host network. They can't even ping their own host 
nor the gateway. The routing table though is the same for both my LXC 
and libvirt-LXC containers:


# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref
Use Iface

default 172.16.0.1  0.0.0.0 UG0 00 eth0
link-local  0.0.0.0 255.255.0.0 U 1021 0
0 eth0

172.16.0.0  0.0.0.0 255.255.0.0 U 0 00 eth0

I'm not sure what LXC magic I am missing to open the containers up to 
the outside network. I'm using the same container template for both my 
LXC and libvirt-LXC tests, and I am using the same host for both. What 
am I missing?


The output of bridge link show br0 with one container running is:

# bridge link show br0
3: bond0 state UP : BROADCAST,MULTICAST,MASTER,UP,LOWER_UP mtu 
1500 master br0 state forwarding priority 32 cost 19
6: virbr0-nic state DOWN : BROADCAST,MULTICAST mtu 1500 master 
virbr0 state disabled priority 32 cost 100
22: veth5BJDXU state UP : BROADCAST,MULTICAST,UP,LOWER_UP mtu 
1500 master virbr0 state forwarding priority 32 cost 2


The veth entry is present only when the container is running. In my 
equivalent setup using libvirt-LXC with one container, the output of 
this command produces essentially the same output except the generated 
name is veth0.


Any advise on how to resolve this issue would be appreciated.

Peter

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users