Re: [Vserver] VServers over NFS

[David MacKinnon]

Tristan Donaldson wrote:
Hi David,
I designed and implemented a setup like this for our company.  We run 
two datastorage servers (primary and backup) which replicate to each 
other using drbd.  We then run 4 servers in front of those which are 
diskless and boot via PXE and mount there root filesystems via NFSv3 
to the backend datastorage servers.  On top of these front end servers 
we run vservers which then allow us to seperate all of our services 
and move them between front end servers to deal with hardware failures 
and load.

Your setup sounds much like what I'm planning. We have a primary and 
secondary storage, just starting to test drbd now. We weren't going to 
go the diskless frontends though. Like you we have all services inside 
vservers now (except for two legacy hosting boxes, which we're still 
migrating things from)

You have to be careful in what applications you run on the front end 
as these need to be nfs nice.  But most things work.  But we did have 
problems with mail queues in postfix, which initially caused lots of 
corruption, but this was fixed by running the mail queue inside a ram 
disk (initially), and then switched to using a loopback device.

That's a bit disappointing, we're looking at moving to postfix for mail, 
I thought postfix actually played nicely with NFS. We currently use 
qmail and I really really want to get away from it (don't ask, or I'll 
probably start ranting :p). I suppose we can have the queue on local 
storage on the front-ends, it's just less elegant. :(

We did have a number of issues with performance of IO.  Since we 
actually have a firewall between the NFS servers and the front end 
servers, we had performance problems with all of the udp traffic 
creating states on the firewall.  We have changed to using NFS over 
TCP.  We also use NFSv3 rather than NFSv2 as it is a lot faster when 
running under the sync option (which you have to run).

I hadn't considered this. Something I'll need to look into.
Another thing to note is we don't run any of our major databases 
across NFS.  We run them inside vservers on the datastorage servers.

We've come to the same conclusion. There are some small databases that 
will probably run over nfs, but our primary ones we'll move to the file 
server. We also do a fair amount of Zope hosting, and we may end up 
moving the ZODBs to a vserver on the file server as well.

We run everything across gigabit ethernet.
For statistics, we are currently running 17 vservers on 3 servers (1 
spare) all mounted across NFS.  Our NFS server is running 5 vservers 
contain different databases.  Our bandwidth to our nfs server sits on 
pretty much 2.5Mbit most of the time (peaks to 25Mbit at times).  Most 
of our vservers are not under heavy use, but we have peaked our out 
going internet traffic from this (http) hosting to about 20Mbit 
without any major performance issues.

We have about 20 vservers at the moment. We're deploying the new setup 
with the servers and two frontends, probably bringing total frontends up 
to about 5 as we migrate vservers off the current boxes (we'll be 
pulling them back to reconfigure as we migrate)

If your hosting is high io intensive.  Then you will probably have 
issues, if its just static html files you should be ok as the file 
should only be loaded over nfs once and cached locally.  For our 
environment all of the high io stuff is inside the database which 
doesn't run over nfs.

Our main intensive stuff is the database at the moment, and the Zopes 
(large ZODBs can cause a lot of disk access). So hopefully we'll 
manage.. :) For reference with bonnie++ over nfs we're seeing about 
40Mbyte/sec writes, 100Mbyte/second reads. That hardly tells the whole 
story, but it's a start.

Thanks to everyone for their comments, certainly some stuff we'll need 
to look at a bit more.

Regards,
-David
___
Vserver mailing list
[EMAIL PROTECTED]
http://list.linux-vserver.org/mailman/listinfo/vserver

Re: [Vserver] error mounting proc

[Björn Steinbrink]

On Mon, 4 Oct 2004 15:06:42 -0600 (MDT)
"Lucas Albers" <[EMAIL PROTECTED]> wrote:

> I'm using vserver as part of the debian util-vserver.

util-vserver 0.30 that is, right? And i guess kernel 2.4-vs1.2x as you don't mention 
any proc-security problems ;) But please tell us if my crystal ball isn't working 
well...

> 
> I setup a vserver by copying it from a working instance to a new instance,
> it appears I did not copy a dev file or similar correclty.

cp -a   should be fine.

> 
> These are the mount entries on the vserver that is not working correctly.
> /dev/hdv1 on / type ext3 (defaults)
> 
> It does not appear that the non functioning vserver is mounting proc or
> dev correctly.
> If I start it up and then mount proc from the host, it appears to work
> correctly.
> If I shut it down and proc is not mounted in the vserver from the host,
> then the shutdown process hangs.

Hmm... interesting, never experienced such a behaviour.

> 
> Any idea what could be wrong:

Do you get any errors/warnings when starting the vserver? Using 'bash -x `which 
vserver` xyz start' you can get a call trace that may also be helpful.

> 
> Here are the /etc/fstab entries for the working vserver,

/etc/fstab is not used within a vserver, all mounts happen within the host (context 0).

HTH
Bjoern
___
Vserver mailing list
[EMAIL PROTECTED]
http://list.linux-vserver.org/mailman/listinfo/vserver

[Vserver] error mounting proc

[Lucas Albers]

I'm using vserver as part of the debian util-vserver.
When I created a vserver it appears to mount the proc directive correctly.
These are the mount entries on a vserver that is working correctly:
--
/dev/hdv1 on / type vfs (none)
proc on /proc type proc (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
--


I setup a vserver by copying it from a working instance to a new instance,
it appears I did not copy a dev file or similar correclty.

These are the mount entries on the vserver that is not working correctly.
/dev/hdv1 on / type ext3 (defaults)

It does not appear that the non functioning vserver is mounting proc or
dev correctly.
If I start it up and then mount proc from the host, it appears to work
correctly.
If I shut it down and proc is not mounted in the vserver from the host,
then the shutdown process hangs.

Any idea what could be wrong:

Here are the /etc/fstab entries for the working vserver,
--
proc/proc   procdefaults0 
 0
--

and the non-working vserver:
--
proc/proc   procdefaults0 
 0
proc/proc   procdefaults0 
 0
--



Thanks for creating such a great product.

-- 
Luke Computer Science System Administrator
Security Administrator,College of Engineering
Montana State University-Bozeman,Montana


___
Vserver mailing list
[EMAIL PROTECTED]
http://list.linux-vserver.org/mailman/listinfo/vserver

Re: [Vserver] 2.6 kernel and links to immutable files

[Herbert Poetzl]

On Mon, Oct 04, 2004 at 03:53:48PM -0400, Gregory (Grisha) Trubetskoy wrote:
> 
> I noticed that in 2.6 kernel you cannot create (hard) 
> links to immutable files.

yes, that's correct ...

> So if I am trying to build a unified server, is my only 
> option to remove the immutable flag temporarily while I 
> link to it? 

no, because you do not use just 'immutable' files

> This seems insecure.

yes, and it probably is ...

> Or am I missing something obvious? 

you are missing something obvious ;)

2.6.9-rc3-vs1.9.3-rc2 (and basically all versions before)

# mkdir /tmp/X
# cd /tmp/X

# touch x1
# setattr --iunlink x1
# showattr x1
UI- x1
# ln x1 x2
# showattr x1 x2
UI- x1
UI- x2

> Has anyone else ran into this?

probably ...

HTH,
Herbert

> Thanks,
> 
> Grisha
> ___
> Vserver mailing list
> [EMAIL PROTECTED]
> http://list.linux-vserver.org/mailman/listinfo/vserver
___
Vserver mailing list
[EMAIL PROTECTED]
http://list.linux-vserver.org/mailman/listinfo/vserver

[Vserver] 2.6 kernel and links to immutable files

[Gregory (Grisha) Trubetskoy]

I noticed that in 2.6 kernel you cannot create (hard) links to immutable 
files.

So if I am trying to build a unified server, is my only option to remove 
the immutable flag temporarily while I link to it? This seems insecure.

Or am I missing something obvious? Has anyone else ran into this?
Thanks,
Grisha
___
Vserver mailing list
[EMAIL PROTECTED]
http://list.linux-vserver.org/mailman/listinfo/vserver

Re: [Vserver] [Release] Stable 1.29

[Sandino Araico Sánchez]

I have just merged the combined-patch 
linux-2.4.27-grsec-2.0.1-vserver-1.29. 


It's available at http://www.sandino.net/parches/vserver/
The following patches were applied:
grsecurity-2.0.1-2.4.27.patch (http://grsecurity.net)
patch-2.4.27-vs1.29.diff.bz2 (http://linux-vserver.org)
patch-2.4.27-vs1.29-q0.14.diff.bz2 (http://linux-vserver.org)
linux-2.4.26-rc1-devmapper-ioctl.patch (http://sources.redhat.com/dm/)
I have not tested it yet (will do it tonight).
The rejects merged by hand were very simple so I expect it to be stable.
Please do conscious testing before using it on production servers.
Herbert Poetzl wrote:
Hi Community!
updated the 1.2 (stable) branch to vs1.29, which
includes a small security improvement and some
warning messages when somebody tries nasty things
with tagged files (like in proc or devpts) ...
nothing else has changed, of course patches have 
been updated for 2.4.27, and a pre3 version of
2.4.28 ...

this release is code-named 'Morrigan' because for
some reason it happened that my significant other 
celebrates her birthday on the 29th of September

as usual, you can get the entire patch as well as
broken out versions from 

   http//www.13thfloor.at/vserver/s_release/v1.29
enjoy,
Herbert
___
Vserver mailing list
[EMAIL PROTECTED]
http://list.linux-vserver.org/mailman/listinfo/vserver
 


--
Sandino Araico Sánchez
-- ... there's no spoon ...
___
Vserver mailing list
[EMAIL PROTECTED]
http://list.linux-vserver.org/mailman/listinfo/vserver

Re: [Vserver] Quickie: loopback mounts

[Herbert Poetzl]

On Mon, Oct 04, 2004 at 01:23:45PM -0300, Ramiro Brito Willmersdorf wrote:
> 
> Hi,
> 
> I need loopback mounts inside a vserver.
> Security is probably not a concern since the root of this vserver
> will always be the same as the the host's root (me).
> 
> I created the loop device loop0 and loop1, but I get

usually providing some info about the kernel and
patches used, maybe even the tools involved provides
valuable information, so that is a good idea ;)

> memlock: Operation not permitted

> When I try to do a loopback mount.

I have no idea why losetup does lock the memory, and
I consider it a little weird, but nevertheless the
required capability is ..

> Is there an extra capability that needs to be set?

/* Allow locking of shared memory segments */
/* Allow mlock and mlockall (which doesn't really have anything to do
   with IPC) */

#define CAP_IPC_LOCK 14

best,
Herbert

> Many thanks!
> 
> -- 
> Ramiro Brito Willmersdorf[EMAIL PROTECTED]  
> GPG key: http://www.demec.ufpe.br/~rbw/GPG/gpg_key.txt
> ___
> Vserver mailing list
> [EMAIL PROTECTED]
> http://list.linux-vserver.org/mailman/listinfo/vserver
___
Vserver mailing list
[EMAIL PROTECTED]
http://list.linux-vserver.org/mailman/listinfo/vserver

[Vserver] Quickie: loopback mounts

[Ramiro Brito Willmersdorf]

Hi,

I need loopback mounts inside a vserver.
Security is probably not a concern since the root of this vserver
will always be the same as the the host's root (me).

I created the loop device loop0 and loop1, but I get

memlock: Operation not permitted

When I try to do a loopback mount.

Is there an extra capability that needs to be set?

Many thanks!

-- 
Ramiro Brito Willmersdorf[EMAIL PROTECTED]  
GPG key: http://www.demec.ufpe.br/~rbw/GPG/gpg_key.txt
___
Vserver mailing list
[EMAIL PROTECTED]
http://list.linux-vserver.org/mailman/listinfo/vserver

Re: [Vserver] VServers over NFS

[Tristan Donaldson]

Hi David,
I designed and implemented a setup like this for our company.  We run 
two datastorage servers (primary and backup) which replicate to each 
other using drbd.  We then run 4 servers in front of those which are 
diskless and boot via PXE and mount there root filesystems via NFSv3 to 
the backend datastorage servers.  On top of these front end servers we 
run vservers which then allow us to seperate all of our services and 
move them between front end servers to deal with hardware failures and load.

You have to be careful in what applications you run on the front end as 
these need to be nfs nice.  But most things work.  But we did have 
problems with mail queues in postfix, which initially caused lots of 
corruption, but this was fixed by running the mail queue inside a ram 
disk (initially), and then switched to using a loopback device.

I did have issues with NFS stale handlers when I tried to mount each 
vserver as a NFS mount.  But we just worked around this by just placing 
everything inside the rootfs of the front server.

We did have a number of issues with performance of IO.  Since we 
actually have a firewall between the NFS servers and the front end 
servers, we had performance problems with all of the udp traffic 
creating states on the firewall.  We have changed to using NFS over TCP. 
 We also use NFSv3 rather than NFSv2 as it is a lot faster when running 
under the sync option (which you have to run).

Here are the options we use for our NFS mounts:
nfsvers=3,hard,nointr,tcp,timeo=600,retrans=2,rsize=32768,wsize=32768
Another thing to note is we don't run any of our major databases across 
NFS.  We run them inside vservers on the datastorage servers.

We run everything across gigabit ethernet.
For statistics, we are currently running 17 vservers on 3 servers (1 
spare) all mounted across NFS.  Our NFS server is running 5 vservers 
contain different databases.  Our bandwidth to our nfs server sits on 
pretty much 2.5Mbit most of the time (peaks to 25Mbit at times).  Most 
of our vservers are not under heavy use, but we have peaked our out 
going internet traffic from this (http) hosting to about 20Mbit without 
any major performance issues.

If your hosting is high io intensive.  Then you will probably have 
issues, if its just static html files you should be ok as the file 
should only be loaded over nfs once and cached locally.  For our 
environment all of the high io stuff is inside the database which 
doesn't run over nfs.

Good luck.
Tristan.
David MacKinnon wrote:
Hey. I know I vaguely recall someone talking about running vservers over 
nfs (ie the vserver directory is hosted on an nfs server, run on a 
front-end). I was just wondering if anyone who has set up something like 
this would care to comment on general performance, and any tuning they did?

We're looking at seting up something like this for our hosting facility, 
to provide greater flexibility, but I didn't realise quite how slow nfs
could be :( We are/were hoping to run over a dozen vservers over this 
setup.

Any comments/suggestions?
-David
___
Vserver mailing list
[EMAIL PROTECTED]
http://list.linux-vserver.org/mailman/listinfo/vserver
___
Vserver mailing list
[EMAIL PROTECTED]
http://list.linux-vserver.org/mailman/listinfo/vserver