Re: routing issue with Jail hosts :: suggestion requested
On Tue, Jan 08, 2013 at 12:39:44PM -0800, Devin Teske wrote: Maybe giving each of the jails their own networking stack would help? Do you know about VIMAGE? I have a boot script that makes it easy to test out this new/experimental (yet very stable) feature: http://druidbsd.sf.net/vimage.shtml VIMAGE (vnet option of jails) can do this but may be a bit overkill. Creating two routing tables and use jail_name_fib in rc.conf to assing each jail to a certain routing table should be enough to do the trick. -- Devin On Jan 8, 2013, at 12:07 PM, Free BSD wrote: Dear List Members I have a scenario where I have an unusual routing need. This is one server with two (or more) interfaces. One of the interface is connected to a public IP network, the other one is connected to the LAN. This box is NOT a gateway machine, just a box serving on two sides of the network. Network diagram below: Interface em1 Public IP Network Connected to Gateway a.b.c.1 ---+ | @|@@ @| @ @ +++@ @ | |@ @ +---+++---++ @ @ | Jail 01 || Jail 02 | @ @ | Public IP a.b.c.4|| Public IP a.b.c.5| @ @ | Gateway a.b.c.1|| Gateway a.b.c.1| @ @ ++++ @ @ @ @ @ @ @ @ ++++ @ @ | Jail 03 || Jail 04 | @ @ | Private IP x.y.z.101 || Private IP x.y.z.102 | @ @ | Gatewayx.y.z.1 || Gatewayx.y.z.1 | @ @ +---+++---++ @ @ | |@ @ +++@ @| @ @ Main Host Server | @ @ Private IP x.y.z.100 | @ @ GW x.y.z.1| @ @| @ @|@@ | Interface em0 | Local Area Network/Privae IP| Connected to GW x.y.z.1 ---+ Now, the problem is, the jails 03 and 04 needs to use the default route for the LAN, since the main firewall on the network does a NAT to these jails. At the same time, the jails 01 and 02 need to use the default route for the public Network, since there are port mappings on them. I will use pf for firewalling, so only certain traffic from certain direction is available. But some traffic are common and could be from any source (i.e., http/S, smtp/S). So, there is http running in jail 01 and jail 03 (two different servers entirely, serving different sites), or smtp/S on jail 02, which too could be from anywhere. Given that /by default/ all jails uses the defaultroute of the host system, I am looking into possible work-around/solution and would appreciate your feedback on this matter. If there were any discussion in the similar line, google failed to yield that to me (had been looking for them for the last two days, but most are dealing with using ipfw and NAT on the same interface -- I am connecting different interface to different network, and would prefer that isolation). If anyone is aware of any such discussion, would appreciate links/pointers to that too. Thanks all. - VFEmail.net - http://www.vfemail.net $14.95 ONETIME Lifetime accounts with Privacy Features! 15GB disk! No bandwidth quotas! Commercial and Bulk Mail Options! ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to freebsd-jail-unsubscr...@freebsd.org _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that
Re: multiple jails with multiple network interfaces
On Sun, Jan 22, 2012 at 10:25:27PM -0600, Valeri Galtsev wrote: Hello! I have a FreeBSD 9.0 host that is registered in DNS to appear with multiple IP addresses: host some.host.com some.host.com has address a.b.c.x some.host.com has address a.b.d.x some.host.com has address a.b.e.x I built multiple jails to run one service in each following mostly: http://www.freebsd.org/doc/handbook/jails-application.html I am trying to start each of the jails with all network interfaces this machine has configured (with the same IP addressed as interfaces are configured on the host system). For that I have in jail related portion of /etc/rc.conf the following jail_enable=YES jail_set_hostname_allow=NO jail_list=http ftp rsync pxe jail_http_hostname=some.host.com jail_http_ip=a.b.c.x,a.b.d.x,a.b.e.x jail_http_rootdir=/jail/http ... jail_ftp_hostname=some.host.com jail_ftp_ip=a.b.c.x,a.b.d.x,a.b.e.x jail_ftp_rootdir=/jail/ftp ... When I start jails: /etc/rc.d/jail start first in the list jail starts perfectly (and I can verify that service configured to run in it is accessible on all three public IP addresses of the machine), all other jails, however, fail to start with the message some# /etc/rc.d/jail start Configuring jails:. Starting jails: some.host.com some.host.com some.host.com ... cannot start jail ftp . If I only leave one IP address in each of the jais, they all start OK. If I configure some jails with different IP (on the same class C network), leaving first jail with multiple IP addresses, e.g.: jail_http_hostname=some.host.com jail_http_ip=a.b.c.x,a.b.d.x,a.b.e.x jail_http_rootdir=/jail/http ... jail_ftp_hostname=some.host.com jail_ftp_ip=a.b.c.y jail_ftp_rootdir=/jail/ftp ... all jails start OK (first with multiple IPs, and other with single different IP). If first (in order of start) jail is with single IP, and next jail is with multiple IPs including the IP of the first one: jail_http_hostname=some.host.com jail_http_ip=a.b.c.x jail_http_rootdir=/jail/http ... jail_ftp_hostname=some.host.com jail_ftp_ip=a.b.c.x,a.b.d.x,a.b.e.x jail_ftp_rootdir=/jail/ftp ... then jail with multiple IPs will not start. I tried to search, but I didn't find anybody mentioning having this problem or having it resolved of just having similar configuration with multiple IPs. Is there something obviously wrong that I'm doing? Is it possible that there is some restriction that will not allow me to have this configuration? See jail(8): ip4.addr ... It is only possible to start multiple jails with the same IP address, if none of the jails has more than this single overlapping IP address assigned to itself. So jails can have the same IP4 address but that has to be the only IP4 address of that jail, otherwise all address must be unique. Kind regards, Paul Schenkeveld ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to freebsd-jail-unsubscr...@freebsd.org
Re: Jexec and access to tty
Hi Ed, On Thu, Aug 25, 2011 at 05:33:56PM +0200, Ed Schouten wrote: Hi all, I am not on this list, but to drop in on this discussion: I don't think you don't need patches of any sort to make jexec + TTYs work in FreeBSD 9.0. This issue has been fixed in r200732. So the changes to jexec(8) are not needed to fix this specific issue. You're my hero! Any idea about MFC? Best regards, -- Ed Schouten e...@80386.nl WWW: http://80386.nl/ Paul Schenkeveld ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to freebsd-jail-unsubscr...@freebsd.org
Jexec and access to tty
Hi, There have been several threads about this issue, some people have come up with work arounds but I think that the issue is more fundamental, that's why I wanted to start this new thread. When using jexec to do interactive work inside an existing jail, people find out that they no longer have access to their tty device. As a result, programs requiring input of passwords or passphrases behave unexpectedly in one of several ways. Ssh says Host key verification failed. and refuses to log in to another system (unless pubkey authentication is user in combination with an agent of course). Some programs fall back to using stdin/stdout and echo the password as it is typed (the mysql clients are popular examples). Work-arounds that have been suggested are 1. Run a sshd inside the jail and log in using ssh 2. Start tmux inside the jail so you get a new pseudo tty slave inside the jail. People trying screen find that it won't work unlike tmux. 3. I tried using 'script -q /dev/null' inside the jail because it is part op the base system and it doesn't change your terminal type and interpret keyboard input and screen output. I found out that I failed when I resized my window :-( I don't like 1 on a machine with many jails, especially if some of them share the same IP address (e.g. sometimes I have to run a mail server on the same IP adress as a webserver but in a distinct jail). 2 is not ideal either because tmux emulates a different terminal on the inside than the terminal on the outside that it runs on. 3 is really a kludge and causes problems when you resize your window. I thought that I found a solution by rewriting jexec such that it will open a pseudo tty and does the passing of data between the jailed pts and the tty from where jexec was started but that's not going to work as the pseudo tty most be opened by the child process inside the jail but the parent outside the jail must have access to the master side of the pseudo tty. So far we are still talking about work-arounds. Why not look at the root cause. Unfortunately I'm not familiar with kernel sources so if I'm wrong, please forgive me, I write this with the best intentions. The root cause of th problem appears to be that pseudo ttys opened outside a jail are not visible nor accessible inside a jail, pseudo ttys created inside a jail are visible and accessible though. Would it be conceivable that by using jexec the controlling tty of jexec magically becomes visible and accessible inside the jail? Preferrable only until jexec dies. I understand that this is not trivial but given the number of threads about this problem, it's a real issue to many people. To me it's worth some $ or EUR to solve this in a clean way. Kind regards, Paul Schenkeveld ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to freebsd-jail-unsubscr...@freebsd.org
Re: New jail(8) with configuration files, not yet in head
Hi, On Sun, Jul 03, 2011 at 11:24:57PM -0600, Jamie Gritton wrote: I'm hoping to get the latest version of jail(8) in before the door slams shut on 9.0. If anyone wants to take a look at the new code and give it a spin, it may help to ease RE's mind about my tardiness. The included diff applies to the current usr.sbin/jail directory. In addition to the new program, it adds a jail.conf(5) man page that explains the config file format (hint: it's a typical C-style block config). Although I really like this new functionality, there is one issue that I am concerned about. Should all this functionality be integrated into the jail(8) command? In UNIX tradition we have simple interfaces[1] in the base system and jail(8) used to fit in quite well. Your new jail(8) tries to be an all singing and dancing subsystem for managing jails but there will always be users that need one more feature[2]. That's why we have ezjail, jailer, jailadmin and others in ports and over time there will be new ones with new features. Jail(8) is the one FreeBSD command to create, modify and destroy jails and is used by many of these wrappers. Your new jail(8) covers part of the functionality of these wrapper but not all. It may grow new features as people submit ideas and you or someone else is willing to code the feature but there will always be wrappers to suit needs not covered by this jail command. I don't want to start a bikeshed discussion and will shut up immediately if I appear to be the only one concerned but if others care like I do, I'd suggest to put this functionality into a separate program and leave jail(8) to be the minimalistic interface to just control the jail(2) system call from a command interpreter. Again, I'm not against your enhancements, I'm just worried about pollution of OS primitives. With kind regards, Paul Schenkeveld [1] Chown/chgrp/chmod are nice examples, I could easily think of at least a dozen enhancements like only operating on files matching a specific user/group or mapping a list of old uids/gids to a list of new ones or looking up user info in LDAP. Sometimes throwing in find(1) will help you solve the problem at hand, sometimes you need to write a script or program. [2] My favorites: starting jails in parallel, dependency graph of jails, monitoring jails and restarting them when they die, migration of jails to other hosts and populating new jails from a release DVD, /usr/src or the source repository and provisioning jails from a database. ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to freebsd-jail-unsubscr...@freebsd.org
Re: Exposing a hierarchy of ZFS datasets inside multiple jails
Hi, On Fri, Jun 17, 2011 at 02:46:59PM -0400, Lars Kellogg-Stedman wrote: Hello all, Hi there, I am trying to expose a hierarchy of home directories to a number of FreeBSD jails. The home directories are configured such that each is a unique ZFS dataset. The jails are used for development work and hence are created and destroyed on a regular basis. My first thought was simply to use nullfs to mount /home inside the jail, but nullfs doesn't provide any way to access subordinate filesystems. My second thought was to export the directories via NFS and then run the automounter daemon (amd) inside each jail. This would have Just Worked...if it were possible to perform NFS mounts inside a jail. But it's not. My third thought was to run amd on the host and provision nullfs mounts into the jails...but amd support for nullfs doesn't exist. My fourth thought was to go back to exporting the directories using NFS, because of course amd works with NFS, right? Unfortunately, rather than mounting a directory on the target mountpoint, amd likes to mount things in a temporary location (/.amd_mnt/...) and then create a symlink...which, of course, is useless inside the jail environment.t So maybe you could use nullfs to expose a subdirectory of /.amd_mnt to the jail? No! This brings us back to my first attempt, in which we find that there is no way to access subordinate filesystems using nullfs. And then my head exploded. Is there a good solution for what I'm trying to do? A bad solution would be to run a script after booting the jail that would create multiple nullfs mountpoints for all the home directories, but this is pretty clunky -- it would need to be run periodically to take into account new directories or removed directories. So basically I would have to write a poorly designed automounter. There must be a better way. How are other folks solving this? It looks like there are discussions going back several years about setting the VFCF_JAIL on NFS filesystems, but it these haven't resulted in any changes to the released code. Is this the best way to go? In theory, if I build a kernel under which NFS is jail friendly I can go ahead and run amd inside the jail Probably not a good solution but to stir the pool of thoughts a bit... Nullfs mounts and NFS mounts operate on filesystems (or datasets) and do not include subordinates. Smbfs operates on directory (sub)trees so have /home and /home/user[123...] datasets outside the jails, run samba there with a share called [home] (not to be confused with the [homes] share that comes with smb.conf.sample) and mount this share using mount_smbfs inside every jail (from fstab.jailname). Just my $.02 Regards, Paul Schenkeveld ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to freebsd-jail-unsubscr...@freebsd.org
Re: Jail starts but doesn't start
On Wed, May 04, 2011 at 04:24:11PM -0700, Mickey Harvey wrote: Hosts /etc/rc.conf 1. ifconfig_bge0=inet 192.168.224.11 netmask 255.255.255.0 2. defaultrouter=192.168.224.1 3. sshd_enable=YES 4. 5. linux_enable=YES 6. zfs_enable=YES 7. jail_enable=YES 8. jail_list=www0 dns0 smarty0 centos 9. 10. ifconfig_bge0_alias0=inet 192.168.224.12 netmask 255.255.255.255 11. jail_www0_rootdir=/tank/jails/www0 12. jail_www0_hostname=www0 13. jail_www0_ip=192.168.224.12 14. jail_www0_devfs_enable=YES 15. jail_www0_exec_stop=/etc/rc.shutdown 16. 17. #JAIL READY TO USE, JUST NEEDS APPROPRIATE FSTAB ENTRIES 18. #ENTRIES ARE IN LOADER.CONF 19. #TRIED TO BOOT WITH REQUIRED FSTAB BUT IT BROKE SO I REVERTED 20. #5/3/11 MH 21. #ifconfig_bge0_alias1=inet 192.168.224.13 netmask 255.255.255.255 22. #jail_deb0_rootdir=/tank/jails/deb0 23. #jail_deb0_hostname=deb0 24. #jail_deb0_ip=192.168.224.13 25. #jail_deb0_devfs_enable=YES 26. #jail_deb0_exec_start=/etc/init.d/rc 3 27. #jail_deb0_exec_stop=/etc/init.d/rc 0 28. #jail_deb0_flags=-l -u root 29. 30. ifconfig_bge0_alias1=inet 192.168.224.14 netmask 255.255.255.255 31. jail_dns0_rootdir=/tank/jails/dns0 32. jail_dns0_hostname=dns0 33. jail_dns0_ip=192.168.224.14 34. jail_dns0_devfs_enable=YES 35. jail_dns0_exec_stop=/etc/rc.shutdown 36. 37. ifconfig_bge0_alias2=inet 192.168.224.15 netmask 255.255.255.255 38. jail_smarty0_rootdir=/tank/jails/smarty0 39. jail_smarty0_hostname=smarty0 40. jail_smarty0_ip=192.168.224.15 41. jail_smarty0_devfs_enable=YES 42. jail_smarty0_exec_stop=/etc/rc.shutdown 43. 44. ifconfig_bge0_alias3=inet 192.168.224.16 netmask 255.255.255.255 45. jail_centos_rootdir=/tank/jails/centos 46. jail_centos_hostname=centos 47. jail_centos_ip=192.168.224.16 48. jail_centos_devfs_enable=YES Result of jls after /etc/rc.d/jail start centos (notice there's no entry for centos) JID IP Address Hostname Path 1 192.168.224.12 www0 /tank/jails/www0 2 192.168.224.14 dns0 /tank/jails/dns0 3 192.168.224.15 smarty0 /tank/jails/smarty0 No error messages when starting or stopping centos jail. /var/run contains jail_centos.id Alias exists on bge0. So I tried jexec 4 /bin/bash figuring jls just isn't showing the centos jail for some reason but: jexec: jail_attach(4): Invalid argument Anybody have any idea about what might be happening here? This usually happens when there are no processes running in the jail to keep it up. By default, jails started thru rc.d/jail are not persistent. Try adding something like cron_enable=YES to rc.conf inside the jail. Regards, Paul Schenkeveld ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to freebsd-jail-unsubscr...@freebsd.org
rc.d/jail issues
Hi,
The order in which jails are started by rc.d/jail is the order in which
jails are listed in $jail_list which is fine. On shutdown, jails are
stopped in the same order they were started which in some cases is not
fine. If jail B depends on functionality provided by jail A, one would
like to start A before B but shutdown B before A. Would it make sense
to reverse the order in which jails are stopped during shutdown by
reversing the nales in $jail_list?
The attached patch reverses $jail_list during shutdown.
Regards,
Paul Schenkeveld
--- etc/rc.d/jail.orig 2009-08-15 14:00:54.0 +0200
+++ etc/rc.d/jail 2011-01-27 13:03:17.0 +0100
@@ -678,7 +678,7 @@
jail_stop()
{
echo -n 'Stopping jails:'
- for _jail in ${jail_list}
+ for _jail in `reverse_list ${jail_list}`
do
if [ -f /var/run/jail_${_jail}.id ]; then
_jail_id=$(cat /var/run/jail_${_jail}.id)
--- etc/rc.d/jail.orig 2009-08-15 14:00:54.0 +0200
+++ etc/rc.d/jail 2011-01-27 13:03:17.0 +0100
@@ -678,7 +678,7 @@
jail_stop()
{
echo -n 'Stopping jails:'
- for _jail in ${jail_list}
+ for _jail in `reverse_list ${jail_list}`
do
if [ -f /var/run/jail_${_jail}.id ]; then
_jail_id=$(cat /var/run/jail_${_jail}.id)
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to freebsd-jail-unsubscr...@freebsd.org
Re: CARP across two jails on one host?
Hi, On Thu, Jul 22, 2010 at 12:51:23PM -0400, Aaron Weeden wrote: My box is running FreeBSD version 8.1-PRERELEASE. I've created two jails and want them to be able to share an IP address via CARP. As I understand it, each host must use the same VHID and IP address on its carp interface in order to work as a failover for the other hosts. I'm also under the impression that jails cannot create interfaces, as my attempt to run 'ifconfig carp0 create' within a jail returned the error 'ifconfig: SIOCIFCREATE2: Operation not permitted'. I'm wondering, then, if it's possible to use CARP for two jails on one host, since attempting to create two carp interfaces with the same vhid on the parent produces the error 'ifconfig: SIOCSVH: File exists'. Does anyone here have experience running CARP in jails? The CARP protocol involves multicast hello packets among the master and backup nodes. Each CARP interface must also be capable of responding to ARP requests if it is operating in MASTER mode. With traditional jails traffic between jails on the same host is sent over the loopback interface which does not support multicasting so thest jails would not be able to see each others hello packets. Since FreeBSD 8 jails support virtual networking (a.k.a. vimage). It looks like it should be possible to do CARP between jails using vnet instances. You'd need to do some network plumbing to get a virtual bus topology network between the jails (ng_ether probably) but I have not yet tried this myself. Also, beware that virtual networking is still not production quality as far as I know and rc.d/jail doesn't know how to set it up (yet). OTOH, is CARP the right solution for your problem? If you would succeed to build the setup using vnet, CARP would only fail over if CARP of the master jail stops sending hello packets. This would normally only occur when the master jail and vnet instance are torn down completely (or the CARP interface in the master jail destroyed). It would not kick in if the application inside the master jail stops responding. If you just want to simulate a multi-host network instead of doing application fail-over then vnet is your best bet. Thank you, Aaron Weeden HTH Paul Schenkeveld ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to freebsd-jail-unsubscr...@freebsd.org
