Re: What's going on with vnets and epairs w/ addresses?

[Bjoern A. Zeeb]

ciple perform
 inp_cred checking after acquiring the lock but before returning.
- If there are no jailed PCBs in a hash chain in_pcblookup_hash_locked()
 always scans the whole chain.
- If we match only one PCB in a lookup, we can probably(?) return that
 PCB without dereferencing the cred pointer at all.  If not, then the
 scan only has to keep track of a fixed number of PCBs before picking
 which one to return.  So it looks like we can perform a lockless scan
 and keep track of matches on the stack, then lock the matched PCBs and
 perform prison checks if necessary, without making the common case
 more expensive.

In fact there is a parallel thread on freebsd-jail which reports that
this inp_cred access is a source of frequent cache misses.  I was
surprised to see that the scan calls prison_flag() before even checking
the PCB's local address.  So if the hash chain is large then we're
potentially performing a lot of unnecessary memory accesses (though
presumably it's common for most of the PCBs to be sharing a single
cred?).  In particular we can perhaps solve two problems at once.


I haven't heard back after I sent the test program there;  I hope that
can be solved independently first and any optimisations can then come.



Any thoughts?  Are there some fundamental reasons this can't work?



--
Bjoern A. Zeeb r15:7

Re: What's going on with vnets and epairs w/ addresses?

[Bjoern A. Zeeb]

On Sun, 18 Dec 2022, Zhenlei Huang wrote:




On Dec 18, 2022, at 3:23 AM, Bjoern A. Zeeb  wrote:

On Sat, 17 Dec 2022, Gleb Smirnoff wrote:


Zhenlei,

On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang wrote:
Z> I managed to repeat this issue on CURRENT/14 with this small snip:
Z>
Z> ---
Z> #!/bin/sh
Z>
Z> # test jail name
Z> n="test_ref_leak"
Z>
Z> jail -c name=$n path=/ vnet persist
Z> # The following line trigger jail pr_ref leak
Z> jexec $n ifconfig lo0 inet 127.0.0.1/8
Z>
Z> jail -R $n
Z>
Z> # wait a moment
Z> sleep 1
Z>
Z> jls -j $n
Z>
Z> After DDB debugging and tracing , it seems that is triggered by a combine of 
[1] and [2]
Z>
Z> [1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 
<https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915>
Z> [2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b 
<https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b>


I can confirm [2] also affects Non-VNET jails.
Prison pr_ref leak cause jail stuck in dying state.


Usually a TCP connection in TW would do this in the old days and things
would solve themselves after a while.  This was always the case even
long before vnet or multi-IP jails.



Z>
Z>
Z> In [1] the per-VNET uma zone is shared with the global one.
Z> `pcbinfo->ipi_zone = pcbstor->ips_zone;`
Z>
Z> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by 
uma_zfree_smr() .
Z>
Z> Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is 
not called immediately ,
Z> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`.
Z>
Z> And it is also not possible to free up the cache by per-VNET SYSUNINIT 
tcp_destroy / udp_destroy / rip_destroy.

This is known issue and I'd prefer not to call it a problem. The "leak" of a 
jail
happens only if machine is idle wrt the networking activity.

Getting back to the problem that started this thread - the epair(4)s not 
immediately
popping back to prison0. IMHO, the problem again lies in the design of if_vmove 
and
epair(4) in particular. The if_vmove shall not exist, instead we should do a 
full
if_attach() and if_detach(). The state of an ifnet when it undergoes if_vmove 
doesn't
carry any useful information. With Alexander melifaro@ we discussed better 
options
for creating or attaching interfaces to jails that if_vmove. Until they are 
ready
the most easy workaround to deal with annoying epair(4) come back problem is to
remove it manually before destroying a jail, like I did in 80fc25025ff.


Ok, move an em0 or cxl0 into the jail;  the problem will be the same I
bet and you need the physical interface to not disappear as then you
cannot re-create a new jail with it.


Re-read sys/kern/kern_jail.c, if pr_ref leaks, vnet_destroy() has no chance to 
be called, thus
if_vmove is not called and epair(4)s or em0, exl0 are not returned to home vnet.

That can be confirmed by setting debug point on vnet_destroy by DDB, and then 
create and destroy vnet jails.

So before the problem prison pr_ref count leaks is resolved, it will cover 
other potential problems such as @glebius
pointed out.

I think the problem that prison ref count leaks should be resolved first.

I'm also reviewing the life cycles of prison / vnet and it seems they could 
still be improved.


But that's the not the problem here as your own test case pointed out.

The point is that if you start a plain vnet jail put an interface in and
destroy the jail that works instantly.
The moment you put an address on any interface (incl. loopback as your
test showed, which will not do ARP/NDP things compared to an ethernet
interface) the jail will no longer die immediately.

Simply putting an address on an interface should not defer things.
So indeed something holds onto things there and is not cleaned up
anymore.  Finding that "something" is the important bit and being able
to clean it up.

I always say, if you have a machine in shutdown -r you don't want it
hanging for hours either (now if you toggle the power switch you can do
a lot more without panicing the rest of the system but with jails we
cannot do that).  And we did have vnet jails shutting down preoperly and
clearing up for years.  People had spent a lot of time on that.  So it is
possible and we need to get back to that state.

/bz

--
Bjoern A. Zeeb r15:7

Re: What's going on with vnets and epairs w/ addresses?

[Bjoern A. Zeeb]

On Sat, 17 Dec 2022, Gleb Smirnoff wrote:


 Zhenlei,

On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang wrote:
Z> I managed to repeat this issue on CURRENT/14 with this small snip:
Z>
Z> ---
Z> #!/bin/sh
Z>
Z> # test jail name
Z> n="test_ref_leak"
Z>
Z> jail -c name=$n path=/ vnet persist
Z> # The following line trigger jail pr_ref leak
Z> jexec $n ifconfig lo0 inet 127.0.0.1/8
Z>
Z> jail -R $n
Z>
Z> # wait a moment
Z> sleep 1
Z>
Z> jls -j $n
Z>
Z> After DDB debugging and tracing , it seems that is triggered by a combine of 
[1] and [2]
Z>
Z> [1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 
<https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915>
Z> [2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b 
<https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b>
Z>
Z>
Z> In [1] the per-VNET uma zone is shared with the global one.
Z> `pcbinfo->ipi_zone = pcbstor->ips_zone;`
Z>
Z> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by 
uma_zfree_smr() .
Z>
Z> Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is 
not called immediately ,
Z> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`.
Z>
Z> And it is also not possible to free up the cache by per-VNET SYSUNINIT 
tcp_destroy / udp_destroy / rip_destroy.

This is known issue and I'd prefer not to call it a problem. The "leak" of a 
jail
happens only if machine is idle wrt the networking activity.

Getting back to the problem that started this thread - the epair(4)s not 
immediately
popping back to prison0. IMHO, the problem again lies in the design of if_vmove 
and
epair(4) in particular. The if_vmove shall not exist, instead we should do a 
full
if_attach() and if_detach(). The state of an ifnet when it undergoes if_vmove 
doesn't
carry any useful information. With Alexander melifaro@ we discussed better 
options
for creating or attaching interfaces to jails that if_vmove. Until they are 
ready
the most easy workaround to deal with annoying epair(4) come back problem is to
remove it manually before destroying a jail, like I did in 80fc25025ff.


Ok, move an em0 or cxl0 into the jail;  the problem will be the same I
bet and you need the physical interface to not disappear as then you
cannot re-create a new jail with it.

/bz

--
Bjoern A. Zeeb r15:7

Re: What's going on with vnets and epairs w/ addresses?

[Bjoern A. Zeeb]

On Fri, 16 Dec 2022, Zhenlei Huang wrote:

Hi,


I managed to repeat this issue on CURRENT/14 with this small snip:

---
#!/bin/sh

# test jail name
n="test_ref_leak"

jail -c name=$n path=/ vnet persist
# The following line trigger jail pr_ref leak
jexec $n ifconfig lo0 inet 127.0.0.1/8

jail -R $n

# wait a moment
sleep 1

jls -j $n


---


After DDB debugging and tracing , it seems that is triggered by a combine of 
[1] and [2]

[1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 
<https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915>
[2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b 
<https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b>


In [1] the per-VNET uma zone is shared with the global one.
`pcbinfo->ipi_zone = pcbstor->ips_zone;`

In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by 
uma_zfree_smr() .

Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not 
called immediately ,
thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`.

And it is also not possible to free up the cache by per-VNET SYSUNINIT 
tcp_destroy / udp_destroy / rip_destroy.


Thanks a lot for tracking it down.

That seems to be a regression then that needs to be fixed before
14.0-RELEASE will happen as it'll break management utilities of people.

Could you open a bug report and flag it as such?

/bz





Best regards,
Zhenlei


On Dec 14, 2022, at 9:56 AM, Zhenlei Huang  wrote:


Hi,

I also encounter this problem while testing gif tunnel between jails.

My script is similar but with additional gif tunnels.


There are reports in mailing list [1], [2], and another one in forum [3] .

Seem to be a long standing issue.

[1] https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086126.html 
<https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086126.html>
[2] https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003357.html 
<https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003357.html>
[3] 
https://forums.freebsd.org/threads/jails-stopping-prolonged-deaths-starting-networking-et-cetera.84200/
 
<https://forums.freebsd.org/threads/jails-stopping-prolonged-deaths-starting-networking-et-cetera.84200/>


Best regards,
Zhenlei


On Dec 14, 2022, at 7:03 AM, Bjoern A. Zeeb mailto:b...@freebsd.org>> wrote:

Hi,

I have used scripts like the below for almost a decade and a half
(obviously doing more than that in the middle).  I haven't used them
much lately but given other questions I just wanted to fire up a test.

I have an end-November kernel doing the below my eapirs do not come back
to be destroyed (immediately).
I have to start polling for the jid to be no longer alive and not in
dying state (hence added the jls/ifconfig -l lines and removed the
error checking from ifconfig destroy).  That seems sometimes rather
unreasonably long (to the point I give up).

If I don't configure the addresses below this isn't a problem.

Sorry I am confused by too many incarnations of the code; I know I once
had a version with an async shutdown path but I believe that never made
it into mainline, so why are we holding onto the epairs now and not
nuking the addresses and returning them and are clean?

It's a bit more funny; I added a twiddle loop at the end and nothing
happened.  So I stop the script and start it again and suddenly another
jail or two have cleaned up and their epairs are back.  Something feels
very very wonky.  Play around with this and see ... and let me know if
you can reproduce this...  I quite wonder why some test cases haven't
gone crazy ...

/bz


#!/bin/sh

set -e
set -x

js=`jail -i -c -n jl host.hostname=left.example.net <http://left.example.net/> 
vnet persist`
jb=`jail -i -c -n jr host.hostname=right.example.net 
<http://right.example.net/> vnet persist`

# Create an epair connecting the two machines (vnet jails).
ep=`ifconfig epair create | sed -e 's/a$//'`

# Add one end to each vnet jail.
ifconfig ${ep}a vnet ${js}
ifconfig ${ep}b vnet ${jb}

# Add an IP address on the epairs in each vnet jail.
# XXX Leave these out and the cleanup seems to work fine.
jexec ${js}  ifconfig ${ep}a inet  192.0.2.1/24
jexec ${jb}  ifconfig ${ep}b inet  192.0.2.2/24

# Clean up.
jail -r ${jb}
jail -r ${js}

# You want to be able to remove this line ...
set +e

# No epairs to destroy with addresses configured; fine otherwise.
ifconfig ${ep}a destroy
# echo $?

# Add this is here only as things are funny ...
# jls -av jid dying
# ifconfig -l

# end
--------

--
Bjoern A. Zeeb r15:7








--
Bjoern A. Zeeb r15:7

Re: prison_flag() check in hot path of in_pcblookup()

[Bjoern A. Zeeb]

On Tue, 13 Dec 2022, Andrew Gallatin wrote:


[ I added pjd, since the original patch came from him ]

Just to make sure I understand, I have a simple yes/no question:

Can  jails and the host ever share the same (local) port and the same IP?


Can they currently (I tested only for TCP)?

- local binds can overlap like they can with just the base system.
  so bind(... {AF_INET, laddr, lport} ... ) works fine (REUSEPORT).

- tcp connect of a 2nd socket to the same {faddr, fport} from the above
  bind will fail with 'Address already in use'  [currently]
  [I believe that would mean your patch could go in? Where does the error come 
from [%]?] [*]

- tcp listen will work on {laddr, lport} if run in paralllel (REUSEPORT)
  or in base and jail at the same time.

[%] likely in_pcbconnect_setup() ?  Also one should check the other
order (jail first then base);  also we assume no other race
conditions in this rather simple testing...

[*] Now someone should run this on a FreeBSD 7.3 / 8.x or later and see how it 
behaves as the stack might have behaved differently.
Also if you have two physical machines or two VMs connected remove the VNET 
layer and just (manually) test the two parts firing up one extra jail on each 
base system.
I just used vnets for simplicity of my testing.

(sorry vnet cleanup currently screwed as it seems 15 years of working also have 
changed due to other changes; you cal always run jail -r jl jr manually).

I haven't done user space socket programming in a while so this was fun (and 
hopefully does what it should for this test case).
I put the simple sources (shell script and C file) up at:
 https://people.freebsd.org/~bz/tmp/jail-in_pcblookup/

HTH /bz

+ pwd
+ STESTBIN=/home/test/socket
+ PORT=7
+ jail -i -c -n jl 'host.hostname=server.example.net' vnet persist 
'children.max=1'
+ js=211
+ jail -i -c -n jr 'host.hostname=base.example.net' vnet persist 
'children.max=1'
+ jb=212
+ jexec 211 ifconfig lo0 inet 127.0.0.1/8 alias up
+ jexec 211 ifconfig lo0 inet6 ::1/128 alias
+ jexec 212 ifconfig lo0 inet 127.0.0.1/8 alias up
+ jexec 212 ifconfig lo0 inet6 ::1/128 alias
+ ifconfig epair create
+ sed -e 's/a$//'
+ ep=epair102
+ ifconfig epair102a vnet 211
+ ifconfig epair102b vnet 212
+ jexec 211 ifconfig epair102a inet 192.0.2.1/24
+ jexec 212 ifconfig epair102b inet 192.0.2.2/24
+ jexec 211 jail -i -c -n jsj 'host.hostname=jails.example.net' 
'ip4.addr=192.0.2.1' persist
+ jexec 211 /home/test/socket 192.0.2.1 7
/home/test/socket pid 23254 listening on [192.0.2.1 7]
+ jsj=213
+ echo 'Listing listening connections from the server (base) system'
+ jexec 213 /home/test/socket 192.0.2.1 7
Listing listening connections from the server (base) system
+ jexec 211 netstat -an
/home/test/socket pid 23257 listening on [192.0.2.1 7]
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address  Foreign Address(state) 
tcp4   0  0 192.0.2.1.7*.*LISTEN 
tcp4   0  0 192.0.2.1.7*.*LISTEN 
+ jexec 212 jail -i -c -n jbj 'host.hostname=jailb.example.net' 'ip4.addr=192.0.2.2' persist

+ jbj=214
+ sleep 1
+ echo 'Starting connection from base jail'
Starting connection from base jail
+ sleep 1
+ jexec 212 /home/test/socket 192.0.2.2 12345 192.0.2.1 7
/home/test/socket pid 23257 accepted [192.0.2.2 12345]
/home/test/socket pid 23261 [192.0.2.2 12345] sleeping 60.
+ echo 'Starting connection from plain-old IP jail'
Starting connection from plain-old IP jail
+ sleep 1
+ jexec 214 /home/test/socket 192.0.2.2 12345 192.0.2.1 7
socket: connect
: Address already in use
+ echo 'Listing server connections from the server (base) system'
Listing server connections from the server (base) system
+ jexec 211 netstat -an
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address  Foreign Address(state) 
tcp4   0  0 192.0.2.1.7192.0.2.2.12345ESTABLISHED
tcp4   0  0 192.0.2.1.7*.*LISTEN 
tcp4   0  0 192.0.2.1.7*.*LISTEN 
+ echo 'Listing client connections from the base system'

Listing client connections from the base system
+ jexec 212 netstat -an
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address  Foreign Address(state) 
tcp4   0  0 192.0.2.2.12345192.0.2.1.7ESTABLISHED

+ sleep 60
^C

--
Bjoern A. Zeeb r15:7

What's going on with vnets and epairs w/ addresses?

[Bjoern A. Zeeb]

Hi,

I have used scripts like the below for almost a decade and a half
(obviously doing more than that in the middle).  I haven't used them
much lately but given other questions I just wanted to fire up a test.

I have an end-November kernel doing the below my eapirs do not come back
to be destroyed (immediately).
I have to start polling for the jid to be no longer alive and not in
dying state (hence added the jls/ifconfig -l lines and removed the
error checking from ifconfig destroy).  That seems sometimes rather
unreasonably long (to the point I give up).

If I don't configure the addresses below this isn't a problem.

Sorry I am confused by too many incarnations of the code; I know I once
had a version with an async shutdown path but I believe that never made
it into mainline, so why are we holding onto the epairs now and not
nuking the addresses and returning them and are clean?

It's a bit more funny; I added a twiddle loop at the end and nothing
happened.  So I stop the script and start it again and suddenly another
jail or two have cleaned up and their epairs are back.  Something feels
very very wonky.  Play around with this and see ... and let me know if
you can reproduce this...  I quite wonder why some test cases haven't
gone crazy ...

/bz


#!/bin/sh

set -e
set -x

js=`jail -i -c -n jl host.hostname=left.example.net vnet persist`
jb=`jail -i -c -n jr host.hostname=right.example.net vnet persist`

# Create an epair connecting the two machines (vnet jails).
ep=`ifconfig epair create | sed -e 's/a$//'`

# Add one end to each vnet jail.
ifconfig ${ep}a vnet ${js}
ifconfig ${ep}b vnet ${jb}

# Add an IP address on the epairs in each vnet jail.
# XXX Leave these out and the cleanup seems to work fine.
jexec ${js}  ifconfig ${ep}a inet  192.0.2.1/24
jexec ${jb}  ifconfig ${ep}b inet  192.0.2.2/24

# Clean up.
jail -r ${jb}
jail -r ${js}

# You want to be able to remove this line ...
set +e

# No epairs to destroy with addresses configured; fine otherwise.
ifconfig ${ep}a destroy
# echo $?

# Add this is here only as things are funny ...
# jls -av jid dying
# ifconfig -l

# end
--------

--
Bjoern A. Zeeb r15:7

Re: prison_flag() check in hot path of in_pcblookup()

[Bjoern A. Zeeb]

On Tue, 13 Dec 2022, Andrew Gallatin wrote:


Are there regression tests for jails where this patch could be run to
ensure it is safe?


Not that I now of but it would certainly good to have one for that case.

But it's likely not going to be deterministic so the question will be
more of the case "theoretically possible or not"?

--
Bjoern A. Zeeb r15:7

Re: prison_flag() check in hot path of in_pcblookup()

[Bjoern A. Zeeb]

On Tue, 13 Dec 2022, James Gritton wrote:

Hi,

Argh, sorry Drew, I looked at the wrong of the two checks in the
function earlier.  Sorry, that's what happens if trying to be helpful
when firefighting elsewhere.


On 2022-12-13 09:18, Andrew Gallatin wrote:

I was trying to improve the performance of in_pcblookup(), as it is a very 
hot path for us (Netflix). One thing I noticed was the prison_flag() check 
in in_pcblookup_hash_locked() can cause a cache miss just by deref'ing the 
cred pointer, and it can also cause multiple misses in tables with 
collisions by causing us to walk the entire chain even after finding a 
perfect match.


I'm curious why this check is needed.  Can you explain it to me?  It 
originated in this commit:


commit 413628a7e3d23a897cd959638d325395e4c9691b
Author: Bjoern A. Zeeb 
Date:   Sat Nov 29 14:32:14 2008 +

MFp4:
Bring in updated jail support from bz_jail branch.

This enhances the current jail implementation to permit multiple
addresses per jail. In addtion to IPv4, IPv6 is supported as well.

My thinking is that a jail will either use the host IP, and share its port 
space, or it will have its own IP entirely (but I know nothing about 
jails).


Well having its own IP address entirely doesn't work with classic jails
as there is only one network stack where they can be configured on
an interface, and that is the base system.

So de-facto all jail address/port space is always shared with the host
(ignoring vnet jails with their own entire virtual network stack).
Whether the host would bind/listen is a different story.  I know 15
years ago people would bind the sshd of the base to a single-IP address
which was not assigned to jails and then the sshd inside a jail would
bind to a different (single) IP address.   If one doesn't do that and the
sshd inside the jail isn't runnig one would end up trying to connect to
the sshd in the base system (sshd being a popular example).  Not sure if
people still do but that's still the case.

There are special cases with classic multi-IP jails in that they then
cannot have overlapping IP-ranges as otherwise it would not be
deterministic which jail would get an inbound connection on
inaddr_any:port_n if two jails were to listen on the same port_n.

Hope that helps for basic understanding.


In either case, a perfect 4-tuple match should be enough to 
uniquely identify the connection.


Even if this somehow is not the case and we have multiple connections 
somehow sharing the same 4-tuple, how does checking the prison flag help 
us?


That logic predates me and came from [1].  The jail_jailed_sockets_first
sysctl got removed in the review process with rwatson.  I am still trying
to see where the SO_REUSEPORT comment (back then) came from.  I know I
only had the first lines initially, so must have been sometime during
review with rwatson as well.   Sadly p4 emails where truncated to 1000
lines so I cannot simply grep for the change (if it is in my mail
archives) or had a useful commit message (but at least would give a
date to check further private email).

My current guess is that if we have the 4-tuple in both the base
and a jail (hence the SO_REUSEPORT comment) we want the jail not getting
a socket of the base system returned as that would mean one could "break
out of prison".  But if the inp belongs to a jail we know we can simply
return.  So if you find the one of the base system first you'll have to
go and look through the others.

XXX-jamie: is that all still true in hierarchical jails?

Now whether fabricating the case of having both inps on the hash is
still theorectically possible I cannot say.  I haven't followed all the
changes to that code lately close enough.


It would prefer the jailed connection over the non jailed, but that 
would shadow a host connection.  And if we had 2 jails sharing the same 
4-tuple, the first jail would win.


I can't see how this check is doing anything useful, so I'd very much like 
to remove this check if possible.   Untested patch attached.


For a complete 4-tuple, it should indeed be the case that a match would only 
ever identify a single prison.  The later part of the function that examines 
wildcards definitely needs the check.  I don't get the XXX comment about both 
being bound with SO_REUSEPORT, because I would only expect that to apply to 
listening, not to full connections. But I also expect Bjoern to know more 
than I do here...


/bz

[1] https://people.freebsd.org/~pjd/patches/jail_2004120901.patch

--
Bjoern A. Zeeb r15:7

Re: how to determine primary (source) IP address in jail

[Bjoern A. Zeeb]

On 28 Feb 2019, at 10:58, Miroslav Lachman wrote:

Is there some easy way to determine the primary (source) address which 
is used in jail with multiple IP addresses?


I came to this problem with running local_unbound in jail. Unbound 
refuses queries originating in this jail because the do not come from 
real 127.0.0.1 (which is the only one allowed by default). Unbound in 
jail see requests come from jails IP. It is easy to determine (in 
shell script) if jail has only one IP.
But what in case where jail has multiple IPs? Is there some sysctl or 
some call to ifconfig or any other util to get the IP which will be 
used as source address for queries on local services in jail?


Bind the listen socket of the local unbound to any IP of your jail and 
other services (unless the source port got bound) will select the same 
IP address as the destination if both are in the same jail.




I know I can allow all IPs of jail in
access-control: a.b.c.d/32 allow
access-control: e.f.g.h/32 allow

I am just curios if there is some way to get "primary" IP in jail 
without calling anything from the host environment.


Open a UDP socket; bind to 127.1; call getsockname;
https://reviews.freebsd.org/D19218   is currently having a similar issue 
solving it exactly that way.



There were people who in the past added a 127.{2,3,4,5,..}  for each 
jail and then used that one instead of 127.1 but I’ve never been a 
huge fan of that, especially given one  may run the resolver for other 
services outside that jail (maybe in others) as well and they need to be 
able to reach that in a reliable way.



/bz
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: init in a jail

[Bjoern A. Zeeb]

On 11 Feb 2019, at 17:23, James Gritton wrote:


On 2019-02-11 08:48, Michael W. Lucas wrote:

Sadly, my google-fu has turned up thousands of man pages but no real
discussion on this.

According to init(8), you can run init inside a jail.

 If init is run in a jail, the security level of the "host 
system" will
 not be affected.  Part of the information set up in the kernel 
to support
 a jail is a per-jail security level.  This allows running a 
higher
 security level inside of a jail than that of the host system.  
See

 jail(8) for more information about jails.


If you actually try, though, the jail dies:

storm~;jail -vc loghost
loghost: run command: /sbin/ifconfig jailether inet 198.51.100.225 
netmask

255.255.255.255 alias
loghost: run command: /sbin/mount -t devfs -oruleset=4 . 
/jail/loghost/dev

loghost: run command: logger trying to start jail loghost...
loghost: jail_set(JAIL_CREATE) persist name=loghost 
path=/jail/loghost

host.hostname=loghost.mwl.io ip4.addr=19 8.51.100.225
loghost: created
loghost: run command in jail: /sbin/init
jail: loghost: /sbin/init: failed
loghost: removed
loghost: run command: /sbin/umount /jail/loghost/dev
loghost: run command: /sbin/ifconfig jailether inet 198.51.100.225 
netmask

255.255.255.255 -alias

Is that init(8) text left over from an earlier jail incarnation? Or 
is

there some other way to run init in a jail?

And WHY would you run init in a jail?


Interesting - I wonder how long it's been since init worked inside 
jails.  From the look of your error messages, probably not since devfs 
started being used.  I wasn't even aware the init(8) had anything to 
say on the matter, but it's clearly erroneous.


Ken Smith added that message to init(8) 15 years ago and from the sounds 
of it, I think it was more related to securelevels.




AS to why it would be good to have a per-jail init, there would be a 
few advantages.  Orphaned processes could then reparent to the jail's 
init instead of the real init, and the jail root could easily reboot 
jails. Doing it right would require presenting jailed init as pid 1, 
but that's not really very hard.


It’s not just PID 1 but yeah;  I have open reviews (which I should 
update) from the vps branch to do a virtualised pid space, real init to 
jails along with it, console, and then init would also manage ttys, ..  
I need to work on the management bits from the host side to make it a 
real thing (ps, kill, etc. to work with a (jid, pid) combination as 
jexec won’t work anymore (possible collisions etc).  But that’s 
unrelated to this thread.



/bz
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: icmp (IPv4) issues with VIMAGE JAILs and IPv6

[Bjoern A. Zeeb]

On 28 Jan 2019, at 12:44, O. Hartmann wrote:


I ran into severe problems on CURRENT ( FreeBSD 13.0-CURRENT #193
r343521: Mon Jan 28 10:26:36 CET 2019 amd64), VIMAGE enabled host with 
jails

utilizing IPv6.


and you forget to mention in the subject that it seems to be an ipfw 
problem and thus missing your target audience most likely.



Stopping all jails, destroying all epairs and bridge0 doesn't change 
anything.


The problems occured when IPv6 came into play on the specific host in 
question.


Does that mean you could reproduce the problem just with bce0 and no 
jail+vnets at all just with IPv4, IPv6, and ipfw on the main host?


If you cannot, would starting a vnet-jail (without the bridge and 
connecting the epair), just starting a jail in persist mode, make a 
difference;  or would strating a jail and applying the ipfw rules 
therein start the problem?




Does anyone have any ideas? I'm out of ideas.


y best guess is to move the thread to freebsd-ipfw (Cc: and Reply-To: 
set) and see if people pick it up there some more and also finding the 
minimalistic case to reproduce.


/bz
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: enforce_statfs showing leading path

[Bjoern A. Zeeb]

On 9 Jan 2019, at 9:42, Alexander Leidinger via freebsd-jail wrote:

Hi,

I’ll be a bit verbose also for mwlucas.

You see the dataset name of zfs without stripping. The mount point is 
correctly stripped. I don't remember how this looks on ufs.


/dev/ada0p19 on / (ufs, local, journaled soft-updates)
/local/data/www1/users/johndoe on /usr/home/johndoe/www1 (nullfs, local)

The “device” is also visible there, as well as file system type and 
specific options.


I also added a nullfs example.  For that the mount point is properly 
treated and lost the /local/jails/whatever/ prefix with enforce_statfs=1 
but the “device” side is just as visible in full as it is for a any 
other real device.



With jailed datasets we would need more than just some code to remove 
parts of the name.


So it's a doc bug (clarity about mount points and dataset names) and a 
zfs issue.


Well, no it’s not a zfs specific issue.  And the docs talk about mount 
points not about the “device” (or dataset in zfs parlance).


If anything for clarity one could add a sentence to the jail(8) page 
saying that the “device” part of the mount output is not being 
restricted or altered.



One of the reasons for enforce_statfs certainly was to limit the amount 
of information;  that also has the side-effect of scripts parsing the 
mount (mount points) output actually finding the paths they might be 
looking for.



The df command output might make some of this all a bit more clear.



Am 8. Januar 2019 8:34:17 nachm. schrieb "Michael W. Lucas" 
:



Hi,

I'm experimenting with enforce_statfs for the jails book, and have 
hit
an inconsistency. Not sure if the bug should go to src or doc. 
Running

last week's -current.

According to jail(8):

When set to 1, only mount points below the jail's chroot
directory are visible.  In addition to that, the path to 
the
jail's chroot directory is removed from the front of 
their path‐

names.

Seems pretty clear that I shouldn't see anything other than

# jls -h name enforce_statfs
...
ioc-www1 1

So, as I read it, the jail's chroot directory should be stripped down
to /. But inside the jail:

root@www1:~ # mount
iocage/iocage/jails/www1/root on / (zfs, local, nfsv4acls)


   it is stripped down to /


devfs on /dev (devfs, local, multilabel)
fdescfs on /dev/fd (fdescfs)

I see the jail's chroot directory.

This seems to contradict the man page, unless I'm misunderstanding.

Is this a software bug? A ZFS thing? A doc bug? Or am I just an 
idiot?


Also, should this path be stripped when enforce_statfs is set to 1 
*or

above*? Or is this strictly when set to 1? If I'm filing a bug, it
might as well be complete...

Thanks,
==ml

--
Michael W. Lucas https://mwl.io/
author of: Absolute OpenBSD, SSH Mastery, git commit murder,
Immortal Clay, PGP & GPG, Absolute FreeBSD, etc, etc, etc...



--
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to 
"freebsd-jail-unsubscr...@freebsd.org"




___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to 
"freebsd-jail-unsubscr...@freebsd.org"

___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Issue with 127.0.0.1 when reconfiguring running Jail

[Bjoern A. Zeeb]

On 6 Aug 2018, at 16:32, Support SimpleRezo wrote:


Hi !

I'm fancing an issue when i'm using "jail -m ip4.addr=..." for
reconfiguring ip4.addr of a running jail: accessing or binding 
127.0.0.1 is

not redirect anymore by kernel to the jail IP.

Is it expected? Do I missing something there?


The change of address should not make a difference;  the primary jail 
address should be used.


Can you show us a very short (scripted) example of the problem?

/bz
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Time for those old global jail sysctls to go

[Bjoern A. Zeeb]

On 22 Mar 2018, at 4:13, James Gritton wrote:

I've got a revision in the works  
to

remove the security.jail.foo_allowed sysctls:


The old jail system had sysctls to set jail permissions for all jails
(e.g. security.jail.mount_allowed), which were superseded by per-jail
permissions (e.g. allow.mount). These old sysctls remain a constant
source of confusion to users, who expect that setting the sysctl will
change the behavior of existing jails. That the sysctl value at the 
time
a jail is created may matter is a backward-compatibility hack that 
does
little or nothing to relieve the confusion. So it's time for them to 
go.



Also, jail(2) has been replaced by jail_set(2) for a number of years
now, and it really ought to retire - at least into the COMPAT world.


This may be of interest to anyone who works with jails.  My hope is 
that
no current software relies on these old sysctls, and they can be 
removed
with little trouble.  But removing old things never seems to go that 
easy.


I think #1 action item is to put them under BURN_BRIDGES or however it 
was spelt if you really want to remove them.
Then for the next major version they could go away ( I’d be all up for 
removing them immediately (incl. from the man pages ) but I remember 
there used to be 2-3 ports which used the jail v1 stuff;  might be worth 
checking that they were updated or are gone?



/bz
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: IPSEC in VNET Jails

[Bjoern A. Zeeb]

On 29 Nov 2017, at 11:40, Kristof Provost wrote:


On 29 Nov 2017, at 12:16, Matthias Meyser wrote:

Hi

i use a IPSEC Tunnel inside a VNET jail without problems.

Annoyingly /etc/rc.d/ipsec dos not run in VNET jails.

This is fixed in head see
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211364

This is NOT MFCed to stable/11 because the author isn't convinced 
that VNET jails are "is sufficiently robust in stable/11 to encourage 
people to use it"


As this fix only makes a difference if you

1) Have compiled a Kernel WITH VIMAGE support
2) Setup and configured a VNET jail.
3) Setup IPSEC inside the VNET jail.

i think this should be MFCed.

I stand by my initial assessment that VNET is not sufficiently stable 
in stable/11 to encourage its use there.
There are still issues with IPSec, even in head. See 
https://reviews.freebsd.org/D13017 for some more information on that.
Those issues are being addressed in head, but I do not expect VNET to 
ever become robust in 11.


Well, whether people will use it or not is their decision.

If they want to give it a try I don’t see any harm why ipsec should 
not start.   It’s a lot more likely to work than some firewalls, given 
I used it years ago under vnet to debug ipcomp problems.


I think in order to not waste more time on this, can we just MFC the 
change to 11?


Feel free to put in   “Urged to by: bz”


Thanks,
/bz
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: testing 11.0-RC1 vnet jails with ipfilter

[Bjoern A. Zeeb]

On 17 Aug 2016, at 1:05, Ernie Luzar wrote:

In light of 11.0 release being published soon there should be 
something posted to the release notes talking about this with sample 
code for a combined rule #5. This would give vnet users a copy & paste 
solution to use until jail(8) gets updated in 11.1.


VIMAGE is not a feature shipping by default in 11.0;  I don’t see the 
point to put more details into the release notes than needed;  we should 
figure out samples/man pages/wiki maybe/handbook though.


___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: testing 11.0-RC1 vnet jails with ipfilter

[Bjoern A. Zeeb]

On 16 Aug 2016, at 21:08, CyberLeo Kitsana wrote:


On 08/16/2016 03:21 PM, Ernie Luzar wrote:


Issuing "ipf -FS -Fa" command from within the vnet jail gives this
message, "open device:no such file or directory. User kernel version
check failed.


According to ipf(8), the ipfilter utilities touch /dev/ipauth , 
/dev/ipl
, and /dev/ipstate . Have you checked that the devfs ruleset applied 
to

your jail has those unhidden?


Issuing "ipfstat -hnio command from within the vnet jail gives this
message, open(IPSTATE_NAME):no such file or directory.


ipfstat(8) also lists /dev/kmem ; I suspect that including this may be 
a

bad idea.


/dev/kmem is a bad idea;  I should go and check what it is using it for 
and if needed we should fix that.



I guess the general thing is that we might want to create another 
default set of devfs rules which include additional nodes we now 
consider safe inside VNET jails;  the jail.conf still needs to know the 
right ruleset to apply, so the jail.conf would need to specify the other 
devfs_ruleset=“..” for vnet jails.  Maybe Jamie could then come up 
with an intelligent solution that would automagically flip things if 
option vnet is set?   I guess jail.conf(5) will need more examples for 
these things as well.



/bz
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: testing 11.0-RC1 vnet jails with ipfilter

[Bjoern A. Zeeb]

On 16 Aug 2016, at 12:47, krad wrote:

is ipfilter supported in vnet jails? Last time I looked and tried pf 
didnt

work (kernel panics), and only ipfw was supported.


In 11-RC* it is present for all 3 firewalls;  like VIMAGE due to memory 
footprint you might have to compile the firewall into the kernel rather 
than kldload it (especially ipfilter).


/bz
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: testing 11.0-RC1 vnet jails with ipfilter

[Bjoern A. Zeeb]

On 15 Aug 2016, at 15:37, Ernie Luzar wrote:


Hello list;

Running 11.0-RC1 with only option vimage compiled into the generic 
kernel.


I can run ipfilter on the host and start vnet jails containing no 
firewalls just fine. But when I try to also have ipfilter run in the 
vnet jail nothing happens. I added this to the vnet jails rc.conf

ipfilter_enable="YES"
ipfilter_rules="/etc/ipf.boot.rules"
ipmon_enable="YES"
ipmon_flags="-Ds"

Then start the vnet jail and its like those ipfilter statements in the 
vnet jails rc.conf are not there. The vnet jails /var/log/messages 
file is not even there. Issuing "ipfstat" inside the running vnet jail 
to display the jails ipfilter rules gives this error message 
"open(IPSTATE_NAME): No such file or directory"
To me this means ipfilter is not running in the vnet jail even though 
I requested it in the vnet jails rc.conf file.


So my question to this list is, has anyone managed to get ipfilter to 
run inside a vnet jail using any of the 11.0 alpha, beta, or rc 
versions? If so would you please share your setup with me?


Maybe I am to close to the bleeding edge for there to be other users 
in the same test loop?



The startup script contains “nojail”.   I think someone opened a bug 
report the other day but I can’t find it anymore;  so the startup 
script won’t automatically run inside a jail.   Can you remove that 
line and try again?



/bz
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Request for VIMAGE testing in 11.0-ALPHA6 and later

[Bjoern A. Zeeb]

Hi,

during the last weeks and months a lot of changes went into the tree to
allow a top-to-bottom network stack teardown to stabilize VNET
shutdown and plug some memory leaks.

In addition some missing parts were virtualised or the virtualisation
was fixed, e.g., pf and ipfilter, ipfw log interface.

I have done some testing and stress testing but it's impossible to catch 
all combinations and setups or even options.  So once 11.0-ALPHA6 is

out please do test (or if you want to do so now r302302 or later).

These changes are only and will only be in FreeBSD 11 for the time being.

You will still need to compile your own kernel;  GENERIC will not have
VIMAGE enabled for 11.0 as that requires at least a performance
analysis (due to extra layer of indirection).   It will also still
print the "experimental" feature line, as we do not want to commit to
KPI/KBI or other things yet and we feel more testing would be good.
I would advise to start testing on dedicated test-systems and not
necessarily production servers but obviously that is your choice.

Also if you are using ports that bring their own ifnet interfaces and
you are experiencing problems please let us know.

If you find problems please file a bug report and make sure to set
"vimage" in the Keywords field but feel also free to post to
freebsd-virtualisation@ which I'll be monitoring.

Thanks a lot to everyone!

Bjoern

--
Bjoern A. Zeeb r15:7
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: jails in different private subnets on the same host

[Bjoern A. Zeeb]

> On 18 May 2016, at 14:00 , Grzegorz Junka  wrote:
> 
> Is it possible to have two jails on the same host each one in a different 
> private subnet, e.g. 192.168.1.0 and 10.33.1.0, and have routing between them 
> working without issues?
> 
> I know it's possible to run jails with IPs in those two subnets but it seems 
> there is no routing and I am not sure if it's because I can't configure my 
> router properly or there is a more fundamental problem. One issue I see is 
> that the jail can't have a different default gateway than the host, and that 
> for now is 192.168.1.1, but I don't see a reason why 10.33.1.0 wouldn't be 
> able to use 192.168.1.1 as it's default gateway provided there is routing 
> between those two subnets.

Given they are both on the same base system host,  both addresses are connected 
locally and thus the kernel knows where to deliver these packets.  If that 
doesn’t work, there is a bug somewhere.

If you want different default gateways then you may want to look into using 
different FIBs for different jails.  See route(8) and jail(8) for parameters to 
set and tune.

/bz

___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Unresponsive jails issues

[Bjoern A. Zeeb]

> On 16 May 2016, at 12:55 , Grzegorz Junka  wrote:
> 
> I have a server running 13 jails for various system services. Recently I 
> added two jails to run simple go applications for testing. They open a 
> network socket and nginx, which is in another jail, and which round robin 
> balances requests to them. I mention that because it may be related, however 
> not necessarily because it was happening earlier.
> 
> The problem is that every 2-3 days jails in my servers stop responding. 
> "jexec jailname tcsh" hangs forever, "service jail stop jailname" hangs 
> forever as well. "top" doesn't show anything suspicious. I can login through 
> SSH to the main server fine. I don't login to jails through SSH so I can't 
> check but it seems that when that happens they stop responding because the 
> services that are running in them stop too (e.g. web server, imap, ...). I 
> tried to "kill -9" the "jexec" process that hangs but that doesn't work.
> 
> My first question is what evidence should I gather when that happens so that 
> I can investigate the issue later on after the server is restarted?
> 
> And the second question, any idea why that might be happening in the first 
> place?
> 
> I am running FreeBSD 10.3 AMD64 updated from 10.2 a couple of weeks ago.

If you can log into the base system and issue commands there;  try to see what 
procstat (-k) thinks about various jailed processes.  You could also check ps 
axl for the WCHAN and see if anything suspicious shows up.

/bz


___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: VNET jails not going away

[Bjoern A. Zeeb]

> On 24 Feb 2016, at 19:20 , Mail Lists  wrote:
> 
> 
> 
> Hi,
> 
> I have the same/similar problem, it's quite annoying, with R10.1 and R10.2:
> jail -r shuts down the (vnet-)jail, jls does not list them anymore, but with 
> jls -d, they are still there - 
> apparently in a 'dying state' ?
> 
> I cannot restart the jail as long as the old jail still appears in 'jls -d'.
> Really annoying.
> 
> But they go away, eventually, sometimes after 5 minutes, sometimes after half 
> an hour or so.

Yeah, if they eventually go away that’s fine.  Hanging around for a bit can be 
expected. That’s, e.g., TCP timeouts from sockets, or similar.

The problem is if they never go away.

Bjoern
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: (VNET) jails not going away

[Bjoern A. Zeeb]

Hi,

sorry for the cross-post, Reply-To: set.


> On 22 Feb 2016, at 13:41 , Bjoern A. Zeeb  
> wrote:
> 
> Hi,
> 
> has anyone else experienced VNET jails to not fully go away anymore on a 
> recent HEAD kernel (or possibly an older kernel)?
> 
> I have test cases with which I can have them in DYING state (see jls -av) for 
> ever or at least more than half a day.   I am in the process of trying to 
> find the cause but would be good to know if anyone else is experiencing this?

Ok, I found more funny behaviour that I can get rid of the previous jail by 
cleaning up the next one.

root@rabbit4:/home/test # jail -i -c -n test19 host.hostname=foo vnet persist
19
root@rabbit4:/home/test # jexec 19 /bin/csh
root@foo:/ # ifconfig lo0 inet 127.19/8
root@foo:/ # exit
root@rabbit4:/home/test # jail -r 19

Jail 19 is in DYING and hangs there forever;  If I repeat this upon exit from 
jail 20, jail 19 will go away.

If I’ll just do this

root@rabbit4:/home/test # jail -i -c -n test20 host.hostname=foo vnet persist
21
root@rabbit4:/home/test # jail -r 21

20 and 21 are going.


I’ll keep tracing this but if it ring a bell for anyone please let me know ;-)



> Thanks,
> Bjoern
> 
> Example (after more than 12 hours of jail -r ..):
> 
> # jls -av
>   JID  Hostname  Path
>Name  State
>CPUSetID
>IP Address(es)
> 1  left.example.net  /
>lef827DYING
>18
> 2  center.example.net/
>mid827DYING
>19
> 3  right.example.net /
>right827  DYING
>20
> 6  right.example.net /
>right923  DYING
>23



— 
Bjoern A. Zeeb  Charles Haddon Spurgeon:
"Friendship is one of the sweetest joys of life.  Many might have failed
 beneath the bitterness of their trial  had they not found a friend."

___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

VNET jails not going away

[Bjoern A. Zeeb]

Hi,

has anyone else experienced VNET jails to not fully go away anymore on a recent 
HEAD kernel (or possibly an older kernel)?

I have test cases with which I can have them in DYING state (see jls -av) for 
ever or at least more than half a day.   I am in the process of trying to find 
the cause but would be good to know if anyone else is experiencing this?


Thanks,
Bjoern

Example (after more than 12 hours of jail -r ..):

# jls -av
   JID  Hostname  Path
Name  State
CPUSetID
IP Address(es)
 1  left.example.net  /
lef827DYING
18
 2  center.example.net/
mid827DYING
19
 3  right.example.net /
right827  DYING
20
 6  right.example.net /
right923  DYING
23
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

VNET teardown changes (part I)

[Bjoern A. Zeeb]

Hi,

sorry for the cross-post;  Reply-To set.

I extracted a patch from projects VNET which tries to get the VNET teardown
more robust (and in a next step plug the remaining [TCP] memory leaks).

If anyone has an interest in testing some parts on a non-production setup
(you have been warned) please do so and report back to me (privately) in case
of success or panics.

There is more to come.

https://people.freebsd.org/~bz/20160222-01-projects-vnets.diff

Thanks,
Bjoern
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: How to implement jail-aware SysV IPC (with my nasty patch)

[Bjoern A. Zeeb]

> On 15 Jun 2015, at 17:10 , kikuc...@uranus.dti.ne.jp wrote:
> 
> On Mon, 15 Jun 2015 09:53:53 +, "Bjoern A. Zeeb" 
>  wrote:
>> Hi,
>> 
>> removed hackers, added virtualization.
>> 
>> 
>>> On 12 Jun 2015, at 01:17 , kikuc...@uranus.dti.ne.jp wrote:
>>> 
>>> Hello,
>>> 
>>> I’m (still) trying to figure out how jail-aware SysV IPC mechanism should 
>>> be.
>> 
>> The best way probably is to finally get the “common” VIMAGE framework into 
>> HEAD to allow easy virtualisation of other services.  That work has been 
>> sitting in perforce for a few years and simply needs updating for sysctls I 
>> think.
>> 
>> Then use that to virtualise things and have a vipc like we have vnets.  The 
>> good news is that you have identified most places and have the cleanup 
>> functions already so it’d be a matter of transforming your changes (assuming 
>> they are correct and working fine; haven’t actually read the patch in 
>> detail;-)  to the different infrastructure.  And that’s the easiest part.
>> 
>> 
>> Bjoern
> 
> Hi Bjoern,
> Thank you for your reply.
> 
> The "common" VIMAGE framework sounds good, I really want it.
> 
> I want to know what the IPC system looks like for user-land after virtualized,
> and what happen if vnet like vipc is implemented.
> 
> For example, jail 1, 2, 3 join vipc group A, and jail 4, 5, 6 join vipc group 
> B ??
> Hmm, it looks good.


That’s not exactly how it works currently and I think the mixing of options 
will be harder and something we’l have to figure out more carefully.
You would be able to say jail 1 has a vipc and jail 2 and 3 and “child jails” 
and inherit it.  (similar for 4 + 5,6) so it’s nested but not side-by-side.

If we want more of the “mixing” and independentness we’ll have to re-think the 
way we “manage” jails.

Bjoern
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: How to implement jail-aware SysV IPC (with my nasty patch)

[Bjoern A. Zeeb]

Hi,

removed hackers, added virtualization.


> On 12 Jun 2015, at 01:17 , kikuc...@uranus.dti.ne.jp wrote:
> 
> Hello,
> 
> I’m (still) trying to figure out how jail-aware SysV IPC mechanism should be.

The best way probably is to finally get the “common” VIMAGE framework into HEAD 
to allow easy virtualisation of other services.  That work has been sitting in 
perforce for a few years and simply needs updating for sysctls I think.

Then use that to virtualise things and have a vipc like we have vnets.  The 
good news is that you have identified most places and have the cleanup 
functions already so it’d be a matter of transforming your changes (assuming 
they are correct and working fine; haven’t actually read the patch in detail;-) 
 to the different infrastructure.  And that’s the easiest part.


Bjoern

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Current state of VIMAGE on 10-STABLE?

[Bjoern A. Zeeb]

> On 26 Mar 2015, at 15:54 , Ernie Luzar  wrote:
> 
> Kai Gallasch wrote:
>> Hi.
>> What is the current state of VIMAGE jails on 10-STABLE?
>> I'm asking, because I saw that Craig Rodrigues and others are working on
>> some long known problems with VIMAGE and there were some related patches
>> committed to the tree.
>> When I experimented with VIMAGE jails on 10.1-RELENG a few weeks ago, I
>> got this when I stopped a running VIMAGE jail.
>> Freed UMA keg (udp_inpcb) was not empty (180 items).  Lost 18 pages of
>> memory.
>> Freed UMA keg (udpcb) was not empty (1757 items). Lost 7 pages
>> of memory.
>> Freed UMA keg (ripcb) was not empty (150 items).  Lost 15
>> pages of memory.
>> hhook_vnet_uninit: hhook_head type=1, id=1 cleanup
>> required
>> hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required
>> Has this already been adressed by recent development?
>> Regards,
>> K.
> 
> This lost memory problem is a long time known bug. 2 - 3 years at least.
> Guess no one has addressed it yet. There are many problems with having a 
> firewall running on host and in the vimage jail at same time.
> 
> Since you say Craig Rodrigues has committed some patches, maybe you should 
> contact him directly and post you finding here for others to read.

All the rest but TCP is my old p4 branch but needs to be carefully reviewed and 
updated due to a couple of years of stack changes.

The one for TCP just emerged as well, and is in reviews (I guess without the 
VIMAGE parts but that will be the easy thing to add if the harder problem is 
indeed solved :-)



— 
Bjoern A. Zeeb  Charles Haddon Spurgeon:
"Friendship is one of the sweetest joys of life.  Many might have failed
 beneath the bitterness of their trial  had they not found a friend."

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: VNET performance

[Bjoern A. Zeeb]

On 10 Jul 2014, at 20:45 , Clint Armstrong  wrote:

> What is the expected network performance of a VNET jail for network
> communication between the jail and the host, or between multiple jails? I
> expected it to approach the 10Gbps of the epair device, but I'm not seeing
> that.
> 
> I see between 800 - 1200 Mbps in standard iperf tests both between the host
> bridge interface and the vnet jail inteface. I see the same poor speeds if
> I make 2 vnet jails and put one side of the epair in each and test between
> them.
> 
> Is the overhead of vnet causing this? Is there anything I can do to improve
> this performance.
> 
> I’ve tested and seen similar performance on 10.0-RELEASE and 11.0-CURRENT.

epair has a netisr queuing in between as you cannot call the input routines 
directly from the output routines.  I was able to get a bit more traffic 
through by doing pinning games.

I wonder what a vale switch for vnets could achieve.

— 
Bjoern A. Zeeb "Come on. Learn, goddamn it.", WarGames, 1983

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: kern/68189 and kern/169751: what jails are allowed to see in a routing socket

[Bjoern A. Zeeb]

On Wed, 2 Jan 2013, Jamie Gritton wrote:

I've been looking at PR kern/169751, which was noting that routing sockets 
don't work inside a jail.  It made the point that setting 
security.jail.socket_unixiproute_only or security.jail.allow_raw_sockets 
didn't help things.  It would seem kind of a given from the "unixiproute" 
name that a route socket ought to work.  And indeed, such a socket is 
permitted to be created in such a jail.  It's just using it that doesn't 
work.


I narrowed this failure down to line 816 of sys/net/rtsock.c, which 
explicitly denies jails from reading routes, regardless of the setting of the 
above two sysctls (or the jail allow.* bits they work with).  And that bit of 
code came about in response to PR kern/68189, which noted that jails could 
see interfaces that aren't theirs (i.e. their address doesn't live on it).


So we have two PRs that are kind of at cross purposes.  It would be nice to 
keep hiding non-jail interfaces from a jail, but it would also be nice to let


jails have no notion of interfaces, only addresses, so by defintiion
there cannot be "non-jail interfaces".


a jailed process know the route to somewhere - at least sometimes.  One 
solution would be to add a much finer layer of control to the jail test in 
rtsock.c, looking at interfaces and seeing if they're somehow connected with 
one of the jail's IP addresses.  But that just seems like a lot of messy 
corner-case code.


Another way around this, and what I'd like to go with if there are no 
objections, is to allow the route sockets to be used by jails that have 
raw_sockets permission.  I know that's kind of a semantic leap, but it seems 
that a jail that has the power of using raw sockets would be able to do 
pretty much as it pleases with routes anyway if it tried hard enough.  Also, 
it would be consistent to allow such operations on jails that aren't 
IP-restricted, or in VIMAGE jails.


I have not further looked at the code but the answer is that we should
not further complicate jails after 14 years when we have a perfect
good solution for the problem;  vnets are there for exactly this.
Someone with enough interest and time should just finish things (tm) ;-)

Meanwhile your suggestion might be ok given simple enough, but I wonder
if a different flag would be helpful still.  I would not be able to
"trust" (the little that is possible anyway) raw_sockets anymore if they
suddently could fiddle with the routing table - even read-only, should
that really be enough.
I would explicitly advertise it as 'do not use - will go away again'
feature and it should the moment vnets are declared non-experimental.

Just my 2cts.

/bz

--
Bjoern A. Zeeb You have to have visions!
 Stop bit received. Insert coin for new address family.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: IPv6 multicast sent to jail

[Bjoern A. Zeeb]

On Wed, 5 Sep 2012, Curtis Villamizar wrote:



In message 
"Bjoern A. Zeeb" writes:


On Sat, 25 Aug 2012, Jamie Gritton wrote:

...

Curtis


Offhand, it does sound like a bug. I imagine the solution would be to
reject the join - at least the easy solution to be done first until
something more complicated can be done to make jails play nice with
multicast.

- Jamie



Jamie,

Certainly not the preferred solution.  Best would be a
jail.allow-ipv6multicast sysctl variable with rejecting the join if 0
and accepting the join and passing in multicast if 1.  Same for v4,
though not of immediate concern since DHCPv4 doesn't need it.

If you (or someone) would like to point me in the right direction, I
would be willing to put some time into learning the relevant code and
proposing a fix.  No promises, but I can put some time into it.  Off
list if you prefer.

Curtis


It'll have to be someone besides me - I don't know enough about
multicast myself to be able to do more than keep it out of jails.


sysctl souns bad to me;  I think it should actually be grouped by
ip4.* and ip6.*.  What dod we currently do for raw sockets?  Can we
have a third level easily, as in ip4.raw.*, ip6.mc.*, ...  which of
course would kill the classic "allow" thing for raw sockets myabe?

/bz


For raw sockets the sysctl variable is:

security.jail.allow_raw_sockets

One sysctl variable for both inet and inet6 AF.  Perhaps a reasonable
name would be:

 security.jail.ip4.allow_multicast
 security.jail.ip6.allow_multicast

Just to be clear, I was hoping to get some help if I were to make an
attempt to allow ipv6 multicast through, though I suspect that the
code would be very similar for ipv4.


The sysctls are mostly not relevant anymore but yes, if we can get
these options we can look at the code.  Defaults to off.
I might be able to help on the v6 trailing end.  Jamie could you
prepare the jail options changes for us?

--
Bjoern A. Zeeb You have to have visions!
 Stop bit received. Insert coin for new address family.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Fixed Jail ID for ZFS -> need proper mgmt?

[Bjoern A. Zeeb]

On Tue, 4 Sep 2012, Jamie Gritton wrote:


It's true that a jail left in the DYING state can't be re-created
normally. But it can with the "-d" flag or the "allow.dying" parameter.
In that case, an existing but dying jail will be re-attached to and this
resurrected. So it can be gotten around, and would be a matter of
education. Or perhaps we could change the default behavior to silently
all re-creation of dying jails. Is there any harm in this? I.e. would
there be any difference noticeable to the user if a jail was created
with some old TCP connections attached to it?


Yes, really bad and TCP is not the only thing in theory.  Assume
your management does not make sure the same users gets the same jail;
you elak a lot of (possibly security related) information.  Would also
make it quite hard in terms of auditing etc. to get this right unless
done knowingly and on purpose.

--
Bjoern A. Zeeb You have to have visions!
 Stop bit received. Insert coin for new address family.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Fixed Jail ID for ZFS -> need proper mgmt?

[Bjoern A. Zeeb]

On Tue, 4 Sep 2012, Jamie Gritton wrote:


Names are unique. And we don't have the dying-jail problem with them, as
creating a jail with the same name as a dying jail is allowed. OK, that
means that jail names aren't quite unique - but they're at least unique
among the non-dying set.


Perfect; all we need.

--
Bjoern A. Zeeb You have to have visions!
 Stop bit received. Insert coin for new address family.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Fixed Jail ID for ZFS -> need proper mgmt?

[Bjoern A. Zeeb]

On Tue, 4 Sep 2012, Pawel Jakub Dawidek wrote:


On Tue, Sep 04, 2012 at 11:33:06AM +0200, Martin Matuska wrote:

On 4. 9. 2012 10:55, Bjoern A. Zeeb wrote:

2) in the case of (1) it should be possible to address jails by name
   as ZFS would be handled automatically and we would not need another
   unique identifier I guess?
   Otherwise I'd prefer for people to be able to delegate ZFS datasets
   to jail names (as well), as long as they are uniquely identifyable
   (i.e. there are no 17 jails running with a name of "filesever").


The binding of a ZFS dataset to a jail has to be exact. So we end up
with id's.
But we could add something like "zfs datasets" to the jail's
configuration file. The jail command would then simply exec "zfs jail
jailid dataset" for each of the datasets on jail creation right before
initiating rc startup and "zfs unjail jailid dataset" for each of the
datasets after jail's rc shutdown but before the jail is destroyed.


Datasets shall not be unjailed. Jailing dataset means that it won't be
mounted in the main system. You need to run 'zfs mount -a' within a
jail, during it start-up to mount its datasets. This is much safer than
mounting anything in jail's directory tree from the main system. We
already had security issues because of that. This is also how it works
in Solaris/IllumOS with zones.

And I can't resist to remind how opposed I was to jail ids in the first
place. Especially because they were dynamically allocated. When they
were introduced I recommended jail names, which we ended up with anyway,
but now we have all this jailid thing to manage in older FreeBSD
versions.

All in all we should move to using jail names, IMHO, the same way zone
names are used in Solaris/IllumOS. When I was adding jail support to ZFS
there were no jail names, only jail hostnames, which weren't an option
really.


I guess we'd need to end up with name and if not uniqe + ID or
something?  Really IDs are not the problem as long as they never
appear anywhere in the config file?  Just not sure given names are not
unique how to handle it the right way?

/bz

--
Bjoern A. Zeeb You have to have visions!
 Stop bit received. Insert coin for new address family.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Fixed Jail ID for ZFS -> need proper mgmt?

[Bjoern A. Zeeb]

Hi,

I had been talking to someone about jail management and it turns out
people are using jail jid=42 to always have a fixed jail ID.  The
reason as I understood is that ZFS datasets are associated by jail id
for delegation? [I admit having no clue about the ZFS side]

If this is true I feel it's a very bad idea as it makes restarting
jails a lot harder in case they remain DYING for say a not fully
closed TCP session.

My memories are: jid are still unique and cannot be re-used, even if
in DYING, names can be re-used and thus are not neccessarily unique.
Jamie, can you confirm this?

Seems we need to sort out one to two problems:

1) can we make sure that the jail management framework can address a
   ZFS dataset for delegation somehow and automatically do that as
   part of the startup?

2) in the case of (1) it should be possible to address jails by name
   as ZFS would be handled automatically and we would not need another
   unique identifier I guess?
   Otherwise I'd prefer for people to be able to delegate ZFS datasets
   to jail names (as well), as long as they are uniquely identifyable
   (i.e. there are no 17 jails running with a name of "filesever").

Do we have documentation for the ZFS features in the man pages or
elsewhere btw?   If not we should add it.

Does this make sense?

/bz

--
Bjoern A. Zeeb You have to have visions!
 Stop bit received. Insert coin for new address family.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: IPv6 multicast sent to jail

[Bjoern A. Zeeb]

On Sat, 25 Aug 2012, Jamie Gritton wrote:

...

Curtis


Offhand, it does sound like a bug. I imagine the solution would be to
reject the join - at least the easy solution to be done first until
something more complicated can be done to make jails play nice with
multicast.

- Jamie



Jamie,

Certainly not the preferred solution.  Best would be a
jail.allow-ipv6multicast sysctl variable with rejecting the join if 0
and accepting the join and passing in multicast if 1.  Same for v4,
though not of immediate concern since DHCPv4 doesn't need it.

If you (or someone) would like to point me in the right direction, I
would be willing to put some time into learning the relevant code and
proposing a fix.  No promises, but I can put some time into it.  Off
list if you prefer.

Curtis


It'll have to be someone besides me - I don't know enough about
multicast myself to be able to do more than keep it out of jails.


sysctl souns bad to me;  I think it should actually be grouped by
ip4.* and ip6.*.  What dod we currently do for raw sockets?  Can we
have a third level easily, as in ip4.raw.*, ip6.mc.*, ...  which of
course would kill the classic "allow" thing for raw sockets myabe?

/bz

--
Bjoern A. Zeeb You have to have visions!
 Stop bit received. Insert coin for new address family.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: [patch] etc/rc.d/jail: allow extra parameters for each jails

[Bjoern A. Zeeb]

On Thu, 16 Aug 2012, Jun Kuriyama wrote:



Hi,

Here is a patch which I'm using for years in my production
environment.

I usually changes parameters documented in jail(8) for each jails, but
current rc.d/jail has no feature to pass extra parameters at starting
jails.

I hope this patch will not prevent jamie's recent/next work for
rc.d/jail.  I'll commit this if there is no objection.


Why not just use his work?

--
Bjoern A. Zeeb You have to have visions!
 Stop bit received. Insert coin for new address family.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: [jail] Allowing root privledged users to renice

[Bjoern A. Zeeb]

On 25. May 2012, at 16:48 , Sean Bruno wrote:

> I've been toying with the idea of letting jails renice processes ... how
> dangerous and/or stupid is this idea?
> 
>  //depot/yahoo/ybsd_9/src/sys/kern/kern_jail.c#5 -
> /home/seanbru/ybsd_9/src/sys/kern/kern_jail.c 
> 270a271,275
> + int   jail_allow_renice = 0;
> + SYSCTL_INT(_security_jail, OID_AUTO, allow_renice, CTLFLAG_RW,
> +&jail_allow_renice, 0,
> +"Prison root can renice processes");
> 
> 3857a3863,3865
> +  case PRIV_SCHED_SETPRIORITY:
> +  if (!jail_allow_renice)
> +   return (EPERM);


I think sysctls are a bad idea given jails have per-jail flags these days.

Maybe also only allow re-nicing to be nicer but not less nice?

/bz

-- 
Bjoern A. Zeeb You have to have visions!
   It does not matter how good you are. It matters what good you do!

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: New jail(8) committed

[Bjoern A. Zeeb]

On 26. Apr 2012, at 20:07 , Jamie Gritton wrote:

Hi,

> I've finally put my jail(8) changes into HEAD.

I meant to say this yesterday already but time flies.  *YEAH!!!*  Thanks a lot.

/bz

-- 
Bjoern A. Zeeb You have to have visions!
   It does not matter how good you are. It matters what good you do!

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: New jail(8) committed

[Bjoern A. Zeeb]

On 27. Apr 2012, at 14:33 , Christer Solskogen wrote:

> On 26/4/2012 10:07 PM, Jamie Gritton wrote:
> 
>> There's still more that I want to do with this, but it's time it was
>> part of current.
>> 
> 
> I've got an idea that you might find useful for future versions of jail. I 
> have created a wrapper script for my own use where I instead of creating a 
> jail with buildworld/installworld, I rather nullmount /bin, /lib, etc. to a 
> jailroot, and run mergemaster on that jailroot. This makes the jail very 
> small (3MB IIRC) and it makes upgrades of the jails very easy.

Yeah, there's other doing this;  ezjail is close to this as well.

But please not that management (configuring, starting, stopping) and 
provisioning can easily be separated and the latter has nothing to do with the 
former in first place.  I hope that ezjail will pick up supporting the new 
config stuff and use that while still providing its own provisioning.

/bz

-- 
Bjoern A. Zeeb You have to have visions!
   It does not matter how good you are. It matters what good you do!

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: intra-jail communication slow

[Bjoern A. Zeeb]

On 15. Apr 2012, at 19:28 , Beeblebrox wrote:

> I would like to know if this is normal for jails:
> 1. 3 different Jails for mysql , http, tinderbox.
> 2. No pf / nat running for the time being
> 3. My ifconfig for jails:
>   inet 192.168.2.1 netmask 0xff00 broadcast 192.168.2.255
> (tinderbox, dhcp, tftp)
>   inet 192.168.2.100 netmask 0x broadcast 192.168.2.100  (http)
>   inet 192.168.2.101 netmask 0x broadcast 192.168.2.101  (mysql)
> 
> Problem: Jails seem to be communicating with host and with each-other
> at an observably slower rate, compared to when all services are
> host-based.

Which communications and how much slower?


> Is this normal for jails or is my netconfig problematic in some way?

In theory not, but in practise please notice that for example with mysql you 
may switch from unix domain sockets (usually used in "localhost" case) to 
normal inet socket communications etc. which may affect your overall feeling of 
"speed".

Knowing details and numbers etc might help here to rule out a jail specific 
problem.

/bz

-- 
Bjoern A. Zeeb You have to have visions!
   It does not matter how good you are. It matters what good you do!

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Jail source address selection broken, patch for ping

[Bjoern A. Zeeb]

On 9. Apr 2012, at 20:11 , Mark Felder wrote:

> On Mon, 09 Apr 2012 14:16:47 -0500, Juan F. Díaz y Díaz  
> wrote:
> 
>> Mark, you can just run a jail with the setfib utility so you don't need to 
>> modify all your scripts.
> 
> I don't think anyone here is understanding the issue and forcing a routing 
> table will not help.

yeah you would need a dedicated FIB per VLAN and then still have the problem 
that a single program like fping would try to reach a node in each VLAN and 
would have to switch FIBs in between and everything; that would require more 
patching of code.   It would be different if it was a VLAN per jail in which 
case you'd probably not have the problem in first place;)

/bz

-- 
Bjoern A. Zeeb You have to have visions!
   It does not matter how good you are. It matters what good you do!

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Jail source address selection broken, patch for ping

[Bjoern A. Zeeb]

On 9. Apr 2012, at 16:20 , Mark Felder wrote:

Hi Mark,

thanks a lot for posting the summary.

> By pure chance I was able to contact bz@ and he provided me with a patch for 
> ping based on his recent work on a similar issue with traceroute. This solved 
> my problem with the system ping utility, but my tests with fping and the ping 
> utility included with our monitoring software still exhibited the same issue.
> 
> bz informed me that he believes he knows where the bug is in the kernel -- I 
> believe he pointed me to the area of sys/netinet/ip_raw.c around line 461. 
> Jails are getting the first IP as a source no matter what.

And maybe to confirm - yes I have told a lot of people in the past to try 
telnet or similar thing as "ping" was special, as it's raw sockets etc.  In 
case you have a PR open about this issue please email me the PR number directly 
(not Cc:ing the list) or ask some FreeBSD committer to assign it to me.

As I had originally left the comment there when committed the multi-IP jail 
source code (or follow-up) and the grief this seems to regularly cause, I will 
try to get it fixed soon:  
http://svnweb.freebsd.org/base/head/sys/netinet/raw_ip.c?annotate=229265#l461

> Anyway, attached is the patch he asked me to post to the mailing list for 
> those that need a workaround for ping. I'm sure fixing this in the kernel 
> will probably require further discussion among those with actual programming 
> skills :-)

It's also available here but it's considered a work-around and prove of concept 
that this really was the issue:
http://people.freebsd.org/~bz/20120407-01-ping-source-addr.diff

/bz

-- 
Bjoern A. Zeeb You have to have visions!
   It does not matter how good you are. It matters what good you do!

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: kern/165769: [rc][jai][ipv6] IPv6 Initialization on external iface is too slow for jail

[Bjoern A. Zeeb]

On 12. Mar 2012, at 12:01 , Oleg Ginzburg wrote:

> Здравствуйте,
> 
> On Monday 12 March 2012 15:27:32 b...@freebsd.org wrote:
>> This has nothing to do with jails, just with interfaces, possibly
>> switch ports and spanning tree.
> 
> /etc/rc.d/netwait the script concerns only in a case "complete"-type jail, 
> not 
> for "service"-type as is written in my example

I don't see a difference but I see what you mean with "on start" which I treated
as "on boot".  The answer unfortunately is - DAD is not supposed to take that
long that it would matter, so the NIC drivers are probably silly or are required
to do silly things.  I wonder if you might even see a DOWN/UP cycle.

The workaround you can apply is to use the prestart option to the jail to add 
that
sleep 1 you need and let the startup script configure the IP addresses for you.
See jail_exec_prestart in man 5 rc.conf.


/bz

-- 
Bjoern A. Zeeb You have to have visions!
   It does not matter how good you are. It matters what good you do!

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Practical limit to number of jails on a given host?

[Bjoern A. Zeeb]

On 6. Feb 2012, at 20:29 , Doug Barton wrote:

> Howdy,
> 
> Thinking about implementing a poor-man's virtualization solution with
> lots'o'jails, and wondering what people think about the practical limits
> of such a system. I realize that part of the answer is going to depend
> on CPU and RAM, so let's assume for the sake of argument that the answer
> to that bit is, "Lots of both."
> 
> So first question is, is there some sort of hard-coded limit somewhere?
> If not, what is the largest number of jails that you've created
> successfully/reliably on a system, and what are the specs for that system?


Yes, jails provide you 6 9s ... though that's not 99.% but 99 is
the maximum number of jails.  And yes, I have started this many before --
without processes or anything.
It took a couple of days, due to some list handling, which could be
optimized.  You will find that once you get there, you'll have a syscall
which never returns...
You notice once the start loop slows down if you print a . every 100 or 1000.

The machine was a 4 or 8 core amd64 with 8G of memory.

I think I had a slide in
there:  
http://www.bsdcan.org/2010/schedule/attachments/130_2010-bz-the-new-vvorld.pdf

I know if using vnets; you can get the 4k (or more) but I also have reports
that it may not scale.

The other limit you'll run into is the number of PIDs.

And eventually scheduling depending on what you want to do.


> And finally, has anyone run into trouble with a large number of IP
> addresses for the jails? ISTR that way back when, the IP addresses
> associated with a particular interface were stored in a linked list, so
> as you added more you would start seeing O(N) slowdown on a lot of
> network stuff in the kernel.

Yeah, we still do list walks here and there.

/bz

-- 
Bjoern A. Zeeb You have to have visions!
   It does not matter how good you are. It matters what good you do!

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: * Re: * Re: Getting Jail v2 working with 9-stable

[Bjoern A. Zeeb]

On 23. Jan 2012, at 16:26 , Devin Teske wrote:

> On Jan 23, 2012, at 3:36 AM, Denny Schierz  wrote:
> 
>> hi,
>> 
>> Am 20.01.2012 um 15:05 schrieb Devin Teske:
>> 
>>> Try my vimage rc.d script for this.
>>> 
>>> http://druidbsd.sf.net/vimage.html
>>> 
>>> http://druidbsd.sourceforge.net/download/vimage-1.4.tbz
>> 
>> I tried it, but doesn't work. I think, it must be a bug in the SPARC Kernel, 
>> because other with I386 and same options  doesn't have the same problems.
>> 
> 
> I don't know if VIMAGE is supported yet on SPARC platform. Maybe someone 
> wants to chime in that's more familiar with which-platforms VIMAGE is 
> supported.

VIMAGE should be arch independent.

-- 
Bjoern A. Zeeb You have to have visions!
   It does not matter how good you are. It matters what good you do!
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: mtr doesn't work in a jail even with security.jail.allow_raw_sockets: 1

[Bjoern A. Zeeb]

On 17. Jan 2012, at 20:41 , Michal Vančo wrote:

> Try with --address option. Address selection doesn't work when mtr is run 
> within jail.

I should or should be fixed.

>> Wanted to use mtr to diagnose an issue in a jail
>> but it seems it totally fails even with
>> security.jail.allow_raw_sockets: 1

which version of freebsd?  Anything newer than incl. 8.0 the systls are not 
what you want anymore; it's per jail flags.

/bz

-- 
Bjoern A. Zeeb You have to have visions!
   It does not matter how good you are. It matters what good you do!

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Passing additional options to jail(8) via rc.conf

[Bjoern A. Zeeb]

On Sep 20, 2011, at 5:25 PM, Moritz Wilhelmy wrote:

> On Tue, Sep 20, 2011 at 16:54:33 +0000, Bjoern A. Zeeb wrote:
>> On Sep 20, 2011, at 3:21 PM, Moritz Wilhelmy wrote:
>> 
>>> Please keep me in CC, I am not subscribed to freebsd-jail.
>> 
>> Which is your problem as the real solution is being discussed there and
>> is really looking for more eyes.  Search for Jamie's posts in the list
>> archive.
> 
> I wasn't subscribed to freebsd-questions either, back in July. I had
> some FreeBSDs running before, but I really wasn't reading any mailing
> lists. I thought I should ask -questions first, because it is the most
> busy one, with all the chatter and noise. Guess I will either patch
> jail(8) or do it via rc.local on 8.2 and wait for the release of 9,
> which seems to be quite an improvement in general. I'm also not that
> actively involved in FreeBSD, and I try not to subscribe too many
> mailing lists in general. I thought the PR was the only thing there was,
> and google tricked me into believing this. Maybe someone could close the
> PR in case this is resolved.
> 
> My apologies. Thanks to Jamie for adding support for the config file.

It's not in 9 for 9.0.  But we'd appreciate if you'd give it a try:)
If you have questions, posting them to freebsd-jail should give you help.

Bjoern

-- 
Bjoern A. Zeeb You have to have visions!
 Stop bit received. Insert coin for new address family.

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Passing additional options to jail(8) via rc.conf

[Bjoern A. Zeeb]

On Sep 20, 2011, at 3:21 PM, Moritz Wilhelmy wrote:

> Please keep me in CC, I am not subscribed to freebsd-jail.

Which is your problem as the real solution is being discussed there and
is really looking for more eyes.  Search for Jamie's posts in the list
archive.

-- 
Bjoern A. Zeeb You have to have visions!
 Stop bit received. Insert coin for new address family.

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: ip settings not work for jail 'service' type

[Bjoern A. Zeeb]

On Jul 20, 2011, at 10:03 PM, Subbsd wrote:

> Hi
> 
> Jail configuration with "complete” works perfectly with IP4, IP6 and
> their secondary.
> But if try jailed 'service' - it does not work. For example:
> 
> 1) have uncomment:
> ftp stream  tcp nowait  root/usr/libexec/ftpd   ftpd -l
> 
> 2) have jail cfg like:
> jail_enable="YES"
> jail_list="example" # Space separated list of names of jails
> jail_example_rootdir="/"
> jail_example_hostname="${hostname}"
> jail_example_interface="lo0"
> jail_example_ip="192.0.2.10"
> #jail_example_ip_multi0="10.0.0.1" # secondary - shouldn't work?
> sockstat show *:21
> jail_example_exec_start="service inetd onestart"
> jail_example_exec_stop="service inetd onestop"
> 
> 3)
> service jail start example

And does inetd run at that point?  Did it log anything to your logfiles?


> 
> 4)
> ftp 192.0.2.10
> (no connection)
> 
> I somewhere was mistaken or it shouldn't work? Thanks
> ___________
> freebsd-jail@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-jail
> To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

-- 
Bjoern A. Zeeb You have to have visions!
 Stop bit received. Insert coin for new address family.

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: pf or ipfw within a jail?

[Bjoern A. Zeeb]

On May 6, 2011, at 8:28 PM, Mickey Harvey wrote:

> Is it possible to run pf or ipfw within a jail? I am running 8.2 and have
> vimage compiled in the kernel.

ipfw might work then; pf not yet. ipfilter in a far distant future.

-- 
Bjoern A. Zeeb You have to have visions!
 Stop bit received. Insert coin for new address family.

___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: ipv6 loopback behaviour inside jail

[Bjoern A. Zeeb]

On Wed, 30 Mar 2011, Rob Evers wrote:


P.S. I can supply any further information needed.


Which verison of FreeBSD are you running?

--
Bjoern A. Zeeb You have to have visions!
 Stop bit received. Insert coin for new address family.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: FTPD not working properly on jail

[Bjoern A. Zeeb]

On Wed, 8 Dec 2010, Redd Vinylene wrote:


On Wed, Dec 8, 2010 at 4:52 PM, Bjoern A. Zeeb <
bzeeb-li...@lists.zabbadoz.net> wrote:



a) have you tried without SSL?
b) have you tried ftpd from base?

It pretty much smells like a bug in vsftpd.

Out of curiosity - which version of freebsd is that?

/bz



Greetings!

a) Good question. Just tested - without a doubt, yes it works perfectly
without SSL. But we need SSL though.


application or configuration issue unrelated to jails.

--
Bjoern A. Zeeb  Welcome a new stage of life.
 Going to jail sucks --  All my daemons like it!
  http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: FTPD not working properly on jail

[Bjoern A. Zeeb]

On Wed, 8 Dec 2010, Redd Vinylene wrote:


Hello!

I'm trying to set up a virtual vsftpd-ssl-2.3.2 server so my band can share
new tracks, production material or what not, but I'm getting all kinds of
strange errors:

http://pastie.org/1358536

Anybody know why? The server runs on a jail. There are no firewalls on
either the host or the jail. I've tried other ftpd's and gotten similar
results so I don't think it's vsftpd there's something wrong with here.


a) have you tried without SSL?
b) have you tried ftpd from base?

It pretty much smells like a bug in vsftpd.

Out of curiosity - which version of freebsd is that?

/bz

--
Bjoern A. Zeeb  Welcome a new stage of life.
 Going to jail sucks --  All my daemons like it!
  http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: loopback in jail

[Bjoern A. Zeeb]

On Thu, 11 Nov 2010, Andrei Kolu wrote:


OK, I tried this way:

ezjail config:
export jail_crashtest_ip="194.xxx.yyy.22,127.0.0.1"

I have loopback ip address now:
lo0: flags=8049 metric 0 mtu 16384
   options=3
   inet 127.0.0.1 netmask 0xff00

But, can't bind anything to 127.0.0.1 anyway. Is this a bug or something?


No, it's intentional. You would bind to your public 194.x.x.x IP.

Quoting from jail(2):

 All connec-
 tions to/from the loopback address (127.0.0.1 for IPv4, ::1 for IPv6)
 will be changed to be to/from the primary address of the jail for the
 given address family.

/bz

--
Bjoern A. Zeeb  Welcome a new stage of life.
 Going to jail sucks --  All my daemons like it!
  http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Mutiple ipv4 and ipv6.

[Bjoern A. Zeeb]

On Mon, 1 Nov 2010, Peter Ankerstål wrote:

Hi,


Im trying to configure a jail with mutiple ipv6 and ipv4-addresses. One 
ipv4-address and multiple ipv6
works fine but not multiple of both.

...

and others:
Starting jails:/etc/rc.d/jail: WARNING: jail_extract_address: type not 
identified
/etc/rc.d/jail: WARNING: jail_extract_address: type not identified
cannot start jail "jailid":
.


can you give us the complete global and configuration for this jail?
It's hard to guess at the moment.

Subsitute the domain name for example.com, the IPv4 prefix with
192.0.2. and the IPv6 prefix with 2001:db8: (which are the
example/docmentation prefixes) or, if you want, send them to me
privately.

/bz

--
Bjoern A. Zeeb  Welcome a new stage of life.
 Going to jail sucks --  All my daemons like it!
  http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: sysvipc in jails + CURRENT

[Bjoern A. Zeeb]

On Thu, 22 Jul 2010, Isaac Levy wrote:

Hi ike,

long time no see.


I could be doing something stupid, or I've dug up an old bug, =
(http://www.mail-archive.com/freebsd-jail@freebsd.org/msg00859.html).

I cannot get good ol' trusty enforce_statfs to work, allowing me to see =
different mounts from within a jail.

--
The example jail command I'm using, (new-style),
 jail -c path=3D$JDIR host.hostname=3D$JHOSTNAME ip4.addr=3D"$INET" =
enforce_statfs=3D1 command=3D/bin/sh /etc/rc

I've tried everything- including attempting to change my sysctls over =
and over, (including /etc/sysctl.conf with rebooting).
Interestingly:
The old standard 'security.jail.enforce_statfs' was not something I =
could modify, *until* I put a sysctl value in /etc/sysctl.conf which was =
not 0 (1 or 2 both will let me set the sysctl value once the system is =
booted).
If I have "security.jail.enforce_statfs=3D0", to my surprise, I cannot =
change that sysctl on the host system as I would usually expect.
(This is what makes me think this smells like a bug)

My extra mounts are UFS volumes, mounted right into the jail directory, =
(on another ufs volume).

What follows, are just machine stats if anyone wants them?

I'd love any thoughts, urls, no matter how brief...


I am confused but maybe I can help you with some explanation:

1) do not change the sysctl anywhere; that is neither in sysctl.conf
   nor by other magic or by hand.   The default on 8 and 9 should be
   2.  You can check that with sysctl security.jail.enforce_statfs
   still I think.

2) Creating a new jail
> jail -c path=/jail/j1 persist
   I can see:
> jexec 1 mount
192.168.5.1:/zoo/bz/HEAD on / (nfs)
   And
> jls -s -j 1 enforce_statfs
enforce_statfs=2
   confirms the default.

3) modifying the jail:
> jail -m jid=1 enforce_statfs=1
   I can now see:
> jexec 1 mount
192.168.5.1:/zoo/bz/HEAD on / (nfs)
devfs on /dev (devfs, local, multilabel)
192.168.5.1:/zoo/bz on /zoo/bz (nfs)
   And jls confirms that the modfication was successful:
> jls -s -j 1 enforce_statfs
enforce_statfs=1

4) If you lower the default by changing the sysctl, all your jails
   that have a higher level will be lowered as well.

5) But if you up the default again, they won't change back up.


I think that you are right, that there is a bug here, as 4) and 5)
should be working the other way round I think.


Anyway, the summary is: if you don't change the default a
    jail -c enforce_statfs=1 ...
should just work fine.


Hope this helps.

/bz

--
Bjoern A. Zeeb   This signature is about you not me.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: selective jail restriction controlling in rc.conf

[Bjoern A. Zeeb]

On Sat, 3 Jul 2010, Harald Schmalzbauer wrote:

Hallo Harald,


Harald Schmalzbauer schrieb am 03.07.2010 10:05 (localtime):
...
One have to seperatly define ip4 and ip6 addresses. The can be with or 
without mask, single oder comma seperated list, doesn't matter, thanks to 
the jail_handle_ips_option() coder, it just works :)


I forgot to change that in defults/rc.conf.
Please find attached the corrected version.


there is currently an ongoing discussion about jail configuration on
the freebsd-jail@ mailing list:

http://lists.freebsd.org/pipermail/freebsd-jail/2010-June/thread.html#1308

I think your comments (and patches) are better sent there, rather than
to sta...@.


Gruesse,
Bjern

--
Bjoern A. ZeebFrom August on I will have a life.  It's now up to you
to do the maths and count to 64. -- Bondorf, Germany, 14th June 2010
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Thoughts on jail.config

[Bjoern A. Zeeb]

On Mon, 28 Jun 2010, Jamie Gritton wrote:

Hi,


On 06/28/10 08:41, Rodrigo Mosconi wrote:


An idea: if it works like a "jaild"? A daemon management the start-up,
shutdown, console redirection?  All the admins task could be done by a
"jailctl"?


I don't know what work a daemon would have to do. I only see it running
tasks on startup, and then waiting until something tells it on shutdown
to wake up and stop the jails. That "something" would have to be that
jailctl you mention. If there's a jail program running anyway, might as
well keep all functionality in that one program.



One functionality I forgot about but was asked for in the past was
"jail reboot"  so that an admin could "restart" a jail completly from
within the jail.  The question is whether we may want a "jailinit" (an
init running inside the jail) for that or if we want to handle it from
the outside.

/bz

--
Bjoern A. ZeebFrom August on I will have a life.  It's now up to you
to do the maths and count to 64. -- Bondorf, Germany, 14th June 2010
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail starts on boot

[Bjoern A. Zeeb]

The following reply was made to PR kern/147162; it has been noted by GNATS.

From: "Bjoern A. Zeeb" 
To: bug-follo...@freebsd.org, tom.dewa...@abvv.be
Cc:  
Subject: Re: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail
 starts on boot
Date: Thu, 3 Jun 2010 14:39:59 + (UTC)

 Hi,
 
 the only kernel changes from 8.0p2 to 8.0p3 are related to NFS imho.
 You are not also using NFS with this machine?
 
 If you are not, this is likely due to some changed timing or similar
 and you are just hitting  generel problem that most likely is entirely
 unrelated to jails and would either be a pf@ or a net@ kernel issue.
 
 /bz
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Panic on 8-STABLE in pfctl with options VIMAGE on a DELL PowerEdge R300 (bge)

[Bjoern A. Zeeb]

On Fri, 26 Feb 2010, Lorenzo Perone wrote:

Hi,

While I was just planning to experiment with VIMAGE, and it is not required 
for production (I'm aware of the message of it being experimental...), I 
thought it might be useful to report it. Please send me a note if I should 
file a pr.


The panic does not occur with the same kernel compiled without options 
VIMAGE.


FAQ from virtualization@ ; pf support for VIMAGE only basically exists
here:  http://svn.freebsd.org/base/user/eri/pf45/head/
but is not fully ready either.

/bz

--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: configuration of multiple IPs for a jail

[Bjoern A. Zeeb]

On Thu, 28 Jan 2010, t...@diogunix.com wrote:

Hi,


Jase,


This behaviour has been addressed in RELENG_7 recently with r202924 [1].


thank you very much. That's what I was watching out for :-).
I somehow could not find that hint in all the resources I used.


This commit allows you to set : sysctl security.jail.ip4_saddrsel 0 ,
which makes the kernel use the first IP passed to jail (8) as the
default source address instead of the default behaviour which picks the
first matching ip for that jail on the interface.


That's not exactly true.  Source address uses the first "matching"
address for the destination on the outgoing interface if possible.
There is a route lookup involved as well.  So if you are serving more
than one subnet it won't necessarily be the first IP of the interface
seen within the jail.

For the case given, it most likely will, though.



Just great. I run 7.2 stable on most machines and thanks to your information
it will be much easier than what I meanwhile did to fix things.


A workaround (if you're not able to update to a RELENG_7 following that
commit) is to reorder your interface aliases in /etc/rc.conf ,so that
your primary jail ip has a lower alias # than any secondary ips for that
jail.


Yes. I've meanwhile found exactly that out the hard way and by trial and
error. Works nice (or however, it works), even when the kernel setting method
of course is much more elegant.


Hope this helps,


I did already.


Though it might help, if you only need it for postfix, using the
smtp_bind_address (and smtp_bind_address6) options might be more
elegant rather than using the hammer of forcing things in the kernel.
See man 5 postconf.

If more services across all jails should be using the intended
behavior using the sysctl and kernel is probably the right thing.

/bz

--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: conf/142972: [jail] [patch] Support JAILv2 and vnet in rc.d/jail

[Bjoern A. Zeeb]

On Mon, 25 Jan 2010, b...@freebsd.org wrote:

Hi,


Synopsis: [jail] [patch] Support JAILv2 and vnet in rc.d/jail

State-Changed-From-To: open->suspended
State-Changed-By: bz
State-Changed-When: Mon Jan 25 11:12:44 UTC 2010
State-Changed-Why:
As was said multiple times before, it is very unlikely that
the current rc script will be changed for the experimental
feature and a more complete mgmt solution is being sought of
for the final support.

http://www.freebsd.org/cgi/query-pr.cgi?pr=142972


that said again, I'll try to get the people involved (poked via Bcc:)
to post a draft about a possible framework here, so we can discuss all
the features, formats, needs, ... everyone has and concentrate on the
final soultion rather than working on hacks on top of hacks that have
long gotton to the point that they are not a feasible anymore.

/bz

--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: kern/142341: [jail] Jail escape when cwd is moved from the host system

[Bjoern A. Zeeb]

The following reply was made to PR kern/142341; it has been noted by GNATS.

From: "Bjoern A. Zeeb" 
To: bug-follo...@freebsd.org, ve...@kajtaz.net
Cc:  
Subject: Re: kern/142341: [jail] Jail escape when cwd is moved from the host
 system
Date: Tue, 5 Jan 2010 19:36:36 + (UTC)

 Hi,
 
 this is the expected behaviour but is probably not explicitly
 documented.  Patches to update the man page are welcome.
 
 -- 
 Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Jail on 2 interfaces?

[Bjoern A. Zeeb]

On Wed, 23 Dec 2009, Mel Flynn wrote:


or later; no official FreeBSD version before had supported
multiple-IPs with a jail.


8.0-p3, yes.


ok


What it did was what you were asking for.  That's the problem.

1) either use ifconfig
2) or use jail + interfaces
3) but do not mix them (especially not overlapping)

So I would suggest to do it like this:

# Base system IPs.
ifconfig_bge0="inet 192.168.177.60/24"
ifconfig_em0="inet 192.168.176.60/24"

jail_squid_rootdir="/usr/squid"
# Either use:
jail_squid_ip="bge0|192.168.177.62/32,em0|192.168.176.62/32"
# or:
jail_squid_ip="bge0|192.168.177.62/32"
jail_squid_ip_multi0="em0|192.168.176.62/32"

but do not use jail_squid_interface=".." as that will be a global
default for that jail.


Is it a global *default* or a global? For example, could I specify:


It's a global default; a more specific interface name that comes with
an address will override it. So you could do what you drafted below.
The entire "ifconfig" feature in rc.d/jail does not really belong
there but people started using it after it was introduced so we lost
that race.


jail_squid_interface="bge0"
jail_squid_ip="192.168.177.62/32"
jail_squid_ip_multi0="192.168.177.63/32"
jail_squid_ip_multi1="em0|192.168.177.62/32"

Below is a patch against HEAD to document the $interface|$ip syntax.


That wasn't done on purpose; man rc.conf has it, if you lookup jail__ip .

/bz

--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Jail on 2 interfaces?

[Bjoern A. Zeeb]

On Wed, 23 Dec 2009, Matthew Seaman wrote:


Mel Flynn wrote:

Hi,

I don't see this documented in jail(8) nor rc(8) nor defaults/rc.conf, so 
is it possible to have 2 IP's on 2 ethernet interfaces? And if so, is it 
settable for rc(8)?


The usage case is to have the same jailed proxy server on two seperate 
internal networks. Ideally, the proxy will use one address for outgoing, so 
I guess I'll need a default route or dive into the squid config.


At present I have:
ifconfig_bge0="inet 192.168.177.60  netmask 255.255.255.0"
ifconfig_em0="inet 192.168.176.60 netmask 255.255.255.0"
ifconfig_em0_alias0="inet 192.168.176.62 netmask 255.255.255.255"
jail_squid_rootdir="/usr/squid"
jail_squid_ip="192.168.177.62"
jail_squid_ip_multi0="192.168.176.62"
jail_squid_interface="bge0"

But this created the IP on bge0 even though one exists on em0. Is it as 
simple as not specifying the interface and add the 177.62 alias on bge0?
Ideally I'd have a jail_$jail_ip_multi$aliasno_interface="foo0", but my 
main worry is that the jail infrastructure understands the routing 
involved.


To do this directly is now possible in 8.0-RELEASE or better.  You will
need a custom kernel with 'options VIMAGE' and I believe the standard jail
startup scripts need a bit of work in order for them to start the jail with
the correct command line arguments to enable the vnet functionality.


No, that's wrong.  FreeBSD 7.2-R and later can do multi-IP jails and
have the IPs on multiple interfaces; there is no need for a dedicated
network stack.

The routing is no much different than if you would do it in the base
system with two IPs.  if it works there, just putting it in a multi-IP
jail with the adresses on the right interface will just work as well.

If you want different routing for a jail use setfib with a multi-FIB
based kernel (you may need to recompile the kernel for that) but you
still won't need mutliple network stacks.



Alternatively, you can achieve much the same effect that you want by using
a simple one-ip jail and writing firewall rules to redirect traffic into it,
and NAT traffic coming out of it.


Using firewall NAT with jails is something I often see and usually
never understand unless people only have a single IP and want to share
that between lots of jails (though if not duplicate services exist,
that will just work as well by default these days as well).

--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Jail on 2 interfaces?

[Bjoern A. Zeeb]

On Tue, 22 Dec 2009, Mel Flynn wrote:

Hi,

first of all this would find more people to help on freebsd-jail as it
has nothing to do with hackers ;-)


I don't see this documented in jail(8) nor rc(8) nor defaults/rc.conf, so is
it possible to have 2 IP's on 2 ethernet interfaces? And if so, is it settable
for rc(8)?

The usage case is to have the same jailed proxy server on two seperate
internal networks. Ideally, the proxy will use one address for outgoing, so I
guess I'll need a default route or dive into the squid config.

At present I have:
ifconfig_bge0="inet 192.168.177.60  netmask 255.255.255.0"
ifconfig_em0="inet 192.168.176.60 netmask 255.255.255.0"
ifconfig_em0_alias0="inet 192.168.176.62 netmask 255.255.255.255"
jail_squid_rootdir="/usr/squid"
jail_squid_ip="192.168.177.62"
jail_squid_ip_multi0="192.168.176.62"
jail_squid_interface="bge0"

But this created the IP on bge0 even though one exists on em0. Is it as simple
as not specifying the interface and add the 177.62 alias on bge0?
Ideally I'd have a jail_$jail_ip_multi$aliasno_interface="foo0", but my main
worry is that the jail infrastructure understands the routing involved.



From what you are writing I assume that you are on FreeBSD 7.2-Release

or later; no official FreeBSD version before had supported
multiple-IPs with a jail.

What it did was what you were asking for.  That's the problem.

1) either use ifconfig
2) or use jail + interfaces
3) but do not mix them (especially not overlapping)

So I would suggest to do it like this:

# Base system IPs.
ifconfig_bge0="inet 192.168.177.60/24"
ifconfig_em0="inet 192.168.176.60/24"

jail_squid_rootdir="/usr/squid"
# Either use:
jail_squid_ip="bge0|192.168.177.62/32,em0|192.168.176.62/32"
# or:
jail_squid_ip="bge0|192.168.177.62/32"
jail_squid_ip_multi0="em0|192.168.176.62/32"

but do not use jail_squid_interface=".." as that will be a global
default for that jail.

As you can see, I removed the ifconfig_em0_alias0 line.  If you want
to keep that and mix things then you could do:

jail_squid_ip="bge0|192.168.177.62/32"
jail_squid_ip_multi0="192.168.176.62/32"

again without the jail_squid_interface=".." line.


HTH
/bz

--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: ioctl call freebsd < 7.2 in jail

[Bjoern A. Zeeb]

On Fri, 18 Dec 2009, Axel Scheepers wrote:

hi,


"Bjoern A. Zeeb"  writes:


I think I remember the patch; I guess it was the "samba patch".  I can
extract it for you; not sure if it'll work easily w/o the other
infrastructure but I'll see what I can do.

I can see no chance that it'll ever make it into 7.1 as an Errata
Notice though, so you would have to keep patching your system
yourself.



Hi Bjoern,

I know it won't make it as an errata, that's not a problem. I'm looking
for a way to fix the problem without having to upgrade all our virtual
hosts to 7.2. The machines in question run a mix of freebsd versions, if
it's not to much trouble it's highly appreciated if you can send me some
more information about the patch you mention. Is this the multi-ip jail
patch you talked about in
http://lists.freebsd.org/pipermail/freebsd-jail/2008-October/000488.html


well, I think that was the earlier workaround; the multi-IP jail
includes one of the patches that actually fix the problem; but the fix
for this case really just is a fraction of the change.

What I was thinking was this commit to head and the PR mentioned
there:
http://svn.freebsd.org/viewvc/base?view=revision&revision=186948

Note that 7.1 or earlier won;t have the prison_check_ip4() function,
different byte order requirements etc (and not IPv6 support).
It's s pretty short thing but has to be "converted" correctly.

I might be able to do that later today.

/bz

--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: ioctl call freebsd < 7.2 in jail

[Bjoern A. Zeeb]

On Fri, 18 Dec 2009, Axel Scheepers wrote:

Hi,


We run Direct Admin in a jailed setup on FreeBSD 7.2 (last supported
release for Direct Admin). We have a problem retrieving the correct ip
address of the jail which makes Direct Admin misbehave.

..

Would it be trivial to backport just the changes which changed this
behaviour and if so how can I get a patch for the changes to
7.2-RELEASE?

I've read that under FreeBSD it's better to use getifaddrs(3) but we
can't change the ioctl call.


I think I remember the patch; I guess it was the "samba patch".  I can
extract it for you; not sure if it'll work easily w/o the other
infrastructure but I'll see what I can do.

I can see no chance that it'll ever make it into 7.1 as an Errata
Notice though, so you would have to keep patching your system
yourself.

/bz

--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: ezjail with vimage

[Bjoern A. Zeeb]

On Mon, 7 Dec 2009, Miroslav Lachman wrote:


Bjoern A. Zeeb wrote:

On Mon, 7 Dec 2009, Miroslav Lachman wrote:

Hi Miroslav,


The last time I wrote with Bjoern A. Zeeb about jailname, cpuset etc.
support in rc.conf (back in March 2009) he stated that "there is no
need to add anything" because it can be done by jail_NAME_flags.
AFAIK current system still doesn't allow me to set cpuset to jail from
rc.conf


Check /etc/defaults/rc.conf for jail_example_exec_afterstart.


You already said that in the past and it was the reason why I found bug in 
cpuset.

http://lists.freebsd.org/pipermail/freebsd-jail/2009-April/000830.html

As I said, exec_afterstart is executed inside the jail and it means that I 
can not use it to bind the jail to specific CPU cores.

...but maybe I am blind.

Can you correct me if I am wrong?


*mumble*  *tired*  *again* ..

Let me cite man rc.conf to not mess it up again:

 jail__exec_afterstart
 (str) Unset by default.  This is the command run as Nth com-
 mand in a jail after jail startup, where N is 1, 2, and so
 on.

 jail__exec_poststart
 (str) Unset by default.  This is the command run as Nth com-
 mand after jail startup, where N is 0, 1, and so on.  It is
 run outside the jail.

HTH

/bz

--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: ezjail with vimage

[Bjoern A. Zeeb]

On Mon, 7 Dec 2009, Miroslav Lachman wrote:

Hi Miroslav,

The last time I wrote with Bjoern A. Zeeb about jailname, cpuset etc. support 
in rc.conf (back in March 2009) he stated that "there is no need to add 
anything" because it can be done by jail_NAME_flags.
AFAIK current system still doesn't allow me to set cpuset to jail from 
rc.conf


Check /etc/defaults/rc.conf for jail_example_exec_afterstart.

/bz

--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: [patch] Improved jail fstab functionality inside rc.d (needs testers and review)

[Bjoern A. Zeeb]

On Sun, 29 Nov 2009, Merijn Verstraaten wrote:

My apologies if these are the wrong lists for this sort of thing but it was 
unclear to me where else to go with additions like this.


You may try freebsd-jail@

Make sure to get a review from simon@ for this.

/bz

--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: AW: Networking from jail - errata

[Bjoern A. Zeeb]

On Tue, 17 Nov 2009, Scheithauer, Lars (FH) wrote:

Hi,


thanks for the clarification, I changed the values according to your 
suggestions. However, it did not resolve the problem.


Did you aslo check resolv.conf inside the jail?
Does host www.freebsd.org work?



I've checked the proxy logfiles and it seems, that the Makefile(s) don't try to 
access the proxy at all while fetching files. Is there any reason, why the 
Makefile(s) should not use the *_PROXY-variables on the jails?


I assume the proxy is squid and that the proxy itself works?
What if you set the http_proxy variables to an IP address rather than
the name (don't use 127.0.0.1 as address, just to rule that out as
well).

/bz

--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Networking from jail - errata

[Bjoern A. Zeeb]

On Tue, 17 Nov 2009, Scheithauer, Lars (FH) wrote:

Hi,


Quick note:
Forgot to replace two values.
Jail - x.y.z.61
Host - x.y.z.60
Router - x.y.z.62


-Ursprüngliche Nachricht-
Von: owner-freebsd-j...@freebsd.org [mailto:owner-freebsd-j...@freebsd.org] Im 
Auftrag von Scheithauer, Lars (FH)
Gesendet: Dienstag, 17. November 2009 10:19
An: freebsd-jail@freebsd.org
Betreff: Networking from jail

Hi everyone!

I'm having a little trouble with my jail's networking and I'm not sure
what to make of it.

My jailhost has an IP of x.y.z.48, my test jail is x.y.z.49. The
jailhost has both IP-adresses, the jail has just it's own:

Jail# ifconfig
bce0: flags=8843 metric 0 mtu
1500

options=1bb
   ether xx:xx:xx:xx:xx:10
   inet x.y.z.60 netmask 0xffc0 broadcast x.y.z.63
   media: Ethernet autoselect (1000baseSX )
   status: active
[...]
Host# ifconfig
bce0: flags=8843 metric 0 mtu
1500

options=1bb
   ether xx:xx:xx:xx:xx:10
   inet x.y.z.61 netmask 0xffc0 broadcast x.y.z.63
   inet x.y.z.60 netmask 0xffc0 broadcast x.y.z.63
   media: Ethernet autoselect (1000baseSX )
   status: active
[...]

I am able to access the ssh-server running on the jail, and I am able to
access the proxyserver of our network via telnet and get some pages of
the internet. However, if I want to install something from the ports,
the jail is unable to fetch it:

Jail# cd /usr/ports/ftp/wget
Jail# make
===>  Vulnerability check disabled, database not found
===>  Found saved configuration for wget-1.11.4_1
=> wget-1.11.4.tar.bz2 doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch from http://ftp.gnu.org/gnu/wget/.
fetch: http://ftp.gnu.org/gnu/wget/wget-1.11.4.tar.bz2: Operation timed
out
=> Attempting to fetch from ftp://ftp.gnu.org/gnu/wget/.
[...]

I've set the appropriate environment variables HTTP_PROXY, HTTPS_PROXY
and FTP_PROXY. If I test the connection with netcat, I get the following
error message:
# nc -zvw 1 -x 'proxy.example.com:8080' www.freebsd.org 80
nc: read failed (0/3): Broken pipe


The usual thing I am interested at that point is - does name
resolution work properly from within the jail?  /etc/resolv.conf setup
correctly etc?




The funny thing is, that I have no problem installing ports from the
Host-system. From what I can tell, all the config files are correct:

Jail# cat /etc/rc.conf
sshd_enable="YES"
ifconfig_bce0="inet x.y.z.60 netmask 255.255.255.192"
defaultrouter="x.y.z.62"
hostname="jail.example.com"


That's not going to work, really (the ifconfig, defaultrouter, and
unless you changed the defaults on the host system not even the
hostname).  You should actually remove those.



Host# cat /etc/rc.conf
sshd_enable="NO"
ifconfig_bce0="inet x.y.z.61 netmask 255.255.255.192"
defaultrouter="x.y.z.62"
hostname="host.example.com"
ipv6_enable="NO"
jail_enable="YES"
jail_set_hostname_allow="NO"
jail_list="jail"
jail_jail_hostname="jail"
jail_jail_ip="x.y.z.60"
jail_jail_rootdir="my/jail/root"
jail_jail_devfs_enable="YES"


That doesn't really match your ifconfig output from above; something
on the host system would have to set the IP address of the host. I
would expect something like (you may have mixed jail and host
addresses so properly sort this):

# host system IP address
ifconfig_bce0=inet x.y.z.61 netmask 255.255.255.192"
# jail IP address
ifconfig_bce0_alias0=inet x.y.z.60 netmask 255.255.255.255"

Note that the alias has a /32 netmask.


/bz

--
Bjoern A. Zeeb It will not break if you know what you are doing.___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Broadcast under Jail problems

[Bjoern A. Zeeb]

On Mon, 16 Nov 2009, Vagif Zeynalov wrote:

Hi,


...I can provide more details if it will be necessary...


error ogs from the application would be interesting to see which
(sys)call return which error so that we can narrow it down.

/bz

--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Setting the jail identifier from /etc/rc.conf

[Bjoern A. Zeeb]

On Thu, 22 Oct 2009, Ed Schouten wrote:


Hi,

I haven't played with Jails for a long time, but I wanted to figure out
how hard it is to make init spawn getties for certain jails. It
shouldn't be too hard (jexec foo /usr/libexec/getty), but I can't seem
to find a way to set the jid to a certain value from within rc.conf.

It also seems jids cannot contain dots, which means I cannot set the jid
equal to the hostname of the jail.

Maybe a Jail hacker can give me some advice here? Wouldn't it be more
sane if the kernel just used the hostname as an identifier if there is
no jail with the same hostname yet? Or maybe we should at least provide
a config tunable for this?


Redirect to freebsd-jail@  ; you may even find the answers to those
int he mail archive (unless those had been private threads I was on
Cc: on;-)


--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Per Jail Memory Limits

[Bjoern A. Zeeb]

On Mon, 5 Oct 2009, Julian Elischer wrote:


Tom Judge wrote:

Adam Vande More wrote:
On Mon, Oct 5, 2009 at 12:47 PM, Tom Judge <mailto:t...@tomjudge.com>> wrote:


Julian Elischer wrote:

Tom Judge wrote:

Hi,

Does anyone know of a patch that will add per jail memory
limits so that a jail can't swallow the resources of the
entire box?


Thanks

Tom

not yet..


I started to port this to 7.1 today:

http://wiki.freebsd.org/JailResourceLimits


What are the peoples opinions on this patch?


Tom


If you're soliciting opinions if this will be used and is needed, I would 
love to see this functionality.  This is the main reason I've had to chose 
XEN over jails.  If you need some help testing, let me know.


--
Adam Vande More

Hi Adam,

I have a patch against 7.1 here: 
http://svn.tomjudge.com/freebsd/patches/jail-resource-limits/jail-limits.patch 




probably the person who should work with this in -current is james (CC'd)


Probably the person who should be contacted is trasz who worked on
hierachical resource limit per .., jail in p4.  Though this is
slightly different.

I think it's ok if people need those things to update the pathes but I
doubt any will probably ever make it into FreeBSD as those things are kind
of contrary to the V_ plans.

BTW, I think the patch referenced is not the latest I had seen and I
thought that we also had one for 7.x or even for 8 already floating around.
Maybe some investigation on list archives etc. might be helpful before
starting to hack things.  Maybe also check the links on
http://wiki.freebsd.org/Jails





I will try to bring the patch up to current when I get a chance but I have 
no real need to do this as we use 7.1 in production.


Notes:

   * CPU limiting is not support is not supported unless you use 
shecd_4bsd.
   * I have not tested this on any system yet, just compile tested, I am 
putting it though its paces right now.


Tom


--
Bjoern A. Zeeb It will not break if you know what you are doing.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Tutorial for Hierarchical Jails?

[Bjoern A. Zeeb]

On Mon, 28 Sep 2009, Edwin Shao wrote:


Hi Jamie,
When I try to change the parameter, nothing happens:
rescue /etc> sudo sysctl security.jail.param.children.max=1
security.jail.param.children.max: 0 -> 0

rescue /etc> sudo sysctl security.jail.param.children.max
security.jail.param.children.max: 0

Am I doing this incorrectly?


Yes. It's a parameter to jail(8).  The security.jail.param sysctls can
be seen as a list of possible options valid to jail(8).  See man 8 jail
for the exact details.

/bz

--
Bjoern A. Zeeb   What was I talking about and who are you again?
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Not getting an IPv6 in a jail

[Bjoern A. Zeeb]

On Tue, 1 Sep 2009, Major Domo wrote:

Hi,


Apologies if this has been discussed already but I searched the web
and the mailing lists and haven't found hints on my problem.

I've got a jail, I assign it a set of IP addresses, and it just won't
take the IP6 I give it.


Uname:
FreeBSD 7.2-STABLE

jail_ns_ip="192.168.0.252,fe80::c0a8:fc"

jls -v:
  JID  Hostname  Path
   Name  State
   CPUSetID
   IP Address(es)
   23  [snip]  /var/jail/ns
 ALIVE
   2
   192.168.0.252
   fe80::c0a8:fc


ifconfig lo252 from the host:
lo252: flags=8049 metric 0 mtu 16384
   inet 192.168.0.252 netmask 0x
   inet6 fe80::c0a8:fc%lo252 prefixlen 128 scopeid 0x5


ifconfig from the jail:
re0: flags=8843 metric 0 mtu 1500
   
options=389b
   ether 00:e0:f4:19:e9:d2
   media: Ethernet autoselect (100baseTX )
   status: active
lo0: flags=8049 metric 0 mtu 16384
pflog0: flags=141 metric 0 mtu 33204
lo252: flags=8049 metric 0 mtu 16384
   inet 192.168.0.252 netmask 0x



This is a rather special case.  For link-local addresses you have to
give the scope as well but it won't take the scope with the %lo252
notation but only in the KAME in-kernel syntax I would assume.
Can you try:

jail_ns_ip="192.168.0.252,fe80:5::c0a8:fc"

Note the added 5 in the second group of hex digits.  That five is the
interface index.  I took it from the "scopeid 0x5". In case your
interface index changes you will need to adjust the address.

I cannot say if it'll work but it would be worth a try.

/bz

--
Bjoern A. Zeeb   What was I talking about and who are you again?
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: 8.0 still allow creating ipv6 udp socket in jail without ipv6 ip

[Bjoern A. Zeeb]

On Wed, 29 Jul 2009, Mykola Dzham wrote:

Hi,


Bjoern A. Zeeb wrote:

On Sat, 25 Jul 2009, Mykola Dzham wrote:

Hi,


After r188146 creating tcp ipv6 socket in jail without ipv6 ip is not
allowed, but udp socket is allowed.


I cannot really follow what you are trying to say as wrt IPv4 and IPv6
sockets and what about UDP.

Your sample further down is trying to use an IPv4 address on an IPv6
Datagram socket which is an error either way.


Some java programms attempt to use ipv6 sockets, then use ipv4 if
socket(AF_INET6,...) fail. My sample imitate this


Prior to FreeBSD 7.2 IPv6 hadn't been supported at all for jails.

With 7.2 it was possible to create IPv6 sockets (but only shortly and
then fail on bind/connect/...).  With the commit you reference the
"Protocol not supported" came back in case there was no address of
that address family for a given jail.

With 8 the primary syntax for jails has changed and the "backward
compat mode" again allows you to create a socket on a jail even if
no address of the same family was configured for the jail.

This should be addressed by the following patch:
http://people.freebsd.org/~bz/20090727-01-jail8-legacy.diff

Can you give it a try and report if that fixes your problem?


Patch aplied cleanly on r195820 , but jail can not start after patching:

# jail -l -U root -i /usr/home/d/guests/tap2 tap2.my.domain.com 10.112.0.151 
/bin/sh /etc/rc
jail: ip6: unknown boolean value "disable"


r195820 is too old; but Jamie has a better solution; I would suggest
to backout the jail(8) patch and wait for the next two commits of
Jamie to HEAD and then update the machine again.

/bz

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: 8.0 still allow creating ipv6 udp socket in jail without ipv6 ip

[Bjoern A. Zeeb]

On Sat, 25 Jul 2009, Mykola Dzham wrote:

Hi,


After r188146 creating tcp ipv6 socket in jail without ipv6 ip is not
allowed, but udp socket is allowed.


I cannot really follow what you are trying to say as wrt IPv4 and IPv6
sockets and what about UDP.

Your sample further down is trying to use an IPv4 address on an IPv6
Datagram socket which is an error either way.

Prior to FreeBSD 7.2 IPv6 hadn't been supported at all for jails.

With 7.2 it was possible to create IPv6 sockets (but only shortly and
then fail on bind/connect/...).  With the commit you reference the
"Protocol not supported" came back in case there was no address of
that address family for a given jail.

With 8 the primary syntax for jails has changed and the "backward
compat mode" again allows you to create a socket on a jail even if
no address of the same family was configured for the jail.

This should be addressed by the following patch:
http://people.freebsd.org/~bz/20090727-01-jail8-legacy.diff

Can you give it a try and report if that fixes your problem?

Regards,
Bjoern

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Multicast in jail?

[Bjoern A. Zeeb]

On Tue, 7 Jul 2009, Alexander Leidinger wrote:

Quoting Bill Marquette  (from Mon, 6 Jul 2009 
20:14:02 -0500 (CDT)):


I'm trying to run Avahi in a jail, much the same as Alexander Leidinger in 
this email from late last year 
http://www.mail-archive.com/freebsd-jail@freebsd.org/msg00587.html.  I 
couldn't find any replies to that thread and it seems that I'm running into 
the same issues - the service announcements make it on the wire and the 
other devices in the network see them.


So far I have nothing working.

I assume that the mcast traffic is not arriving at all IPs. guess>Either because on overly restrictive jail check, and/or just because 
there's no code which is distributing the traffic to all IPs.


It seems kern_jail.c is a place to check if there's some code which handles


No, in_pcb.c in6_pcb.c in_m*.[ch] in6_m*.[ch] are the files you need
as a starting point; there's more and more and more.

this. Maybe prison_check_ip[46] if mcast is on top of this, or something new 
to write if mcast is a different "AF". Again, this is a wild guess, I don't 
have enough understanding of the network code in the kernel to even make 
educated guesses about the real reason.


But you first will have to understand all implications, that need a
proper design plan and after that thoughtout implementation.

Alternatively I wouldn't wonder if enabling raw sockets would give
what you want or you'll wait for virtualization to be ready.

/bz

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Can't login Jailed system

[Bjoern A. Zeeb]

On Mon, 29 Jun 2009, Sam Wun wrote:

Hi,

we've got a freebsd-jail list that I am Cc:ing.



With FreeBSD 7.2Stable,
I have done this many times before.
After about a month left the "jail" behind, now when I done a
"/etc/rc.d/jail start" and ssh into it, I ended up login to the host
system.
Here is the network configuraiton of the host system and the jail system:

# ifconfig
rl0: flags=8843 metric 0 mtu 1500
   options=8
   ether 00:00:21:ef:27:f7
   media: Ethernet autoselect (100baseTX )
   status: active
rl1: flags=8802 metric 0 mtu 1500
   options=8
   ether 00:50:fc:65:78:c0
   media: Ethernet autoselect
   status: no carrier
fxp0: flags=8843 metric 0 mtu 1500
   options=8
   ether 00:13:20:65:a9:be
   inet 192.168.1.246 netmask 0xff00 broadcast 192.168.1.255
   inet 192.168.1.245 netmask 0xff00 broadcast 192.168.1.255
   inet 192.168.1.235 netmask 0xff00 broadcast 192.168.1.255
   inet 192.168.1.242 netmask 0x broadcast 192.168.1.242
   media: Ethernet autoselect (100baseTX )
   status: active
plip0: flags=108810 metric 0 mtu 1500
enc0: flags=0<> metric 0 mtu 1536
pflog0: flags=141 metric 0 mtu 33204
pfsync0: flags=0<> metric 0 mtu 1460
   syncpeer: 224.0.0.240 maxupd: 128
lo0: flags=8049 metric 0 mtu 16384
   inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
   inet6 ::1 prefixlen 128
   inet 127.0.0.1 netmask 0xff00
twp1:# jls
  JID  IP Address  Hostname  Path
5  192.168.1.242   twp5.ip6.com.au   /usr/jail2/twp5

192.168.1.242 is the jailed system,
twp1 is the host system.

After I login 192.168.1.242, I ended up logged in twp1 which is my host system.
Now I am stuck. I don't know how I logged in the jailed system a month ago.

Can anyone shred some lights on me?


Try to jexec 5 /bin/sh (5 is the jailID from the jls output)  and check
with ps if sshd is running inside the jail, and check the usual things
are up and there.


/bz

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Switching /etc/rc.d/jail to new syntax (+ new features)

[Bjoern A. Zeeb]

On Sat, 27 Jun 2009, Alexander Leidinger wrote:


On Sat, 27 Jun 2009 10:47:47 + (UTC) "Bjoern A. Zeeb"
 wrote:



On Sat, 27 Jun 2009, Alexander Leidinger wrote:


at http://www.leidinger.net/FreeBSD/current-patches/jail.diff I
have a patch to switch the jail rc script to the new jail
(8-current) syntax. This includes new config options for a jail
(see etc/defaults/rc.conf after patching). The patch also contains
my X-in-a-jail stuff (feel free to ignore this part, it's disabled
by default).

If you do not make any config change, you will be able to see all
mounted filesystems of the entire machine. To get back to the
previous behavior, you have to add a config option:
 jail_XXX_startparams="enforce_statfs=2"

This config option can also take other jail parameters like
allow.sysvipc and other ones described in the jail man-page
(additional parameters need to be space separated).

Feedback welcome.


1) it break various things that will no longer work


As mentioned, it "breaks" the statfs part. If there's anything else, be
more specific please.


v6, noIP, ...



2) it's not a poper solution


The proper solution for the statfs part would be, that jail(8) defaults
to =2 if nothing is specified. Alternatively I can get convinced that
we should do a default for it in defaults/rc.conf if nothing is specied
for startparams for a particular jail (like we have for some other
things), but this would not be as good as if jail(8) would handle it
itself.

If you do not talk about the statfs part but in a more generic way,
what would be a proper solution in your eyes?


A proper solution would be a proper mgmt system ready for the future
instead of continuting to hack up rc.d/jail via option fo bar baz and
another 17000 of them.
But this is nothing I'll discuss today while things aren't fully
shaken out yet.

For now what used to work should continue to work and not break.
Everything else on top of that needs to be done properly and not in a
rainy-midnight-drive-by.

/bz

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Switching /etc/rc.d/jail to new syntax (+ new features)

[Bjoern A. Zeeb]

On Sat, 27 Jun 2009, Alexander Leidinger wrote:


at http://www.leidinger.net/FreeBSD/current-patches/jail.diff I have a
patch to switch the jail rc script to the new jail (8-current) syntax.
This includes new config options for a jail (see etc/defaults/rc.conf
after patching). The patch also contains my X-in-a-jail stuff (feel
free to ignore this part, it's disabled by default).

If you do not make any config change, you will be able to see all
mounted filesystems of the entire machine. To get back to the previous
behavior, you have to add a config option:
 jail_XXX_startparams="enforce_statfs=2"

This config option can also take other jail parameters like
allow.sysvipc and other ones described in the jail man-page (additional
parameters need to be space separated).

Feedback welcome.


1) it break various things that will no longer work
2) it's not a poper solution

/bz

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: sysctl variables not propagating to children jails

[Bjoern A. Zeeb]

On Tue, 9 Jun 2009, Edwin Shao wrote:

Hi,


In the most recent -current, I've noticed that sysctl variables no
longer propagate to jails and thus it is impossible to allow raw
sockets, allow mounting, etc. This might be related to
<http://www.mail-archive.com/freebsd-jail@freebsd.org/msg00847.html>.

..

Please let me know if I'm doing something stupid!


No, nothing apart from probably not spotting that the problem was
already well understood and there had been workarounds suggested at
the end of the above thread.


That said, expect the problem to be fixed within 24 hours.  You will
only have to rebuild your jail(8) command line tool, once you spot the
commit by:

1) update your source and make sure to have the new version of jail.c
2) cd src/usr.sbin/jail
3) make obj && make depend && make all
4) sudo make install
5) try again.

In case you still see problems afterwards, cry again, loud and in
here - in case that will fix the problem a short note will be
welcome as well;-)

/bz

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: kern/133265: [jail] is there a solution how to run nfs client in jail environment?

[Bjoern A. Zeeb]

The following reply was made to PR kern/133265; it has been noted by GNATS.

From: "Bjoern A. Zeeb" 
To: bug-follo...@freebsd.org, p...@fincombank.com
Cc:  
Subject: Re: kern/133265: [jail] is there a solution how to run nfs client
 in jail environment?
Date: Mon, 8 Jun 2009 17:18:35 + (UTC)

 The general answer is:  it is not possible.
 
 You could do the NFS mount from the base system and have the
 mountpoint within the visbility of the jail.
 
 You may get around enabling raw_sockets but if that works somehow I
 wouldn't rely on it and you'll have to be aware of what globally
 enabling raw sockets means.
 
 With FreeBSD 8 it will hopefully be possible as you may have your own
 network stack oer jail.  I am just not sure if the NFS code is there
 ("fully virtualized") yet to make it work.
 
 -- 
 Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: sysvipc in jails + CURRENT

[Bjoern A. Zeeb]

On Thu, 4 Jun 2009, Boris Samorodov wrote:

Hi,


There is definitely some inconsistency. JAIL(8) at recent
CURRENT talk about security.jail.param.allow.sysvipc and
it is listed via "sysctl -d security.jail.param". But seems
not to be used:
- at the jail -
# sysctl security.jail.param.allow.sysvipc
#
-


If you can use an old jail binary things should work for you for the
moment.  The jail(8) compat code that still supports the old syntax
but already uses the new syscall does not take the old sysctls into
account -  the problem you are seeing.

Alternatively you could try updating the jail by hand using the new
syntax and switch sysvipc on.

The bug will probably be fixed latest somewhen next week and I just
got back and have a huge backlog and Jamie will be back in a few days
I think.


/bz

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: sysvipc in jails + CURRENT

[Bjoern A. Zeeb]

On Sun, 31 May 2009, Boris Samorodov wrote:

Hi,


has something changed at CURRENT with sysvipc jail handling?
This jail has been working fine for almost a year.

I've upgrade CURRENT to yesterday's sources and can't start
postgresql in a jail anymore:
- the jail -
% tail -2 /var/log/messages
May 31 18:22:47 pg postgres[55425]: [1-1] FATAL:  could not create shared 
memory segment: Function not implemented
May 31 18:22:47 pg postgres[55425]: [1-2] DETAIL:  Failed system call was 
shmget(key=5432001, size=30384128, 03600).
% sysctl security.jail.sysvipc_allowed
security.jail.sysvipc_allowed: 0
% grep sysvipc /etc/sysctl.conf
security.jail.sysvipc_allowed=1
- the host -
% uname -a
FreeBSD tba.bsam.ru 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Sun May 31 11:28:31 MSD 
2009 r...@tba.bsam.ru:/usr/obj/usr/src/sys/TBA  amd64
% sysctl security.jail.sysvipc_allowed
security.jail.sysvipc_allowed: 1
-


I'll look into that; possibly the default option is not properly taken
into account for the new jail framework.

/bz

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: bind()/sendto() behavior in RELENG_7

[Bjoern A. Zeeb]

On Fri, 8 May 2009, Brian A. Seklecki wrote:

Hi,


All:

Did the behavior of bind()/sendto() functions WRT jails change in
proximity to the RELENG_7_2 branch?

I just spent 1.5 days chasing, what I thought was a bug in Courier-MTA's
IPv6 socket selection code within Jails, to realize a paradox of a
configuration scenario:

My ESTMP client libraries in Courier were programed to explicitly bind()
to a specific source address.  The system in question was RELENG_7 from
last month; but was upgraded to 7.2-R last week, when this problem was
observed.  After which, I began to receive:
  "Can't assign requested address", as expected.

Unfortunately, we also enabled IPv6 on the system at the same time,
complicating troubleshooting.

The configuration for Courier in the jail is being rsync(1)'d every hour
from a production environment (where explicit binding for System-Service
abstraction is a security policy requirement) to a DRP system within a
Jail.

So as far as I know, the explicit bind was always present in the DRP
jail and in theory, should never have worked.

I hypothesize that after 7.2-R was installed, the correct behavior of
bind() began to occur, and that prior to that, it was gracefully
allowing Courier to bind() to an IP that wasn't present in the jail.

Unfortunately, I don't have any records of what the RELENG_7 build date
was of the original jail environment to test this hypothesis.


So I am having trouble understanding the actual problem with what on
which system what fails and enough things are coming together> So let
me ask a few questions/explain:

1) Had you been running the multi-IP jail work on the 7-STABLE before
already?

2) In the past you did bind to an IPv4 address and the same address
worked on machines even if the IP wasn't there. Right?

3) Now you switched on IPv6 as well 2) no longer works?

4) can you give me the output of sysctl net.inet6.ip6.v6only ?

/bz

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: HEADS UP: multi-IPv4/v6/no-IP jails now in 7-STABLE

[Bjoern A. Zeeb]

On Thu, 30 Apr 2009, Stefan Lambrev wrote:


Hi,

On Apr 22, 2009, at 11:25 PM, Miroslav Lachman wrote:


Stefan Lambrev wrote:

Hi,
Does this allow multiple network interfaces to be used by a single  jail 
instance?


Yes, I am using it.


- cut -

Basically it works, but I found another problem.
I have created on two servers jails with 2 IPs on different interfaces.
First IP is on "external" interface and second IP is on internal interface.
As expected if I send packets from the host (outside jail) their source 
address match the IP of the interface (from which they are leaving the 
machine),
but if I send packets from jail they always go out with source address equal 
to the first IP of the jail even when they are going out

through the second interface.

I do not know if this matters but in my case, internal interface have few 
vlans and the IP is set on the vlan not directly on the interface.


Here is some output from the jail which can be useful:

igb0: flags=8843 metric 0 mtu 1500
options=19b
ether 00:30:48:9c:3a:0a
inet 192.168.3.100 netmask 0x broadcast 192.168.3.100
media: Ethernet autoselect (100baseTX )
status: active

igb1.2: flags=8843 metric 0 mtu 1500
options=3
ether 00:30:48:9c:3a:0b
inet 10.35.1.1 netmask 0xff00 broadcast 10.35.1.255
media: Ethernet autoselect (1000baseTX )
status: active
vlan: 2 parent interface: igb1

And here is the tcpdump from igb1.2 when trying to ping 10.35.1.2 from inside 
jail:


17:20:04.109972 IP 192.168.3.100 > 10.35.1.2: ICMP echo request, id 28421, 
seq 0, length 64
17:20:05.110321 IP 192.168.3.100 > 10.35.1.2: ICMP echo request, id 28421, 
seq 1, length 64


Any idea how this can be fixed?

P.S. I know I can rewrite outgoing packets with firewall, but it's not 
performance wise,
and I expect lot of udp multicast through igb1.2, that's why this doesn't 
look like a proper solution for me.



1) you turned on a non-default feature permitting raw-ip-sockets from
   inside jails. You lost supp^Wpredicatability. Well not really but
   this is just the beware-of reminder.
2) you are using 1) with ping to test source address selection which
   will not work well. There is more magic involved.  Does it work
   properly and as requested with ping -S  ?
3) turn off 1) and/or use telnet, ssh, or nc to test outgoing connections
   in each direction. Does source address selection work here as
   expected?
4) jails do not support MC. You'll have to wait for full-blown network
   stack virtualization.



--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: changing cpuset of jail from inside of jail - is it feature?

[Bjoern A. Zeeb]

On Fri, 24 Apr 2009, Miroslav Lachman wrote:


Bjoern A. Zeeb wrote:

[...]


Ok, I am not sure what is going wrong here; well I know but I don't
know if it's intended in cpuset.  Trying to talk to the right people
but they seen to be AWOL atm.


If you are brave, you could try:

http://people.freebsd.org/~bz/20090423-01-cpuset-jails.diff

I haven't even compiled it yet. It may work, it may not work, it may
make your machine panicing, ... just to warn you.

it should still allow you to create further sets within a jail but you
should not be able to change the "root set" of the jail from inside
the jail anymore (in case it works;)


I did just a quick test. (OK, not so quick, because compilation inside Qemu 
on my old PC takes 2 hours ;])

It compiles without problems and did what I expect:


...
I have no real multicore machine to test it more deeply. (can't test it on 
production servers and spare machine is blocked by another task)


Will this fix be included in 7.2-RELEASE or is it too late to commit this 
fix?


FreeBSD 7/7.2 just got a BUGS entry for the man pages.  The patch will
not make it;  it's still waiting review for HEAD and possibly
discussion if a super user inside a jail would still be allowed to
further restrict the cpuset (but not extend it).

/bz

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: changing cpuset of jail from inside of jail - is it feature?

[Bjoern A. Zeeb]

On Wed, 22 Apr 2009, Miroslav Lachman wrote:

Hi,


Bjoern A. Zeeb wrote:


On Wed, 22 Apr 2009, Miroslav Lachman wrote:

Hi,

I am running system FreeBSD 7.1-STABLE amd64 GENERIC (Wed Feb 11 09:56:08 
CET 2009) hosting few jails.
The machine has dual core CPU and some jails are set to run only on one 
core (core 0 in this example):


   host# cpuset -l 0 -j 25

As I tested today, root user inside the jail can change this by the same 
command as I am doing it from the host system:


  injail# cpuset -l 0,1 -j 25

And from now, jail with JID 25 is running on both cores.

Is it expected behavior of cpuset to allow user inside the jail change 
cpuset of the jail itself or is it a bug?


It seems to me as undesirable.



it is (undesirable) and it seems to be a bug as even if you do

host# cpuset -l 0 -r -j 25

you can get back to 0,1 from within the jail.

I'll check how/why this is possible.

/bz

PS: moving this to freebsd-jail@


Ok, I am not sure what is going wrong here; well I know but I don't
know if it's intended in cpuset.  Trying to talk to the right people
but they seen to be AWOL atm.


If you are brave, you could try:

http://people.freebsd.org/~bz/20090423-01-cpuset-jails.diff

I haven't even compiled it yet. It may work, it may not work, it may
make your machine panicing, ... just to warn you.

it should still allow you to create further sets within a jail but you
should not be able to change the "root set" of the jail from inside
the jail anymore (in case it works;)

/bz

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: changing cpuset of jail from inside of jail - is it feature?

[Bjoern A. Zeeb]

On Wed, 22 Apr 2009, Miroslav Lachman wrote:

Hi,

I am running system FreeBSD 7.1-STABLE amd64 GENERIC (Wed Feb 11 09:56:08 CET 
2009) hosting few jails.
The machine has dual core CPU and some jails are set to run only on one core 
(core 0 in this example):


   host# cpuset -l 0 -j 25

As I tested today, root user inside the jail can change this by the same 
command as I am doing it from the host system:


  injail# cpuset -l 0,1 -j 25

And from now, jail with JID 25 is running on both cores.

Is it expected behavior of cpuset to allow user inside the jail change cpuset 
of the jail itself or is it a bug?


It seems to me as undesirable.


it is (undesirable) and it seems to be a bug as even if you do

host# cpuset -l 0 -r -j 25

you can get back to 0,1 from within the jail.

I'll check how/why this is possible.

/bz

PS: moving this to freebsd-jail@

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Regarding multi-ip Bjoern head patch

[Bjoern A. Zeeb]

On Wed, 22 Apr 2009, Vivek Gite wrote:


Hi,

I'm running FreeBSD 7.1_AMD_P4 release with 4 jails. Recently, our ISP 
provided us IPv6 and I'd like to use both multiple IPv4 and IPv6 for my 
jails. According to FreeBSD wiki (http://wiki.freebsd.org/Jails) - there is a 
patch  http://svn.freebsd.org/viewvc/base?view=revision&revision=188281 ; 
which  is committed to FreeBSD. But I'm not able to use it under said 
version. So I'm looking to grab this one and manually patch it up. Is there 
any tar-ball to grab a patch? Is it included in FreeBSD 7.2RC1? How do I grab 
HEADS UP r185435?


Yes, all of FreeBSD 7.2 (BETA, RC1, upcomig RC2 and RELEASE) have and
will have it.  So if you are going to update your system to any of
those versions you'll have it.

/bz

PS: in case of reply please remove the -virtualization Cc:

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: lo0's IPv6 address overwritten

[Bjoern A. Zeeb]

On Fri, 10 Apr 2009, JAKO Andras wrote:


I found that when I start a telnet in the jail to an arbitrary global IPv6
address, lo0's ::1 changes to the jail's IPv6 address. The routing table
doesn't change.


telnet to where? To the jail IP? To an IP of the base system? To world?


I started telnet inside the jail, to the world:

[r...@splash /usr/home/goya]# jail -l -U root -i /usr/jail/ro.noc
r-noc.net.bme.hu "2001:738:2001:1000::2" /bin/sh
1
# ifconfig lo0 ; ifconfig lo1
lo0: flags=8049 metric 0 mtu 16384
lo1: flags=8049 metric 0 mtu 16384
   inet6 2001:738:2001:1000::2 prefixlen 128
# telnet 2001:738::abcd
Trying 2001:738::abcd...
^C#
# ifconfig lo0 ; ifconfig lo1
lo0: flags=8049 metric 0 mtu 16384
   inet6 2001:738:2001:1000::2 prefixlen 128
lo1: flags=8049 metric 0 mtu 16384
   inet6 2001:738:2001:1000::2 prefixlen 128


*wow*, that's indeed ... confusing. I'll try to (get someone to) look
into this.

/bz

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: lo0's IPv6 address overwritten

[Bjoern A. Zeeb]

On Fri, 10 Apr 2009, JAKO Andras wrote:

Hi,


This works fine here too.


Good.


I wonder what's going onfor you.  Can you check with
netstat -rn -f inet6
that what you are seeing is indeed true?


It's always the same:


I cannot see the /128 on lo0 so that's fine too.



Can you try starting the jail to get an interactive shell and not
running any scripts like I did and check what happens then?


That works, and ifconfig doesn't show any change on lo0.


Good, as I said above.



I found that when I start a telnet in the jail to an arbitrary global IPv6
address, lo0's ::1 changes to the jail's IPv6 address. The routing table
doesn't change.


telnet to where? To the jail IP? To an IP of the base system? To world?


Which version of RELENG_7 are you on (as what does a few days mean)?

/bz

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: lo0's IPv6 address overwritten

[Bjoern A. Zeeb]

On Fri, 10 Apr 2009, JAKO Andras wrote:


Hi,

Starting a jail with one IPv4 and one IPv6 address on a few days old
RELENG_7 overwrites lo0's ::1 with the jail's IPv6 address. (The jail's
addresses are preconfigured on lo1.)

Is this expected behaviour? Or did I made something the wrong way?

Here's ifconfig(8)'s output before and after executing jail(8), and also
from inside of the jail.


testing this on a bit older HEAD:

ifconfig lo1 create inet6 2001:738:2001:1000::2/128
ifconfig lo0 ; ifconfig lo1
lo0: flags=8049 metric 0 mtu 16384
options=3
inet 127.0.0.1 netmask 0xff00
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
lo1: flags=8049 metric 0 mtu 16384

options=3
inet6 2001:738:2001:1000::2 prefixlen 128 
jail -l -U root -i / hostname "2001:738:2001:1000::2" /bin/sh

1
# sysctl security.jail.jailed
security.jail.jailed: 1
# ifconfig lo0; ifconfig lo1
lo0: flags=8049 metric 0 mtu 16384
options=3
lo1: flags=8049 metric 0 mtu 16384
options=3
inet6 2001:738:2001:1000::2 prefixlen 128

[ another xterm ]

bz@:~> sysctl security.jail.jailed
security.jail.jailed: 0
bz@:~> ifconfig lo0 ; ifconfig lo1
lo0: flags=8049 metric 0 mtu 16384
options=3
inet 127.0.0.1 netmask 0xff00
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
lo1: flags=8049 metric 0 mtu 16384

options=3
inet6 2001:738:2001:1000::2 prefixlen 128 
..

ifconfig lo1 destroy


I wonder what's going onfor you.  Can you check with
netstat -rn -f inet6
that what you are seeing is indeed true?

Can you try starting the jail to get an interactive shell and not
running any scripts like I did and check what happens then?

/bz

--
Bjoern A. Zeeb  The greatest risk is not taking one.
___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"

Re: Adding ips to running jail.

[Bjoern A. Zeeb]

On Thu, 2 Apr 2009, Peter Ankerstål wrote:


Is it possible to add ip-addresses to an already running jail?


Not yet but possibly soon (in FreeBSD 8).

--
Bjoern A. Zeeb  The greatest risk is not taking one.___
freebsd-jail@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"