Re: UNIX domain sockets on nullfs still broken?
On Mon, 30 Nov 2009, xorquew...@googlemail.com wrote:
jackd (audio/jack) creates a directory in /tmp with a UNIX domain socket in
it. Clients connect to this socket to communicate with the server.
We currently support the sharing of UNIX domain sockets between file system
layers on either nullfs or unionfs. In the former case, this is a bug, and in
the latter case, it is a feature.
The specific nature of the bug is that you can't just copy the socket pointer
between layers in the vnode stack without additional reference counting (and
other similar state propagation), so if we allowed inter-layer access it would
lead to use-after-free panics and similar sorts of problems.
This occurs, BTW, because the socket pointer is directly in struct vnode, and
not queried by a VOP, which could be forwarded by nullfs down a layer. The
fixes here aren't easy, so I would anticipate UNIX domain sockets not working
across nullfs layers for some time to come. It's not immediately clear to me
which approach is the best way to fix it, since it likely requires UNIX domain
sockets to learn about stacked file systems in some form, which will
significantly complicate an already complicated relationship.
Robert N M Watson
Computer Laboratory
University of Cambridge
$ jackd -d oss -r 44100 -p 128
$ ls -alF /tmp/jack-11001/default
total 4
drwx-- 2 xw wheel 512 30 Nov 14:19 ./
drwx-- 3 xw wheel 512 30 Nov 14:19 ../
prw-r--r-- 1 xw wheel0 30 Nov 14:19 jack-ack-fifo-54211-0|
prw-r--r-- 1 xw wheel0 30 Nov 14:19 jack-ack-fifo-54211-1|
prw-r--r-- 1 xw wheel0 30 Nov 14:19 jack-ack-fifo-54211-2|
srwxr-xr-x 1 xw wheel0 30 Nov 14:19 jack_0=
srwxr-xr-x 1 xw wheel0 30 Nov 14:19 jack_ack_0=
$ sudo mount_nullfs /tmp/ /jail/k4m/tmp
In the jail:
k4m$ ls -alF /tmp/jack-11001/default
drwx-- 2 xw wheel 512 30 Nov 14:19 ./
drwx-- 3 xw wheel 512 30 Nov 14:19 ../
prw-r--r-- 1 xw wheel0 30 Nov 14:19 jack-ack-fifo-54211-0|
prw-r--r-- 1 xw wheel0 30 Nov 14:19 jack-ack-fifo-54211-1|
prw-r--r-- 1 xw wheel0 30 Nov 14:19 jack-ack-fifo-54211-2|
srwxr-xr-x 1 xw wheel0 30 Nov 14:19 jack_0=
srwxr-xr-x 1 xw wheel0 30 Nov 14:19 jack_ack_0=
k4m$ ktrace jack_showtime
jack server not running?
k4m$ kdump | grep '/tmp/jack-11001'
76030 initial thread STRU struct sockaddr { AF_LOCAL,
/tmp/jack-11001/default/jack_0 }
76030 initial thread NAMI /tmp/jack-11001/default/jack_0
76030 initial thread RET connect -1 errno 61 Connection refused
$ uname -a
FreeBSD viper.internal.network 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat Nov 21
15:02:08 UTC 2009 r...@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
amd64
xw
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: UNIX domain sockets on nullfs still broken?
On Mon, 30 Nov 2009, Ivan Voras wrote: What's the sane solution, then, when the only method of communication is unix domain sockets? It is a security problem. I think the long-term solution would be to add a sysctl analogous to security.jail.param.securelevel to handle this. I don't think there is a workaround right now. I'm not sure I agree on the above, hence my comments about nullfs and unionfs. I see nullfs as intended to provide references (possibly masked to read-only) to the same fundamental object, and unionfs to provide independence between different consumers that see objects via different file system mounts. As such, I'd expect UNIX domain sockets to work for inter-jail communication when using nullfs, and not work when using unionfs. It's simply a property of the implementation of the linkage between VFS and UNIX domain sockets that they are currently both broken (in fact, someone tried to fix it with union mounts recenty, running into the use-after-free bugs I mentioned, but also breaking the semantics in my view). Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: UNIX domain sockets on nullfs still broken?
On Tue, 1 Dec 2009, Linda Messerschmidt wrote: On Mon, Nov 30, 2009 at 10:14 AM, Ivan Voras ivo...@freebsd.org wrote: What's the sane solution, then, when the only method of communication is unix domain sockets? It is a security problem. I think the long-term solution would be to add a sysctl analogous to security.jail.param.securelevel to handle this. Out of curiosity, why is allowing accessing to a Unix domain socket in a filesystem to which a jail has explicitly been allowed access more or less secure than allowing access to a file or a devfs node in a filesystem to which a jail has explicitly been allowed access? (I seem to have caught this thread rather late in the game due to being on travel) -- Ivan is wrong about nullfs, it's broken due to a bug, not a feature, and that bug is not present when using a single file system. He's thinking of unionfs semantics, where if it worked it would be a bug. :-) Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: UNIX domain sockets on nullfs still broken?
On Thu, 10 Dec 2009, Robert Watson wrote:
On Mon, 30 Nov 2009, xorquew...@googlemail.com wrote:
jackd (audio/jack) creates a directory in /tmp with a UNIX domain socket in
it. Clients connect to this socket to communicate with the server.
We currently support the sharing of UNIX domain sockets between file system
layers on either nullfs or unionfs. In the former case, this is a bug, and
Should read neither ... nor.
Robert N M Watson
Computer Laboratory
University of Cambridge
in the latter case, it is a feature.
The specific nature of the bug is that you can't just copy the socket pointer
between layers in the vnode stack without additional reference counting (and
other similar state propagation), so if we allowed inter-layer access it
would lead to use-after-free panics and similar sorts of problems.
This occurs, BTW, because the socket pointer is directly in struct vnode, and
not queried by a VOP, which could be forwarded by nullfs down a layer. The
fixes here aren't easy, so I would anticipate UNIX domain sockets not working
across nullfs layers for some time to come. It's not immediately clear to me
which approach is the best way to fix it, since it likely requires UNIX
domain sockets to learn about stacked file systems in some form, which will
significantly complicate an already complicated relationship.
Robert N M Watson
Computer Laboratory
University of Cambridge
$ jackd -d oss -r 44100 -p 128
$ ls -alF /tmp/jack-11001/default
total 4
drwx-- 2 xw wheel 512 30 Nov 14:19 ./
drwx-- 3 xw wheel 512 30 Nov 14:19 ../
prw-r--r-- 1 xw wheel0 30 Nov 14:19 jack-ack-fifo-54211-0|
prw-r--r-- 1 xw wheel0 30 Nov 14:19 jack-ack-fifo-54211-1|
prw-r--r-- 1 xw wheel0 30 Nov 14:19 jack-ack-fifo-54211-2|
srwxr-xr-x 1 xw wheel0 30 Nov 14:19 jack_0=
srwxr-xr-x 1 xw wheel0 30 Nov 14:19 jack_ack_0=
$ sudo mount_nullfs /tmp/ /jail/k4m/tmp
In the jail:
k4m$ ls -alF /tmp/jack-11001/default
drwx-- 2 xw wheel 512 30 Nov 14:19 ./
drwx-- 3 xw wheel 512 30 Nov 14:19 ../
prw-r--r-- 1 xw wheel0 30 Nov 14:19 jack-ack-fifo-54211-0|
prw-r--r-- 1 xw wheel0 30 Nov 14:19 jack-ack-fifo-54211-1|
prw-r--r-- 1 xw wheel0 30 Nov 14:19 jack-ack-fifo-54211-2|
srwxr-xr-x 1 xw wheel0 30 Nov 14:19 jack_0=
srwxr-xr-x 1 xw wheel0 30 Nov 14:19 jack_ack_0=
k4m$ ktrace jack_showtime
jack server not running?
k4m$ kdump | grep '/tmp/jack-11001'
76030 initial thread STRU struct sockaddr { AF_LOCAL,
/tmp/jack-11001/default/jack_0 }
76030 initial thread NAMI /tmp/jack-11001/default/jack_0
76030 initial thread RET connect -1 errno 61 Connection refused
$ uname -a
FreeBSD viper.internal.network 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat Nov
21 15:02:08 UTC 2009
r...@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
xw
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: UNIX domain sockets on nullfs still broken?
2009/12/10 Robert Watson rwat...@freebsd.org: On Tue, 1 Dec 2009, Linda Messerschmidt wrote: On Mon, Nov 30, 2009 at 10:14 AM, Ivan Voras ivo...@freebsd.org wrote: What's the sane solution, then, when the only method of communication is unix domain sockets? It is a security problem. I think the long-term solution would be to add a sysctl analogous to security.jail.param.securelevel to handle this. Out of curiosity, why is allowing accessing to a Unix domain socket in a filesystem to which a jail has explicitly been allowed access more or less secure than allowing access to a file or a devfs node in a filesystem to which a jail has explicitly been allowed access? (I seem to have caught this thread rather late in the game due to being on travel) -- Ivan is wrong about nullfs, it's broken due to a bug, not a feature, and that bug is not present when using a single file system. He's thinking of unionfs semantics, where if it worked it would be a bug. :-) You have a point there. I was actually thinking more of sysvshm - which doesn't have anything to do with any of the issues here - but has some of the same properties (and is also used by databases - e.g. postgresql, which I'm using daily so it sort of cross-linked). The reason I'd like the nullfs barrier kept is that it (like shm) is used for IPC, and in this case, IPC across different jails (though a file system itself also be used so...). It's not a big issue - I'll also accept that it's the operator's fault if he doesn't know sharing file systems will also share sockets and fifos on it... ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: UNIX domain sockets on nullfs still broken?
On 10 Dec 2009, at 09:59, Ivan Voras wrote: You have a point there. I was actually thinking more of sysvshm - which doesn't have anything to do with any of the issues here - but has some of the same properties (and is also used by databases - e.g. postgresql, which I'm using daily so it sort of cross-linked). The reason I'd like the nullfs barrier kept is that it (like shm) is used for IPC, and in this case, IPC across different jails (though a file system itself also be used so...). It's not a big issue - I'll also accept that it's the operator's fault if he doesn't know sharing file systems will also share sockets and fifos on it... Yeah, what this really comes down to is IPC namespaces. We have a lot, and they have different properties, unfortunately, leading to different interactions with Jail, which is largely about namespace subsetting. Very little is about IPC between jails, but rather, whether the IPC namespace supported easy subsetting/virtualization. In the new vimage world order, it should now be easy to virtualize all of the remaining IPC namespaces (small matter of code). Robert___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: old/unupdated xterm entries in termcap db
Hello Alexander, others, * Alexander Leidinger alexan...@leidinger.net wrote: The practical attitude should be coordinated with ed@ (CCed), as he switched the console in 9-current to be an xterm, and AFAIR it does not support as much colors as the real xterm. Maybe there is a reason to not update it. So the idea is to make TERM=xterm use 256 colors? Even though I think having more colors would be awesome, I think many things would break. For example: - As Alexander mentioned, our console driver doesn't support more than 16 (well, 8 on the background) colors. Fortunately our implementation in HEAD smashes down the 256 colors back to 8, so it shouldn't cause any serious artifacts. Just a slight inconvenience. - I know Apple's Terminal.app for example doesn't support 256 colors and badly misinterprets the escape sequences, causing portions of the screen to blink (because the sequence to switch to one of the extended colors contains a 5, which is blink). But if someone is interested in updating the entries in the termcap file to something newer (but no 256 colors), please do! Patches welcome! I will even MFC it! Our current entry for xterm isn't entirely compatible with Apple's Terminal.app either. I've noticed that an Erase Line (EL, ^[[K) with xterm uses the terminal's selected attributes to blank the terminal, while Apple's implementation uses the default terminal attributes (i.e. black background). -- Ed Schouten e...@80386.nl WWW: http://80386.nl/ pgpj6FdeBfU5Y.pgp Description: PGP signature
Re: old/unupdated xterm entries in termcap db
On Thu, 10 Dec 2009 13:42:22 +0100 Ed Schouten e...@80386.nl wrote: But if someone is interested in updating the entries in the termcap file to something newer (but no 256 colors), please do! Patches welcome! I will even MFC it! Our current entry for xterm isn't entirely compatible with Apple's Terminal.app either. I've noticed that an Erase Line (EL, ^[[K) with xterm uses the terminal's selected attributes to blank the terminal, while Apple's implementation uses the default terminal attributes (i.e. black background). IIRC there was a patch in the original post which may be a good starting point. --- Gary Jennejohn ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: old/unupdated xterm entries in termcap db
* Gary Jennejohn gary.jennej...@freenet.de wrote: IIRC there was a patch in the original post which may be a good starting point. I just tried the patch, but when I run `make' in share/termcap, I get the following: | gzip -cn termcap.5 termcap.5.gz | TERM=dumb TERMCAP=dumb: ex - /store/home/ed/projects/freebsd-head/share/termcap/termcap.src /store/home/ed/projects/freebsd-head/share/termcap/reorder | script, 36: Pattern not found | script, 36: Ex command failed: pending commands discarded | *** Error code 1 | | Stop in /store/home/ed/projects/freebsd-head/share/termcap. -- Ed Schouten e...@80386.nl WWW: http://80386.nl/ pgp2KFkHlrhPx.pgp Description: PGP signature
[Patch] Updated termcap entries for xterm
* Ed Schouten e...@80386.nl wrote: I just tried the patch, but when I run `make' in share/termcap, I get the following: | gzip -cn termcap.5 termcap.5.gz | TERM=dumb TERMCAP=dumb: ex - /store/home/ed/projects/freebsd-head/share/termcap/termcap.src /store/home/ed/projects/freebsd-head/share/termcap/reorder | script, 36: Pattern not found | script, 36: Ex command failed: pending commands discarded | *** Error code 1 | | Stop in /store/home/ed/projects/freebsd-head/share/termcap. The attached patch should bring the entries up-to-date. Unfortunately it still seems the issue with Apple's Terminal.app is present, but that's just Apple's fault. Because of that, I don't see a reason (yet) to MFC this. Any testers, before I commit this patch to HEAD? -- Ed Schouten e...@80386.nl WWW: http://80386.nl/ Index: share/termcap/termcap.src === --- share/termcap/termcap.src (revision 200186) +++ share/termcap/termcap.src (working copy) @@ -2784,48 +2784,46 @@ :ts=\E_:fs=\E\\:ds=\E_\E\\:tc=screen: SW|screen-w|VT 100/ANSI X3.64 virtual terminal with 132 cols:\ :co#132:tc=screen: -# $Xorg: termcap,v 1.3 2000/08/17 19:55:10 cpqbld Exp $ +# $XTermId: termcap,v 1.78 2009/11/09 00:24:26 tom Exp $ # # Note: # termcap format is limited to 1023 characters. This set of descriptions # is a subset of the terminfo, since not all features can be fit into # that limit. The 'xterm' description supports color. The monochrome -# 'xtermm' drops color in favor of additional function keys. If you need -# both, use terminfo. +# 'xterm-mono' drops color in favor of additional function keys. If you +# need both, use terminfo. # # The 1023-character limit applies to each entry after resolving the # tc= strings. Some implementations may discount all or part of the # formatting characters in the entry (i.e., the backslash newline tab # colon). GNU termcap does not have this limit. # -# I checked the limits using ncurses captoinfo -CrTv, which prints +# I checked the limits using ncurses captoinfo -CrTUvx, which prints # the resolved length of each entry in a comment at the end - T.Dickey # -# $XFree86: xc/programs/xterm/termcap,v 3.28 2001/01/17 23:46:39 dawes Exp $ +xterm-new|modern xterm:\ + :*6=\EOF:@7=\EOF:F1=\E[23~:F2=\E[24~:K2=\EOE:Km=\E[M:\ + :k1=\EOP:k2=\EOQ:k3=\EOR:k4=\EOS:k5=\E[15~:k6=\E[17~:\ + :k7=\E[18~:k8=\E[19~:k9=\E[20~:k;=\E[21~:kH=\EOF:kI=\E[2~:\ + :kN=\E[6~:kP=\E[5~:kd=\EOB:kh=\EOH:kl=\EOD:kr=\EOC:ku=\EOA:\ + :tc=xterm-basic: # -xterm-xfree86|XFree86 xterm:\ - :k1=\EOP:k2=\EOQ:k3=\EOR:k4=\EOS:\ - :k5=\E[15~:k6=\E[17~:k7=\E[18~:k8=\E[19~:\ - :k9=\E[20~:k;=\E[21~:F1=\E[23~:F2=\E[24~:\ - :@7=\EOF:@8=\EOM:kI=\E[2~:\ - :kh=\EOH:kP=\E[5~:kN=\E[6~:\ - :ku=\EOA:kd=\EOB:kr=\EOC:kl=\EOD:Km=\E[M:tc=xterm-basic: -# # This chunk is used for building the VT220/Sun/PC keyboard variants. -xterm-basic|xterm common (XFree86):\ - :li#24:co#80:am:kn#12:km:mi:ms:xn:AX:bl=^G:\ - :is=\E[!p\E[?3;4l\E[4l\E:rs=\E[!p\E[?3;4l\E[4l\E:le=^H:\ - :AL=\E[%dL:DL=\E[%dM:DC=\E[%dP:al=\E[L:dc=\E[P:dl=\E[M:\ - :UP=\E[%dA:DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:\ - :ho=\E[H:cd=\E[J:ce=\E[K:cl=\E[H\E[2J:cm=\E[%i%d;%dH:cs=\E[%i%d;%dr:\ - :im=\E[4h:ei=\E[4l:ks=\E[?1h\E=:ke=\E[?1l\E:kD=\E[3~:kb=^H:\ - :sf=\n:sr=\EM:st=\EH:ct=\E[3g:sc=\E7:rc=\E8:\ - :eA=\E(B\E)0:as=\E(0:ae=\E(B:ml=\El:mu=\Em:up=\E[A:nd=\E[C:\ - :md=\E[1m:me=\E[m:mr=\E[7m:so=\E[7m:se=\E[27m:us=\E[4m:ue=\E[24m:\ - :ti=\E[?1049h:te=\E[?1049l:vi=\E[?25l:ve=\E[?25h:\ - :ut:Co#8:pa#64:op=\E[39;49m:AB=\E[4%dm:AF=\E[3%dm: +xterm-basic|modern xterm common:\ + :am:bs:km:mi:ms:ut:xn:AX:\ + :Co#8:co#80:kn#12:li#24:pa#64:\ + :AB=\E[4%dm:AF=\E[3%dm:AL=\E[%dL:DC=\E[%dP:DL=\E[%dM:\ + :DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:UP=\E[%dA:ae=\E(B:al=\E[L:\ + :as=\E(0:bl=^G:cd=\E[J:ce=\E[K:cl=\E[H\E[2J:\ + :cm=\E[%i%d;%dH:cs=\E[%i%d;%dr:ct=\E[3g:dc=\E[P:dl=\E[M:\ + :ei=\E[4l:ho=\E[H:im=\E[4h:is=\E[!p\E[?3;4l\E[4l\E:\ + :kD=\E[3~:kb=^H:ke=\E[?1l\E:ks=\E[?1h\E=:le=^H:md=\E[1m:\ + :me=\E[m:ml=\El:mr=\E[7m:mu=\Em:nd=\E[C:op=\E[39;49m:\ + :rc=\E8:rs=\E[!p\E[?3;4l\E[4l\E:sc=\E7:se=\E[27m:sf=^J:\ + :so=\E[7m:sr=\EM:st=\EH:te=\E[?1049l:ti=\E[?1049h:\ + :ue=\E[24m:up=\E[A:us=\E[4m:ve=\E[?12l\E[?25h:vi=\E[?25l:vs=\E[?12;25h: -# The xterm-xfree86 description has all of the features, but is not completely +# The xterm-new description has all of the features, but is not completely # compatible with vt220. If you are using a Sun or PC keyboard, set the # sunKeyboard resource to true: # + maps the editing keypad @@ -2835,68 +2833,91 @@ # + uses DEC-style control sequences for the application keypad. # xterm-vt220|xterm emulating vt220:\ - :kH=\E[4~::@7=\E[4~:*6=\E[4~:kh=\E[1~:Km=\E[M:tc=xterm-basic: + :*6=\E[4~:@7=\E[4~:K2=\EOu:Km=\E[M:kH=\E[4~:kh=\E[1~:\ + :tc=xterm-basic: xterm-24|xterms|vs100|24x80 xterm:\ - :li#24:\ - :tc=xterm: + :li#24:tc=xterm-old: xterm-65|65x80 xterm:\ - :li#65:tc=xterm: + :li#65:tc=xterm-old: xterm-bold|xterm
Re: old/unupdated xterm entries in termcap db
Quoting Gary Jennejohn gary.jennej...@freenet.de (from Wed, 9 Dec 2009 12:32:46 +0100): On Wed, 09 Dec 2009 12:29:21 +0100 Dag-Erling Sm__rgrav d...@des.no wrote: Gary Jennejohn gary.jennej...@freenet.de writes: Leonidas Tsampros ltsamp...@upnet.gr writes: Why aren't these entries updated in order to match the ones that ship with xterm? Am I missing something? Probably because xterm is under ports and termcap under src and it would not be easy to track changes in ports under src. The only practical way to keep termcap up to date would be for the committer updating the port to also check and update termcap under src. The problem with this is that most ports committers aren't authorized to make commits under src. That's not an issue - termcaps don't change all that often. We should just import the new definitions. That's a practical attitude, but it begs the question why it hasn't happened in the past. The practical attitude should be coordinated with ed@ (CCed), as he switched the console in 9-current to be an xterm, and AFAIR it does not support as much colors as the real xterm. Maybe there is a reason to not update it. Bye, Alexander. -- You can't depend on the man who made the mess to clean it up. -- Richard Nixon, 1952 http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Superpages on amd64 FreeBSD 7.2-STABLE
On Wed, Dec 9, 2009 at 9:07 AM, John Baldwin j...@freebsd.org wrote: There is lower hanging fruit in other areas in the VM that will probably be worked on first. OK, as long as somebody who knows more than me knows whats going on, that's good enough for me. :) Thanks! ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Superpages on amd64 FreeBSD 7.2-STABLE
On Wed, Dec 09, 2009 at 09:07:33AM -0500, John Baldwin wrote: On Thursday 26 November 2009 10:14:20 am Linda Messerschmidt wrote: It's not clear to me if this might be a problem with the superpages implementation, or if squid does something particularly horrible to its memory when it forks to cause this, but I wanted to ask about it on the list in case somebody who understands it better might know whats going on. :-) I talked with Alan Cox some about this off-list and there is a case that can cause this behavior if the parent squid process takes write faults on a superpage before the child process has called exec() then it can result in superpages being fragmented and never reassembled. Using vfork() should prevent this from happening. It is a known issue, but it will probably be some time before it is addressed. There is lower hanging fruit in other areas in the VM that will probably be worked on first. For me the whole threads puzzles me. Especially because vfork is often called a solution. Scenario A Parent with super page fork/exec This problem can happen because there is a race. The parent now has it's super pages fragmented permanently!? the child throws away his pages because of the exec!? Scenario B Parent with super page vfork/exec This problem won't happen because the child has no pseudo copy of the parents memory and then starts with a completely new map. Scenario C Parent with super page fork/ no exec The problem can happen because the child shares the same memory over it's complete lifetime. The parent can get it's super pages fragmented over time. I don't see a use case for scenario A, because vfork is there since over 16 years. I use fork myself, because it is easier sometimes, but people writing big programms such as squid should know better. If squid doesn't use vfork they likely have a reason. With scenario C I don't see how vfork can help, since this is not a legal case for vfork. I use quid myself, but don't know how it handles it's childs. But isn't the whole story about such slave childs that they share memory with the master? - How can vfork be solution for this case? How can fragmentation of super pages be avoided at all? I obviously don't have enough clue about this to understand those details. Hope that someone can enlighten me. -- B.Walter be...@bwct.de http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Superpages on amd64 FreeBSD 7.2-STABLE
On Thu, Dec 10, 2009 at 9:50 AM, Bernd Walter ti...@cicely7.cicely.de wrote: I obviously don't have enough clue about this to understand those details. Hope that someone can enlighten me. I think what he is saying is that they are aware that the current situation is not ideal. vfork() is suggested as a workaround, definitely not a fix. The problem is that there is limited effort available to be put into the VM subsystem and since mostly-working superpages is still *way* better than no superpages, that limited effort is best spent elsewhere at present, presumably dealing with other, bigger VM subsystem issues. Hope I got that right. :) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Superpages on amd64 FreeBSD 7.2-STABLE
Also... On Thu, Dec 10, 2009 at 9:50 AM, Bernd Walter ti...@cicely7.cicely.de wrote: I use fork myself, because it is easier sometimes, but people writing big programms such as squid should know better. If squid doesn't use vfork they likely have a reason. Actually they are probably going to switch to vfork(). They were previously not using it because they thought there was some ambiguity about whether it was going to be around long term. I actually am not a huge fan of vfork() since it stalls the parent process until the child exec()'s. To me, this case actually highlights why that's an issue. If the explanation is that stuff is happening in the parent process between fork() and the child's exec() causes the fragmentation, that's stuff that would be deferred in a vfork() regime, with unknown potential consequences. (At a minimum, decreased performance.) But that's personal and largely uninformed opinion. :) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: [Patch] Updated termcap entries for xterm
On Thu, 10 Dec 2009 14:25:54 +0100 Ed Schouten e...@80386.nl wrote: * Ed Schouten e...@80386.nl wrote: I just tried the patch, but when I run `make' in share/termcap, I get the following: | gzip -cn termcap.5 termcap.5.gz | TERM=dumb TERMCAP=dumb: ex - /store/home/ed/projects/freebsd-head/share/termcap/termcap.src /store/home/ed/projects/freebsd-head/share/termcap/reorder | script, 36: Pattern not found | script, 36: Ex command failed: pending commands discarded | *** Error code 1 | | Stop in /store/home/ed/projects/freebsd-head/share/termcap. The attached patch should bring the entries up-to-date. Unfortunately it still seems the issue with Apple's Terminal.app is present, but that's just Apple's fault. Because of that, I don't see a reason (yet) to MFC this. Any testers, before I commit this patch to HEAD? I tried it with a real xterm and mrxvt and see no regressions. However, I didn't try it with a VT as xterm. --- Gary Jennejohn ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Superpages on amd64 FreeBSD 7.2-STABLE
On Thu, 10 Dec 2009, Linda Messerschmidt wrote: Also... On Thu, Dec 10, 2009 at 9:50 AM, Bernd Walter ti...@cicely7.cicely.de wrote: I use fork myself, because it is easier sometimes, but people writing big programms such as squid should know better. If squid doesn't use vfork they likely have a reason. Actually they are probably going to switch to vfork(). They were previously not using it because they thought there was some ambiguity about whether it was going to be around long term. Well, the worst that would likely happen to vfork() is it would become an alias of fork(), and you'd be back to where you are now (or better if fork() were fixed in the meantime). I'd be more worried about the mysterious bugs which it's so easy to introduce with vfork() if you do anything at all nontrivial before exec() and accidentally touch the parent's memory. What about using posix_spawn(3)? This is implemented in terms of vfork(), so you'll gain the same performance advantages, but it avoids many of vfork's pitfalls. Also, since it's a POSIX standard function, you needn't worry that it will go away or change its semantics someday. I actually am not a huge fan of vfork() since it stalls the parent process until the child exec()'s. If you're doing so much work between vfork() and exec() that this delay is significant, then I would think you're really abusing vfork(). To me, this case actually highlights why that's an issue. If the explanation is that stuff is happening in the parent process between fork() and the child's exec() causes the fragmentation, that's stuff that would be deferred in a vfork() regime, with unknown potential consequences. (At a minimum, decreased performance.) Not necessarily. In the fork() case, presumably copy-on-write is to blame for the fragmentation. In the vfork() case, there's no copy at all. -- Nate Eldredge n...@thatsmathematics.com ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Superpages on amd64 FreeBSD 7.2-STABLE
On Mon, Dec 07, 2009 at 04:20:03PM +0100, Mel Flynn wrote: On Thursday 26 November 2009 18:11:10 Linda Messerschmidt wrote: I did not mean to suggest that we were asking for help solving a problem with squid rotation. I provided that information as background to discuss what we observed as a potential misbehavior in the new VM superpages feature, in the hope that if there is a problem with the new feature, we can help find/resolve it or, if this is working as intended, hopefully gain some insight as to what's going on. I tend to agree with this, though I don't know the nitty gritty of the implementation, it seems that: a) superpages aren't copied efficiently (at all?) on fork and probably other workloads b) vfork is encouraged for memory intensive applications, yet: BUGS This system call will be eliminated when proper system sharing mechanisms are implemented. Users should not depend on the memory sharing semantics of vfork() as it will, in that case, be made synonymous to fork(2). FYI, this comment has been removed a couple of weeks ago in HEAD and the STABLE branches. - Christian -- Christian Brueffer ch...@unixpages.org bruef...@freebsd.org GPG Key: http://people.freebsd.org/~brueffer/brueffer.key.asc GPG Fingerprint: A5C8 2099 19FF AACA F41B B29B 6C76 178C A0ED 982D pgpuTJBCAUIrV.pgp Description: PGP signature
Re: Superpages on amd64 FreeBSD 7.2-STABLE
Depending upon the IPC method being used, the fork() may be followed with calls to socket() and connect(), which may take a while. The main process will stall if you have a busy proxy and there's some temporary shortage of something which makes connect() take longer than usual, the main process will stall, potentially causing the shortage to become worse. 2c, Adrian (With his (ex-, kinda) Squid hacker hat on.) 2009/12/10 Linda Messerschmidt linda.messerschm...@gmail.com: Also... On Thu, Dec 10, 2009 at 9:50 AM, Bernd Walter ti...@cicely7.cicely.de wrote: I use fork myself, because it is easier sometimes, but people writing big programms such as squid should know better. If squid doesn't use vfork they likely have a reason. Actually they are probably going to switch to vfork(). They were previously not using it because they thought there was some ambiguity about whether it was going to be around long term. I actually am not a huge fan of vfork() since it stalls the parent process until the child exec()'s. To me, this case actually highlights why that's an issue. If the explanation is that stuff is happening in the parent process between fork() and the child's exec() causes the fragmentation, that's stuff that would be deferred in a vfork() regime, with unknown potential consequences. (At a minimum, decreased performance.) But that's personal and largely uninformed opinion. :) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Superpages on amd64 FreeBSD 7.2-STABLE
On Thursday 10 December 2009 9:50:52 am Bernd Walter wrote: On Wed, Dec 09, 2009 at 09:07:33AM -0500, John Baldwin wrote: On Thursday 26 November 2009 10:14:20 am Linda Messerschmidt wrote: It's not clear to me if this might be a problem with the superpages implementation, or if squid does something particularly horrible to its memory when it forks to cause this, but I wanted to ask about it on the list in case somebody who understands it better might know whats going on. :-) I talked with Alan Cox some about this off-list and there is a case that can cause this behavior if the parent squid process takes write faults on a superpage before the child process has called exec() then it can result in superpages being fragmented and never reassembled. Using vfork() should prevent this from happening. It is a known issue, but it will probably be some time before it is addressed. There is lower hanging fruit in other areas in the VM that will probably be worked on first. For me the whole threads puzzles me. Especially because vfork is often called a solution. Scenario A Parent with super page fork/exec This problem can happen because there is a race. The parent now has it's super pages fragmented permanently!? the child throws away his pages because of the exec!? Scenario B Parent with super page vfork/exec This problem won't happen because the child has no pseudo copy of the parents memory and then starts with a completely new map. Actually, the fact that vfork() doesn't let the parent execute until the child has called exec() also closes the race, as it were, and that was the primary reason in my mind for saying that vfork() would prevent it. Scenario C Parent with super page fork/ no exec The problem can happen because the child shares the same memory over it's complete lifetime. The parent can get it's super pages fragmented over time. I don't see a use case for scenario A, because vfork is there since over 16 years. I use fork myself, because it is easier sometimes, but people writing big programms such as squid should know better. If squid doesn't use vfork they likely have a reason. With scenario C I don't see how vfork can help, since this is not a legal case for vfork. I use quid myself, but don't know how it handles it's childs. But isn't the whole story about such slave childs that they share memory with the master? - How can vfork be solution for this case? How can fragmentation of super pages be avoided at all? In Linda's case it was a fork to run a log rotation binary, so it was A). For C) I think you would want to map the pages MAP_SHARED (or use minherit(2) with INHERIT_SHARE) in which case they would not be COW'd on fork() and you would keep the superpages. Assuming that you explicitly want to share the memory with your child processes you already have to do this to really do sharing anyway. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: [Patch] Updated termcap entries for xterm
* Gary Jennejohn gary.jennej...@freenet.de wrote:
On Thu, 10 Dec 2009 14:25:54 +0100
Ed Schouten e...@80386.nl wrote:
Any testers, before I commit this patch to HEAD?
I tried it with a real xterm and mrxvt and see no regressions. However,
I didn't try it with a VT as xterm.
I couldn't find any regressions either, so I just committed it to HEAD.
It turns out it did improve the situation for Terminal.app a little, so
I am going to MFC it after all.
Thanks for {testing,reporting,etc}!
--
Ed Schouten e...@80386.nl
WWW: http://80386.nl/
pgp7XwgDaZf3G.pgp
Description: PGP signature
Re: old/unupdated xterm entries in termcap db
On Thu, 10 Dec 2009 13:42:22 +0100, Ed Schouten e...@80386.nl wrote: Hello Alexander, others, * Alexander Leidinger alexan...@leidinger.net wrote: The practical attitude should be coordinated with ed@ (CCed), as he switched the console in 9-current to be an xterm, and AFAIR it does not support as much colors as the real xterm. Maybe there is a reason to not update it. So the idea is to make TERM=xterm use 256 colors? Even though I think having more colors would be awesome, I think many things would break. How about an xterm entry with 8 fg and 8 bg colors and a separate xterm-256color entry with 256 colors? I know, from our personal chat sessions, that the original drive behind Leonidas' patch to termcap was to make it possible for Emacs and vim to highlight/format code with more than 8 colors. This *is* useful for X11-based xterm windows but it may be less useful for vty terminals. pgpUoViQjf9bX.pgp Description: PGP signature
