Re: RFC: import of posix_spawn GSoC results

2012-02-26 Thread YAMAMOTO Takashi
hi, (I'll check the other issues you pointed out ASAP) if an error happens between execve_loadvm and execve_runproc, who frees what execve_loadvm allocated? (ed_argp, ep_fd, ep_vp, vmcmds,...) the error_exit: path seems to leak p_reflock and spawn_data. exec_lock still seems deadlock-prune.

Re: RFC: import of posix_spawn GSoC results

2012-02-13 Thread Martin Husemann
On Mon, Feb 13, 2012 at 06:56:00AM +, YAMAMOTO Takashi wrote: Good catch - I'll defer the release untill the child has done it's rw_enter(). it would deadlock if in the mean time another thread does rw_enter(WRITER) on the lock. Hmm, but we can't just use the existing parent lock in

Re: RFC: import of posix_spawn GSoC results

2012-02-13 Thread Martin Husemann
On Mon, Feb 13, 2012 at 09:43:10AM +, YAMAMOTO Takashi wrote: in this particular case, i think that the child does not actually need to acquire the lock because the parent always keeps it locked. probably some comments and assertion updates are necessary, tho. Yeah, will fix it that way

Re: RFC: import of posix_spawn GSoC results

2012-02-12 Thread YAMAMOTO Takashi
hi, On Fri, Feb 10, 2012 at 11:20:38AM +, YAMAMOTO Takashi wrote: is it safe to release exec_lock? and how about p_reflock? Good catch - I'll defer the release untill the child has done it's rw_enter(). it would deadlock if in the mean time another thread does rw_enter(WRITER) on the

Re: RFC: import of posix_spawn GSoC results

2012-02-10 Thread YAMAMOTO Takashi
hi, I updated my patches after fixing a sever bug that caused fallout in lots of (apparently unrelated) atf tests by fooling ld.elf_so into insecure mode, thus not allowing LD_PRELOAD and breaking all the rump clients...

Re: RFC: import of posix_spawn GSoC results

2012-02-10 Thread Martin Husemann
On Fri, Feb 10, 2012 at 11:20:38AM +, YAMAMOTO Takashi wrote: is it safe to release exec_lock? Good catch - I'll defer the release untill the child has done it's rw_enter(). Martin

Re: RFC: import of posix_spawn GSoC results

2012-01-26 Thread YAMAMOTO Takashi
hi, On Wed, Jan 25, 2012 at 01:14:18PM +, YAMAMOTO Takashi wrote: hi, On Wed, Jan 25, 2012 at 12:50:12PM +0100, Martin Husemann wrote: A working alternative aproach to this patch is to use a vfork based emulation in userland only (FreeBSD is doing that). It is less kernel code,

Re: RFC: import of posix_spawn GSoC results

2012-01-26 Thread Joerg Sonnenberger
On Thu, Jan 26, 2012 at 03:11:14PM +, YAMAMOTO Takashi wrote: do you mean rumphijack? iirc it sits between application and libc. It overwrites the strong syscall symbols, so it covers libc itself as well. Joerg

Re: RFC: import of posix_spawn GSoC results

2012-01-26 Thread Martin Husemann
On Thu, Jan 26, 2012 at 04:16:35PM +0100, Joerg Sonnenberger wrote: It overwrites the strong syscall symbols, so it covers libc itself as well. Rump will probably want to override posix_spawn completely to pass information around (similar to the pre/post exec stuff it does). Worst case it need

Re: RFC: import of posix_spawn GSoC results

2012-01-26 Thread Joerg Sonnenberger
On Thu, Jan 26, 2012 at 04:23:14PM +0100, Martin Husemann wrote: But what do you mean vs. mutexes? The vfork child is still in the same process, so our mutexes should just work - and as soon as it is not (i.e. the exec happens), it does not matter any more. No, it is in the same address

Re: RFC: import of posix_spawn GSoC results

2012-01-26 Thread YAMAMOTO Takashi
hi, On Thu, Jan 26, 2012 at 03:11:14PM +, YAMAMOTO Takashi wrote: do you mean rumphijack? iirc it sits between application and libc. It overwrites the strong syscall symbols, so it covers libc itself as well. a libc implementation of posix_spawn can even bypass plt if it wants, can't

Re: RFC: import of posix_spawn GSoC results

2012-01-25 Thread Joerg Sonnenberger
On Wed, Jan 25, 2012 at 12:50:12PM +0100, Martin Husemann wrote: A working alternative aproach to this patch is to use a vfork based emulation in userland only (FreeBSD is doing that). It is less kernel code, but to me sounds a bit hackish and/or fragile - maybe a matter of personal taste. As

Re: RFC: import of posix_spawn GSoC results

2012-01-25 Thread YAMAMOTO Takashi
hi, On Wed, Jan 25, 2012 at 12:50:12PM +0100, Martin Husemann wrote: A working alternative aproach to this patch is to use a vfork based emulation in userland only (FreeBSD is doing that). It is less kernel code, but to me sounds a bit hackish and/or fragile - maybe a matter of personal

Re: RFC: import of posix_spawn GSoC results

2012-01-25 Thread Joerg Sonnenberger
On Wed, Jan 25, 2012 at 01:14:18PM +, YAMAMOTO Takashi wrote: hi, On Wed, Jan 25, 2012 at 12:50:12PM +0100, Martin Husemann wrote: A working alternative aproach to this patch is to use a vfork based emulation in userland only (FreeBSD is doing that). It is less kernel code, but to

Re: RFC: import of posix_spawn GSoC results

2011-12-28 Thread Joerg Sonnenberger
On Wed, Dec 28, 2011 at 04:07:45AM +, YAMAMOTO Takashi wrote: my understanding: there is no need to stop other threads as far as posix_spawn is concerned. so there is no big performace problems with a vfork-based implementation. because our current implementation of vfork suspends the

Re: RFC: import of posix_spawn GSoC results

2011-12-28 Thread Dennis Ferguson
On 28 Dec, 2011, at 19:08 , Joerg Sonnenberger wrote: On Wed, Dec 28, 2011 at 04:07:45AM +, YAMAMOTO Takashi wrote: my understanding: there is no need to stop other threads as far as posix_spawn is concerned. so there is no big performace problems with a vfork-based implementation.

Re: RFC: import of posix_spawn GSoC results

2011-12-28 Thread Joerg Sonnenberger
On Wed, Dec 28, 2011 at 07:53:00PM +, Eduardo Horvath wrote: All threaded programs have problems with #1 which is why they have thread safe libraries and locks. Or do locks not work after the vfork() call? Locks can and do interact very badly with both fork and vfork. They are not pure

Re: RFC: import of posix_spawn GSoC results

2011-12-27 Thread YAMAMOTO Takashi
hi, On Tue, Dec 20, 2011 at 02:34:22PM +, YAMAMOTO Takashi wrote: i haven't explored either. Ok, I will give it a closer look (but that will take a few days). well, i confess that i don't understand why in-kernel implementation is desirable in the first place. I don't know what

Re: RFC: import of posix_spawn GSoC results

2011-12-27 Thread Martin Husemann
On Tue, Dec 27, 2011 at 09:19:48AM +, YAMAMOTO Takashi wrote: vfork based implementation has its advantages. eg. less kernel code i'm not sure what kind of dirty libpthread changes. can you explain? We would need to guarantee thread safeness of posix_spawn() - but vfork itself is not.

Re: RFC: import of posix_spawn GSoC results

2011-12-27 Thread YAMAMOTO Takashi
hi, On Tue, Dec 27, 2011 at 09:19:48AM +, YAMAMOTO Takashi wrote: vfork based implementation has its advantages. eg. less kernel code i'm not sure what kind of dirty libpthread changes. can you explain? We would need to guarantee thread safeness of posix_spawn() - but vfork itself

Re: RFC: import of posix_spawn GSoC results

2011-12-27 Thread Martin Husemann
On Wed, Dec 28, 2011 at 04:07:45AM +, YAMAMOTO Takashi wrote: my understanding: there is no need to stop other threads as far as posix_spawn is concerned. so there is no big performace problems with a vfork-based implementation. because our current implementation of vfork suspends the

Re: RFC: import of posix_spawn GSoC results

2011-12-21 Thread Martin Husemann
On Tue, Dec 20, 2011 at 02:34:22PM +, YAMAMOTO Takashi wrote: i haven't explored either. Ok, I will give it a closer look (but that will take a few days). well, i confess that i don't understand why in-kernel implementation is desirable in the first place. I don't know what alternatives

Re: RFC: import of posix_spawn GSoC results

2011-12-20 Thread Martin Husemann
On Tue, Dec 20, 2011 at 01:31:57AM +, YAMAMOTO Takashi wrote: isn't it better to just swich to the newly created lwp and let it do these? And have the parent lwp block on a cv untill the child signals it? The parent would have to do all the copyin dance before and pass the full information

Re: RFC: import of posix_spawn GSoC results

2011-12-20 Thread YAMAMOTO Takashi
hi, On Tue, Dec 20, 2011 at 01:31:57AM +, YAMAMOTO Takashi wrote: isn't it better to just swich to the newly created lwp and let it do these? And have the parent lwp block on a cv untill the child signals it? yes. The parent would have to do all the copyin dance before and pass the

Re: RFC: import of posix_spawn GSoC results

2011-12-20 Thread Mouse
What's clean about importing the VMS process model to Unix? That's hardly the VMS process model - or at least it wasn't back in the '80s when I used VMS. In particular, in the VMS paradigm, the CLI (as close as VMS gets to the shell) and the program being run run in the same process. Well,

Re: RFC: import of posix_spawn GSoC results

2011-12-20 Thread Mouse
From the annals of the POSIX wars: the rationale for posix_spawn() was to support systems without MMUs, where fork() is expensive, and vfork() impossible. I would quibble with calling vfork() `impossible'. Perhaps I'm missing osmehting, but vfork() seems particularly well-suited to such a

Re: RFC: import of posix_spawn GSoC results

2011-12-20 Thread David Holland
On Tue, Dec 20, 2011 at 05:03:51PM -0500, Mouse wrote: From the annals of the POSIX wars: the rationale for posix_spawn() was to support systems without MMUs, where fork() is expensive, and vfork() impossible. I would quibble with calling vfork() `impossible'. Perhaps I'm missing

Re: RFC: import of posix_spawn GSoC results

2011-12-19 Thread haad
Hi, On Mon, Dec 19, 2011 at 1:23 AM, Martin Husemann mar...@duskware.de wrote: Hi folks, during this years Summer of Code Charles Zhang and I implemented the posix_spawn syscall. We are now at a point that, with some further minor cleanup and debugging, this is ready to commit. The main

Re: RFC: import of posix_spawn GSoC results

2011-12-19 Thread Martin Husemann
On Mon, Dec 19, 2011 at 11:55:20AM +0100, haad wrote: I can't see any test cases in newfiles.tar.gz. Maybe we should have posix_spawn test cases before import. As I said, there are test cases, but they are not fully ready yet. Of course they will be at the time of commit. Is RUMP

Re: RFC: import of posix_spawn GSoC results

2011-12-19 Thread Joerg Sonnenberger
On Mon, Dec 19, 2011 at 03:26:07PM -0500, Mouse wrote: This either has a very simplistic idea of what child processes do between forking and execing/exiting, or it doesn't actually buy you much. (What do I suspect it's missing? flock(). sigaction(). Writing to shared memory. To name just

Re: RFC: import of posix_spawn GSoC results

2011-12-19 Thread Thor Lancelot Simon
On Mon, Dec 19, 2011 at 09:43:57PM +0100, Martin Husemann wrote: On Mon, Dec 19, 2011 at 03:26:07PM -0500, Mouse wrote: This either has a very simplistic idea of what child processes do between forking and execing/exiting, or it doesn't actually buy you much. Well, it only offers the

Re: RFC: import of posix_spawn GSoC results

2011-12-19 Thread Martin Husemann
On Mon, Dec 19, 2011 at 04:18:38PM -0500, Thor Lancelot Simon wrote: If it doesn't perform better -- do I misunderstand, or is that in fact the case -- why dirty up the system with this superfluous interface? No, I'm just saying that it does not make the existing fork/exec path slower. We don't

Re: RFC: import of posix_spawn GSoC results

2011-12-19 Thread Thor Lancelot Simon
On Mon, Dec 19, 2011 at 10:28:42PM +0100, Martin Husemann wrote: On Mon, Dec 19, 2011 at 04:18:38PM -0500, Thor Lancelot Simon wrote: If it doesn't perform better -- do I misunderstand, or is that in fact the case -- why dirty up the system with this superfluous interface? No, I'm just

Re: RFC: import of posix_spawn GSoC results

2011-12-19 Thread Mouse
What's clean about importing the VMS process model to Unix? That's hardly the VMS process model - or at least it wasn't back in the '80s when I used VMS. In particular, in the VMS paradigm, the CLI (as close as VMS gets to the shell) and the program being run run in the same process. /~\ The

Re: RFC: import of posix_spawn GSoC results

2011-12-19 Thread Joerg Sonnenberger
On Mon, Dec 19, 2011 at 04:56:08PM -0500, Thor Lancelot Simon wrote: I prefer a clean posix_spawn over a vfork hack any time. What's clean about importing the VMS process model to Unix? vfork and multi-threading don't play well with each other. Essentially there are three options without

Re: RFC: import of posix_spawn GSoC results

2011-12-19 Thread YAMAMOTO Takashi
hi, Now what caused all the intrusiveness of the patch? Since posix_spawn, before doing the exec part, needs to manipulate file handles on behalf of the user's request for the new process, we need to pass a lwp argument to a few of the already existing file descriptor manipulating functions.

Re: RFC: import of posix_spawn GSoC results

2011-12-19 Thread Christos Zoulas
In article 20111219215608.ga12...@panix.com, Thor Lancelot Simon t...@panix.com wrote: On Mon, Dec 19, 2011 at 10:28:42PM +0100, Martin Husemann wrote: On Mon, Dec 19, 2011 at 04:18:38PM -0500, Thor Lancelot Simon wrote: If it doesn't perform better -- do I misunderstand, or is that in fact

Re: RFC: import of posix_spawn GSoC results

2011-12-19 Thread Thor Lancelot Simon
On Tue, Dec 20, 2011 at 02:43:08AM +, Christos Zoulas wrote: You realize that currently vfork() does not suspend all the threads in a threaded program, making it difficult to use... I didn't. That's kind of nifty, but I agree it's not exactly easy to put to constructive use. So the

Re: RFC: import of posix_spawn GSoC results

2011-12-19 Thread bill o gallmeister
From the annals of the POSIX wars: the rationale for posix_spawn() was to support systems without MMUs, where fork() is expensive, and vfork() impossible. The idea was to allow MMUless systems a decent means of starting other processes, without requiring the full generality of fork(). It's

RFC: import of posix_spawn GSoC results

2011-12-18 Thread Martin Husemann
Hi folks, during this years Summer of Code Charles Zhang and I implemented the posix_spawn syscall. We are now at a point that, with some further minor cleanup and debugging, this is ready to commit. The main changes are pretty local, but to avoid code duplication a few of the existing file