> > > > > > > > No, wait. You are getting down to the details so I want to discuss :) > > > > OSProcess does not use popen()/system() but rather fork()+exec(). That > means > > that OSProcess has implementing the very very low building blocks that > would > > allow you do to almost everything. But then it has it's complicated > matters. > > fork() + exec() cannot be implemented via FFI (this was already > discussed in > > another thread). > > Can you link that thread? I'm curious to learn this.
It's a bit long discussion, but here it is: http://forum.world.st/ANN-Limbo-a-simple-shell-wrapper-for-NativeBoost-td4655894i20.html Is it > multi-thread related? I read [a]... > ..."The biggest problem with using fork+exec in this way is that you > can't guarantee *nothing* happens between the fork call and the exec > call. If, for example, the JVM decides to GC or move memory around, > you can have a fatal crash at the JVM process level. Because of that, > I don't recommend using fork + exec via FFI in JRuby, even though it's > pretty cool. " > > Yes, exactly. Pretty much the same. Since fork() clones and both parent and child execute same code (smalltalk vm in our case) until you call exec(), for that period of time, you will have the child running too, and that can screw up everything. ..."However, since this post I've learned of the "posix_spawn" [b] > function available on most Unix variants. It's basically fork + exec > in a single function, plus most of the typical security and IO tweaks > you might do after forking and before execing. It's definitely my > recommended alternative to fork+exec for JRuby." > *WOW. This seems to be one of the most useful recommendations I got. Thank you VERY MUCH.* I was thinking myself "how could I be the only one trying to call a fork+exec kind of function from FFI". Here is the answer. BTW...I have already started coding a FFI wrapper for such a function. I have very basic first command working :) > > And posix_spawn also seems recommended for use with the JVM [d] > Nice. > and that Windows and POSIX might have similar usage, and even better > match Windows CreateProcess than fork/exec [f] [g] [h] > > And since with 64-bit Spur we can expect very large images, spawn may > be preferred over fork/exec to minimize memory usage for creating > application subprocesses [i] > > Interesting. > > Fork versus threads section of [c] > > or should we be making use of pthread_atfork [e] ? > > Overall I learnt something new about forking in a multi-threaded > environment > "When fork() is called, only the calling thread is duplicated in the > child process." > I learned that today too :) > > [a] > http://blog.headius.com/2009/05/fork-and-exec-on-jvm-jruby-to-rescue.html > [b] > http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html > [d] http://skife.org/java/2012/01/24/java_daemonization_with_gressil.html > [c] http://repository.readscheme.org/ftp/papers/sw2002/gasbichler.pdf > [e] > http://stackoverflow.com/questions/4223553/c-does-exec-have-to-immediately-follow-fork-in-a-multi-threaded-process > [f] > http://stackoverflow.com/questions/33249253/what-is-the-difference-between-spawn-and-fork > [g] http://stackoverflow.com/questions/5883462/linux-createprocess > [h] > http://www.linuxquestions.org/questions/programming-9/fork-exec-versus-posix_spawn-835385/ > [i] > http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html > > Thanks for all the links, very useful! > > However, there is a lot of other stuff that we believe it > > could. So... what we wanted to see if it is possible is to have one only > (or > > very few) primitives (like forkAndExec) and all the rest via FFI. And > then > > all the features I listed in the first email of this thread. > > > > With that in mind, we believe that: > > > > 1) Before starting dealing with forkAndExec, we could try a first > prototype > > using popen(). That would allow us test a 100% FFI based approeach, deal > > with writing and reading from pipes via stream, face the blocking issue, > see > > a prototype on how to read async from pipes etc etc etc. If we see this > > works nicely, on a second step we can build another approach that > instead of > > popen() uses forkAndExec primitive. > > Is popen() being restricted to unidirectional part of this consideration > [j] ? > > Yep. You can either define the read more or the write mode (stdout vs stdin) but not both, since popen() opens ONE pipe. > Also I read [j] that a problem with popen is that you do not get pid > of child so you cannot wait on command to complete. Could posix_spawn > be preferrable to popen? > > Exactly. There is no standard way to get child pid from popen() so you can't do polling in image side to wait it's completion. So yeah, posix_spawn() does fix this issue as well as many many other. > and just btw, mildly interesting is using sockets rather than pipes > for interprocess communication [l] > > [j] > http://stackoverflow.com/questions/3884103/can-popen-make-bidirectional-pipes-like-pipe-fork > [k] http://stackoverflow.com/questions/6743771/popen-alternative > [l] > http://stackoverflow.com/questions/25161377/open-a-cmd-program-with-full-functionality-i-o/25177958#25177958 > > Thanks. I will check that on a second step ;) > > > > 2) We believe that there is a large number of OSProcess users that use > > OSProcess only for executing OS commands AND popen() approach would be > more > > than enough (we will do a survey about this). So, a very simply solution > via > > FFI for these users rather than having all OSProcess, Command Shell, the > > plugins etc, may be worth. > > > > What do you think? > > > >> > >> Now, given some of the difficulties seen when dealing with external > >> processes, I'd be wary of the "async" side of things. Random failures > >> seemingly linked to signal loss are certainly not something I'd like to > >> debug. > >> > > > > Indeed, it won't be easy I guess. But sometimes the process you execute > may > > take quite some time and blocking the VM for that period is not a good > > choice. So... what would you prefer to solve that? Do you prefer > image-based > > pooling as OSProcess does when IOPlugin is not used? > > > How much might your work overlap with the Threaded FFI project for Cog?... > ..."The Cog VM is single-threaded, providing green threads to the > Smalltalk level, as do most Smalltalk VMs. There is a prototype > subclass of the CoInterpreter, CoInterpreterMT, which provides a > Python-like multi-threaded VM where any number of threads can share > the VM with only one running the VM at any one time. This uses an > extremely simple low-level scheme for releasing and re-acquiring > ownership of the VM designed by David Simmons. This results in any > and all FFI calls being non-blocking since there is such a low > overhead to making a non-blocking FFI call that all FFI calls can be > non-blocking. Eliot has worked on this, implementing the prototype > and the reentrant FFI plugin it requires. But this needs to be > revived. It would be a really powerful enhancement." > > Well, Eliot said I may need to give a hand to them in order to finish the Threaded FFI. However, I am still not 100% certain if the threaded FFI calls to read from pipes would be worth. Eliot said that yes. Anyway, once I have the first version working I will digest this and goes to the next level and see how I can make things better than polling, if polling proves to be a problem. > cheers -ben > > P.S. Just another article warning against mixing fork and > multiple-threads (which it seems posix_spawn may avoid)... > > http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them > > Thanks for the reference. Cheers, -- Mariano http://marianopeck.wordpress.com
