I was going to suggest shared memory as an alternate way to co-ordinate processes but it looks like you've already looked into that. I don't think I can be of much help as the parallel apps I've done have all been very simple by design and lack of necessity to do otherwise. That's why I mentioned the socket communication idea - it would be a step up in sophistication compared to what I've done.
Shared memory would be bigger step up but also, in some sense, a step back because of the requirement that the processes be able to share memory. The multi-process and the socket methods are amenable to spreading the work across networked machines whereas the shared memory way is not. Also, any kind of real-time requirement complicates things as well. On Mon, Jul 15, 2013 at 9:01 AM, I.T. Daniher <[email protected]>wrote: > Multiprocessing is a decent solution for concurrency, but alas there are > some situations where message passing is insufficient. > > LibUSB has an asynchronous API ( > http://libusb.sourceforge.net/api-1.0/group__asyncio.html), and you're > pretty much stuck using it if you wish to achieve anywhere close to the > throughput USB is capable of reaching on a modern computer, or if you wish > to provide your application with a modicum of jitter-resistance by having > an outstanding transfer queue. Using this API requires an event handler to > be called either frequently or in a separate thread. I don't think > multiprocess could be made to work here (one process for running data > handling callbacks, one process for polling the handler function) due to > the requirements for the polling task sharing memory with the application > task. > > Alternatively, on Linux and OSX, LibUSB can spit out file descriptors that > can be monitored via poll/select in the main event loop, something along > the lines of how sdasync works. Unfortunately, I don't think the libJ event > loop / jconsole event loop is exposed, at least not to the extent of > registering additional file descriptors for monitoring. > > Any thoughts here? I've been able to prototype my application well enough > with the synchronous API, but I'd like to bring jitter* down and throughput > up, and it looks like this is going to be sufficiently difficult to pull > off in J. > > Thanks! > -- > Ian > > * I'm concerned about the 1-10ms stalls frequently demonstrated by USB bulk > transfers under moderate system load, not the hundred microseconds in > either direction that I've observed to be typical of an interpreted J > program. > > On Wed, Jul 10, 2013 at 7:05 AM, Jack Andrews <[email protected]> wrote: > > > On 10 July 2013 02:42, I.T. Daniher <[email protected]> > > wrote in thread "ymacs": > > > .... This is perhaps more a petition for better > > > threading / async support in J in general > > > > As Raul said in the ymacs thread, concurrency (async) can be acheived > > using multiple processes [multiprocessing]. Trendy things like .NET > > and Java do multithreading. > > > > A language implementation that doesn't support multithreading is not > > necessarily deficient. For example, through the language 'k' (an > > APL inspired language), kdb+ has blown away competitors in a STAC M3 > > benchmark [focused on finance and automated trading] partially with a > > new multiprocessing feature, which performs better than multithreading > > in this context. > > > > Multiple processes can communicate using sockets (as Raul illustrated) > > Using sockets involves "message passing" which is a lot easier than > > using threads and locks. Erlang, a language built around concurrency, > > uses message passing. > > > > On 10 July 2013 02:42, I.T. Daniher <[email protected]> wrote: > > > Unfortunately, the collection of data isn't always a low-overhead task, > > so > > >... workflow consists of writing an experiment, running the script, and > > viewing > > > / processing the resulting data. > > > ... J IDEs to be ill suited to a more realtime/interactive paradigm due > > to UI > > > requests, and in some cases, interaction with the text editor window > > > itself, being blocked by "long-running" data collection. > > > > Here's a recipe for multiprocessing your example: > > - start a J process that listens for 'experiments' on socket with port > 4321 > > - start the J IDE and send an experiment message (eg: send 'sin 0.2 * > > i.60') to port 4321. > > - continue to refine your experiment in the IDE > > - when the the result message arrives, the J IDE can pop up a plot. . > > > > In addition to Raul's pointers, this might be a good place to start: > > http://www.jsoftware.com/jwiki/Studio/Socket%20Driver > > > > Ta, Jack. > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > -- Devon McCormick, CFA ^me^ at acm. org is my preferred e-mail ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
