Why would array programming require a large team? The problem with large teams is coordination of many people. I would think that if anything array programming groups would be smaller. On Sep 27, 2015 11:41 AM, "Devon McCormick" <[email protected]> wrote:
> At a Kdb talk I attended recently, Fintan Quill mentioned that K does > multi-processing but not multi-threading. He didn't go into details on > this but, based on what they say in this talk and what I've heard > elsewhere, the reason seems obvious: multi-threading is nearly impossible > to get to work properly. > > As they say in this talk about how "Shared Mutable State" in multi-threaded > programming should be some of the most feared words in computing, Martin > Thompson elaborates "It's a complete nightmare even with really, really > good people." He then goes on to add "I haven't come across any > multi-threaded application where people are mutating the same space, that > isn't bug-ridden." > > So, in short, don't do this. Don't even try to do this: fine-grained, > multi-threaded parallelism is probably a waste of time with the current > level of tools we have for it. > > Multi-processing, on the other hand, is dead-easy to do in J for anything > that can be parallelized at a coarse-grained level, which is many if not > most things. I've written about this extensively over the past few years > (all at jcode.jsoftware.com/wiki/): > User:Devon_McCormick/ParallelizedJCodeExamples, > Community/Conference2012/Talks/ParallelSimulationInJ, > NYCJUG/2011-04-12/RedOrBlackGameSimulation > . These latter two give detailed examples on how to achieve > multi-processor parallelization. > > On another note, there's been some mention of processes like Agile for > speeding up the development process. Keep in mind that the context for > these kinds of "methodologies" is an idea that a small team might be five > people. In the array-programming world, this would be a large team. > > > On Sat, Sep 26, 2015 at 7:19 PM, Don Guinn <[email protected]> wrote: > > > Given a fork (f g h) f and h can be processed in parallel. But if f and h > > have side effects (shared global names that are modified) then they > should > > not run in parallel. A programming practice that should be avoided > > anyway. This is not a pipeline, but multiple processors could be used. > But > > even not running them in parallel I see nothing stating which, f or h, is > > run first. So one should avoid code that depended on the order of f and h > > execution anyway. > > > > On Sat, Sep 26, 2015 at 4:42 PM, Raul Miller <[email protected]> > > wrote: > > > > > On Sat, Sep 26, 2015 at 3:04 PM, Vijay Lulla <[email protected]> > > wrote: > > > > quite easy. But what I'm very unclear about is how does one do > > > > pipelining in J? Say we have functions f, g, and h (all used > > > > monadically) and it is applied like f@g@h y and function g was > > > > particularly costly how can we parallelize (maybe the user has to > > > > program it himself or the interpreter can do some cost analysis [like > > > > query planning in SQL databases]) it to make it faster? Isn't this > > > > what the presenters were mentioning when they were using the example > > > > of airbus pipeline system? > > > > > > "Pipelining" seems to describe a variety of topics. > > > > > > https://en.wikipedia.org/wiki/Pipeline_(computing) > > > > > > So, I would have to say that there is no general technique. If f, g > > > and h are black boxes, you cannot pipeline them. If you want to > > > reschedule g or make it more efficient, you'll need to know details > > > about g. The more you know, the greater the odds are that you can do > > > it (or the important parts of it) differently, in a more efficient > > > manner. > > > > > > That said, I should also point out that a lot of the automated query > > > planning systems are workarounds for bogus constraints underlying the > > > sql standard. That effort could have gone in much more useful > > > directions if people hadn't bought into those ideas. (But at this > > > point, it has turned into a multi-billion dollar industry, so it's not > > > going away. And there are some applications where the flaws are not > > > all that important.) > > > > > > Thanks, > > > > > > -- > > > Raul > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > > > -- > Devon McCormick, CFA > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
