I fixed up this system using 5!:5 and ". to convert to and from strings, so
that it now handles arbitrary arguments (as long as the verb is executed
monadically).
NB. (u parallelize n) y
NB. assumes (u -: u"_1) y
NB. divides y into n parts and executes u on them in parallel.
parallelize=:2 :0
parts=.(3 :'5!:5 <''y''')&.> (</.~ [:<.n* (%~i.)@#) y
inputs=.(<jpath '~user\input') ,&.> (":&.>i.n) ,&.> <'.txt'
parts fwrite&.> inputs
(u buildtemplate) runkernels n
WAITTIME=.0.05
result=.$0
for_i. i.n do.
whilst. r-:_1 do.
6!:3 WAITTIME
r=.fread (jpath '~user\output') , (":i) , '.txt'
end.
if. result-:$0 do. result=.".r
else. result=.result,".r
end.
end.
ferase@> (<jpath'~user\') ,&.> (<'.txt') ,~&.> ('input';'output';'kernel')
,&.>/ (":&.>i.n)
result
)
NB. run first y kernels
runkernels=:3 :0
(fread jpath '~user\template.txt') runkernels y
:
JEXE=: jpath '~bin\jconsole.exe '
for_n. ":"0 i.y do.
'input output'=.(<'''',jpath'~user\') ,&.> ('input';'output') ,&.>
<n,'.txt'''
(x rplc '{input}';input;'{output}';output) fwrite xflnm=. jpath
'~user\kernel',n,'.txt'
fork JEXE,xflnm
end.
)
NB. built a template string to use for runkernels.
NB. assumes u is monadic.
buildtemplate=:1 :0
verbdef=. LF ,~ 'f=.',5!:5<'u'
verbdef, '{output} fwrite~ 5!:5 <''a'' [a=.f ". fread {input}'
)
For a fairly short computation, gains are modest relative to regular
execution, but we can see that if we account for the overhead in writing to
files from parallelizing, speed roughly doubles (I am using a dual-core
processor).
6!:2 '+/@("."0@":)@:!"0 (i.4)+10000x'
1.76074
6!:2 '+/@("."0@":)@:!"0 parallelize 1 (i.4)+10000x'
2.44481
6!:2 '+/@("."0@":)@:!"0 parallelize 2 (i.4)+10000x'
1.25969
6!:2 '+/@("."0@":)@:!"0 parallelize 4 (i.4)+10000x'
1.22391
Marshall
-----Original Message-----
From: Marshall Lochbaum [mailto:[email protected]]
Sent: Thursday, April 07, 2011 10:17 PM
To: 'Programming forum'
Subject: RE: [Jprogramming] "Red or Black" game simulation
Using methods like yours to parallelize execution, I made a conjunction that
divides a verbs execution into several processes. Currently verbs can only
take and return strings because these can be written to text files.
Hopefully this will be a step on the way to making those "embarrassingly
parallel" problems (ie. ((u -: u"_1) y) in J) two or four times faster.
NB. (u parallelize n) y
NB. assumes (u -: u"_1) y
NB. divides y into n parts and executes u on them in parallel.
parallelize=:2 :0
parts=.(</.~ [:<.n* (%~i.)@#) y
inputs=.(<jpath '~user\input') ,&.> (":&.>i.n) ,&.> <'.txt'
parts fwrite&.> inputs
(u buildtemplate) runkernels n
WAITTIME=.0.05
result=.$0
for_i. i.n do.
whilst. r-:_1 do.
6!:3 WAITTIME
r=.fread (jpath '~user\output') , (":i) , '.txt'
end.
if. result-:$0 do. result=.,:r
else. result=.result,r
end.
end.
ferase@> (<jpath'~user\') ,&.> (<'.txt') ,~&.> ('input';'output';'kernel')
,&.>/ (":&.>i.n)
result
)
NB. run first y kernels
runkernels=:3 :0
(fread jpath '~user\template.txt') runkernels y
:
JEXE=: jpath '~bin\jconsole.exe '
for_n. ":"0 i.y do.
'input output'=.(<'''',jpath'~user\') ,&.> ('input';'output') ,&.>
<n,'.txt'''
(x rplc '{input}';input;'{output}';output) fwrite xflnm=. jpath
'~user\kernel',n,'.txt'
fork JEXE,xflnm
end.
)
NB. built a template string to use for runkernels.
NB. assumes u is monadic.
buildtemplate=:1 :0
verbdef=. LF ,~ 'f=.',5!:5<'u'
verbdef, '{output} fwrite~ f fread {input}'
)
Proof of concept:
|. parallelize 3 ('abcd','1234',:'abcd') dcba
4321
dcba
6!:2 '(''1''[6!:3@:1:) parallelize 2 (''abcd'',:''1234'')'
1.2025
The code is pretty messy right now, and if anyone can suggest better methods
that would be great. Main problems now:
- Only text can be processed. What I would like for this are generic verbs
that convert arbitrary J nouns to and from text format.
- Related to this, two dimensional or more data gets flattened:
|. parallelize 2 ('abcd','1234',:'abcd') 4321dcba
dcba
- Is there a better method of determining when the procedure is done? My
script just keeps checking the file at intervals until it exists, then gives
the results.
Also, more tests would be great (ie. verbs that actually require processing
time, unlike 6!:3). I haven't done anything more than the few examples I've
shown here.
Marshall
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Devon McCormick
Sent: Monday, April 04, 2011 12:24 AM
To: J-programming forum
Subject: [Jprogramming] "Red or Black" game simulation
I've posted some work I've done recently on calculating the score obtained
by using a simple strategy in the "Red or Black" game about which there was
some traffic on the forum a month or so ago:
http://www.jsoftware.com/jwiki/NYCJUG/2011-04-12/RedOrBlackGameSimulation
.
--
Devon McCormick, CFA
^me^ at acm.
org is my
preferred e-mail
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
