On my pc, Model2 is best, even if it takes 3% longer to complete than model 1.
jpm confirms that attention to the data structure is an important part of the solution. In Model2, it takes more than twice as long to reformat the results as to make the actual calculation. Thanks again. There is still some verification before I really can use the routines I have written, this leaves me still in the development stage, and I will revise my solution and try to incorporate your recommendations. Presently my timing is at 1.06 overall. With a drastic cut in this timing and a bit of graphics I would consider putting this thing online for a few selected users. Robert Cyr On Sun, Apr 5, 2009 at 3:59 PM, R.E. Boss <[email protected]> wrote: > Stripping the structure gives better performance. > > [sc=:scores(":,.3 4 5 6)<@,"1 ' test 55' > 0.036677156 12541376 > 0.023493124 16624128 > 0.025254491 12589888 > 0.023507607 10455808 > > ('ProjOutf';'ProjModel1';'ProjModel1a';'ProjModel2')dspl rn=:rnkng sc > +-----------+----+-------+------+------+ > |verb |rank|rlt_prf|rlt_et|rlt_sz| > +-----------+----+-------+------+------+ > |ProjOutf | 3 |1.87 |1.56 |1.20 | > +-----------+----+-------+------+------+ > |ProjModel1 | 2 |1.59 |1.00 |1.59 | > +-----------+----+-------+------+------+ > |ProjModel1a| 1 |1.29 |1.07 |1.20 | > +-----------+----+-------+------+------+ > |ProjModel2 | 0 |1.00 |1.00 |1.00 | > +-----------+----+-------+------+------+ > > ProjModel2 is 30%, 60% and 90% more efficient. > > ProjModel2=: 3 : 0 > 'pop new srv'=.y > b=.1#~(1 j.~|....@i.) c=.1{$pop > pop=.(0-.~,@|:)"2 pop > srv=.(0-.~ ,@}:@|:@}:)"2 srv > new=.1 0 2|: new > z=. ,: a=.pop > for_t. new do. > z=.z, a=.t ,"1 srv * b #"1 a > end. > 1 0 3 2|:(2#c) $"1(1#~((1&|.j.~|.@:>:)@i.)c)#^:_1"1 z > ) > > It seems rather important how the input is delivered an how the output is > to > be presented. Simplifications in one or both could result in considerable > performance improvements. > > > R.E. Boss > > > > -----Oorspronkelijk bericht----- > Van: [email protected] > [mailto:[email protected]] Namens Robert Cyr > Verzonden: woensdag 1 april 2009 23:46 > Aan: Programming forum > Onderwerp: Re: [Jprogramming] Oblique can be slow > > In my view these are great solutions. > > My ultimate aim is to evaluate the consequence of the market tsunami on the > projected improved indexing. It has been promised by unions and the plan > sponsor but I doubt they will be able to deliver. > > Initially I was not concerned about the use of memory, but in the end I > will > have to keep an eye on it, and your techniques will be quite handy. The > arrays tend to become rather large as I am trying to keep track of the year > a unit of pension is credited, the aging of the contributing population and > the characteristics of the retired as they age. > > Thank you. > > Robert Cyr > > On Tue, Mar 31, 2009 at 1:27 PM, R.E. Boss <[email protected]> wrote: > > > Some alternatives from remodeling the data are more efficient. > > > > test=: 3 : 0 > > NB. x: 0 for projection, 1 for rotation > > NB. y: number of age and service > > : > > POP=:((>:/)~i.y)*"2 (1000*i.2)+/(100*i.y)+/1+i.y NB. SERVICE AGE > > QR=: ((>:/)~i.y)*"2 (0.0001* (1000*i.2)+/(100*i.-y)+/1+i.-y) NB. > SERVICE > > AGE > > NV=: (2, y+4,0) $,10000+(100*i.y+4)+/i.y NB. ANS AGE > > projrotate`projoblique`projloop`projoutf`projmodel1`[email protected] > > POP;NV;QR > > ) > > > > ProjModel1=: 3 : 0 > > 'pop new srv'=.y > > 'srv pop'=.(i...@# |."0 1 |:)"2 srv,:pop > > srv=.(* <:@# >+/~@:i...@#)"2 srv > > 'new'=. 1 0 2 |: new > > 1 0 3 2|:(-...@i.@# |."0 1 ])"2>(,"1 2 [:}:"2 srv&*)&.>/\. &.|.pop;;/new > > ) > > > > ProjModel1a=: 3 : 0 NB. same as before but with for-loop > > 'pop new srv'=.y > > 'srv pop'=.(i...@# |."0 1 |:)"2 srv,:pop > > srv=.(* <:@# >+/~@:i...@#)"2 srv > > 'new'=. 1 0 2 |: new > > z=. ,: pop > > for_t. new do. > > z=.z, t ,"1 2 }:"2 srv * {:z > > end. > > 1 0 3 2|:(-...@i.@# |."0 1 ])"2 z > > ) > > > > [sc=:scores(":,.i.6)<@,"1 ' test 55' > > 0.33658643 76295872 > > 1.4320343 58728320 > > 0.082450888 16817472 > > 0.056301756 12590144 > > 0.040160138 16623744 > > 0.043919412 12588608 > > > > [rn=:rnkng sc > > 4 46.447614 8.3811073 6.0607076 > > 5 152.11316 35.658103 4.6651957 > > 3 2.507967 2.0530529 1.3359278 > > 2 1.2820897 1.4019313 1.000122 > > 1 1.2075074 1 1.3205387 > > 0 1 1.0936071 1 > > > > > > > > > > ('ProjRotate';'ProjOblique';'ProjLoop';'ProjOutf';'ProjModel1';'ProjModel1a' > > )dspl rn > > +-----------+----+-------+------+------+ > > |verb |rank|rlt_prf|rlt_et|rlt_sz| > > +-----------+----+-------+------+------+ > > |ProjRotate | 4 | 46.45 | 8.38 |6.06 | > > +-----------+----+-------+------+------+ > > |ProjOblique| 5 |152.11 |35.66 |4.67 | > > +-----------+----+-------+------+------+ > > |ProjLoop | 3 | 2.51 | 2.05 |1.34 | > > +-----------+----+-------+------+------+ > > |ProjOutf | 2 | 1.28 | 1.40 |1.00 | > > +-----------+----+-------+------+------+ > > |ProjModel1 | 1 | 1.21 | 1.00 |1.32 | > > +-----------+----+-------+------+------+ > > |ProjModel1a| 0 | 1.00 | 1.09 |1.00 | > > +-----------+----+-------+------+------+ > > > > ProjModel1a is the overall champion, ProjModel1 is fastest. > > > > 2-:/\(i.6) test"0[ 55 > > 1 1 1 1 1 > > > > Their output match. > > > > > > ts=: 6!:2 , 7!:2...@] > > > > scores=: 5 ts&> ] > > > > rnkng=: 3 : '([:(,~/:@/:@{.) [:(%<./)"1(,~*/)) &.|: y' > > > > dspl=: 4 : 0 > > z=: '2.0,0.2,0.2,0.2'(8!:0) y > > z=: ('rank';'rlt_prf';'rlt_et';'rlt_sz'),z > > ('verb';x),.z > > ) > > > > > > R.E. Boss > > > > > > -----Oorspronkelijk bericht----- > > Van: [email protected] > > [mailto:[email protected]] Namens Robert Cyr > > Verzonden: maandag 23 maart 2009 18:44 > > Aan: Programming forum > > Onderwerp: Re: [Jprogramming] Oblique can be slow > > > > Obviously, thank you. > > > > I like the way J encourages the manipulation of these multidimensional > > arrays. > > > > On Mon, Mar 23, 2009 at 11:10 AM, R.E. Boss <[email protected]> wrote: > > > > > Not very much faster but more elegant, replacing the for-loop: > > > > > > 10 ts 'loop test 55' > > > 0.057558227 16816768 > > > > > > 10 ts 'outf test 55' > > > 0.052024896 20978048 > > > > > > (outf test 55)-:loop test 55 > > > 1 > > > > > > with > > > > > > 'rotate oblique loop outf' =: i.4 > > > > > > and > > > > > > ProjOutf=: 3 : 0 > > > 'pop new srv'=:y > > > new=. 1 0 2 |: new NB. it makes it easy to loop on the first dimention > > > p=. pop,"3 4(new,"1(0)),"0 1"2 3(0,"2 srv) > > > q=.(*(-1 1)|.!.1 "1 2 ])/\.&.|. p > > > 1 0 2 3 |:}:"1}:"2 q > > > ) > > > > > > > > > R.E. Boss > > > > > > > > > -----Oorspronkelijk bericht----- > > > Van: [email protected] > > > [mailto:[email protected]] Namens Robert Cyr > > > Verzonden: zaterdag 21 maart 2009 15:55 > > > Aan: Programming forum > > > Onderwerp: Re: [Jprogramming] Oblique can be slow > > > > > > In this case, the simple looping solution is 20 times faster than the > > > oblique solution, as it avoids some unnecessary calculations and > > > manipulation of data. It is also makes for easier reading and > > maintenance > > > > > > Noting that the population data is basically triangular(age service) > > where > > > age-service >= 0 is of little use as we cannot easily create create and > > use > > > a triangular matrix. > > > > > > try these: > > > > > > 'rotate oblique loop' =. i.3 > > > > > > ts 'rotate test 55' > > > 0.203641 9.71485e7 > > > > > > ts 'oblique test 55' > > > 0.854339 6.29294e7 > > > > > > ---------------------------------------- > > > and the winner is... > > > ---------------------------------------- > > > ts 'loop test 55' > > > 0.0385865 1.68559e7 > > > ---------------------------------------- > > > > > > test=: 3 : 0 > > > NB. x: 0 for projection, 1 for rotation > > > NB. y: number of age and service > > > : > > > POP=:((>:/)~i.y)*"2 (1000*i.2)+/(100*i.y)+/1+i.y NB. SERVICE AGE > > > QR=: ((>:/)~i.y)*"2 (0.0001* (1000*i.2)+/(100*i.-y)+/1+i.-y) NB. > > SERVICE > > > AGE > > > NV=: (2, y+4,0) $,10000+(100*i.y+4)+/i.y NB. ANS AGE > > > projrotate`projoblique`[email protected] POP;NV;QR > > > ) > > > > > > ProjLoop=: 3 : 0 > > > 'pop new srv'=.y > > > new=. 1 0 2 |: new NB. it makes it easy to loop on the first dimention > > > p=. pop,"3 4(new,"1(0)),"0 1"2 3(0,"2 srv) > > > for_t. i._1+#p do. > > > p=.(((t+1){p)*(-1 1)|.!.1 "1 2 t{p)(t+1)}p > > > end. > > > 1 0 2 3 |:}:"1}:"2 p > > > ) > > > > > > ProjRotate=: 3 : 0 > > > NB. Population projection using rotation > > > NB. result by sex year age service will allow us to determine the > > probable > > > aging of our population. > > > NB. pop: number of active employees by sex(2), age(55=16 to 70), > > service(55 > > > = 0 to 54) > > > NB. new: number of new employees by sex(2), year of employment(50=years > > of > > > projection), age(55) > > > NB. srv: probability of survival (not dying,quitting or retiring) by > > > sex(2),age(55),service(55) > > > 'pop new srv'=.y > > > p=. pop,"2 3 (new,"1 (0)),"0 1"1 2 (0,"2 srv) NB. setup a matrix : > > > sex,year of projection, age,service > > > 'a n'=. (1 0)+ 1}.$new > > > r=. (,.)~i.a NB. to shift each slice to the left and to the > > top > > > by 1 for each year of projection > > > t=.-(0,0,a,a)+$p NB. to make space for our rotation > > > f=.2,a,1+n,n NB. to return to the original size > > > }:"2}:"1(-f){.(-r)|.!.0"1 2"2 3 */\"3 r |.!.1"1 2"2 3 t{.!.1 p > > > ) > > > > > > > > > ProjOblique=: 3 : 0 > > > 'pop new srv'=.y > > > p=. pop,"2 3 (new,"1 (0)),"0 1"1 2 (0,"2 srv) > > > 'a n'=. (1 0)+ 1}.$new > > > r=. |."1|."2*//./.\"3|."2|."1 p NB. this is the 3d oblique > > > cumulative product, by year of projection > > > b=. (1+i.a)|."0 1 ((a,n)#0 1) NB. eliminate the unused rows > of > > > the oblique > > > s=.|:n-(i.n)<./i.a NB. a rotation will position > the > > > required results > > > n{."1 s|."0 1 " 2 3 b#"1 2"2 3 r NB. we retain the correct > number > > of > > > columns > > > ) > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
