It's kind of hacky but you could write each column to a string or a
file and then read it back in...
forgive my crude mechanism of adding LFs. I
b5 =: 3 : 0
(, > L:0 > ,&LF each ": each i. y) fwrite 'c:/temp/foo.txt'
arr=: ". each LF cut fread 'c:/temp/foo.txt'
)
timespacex 'b5 1e3'
0.00271386 1.24134e6
timespacex 'b5 1e4'
0.0355411 1.24244e7
timespacex 'b5 1e5'
0.433461 1.241e8
timespacex 'b5 1e6'
5.35771 1.23453e9
or using the linear representation and no files (which is probably better):
data=:''
lr=:3 : '5!:5 <''y'''
appendBox =: 3 : 'data=:data,((lr y),LF)'
testAppend =: 3 : 0
data=:''
for_i. i. y do.
appendBox 'hi';1
end.
arr=: ". each LF cut data
)
timespacex 'testAppend 1e4'
0.0817675 1.91711e7
timespacex 'testAppend 1e5'
0.927324 1.92051e8
timespacex 'testAppend 1e6'
13.9085 1.91448e9
On Mon, Mar 10, 2014 at 1:12 PM, Raul Miller <[email protected]> wrote:
> Updating preallocated arrays works for fixed sized data, but J's boxed
> array implementation doesn't really fit that model.
>
> Thanks,
>
> --
> Raul
>
>
>
> On Mon, Mar 10, 2014 at 12:19 PM, Devon McCormick <[email protected]>wrote:
>
>> Raul -
>>
>> I don't know if it will help with your particular problem, but, in general,
>> a way to avoid the O(n^2) behavior of repeated concatenation is to
>> initialize a place-holder array of the right shape, then fill it in.
>>
>> For example, using Joe's "bld2" as an example of what to accomplish:
>>
>> bld3=: 3 : 0
>> (<'.') bld3 y
>> :
>> (x) (i.y)}y$a:
>> )
>> bld3 10
>> +-+-+-+-+-+-+-+-+-+-+
>> |.|.|.|.|.|.|.|.|.|.|
>> +-+-+-+-+-+-+-+-+-+-+
>> 6!:2 'bld3 1e2'
>> 4.80314e_5
>> 6!:2 'bld3 1e3'
>> 0.000126031
>> 6!:2 'bld3 1e4'
>> 0.00218933
>> 6!:2 'bld3 1e5'
>> 0.00901553
>> 6!:2 'bld3 1e6'
>> 0.0967073
>>
>> Of course, it's hard to do this if you don't know the final shape in
>> advance. It's possible to reach a compromise, complicating the code, by
>> pre-allocating blocks of boxes but I don't know how feasible this is for
>> you.
>>
>>
>> On Mon, Mar 10, 2014 at 11:22 AM, Raul Miller <[email protected]>
>> wrote:
>>
>> > Yes, exactly.
>> >
>> > I'm working on a project where I am parsing xml files and building up
>> boxed
>> > representations of the results. The final result will be on the order of
>> 30
>> > million boxes long (and have approaching 100 distinct "columns" of
>> boxes).
>> >
>> > It's been more painful than I expected, in a variety of ways. I've found
>> > new and innovative ways of crashing J (and in my copious free time I'll
>> > need to spend some time isolating those issues). For now, it looks like
>> > I'll be needing to do my xml parsing in 32 bit j602 and then assemble the
>> > results in a 64 bit version of J.
>> >
>> > But since each xml file only contributes one box to each of the "columns"
>> > it contributes to, there isn't really any better way of building the
>> > intermediate results other than using ,
>> >
>> > Hypothetically speaking, I might need to switch to a flat intermediate
>> > representation. I've done some drafts of code using flat representations
>> > and that's certainly doable (but a bit more complicated and at the time I
>> > was experimenting with them I did not see any benefit to the additional
>> > code complexity - timing was about the same).
>> >
>> > So instead, for now, I'm going to rely on "checkpointing" at various
>> orders
>> > of magnitude. With this much data I already have to deal with the fact
>> that
>> > the machines can fail for any of a variety of reasons, and computational
>> > limits and bugs in the interface to sax just get included on that list.
>> >
>> > You can't let reasons become excuses or you don't get stuff done.
>> >
>> > Thanks,
>> >
>> > --
>> > Raul
>> >
>> >
>> >
>> > On Mon, Mar 10, 2014 at 7:33 AM, Joe Bogner <[email protected]> wrote:
>> >
>> > > Is this an example of what you're referring to?
>> > >
>> > > bld2=: 3 : 0
>> > > (<'.') 4 : 'y , x' ^:y ''
>> > > )
>> > >
>> > > ts 'l=:bld2 1e2'
>> > > 0.00177792 6400
>> > > ts 'l=:bld2 1e3'
>> > > 0.0850437 20544
>> > > ts 'l=:bld2 1e4'
>> > > 8.28457 217152
>> > >
>> > > $ l
>> > > 10000
>> > >
>> > > Looping explicitly is similar
>> > >
>> > > bld4 =: 3 : 0
>> > > l=:''
>> > > for. i. y do. l=:l,(<'.') end.
>> > > )
>> > >
>> > > ts 'l=:bld4 1e4'
>> > > 5.41629 199104
>> > >
>> > >
>> > > If so, I agree there needs to be a more efficient way
>> > >
>> > > On Mon, Mar 10, 2014 at 7:05 AM, Linda Alvord <[email protected]
>> >
>> > > wrote:
>> > > >
>> > > > Raul, Since I have a math background, I'm rather fond of x and y
>> > > and am
>> > > > not in any hurry to eliminate them.
>> > > > However, I like boxes and will ponder your ideas - at least
>> > > conceptually.
>> > > >
>> > > > Thanks for all your coaching!
>> > > >
>> > > > Linda
>> > > >
>> > > >
>> > > > -----Original Message-----
>> > > > From: [email protected]
>> > > > [mailto:[email protected]] On Behalf Of bill
>> > lam
>> > > > Sent: Monday, March 10, 2014 3:30 AM
>> > > > To: Programming forum
>> > > > Subject: Re: [Jprogramming] strategies for building long lists of
>> boxes
>> > > >
>> > > > we can build internal representation (3!:1 or 3) of the box array and
>> > > > convert it using 3!:2, not sure if this can improve time or space
>> > > > efficiency.
>> > > >
>> > > > On Mon, Mar 10, 2014 at 2:37 PM, Raul Miller <[email protected]>
>> > > wrote:
>> > > >> Since using , to build boxed arrays does not currently have any code
>> > to
>> > > >> support it, time is O(n^2). In other words: inefficient for long
>> lists
>> > > of
>> > > >> boxes.
>> > > >>
>> > > >> So let's say we wanted to build lists of 30000 boxes, how could we
>> do
>> > > that
>> > > >> efficiently?
>> > > >>
>> > > >> It seems to me that the right thing to do would be: pick a threshold
>> > > > (maybe
>> > > >> 1000 boxes) and when your list gets that long, append that
>> > intermediate
>> > > >> result to a result list and start a fresh instance of the working
>> > list.
>> > > >> Repeat until done (and don't forget to append the last intermediate
>> > list
>> > > > to
>> > > >> the result).
>> > > >>
>> > > >> Conceptually speaking, this is still O(n^2). But it should also be
>> > > orders
>> > > >> of magnitude faster (at the cost of some complexity) than use of
>> > > unadorned
>> > > >> comma. (And conceptually speaking one might be able to define some
>> > kind
>> > > of
>> > > >> "infinite" representation of this algorithm which has better than
>> > O(n^2)
>> > > >> performance. Maybe O(n log n)?
>> > > >>
>> > > >> Thanks,
>> > > >>
>> > > >> --
>> > > >> Raul
>> > > >>
>> ----------------------------------------------------------------------
>> > > >> For information about J forums see
>> > http://www.jsoftware.com/forums.htm
>> > > >
>> ----------------------------------------------------------------------
>> > > > For information about J forums see
>> http://www.jsoftware.com/forums.htm
>> > > >
>> > > >
>> ----------------------------------------------------------------------
>> > > > For information about J forums see
>> http://www.jsoftware.com/forums.htm
>> > > ----------------------------------------------------------------------
>> > > For information about J forums see http://www.jsoftware.com/forums.htm
>> > >
>> > ----------------------------------------------------------------------
>> > For information about J forums see http://www.jsoftware.com/forums.htm
>> >
>>
>>
>>
>> --
>> Devon McCormick, CFA
>> ----------------------------------------------------------------------
>> For information about J forums see http://www.jsoftware.com/forums.htm
>>
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
