Well, 65000*65000 is 4,225,000,000 ~= 4 billion. If your distances are integral and integers are 4 bytes, then that's already >16GB. If your distances are floating point or integers are 8 bytes wide (I think they are on 64-bit architectures?), then that's >32GB.
Even if you had enough RAM to make this feasible, it would probably be a waste, given that the output is only plotted. So probably easiest to do some sampling to reduce the data set. Our local mathematicians can advise you on that better than I can. One question though: you indicate the plot is 3D, but your array only has 2 dimensions (65e3 x 65e3). Where's the 3rd dimension? -Dan -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Charles Holder Sent: Wednesday, May 25, 2011 7:38 PM To: [email protected] Subject: Re: [Jprogramming] Programming Digest, Vol 68, Issue 43 - array limit I ask because I am shaping an array 65000 by 65000 for ploting LY distances in Boeing Graph (3D). > From: [email protected] > Subject: Programming Digest, Vol 68, Issue 43 > To: [email protected] > Date: Wed, 25 May 2011 22:22:39 +0800 > > Send Programming mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > http://jsoftware.com/cgi-bin/mailman/listinfo/programming > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Programming digest..." > > > Today's Topics: > > 1. J arrays (Charles Holder) > 2. Re: J arrays (Roger Hui) > 3. Sorting (Gian Medri) > 4. Re: Sorting (Pablo Landherr) > 5. Re: Sorting (bill lam) > 6. Re: Sorting (R.E. Boss) > 7. Re: Sorting (bill lam) > 8. Re: J arrays (Dan Bron) > 9. Re: Sorting (Devon McCormick) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 25 May 2011 05:23:23 +0000 > From: Charles Holder <[email protected]> > Subject: [Jprogramming] J arrays > To: <[email protected]> > Message-ID: <[email protected]> > Content-Type: text/plain; charset="iso-8859-1" > > > What is the max items an array can have? > > > ------------------------------ > > Message: 2 > Date: Tue, 24 May 2011 22:47:47 -0700 > From: Roger Hui <[email protected]> > Subject: Re: [Jprogramming] J arrays > To: Programming forum <[email protected]> > Message-ID: <[email protected]> > Content-Type: text/plain; charset=us-ascii > > _1+2^n-1 where n is the number of bits in a machine word. > (n is 32 or 64). > > > > ----- Original Message ----- > From: Charles Holder <[email protected]> > Date: Tuesday, May 24, 2011 22:24 > Subject: [Jprogramming] J arrays > To: [email protected] > > > What is the max items an array can have? > > > > ------------------------------ > > Message: 3 > Date: Wed, 25 May 2011 10:00:42 +0200 > From: Gian Medri <[email protected]> > Subject: [Jprogramming] Sorting > To: [email protected] > Message-ID: <[email protected]> > Content-Type: text/plain; charset=UTF-8 > > Hi! > 1) I have a problem with sorting with the Swedish alphabet that contains 3 > extra letters after z. > Example: > /:~'qwe?rta?as?sdf' > aadefqrsstw?????? > > Perhaps it depends on the limitation of "a. "? > > 2) Often I encounter the necessity to replace element x by element y in an > array. > I solved the problem in an APL fashion such as: > repl=: 4 : '($y)$ (1{x) (((0{x)=,y)#i.*/$y)},y' NB. replace x1 in array y > by x2 > > This verb works OK for any array, but I wonder if there is a better J-way > to do it. > > Thanks > Gian Medri > > > ------------------------------ > > Message: 4 > Date: Wed, 25 May 2011 10:38:04 +0200 > From: Pablo Landherr <[email protected]> > Subject: Re: [Jprogramming] Sorting > To: Programming forum <[email protected]> > Message-ID: <[email protected]> > Content-Type: text/plain; charset=UTF-8 > > And I get > > 'qwe?rta?as?sdf'/:'abcdefghijklmnopqrstuvwxyz???' i. 'qwe?rta?as?sdf' > aadefqrsstw? > 9!:14 '' > j602/2008-03-03/16:45 > > On Wed, May 25, 2011 at 10:00 AM, Gian Medri <[email protected]> wrote: > > > Hi! > > 1) I have a problem with sorting with the Swedish alphabet that contains 3 > > extra letters after z. > > Example: > > /:~'qwe?rta?as?sdf' > > aadefqrsstw?????? > > > > Perhaps it depends on the limitation of "a. "? > > > > 2) Often I encounter the necessity to replace element x by element y in an > > array. > > I solved the problem in an APL fashion such as: > > repl=: 4 : '($y)$ (1{x) (((0{x)=,y)#i.*/$y)},y' NB. replace x1 in array y > > by x2 > > > > This verb works OK for any array, but I wonder if there is a better J-way > > to do it. > > > > Thanks > > Gian Medri > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > ------------------------------ > > Message: 5 > Date: Wed, 25 May 2011 16:42:28 +0800 > From: bill lam <[email protected]> > Subject: Re: [Jprogramming] Sorting > To: [email protected] > Message-ID: <[email protected]> > Content-Type: text/plain; charset=utf-8 > > You may try converting it to wide character first > > /:~ucp'qwe?rta?as?sdf' > aadefqrsstw??? > > or if you want utf8 back > > /:~&.ucp'qwe?rta?as?sdf' > aadefqrsstw??? > > if x is a rank-1 vector of 2 atom, I think it can be (untested) > > ($y) $ (1{x) (I.(0{x)=,y) } ,y > > please also refer to the verb charsub , that amend idiom should be supported by > special code. > > ???, 25 ??? 2011, Gian Medri ?????(?): > > Hi! > > 1) I have a problem with sorting with the Swedish alphabet that contains 3 > > extra letters after z. > > Example: > > /:~'qwe?rta?as?sdf' > > aadefqrsstw?????? > > > > Perhaps it depends on the limitation of "a. "? > > > > 2) Often I encounter the necessity to replace element x by element y in an > > array. > > I solved the problem in an APL fashion such as: > > repl=: 4 : '($y)$ (1{x) (((0{x)=,y)#i.*/$y)},y' NB. replace x1 in array y > > by x2 > > > > This verb works OK for any array, but I wonder if there is a better J-way > > to do it. > > > > Thanks > > Gian Medri > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > -- > regards, > ==================================================== > GPG key 1024D/4434BAB3 2008-08-24 > gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 > > > ------------------------------ > > Message: 6 > Date: Wed, 25 May 2011 11:11:14 +0200 > From: "R.E. Boss" <[email protected]> > Subject: Re: [Jprogramming] Sorting > To: "'Programming forum'" <[email protected]> > Message-ID: <[email protected]> > Content-Type: text/plain; charset="UTF-8" > > rpl2a=: ] - (-/ , 0:)@[ {~ {.@[ i. ] > > by Hui from http://www.jsoftware.com/pipermail/programming/2007-July/007303.html is the most elegant known (to me). > > If x and y are from a small domain (e.g. characters), then rpl3a is faster and leaner: http://www.jsoftware.com/pipermail/programming/2007-July/007302.html > > rpl3a=: 4 : 0 > 'x0 x1'=. x > ((x1,a.) {~ (x0,a.) i. ]) y > ) > > > R.E. Boss > > > > -----Oorspronkelijk bericht----- > > Van: [email protected] [mailto:programming- > > [email protected]] Namens Gian Medri > (...) > > > 2) Often I encounter the necessity to replace element x by element y in an > > array. > > I solved the problem in an APL fashion such as: > > repl=: 4 : '($y)$ (1{x) (((0{x)=,y)#i.*/$y)},y' NB. replace x1 in array y by x2 > > > > This verb works OK for any array, but I wonder if there is a better J-way to do > > it. > > > > > ------------------------------ > > Message: 7 > Date: Wed, 25 May 2011 18:50:57 +0800 > From: bill lam <[email protected]> > Subject: Re: [Jprogramming] Sorting > To: [email protected] > Message-ID: <[email protected]> > Content-Type: text/plain; charset=utf-8 > > What you see in jconsole or ide is _not_ the same as that inside J engine. For > example, the character ? is one character in ide but it is two bytes in J > engine, furthermore, J engine has no idea that these two bytes should be > regarded as a single character, or know that you want them to be > interpreted in utf8 encoding, or anything special. So nearly all string > operations on these non-ascii characters will fail. > > There are some articles on unicodes in jwiki, please enter 'unicode' in the > search box to locate them. They should provide some background materials. > > ???, 25 ??? 2011, Pablo Landherr ?????(?): > > And I get > > > > 'qwe?rta?as?sdf'/:'abcdefghijklmnopqrstuvwxyz???' i. 'qwe?rta?as?sdf' > > aadefqrsstw? > > 9!:14 '' > > j602/2008-03-03/16:45 > > > > On Wed, May 25, 2011 at 10:00 AM, Gian Medri <[email protected]> wrote: > > > > > Hi! > > > 1) I have a problem with sorting with the Swedish alphabet that contains 3 > > > extra letters after z. > > > Example: > > > /:~'qwe?rta?as?sdf' > > > aadefqrsstw?????? > > > > > > Perhaps it depends on the limitation of "a. "? > > > > > > 2) Often I encounter the necessity to replace element x by element y in an > > > array. > > > I solved the problem in an APL fashion such as: > > > repl=: 4 : '($y)$ (1{x) (((0{x)=,y)#i.*/$y)},y' NB. replace x1 in array y > > > by x2 > > > > > > This verb works OK for any array, but I wonder if there is a better J-way > > > to do it. > > > > > > Thanks > > > Gian Medri > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > -- > regards, > ==================================================== > GPG key 1024D/4434BAB3 2008-08-24 > gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 > > > ------------------------------ > > Message: 8 > Date: Wed, 25 May 2011 12:02:44 +0000 > From: "Dan Bron" <[email protected]> > Subject: Re: [Jprogramming] J arrays > To: "J Programming" <[email protected]> > Message-ID: > <1182149784-1306324965-cardhu_decombobulator_blackberry.rim.net-1490800668-@ b18.c25.bise6.blackberry> > > Content-Type: text/plain > > Which is to say: exactly as much as your machine can handle (note the formula below also defines the limits of memory pointers on modern architectures). > > Which should also tell you that the max number of items depends on the size (in memory) of 1 item. The smallest J item is one byte (a literal atom), so you could have 4 billion of those on a 32 bit machine (modulo OS taxes). But if your items were, say, 4-byte integers, you'd max out at 1 billion. And if your items were tuples of 3 integers (R,G,B pixels, for example) you'd top out at ~300 million, etc. > > > Of course, J also supports sparse arrays, which have the same theoretical limit (2^n etc) but much higher practical limits, as they only store the relevant parts of arrays (of course, if all the items of your array are relevant, you're back to the same memory limit). > > Why do you ask? > > -Dan > > PS: Roger's formula refers to the limit of a single dimension of a J array (the number of "items", as opposed to "atoms"). But J arrays are multidimensional, and each dimension could (in theory) have that many items, and of course the array's total volume is the product of all its dimensions. The max # of dimensions for a J array also shares Roger's limit, so the max shape J's internals can handle is > > shape =. $~ _1+2^n-1 > > And correspondingly its volume would be */shape which is shape^shape or about 2^(n-1)^2 atoms. Which of course no machine can handle. But it does let you play silly games like carrying payload data around in the shape of an (empty) array :) > > And while we're discussing theory, it's worth pointing out that the specification of J (the Dictionary) imposes no limits on array size. So in theory, a different implementation of J could have arrays that stretched back onto disk (like a RDBMS), or across mutliple disks, or across multiple networked machines, or across the globe.... > > The notation wouldn't change. Just the implementation. > > > > Please excuse typos; composed on a handheld device. > > -----Original Message----- > From: Roger Hui <[email protected]> > Sender: [email protected] > Date: Tue, 24 May 2011 22:47:47 > To: Programming forum<[email protected]> > Reply-To: Programming forum <[email protected]> > Subject: Re: [Jprogramming] J arrays > > _1+2^n-1 where n is the number of bits in a machine word. > (n is 32 or 64). > > > > ----- Original Message ----- > From: Charles Holder <[email protected]> > Date: Tuesday, May 24, 2011 22:24 > Subject: [Jprogramming] J arrays > To: [email protected] > > > What is the max items an array can have? > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > > ------------------------------ > > Message: 9 > Date: Wed, 25 May 2011 10:22:03 -0400 > From: Devon McCormick <[email protected]> > Subject: Re: [Jprogramming] Sorting > To: Programming forum <[email protected]> > Message-ID: <[email protected]> > Content-Type: text/plain; charset=UTF-8 > > I like the boolean variant of replace: > > replace1by1=: 3 : 0 > 'toreplace replacement target'=. y > (toreplace=target)}target,:replacement > ) > > replace1by1 'e';'X';'Replace one element by another' > RXplacX onX XlXmXnt by anothXr > > $mat=. ><;._1 ' Replace one element by another' > 5 7 > > replace1by1 'e';'X';mat > RXplacX > onX > XlXmXnt > by > anothXr > > On Wed, May 25, 2011 at 6:50 AM, bill lam <[email protected]> wrote: > > > What you see in jconsole or ide is _not_ the same as that inside J engine. > > For > > example, the character ? is one character in ide but it is two bytes in J > > engine, furthermore, J engine has no idea that these two bytes should be > > regarded as a single character, or know that you want them to be > > interpreted in utf8 encoding, or anything special. So nearly all string > > operations on these non-ascii characters will fail. > > > > There are some articles on unicodes in jwiki, please enter 'unicode' in the > > search box to locate them. They should provide some background materials. > > > > ???, 25 ??? 2011, Pablo Landherr ?????(?): > > > And I get > > > > > > 'qwe?rta?as?sdf'/:'abcdefghijklmnopqrstuvwxyz???' i. 'qwe?rta?as?sdf' > > > aadefqrsstw? > > > 9!:14 '' > > > j602/2008-03-03/16:45 > > > > > > On Wed, May 25, 2011 at 10:00 AM, Gian Medri <[email protected]> > > wrote: > > > > > > > Hi! > > > > 1) I have a problem with sorting with the Swedish alphabet that > > contains 3 > > > > extra letters after z. > > > > Example: > > > > /:~'qwe?rta?as?sdf' > > > > aadefqrsstw?????? > > > > > > > > Perhaps it depends on the limitation of "a. "? > > > > > > > > 2) Often I encounter the necessity to replace element x by element y in > > an > > > > array. > > > > I solved the problem in an APL fashion such as: > > > > repl=: 4 : '($y)$ (1{x) (((0{x)=,y)#i.*/$y)},y' NB. replace x1 in > > array y > > > > by x2 > > > > > > > > This verb works OK for any array, but I wonder if there is a better > > J-way > > > > to do it. > > > > > > > > Thanks > > > > Gian Medri > > > > ---------------------------------------------------------------------- > > > > For information about J forums see http://www.jsoftware.com/forums.htm > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > -- > > regards, > > ==================================================== > > GPG key 1024D/4434BAB3 2008-08-24 > > gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > > > -- > Devon McCormick, CFA > ^me^ at acm. > org is my > preferred e-mail > > > ------------------------------ > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > > End of Programming Digest, Vol 68, Issue 43 > ******************************************* ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
