I don't have a good grasp of order, but a different practical interpretation of non-list shapes:
1. You mentioned distinguishing atoms from lists of 1 element. J already manages this well, and making an atomic shape result would not be of any help, IMO. 2. A table shape could be a code for ragged arrays, or arrays of arrays. The following shape: 3 2 2 4 0 0 Would be 2 arrays, the first is shape 3 2 2, and the 2nd is shape 4. Trailing 0s would be fills. This precludes the ultra rare need of needing an "embedded" shape 4 0 or shape 4 0 0 in your data. A list of 3 strings/lists with individual lengths of 2 3 4 would have the shape: 2 3 4 If there is an extention of the meaning of shape, I'd prefer it be in this direction rather than ultra high dimensional math that hurts my head. On Thursday, March 3, 2022, 05:13:27 a.m. EST, Elijah Stone <[email protected]> wrote: A little thought exercise I embarked upon. Probably not very clearly expressed, and certainly of no practical use, but I think there are some interesting ideas: Why must the result of $y always be a vector? What would happen if we let it be a higher-ranked array? What would that _mean_? Well, say that 2 3 4-:$y. That means that y is a mapping from arrays x that have the same shape as 2 3 4 (and whose elements are natural and bounded by 2 3 4). In other words, if (x -:&$ 2 3 4) *. -. 0 e. x<2 3 4 then (<x){y is well-defined as an atom. (I am intentially ignoring the possibility of sentences such as ((<1 1){y) ((<,1){y) ((<1){y) and so on; consider these sentences ill-formed for the time being.) Generalising: if (x -:&$ $y) *. -. 0 e. x<$y then (<x){y is well-defined as an atom. But this sentence makes no restrictions whatsoever on the rank of x! So it would seem that we've discovered a way to make a y whose shape is an array of any rank. But we must be careful. Really, all we've constructed is a strange entity y--call it an array-as-we-do-not-know-it--and, on arrays-as-we-do-not-know-them, we have defined that: $y is well-defined, and evaluates to an array-as-we-know-it. x{y is well-defined for some subset of arrays-as-we-know-them x, and evaluates to an atom-as-we-know-it. I will call this strange creature a 4th-order array. And all arrays-as-we-know-them are 3rd-order (this includes scalars, vectors, and higher-rank arrays). A 4th-order array has a shape which is a 3rd-order array, and maps from 3rd-order arrays to atoms. This suggests the existence of lower-order arrays, and higher-order ones. Following the same logic, then, a 3rd-order array would map 2nd-order arrays to atoms, and its shape would be a 2nd-order array. But--for 3rd-order y, the x in (<x){y for 3rd-order y is currently also a 3rd-order array, and $y is currently also 3rd-order. Or is it, really? We've already established that $y must always be a vector for ordinary, familiar, 3rd-order y; and that x must be too for (<x){y. (A vector is a 3rd-order array y which satisfies (,1)-:$$y.) I propose that there is another strange creature, which is a 2nd-order array, or _list_. This is no less strange than 4th-order array, and j does not have them. A list is very similar to a vector, though, in that it is an ordered sequence of atoms. And there is an obvious 1:1 mapping between vectors and lists. Therefore, when you write (<x){y, and x is a vector, it is first _converted_ to the corresponding list, and that list is used to index y. Similarly, when you write $y, and y is a third-order array, its shape is actually a list; that list is _converted_ to the corresponding vector before being returned. Then, fairly straightforwardly, we can define the 1st-order array, or _atom_, another strange creature. Atoms are used to index lists, and the shape of a list is an atom. And every atom (1st-order array) also has a counterpart in familiar 3rd-order land, as a _scalar_. (A scalar is a 3rd-order array y which satisfies ((,0)$0)-:$y.) The obvious difference between a vector and a list is that, if we want element 5 of y, and y is a list, we write (<5){y; but if y is a vector, we must write (<,5){y. But to make the distinction that way misses the point, I think. A 2nd-order array is a particular sort of array which is restricted in that it can only contain unidirectional sequences of atoms. A vector is of a different sort of array, which is less restricted, but it happens to belong to a particular subset which maps to lists. (The same relationship holds for scalars and atoms.) Now, imagine a hypothetical version of j which could represent arrays of orders other than 3. It is difficult to talk about such a j, because--for instance--there might be an atom value 3 and a scalar value 3, and these would be distinct; if I simply write '3', it is ambiguous which is referred to. So I will err on the side of verbosity rather than ambiguity. One thing immediately becomes clear: the result of $y in such a j should be the _actual_ shape of y; that is, it should have order one lower than y. What, then is the shape of an atom? It might be an error, but I should rather make it the _unit_. The unit is the singular value which is an order-0 array. The unit is also its own shape. Then, we can also say that an order n array is one which requires n applications of $ to reach a fixed point. Personally, I think a fixed point of a unit is nicer than a fixed point of ,1 (which is what we currently have, with all results squashed into 3rd-order arrays), but this is a point of style. Another nice consistency that results is that the idiom $$y evaluates to an atom when y is a 3rd-order array. That is, it gives you the rank of y, rather than a 1-element vector containing the rank. Another observation: this provides a way to relate j to k; the difference between them is one of degree, not kind. In k, everything is a list or a scalar; that is, all values have order 1 or order 2. In j, all values have order 3. (There is another difference, which is that k has nests where j has boxes, but this is orthogonal.) We can think of 3rd-order arrays as functions; for instance, an array of rank 4 is like a function of 4 parameters. If we index such an array using a list of length 4, that is like applying the corresponding function, where each element of the list is associated with the corresponding argument. By convention, these functions are curried. Even if y has rank 4, it is legal in j to write (<1 2 0){y, even though 1 2 0 has length 3. (Previously, I said I was considering this illegal; now, I am defining it again.) This works by currying y; producing an array which has its first 3 arguments fixed (to 1, 2, and 0, respectively), requiring only 1 more; because it only requires 1 more argument, we say it has rank 1. This works well enough for 3rd-order arrays, but what about higher-order ones? When is something a valid 'partial index', and what are the 'first arguments'? A better definition comes from , . If p is a valid index into y (as defined in the 3rd paragraph), then (<p){y applies the indexing procedure as usual. Otherwise, it is the array for which (<q){(<p){y is equivalent to (<p,q){y. (This only holds if there is q for which, given p and y, p,q is a valid index into y; otherwise it is an index error.) This is adequate to describe existing indexing behaviour for 3rd-order arrays . And it works without a hitch for 4th-order arrays. For higher-order arrays, it is simply necessary to extend , to work on higher-order arrays; given such an extension, that, too will work without a hitch (and probably (?) give sensible results). I think the foregoing definition goes most of the way towards explaining why we may index arrays using scalars; it is more than a convenience feature, and the scalar is not simply converted to a vector. (Going the rest of the way would require a complete and general model of , .) Now, some questions: 1. It there any practical value to this scheme? In some respects, an array with a non-list shape just seems like an array with a list shape, and bells on. But I am a low-dimensional creature, and perhaps if I had more use for very high-dimensional arrays, I would find it useful to structure my axes (and then structure the structure of my axes, and so on, leading to arrays of ever higher orders). The simplest 4th-order array which is 'interesting' has shape 2 2$2; that is, it corresponds roughly to a 4-dimensional array; that is rare enough. The simplest 5th-order array which is 'interesting' has shape (2 2$2)$2, that is 16-dimensional-ish. I generally have little use for arrays of such high dimension (4 is reasonable, 16 is quite unlikely), and have not found a linear organization stifling or problematic. The jump from the unit to the atom seems most significant, as it allows you to communicate any information at all; the unit carries no information. Next in significance is the jump from atoms to lists, which permit the communication of inherently _structured_ information. This is good enough for many applications (and k gets along just fine); moving to 3rd-order arrays simply increases the degree of structure available, which can be useful in some (ok, many) situations. So it would not be surprising if the added benefit of moving to higher orders continued to peter out, such that 3 hit a good sweet spot. 2. Is there any theoretical value to this scheme, or anything interesting which can be described only using it? I used it to explain a number of small inconsistencies, but that is marginal at best. Is there any existing mathematical precedent? I know that what j calls arrays the literature calls tensors. So I googled for 'higher-order tensors', but only found information on what we would call higher-_rank_ arrays, not what I distinguished here as higher-_order_ arrays. I also searched for non-cartesian coordinate systems, but could not find anything where the components of the coordinates were not sequential and flat. 3. Are the different orders of arrays qualitatively different? E.G. a 3rd-order array can approximate both a 1st-order array (using a scalar) and a 2nd-order array (using a vector); but a 2nd-order array can not approximate a 1st-order array. (This is why, to my understanding, a scalar is an array in j, but it is not a list in k.) (On the other hand, 3rd-order arrays can not represent the unit.) I think all higher-order arrays can approximate any 3rd-order array; but do they gain any new powers? 4. I called arrays of order 0, 1, and 2 units, atoms, and lists, respectively. What should arrays of order 3 be called? Perhaps 'spaces'? Is there a sensible name for 4th- or 5th-order arrays? Is the name 'shape' still sensible if extended to different orders? 5. How would the j primitives work if extended to work on arrays of order other than 3? What are the rules of conformability? Can you add an atom to a scalar? A list to a matrix? A list to a vector? When can you catenate two 5th-order arrays? How does rank work on 4th-order arrays? 6. Is there a good intuition for higher-order arrays? I can more or less imagine what 4th-dimensional space is like (though I can't visualize it), but I have no idea what a 'space' of shape 2 2$y is like, even though it has the same number of 'degrees of freedom'. Is there a well-defined notion of distance defined in higher-order 'space'? If so, is it still scalar? Thoughts? -E ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
