A little thought exercise I embarked upon.
Probably not very clearly expressed, and certainly of no practical use,
but I think there are some interesting ideas:
Why must the result of $y always be a vector? What would happen if we let
it be a higher-ranked array? What would that _mean_?
Well, say that 2 3 4-:$y. That means that y is a mapping from arrays x
that have the same shape as 2 3 4 (and whose elements are natural and
bounded by 2 3 4). In other words, if (x -:&$ 2 3 4) *. -. 0 e. x<2 3 4
then (<x){y is well-defined as an atom. (I am intentially ignoring the
possibility of sentences such as ((<1 1){y) ((<,1){y) ((<1){y) and so
on; consider these sentences ill-formed for the time being.)
Generalising: if (x -:&$ $y) *. -. 0 e. x<$y then (<x){y is well-defined
as an atom.
But this sentence makes no restrictions whatsoever on the rank of x! So
it would seem that we've discovered a way to make a y whose shape is an
array of any rank. But we must be careful. Really, all we've constructed
is a strange entity y--call it an array-as-we-do-not-know-it--and, on
arrays-as-we-do-not-know-them, we have defined that: $y is well-defined,
and evaluates to an array-as-we-know-it. x{y is well-defined for some
subset of arrays-as-we-know-them x, and evaluates to an
atom-as-we-know-it.
I will call this strange creature a 4th-order array. And all
arrays-as-we-know-them are 3rd-order (this includes scalars, vectors, and
higher-rank arrays). A 4th-order array has a shape which is a 3rd-order
array, and maps from 3rd-order arrays to atoms. This suggests the
existence of lower-order arrays, and higher-order ones.
Following the same logic, then, a 3rd-order array would map 2nd-order
arrays to atoms, and its shape would be a 2nd-order array. But--for
3rd-order y, the x in (<x){y for 3rd-order y is currently also a 3rd-order
array, and $y is currently also 3rd-order.
Or is it, really? We've already established that $y must always be a
vector for ordinary, familiar, 3rd-order y; and that x must be too for
(<x){y. (A vector is a 3rd-order array y which satisfies (,1)-:$$y.) I
propose that there is another strange creature, which is a 2nd-order
array, or _list_. This is no less strange than 4th-order array, and j
does not have them. A list is very similar to a vector, though, in that
it is an ordered sequence of atoms. And there is an obvious 1:1 mapping
between vectors and lists. Therefore, when you write (<x){y, and x is a
vector, it is first _converted_ to the corresponding list, and that list
is used to index y. Similarly, when you write $y, and y is a third-order
array, its shape is actually a list; that list is _converted_ to the
corresponding vector before being returned.
Then, fairly straightforwardly, we can define the 1st-order array, or
_atom_, another strange creature. Atoms are used to index lists, and the
shape of a list is an atom. And every atom (1st-order array) also has a
counterpart in familiar 3rd-order land, as a _scalar_. (A scalar is a
3rd-order array y which satisfies ((,0)$0)-:$y.)
The obvious difference between a vector and a list is that, if we want
element 5 of y, and y is a list, we write (<5){y; but if y is a vector, we
must write (<,5){y. But to make the distinction that way misses the
point, I think. A 2nd-order array is a particular sort of array which is
restricted in that it can only contain unidirectional sequences of atoms.
A vector is of a different sort of array, which is less restricted, but it
happens to belong to a particular subset which maps to lists. (The same
relationship holds for scalars and atoms.)
Now, imagine a hypothetical version of j which could represent arrays of
orders other than 3. It is difficult to talk about such a j, because--for
instance--there might be an atom value 3 and a scalar value 3, and these
would be distinct; if I simply write '3', it is ambiguous which is
referred to. So I will err on the side of verbosity rather than
ambiguity.
One thing immediately becomes clear: the result of $y in such a j should
be the _actual_ shape of y; that is, it should have order one lower than
y. What, then is the shape of an atom? It might be an error, but I
should rather make it the _unit_. The unit is the singular value which is
an order-0 array. The unit is also its own shape. Then, we can also say
that an order n array is one which requires n applications of $ to reach a
fixed point. Personally, I think a fixed point of a unit is nicer than a
fixed point of ,1 (which is what we currently have, with all results
squashed into 3rd-order arrays), but this is a point of style.
Another nice consistency that results is that the idiom $$y evaluates to
an atom when y is a 3rd-order array. That is, it gives you the rank of y,
rather than a 1-element vector containing the rank.
Another observation: this provides a way to relate j to k; the difference
between them is one of degree, not kind. In k, everything is a list or a
scalar; that is, all values have order 1 or order 2. In j, all values
have order 3. (There is another difference, which is that k has nests
where j has boxes, but this is orthogonal.)
We can think of 3rd-order arrays as functions; for instance, an array of
rank 4 is like a function of 4 parameters. If we index such an array
using a list of length 4, that is like applying the corresponding
function, where each element of the list is associated with the
corresponding argument. By convention, these functions are curried.
Even if y has rank 4, it is legal in j to write (<1 2 0){y, even though 1
2 0 has length 3. (Previously, I said I was considering this illegal;
now, I am defining it again.) This works by currying y; producing an
array which has its first 3 arguments fixed (to 1, 2, and 0,
respectively), requiring only 1 more; because it only requires 1 more
argument, we say it has rank 1. This works well enough for 3rd-order
arrays, but what about higher-order ones? When is something a valid
'partial index', and what are the 'first arguments'?
A better definition comes from , . If p is a valid index into y (as
defined in the 3rd paragraph), then (<p){y applies the indexing procedure
as usual. Otherwise, it is the array for which (<q){(<p){y is equivalent
to (<p,q){y. (This only holds if there is q for which, given p and y, p,q
is a valid index into y; otherwise it is an index error.) This is
adequate to describe existing indexing behaviour for 3rd-order arrays .
And it works without a hitch for 4th-order arrays. For higher-order
arrays, it is simply necessary to extend , to work on higher-order arrays;
given such an extension, that, too will work without a hitch (and probably
(?) give sensible results).
I think the foregoing definition goes most of the way towards explaining
why we may index arrays using scalars; it is more than a convenience
feature, and the scalar is not simply converted to a vector. (Going the
rest of the way would require a complete and general model of , .)
Now, some questions:
1. It there any practical value to this scheme? In some respects, an
array with a non-list shape just seems like an array with a list shape,
and bells on. But I am a low-dimensional creature, and perhaps if I had
more use for very high-dimensional arrays, I would find it useful to
structure my axes (and then structure the structure of my axes, and so on,
leading to arrays of ever higher orders).
The simplest 4th-order array which is 'interesting' has shape 2 2$2; that
is, it corresponds roughly to a 4-dimensional array; that is rare enough.
The simplest 5th-order array which is 'interesting' has shape (2 2$2)$2,
that is 16-dimensional-ish. I generally have little use for arrays of
such high dimension (4 is reasonable, 16 is quite unlikely), and have not
found a linear organization stifling or problematic.
The jump from the unit to the atom seems most significant, as it allows
you to communicate any information at all; the unit carries no
information. Next in significance is the jump from atoms to lists, which
permit the communication of inherently _structured_ information. This is
good enough for many applications (and k gets along just fine); moving to
3rd-order arrays simply increases the degree of structure available, which
can be useful in some (ok, many) situations. So it would not be
surprising if the added benefit of moving to higher orders continued to
peter out, such that 3 hit a good sweet spot.
2. Is there any theoretical value to this scheme, or anything interesting
which can be described only using it? I used it to explain a number of
small inconsistencies, but that is marginal at best.
Is there any existing mathematical precedent? I know that what j calls
arrays the literature calls tensors. So I googled for 'higher-order
tensors', but only found information on what we would call higher-_rank_
arrays, not what I distinguished here as higher-_order_ arrays. I also
searched for non-cartesian coordinate systems, but could not find anything
where the components of the coordinates were not sequential and flat.
3. Are the different orders of arrays qualitatively different? E.G. a
3rd-order array can approximate both a 1st-order array (using a scalar)
and a 2nd-order array (using a vector); but a 2nd-order array can not
approximate a 1st-order array. (This is why, to my understanding, a
scalar is an array in j, but it is not a list in k.) (On the other hand,
3rd-order arrays can not represent the unit.) I think all higher-order
arrays can approximate any 3rd-order array; but do they gain any new
powers?
4. I called arrays of order 0, 1, and 2 units, atoms, and lists,
respectively. What should arrays of order 3 be called? Perhaps 'spaces'?
Is there a sensible name for 4th- or 5th-order arrays?
Is the name 'shape' still sensible if extended to different orders?
5. How would the j primitives work if extended to work on arrays of order
other than 3? What are the rules of conformability? Can you add an atom
to a scalar? A list to a matrix? A list to a vector? When can you
catenate two 5th-order arrays? How does rank work on 4th-order arrays?
6. Is there a good intuition for higher-order arrays? I can more or less
imagine what 4th-dimensional space is like (though I can't visualize it),
but I have no idea what a 'space' of shape 2 2$y is like, even though it
has the same number of 'degrees of freedom'.
Is there a well-defined notion of distance defined in higher-order
'space'? If so, is it still scalar?
Thoughts?
-E
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
