I used sparse boxed arrays when I was attempting the Netflix challenge. The video rental company Netflix made available a list of about half a million users - each represented simply by an integer - a list of about 17,000 movies - by title - and sets of user ratings of particular movies on a scale of one to five.
The objective was to estimate existing users' ratings of some movies they hadn't rated. In theory, you could represent this as a half million by 17,000 table of integers with zero for no rating or one to five for a given user's rating of a given movie. However, this table is intractably large but it is also sparse. I'd written some preliminary code in J6 to attempt to solve this problem but didn't get very far. When I returned to the code many months later, I was getting errors on parts of it I could have sworn used to work. Eventually, I figured out it was because I'd used sparse boxed arrays that were not disallowed in an earlier version of J 6 but not a later one. Some discussion of the Netflix challenge is at http://www.jsoftware.com/jwiki/NYCJUG/2007-01-09 . On Sat, Nov 10, 2012 at 8:50 AM, Raul Miller <[email protected]> wrote: > I have only vague memory of this implementation. > > But J7 is open source, so in principle any of us could write support > for this type. > > Personally, though, I'd want to have a deep understanding of the > issues that control the performance of J's memory management in the > context of boxed types before I would want to tackle "sparse boxed". > > I'd also want to have in mind some good ideas of why this would be useful. > > -- > Raul > > On Wed, Nov 7, 2012 at 12:42 PM, Devon McCormick <[email protected]> > wrote: > > I'd not thought of the parts of speech as a type system before. On the > > data types, has "sparse boxed" been (re-) implemented in J7? It was > > present in J6 for a while, then removed. > > > > > > On Tue, Nov 6, 2012 at 9:01 PM, Raul Miller <[email protected]> > wrote: > > > >> I have had my attention called to > >> > >> > http://existentialtype.wordpress.com/2011/03/19/dynamic-languages-are-static-languages/ > >> repeatedly, and perhaps discussions of types are interesting. > >> > >> J has two systems of types: > >> > >> J's static type system has nouns, verbs, adverbs and conjunctions (J's > >> copula might also be thought of as having static type, but copula > >> cannot be used in isolation so I will not consider them to be a part > >> of the type system). > >> > >> J's dynamic type system currently has boolean, literal, integer, > >> floating point, complex, boxed, extended integer, rational, sparse > >> boolean, sparse literal, sparse integer, sparse floating point, sparse > >> complex, sparse boxed, symbol and unicode. Note that J's gerunds do > >> not have their own type -- they use the boxed type. > >> > >> Additionally, J's static type system is extensible in a restricted, > >> but open ended sense: Adverbs and Conjunctions can produce any of the > >> main four static types, and this result is a part of the complete > >> static type. In some sense, this is like a macro, or a define-syntax > >> system, it's constrained -- in essence all you can do here is obtain > >> additional arguments (beyond the "zero extra arguments" that a noun > >> result can reference). > >> > >> J's dynamic type system is not extensible, in current implementations > >> of J. This is [arguably, but in my opinion] a good thing. > >> > >> We could imagine [for example] a facility (analogous to f.) which > >> transforms a J verb according to some static type inference system > >> which has its origins in the types supported by J's dynamic type > >> system. Here, a constraint on dynamic type within the verb could have > >> its influence propagated to the beginning of the verb -- if at some > >> point a value must be a member of a restricted set of types, the verb > >> could refuse to run if an argument were not a member of an appropriate > >> set (thus avoiding the need to perform those tests in an iterated > >> context). See also: http://www.chilton.com/~jimw/apl85.html > >> > >> That said, note that a type language is typically a low power language > >> with guaranteed termination semantics ("grammar"), which allows for > >> some [necessarily incomplete] early consistency tests on large > >> programs. I do not use J to write large programs, so I do not miss > >> this kind of meta-language in J. I mostly use J to understand > >> concepts, and I am usually only interested in concise J expressions. > >> Some other people's needs will of course be different. > >> > >> Finally, note that a key feature of J's type system is its regularity, > >> which is inspired a variety of mathematical issues. Math, in general, > >> is not necessarily regular, but a regular > >> > >> I believe the above description is consistent with typical uses of > >> these words and phrases. But note that strictly speaking the types > >> which I have labeled as "static" are static within an instance of an > >> expression. This is unambiguous for tacit expressions, but explicit > >> expressions are re-parsed during execution in the current > >> implementation of J. > >> > >> FYI, > >> > >> -- > >> Raul > >> ---------------------------------------------------------------------- > >> For information about J forums see http://www.jsoftware.com/forums.htm > >> > > > > > > > > -- > > Devon McCormick, CFA > > ^me^ at acm. > > org is my > > preferred e-mail > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > -- Devon McCormick, CFA ^me^ at acm. org is my preferred e-mail ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
