One idea I was thinking about was to have a new class of object that referred to data in a file on disk, and which had all the standard methods of matrices and arrays, i.e., subsetting ("["), dim, dimnames, etc. The object in memory would only store the array attributes, while the actual array data (the elements) would reside in a file. When some extraction method was called, it would access data in the file and return the appropriate data. With sensible use of seek operations, the data access could probably be quite fast. The file format of the object on disk could possibly be the standard serialized binary format as used in .RData files. Of course, if the object was larger than would fit in memory, then trying to extract too large a subarray would exhaust memory, but it should be possible to efficiently extract reasonably sized subarrays. To be more useful, one would want want apply() to work with such arrays. That would be doable, either by creating a new method for apply, or possibly just for aperm.
Some difficulties that might arise could have to do with functions like "typeof" and "mode" and "storage.mode" -- what should these return for such an object? Such a class would probably break the common relationships such as x[1] having the same storage mode as x. I don't know if difficulties like these would ultimately make such a "virtual array" class unworkable.
Does anyone have any opinions as to the merits of this idea? Would there be any interest in seeing such a class in R?
-- Tony Plate
______________________________________________ [EMAIL PROTECTED] mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
