Wow. Never thought of such a simple design. Leveraging off the existing
file system is a great idea. Ultimately, however, I wouldn't do it that way
for two main reasons:
1. The overhead of opening/creating, read/writing, and closing a file with
each record access is huge when many records are accessed.
2. I've already written component file systems and would be able to
leverage off of pre-existing / pre-tested code. (I suppose I'd have to
find it though...)
Great idea.
Given the ubiquity of SQL databases, and the fact that you get keyed or
indexed access for free, I think I would go that way. I see it looking
like this:
CREATE TABLE my_table (
numeric_key bigint,
string_key character varying(100),
short_data character varying(1000),
long_data text
);
numeric_key and string_key would both be indexed. You would use the one
that made the most sense on a record by record basis. short_data would be
used for data that fits in that space. long_data for the larger stuff.
I have APL code that converts arbitrary nested arrays to/from plain strings.
I feel strongly that a data persist mechanism is an absolutely necessity.
This would be an easy and portable solution.
Thanks.
Blake
On Tue, Apr 1, 2014 at 11:05 AM, Nick Lobachevsky <[email protected]>wrote:
> As for file systems, I did something like this years ago for a client
> who (A) refused to have his data tied up in a proprietary format, and
> (B) wanted to share data across Dyalog APL and APL2000. This dates
> back to the Windows 8.3 FAT file system days, this could be far better
> (or worse) with the right file system. File locking ([]FHOLD) was not
> needed and not implemented. File access control was implemented with
> Novell or whatever the client was using at the time.
>
> - A conceptual APL component file is a Directory.
>
> - File system components are individual physical files named something
> where component 42 would be "00000042.ncf", .ncf stood for "Native
> Component File"
>
> - File metadata was stored in Component Zero, i.e. "00000000.ncf"
>
> - []FCREATE was mkdir, []FAPPEND created a new file, []FREPLACE
> rewrote the entire file, []FDROP erased individual files, []FERASE
> deleted the entire directory, etc.
>
> - []FSIZE info was stored in the metadata component
>
> - Component data was something like this:
> (thorn (shape encoded data), ([]DR data), shape data), 82 []DR data
>
> Monadic []DR returned the encoding of the data type, 82 was char, 163
> was 16 bit integer, etc. Dyadic 82 []DR converted the data content to
> character. Character data was always stored translated to Dyalog's
> []AV. To re-materialise the data, you would split the component from
> its header and data and use []DR to bring the data back to its
> original type. Not all data types were available on both systems. I
> did not deal with nested or mixed arrays.
>
> Performance was adequate.
>
> I suppose you can design a database this way.
>
