I may be a biased judge, but I think the code is **_pretty straightforward_**.
My more broad abstract-speak may be harder to follow without background, but I
do try to be more concrete. If you give that code some time, I think you'll do
fine. :-) People may be interested to hear your perf numbers, but I don't know
**_if Windows has a RAM filesystem_**. That likely only matters for the "write"
phase anyway, and as mentioned that is the least important phase.
If what you care about is local machine back testing performance then **_no
other approach_** will really even approach just casting the object type. You
can even squirrel away { like nuts! :-) } the indices into different time
regions to back test on a few for debugging before burning your more recent
data (though of course you should always paper trade true out of sample before
risking much money...) or make an easy way to look them up.
If you had **_actually big data_** like terabytes then you could get some juice
out of compressing it where you probably would lose random access (though you
could compress the binary output in "chunks" like 2019, 2020, etc and just use
those as your makeshift interval segmentation). If you ever do go that way, I
recommend Zstd because its parallel decompression gives the highest output
rates and you are clearly in a read-many scenario. You would still be well
advised to just `readBuffer` some cast addr rather than this marshal/unmarshal
pointless (for your application) activity circus wasting 99% of your time.
There are some Unix filesystems that offer builtin transparent compression
(Btrfs, ZFS). No idea if Windows has this ability. But your data seems small.
Maybe big for Excel with some 2500x penalty, but not big for a systems language
like Nim.
I am also unfamiliar with **_any port of Windows_** to a big endian CPU, but I
really don't follow it much. Araq might know. IBM POWER cpus have been the last
CPUs doing it that way for a long time, AFAIK. Apple's M1 seems ARM-related
which probably means little endian, but I don't know for sure. It's become so
rare that I don't even know if anyone's run the Nim test suite on a big endian
CPU at all in the past N years. Araq might know that, too. Maybe someone has a
13+ year old Mac PowerPC or tests in some CPU simulator. Anyway, others can
chime in but my best guess is that "on Windows implies little endian, full
stop." and this is simply not a real concern for you.
**_You can always run this to find out_** :
import strutils
var bytes = "01234567"
var i = cast[ptr int64](bytes[0].addr)[]
echo toHex(i) # little 3736353433323130; big: 3031323334353637
Run
Hex 0x30..0x37 are the ASCII codes for characters '0'..'7'. That cast with a
`[]` at the end is Nim-ese for interpreting that memory as an integer and
loading it into a register. Then you convert that integer to Hex. Since there
is no ambiguity of the order of bytes in the string, this tells you what you
want to know. In my experience you would have to **_really hunt for a
machine_** to get that big endian answer, though. Gone are the days of Sun
Microsystems SPARC and all the others.
Because the Internet began in the 70s & 80s when big endian CPUs were more
popular, the binary fields of internet protocol headers are typically all big
endian. Because of **_that_** , there is a lot of support for converting
to/from "network order" and "host order" which for you would be big & little
endian. Because of how often that happens and how sensitive people are to
network software stack performance, Intel even has some ["bswap" CPU
instructions](https://www.felixcloutier.com/x86/bswap) (for byte swap) tailored
for the task, though these days SIMD/SSE permute instructions might be even
faster. So, you could convert at some blazing speed - maybe full L1 cache
bandwidth of like 200 GB/s which practically means simply very limited by your
actual IO be it memory/disk/network. I really doubt you will ever need to
convert..like ever, though. By the time you do, there will probably be a `nio
bswap` to do it for you if you stay in a simple CPU-type packed object format
which I doubt you will have a pressing need to leave. :-)
