@cblake Many thanks for your intriguing post. It certainly makes sense, and opens up the prospect of very low-cost and rapid caching to disk which might change my design somewhat.
I will cogitate and digest and try to understand the code you have linked. But more broadly I'd ask you to reflect on the yawning gap between a confident systems programmer like yourself, and someone like myself who has mainly worked with high-level line-of-business development. Ask me to design you a nicely abstracted database or validated data-flow and I'm your man, but all this talk of bits and bytes leaves me a bit at sea. I'm working through a book on C to try and widen my education, but systems work is still very unfamiliar. So if you can spare the time to offer a few more kindergarten-level pointers or even some example code it would be very helpful. Judging by the readership the thread is attracting, they are others who are benefiting too. For my project, memory safety isn't critical - the serialisation is mainly for backtesting on my own machine, so if it blows up I'm on hand to fix it. The work is very CPU intensive, so speed is the priority. As for endianness, for production I'll be running a conservative subset of the functionality on a remote VPS - probably Windows as that's more familiar to my partners in this little venture. I'm not very familiar with the underlying hardware they use - is there a significant risk that this might clash with the Intel Xeons I'm running locally? I've never heard of this being an issue. But then again, most people aren't using the low-end tricks you are proposing...
