Since a lot of people get kind of stuck in their IO ways and for whatever
reasons seem to not just trust my word for it and since @Scotpip seems new to
Nim and might want some hand holding, here is some **_code to get him/others
started_** :
import times, os, memfiles, strutils # This is barIO.nim
type
Bar* = object {.packed.}
t*: int64 # ns since 1970-01-01 gives +-292 year range!
o*, h*, l*, c*, v*: float # open,hi,lo,close,vlm
Bars* = UncheckedArray[Bar]
BarFile* = object
barf: MemFile
bars: ptr Bars
nBar: int #XXX should define `[]` bounds checking this
proc bopen*(path: string): BarFile =
result.barf = memfiles.open path
result.bars = cast[ptr Bars](result.barf.mem)
result.nBar = result.barf.size div Bar.sizeof
proc close*(bf: var BarFile) = bf.barf.close
when isMainModule:
if paramCount() != 3:
stderr.write "Use: bario make|vlmTot path.bars num|idx\n"
quit(1)
let path = paramStr(2)
let num = parseInt paramStr(3)
let t0 = epochTime()
case paramStr(1)
of "make":
let f = system.open(path, fmWrite)
var bar: Bar # real data have more than `.v` ;)
for i in 1..num:
bar.v = float(i) #XXX handle disk full below
discard f.writeBuffer(bar.addr, bar.sizeof)
of "vlmTot":
var bf = bopen(path)
var total = 0.0
for i in bf.nBar - num ..< bf.nBar:
total += bf.bars[i].v #XXX bf[i].v once `[]` exists
echo "volume total of last ", num, " records is ", total
echo epochTime() - t0, " seconds"
Run
That's it - define, create, read, benchmark all in under 40 lines. Now for some
**_benchmark numbers that might inspire you to care_**. Below are times for a
Unix RAM test; 72 MB likely VERY cachable and data like this is typically
write-once read-many. So, round-trip is much less relevant than teased apart
write/read measurement.
$ ./barIO make /dev/shm/test.bars 1_500_000
0.039844 seconds
$ ./barIO vlmTot /dev/shm/test.bars 1_500_000
volume total of last 1500000 records is 1125000750000.0
0.005187 seconds
$ ./barIO vlmTot /dev/shm/test.bars 100_000
volume total of last 100000 records is 145000050000.0
0.000386 seconds # 337X faster than 260ms/2 (for RT)
# Demo some NIO stuff while we're at it:
$ ln -s /dev/shm/test.bars /dev/shm/tohlcv.Nlddddd
$ nio print /dev/shm/tohlcv.Nlddddd | tail -n100000 |
awk '{print sum += $6}' | tail -n1
145000050000 # Checks out; Just 2.65s=7000X slower.. ;)
# Ok, ok..
$ nio cut -d :5 /dev/shm/tohlcv.Nlddddd |
nio tails -t 100000 .Nd | nio pr .Nd |
awk '{print sum+=\$1}' | tail -n1
145000050000 # Also verifies; 122 millisec; 360X slower. ;)
Run
Now, 72 MB/.005187s =~ 13.9 GB/s. **_Not terrible, but also not great_**. That
machine can do like 45 GB/s, but getting 3 cores hot sucking data out of DIMMs
is out of scope. You are on your own for the last factor of 3..12X. :-D { That
will also obviously depend upon your exact calculations anyway, but VWAP or
EWMA or other things should be pretty similar to volume totals. And 5 ms may be
fast enough and who knows what your mem bw situation is anyway. }
