Thank you, this works.
Good point,
Once it's indexed, you can use Nim streams and setPosition:
[https://nim-lang.org/docs/streams.html#setPosition%2CStream%2Cint](https://nim-lang.org/docs/streams.html#setPosition%2CStream%2Cint).
So you would:
1. Single threaded index the file
2. Single threaded openFile to get
Hm, so you mean something along this?
var linePtr: seq[pointer]
var input = memfiles.open("file.txt", mode = fmRead)
for line in memSlices(input):
linePtr.add(line.data)
Run
and then i divide the length of linePtr by my core count. This gives me the
You need to do 2 passes.
The first pass will lex the files and store either in memory or on disk an
array of the starting bytes of each line.
Then you divide those lines by the number of cores and process them.
I want to split a file into multiple pieces and give one piece to one thread. I
figured memfiles should be the best option, but I'm struggling as to how I
should use them to divide my file into multiple slices.
The file itself contains in each line a single datum.
Does anyone have any hints on
