Hi,
i currently use this code to divide work between worker threads:
var linePtr: seq[pointer]
for line in memSlices(memfiles.open($args["--inputFile"], mode = fmRead)):
linePtr.add(line.data)
let batchSize = (linePtr.len / parseInt($args["--cores"])).ceil().toInt()
let f = system.open($args["--inputFile"])
{.push experimental: "parallel".}
parallel:
for i in 0..(parseInt($args["--cores"])-1):
spawn worker(f, linePtr.at((batchSize * i)), batchSize)
{.pop.}
Run
and this worker function:
proc worker(fd: File, offset, size: int) {.thread.} =
let fs = newFileStream(fd)
fs.setPosition(offset)
var i = 0
while i < size:
echo i, " ", size, " ", fs.atEnd(), " ", fs.readLine()
inc(i)
Run
now the thing is: the program throws after a random amount of lines an IOError;
when i wrap the while-loop as follows:
while i < size:
try:
echo i, " ", size, " ", fs.atEnd(), " ", fs.readLine()
except IOError:
echo i, " ", size, " ", fs.atEnd()
inc(i)
Run
i see that fs.atEnd() returns true even if i shuld have lines left. Why is this
happening and how do i crcumvent it?
Also a little side question: i currently divide into chunks of size
ceil(total_lines/core_count) this means the last chunk is bigger or smaller
than the previous ones. Is it possible to use fs.atEnd() as in while i<size and
not fs.atEnd()? Because i didn't mange to get this to work either :(.
Thx in advance, ff