Good point,

Once it's indexed, you can use Nim streams and setPosition: 
[https://nim-lang.org/docs/streams.html#setPosition%2CStream%2Cint](https://nim-lang.org/docs/streams.html#setPosition%2CStream%2Cint).

So you would:

  1. Single threaded index the file
  2. Single threaded openFile to get a file descriptor
  3. create new threads
  4. Pass the file descriptor to those threads with newFileStream (if you pass 
the path directly you will have the file opens once per thread on your machine)
  5. setPosition(i) on each thread
  6. Read until they reach the position of the previous thread
  7. make sure the result is sent back to the main thread
  8. join the threads



One thing that will be a bit painful is the thread-local GC. The strings are 
currently gc-ed unless you use gc:destructors so you can't use a global shared 
sequence with each thread updating their string fragment.

Either you compile with --gc:destructors which would allow that, or you use 
channels to communicate the final string between the main thread and the 
workers. This creates additional copy overhead.

Reply via email to