Re: How to divide file in Chunks for multithreaded processing?

[mratsim]

You need to do 2 passes.

The first pass will lex the files and store either in memory or on disk an 
array of the starting bytes of each line.

Then you divide those lines by the number of cores and process them.