On 2013-02-07 00:41, Lee Braiden wrote:
I wasn't going to mention this as I thought the CPU usage might be trivial, but if both CPU and IO are factors, then it would probably be beneficial to have a separate IO thread/task.
This wasn't an issue in my version of the program. It took 1m55s to process the file, but then again it takes 1m44s just to read it (as shown previously).
Also, if you don't strictly need to parse the file in order, then you could divide and conquer it by breaking it into more sections/tasks. For example, if you're parsing records, you cold split the file in half, find the remaining parts of the record in the second half, move it to the first, and then process the two halves in two threads. If you've a nice function to do that split cleanly, and n cpus, then just call it some more.
Now, this could make a big difference! If only parsing out of order is acceptable in this case.
