Re: Optimizing file I/O

[aedt]

I see that my readLine was messing with my buffer optimizations. Thanks 
KevinGolding. Udiknedormin's code is actually the fastest. Still, it's worth 
mentioning that pypy is considerably fast. I made a csv with 10000000 rows and 
10 columns with
    
    
    import os, strutils
    
    let
      rows = parseInt($paramStr 1)
      columns = parseInt($paramStr 2)
    
    var output = open($paramStr(3), fmWrite, 4096)
    
    for _ in 1 .. rows:
      for column in 1 .. columns - 1:
        output.write("field", column, ",")
      output.write("field", columns, "\n")
    

Kevin's code is similar to that of python. Here is another benchmark.
    
    
    [user0@user0-pc expressive]$ >kevingolding.csv && time ./kevingolding 
biginput.csv field8 TEST kevingolding.csv
    
    real        0m16.727s
    user        0m13.651s
    sys 0m0.867s
    [user0@user0-pc expressive]$ >nim.csv && time ./nim biginput.csv field8 
TEST nim.csv    # mine
    
    real        0m8.392s
    user        0m6.484s
    sys 0m0.803s
    [user0@user0-pc expressive]$ >python.csv && time pypy python.py 
biginput.csv field8 TEST python.csv
    
    real        0m8.020s
    user        0m5.048s
    sys 0m1.395s
    [user0@user0-pc expressive]$ >udiknedromin.csv && time ./udiknedromin 
biginput.csv field8 TEST udiknedromin.csv
    
    real        0m6.236s
    user        0m3.082s
    sys 0m0.881s