On 05/26/2014 04:38 PM, Peng Yu wrote: > Hi, > > I tried "sort" on some large file. But the memory usage of "sort" does > not seem to be large. This seems to be strange to me, as I think that > sort need to see all the data before completing the sorting process. > Shouldn't the memory usage of "sort" increase as the input size > increases? Thanks.
Sort takes a divide and conquer approach, by sorting parts of the input to temporary files, and then merging the results with a bounded amount of memory. sort currently defaults to using a large memory buffer to minimize overhead associated with writing and reading temp files, so you may be seeing just this large memory allocation each time. The memory allocation can be controlled with --buffer-size thanks, Pádraig.
