fetching data from many small .txt files

[treeform]

Have you though of maybe doing an initial step puts all of your small files 
into a database or some sort of in memory store? Using a bunch of small files 
as normal runtime operation is not a great idea. You are at the mercy of 
inefficient file system no matter which way you slice it.