May worth to try a) use async b) use caching, so you don't have to process those files every time.
- fetching data from many small .txt files tcheran
- fetching data from many small .txt files cblake
- fetching data from many small .txt files cmc
- fetching data from many small .txt files alexeypetrushin