It sounds like you will have many regular files (i.e., random access/seekable
inputs as opposed to things like Unix pipes). On Linux with glibc,
memfiles.open is probably the fastest approach which uses memchr internally to
find line boundaries. E.g. (right from memfiles documentation),
import memfiles
for line in lines(memfiles.open("foo")):
inc(i)
Your mileage on this may vary from OS to OS or libc to libc. I have no idea
which if any Microsoft/Windows versions have well-optimized libc memchr()
implementations.
