Small file creation is a well-documented major problem (and bottleneck) in HDFS. You can either roll your own protocol, or use MapR which is about 100x faster and 1000x scalable than HDFS for this particular problem.
- HDFs file-create performance John Lilley
- Re: HDFs file-create performance M. C. Srivas
