@Ram: You are entirely correct, I made the exact same mistakes of mixing up Large and minor compaction. By looking closely, what I see is that at around 200 HFiles per region it starts minor compacting files per group of 10 HFiles. The "problem" seems that this minor compacting never stops even when there are about 20 HFiles left. It just keep on going and on taking more and more time (I guess because the files to compact are getting bigger).
Of course in parallel we keep on adding more and more data. @J-D: "It seems to me that it would be better if you were able to do a single load for all your files." Yes, I agree.. but that is not what we are testing, our use case is to use 1min batch files.
