Hi, I was not talking about differences in hardware. I know using different hardware will result in different numbers.
I was worried about results would be different if you run one test alone versus if you run all tests. That would indicate a problem in the benchmark (framework) itself. But luckily, that doesn't seem to be the case. I updated the code and ran the tests again, and now the results are different. It seems there were changes recently that improved SmallFileWriteTest for Oak-Tar about 5 times; that's great. The result I got now are: https://gist.github.com/anonymous/5697099 Specially the SmallFileWriteTest seems slow with Oak. The problem doesn't seem to be actual blob handling; the profiling result shows the bottleneck is with the (few) nodes. If I change the blob size to 0 (that is, 100 nodes with the same zero-length blob each, instead of 100 nodes with 10 KB each), I get basically the same result, with both MongoMK and the Oak-Tar. Maybe it's a first sign of slow "many child nodes"? Regards, Thomas On 6/3/13 10:27 AM, "Jukka Zitting" <[email protected]> wrote: >Hi, > >On Mon, Jun 3, 2013 at 11:09 AM, Thomas Mueller <[email protected]> wrote: >> A bit weird is, when I run the tests separately I get different numbers: > >The results depend on the hardware you're using, so in general numbers >from two different environments are not directly comparable. > >> In your case, the N was 304 versus 3574 (more than 10 times different), >>in >> my case it was 528 versus 1085 (factor 2). > >Even relative numbers across fixtures can be different depending on >the varying IO/CPU/memory access costs on different environments. For >example an SSD disk will reduce the relative advantage of the TarMK >that "cheats" by mmapping the entire repository to memory. > >> How did you run the test? I will try the same command line and post my >> results. > >I'm using a ec2 m1.medium instance to keep the environment stable over >time. It would be nice to keep track of results also on different >hardware. > >The command line I've used so far is simply: > > $ java -jar oak-run-*.jar benchmark \ > ReadPropertyTest SmallFileReadTest SmallFileWriteTest \ > Jackrabbit Oak-Default Oak-Mongo Oak-Segment Oak-Tar > >BR, > >Jukka Zitting
