Hmm... Interesting. I expected there to be a better improvement from smaller blocks.
So it's really just IOPS (and block size does not matter), in which case, yes, HBase checksum will save you 50% IOPS for each data block (and since index blocks are cache) it'll save 50% total IOPS. ________________________________ From: Jan Schellenberger <[email protected]> To: [email protected] Sent: Saturday, February 1, 2014 9:38 PM Subject: Re: Slow Get Performance (or how many disk I/O does it take for one non-cached read?) I've experimented with the block size. Here are results: 4k - 60reads/sec - 1.2GB totalStaticIndexSize 8k - 80reads/sec - 660MB totalStaticIndexSize 16k - 90reads/sec - 330MB totalStaticIndexSize and previously 64k - 80reads/sec - ~100mb totalStaticIndexSize Also, I turned off caching and you are correct, the index blocks seem to be cached always - the blockCachedSize grows until it reaches totalStaticIndexSize and then stops growing. If you turn caching on, it will grow until the maxHeap * blockCacheSize (.4 in my case). I saw no performance difference between caching off/on so I guess off is fine. Yes, I always do a major_compact before testing. I think this probably concludes my question - I will try to upgrade to a newer hbase version to get the CRC32/HDFS check fix and we will probably have to evaluate SSD's. Cheers, everyone. -- View this message in context: http://apache-hbase.679495.n3.nabble.com/Slow-Get-Performance-or-how-many-disk-I-O-does-it-take-for-one-non-cached-read-tp4055545p4055582.html Sent from the HBase User mailing list archive at Nabble.com.
