[jira] [Updated] (HBASE-12935) Does any one consider the performance of HBase on SSD?
[ https://issues.apache.org/jira/browse/HBASE-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Lee updated HBASE-12935: -- Description: Some features of HBase doesn't mathch features of SSD. Such as compaction is harmful for SSD life span. (was: Some features of HBase doesn't mathch features of SSD. Such as comapction is harmful for SSD life span.) > Does any one consider the performance of HBase on SSD? > --- > > Key: HBASE-12935 > URL: https://issues.apache.org/jira/browse/HBASE-12935 > Project: HBase > Issue Type: Improvement >Reporter: Liang Lee > > Some features of HBase doesn't mathch features of SSD. Such as compaction is > harmful for SSD life span. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12935) Does any one consider the performance of HBase on SSD?
Liang Lee created HBASE-12935: - Summary: Does any one consider the performance of HBase on SSD? Key: HBASE-12935 URL: https://issues.apache.org/jira/browse/HBASE-12935 Project: HBase Issue Type: Improvement Reporter: Liang Lee Some features of HBase doesn't mathch features of SSD. Such as comapction is harmful for SSD life span. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-7404) Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
[ https://issues.apache.org/jira/browse/HBASE-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13667490#comment-13667490 ] Liang Lee commented on HBASE-7404: -- Thanks,Ted. I modified the configuration and the lruCacheSize is 1G ,the bucketCacheSize isv 1G. The Heapszie is set to 4G while the machine memory is 8G . I use PerformanceEvaluation tool based on hbase to test the Performance. Before this path, the randomRead speed is about 6hours ,which used 10 clients to read 1000W rows(which means each client read 100W rows). But After patch ,the randomRead speed is about 7hours,I want to know what's the matter? Thanks! > Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE > -- > > Key: HBASE-7404 > URL: https://issues.apache.org/jira/browse/HBASE-7404 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.3 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.95.0 > > Attachments: 7404-trunk-v10.patch, 7404-trunk-v11.patch, > 7404-trunk-v12.patch, 7404-trunk-v13.patch, 7404-trunk-v13.txt, > 7404-trunk-v14.patch, BucketCache.pdf, hbase-7404-94v2.patch, > HBASE-7404-backport-0.94.patch, hbase-7404-trunkv2.patch, > hbase-7404-trunkv9.patch, Introduction of Bucket Cache.pdf > > > First, thanks @neil from Fusion-IO share the source code. > Usage: > 1.Use bucket cache as main memory cache, configured as the following: > –"hbase.bucketcache.ioengine" "heap" > –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of > max heap size) > 2.Use bucket cache as a secondary cache, configured as the following: > –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path > where to store the block data) > –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 > means 1GB) > –"hbase.bucketcache.combinedcache.enabled" false (default value being true) > See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache > What's Bucket Cache? > It could greatly decrease CMS and heap fragment by GC > It support a large cache space for High Read Performance by using high speed > disk like Fusion-io > 1.An implementation of block cache like LruBlockCache > 2.Self manage blocks' storage position through Bucket Allocator > 3.The cached blocks could be stored in the memory or file system > 4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), > combined with LruBlockCache to decrease CMS and fragment by GC. > 5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to > store block) to enlarge cache space > How about SlabCache? > We have studied and test SlabCache first, but the result is bad, because: > 1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds > of block size, especially using DataBlockEncoding > 2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache > and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , > it causes CMS and heap fragment don't get any better > 3.Direct heap performance is not good as heap, and maybe cause OOM, so we > recommend using "heap" engine > See more in the attachment and in the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7404) Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
[ https://issues.apache.org/jira/browse/HBASE-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663968#comment-13663968 ] Liang Lee commented on HBASE-7404: -- OK ,Thanks ,I will have a try! > Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE > -- > > Key: HBASE-7404 > URL: https://issues.apache.org/jira/browse/HBASE-7404 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.3 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.95.0 > > Attachments: 7404-trunk-v10.patch, 7404-trunk-v11.patch, > 7404-trunk-v12.patch, 7404-trunk-v13.patch, 7404-trunk-v13.txt, > 7404-trunk-v14.patch, BucketCache.pdf, hbase-7404-94v2.patch, > HBASE-7404-backport-0.94.patch, hbase-7404-trunkv2.patch, > hbase-7404-trunkv9.patch, Introduction of Bucket Cache.pdf > > > First, thanks @neil from Fusion-IO share the source code. > Usage: > 1.Use bucket cache as main memory cache, configured as the following: > –"hbase.bucketcache.ioengine" "heap" > –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of > max heap size) > 2.Use bucket cache as a secondary cache, configured as the following: > –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path > where to store the block data) > –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 > means 1GB) > –"hbase.bucketcache.combinedcache.enabled" false (default value being true) > See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache > What's Bucket Cache? > It could greatly decrease CMS and heap fragment by GC > It support a large cache space for High Read Performance by using high speed > disk like Fusion-io > 1.An implementation of block cache like LruBlockCache > 2.Self manage blocks' storage position through Bucket Allocator > 3.The cached blocks could be stored in the memory or file system > 4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), > combined with LruBlockCache to decrease CMS and fragment by GC. > 5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to > store block) to enlarge cache space > How about SlabCache? > We have studied and test SlabCache first, but the result is bad, because: > 1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds > of block size, especially using DataBlockEncoding > 2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache > and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , > it causes CMS and heap fragment don't get any better > 3.Direct heap performance is not good as heap, and maybe cause OOM, so we > recommend using "heap" engine > See more in the attachment and in the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7404) Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
[ https://issues.apache.org/jira/browse/HBASE-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663819#comment-13663819 ] Liang Lee commented on HBASE-7404: -- When I used this patch on my cluster ,I found that the randomRead speed is very slow which is about 70% with the speed before patch. I want to know waht is the problem? The HBase version is 0.94.2 and my cluster consists by one Master and three regionservers. The Heapszie is set to 4G while the machine memory is 8G . I use the first method,the ocnfiguration is as follow: hbase.bucketcache.ioengine heap hbase.bucketcache.size 0.4 hbase.bucketcache.combinedcache.percentage 0.9f Thanks > Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE > -- > > Key: HBASE-7404 > URL: https://issues.apache.org/jira/browse/HBASE-7404 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.3 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.95.0 > > Attachments: 7404-trunk-v10.patch, 7404-trunk-v11.patch, > 7404-trunk-v12.patch, 7404-trunk-v13.patch, 7404-trunk-v13.txt, > 7404-trunk-v14.patch, BucketCache.pdf, hbase-7404-94v2.patch, > HBASE-7404-backport-0.94.patch, hbase-7404-trunkv2.patch, > hbase-7404-trunkv9.patch, Introduction of Bucket Cache.pdf > > > First, thanks @neil from Fusion-IO share the source code. > Usage: > 1.Use bucket cache as main memory cache, configured as the following: > –"hbase.bucketcache.ioengine" "heap" > –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of > max heap size) > 2.Use bucket cache as a secondary cache, configured as the following: > –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path > where to store the block data) > –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 > means 1GB) > –"hbase.bucketcache.combinedcache.enabled" false (default value being true) > See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache > What's Bucket Cache? > It could greatly decrease CMS and heap fragment by GC > It support a large cache space for High Read Performance by using high speed > disk like Fusion-io > 1.An implementation of block cache like LruBlockCache > 2.Self manage blocks' storage position through Bucket Allocator > 3.The cached blocks could be stored in the memory or file system > 4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), > combined with LruBlockCache to decrease CMS and fragment by GC. > 5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to > store block) to enlarge cache space > How about SlabCache? > We have studied and test SlabCache first, but the result is bad, because: > 1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds > of block size, especially using DataBlockEncoding > 2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache > and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , > it causes CMS and heap fragment don't get any better > 3.Direct heap performance is not good as heap, and maybe cause OOM, so we > recommend using "heap" engine > See more in the attachment and in the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5844) Delete the region servers znode after a regions server crash
[ https://issues.apache.org/jira/browse/HBASE-5844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662615#comment-13662615 ] Liang Lee commented on HBASE-5844: -- And I still have another question:in the hbase-daemon.sh file,in the cleanZNode() funtion ,which znode is deleted ? Under the normal scenario,when RegionServer starts ,it will create temporary znode on Zookeeper in the path /habse/rs,and what we delete in the cleanZNode() funtion ? Thank you! > Delete the region servers znode after a regions server crash > > > Key: HBASE-5844 > URL: https://issues.apache.org/jira/browse/HBASE-5844 > Project: HBase > Issue Type: Improvement > Components: regionserver, scripts >Affects Versions: 0.95.2 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.95.0 > > Attachments: 5844.v1.patch, 5844.v2.patch, 5844.v3.patch, > 5844.v3.patch, 5844.v4.patch > > > today, if the regions server crashes, its znode is not deleted in ZooKeeper. > So the recovery process will stop only after a timeout, usually 30s. > By deleting the znode in start script, we remove this delay and the recovery > starts immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5844) Delete the region servers znode after a regions server crash
[ https://issues.apache.org/jira/browse/HBASE-5844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13661824#comment-13661824 ] Liang Lee commented on HBASE-5844: -- Hi,stack ,Could you please provide some document about how to use this pach like HBASE-7404? Where does the core configution HBASE_ZNODE_FILE should be configured? Thanks > Delete the region servers znode after a regions server crash > > > Key: HBASE-5844 > URL: https://issues.apache.org/jira/browse/HBASE-5844 > Project: HBase > Issue Type: Improvement > Components: regionserver, scripts >Affects Versions: 0.95.2 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.95.0 > > Attachments: 5844.v1.patch, 5844.v2.patch, 5844.v3.patch, > 5844.v3.patch, 5844.v4.patch > > > today, if the regions server crashes, its znode is not deleted in ZooKeeper. > So the recovery process will stop only after a timeout, usually 30s. > By deleting the znode in start script, we remove this delay and the recovery > starts immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6134) Improvement for split-worker to speed up distributed log splitting
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655106#comment-13655106 ] Liang Lee commented on HBASE-6134: -- Hi,when I apply the patch in Hbase-0.94.2 ,I find error in the HLogSplitter.java: The object DistributedLogSplittingHelper can't find its defination,what's the matter? Thanks > Improvement for split-worker to speed up distributed log splitting > -- > > Key: HBASE-6134 > URL: https://issues.apache.org/jira/browse/HBASE-6134 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.95.0 > > Attachments: 6134v4.patch, HBASE-6134.patch, HBASE-6134v2.patch, > HBASE-6134v3-92.patch, HBASE-6134v3.patch, HBASE-6134v4-94.patch, > HBASE-6134v4-94v2.patch, HBASE-6134v4.patch > > > First，we do the test between local-master-splitting and > distributed-log-splitting > Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths > splitting work), 400 regions in one hlog file > local-master-split:60s+ > distributed-log-splitting:165s+ > In fact, in our production environment, distributed-log-splitting also took > 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) > We found split-worker split one log file took about 20s > (30ms~50ms per writer.close(); 10ms per create writers ) > I think we could do the improvement for this: > Parallelizing the create and close writers in threads > In the patch, change the logic for distributed-log-splitting same as the > local-master-splitting and parallelizing the close in threads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
