[jira] [Updated] (HBASE-12935) Does any one consider the performance of HBase on SSD?

2015-02-25 Thread Liang Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liang Lee updated HBASE-12935:
--
Description: Some features of HBase doesn't mathch features of SSD. Such as 
compaction is harmful for SSD life span.  (was: Some features of HBase doesn't 
mathch features of SSD. Such as comapction is harmful for SSD life span.)

 Does any one consider the performance of HBase on SSD? 
 ---

 Key: HBASE-12935
 URL: https://issues.apache.org/jira/browse/HBASE-12935
 Project: HBase
  Issue Type: Improvement
Reporter: Liang Lee

 Some features of HBase doesn't mathch features of SSD. Such as compaction is 
 harmful for SSD life span.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (HBASE-12935) Does any one consider the performance of HBase on SSD?

2015-01-27 Thread Liang Lee (JIRA)
Liang Lee created HBASE-12935:
-

 Summary: Does any one consider the performance of HBase on SSD? 
 Key: HBASE-12935
 URL: https://issues.apache.org/jira/browse/HBASE-12935
 Project: HBase
  Issue Type: Improvement
Reporter: Liang Lee


Some features of HBase doesn't mathch features of SSD. Such as comapction is 
harmful for SSD life span.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-7404) Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE

2013-05-27 Thread Liang Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13667490#comment-13667490
 ] 

Liang Lee commented on HBASE-7404:
--

Thanks,Ted.
I modified the configuration and the lruCacheSize is 1G ,the bucketCacheSize 
isv 1G.
The Heapszie is set to 4G while the machine memory is 8G .
I use PerformanceEvaluation tool based on hbase to test the Performance.
Before this path, the randomRead speed is about 6hours ,which used 10 clients 
to read 1000W rows(which means each client read 100W rows).
But After patch ,the randomRead speed is about 7hours,I want to know what's the 
matter?
Thanks!

 Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
 --

 Key: HBASE-7404
 URL: https://issues.apache.org/jira/browse/HBASE-7404
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
 Fix For: 0.95.0

 Attachments: 7404-trunk-v10.patch, 7404-trunk-v11.patch, 
 7404-trunk-v12.patch, 7404-trunk-v13.patch, 7404-trunk-v13.txt, 
 7404-trunk-v14.patch, BucketCache.pdf, hbase-7404-94v2.patch, 
 HBASE-7404-backport-0.94.patch, hbase-7404-trunkv2.patch, 
 hbase-7404-trunkv9.patch, Introduction of Bucket Cache.pdf


 First, thanks @neil from Fusion-IO share the source code.
 Usage:
 1.Use bucket cache as main memory cache, configured as the following:
 –hbase.bucketcache.ioengine heap
 –hbase.bucketcache.size 0.4 (size for bucket cache, 0.4 is a percentage of 
 max heap size)
 2.Use bucket cache as a secondary cache, configured as the following:
 –hbase.bucketcache.ioengine file:/disk1/hbase/cache.data(The file path 
 where to store the block data)
 –hbase.bucketcache.size 1024 (size for bucket cache, unit is MB, so 1024 
 means 1GB)
 –hbase.bucketcache.combinedcache.enabled false (default value being true)
 See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and 
 org.apache.hadoop.hbase.io.hfile.bucket.BucketCache
 What's Bucket Cache? 
 It could greatly decrease CMS and heap fragment by GC
 It support a large cache space for High Read Performance by using high speed 
 disk like Fusion-io
 1.An implementation of block cache like LruBlockCache
 2.Self manage blocks' storage position through Bucket Allocator
 3.The cached blocks could be stored in the memory or file system
 4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), 
 combined with LruBlockCache to decrease CMS and fragment by GC.
 5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to 
 store block) to enlarge cache space
 How about SlabCache?
 We have studied and test SlabCache first, but the result is bad, because:
 1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds 
 of block size, especially using DataBlockEncoding
 2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache 
 and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , 
 it causes CMS and heap fragment don't get any better
 3.Direct heap performance is not good as heap, and maybe cause OOM, so we 
 recommend using heap engine 
 See more in the attachment and in the patch

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (HBASE-7404) Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE

2013-05-22 Thread Liang Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663819#comment-13663819
 ] 

Liang Lee commented on HBASE-7404:
--

When I used this patch on my cluster ,I found that the randomRead speed is very 
slow which is about 70% with the speed before patch.
I want to know waht is the problem? 
The HBase version is 0.94.2 and my cluster consists by one Master and three 
regionservers.
The Heapszie is set to 4G while the machine memory is 8G .
I use the first method,the ocnfiguration is as follow:
hbase.bucketcache.ioengine  heap
hbase.bucketcache.size   0.4
hbase.bucketcache.combinedcache.percentage  0.9f

Thanks


 Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
 --

 Key: HBASE-7404
 URL: https://issues.apache.org/jira/browse/HBASE-7404
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
 Fix For: 0.95.0

 Attachments: 7404-trunk-v10.patch, 7404-trunk-v11.patch, 
 7404-trunk-v12.patch, 7404-trunk-v13.patch, 7404-trunk-v13.txt, 
 7404-trunk-v14.patch, BucketCache.pdf, hbase-7404-94v2.patch, 
 HBASE-7404-backport-0.94.patch, hbase-7404-trunkv2.patch, 
 hbase-7404-trunkv9.patch, Introduction of Bucket Cache.pdf


 First, thanks @neil from Fusion-IO share the source code.
 Usage:
 1.Use bucket cache as main memory cache, configured as the following:
 –hbase.bucketcache.ioengine heap
 –hbase.bucketcache.size 0.4 (size for bucket cache, 0.4 is a percentage of 
 max heap size)
 2.Use bucket cache as a secondary cache, configured as the following:
 –hbase.bucketcache.ioengine file:/disk1/hbase/cache.data(The file path 
 where to store the block data)
 –hbase.bucketcache.size 1024 (size for bucket cache, unit is MB, so 1024 
 means 1GB)
 –hbase.bucketcache.combinedcache.enabled false (default value being true)
 See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and 
 org.apache.hadoop.hbase.io.hfile.bucket.BucketCache
 What's Bucket Cache? 
 It could greatly decrease CMS and heap fragment by GC
 It support a large cache space for High Read Performance by using high speed 
 disk like Fusion-io
 1.An implementation of block cache like LruBlockCache
 2.Self manage blocks' storage position through Bucket Allocator
 3.The cached blocks could be stored in the memory or file system
 4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), 
 combined with LruBlockCache to decrease CMS and fragment by GC.
 5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to 
 store block) to enlarge cache space
 How about SlabCache?
 We have studied and test SlabCache first, but the result is bad, because:
 1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds 
 of block size, especially using DataBlockEncoding
 2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache 
 and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , 
 it causes CMS and heap fragment don't get any better
 3.Direct heap performance is not good as heap, and maybe cause OOM, so we 
 recommend using heap engine 
 See more in the attachment and in the patch

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (HBASE-7404) Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE

2013-05-22 Thread Liang Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663968#comment-13663968
 ] 

Liang Lee commented on HBASE-7404:
--

OK ,Thanks ,I will have a try!

 Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
 --

 Key: HBASE-7404
 URL: https://issues.apache.org/jira/browse/HBASE-7404
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
 Fix For: 0.95.0

 Attachments: 7404-trunk-v10.patch, 7404-trunk-v11.patch, 
 7404-trunk-v12.patch, 7404-trunk-v13.patch, 7404-trunk-v13.txt, 
 7404-trunk-v14.patch, BucketCache.pdf, hbase-7404-94v2.patch, 
 HBASE-7404-backport-0.94.patch, hbase-7404-trunkv2.patch, 
 hbase-7404-trunkv9.patch, Introduction of Bucket Cache.pdf


 First, thanks @neil from Fusion-IO share the source code.
 Usage:
 1.Use bucket cache as main memory cache, configured as the following:
 –hbase.bucketcache.ioengine heap
 –hbase.bucketcache.size 0.4 (size for bucket cache, 0.4 is a percentage of 
 max heap size)
 2.Use bucket cache as a secondary cache, configured as the following:
 –hbase.bucketcache.ioengine file:/disk1/hbase/cache.data(The file path 
 where to store the block data)
 –hbase.bucketcache.size 1024 (size for bucket cache, unit is MB, so 1024 
 means 1GB)
 –hbase.bucketcache.combinedcache.enabled false (default value being true)
 See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and 
 org.apache.hadoop.hbase.io.hfile.bucket.BucketCache
 What's Bucket Cache? 
 It could greatly decrease CMS and heap fragment by GC
 It support a large cache space for High Read Performance by using high speed 
 disk like Fusion-io
 1.An implementation of block cache like LruBlockCache
 2.Self manage blocks' storage position through Bucket Allocator
 3.The cached blocks could be stored in the memory or file system
 4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), 
 combined with LruBlockCache to decrease CMS and fragment by GC.
 5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to 
 store block) to enlarge cache space
 How about SlabCache?
 We have studied and test SlabCache first, but the result is bad, because:
 1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds 
 of block size, especially using DataBlockEncoding
 2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache 
 and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , 
 it causes CMS and heap fragment don't get any better
 3.Direct heap performance is not good as heap, and maybe cause OOM, so we 
 recommend using heap engine 
 See more in the attachment and in the patch

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (HBASE-5844) Delete the region servers znode after a regions server crash

2013-05-20 Thread Liang Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13661824#comment-13661824
 ] 

Liang Lee commented on HBASE-5844:
--

Hi,stack ,Could you please provide some document about how to use this pach 
like HBASE-7404?
Where does the core configution HBASE_ZNODE_FILE should be configured?
Thanks
 

 Delete the region servers znode after a regions server crash
 

 Key: HBASE-5844
 URL: https://issues.apache.org/jira/browse/HBASE-5844
 Project: HBase
  Issue Type: Improvement
  Components: regionserver, scripts
Affects Versions: 0.95.2
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.95.0

 Attachments: 5844.v1.patch, 5844.v2.patch, 5844.v3.patch, 
 5844.v3.patch, 5844.v4.patch


 today, if the regions server crashes, its znode is not deleted in ZooKeeper. 
 So the recovery process will stop only after a timeout, usually 30s.
 By deleting the znode in start script, we remove this delay and the recovery 
 starts immediately.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (HBASE-5844) Delete the region servers znode after a regions server crash

2013-05-20 Thread Liang Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662615#comment-13662615
 ] 

Liang Lee commented on HBASE-5844:
--

And I still have another question:in the hbase-daemon.sh file,in the 
cleanZNode() funtion ,which znode is deleted ?
Under the normal scenario,when RegionServer starts ,it will create temporary 
znode on Zookeeper in the path /habse/rs,and what we delete in the cleanZNode() 
funtion ?
Thank you!

 Delete the region servers znode after a regions server crash
 

 Key: HBASE-5844
 URL: https://issues.apache.org/jira/browse/HBASE-5844
 Project: HBase
  Issue Type: Improvement
  Components: regionserver, scripts
Affects Versions: 0.95.2
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.95.0

 Attachments: 5844.v1.patch, 5844.v2.patch, 5844.v3.patch, 
 5844.v3.patch, 5844.v4.patch


 today, if the regions server crashes, its znode is not deleted in ZooKeeper. 
 So the recovery process will stop only after a timeout, usually 30s.
 By deleting the znode in start script, we remove this delay and the recovery 
 starts immediately.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (HBASE-6134) Improvement for split-worker to speed up distributed log splitting

2013-05-10 Thread Liang Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655106#comment-13655106
 ] 

Liang Lee commented on HBASE-6134:
--

Hi,when I apply the patch in Hbase-0.94.2 ,I find error in the 
HLogSplitter.java:
The object DistributedLogSplittingHelper can't find its defination,what's the 
matter?
Thanks

 Improvement for split-worker to speed up distributed log splitting
 --

 Key: HBASE-6134
 URL: https://issues.apache.org/jira/browse/HBASE-6134
 Project: HBase
  Issue Type: Improvement
  Components: wal
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.95.0

 Attachments: 6134v4.patch, HBASE-6134.patch, HBASE-6134v2.patch, 
 HBASE-6134v3-92.patch, HBASE-6134v3.patch, HBASE-6134v4-94.patch, 
 HBASE-6134v4-94v2.patch, HBASE-6134v4.patch


 First,we do the test between local-master-splitting and 
 distributed-log-splitting
 Environment:34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths 
 splitting work), 400 regions in one hlog file
 local-master-split:60s+
 distributed-log-splitting:165s+
 In fact, in our production environment, distributed-log-splitting also took 
 60s with 30 regionservers for 34 hlog files (regionserver may be in high load)
 We found split-worker split one log file took about 20s
 (30ms~50ms per writer.close(); 10ms per create writers )
 I think we could do the improvement for this:
 Parallelizing the create and close writers in threads
 In the patch, change the logic for  distributed-log-splitting same as the 
 local-master-splitting and parallelizing the close in threads.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira