[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447554#comment-16447554 ] Chance Li commented on HBASE-20312: --- [~mdrob], sir, I will separate this patch to two parts as [~carp84] suggested after completing the document. I have to pay attention to the precommit and review first, and now failed UT cases seem to irrelevant, I try to analyze it. separating patch is more easy, as these two parts are very logically clear in this patch, and I've always been focusing on readability. Thanks for great suggestion and review. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, HBASE-20312-master.v8.patch, > HBASE-20312-master.v9.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447084#comment-16447084 ] Chance Li commented on HBASE-20312: --- Thanks, [~yuzhih...@gmail.com] HBASE-20312-master.v9.patch for review and check-style. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, HBASE-20312-master.v8.patch, > HBASE-20312-master.v9.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20312: -- Attachment: HBASE-20312-master.v9.patch > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, HBASE-20312-master.v8.patch, > HBASE-20312-master.v9.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446891#comment-16446891 ] Chance Li commented on HBASE-20312: --- HBASE-20312-master.v8.patch update for review and check-style. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, HBASE-20312-master.v8.patch, > ccsmap-branch-1.1.patch, hits.png, jira1.png, jira2.png, jira3.png, > off-heap-test-put-master.png, on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20312: -- Attachment: HBASE-20312-master.v8.patch > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, HBASE-20312-master.v8.patch, > ccsmap-branch-1.1.patch, hits.png, jira1.png, jira2.png, jira3.png, > off-heap-test-put-master.png, on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443617#comment-16443617 ] Chance Li commented on HBASE-20312: --- [~mdrob] sir, Making another project is not my propose. Let's me give a brief introduction. First, CCSMap is totally designed for HBase. -- Global chunkPool, including the functions of MemStoreLAB. Second, for reliability and developing iterative ability , also for readability, we make CCSMap more independent. that means, CCSMap is easy to use. The purpose of CCSMap is to improve the throughput(cuts down YGC time). And it work better than other memstores. So, Maybe I think now is Stage-1. next, we maybe can consider making CCSMap integrated with HBase. And the doc is going on, I think it will be done in the week. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[
https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443092#comment-16443092
]
Chance Li commented on HBASE-20312:
---
HBASE-20312-master.v7.patch update for UT and check-style again.
I wonder how can I process this check-style-error as the following. Should I
refactor it?
{code:java}
./hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/skiplist/core/TestCCSMap.java:105:
@SuppressWarnings("CheckStyle"):3: Method length is 192 lines (max allowed is
150). [MethodLength]
./hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/skiplist/core/TestCompactedConcurrentSkipList.java:153:
@SuppressWarnings("CheckStyle"):3: Method length is 239 lines (max allowed is
150). [MethodLength]
./hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/skiplist/core/TestCompactedConcurrentSkipList.java:597:
@SuppressWarnings("CheckStyle"):3: Method length is 219 lines (max allowed is
150). [MethodLength]
./hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/skiplist/core/TestSubCompactedConcurrentSkipList.java:44:
@SuppressWarnings("CheckStyle"):3: Method length is 312 lines (max allowed is
150). [MethodLength]
{code}
and bug report, it look like unnecessary too.
{code}
org.apache.hadoop.hbase.regionserver.skiplist.hbase.CCSMapCellComparatorDirectly.compareTo(Cell,
ByteBuffer, int, int) negates the return value of
org.apache.hadoop.hbase.regionserver.skiplist.hbase.CCSMapCellComparatorDirectly.compareTo(ByteBuffer,
int, int, Cell)
{code}
> CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
> --
>
> Key: HBASE-20312
> URL: https://issues.apache.org/jira/browse/HBASE-20312
> Project: HBase
> Issue Type: New Feature
> Components: regionserver
>Reporter: Xiang Wang
>Assignee: Chance Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch,
> HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch,
> HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch,
> HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch,
> ccsmap-branch-1.1.patch, hits.png, jira1.png, jira2.png, jira3.png,
> off-heap-test-put-master.png, on-heap-test-put-master.png
>
>
> Now hbase use ConcurrentSkipListMap as memstore's data structure.
> Although MemSLAB reduces memory fragment brought by key-value pairs.
> Hundred of millions key-value pairs still make young generation
> garbage-collection(gc) stop time long.
>
> These are 2 gc problems of ConcurrentSkipListMap:
> 1. HBase needs 3 objects to store one key-value on expectation. One
> Index(skiplist's average node height is 1), one Node, and one KeyValue. Too
> many objects are created for memstore.
> 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned
> on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc
> algorithm) will change frequently on high writing throughput, which makes YGC
> slow.
>
> We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap
> for short),
> which provides similary features like ConcurrentSkipListMap but get rid of
> Objects for every key-value pairs.
> CCSMap's memory structure is like this picture:
> !jira1.png!
>
> One CCSMap consists of a certain number of Chunks. One Chunk consists of a
> certain number of nodes. One node is corresspding one element. This element's
> all information and its key-value is encoded on a continuous memory segment
> without any objects.
> Features:
> 1. all insert,update,delete operations is lock-free on CCSMap.
> 2. Consume less memory, it brings 40% memory saving for 50Byte key-value.
> 3. Faster on small key-value because of better cacheline usage. 20~30% better
> read/write troughput than ConcurrentSkipListMap for 50Byte key-value.
> CCSMap do not support recyle space when deleting element. But it doesn't
> matter for hbase because of region flush.
> CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts
> down YGC time significantly. here are 2 graph of before and after.
> !jira2.png!
> !jira3.png!
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20312: -- Attachment: HBASE-20312-master.v7.patch > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20312: -- Attachment: HBASE-20312-master.v6.patch > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > ccsmap-branch-1.1.patch, hits.png, jira1.png, jira2.png, jira3.png, > off-heap-test-put-master.png, on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442560#comment-16442560 ] Chance Li commented on HBASE-20312: --- HBASE-20312-master.v6.patch is for check-style and review. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > ccsmap-branch-1.1.patch, hits.png, jira1.png, jira2.png, jira3.png, > off-heap-test-put-master.png, on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440902#comment-16440902 ] Chance Li commented on HBASE-20312: --- HBASE-20312-master.v5.patch fix a bug in close method. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20312: -- Attachment: HBASE-20312-master.v5.patch > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[
https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429615#comment-16429615
]
Chance Li commented on HBASE-20312:
---
key configurations:
1. hbase-env.sh hbase-site.xml
- -Xmx36864m -Xmx36864m -Xmn10240m -XX:SurvivorRatio=3
- -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=4096
- hfile.block.cache.size => 0.08
- hbase.regionserver.metahandler.count => 90
- hbase.regionserver.handler.count => 192
- hbase.wal.provider => multiwal
- hbase.wal.regiongrouping.numgroups => 4
- hbase.regionserver.maxlogs => 32
- memstore on-heap:
hbase.regionserver.offheap.global.memstore.size=0
hbase.regionserver.global.memstore.size=0.3
- memstore off-heap:
hbase.regionserver.offheap.global.memstore.size=10240
2. Ycsb
- 1 region server
- 4 ycsb machines，10 clients/machine ，100 threads/client
- uniform
- fieldcount=1
- fieldlength=1024
- insertproportion=1
3. Table：presharding 10 region
table schema： \{NAME => 'cf', DATA_BLOCK_ENCODING => 'DIFF', VERSIONS=> '1',
COMPRESSION => 'SNAPPY', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
,{SPLITS => (1..9).map
Unknown macro: \{|i| "user#{1000+i*(-1000)/9}"}
, MAX_FILESIZE=>'223372036854775807',DURABILITY=>'SKIP_WAL',METADATA =>
Unknown macro: \{'hbase.hstore.block.storage.policy' => 'ALL_SSD'}
}
> CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
> --
>
> Key: HBASE-20312
> URL: https://issues.apache.org/jira/browse/HBASE-20312
> Project: HBase
> Issue Type: New Feature
> Components: regionserver
>Reporter: Xiang Wang
>Assignee: Chance Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch,
> HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch,
> HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch,
> ccsmap-branch-1.1.patch, hits.png, jira1.png, jira2.png, jira3.png,
> off-heap-test-put-master.png, on-heap-test-put-master.png
>
>
> Now hbase use ConcurrentSkipListMap as memstore's data structure.
> Although MemSLAB reduces memory fragment brought by key-value pairs.
> Hundred of millions key-value pairs still make young generation
> garbage-collection(gc) stop time long.
>
> These are 2 gc problems of ConcurrentSkipListMap:
> 1. HBase needs 3 objects to store one key-value on expectation. One
> Index(skiplist's average node height is 1), one Node, and one KeyValue. Too
> many objects are created for memstore.
> 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned
> on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc
> algorithm) will change frequently on high writing throughput, which makes YGC
> slow.
>
> We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap
> for short),
> which provides similary features like ConcurrentSkipListMap but get rid of
> Objects for every key-value pairs.
> CCSMap's memory structure is like this picture:
> !jira1.png!
>
> One CCSMap consists of a certain number of Chunks. One Chunk consists of a
> certain number of nodes. One node is corresspding one element. This element's
> all information and its key-value is encoded on a continuous memory segment
> without any objects.
> Features:
> 1. all insert,update,delete operations is lock-free on CCSMap.
> 2. Consume less memory, it brings 40% memory saving for 50Byte key-value.
> 3. Faster on small key-value because of better cacheline usage. 20~30% better
> read/write troughput than ConcurrentSkipListMap for 50Byte key-value.
> CCSMap do not support recyle space when deleting element. But it doesn't
> matter for hbase because of region flush.
> CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts
> down YGC time significantly. here are 2 graph of before and after.
> !jira2.png!
> !jira3.png!
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429587#comment-16429587 ] Chance Li commented on HBASE-20312: --- HBASE-20312-master.v4.patch only fix the config error with CompactingMemstore for easy testing. on master branch, and focusing on the memstore(CCSMapMemstore,CompactingMemstore,DefaultMemstore), I only test SKIP_WAL with the 100% put workload(insertproportion=1), both on-heap and off-heap. On-heap: !on-heap-test-put-master.png! off-heap: !off-heap-test-put-master.png! >From the numbers there are some more interesting things: 1. writting 1 col with DefaultMemstore, the Master has a more performance than 1.1.2 , such as GC and Mem used. I can't figure out what improved it. It's very great. 2. in some cases, off-heap is worse than on-heap. when writting 100 cols, the CCSMapMemstore has much better than others. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > ccsmap-branch-1.1.patch, hits.png, jira1.png, jira2.png, jira3.png, > off-heap-test-put-master.png, on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20312: -- Attachment: off-heap-test-put-master.png > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > ccsmap-branch-1.1.patch, hits.png, jira1.png, jira2.png, jira3.png, > off-heap-test-put-master.png, on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20312: -- Attachment: on-heap-test-put-master.png > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > ccsmap-branch-1.1.patch, hits.png, jira1.png, jira2.png, jira3.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20312: -- Attachment: HBASE-20312-master.v4.patch > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > ccsmap-branch-1.1.patch, hits.png, jira1.png, jira2.png, jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424833#comment-16424833 ] Chance Li commented on HBASE-20312: --- [~stack] sir, in our 1.1.2 branch, the result is: !1.1.2-ccsmap-number.png! I will check it in master version again. It's more strange. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, ccsmap-branch-1.1.patch, jira1.png, jira2.png, > jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20312: -- Attachment: 1.1.2-ccsmap-number.png > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, ccsmap-branch-1.1.patch, jira1.png, jira2.png, > jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423591#comment-16423591 ] Chance Li commented on HBASE-20312: --- Thanks. [~carp84],[~stack] It's my fault. I will fix it on next patch. now, to use CompactingMemsotre, We must set "hbase.regionserver.memstore.class"=org.apache.hadoop.hbase.regionserver.DefaultMemStore. That's terrible. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, > HBASE-20312-master.v2.patch, HBASE-20312-master.v3.patch, > ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[
https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423528#comment-16423528
]
Chance Li commented on HBASE-20312:
---
stack
{quote}regionserver.HStore: Memstore class name is
org.apache.hadoop.hbase.regionserver.DefaultMemStore ;{quote}
This log shows the Meta is DefaultMemstore. Which patch did you apply? is it
HBASE-20312-master.v3.patch?
> CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
> --
>
> Key: HBASE-20312
> URL: https://issues.apache.org/jira/browse/HBASE-20312
> Project: HBase
> Issue Type: New Feature
> Components: regionserver
>Reporter: Xiang Wang
>Assignee: Chance Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch,
> HBASE-20312-master.v2.patch, HBASE-20312-master.v3.patch,
> ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png
>
>
> Now hbase use ConcurrentSkipListMap as memstore's data structure.
> Although MemSLAB reduces memory fragment brought by key-value pairs.
> Hundred of millions key-value pairs still make young generation
> garbage-collection(gc) stop time long.
>
> These are 2 gc problems of ConcurrentSkipListMap:
> 1. HBase needs 3 objects to store one key-value on expectation. One
> Index(skiplist's average node height is 1), one Node, and one KeyValue. Too
> many objects are created for memstore.
> 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned
> on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc
> algorithm) will change frequently on high writing throughput, which makes YGC
> slow.
>
> We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap
> for short),
> which provides similary features like ConcurrentSkipListMap but get rid of
> Objects for every key-value pairs.
> CCSMap's memory structure is like this picture:
> !jira1.png!
>
> One CCSMap consists of a certain number of Chunks. One Chunk consists of a
> certain number of nodes. One node is corresspding one element. This element's
> all information and its key-value is encoded on a continuous memory segment
> without any objects.
> Features:
> 1. all insert,update,delete operations is lock-free on CCSMap.
> 2. Consume less memory, it brings 40% memory saving for 50Byte key-value.
> 3. Faster on small key-value because of better cacheline usage. 20~30% better
> read/write troughput than ConcurrentSkipListMap for 50Byte key-value.
> CCSMap do not support recyle space when deleting element. But it doesn't
> matter for hbase because of region flush.
> CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts
> down YGC time significantly. here are 2 graph of before and after.
> !jira2.png!
> !jira3.png!
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423109#comment-16423109 ] Chance Li commented on HBASE-20312: --- And Thanks [~yuzhih...@gmail.com]. :) > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, > HBASE-20312-master.v2.patch, HBASE-20312-master.v3.patch, > ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423101#comment-16423101 ] Chance Li commented on HBASE-20312: --- HBASE-20312-master.v3.patch fix UT. quickly start: 1. turn off CompactingMemsotre(set "hbase.hregion.compacting.memstore.type" to NONE), And CCSMapMemstore will be enabled by default. to change it by setting "hbase.regionserver.memstore.class"(e.g. org.apache.hadoop.hbase.regionserver.DefaultMemStore). 2. CCSMapMemstore will use the existed configure: - with write-path offheap, using hbase.regionserver.offheap.global.memstore.size, and CCSMap is off-heap. - with write-patch onheap, using hbase.regionserver.global.memstore.size, and CCSMap is on-heap. 3. internally, CCSMapMemstore will bypass the HRegionServer's ChunkCreator and MemStoreLAB, and replace to itself's. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, > HBASE-20312-master.v2.patch, HBASE-20312-master.v3.patch, > ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20312: -- Attachment: HBASE-20312-master.v3.patch > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, > HBASE-20312-master.v2.patch, HBASE-20312-master.v3.patch, > ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422879#comment-16422879 ] Chance Li commented on HBASE-20312: --- HBASE-20312-master.v2.patch fix the error for META using CCSMap. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, > HBASE-20312-master.v2.patch, ccsmap-branch-1.1.patch, jira1.png, jira2.png, > jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422873#comment-16422873 ] Chance Li commented on HBASE-20312: --- [~yuzhih...@gmail.com] sorry for that. because master's version have some different, I change the META to use CCSMap when I made patch for master. because it's not much useful for META to use CCSMap, And I don't make a better CCSMapMetaCellComparator(now it is CCSMapMetaCellComparatorDefault)–the better Comparator is like CCSMapCellComparatorDirectly that have a better performance. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, > HBASE-20312-master.v2.patch, ccsmap-branch-1.1.patch, jira1.png, jira2.png, > jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20312: -- Attachment: HBASE-20312-master.v2.patch > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, > HBASE-20312-master.v2.patch, ccsmap-branch-1.1.patch, jira1.png, jira2.png, > jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[
https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422096#comment-16422096
]
Chance Li commented on HBASE-20312:
---
[~yuzhih...@gmail.com]
{quote}does it make sense to give individual table the ability to switch CCSMap
on / off
{quote}
Maybe not need do that. The managed memory(global) maybe don't fit this
strategy. And CCSMap has very good performance.
> CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
> --
>
> Key: HBASE-20312
> URL: https://issues.apache.org/jira/browse/HBASE-20312
> Project: HBase
> Issue Type: New Feature
> Components: regionserver
>Reporter: Xiang Wang
>Assignee: Chance Li
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch,
> ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png
>
>
> Now hbase use ConcurrentSkipListMap as memstore's data structure.
> Although MemSLAB reduces memory fragment brought by key-value pairs.
> Hundred of millions key-value pairs still make young generation
> garbage-collection(gc) stop time long.
>
> These are 2 gc problems of ConcurrentSkipListMap:
> 1. HBase needs 3 objects to store one key-value on expectation. One
> Index(skiplist's average node height is 1), one Node, and one KeyValue. Too
> many objects are created for memstore.
> 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned
> on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc
> algorithm) will change frequently on high writing throughput, which makes YGC
> slow.
>
> We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap
> for short),
> which provides similary features like ConcurrentSkipListMap but get rid of
> Objects for every key-value pairs.
> CCSMap's memory structure is like this picture:
> !jira1.png!
>
> One CCSMap consists of a certain number of Chunks. One Chunk consists of a
> certain number of nodes. One node is corresspding one element. This element's
> all information and its key-value is encoded on a continuous memory segment
> without any objects.
> Features:
> 1. all insert,update,delete operations is lock-free on CCSMap.
> 2. Consume less memory, it brings 40% memory saving for 50Byte key-value.
> 3. Faster on small key-value because of better cacheline usage. 20~30% better
> read/write troughput than ConcurrentSkipListMap for 50Byte key-value.
> CCSMap do not support recyle space when deleting element. But it doesn't
> matter for hbase because of region flush.
> CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts
> down YGC time significantly. here are 2 graph of before and after.
> !jira2.png!
> !jira3.png!
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422086#comment-16422086 ] Chance Li commented on HBASE-20312: --- HBASE-20312-master.v1.patch. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, > ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20312: -- Fix Version/s: 3.0.0 Release Note: TODO Status: Patch Available (was: Open) > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, > ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20312: -- Attachment: HBASE-20312-master.v1.patch > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Minor > Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, > ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421392#comment-16421392 ] Chance Li commented on HBASE-19389: --- [~stack] sorry for the unclear comment. the currentParallelPutCount is not only for logging, but also recording the concurrency of in-progress parallel put. And only enable TRACE when we suspect an issue , but sometimes the log is still to much. so we added the parallelPutCountPrintThreshold that is only for log and avoiding log to much. > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li >Priority: Critical > Fix For: 3.0.0, 2.1.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V10.patch, HBASE-19389-branch-2-V2.patch, > HBASE-19389-branch-2-V3.patch, HBASE-19389-branch-2-V4.patch, > HBASE-19389-branch-2-V5.patch, HBASE-19389-branch-2-V6.patch, > HBASE-19389-branch-2-V7.patch, HBASE-19389-branch-2-V8.patch, > HBASE-19389-branch-2-V9.patch, HBASE-19389-branch-2.patch, > HBASE-19389.master.patch, HBASE-19389.master.v2.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[
https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420450#comment-16420450
]
Chance Li commented on HBASE-19389:
---
[~stack] Thanks for advice, boss。
{quote}Should we not rather not keep counters unless trace is enabled?
{quote}
To make counters right, it can't be trigged by enabling trace. but maybe we can
optimize to reduce a comparison:
{code:java}
int currentConcurrency = this.currentParallelPutCount.getAndIncrement();
if (LOG.isTraceEnabled()) {
if (currentConcurrency > this.parallelPutCountPrintThreshold) {
LOG.trace("");
}
}
{code}
{quote}Is this how you folks used it?
{quote}
Yes, We need this logs to quickly figure out what happened, sometimes it's not
even the user wanted .
> Limit concurrency of put with dense (hundreds) columns to prevent write
> handler exhausted
> -
>
> Key: HBASE-19389
> URL: https://issues.apache.org/jira/browse/HBASE-19389
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 2.0.0
> Environment: 2000+ Region Servers
> PCI-E ssd
>Reporter: Chance Li
>Assignee: Chance Li
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: CSLM-concurrent-write.png,
> HBASE-19389-branch-2-V10.patch, HBASE-19389-branch-2-V2.patch,
> HBASE-19389-branch-2-V3.patch, HBASE-19389-branch-2-V4.patch,
> HBASE-19389-branch-2-V5.patch, HBASE-19389-branch-2-V6.patch,
> HBASE-19389-branch-2-V7.patch, HBASE-19389-branch-2-V8.patch,
> HBASE-19389-branch-2-V9.patch, HBASE-19389-branch-2.patch,
> HBASE-19389.master.patch, HBASE-19389.master.v2.patch, metrics-1.png,
> ycsb-result.png
>
>
> In a large cluster, with a large number of clients, we found the RS's
> handlers are all busy sometimes. And after investigation we found the root
> cause is about CSLM, such as compare function heavy load. We reviewed the
> related WALs, and found that there were many columns (more than 1000 columns)
> were writing at that time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20303) RS RPC server should not allow the response queue size to be too large
[ https://issues.apache.org/jira/browse/HBASE-20303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418590#comment-16418590 ] Chance Li commented on HBASE-20303: --- [~Apache9] Thanks, sir. I totally agree with 'netty like' way. I will do it. How about SimpleRpcServer? Do I need to do it? > RS RPC server should not allow the response queue size to be too large > -- > > Key: HBASE-20303 > URL: https://issues.apache.org/jira/browse/HBASE-20303 > Project: HBase > Issue Type: Improvement > Components: rpc >Affects Versions: 3.0.0 > Environment: 2000+ Region Servers >Reporter: Chance Li >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20303.master.v1.patch > > > With async clients, in some scenarios RS RPC server will occur Full GC > because of the large reponse queue. > Netty provides a WriteBufferHighWaterMark on channel, But this doesn't meet > RS's needs(Consider 5k ~ 10k sockets in one RS server, it will need 50G ~ > 100G heap for 10M per channel). > RS rpc server will add a gloabl response buffer water mark(2G by default). > when reaching the throttle, RS will not serve any request. > And RS rpc server will add a water mark for channel (100M by default), > because it's mostly possible that this client is abnomal. > We created a unit test to simulate abnormal case: a client that has only 1 > socket can lead RS server to occupy heap up to 100M. > > Notes: > 1. For client compability, we still use existed > exception(CALL_QUEUE_TOO_BIG_EXCEPTION) but error message is different. > 2. Not for SimpleRpcServer, because It's rarely used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20303) RS RPC server should not allow the response queue size to be too large
[
https://issues.apache.org/jira/browse/HBASE-20303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418556#comment-16418556
]
Chance Li commented on HBASE-20303:
---
[~Apache9]
{quote}I think for SimpleRpcServer we will also have the same problem?{quote}
Yes sir. If it's need, I will do it. SimpleRpcServer is rarely used.
{quote}And for netty, I think we can do something like this?{quote}
Thanks sir. For more reliable, I used background task that can avoid any
bugs(maybe netty's). also I don't add anything in the core write-code-path for
keeping performance. do I need change it?
> RS RPC server should not allow the response queue size to be too large
> --
>
> Key: HBASE-20303
> URL: https://issues.apache.org/jira/browse/HBASE-20303
> Project: HBase
> Issue Type: Improvement
> Components: rpc
>Affects Versions: 3.0.0
> Environment: 2000+ Region Servers
>Reporter: Chance Li
>Assignee: Chance Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20303.master.v1.patch
>
>
> With async clients, in some scenarios RS RPC server will occur Full GC
> because of the large reponse queue.
> Netty provides a WriteBufferHighWaterMark on channel, But this doesn't meet
> RS's needs(Consider 5k ~ 10k sockets in one RS server, it will need 50G ~
> 100G heap for 10M per channel).
> RS rpc server will add a gloabl response buffer water mark(2G by default).
> when reaching the throttle, RS will not serve any request.
> And RS rpc server will add a water mark for channel (100M by default),
> because it's mostly possible that this client is abnomal.
> We created a unit test to simulate abnormal case: a client that has only 1
> socket can lead RS server to occupy heap up to 100M.
>
> Notes:
> 1. For client compability, we still use existed
> exception(CALL_QUEUE_TOO_BIG_EXCEPTION) but error message is different.
> 2. Not for SimpleRpcServer, because It's rarely used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20303) RS RPC server should not allow the response queue size to be too large
[ https://issues.apache.org/jira/browse/HBASE-20303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20303: -- Affects Version/s: 3.0.0 Status: Patch Available (was: Open) > RS RPC server should not allow the response queue size to be too large > -- > > Key: HBASE-20303 > URL: https://issues.apache.org/jira/browse/HBASE-20303 > Project: HBase > Issue Type: Improvement > Components: rpc >Affects Versions: 3.0.0 > Environment: 2000+ Region Servers >Reporter: Chance Li >Assignee: Chance Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20303.master.v1.patch > > > With async clients, in some scenarios RS RPC server will occur Full GC > because of the large reponse queue. > Netty provides a WriteBufferHighWaterMark on channel, But this doesn't meet > RS's needs(Consider 5k ~ 10k sockets in one RS server, it will need 50G ~ > 100G heap for 10M per channel). > RS rpc server will add a gloabl response buffer water mark(2G by default). > when reaching the throttle, RS will not serve any request. > And RS rpc server will add a water mark for channel (100M by default), > because it's mostly possible that this client is abnomal. > We created a unit test to simulate abnormal case: a client that has only 1 > socket can lead RS server to occupy heap up to 100M. > > Notes: > 1. For client compability, we still use existed > exception(CALL_QUEUE_TOO_BIG_EXCEPTION) but error message is different. > 2. Not for SimpleRpcServer, because It's rarely used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20303) RS RPC server should not allow the response queue size to be too large
[ https://issues.apache.org/jira/browse/HBASE-20303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-20303: -- Attachment: HBASE-20303.master.v1.patch > RS RPC server should not allow the response queue size to be too large > -- > > Key: HBASE-20303 > URL: https://issues.apache.org/jira/browse/HBASE-20303 > Project: HBase > Issue Type: Improvement > Components: rpc > Environment: 2000+ Region Servers >Reporter: Chance Li >Assignee: Chance Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20303.master.v1.patch > > > With async clients, in some scenarios RS RPC server will occur Full GC > because of the large reponse queue. > Netty provides a WriteBufferHighWaterMark on channel, But this doesn't meet > RS's needs(Consider 5k ~ 10k sockets in one RS server, it will need 50G ~ > 100G heap for 10M per channel). > RS rpc server will add a gloabl response buffer water mark(2G by default). > when reaching the throttle, RS will not serve any request. > And RS rpc server will add a water mark for channel (100M by default), > because it's mostly possible that this client is abnomal. > We created a unit test to simulate abnormal case: a client that has only 1 > socket can lead RS server to occupy heap up to 100M. > > Notes: > 1. For client compability, we still use existed > exception(CALL_QUEUE_TOO_BIG_EXCEPTION) but error message is different. > 2. Not for SimpleRpcServer, because It's rarely used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20303) RS RPC server should not allow the response queue size to be too large
Chance Li created HBASE-20303: - Summary: RS RPC server should not allow the response queue size to be too large Key: HBASE-20303 URL: https://issues.apache.org/jira/browse/HBASE-20303 Project: HBase Issue Type: Improvement Components: rpc Environment: 2000+ Region Servers Reporter: Chance Li Assignee: Chance Li Fix For: 3.0.0 With async clients, in some scenarios RS RPC server will occur Full GC because of the large reponse queue. Netty provides a WriteBufferHighWaterMark on channel, But this doesn't meet RS's needs(Consider 5k ~ 10k sockets in one RS server, it will need 50G ~ 100G heap for 10M per channel). RS rpc server will add a gloabl response buffer water mark(2G by default). when reaching the throttle, RS will not serve any request. And RS rpc server will add a water mark for channel (100M by default), because it's mostly possible that this client is abnomal. We created a unit test to simulate abnormal case: a client that has only 1 socket can lead RS server to occupy heap up to 100M. Notes: 1. For client compability, we still use existed exception(CALL_QUEUE_TOO_BIG_EXCEPTION) but error message is different. 2. Not for SimpleRpcServer, because It's rarely used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20003) WALLess HBase on Persistent Memory
[
https://issues.apache.org/jira/browse/HBASE-20003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413649#comment-16413649
]
Chance Li commented on HBASE-20003:
---
[~anoop.hbase]
{quote}
we may see the latest data (Which is not committed to user) if tertiary was
selected to become new primary or not see it is secondary became new primary.
IMHO this situation is very similar to what you said. WDYT?
{quote}
sir, I think it's totally different. on WAL replay , if the data come back, the
data will always be seen. But here, because all replica can serve read
request, so the data will NOT always be seen, no matter who became to the
primary.
> WALLess HBase on Persistent Memory
> --
>
> Key: HBASE-20003
> URL: https://issues.apache.org/jira/browse/HBASE-20003
> Project: HBase
> Issue Type: New Feature
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Major
>
> This JIRA aims to make use of persistent memory (pmem) technologies in HBase.
> One such usage is to make the Memstore to reside on pmem. Making a persistent
> memstore would remove the need for WAL and paves way for a WALLess HBase.
> The existing region replica feature could be used here and ensure the data
> written to memstores are synchronously replicated to the replicas and ensure
> strong consistency of the data. (pipeline model)
> Advantages :
> - Data Availability : Since the data across replicas are consistent
> (synchronously written) our data is always 100 % available.
> - Lower MTTR : It becomes easier/faster to switch over to the replicas on a
> primary region failure as there is no WAL replay involved. Building the
> memstore map data also is much faster than reading the WAL and replaying the
> WAL.
> - Possibility of bigger memstores : These pmems are designed to have more
> memory than DRAMs so it would also enable us to have bigger sized memstores
> which leads to lesser flushes/compaction IO.
> - Removes the dependency of HDFS on the write path
> Initial PoC has been designed and developed. Testing is underway and we would
> publish the PoC results along with the design doc sooner. The PoC doc will
> talk about the design decisions, the libraries considered to work with these
> pmem devices, pros and cons of those libraries and the performance results.
> Note : Next gen memory technologies using 3DXPoint gives persistent memory
> feature. Such memory DIMMs are soon to appear in the market. The PoC is done
> around Intel's ApachePass (AEP)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20003) WALLess HBase on Persistent Memory
[ https://issues.apache.org/jira/browse/HBASE-20003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413378#comment-16413378 ] Chance Li commented on HBASE-20003: --- [~anoop.hbase] Sir, Excellent doc, I have a little question. To consider a much more special case where the Tertiary success but the Secondary failure(not down), and then the primary goes down without changing the META. This scenario looks like that the RS hosting the Secondary replica reaches the Global Memstore size upper limit(maybe because other table's regions' heavy workload). It's not similar to the case you mentioned. The questions are: 1) Which replica is eligible to become a new primary? (the last mutation(s)(batched) are only in Tertiary but not in Secondary). 2) Losing the state of replica, these two replica will keep the inconsistent state for a while until new replica goes up (the new primary will start flushing). That means the client sometimes can get data, sometimes can't. It's a very bad thing. How can we avoid it? Basically, the architecture looks like moving the pipline from the WAL to region replication, and it will improve MTTR , so it's acceptable. But the different is there will have more thing that will impact the pipeline, such as Global Memstore size upper limit, then the replica's pipeline may be more volatile than WAL's, and this may lead to uncontrolled replica failed, also lead to uncontrolled flush. Finally, Do Hbase consider the geographic locality? > WALLess HBase on Persistent Memory > -- > > Key: HBASE-20003 > URL: https://issues.apache.org/jira/browse/HBASE-20003 > Project: HBase > Issue Type: New Feature >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > > This JIRA aims to make use of persistent memory (pmem) technologies in HBase. > One such usage is to make the Memstore to reside on pmem. Making a persistent > memstore would remove the need for WAL and paves way for a WALLess HBase. > The existing region replica feature could be used here and ensure the data > written to memstores are synchronously replicated to the replicas and ensure > strong consistency of the data. (pipeline model) > Advantages : > - Data Availability : Since the data across replicas are consistent > (synchronously written) our data is always 100 % available. > - Lower MTTR : It becomes easier/faster to switch over to the replicas on a > primary region failure as there is no WAL replay involved. Building the > memstore map data also is much faster than reading the WAL and replaying the > WAL. > - Possibility of bigger memstores : These pmems are designed to have more > memory than DRAMs so it would also enable us to have bigger sized memstores > which leads to lesser flushes/compaction IO. > - Removes the dependency of HDFS on the write path > Initial PoC has been designed and developed. Testing is underway and we would > publish the PoC results along with the design doc sooner. The PoC doc will > talk about the design decisions, the libraries considered to work with these > pmem devices, pros and cons of those libraries and the performance results. > Note : Next gen memory technologies using 3DXPoint gives persistent memory > feature. Such memory DIMMs are soon to appear in the market. The PoC is done > around Intel's ApachePass (AEP) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397987#comment-16397987 ] Chance Li commented on HBASE-19389: --- HBASE-19389-branch-2-V10.patch for checkstyle updates, same as [^HBASE-19389.master.v2.patch] > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li >Priority: Critical > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V10.patch, HBASE-19389-branch-2-V2.patch, > HBASE-19389-branch-2-V3.patch, HBASE-19389-branch-2-V4.patch, > HBASE-19389-branch-2-V5.patch, HBASE-19389-branch-2-V6.patch, > HBASE-19389-branch-2-V7.patch, HBASE-19389-branch-2-V8.patch, > HBASE-19389-branch-2-V9.patch, HBASE-19389-branch-2.patch, > HBASE-19389.master.patch, HBASE-19389.master.v2.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: HBASE-19389-branch-2-V10.patch > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li >Priority: Critical > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V10.patch, HBASE-19389-branch-2-V2.patch, > HBASE-19389-branch-2-V3.patch, HBASE-19389-branch-2-V4.patch, > HBASE-19389-branch-2-V5.patch, HBASE-19389-branch-2-V6.patch, > HBASE-19389-branch-2-V7.patch, HBASE-19389-branch-2-V8.patch, > HBASE-19389-branch-2-V9.patch, HBASE-19389-branch-2.patch, > HBASE-19389.master.patch, HBASE-19389.master.v2.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394738#comment-16394738 ] Chance Li commented on HBASE-19389: --- checkstyle, run UT again > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li >Priority: Critical > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch, > HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2-V7.patch, > HBASE-19389-branch-2-V8.patch, HBASE-19389-branch-2-V9.patch, > HBASE-19389-branch-2.patch, HBASE-19389.master.patch, > HBASE-19389.master.v2.patch, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: HBASE-19389.master.v2.patch > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li >Priority: Critical > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch, > HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2-V7.patch, > HBASE-19389-branch-2-V8.patch, HBASE-19389-branch-2-V9.patch, > HBASE-19389-branch-2.patch, HBASE-19389.master.patch, > HBASE-19389.master.v2.patch, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394484#comment-16394484 ] Chance Li commented on HBASE-19389: --- update patch HBASE-19389-branch-2-V8.patch, add HBASE-19389.master.patch for master. Fixed an issue when handling the STORE_TOO_BUSY code or a atomic request,. > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li >Priority: Critical > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch, > HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2-V7.patch, > HBASE-19389-branch-2-V8.patch, HBASE-19389-branch-2-V9.patch, > HBASE-19389-branch-2.patch, HBASE-19389.master.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: HBASE-19389.master.patch > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li >Priority: Critical > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch, > HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2-V7.patch, > HBASE-19389-branch-2-V8.patch, HBASE-19389-branch-2-V9.patch, > HBASE-19389-branch-2.patch, HBASE-19389.master.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: HBASE-19389-branch-2-V9.patch > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li >Priority: Critical > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch, > HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2-V7.patch, > HBASE-19389-branch-2-V8.patch, HBASE-19389-branch-2-V9.patch, > HBASE-19389-branch-2.patch, HBASE-19389.master.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391023#comment-16391023 ] Chance Li commented on HBASE-19389: --- [~uagashe], Thanks, sir. I should re-write javadoc or comments of the patch, it's really not good. I will update it soon. > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li >Priority: Critical > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch, > HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2-V7.patch, > HBASE-19389-branch-2-V8.patch, HBASE-19389-branch-2.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[
https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388970#comment-16388970
]
Chance Li commented on HBASE-19389:
---
[~stack]
thanks, sir.
{quote}less throughput, but way less CPU used (better 95th percentiles, etc),
and handlers free for other tasks
{quote}
Yes sir, this is our purpose.
{quote}Would it make sense to put this limit behind a configuration gate?
{quote}
I think we need it. It is mainly for to enable us to 'remove' it , even though
we can't totally remove it because of the added counter on the path.
{quote}Also of note is how your compares show the asyncwal being slightly
better than old wal.
{quote}
Let me check again. And this async_wal is one of Durablilty types. it's not WAL.
[~carp84] ok, I will work on it.
> Limit concurrency of put with dense (hundreds) columns to prevent write
> handler exhausted
> -
>
> Key: HBASE-19389
> URL: https://issues.apache.org/jira/browse/HBASE-19389
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 2.0.0
> Environment: 2000+ Region Servers
> PCI-E ssd
>Reporter: Chance Li
>Assignee: Chance Li
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: CSLM-concurrent-write.png,
> HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch,
> HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch,
> HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2-V7.patch,
> HBASE-19389-branch-2-V8.patch, HBASE-19389-branch-2.patch, metrics-1.png,
> ycsb-result.png
>
>
> In a large cluster, with a large number of clients, we found the RS's
> handlers are all busy sometimes. And after investigation we found the root
> cause is about CSLM, such as compare function heavy load. We reviewed the
> related WALs, and found that there were many columns (more than 1000 columns)
> were writing at that time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HBASE-12439) Procedure V2
[
https://issues.apache.org/jira/browse/HBASE-12439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chance Li updated HBASE-12439:
--
Comment: was deleted
(was: [~stack]
thanks, sir.
{quote}less throughput, but way less CPU used (better 95th percentiles, etc),
and handlers free for other tasks\{quote}
Yes sir, this is our purpose.
{quote}Would it make sense to put this limit behind a configuration
gate?\{quote}
I think we need it. It is mainly for to enable us to 'remove' it , even though
we can't totally remove it because of the added counter on the path.
{quote}Also of note is how your compares show the asyncwal being slightly
better than old wal.\{quote}
Let me check again. And this async_wal is one of Durablilty types. it's not WAL.
[~carp84] ok, I will work on it.)
> Procedure V2
>
>
> Key: HBASE-12439
> URL: https://issues.apache.org/jira/browse/HBASE-12439
> Project: HBase
> Issue Type: New Feature
> Components: master
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Priority: Major
> Labels: reliability
> Attachments: ProcedureV2b.pdf, Procedurev2Notification-Bus.pdf,
> Procedurev2Notification-BusRoadmap.pdf
>
>
> Procedure v2 (aka Notification Bus) aims to provide a unified way to build:
> * multi-steps procedure with a rollback/rollforward ability in case of
> failure (e.g. create/delete table)
> ** HBASE-12070
> * notifications across multiple machines (e.g. ACLs/Labels/Quotas cache
> updates)
> ** Make sure that every machine has the grant/revoke/label
> ** Enforce "space limit" quota across the namespace
> ** HBASE-10295 eliminate permanent replication zk node
> * procedures across multiple machines (e.g. Snapshots)
> * coordinated long-running procedures (e.g. compactions, splits, ...)
> * Synchronous calls, with the ability to see the state/result in case of
> failure.
> ** HBASE-11608 sync split
> still work in progress/initial prototype: https://reviews.apache.org/r/27703/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-12439) Procedure V2
[
https://issues.apache.org/jira/browse/HBASE-12439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388965#comment-16388965
]
Chance Li commented on HBASE-12439:
---
[~stack]
thanks, sir.
{quote}less throughput, but way less CPU used (better 95th percentiles, etc),
and handlers free for other tasks\{quote}
Yes sir, this is our purpose.
{quote}Would it make sense to put this limit behind a configuration
gate?\{quote}
I think we need it. It is mainly for to enable us to 'remove' it , even though
we can't totally remove it because of the added counter on the path.
{quote}Also of note is how your compares show the asyncwal being slightly
better than old wal.\{quote}
Let me check again. And this async_wal is one of Durablilty types. it's not WAL.
[~carp84] ok, I will work on it.
> Procedure V2
>
>
> Key: HBASE-12439
> URL: https://issues.apache.org/jira/browse/HBASE-12439
> Project: HBase
> Issue Type: New Feature
> Components: master
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Priority: Major
> Labels: reliability
> Attachments: ProcedureV2b.pdf, Procedurev2Notification-Bus.pdf,
> Procedurev2Notification-BusRoadmap.pdf
>
>
> Procedure v2 (aka Notification Bus) aims to provide a unified way to build:
> * multi-steps procedure with a rollback/rollforward ability in case of
> failure (e.g. create/delete table)
> ** HBASE-12070
> * notifications across multiple machines (e.g. ACLs/Labels/Quotas cache
> updates)
> ** Make sure that every machine has the grant/revoke/label
> ** Enforce "space limit" quota across the namespace
> ** HBASE-10295 eliminate permanent replication zk node
> * procedures across multiple machines (e.g. Snapshots)
> * coordinated long-running procedures (e.g. compactions, splits, ...)
> * Synchronous calls, with the ability to see the state/result in case of
> failure.
> ** HBASE-11608 sync split
> still work in progress/initial prototype: https://reviews.apache.org/r/27703/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[
https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chance Li updated HBASE-19389:
--
Comment: was deleted
(was: [~stack]
thanks, sir.
{quote} less throughput, but way less CPU used (better 95th percentiles, etc),
and handlers free for other tasks
Yes sir, this is our purpose.
{quote}Would it make sense to put this limit behind a configuration gate?
I think we need it. It is mainly for to enable us to 'remove' it , even though
we can't totally remove it because of the added counter on the path.
{quote}Also of note is how your compares show the asyncwal being slightly
better than old wal.
Let me check again. And this async_wal is one of Durablilty types. it's not
WAL.
[~carp84] ok, I will work on it.)
> Limit concurrency of put with dense (hundreds) columns to prevent write
> handler exhausted
> -
>
> Key: HBASE-19389
> URL: https://issues.apache.org/jira/browse/HBASE-19389
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 2.0.0
> Environment: 2000+ Region Servers
> PCI-E ssd
>Reporter: Chance Li
>Assignee: Chance Li
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: CSLM-concurrent-write.png,
> HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch,
> HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch,
> HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2-V7.patch,
> HBASE-19389-branch-2-V8.patch, HBASE-19389-branch-2.patch, metrics-1.png,
> ycsb-result.png
>
>
> In a large cluster, with a large number of clients, we found the RS's
> handlers are all busy sometimes. And after investigation we found the root
> cause is about CSLM, such as compare function heavy load. We reviewed the
> related WALs, and found that there were many columns (more than 1000 columns)
> were writing at that time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[
https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388962#comment-16388962
]
Chance Li commented on HBASE-19389:
---
[~stack]
thanks, sir.
{quote} less throughput, but way less CPU used (better 95th percentiles, etc),
and handlers free for other tasks
Yes sir, this is our purpose.
{quote}Would it make sense to put this limit behind a configuration gate?
I think we need it. It is mainly for to enable us to 'remove' it , even though
we can't totally remove it because of the added counter on the path.
{quote}Also of note is how your compares show the asyncwal being slightly
better than old wal.
Let me check again. And this async_wal is one of Durablilty types. it's not
WAL.
[~carp84] ok, I will work on it.
> Limit concurrency of put with dense (hundreds) columns to prevent write
> handler exhausted
> -
>
> Key: HBASE-19389
> URL: https://issues.apache.org/jira/browse/HBASE-19389
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 2.0.0
> Environment: 2000+ Region Servers
> PCI-E ssd
>Reporter: Chance Li
>Assignee: Chance Li
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: CSLM-concurrent-write.png,
> HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch,
> HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch,
> HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2-V7.patch,
> HBASE-19389-branch-2-V8.patch, HBASE-19389-branch-2.patch, metrics-1.png,
> ycsb-result.png
>
>
> In a large cluster, with a large number of clients, we found the RS's
> handlers are all busy sometimes. And after investigation we found the root
> cause is about CSLM, such as compare function heavy load. We reviewed the
> related WALs, and found that there were many columns (more than 1000 columns)
> were writing at that time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20003) WALLess HBase on Persistent Memory
[ https://issues.apache.org/jira/browse/HBASE-20003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375425#comment-16375425 ] Chance Li commented on HBASE-20003: --- Cool. But it's maybe difficult for pipeline model to provide the strong consistency guarantee, may need something like Paxos-based replication. > WALLess HBase on Persistent Memory > -- > > Key: HBASE-20003 > URL: https://issues.apache.org/jira/browse/HBASE-20003 > Project: HBase > Issue Type: New Feature >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > > This JIRA aims to make use of persistent memory (pmem) technologies in HBase. > One such usage is to make the Memstore to reside on pmem. Making a persistent > memstore would remove the need for WAL and paves way for a WALLess HBase. > The existing region replica feature could be used here and ensure the data > written to memstores are synchronously replicated to the replicas and ensure > strong consistency of the data. (pipeline model) > Advantages : > - Data Availability : Since the data across replicas are consistent > (synchronously written) our data is always 100 % available. > - Lower MTTR : It becomes easier/faster to switch over to the replicas on a > primary region failure as there is no WAL replay involved. Building the > memstore map data also is much faster than reading the WAL and replaying the > WAL. > - Possibility of bigger memstores : These pmems are designed to have more > memory than DRAMs so it would also enable us to have bigger sized memstores > which leads to lesser flushes/compaction IO. > - Removes the dependency of HDFS on the write path > Initial PoC has been designed and developed. Testing is underway and we would > publish the PoC results along with the design doc sooner. The PoC doc will > talk about the design decisions, the libraries considered to work with these > pmem devices, pros and cons of those libraries and the performance results. > Note : Next gen memory technologies using 3DXPoint gives persistent memory > feature. Such memory DIMMs are soon to appear in the market. The PoC is done > around Intel's ApachePass (AEP) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19959) How much RAM space is to be really consumed by the memstore?
[ https://issues.apache.org/jira/browse/HBASE-19959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359868#comment-16359868 ] Chance Li commented on HBASE-19959: --- To flush all the data to make the memstore size is 0 . > How much RAM space is to be really consumed by the memstore? > > > Key: HBASE-19959 > URL: https://issues.apache.org/jira/browse/HBASE-19959 > Project: HBase > Issue Type: Brainstorming > Components: regionserver >Reporter: Chance Li >Priority: Minor > > Let's consider this scenario where memstoreLAB and ChunkPool is enable and > max memstore size is 10G, and after some time all pooled chunk have been > created, then flush all data, now memstore size is 0 but RAM actually have > consumed 10G, then continue writing big cell which will not use the chunk > pool but jvm heap, then memstore size will be increased to 10G(maybe more > because overhead). now we can see RAM actually consumed 20G (10G of pooled > chunk + 10G java objects), but the max memstore size is only 10G. > what I say is the max memstore size not only take care about the cell "size" > but also RAM really used. This will be a strict memory management: the max > memstore size limit the RAM space which the memstore or related module can be > used. > This really rarely occured. It's just for a robust memory managemant > semantically. > What do you think? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19959) How much RAM space is to be really consumed by the memstore?
[ https://issues.apache.org/jira/browse/HBASE-19959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358014#comment-16358014 ] Chance Li commented on HBASE-19959: --- In this scenario, every chunk in pool was used at least once , and the number of pooled chunk has reached the upper limit. That means chunk pool have take the physical RAM. > How much RAM space is to be really consumed by the memstore? > > > Key: HBASE-19959 > URL: https://issues.apache.org/jira/browse/HBASE-19959 > Project: HBase > Issue Type: Brainstorming > Components: regionserver >Reporter: Chance Li >Priority: Minor > > Let's consider this scenario where memstoreLAB and ChunkPool is enable and > max memstore size is 10G, and after some time all pooled chunk have been > created, then flush all data, now memstore size is 0 but RAM actually have > consumed 10G, then continue writing big cell which will not use the chunk > pool but jvm heap, then memstore size will be increased to 10G(maybe more > because overhead). now we can see RAM actually consumed 20G (10G of pooled > chunk + 10G java objects), but the max memstore size is only 10G. > what I say is the max memstore size not only take care about the cell "size" > but also RAM really used. This will be a strict memory management: the max > memstore size limit the RAM space which the memstore or related module can be > used. > This really rarely occured. It's just for a robust memory managemant > semantically. > What do you think? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-19959) How much RAM space is to be really consumed by the memstore?
Chance Li created HBASE-19959: - Summary: How much RAM space is to be really consumed by the memstore? Key: HBASE-19959 URL: https://issues.apache.org/jira/browse/HBASE-19959 Project: HBase Issue Type: Brainstorming Components: regionserver Reporter: Chance Li Let's consider this scenario where memstoreLAB and ChunkPool is enable and max memstore size is 10G, and after some time all pooled chunk have been created, then flush all data, now memstore size is 0 but RAM actually have consumed 10G, then continue writing big cell which will not use the chunk pool but jvm heap, then memstore size will be increased to 10G(maybe more because overhead). now we can see RAM actually consumed 20G (10G of pooled chunk + 10G java objects), but the max memstore size is only 10G. what I say is the max memstore size not only take care about the cell "size" but also RAM really used. This will be a strict memory management: the max memstore size limit the RAM space which the memstore or related module can be used. This really rarely occured. It's just for a robust memory managemant semantically. What do you think? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355534#comment-16355534 ] Chance Li commented on HBASE-18294: --- [~anastas] Accutally, It's not a new thing. All cells in memstore will be put in a self-managed and contiguous RAM(on or off heap), with necessary meta(index) which physically co-locate the cell data. no overhead. I m new one , and trying to get your opinions. IMO the memory management of RS(RegionServerAccounting) need to be abstractly, which duty is only protecting the RAM(whatever on/off heap) overflow. This will be more clearly and efficiently. And the store(memstore) will have different rules to handle the size(on heap size or off heap size or overhead size or any combination of these sizes), And it's duty is avoiding flush small files. There have a little bad code smell when adding a new module. > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch, > HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch, > HBASE-18294.11.patch, HBASE-18294.11.patch, HBASE-18294.12.patch, > HBASE-18294.13.patch, HBASE-18294.15.patch, HBASE-18294.16.patch, > HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352221#comment-16352221 ] Chance Li commented on HBASE-18294: --- [~eshcar] The store will known the detail about the on-heap/off-heap/data size, it's strategy. The RegionServerAccounting not need to do this. I mean not all the memStore need to count data-size,on-heap size,off-heap size both. so it's not need to do in RegionServerAccounting, but in the store. now the store's memstore can be on-heap or offheap or both, LAB or not, chunkPool or not, include CellArrayMap and CellChunkMap, And new CCSLM (self-managed memory Memstore, it's have not any over-heap). > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch, > HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch, > HBASE-18294.11.patch, HBASE-18294.11.patch, HBASE-18294.12.patch, > HBASE-18294.13.patch, HBASE-18294.15.patch, HBASE-18294.16.patch, > HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352158#comment-16352158 ] Chance Li commented on HBASE-18294: --- [~eshcar] How about we conside the RAM of the Memstore as a whole? We don't need to care this RAM is used by On-Heap or Off-Heap or both. What about we care is this RAM's water. Do flush when it access the Low Mark, Do blocking when access High Mark. Especially, when the memStore is self-managed memory, And in the furture this RAM maybe be used by diff memStores at same time (DefaultMemsotre and as SelfManageMemoryMemstore ). > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch, > HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch, > HBASE-18294.11.patch, HBASE-18294.11.patch, HBASE-18294.12.patch, > HBASE-18294.13.patch, HBASE-18294.15.patch, HBASE-18294.16.patch, > HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285435#comment-16285435 ] Chance Li commented on HBASE-19389: --- checkstyle, run UT again. > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch, > HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2-V7.patch, > HBASE-19389-branch-2-V8.patch, HBASE-19389-branch-2.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: HBASE-19389-branch-2-V8.patch > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch, > HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2-V7.patch, > HBASE-19389-branch-2-V8.patch, HBASE-19389-branch-2.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285326#comment-16285326 ] Chance Li commented on HBASE-19389: --- sorry for the UT error. update patch HBASE-19389-branch-2-V7.patch , and run UT again > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch, > HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2-V7.patch, > HBASE-19389-branch-2.patch, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: HBASE-19389-branch-2-V7.patch > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch, > HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2-V7.patch, > HBASE-19389-branch-2.patch, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285168#comment-16285168 ] Chance Li commented on HBASE-16890: --- Hi Stack, do 24 thread(or 48 threads) mean "threadcount=24" on one YCSB client? hbase.wal.async.use-shared-event-loop is useful on my test env, It's difference with your and ZhangDuo's test result. maybe not only because PCI-e. and because of CPU-BOUND, with multiwal the asyncwal is more better than FSHLog now on my test env. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: AsyncWAL_disruptor.patch, AsyncWAL_disruptor_1 > (2).patch, AsyncWAL_disruptor_3.patch, AsyncWAL_disruptor_3.patch, > AsyncWAL_disruptor_4.patch, AsyncWAL_disruptor_6.patch, > HBASE-16890-rc-v2.patch, HBASE-16890-rc-v3.patch, > HBASE-16890-remove-contention-v1.patch, HBASE-16890-remove-contention.patch, > Screen Shot 2016-10-25 at 7.34.47 PM.png, Screen Shot 2016-10-25 at 7.39.07 > PM.png, Screen Shot 2016-10-25 at 7.39.48 PM.png, Screen Shot 2016-11-04 at > 5.21.27 PM.png, Screen Shot 2016-11-04 at 5.30.18 PM.png, async.svg, > classic.svg, contention.png, contention_defaultWAL.png, > ycsb_FSHlog.vs.Async.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285145#comment-16285145 ] Chance Li commented on HBASE-19389: --- rebased, HBASE-19389-branch-2-V6.patch > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch, > HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: HBASE-19389-branch-2-V6.patch > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch, > HBASE-19389-branch-2-V6.patch, HBASE-19389-branch-2.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284728#comment-16284728 ] Chance Li commented on HBASE-19389: --- [~carp84] Thanks for the reviews. Update the patch. > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch, > HBASE-19389-branch-2.patch, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: HBASE-19389-branch-2-V5.patch > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2-V5.patch, > HBASE-19389-branch-2.patch, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: HBASE-19389-branch-2-V4.patch > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281215#comment-16281215 ] Chance Li commented on HBASE-19389: --- patch v4 for checkstyle > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280308#comment-16280308 ] Chance Li commented on HBASE-19389: --- done [~ted_yu] > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2.patch, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280301#comment-16280301 ] Chance Li commented on HBASE-19389: --- [~syuanjiang] [~ted_yu] Thanks for the reviews. Update the patch. > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2.patch, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: HBASE-19389-branch-2-V3.patch > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2.patch, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18962) Support atomic BatchOperations through batchMutate()
[
https://issues.apache.org/jira/browse/HBASE-18962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280242#comment-16280242
]
Chance Li commented on HBASE-18962:
---
[~uagashe]
I will do some UT for this later. : ) I only see when atomic is true,
doBatchOp() method throw the exception directly, and not sets the index at all
{code}
if (atomic) {
throw ie;
}
{code}
> Support atomic BatchOperations through batchMutate()
>
>
> Key: HBASE-18962
> URL: https://issues.apache.org/jira/browse/HBASE-18962
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0-alpha-3
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Fix For: 2.0.0-beta-1
>
> Attachments: hbase-18962.master.001.patch,
> hbase-18962.master.002.patch, hbase-18962.master.003.patch,
> hbase-18962.master.004.patch
>
>
> Support all mutations in BatchOperations to be applied atomically (all or
> none) by locking all rows corresponding to mutations exclusively.
> mutateRows() which uses MultiRowMutationProcessor applies all mutations
> atomically and batchMutate() is non-atomic. To unify code paths, isAtomic()
> attribute can be added to BatchOperations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18962) Support atomic BatchOperations through batchMutate()
[
https://issues.apache.org/jira/browse/HBASE-18962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279954#comment-16279954
]
Chance Li commented on HBASE-18962:
---
[~uagashe]
I have not done the test now, just read the code.
So what's index of the #resultOrException when catching the #doBatchOp
exception?
I think it should be builder#setException(pair) to set RegionActionResult fail,
or do resultOrExceptionBuilder.setException one by one for all mutations ，
like
{code}
for (Action mutation : mutations) {
builder.addResultOrException(getResultOrException(ie,
mutation.getIndex()));
}
{code}
> Support atomic BatchOperations through batchMutate()
>
>
> Key: HBASE-18962
> URL: https://issues.apache.org/jira/browse/HBASE-18962
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0-alpha-3
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Fix For: 2.0.0-beta-1
>
> Attachments: hbase-18962.master.001.patch,
> hbase-18962.master.002.patch, hbase-18962.master.003.patch,
> hbase-18962.master.004.patch
>
>
> Support all mutations in BatchOperations to be applied atomically (all or
> none) by locking all rows corresponding to mutations exclusively.
> mutateRows() which uses MultiRowMutationProcessor applies all mutations
> atomically and batchMutate() is non-atomic. To unify code paths, isAtomic()
> attribute can be added to BatchOperations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write hander exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278866#comment-16278866 ] Chance Li commented on HBASE-19389: --- [~te...@apache.org] Thanks for the reviews. Update the patch. > Limit concurrency of put with dense (hundreds) columns to prevent write > hander exhausted > > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write hander exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: HBASE-19389-branch-2-V2.patch > Limit concurrency of put with dense (hundreds) columns to prevent write > hander exhausted > > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write hander exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276971#comment-16276971 ] Chance Li commented on HBASE-19389: --- https://reviews.apache.org/r/64300/ > Limit concurrency of put with dense (hundreds) columns to prevent write > hander exhausted > > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, HBASE-19389-branch-2.patch, > metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write hander exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: (was: HBASE-19389-branch-2.patch.POC) > Limit concurrency of put with dense (hundreds) columns to prevent write > hander exhausted > > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, HBASE-19389-branch-2.patch, > metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write hander exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276914#comment-16276914 ] Chance Li commented on HBASE-19389: --- [~carp84] I make a new patch. The result of test meet our expectations: normal case no regression, dense (hundreds) columns to prevent write handler exhausted. And the doubtful question of HBASE-18962 is unrelated. I'll confirm it later. btw, I was trying to use #Coprocessor, it's will more clear, But it looks like it's not fit for this. pls help review the patch. Any suggestion will be appreciated. > Limit concurrency of put with dense (hundreds) columns to prevent write > hander exhausted > > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, HBASE-19389-branch-2.patch, > metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write hander exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: HBASE-19389-branch-2.patch > Limit concurrency of put with dense (hundreds) columns to prevent write > hander exhausted > > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, HBASE-19389-branch-2.patch, > HBASE-19389-branch-2.patch.POC, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write hander exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Fix Version/s: (was: 3.0.0) Status: Patch Available (was: Open) > Limit concurrency of put with dense (hundreds) columns to prevent write > hander exhausted > > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2.patch.POC, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write hander exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: HBASE-19389-branch-2.patch.POC > Limit concurrency of put with dense (hundreds) columns to prevent write > hander exhausted > > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0, 3.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2.patch.POC, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write hander exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276058#comment-16276058 ] Chance Li commented on HBASE-19389: --- The result is not as good as 1.x with patch(But it's still worked) , and the main reason is the modification of HBASE-18962. I need think more. So I changed patch to POC (patch is simple), There may be better suggestions for better rs self-protection . > Limit concurrency of put with dense (hundreds) columns to prevent write > hander exhausted > > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0, 3.0.0 > > Attachments: CSLM-concurrent-write.png, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18962) Support atomic BatchOperations through batchMutate()
[
https://issues.apache.org/jira/browse/HBASE-18962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16275635#comment-16275635
]
Chance Li edited comment on HBASE-18962 at 12/2/17 4:13 PM:
```
try {
doBatchOp(builder, region, quota, mutations, cellScanner,
spaceQuotaEnforcement, false);
} catch (IOException ioe) {
rpcServer.getMetrics().exception(ioe);
NameBytesPair pair = ResponseConverter.buildException(ioe);
resultOrExceptionBuilder.setException(pair);
context.incrementResponseExceptionSize(pair.getSerializedSize());
builder.addResultOrException(resultOrExceptionBuilder.build());
}
```
#resultOrExceptionBuilder.setException(pair) here maybe need replace to
#builder.setException(pair)? [~uagashe]
was (Author: chancelq):
try {
doBatchOp(builder, region, quota, mutations, cellScanner,
spaceQuotaEnforcement, false);
} catch (IOException ioe) {
rpcServer.getMetrics().exception(ioe);
NameBytesPair pair = ResponseConverter.buildException(ioe);
resultOrExceptionBuilder.setException(pair);
context.incrementResponseExceptionSize(pair.getSerializedSize());
builder.addResultOrException(resultOrExceptionBuilder.build());
}
#resultOrExceptionBuilder.setException(pair) here maybe need replace to
#builder.setException(pair)? [~uagashe]
> Support atomic BatchOperations through batchMutate()
>
>
> Key: HBASE-18962
> URL: https://issues.apache.org/jira/browse/HBASE-18962
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0-alpha-3
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Fix For: 2.0.0-beta-1
>
> Attachments: hbase-18962.master.001.patch,
> hbase-18962.master.002.patch, hbase-18962.master.003.patch,
> hbase-18962.master.004.patch
>
>
> Support all mutations in BatchOperations to be applied atomically (all or
> none) by locking all rows corresponding to mutations exclusively.
> mutateRows() which uses MultiRowMutationProcessor applies all mutations
> atomically and batchMutate() is non-atomic. To unify code paths, isAtomic()
> attribute can be added to BatchOperations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18962) Support atomic BatchOperations through batchMutate()
[
https://issues.apache.org/jira/browse/HBASE-18962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16275635#comment-16275635
]
Chance Li commented on HBASE-18962:
---
try {
doBatchOp(builder, region, quota, mutations, cellScanner,
spaceQuotaEnforcement, false);
} catch (IOException ioe) {
rpcServer.getMetrics().exception(ioe);
NameBytesPair pair = ResponseConverter.buildException(ioe);
resultOrExceptionBuilder.setException(pair);
context.incrementResponseExceptionSize(pair.getSerializedSize());
builder.addResultOrException(resultOrExceptionBuilder.build());
}
#resultOrExceptionBuilder.setException(pair) here maybe need replace to
#builder.setException(pair)? [~uagashe]
> Support atomic BatchOperations through batchMutate()
>
>
> Key: HBASE-18962
> URL: https://issues.apache.org/jira/browse/HBASE-18962
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0-alpha-3
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Fix For: 2.0.0-beta-1
>
> Attachments: hbase-18962.master.001.patch,
> hbase-18962.master.002.patch, hbase-18962.master.003.patch,
> hbase-18962.master.004.patch
>
>
> Support all mutations in BatchOperations to be applied atomically (all or
> none) by locking all rows corresponding to mutations exclusively.
> mutateRows() which uses MultiRowMutationProcessor applies all mutations
> atomically and batchMutate() is non-atomic. To unify code paths, isAtomic()
> attribute can be added to BatchOperations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write hander exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274831#comment-16274831 ] Chance Li commented on HBASE-19389: --- My writing is a bit confusing between parallel and concurrency. It should all be parallel. > Limit concurrency of put with dense (hundreds) columns to prevent write > hander exhausted > > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0, 3.0.0 > > Attachments: CSLM-concurrent-write.png, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write hander exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272941#comment-16272941 ] Chance Li edited comment on HBASE-19389 at 12/1/17 7:07 PM: We have done some small tests for concurrent writing to CSLM, see test number below. We can see the RT growth is very fast (performance reduction) in the case of concurrency. !CSLM-concurrent-write.png! About solution, one choice is to protect CSLM avoiding large concurrency writing, another is to improve the CSLM. By the way, In our scene we don't want to use Qos(request-throttling). We have chosen a more engineering solution which is to protect CSLM avoiding large concurrency and many columns writing. In this way, we can avoid the all RS handlers doing a more slow call. In another word, the other calls have chance to be handled. about the patch: 1. Dynamic configuration: such as #parallelPutToStoreThreadLimitCheckMinColumnNum and #parallelPutToStoreThreadLimit. 2. Return #RegionTooBusyException When it exceeds the threshold. 3. It's not strong limit, we wan't use lock. so handler maybe busy in short time. 4. Only for multi op, not Append. ycsb result with patch: !ycsb-result.png! metrics: !metrics-1.png! Welcome any suggestion. And I will upload the patch in 2 days , and upload more test number. was (Author: chancelq): We have done some small tests for concurrent writing to CSLM, see test number below. We can see the RT growth is very fast (performance reduction) in the case of concurrency. !CSLM-concurrent-write.png! About solution, one choice is to protect CSLM avoiding large concurrency writing, another is to improve the CSLM. By the way, In our scene we don't want to use Qos(request-throttling). We have chosen a more engineering solution which is to protect CSLM avoiding large concurrency and many columns writing. In this way, we can avoid the all RS handlers doing a more slow call. In another word, the other calls have chance to be handled. about the patch: 1. Dynamic configuration: such as min column num and concurrent num. 2. Return #RegionTooBusyException When it exceeds the threshold. 3. It's not strong limit, we wan't use lock. so handler maybe busy in short time. 4. Only for multi op, not Append. ycsb result with patch: !ycsb-result.png! metrics: !metrics-1.png! Welcome any suggestion. And I will upload the patch in 2 days , and upload more test number. > Limit concurrency of put with dense (hundreds) columns to prevent write > hander exhausted > > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0, 3.0.0 > > Attachments: CSLM-concurrent-write.png, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write hander exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274786#comment-16274786 ] Chance Li commented on HBASE-19389: --- bq. In the solution - abt the concurrent max #columns . So this is the max number of different columns for same CF but across rows? 1. for same CF maybe across rows, on test the row is random. 2. when the PUT's cell more than #parallelPutToStoreThreadLimitCheckMinColumnNum, and the concurrency of adding to the CSLM exceed the #parallelPutToStoreThreadLimit, then result in RegionTooBusyException. We want to protected RS avoiding do such slow call ( see the CSLM's concurrency writing test number above), I'd say this is a part of RS self-protected (improve the reliability of RS) under some extreme case like as the abused client. > Limit concurrency of put with dense (hundreds) columns to prevent write > hander exhausted > > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0, 3.0.0 > > Attachments: CSLM-concurrent-write.png, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19389) RS's handlers are all busy when writing many columns (more than 1000 columns)
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273004#comment-16273004 ] Chance Li commented on HBASE-19389: --- bq. For metrics-1.png, was the stat collected with or without protection ? Only with protection, sir. The three parts is about #SKIP_WAL, #ASYNC_WAL and #SYNC_WAL. Without protection, all handlers were always busy. bq. Does that mean the other half handlers were idle ? yes sir. Maybe the reason is client do some sleep when it receive exception. I'll check it. But actually, we aim to ensure that the #handler avoiding to do slower call(This actually reduces the performance of the server). All handlers are busy is not a problem. The other calls come from other client for normal region(store) will be have chance to be handled. In my test, there are no the other calls . Thanks, sir. [~te...@apache.org] > RS's handlers are all busy when writing many columns (more than 1000 columns) > -- > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li >Priority: Minor > Fix For: 2.0.0, 3.0.0 > > Attachments: CSLM-concurrent-write.png, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19389) RS's handlers are all busy when writing many columns (more than 1000 columns)
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272941#comment-16272941 ] Chance Li commented on HBASE-19389: --- We have done some small tests for concurrent writing to CSLM, see test number below. We can see the RT growth is very fast (performance reduction) in the case of concurrency. !CSLM-concurrent-write.png! About solution, one choice is to protect CSLM avoiding large concurrency writing, another is to improve the CSLM. By the way, In our scene we don't want to use Qos(request-throttling). We have chosen a more engineering solution which is to protect CSLM avoiding large concurrency and many columns writing. In this way, we can avoid the all RS handlers doing a more slow call. In another word, the other calls have chance to be handled. about the patch: 1. Dynamic configuration: such as min column num and concurrent num. 2. Return #RegionTooBusyException When it exceeds the threshold. 3. It's not strong limit, we wan't use lock. so handler maybe busy in short time. 4. Only for multi op, not Append. ycsb result with patch: !ycsb-result.png! metrics: !metrics-1.png! Welcome any suggestion. And I will upload the patch in 2 days , and upload more test number. > RS's handlers are all busy when writing many columns (more than 1000 columns) > -- > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li >Priority: Minor > Fix For: 2.0.0, 3.0.0 > > Attachments: CSLM-concurrent-write.png, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) RS's handlers are all busy when writing many columns (more than 1000 columns)
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: metrics-1.png > RS's handlers are all busy when writing many columns (more than 1000 columns) > -- > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li >Priority: Minor > Fix For: 2.0.0, 3.0.0 > > Attachments: CSLM-concurrent-write.png, metrics-1.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) RS's handlers are all busy when writing many columns (more than 1000 columns)
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: ycsb-result.png > RS's handlers are all busy when writing many columns (more than 1000 columns) > -- > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li >Priority: Minor > Fix For: 2.0.0, 3.0.0 > > Attachments: CSLM-concurrent-write.png, ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) RS's handlers are all busy when writing many columns (more than 1000 columns)
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: CSLM-concurrent-write.png > RS's handlers are all busy when writing many columns (more than 1000 columns) > -- > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li >Priority: Minor > Fix For: 2.0.0, 3.0.0 > > Attachments: CSLM-concurrent-write.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19389) RS's handlers are all busy when writing many columns (more than 1000 columns)
Chance Li created HBASE-19389: - Summary: RS's handlers are all busy when writing many columns (more than 1000 columns) Key: HBASE-19389 URL: https://issues.apache.org/jira/browse/HBASE-19389 Project: HBase Issue Type: Improvement Components: hbase Affects Versions: 2.0.0 Environment: 2000+ Region Servers PCI-E ssd Reporter: Chance Li Assignee: Chance Li Priority: Minor Fix For: 2.0.0, 3.0.0 In a large cluster, with a large number of clients, we found the RS's handlers are all busy sometimes. And after investigation we found the root cause is about CSLM, such as compare function heavy load. We reviewed the related WALs, and found that there were many columns (more than 1000 columns) were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19344) improve asyncWAL by using Independent thread for netty #IO in FanOutOneBlockAsyncDFSOutput
[ https://issues.apache.org/jira/browse/HBASE-19344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272066#comment-16272066 ] Chance Li edited comment on HBASE-19344 at 11/30/17 3:34 AM: - These tests are based on SSD. And the result is what we expect. was (Author: chancelq): These tests are based on SSD. And the result is what we expect. !HBASE-19344-branch.ycsb.png! > improve asyncWAL by using Independent thread for netty #IO in > FanOutOneBlockAsyncDFSOutput > --- > > Key: HBASE-19344 > URL: https://issues.apache.org/jira/browse/HBASE-19344 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0-beta-1 >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: HBASE-19344-branch-ycsb-1.png, > HBASE-19344-branch.ycsb.png, HBASE-19344-branch.ycsb.png, > HBASE-19344-branch2.patch, HBASE-19344-branch2.patch.2.POC, > wal-1-test-result.png, wal-8-test-result.png, > ycsb_result_apache20_async_wal.pdf > > > The logic now is that the netty #IO thread and asyncWal's thread are the same > one. > Improvement proposal: > 1, Split into two. > 2, All multiWal share the netty #IO thread pool. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Issue Comment Deleted] (HBASE-19344) improve asyncWAL by using Independent thread for netty #IO in FanOutOneBlockAsyncDFSOutput
[ https://issues.apache.org/jira/browse/HBASE-19344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19344: -- Comment: was deleted (was: !HBASE-19344-branch-ycsb-1.png!) > improve asyncWAL by using Independent thread for netty #IO in > FanOutOneBlockAsyncDFSOutput > --- > > Key: HBASE-19344 > URL: https://issues.apache.org/jira/browse/HBASE-19344 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0-beta-1 >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: HBASE-19344-branch-ycsb-1.png, > HBASE-19344-branch.ycsb.png, HBASE-19344-branch.ycsb.png, > HBASE-19344-branch2.patch, HBASE-19344-branch2.patch.2.POC, > wal-1-test-result.png, wal-8-test-result.png, > ycsb_result_apache20_async_wal.pdf > > > The logic now is that the netty #IO thread and asyncWal's thread are the same > one. > Improvement proposal: > 1, Split into two. > 2, All multiWal share the netty #IO thread pool. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19344) improve asyncWAL by using Independent thread for netty #IO in FanOutOneBlockAsyncDFSOutput
[ https://issues.apache.org/jira/browse/HBASE-19344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272066#comment-16272066 ] Chance Li edited comment on HBASE-19344 at 11/30/17 3:32 AM: - These tests are based on SSD. And the result is what we expect. !HBASE-19344-branch.ycsb.png! was (Author: chancelq): These tests are based on SSD. And the result is what we expect. !HBASE-19344-branch.ycsb.png! > improve asyncWAL by using Independent thread for netty #IO in > FanOutOneBlockAsyncDFSOutput > --- > > Key: HBASE-19344 > URL: https://issues.apache.org/jira/browse/HBASE-19344 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0-beta-1 >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: HBASE-19344-branch-ycsb-1.png, > HBASE-19344-branch.ycsb.png, HBASE-19344-branch.ycsb.png, > HBASE-19344-branch2.patch, HBASE-19344-branch2.patch.2.POC, > wal-1-test-result.png, wal-8-test-result.png, > ycsb_result_apache20_async_wal.pdf > > > The logic now is that the netty #IO thread and asyncWal's thread are the same > one. > Improvement proposal: > 1, Split into two. > 2, All multiWal share the netty #IO thread pool. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19344) improve asyncWAL by using Independent thread for netty #IO in FanOutOneBlockAsyncDFSOutput
[ https://issues.apache.org/jira/browse/HBASE-19344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272108#comment-16272108 ] Chance Li commented on HBASE-19344: --- !HBASE-19344-branch-ycsb-1.png! > improve asyncWAL by using Independent thread for netty #IO in > FanOutOneBlockAsyncDFSOutput > --- > > Key: HBASE-19344 > URL: https://issues.apache.org/jira/browse/HBASE-19344 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0-beta-1 >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: HBASE-19344-branch-ycsb-1.png, > HBASE-19344-branch.ycsb.png, HBASE-19344-branch.ycsb.png, > HBASE-19344-branch2.patch, HBASE-19344-branch2.patch.2.POC, > wal-1-test-result.png, wal-8-test-result.png, > ycsb_result_apache20_async_wal.pdf > > > The logic now is that the netty #IO thread and asyncWal's thread are the same > one. > Improvement proposal: > 1, Split into two. > 2, All multiWal share the netty #IO thread pool. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
