[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21256: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: HBASE-21256-branch-1.patch, HBASE-21256-v1.patch, > HBASE-21256-v10.patch, HBASE-21256-v11.patch, HBASE-21256-v12.patch, > HBASE-21256-v2.patch, HBASE-21256-v3.patch, HBASE-21256-v4.patch, > ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21256: --- Attachment: HBASE-21256-branch-1.patch > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: HBASE-21256-branch-1.patch, HBASE-21256-v1.patch, > HBASE-21256-v10.patch, HBASE-21256-v11.patch, HBASE-21256-v12.patch, > HBASE-21256-v2.patch, HBASE-21256-v3.patch, HBASE-21256-v4.patch, > ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21256: -- Fix Version/s: 2.2.0 1.5.0 > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v10.patch, > HBASE-21256-v11.patch, HBASE-21256-v12.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, HBASE-21256-v4.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: HBASE-21256-v12.patch > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v10.patch, > HBASE-21256-v11.patch, HBASE-21256-v12.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, HBASE-21256-v4.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: HBASE-21256-v11.patch > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v10.patch, > HBASE-21256-v11.patch, HBASE-21256-v2.patch, HBASE-21256-v3.patch, > HBASE-21256-v4.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: HBASE-21256-v10.patch > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v10.patch, > HBASE-21256-v2.patch, HBASE-21256-v3.patch, HBASE-21256-v4.patch, > ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: HBASE-21256-v4.patch > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, HBASE-21256-v4.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: HBASE-21256-v3.patch > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: (was: HBASE-21256-v3.patch) > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: HBASE-21256-v3.patch > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: (was: HBASE-21256-v3.patch) > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, ITBLL-1.png, > ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: HBASE-21256-v3.patch > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: HBASE-21256-v2.patch Update patch v2: against hbase.client.keyvalue.maxsize > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, ITBLL-1.png, > ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21256: --- Status: Patch Available (was: Open) > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Description: Recently, I use ITBLL to test some features in our company. I have encountered the following problems: 1. Generator is too slow at the generating stage, the root cause is SecureRandom. There is a global lock in SecureRandom( See the following picture). I use Random instead of SecureRandom, and it could speed up this stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but speaking of generating random bytes, in my opnion, it is the same with Random. !ITBLL-1.png! 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by create keyString variable. However, keyString may never be used if test result is correct.(and that's in most cases). Just delay creating keyString can yield huge performance boost in verifing stage. !ITBLL-2.png! 3.Arguments check is needed, because there's constraint between arguments. If we broken this constraint, we can not get a correct circular list. 4.Let big family value size could be configured. 5.Avoid starting RS at backup master was: Recently, I use ITBLL to test some features in our company. I have encountered the following problems: 1. Generator is too slow at the generating stage, the root cause is SecureRandom. There is a global lock in SecureRandom( See the following picture). I use Random instead of SecureRandom, and it could speed up this stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but speaking of generating random bytes, in my opnion, it is the same with Random. !ITBLL-1.png! 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by create keyString variable. However, keyString is never be used if test result is correct.(and that's in most cases). Just delay creating keyString can yield huge performance boost in verifing stage. !ITBLL-2.png! 3.Arguments check is needed, because there's constraint between arguments. If we broken this constraint, we can not get a correct circular list. 4.Let big family value size could be configured. 5.Avoid starting RS at backup master > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Description: Recently, I use ITBLL to test some features in our company. I have encountered the following problems: 1. Generator is too slow at the generating stage, the root cause is SecureRandom. There is a global lock in SecureRandom( See the following picture). I use Random instead of SecureRandom, and it could speed up this stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but speaking of generating random bytes, in my opnion, it is the same with Random. !ITBLL-1.png! 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by create keyString variable. However, keyString is never be used if test result is correct.(and that's in most cases). Just delay creating keyString can yield huge performance boost in verifing stage. !ITBLL-2.png! 3.Arguments check is needed, because there's constraint between arguments. If we broken this constraint, we can not get a correct circular list. 4.Let big family value size could be configured. 5.Avoid starting RS at backup master was: Recently, I use ITBLL to test some features in our company. I have encountered the following problems: 1. Generator is too slow at the generating stage, the root cause is SecureRandom. There is a global lock in SecureRandom( See the following picture). I use Random instead of SecureRandom, and it could speed up this stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but speaking of generating random bytes, in my opnion, it is the same with Random. !ITBLL-1.png! 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by create keyString variable. However, keyString is never be used if test result is correct.(and that's in most cases). Just delay creating keyString can yield huge performance boost in verifing stage. !ITBLL-2.png! 3.Arguments check is needed, because there's constraint between arguments. If we broken this constraint, we can not get a correct circular list. 4.Let big family value size could be configured. > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString is never be used if test result > is correct.(and that's in most cases). Just delay creating keyString can > yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: HBASE-21256-v1.patch > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString is never be used if test result > is correct.(and that's in most cases). Just delay creating keyString can > yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
