[jira] [Commented] (HBASE-25346) hbase2.x the performance is lower than hbase 1.x ？
[
https://issues.apache.org/jira/browse/HBASE-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247726#comment-17247726
]
nilonealex commented on HBASE-25346:
By the following methods reak the limit on the number of fields （now the test
is on Hbase 2.1.2）
{code:java}
disable 'test_person'
alter 'test_person', CONFIGURATION =>
{'hbase.region.store.parallel.put.limit.min.column.count' => 200,
'hbase.region.store.parallel.put.limit' => 100}
enable 'test_person'
{code}
> hbase2.x the performance is lower than hbase 1.x ？
> ---
>
> Key: HBASE-25346
> URL: https://issues.apache.org/jira/browse/HBASE-25346
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.2
>Reporter: nilonealex
>Priority: Critical
> Attachments: error_pe_randomWrite.log, error_pe_randomWrite.log,
> hbase-pe-performace-test.log, hbase-site.xml, test_for_randomWrite.log,
> test_for_randomWrite_hbase1.2.1.log
>
>
> Recently we found that the newly built production hbase cluster is running a
> bit slow , the hadoop version is Hbase2.0.2 ( HDP3.1.1) and it has 100
> nodes.Then we begin to do load & query performance verification between
> Hbase2.0.2 ( HDP3.1.1) & Hbase1.2.0 ( CDH5.13.3 ) test environment （4nodes）,
> found that : put data based on hbase2.0 is much slower than hbase1.x (the
> former is almost half of the latter), I use BufferedMutator and
> BufferedMutatorParams term for batch put to improve efficiency. More
> confusing is the performance of the production environment is worse than my
> test environment
> Some of the codes are as follows:
> ---
> {color:#4C9AFF}List mutator = new ArrayList<>();
> BufferedMutator table = null;
> BufferedMutatorParams params = new
> BufferedMutatorParams(TableName.valueOf(fileHbRule.getHbaseTableName()));
> params.writeBufferSize(fileHbRule.getFlushBuffer().intValue()*1024*1024);
> table = connection.getBufferedMutator(params);
>
> mutator.add(p);
> if(totalCnts % 5000 == 0 ) {
> table.mutate(mutator);
> mutator.clear();
> }{color}
> ---
> The file to put is a text format file: 2 million rows comma-separated text
> file, each row records 110 columns, total size is about 1G. In addition to
> the main parameter configuration such as heap memory, I kept the default
> parameter values ??for most of the hbase services.
> The load program is designed for single thread.
> The following is the progress information :
> --- Hbase1.2.0 ( CDH5.13.3 )
>
> 2020-12-01 16:48:18 inserted: 10
> 2020-12-01 16:48:36 inserted: 20
> 2020-12-01 16:48:52 inserted: 30
> 2020-12-01 16:49:08 inserted: 40
> 2020-12-01 16:49:23 inserted: 50
> 2020-12-01 16:49:39 inserted: 60
> 2020-12-01 16:49:56 inserted: 70
> 2020-12-01 16:50:12 inserted: 80
> 2020-12-01 16:50:29 inserted: 90
> 2020-12-01 16:50:45 inserted: 100
> 2020-12-01 16:51:01 inserted: 110
> 2020-12-01 16:51:17 inserted: 120
> 2020-12-01 16:51:34 inserted: 130
> 2020-12-01 16:51:49 inserted: 140
> 2020-12-01 16:52:05 inserted: 150
> 2020-12-01 16:52:21 inserted: 160
> 2020-12-01 16:52:40 inserted: 170
> 2020-12-01 16:52:57 inserted: 180
> 2020-12-01 16:53:19 inserted: 190
> 2020-12-01 16:53:42 inserted: 200
> 2020-12-01 16:53:48 inserted: 200
> imp finished ok!
> --job finished--
> ---Hbase.2.0.2 (
> HDP3.1.1)-
> 2020-12-01 17:25:24 inserted: 10
> 2020-12-01 17:26:03 inserted: 20
> 2020-12-01 17:26:39 inserted: 30
> 2020-12-01 17:27:13 inserted: 40
> 2020-12-01 17:27:47 inserted: 50
> 2020-12-01 17:28:23 inserted: 60
> 2020-12-01 17:29:03 inserted: 70
> 2020-12-01 17:29:40 inserted: 80
> 2020-12-01 17:30:15 inserted: 90
> 2020-12-01 17:30:51 inserted: 100
> 2020-12-01 17:31:27 inserted: 110
> 2020-12-01 17:32:03 inserted: 120
> 2020-12-01 17:32:39 inserted: 130
> 2020-12-01 17:33:14 inserted: 140
> 2020-12-01 17:33:50 inserted: 150
> 2020-12-01 17:34:25 inserted: 160
> 2020-12-01 17:35:01 inserted: 170
> 2020-12-01 17:35:38 inserted: 180
> 2020-12-01 17:36:14 inserted: 190
> 2020-12-01 17:36:51 inserted: 200
> 2020-12-01 17:36:55 inserted: 200
> imp finished ok!
> --job finished--
> returnCode=0
> In addition, we also did some benchmark tests on the production cluster.The
> delay is seem to be a bit high. The detailed report is in the attachment.
> Are there any key points that I have not done configuration?
[jira] [Commented] (HBASE-25346) hbase2.x the performance is lower than hbase 1.x ？
[
https://issues.apache.org/jira/browse/HBASE-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247700#comment-17247700
]
nilonealex commented on HBASE-25346:
I have a new discovery： The number of columns has a very significant
impact。When set 100 columns ,the PE run ok, when set 101 columns , the PE
failed， and my program just use 110 columns。
so what does this mean ?
[^error_pe_randomWrite.log]
> hbase2.x the performance is lower than hbase 1.x ？
> ---
>
> Key: HBASE-25346
> URL: https://issues.apache.org/jira/browse/HBASE-25346
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.2
>Reporter: nilonealex
>Priority: Critical
> Attachments: error_pe_randomWrite.log, hbase-pe-performace-test.log,
> hbase-site.xml, test_for_randomWrite.log, test_for_randomWrite_hbase1.2.1.log
>
>
> Recently we found that the newly built production hbase cluster is running a
> bit slow , the hadoop version is Hbase2.0.2 ( HDP3.1.1) and it has 100
> nodes.Then we begin to do load & query performance verification between
> Hbase2.0.2 ( HDP3.1.1) & Hbase1.2.0 ( CDH5.13.3 ) test environment （4nodes）,
> found that : put data based on hbase2.0 is much slower than hbase1.x (the
> former is almost half of the latter), I use BufferedMutator and
> BufferedMutatorParams term for batch put to improve efficiency. More
> confusing is the performance of the production environment is worse than my
> test environment
> Some of the codes are as follows:
> ---
> {color:#4C9AFF}List mutator = new ArrayList<>();
> BufferedMutator table = null;
> BufferedMutatorParams params = new
> BufferedMutatorParams(TableName.valueOf(fileHbRule.getHbaseTableName()));
> params.writeBufferSize(fileHbRule.getFlushBuffer().intValue()*1024*1024);
> table = connection.getBufferedMutator(params);
>
> mutator.add(p);
> if(totalCnts % 5000 == 0 ) {
> table.mutate(mutator);
> mutator.clear();
> }{color}
> ---
> The file to put is a text format file: 2 million rows comma-separated text
> file, each row records 110 columns, total size is about 1G. In addition to
> the main parameter configuration such as heap memory, I kept the default
> parameter values ??for most of the hbase services.
> The load program is designed for single thread.
> The following is the progress information :
> --- Hbase1.2.0 ( CDH5.13.3 )
>
> 2020-12-01 16:48:18 inserted: 10
> 2020-12-01 16:48:36 inserted: 20
> 2020-12-01 16:48:52 inserted: 30
> 2020-12-01 16:49:08 inserted: 40
> 2020-12-01 16:49:23 inserted: 50
> 2020-12-01 16:49:39 inserted: 60
> 2020-12-01 16:49:56 inserted: 70
> 2020-12-01 16:50:12 inserted: 80
> 2020-12-01 16:50:29 inserted: 90
> 2020-12-01 16:50:45 inserted: 100
> 2020-12-01 16:51:01 inserted: 110
> 2020-12-01 16:51:17 inserted: 120
> 2020-12-01 16:51:34 inserted: 130
> 2020-12-01 16:51:49 inserted: 140
> 2020-12-01 16:52:05 inserted: 150
> 2020-12-01 16:52:21 inserted: 160
> 2020-12-01 16:52:40 inserted: 170
> 2020-12-01 16:52:57 inserted: 180
> 2020-12-01 16:53:19 inserted: 190
> 2020-12-01 16:53:42 inserted: 200
> 2020-12-01 16:53:48 inserted: 200
> imp finished ok!
> --job finished--
> ---Hbase.2.0.2 (
> HDP3.1.1)-
> 2020-12-01 17:25:24 inserted: 10
> 2020-12-01 17:26:03 inserted: 20
> 2020-12-01 17:26:39 inserted: 30
> 2020-12-01 17:27:13 inserted: 40
> 2020-12-01 17:27:47 inserted: 50
> 2020-12-01 17:28:23 inserted: 60
> 2020-12-01 17:29:03 inserted: 70
> 2020-12-01 17:29:40 inserted: 80
> 2020-12-01 17:30:15 inserted: 90
> 2020-12-01 17:30:51 inserted: 100
> 2020-12-01 17:31:27 inserted: 110
> 2020-12-01 17:32:03 inserted: 120
> 2020-12-01 17:32:39 inserted: 130
> 2020-12-01 17:33:14 inserted: 140
> 2020-12-01 17:33:50 inserted: 150
> 2020-12-01 17:34:25 inserted: 160
> 2020-12-01 17:35:01 inserted: 170
> 2020-12-01 17:35:38 inserted: 180
> 2020-12-01 17:36:14 inserted: 190
> 2020-12-01 17:36:51 inserted: 200
> 2020-12-01 17:36:55 inserted: 200
> imp finished ok!
> --job finished--
> returnCode=0
> In addition, we also did some benchmark tests on the production cluster.The
> delay is seem to be a bit high. The detailed report is in the attachment.
> Are there any key points that I have not done configuration? or，, this
> version has performance defects ?
--
This message was sent by Atlassian Jira
[jira] [Commented] (HBASE-25346) hbase2.x the performance is lower than hbase 1.x ？
[
https://issues.apache.org/jira/browse/HBASE-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247692#comment-17247692
]
ramkrishna.s.vasudevan commented on HBASE-25346:
>From both the logs that you have attached (the PE output) of 2.x and 1.2.x it
>seems the total run time difference is about 2 secs. you mean this 2 secs is
>consistently observed? (like around 3%).
> hbase2.x the performance is lower than hbase 1.x ？
> ---
>
> Key: HBASE-25346
> URL: https://issues.apache.org/jira/browse/HBASE-25346
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.2
>Reporter: nilonealex
>Priority: Critical
> Attachments: hbase-pe-performace-test.log, hbase-site.xml,
> test_for_randomWrite.log, test_for_randomWrite_hbase1.2.1.log
>
>
> Recently we found that the newly built production hbase cluster is running a
> bit slow , the hadoop version is Hbase2.0.2 ( HDP3.1.1) and it has 100
> nodes.Then we begin to do load & query performance verification between
> Hbase2.0.2 ( HDP3.1.1) & Hbase1.2.0 ( CDH5.13.3 ) test environment （4nodes）,
> found that : put data based on hbase2.0 is much slower than hbase1.x (the
> former is almost half of the latter), I use BufferedMutator and
> BufferedMutatorParams term for batch put to improve efficiency. More
> confusing is the performance of the production environment is worse than my
> test environment
> Some of the codes are as follows:
> ---
> {color:#4C9AFF}List mutator = new ArrayList<>();
> BufferedMutator table = null;
> BufferedMutatorParams params = new
> BufferedMutatorParams(TableName.valueOf(fileHbRule.getHbaseTableName()));
> params.writeBufferSize(fileHbRule.getFlushBuffer().intValue()*1024*1024);
> table = connection.getBufferedMutator(params);
>
> mutator.add(p);
> if(totalCnts % 5000 == 0 ) {
> table.mutate(mutator);
> mutator.clear();
> }{color}
> ---
> The file to put is a text format file: 2 million rows comma-separated text
> file, each row records 110 columns, total size is about 1G. In addition to
> the main parameter configuration such as heap memory, I kept the default
> parameter values ??for most of the hbase services.
> The load program is designed for single thread.
> The following is the progress information :
> --- Hbase1.2.0 ( CDH5.13.3 )
>
> 2020-12-01 16:48:18 inserted: 10
> 2020-12-01 16:48:36 inserted: 20
> 2020-12-01 16:48:52 inserted: 30
> 2020-12-01 16:49:08 inserted: 40
> 2020-12-01 16:49:23 inserted: 50
> 2020-12-01 16:49:39 inserted: 60
> 2020-12-01 16:49:56 inserted: 70
> 2020-12-01 16:50:12 inserted: 80
> 2020-12-01 16:50:29 inserted: 90
> 2020-12-01 16:50:45 inserted: 100
> 2020-12-01 16:51:01 inserted: 110
> 2020-12-01 16:51:17 inserted: 120
> 2020-12-01 16:51:34 inserted: 130
> 2020-12-01 16:51:49 inserted: 140
> 2020-12-01 16:52:05 inserted: 150
> 2020-12-01 16:52:21 inserted: 160
> 2020-12-01 16:52:40 inserted: 170
> 2020-12-01 16:52:57 inserted: 180
> 2020-12-01 16:53:19 inserted: 190
> 2020-12-01 16:53:42 inserted: 200
> 2020-12-01 16:53:48 inserted: 200
> imp finished ok!
> --job finished--
> ---Hbase.2.0.2 (
> HDP3.1.1)-
> 2020-12-01 17:25:24 inserted: 10
> 2020-12-01 17:26:03 inserted: 20
> 2020-12-01 17:26:39 inserted: 30
> 2020-12-01 17:27:13 inserted: 40
> 2020-12-01 17:27:47 inserted: 50
> 2020-12-01 17:28:23 inserted: 60
> 2020-12-01 17:29:03 inserted: 70
> 2020-12-01 17:29:40 inserted: 80
> 2020-12-01 17:30:15 inserted: 90
> 2020-12-01 17:30:51 inserted: 100
> 2020-12-01 17:31:27 inserted: 110
> 2020-12-01 17:32:03 inserted: 120
> 2020-12-01 17:32:39 inserted: 130
> 2020-12-01 17:33:14 inserted: 140
> 2020-12-01 17:33:50 inserted: 150
> 2020-12-01 17:34:25 inserted: 160
> 2020-12-01 17:35:01 inserted: 170
> 2020-12-01 17:35:38 inserted: 180
> 2020-12-01 17:36:14 inserted: 190
> 2020-12-01 17:36:51 inserted: 200
> 2020-12-01 17:36:55 inserted: 200
> imp finished ok!
> --job finished--
> returnCode=0
> In addition, we also did some benchmark tests on the production cluster.The
> delay is seem to be a bit high. The detailed report is in the attachment.
> Are there any key points that I have not done configuration? or，, this
> version has performance defects ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25346) hbase2.x the performance is lower than hbase 1.x ？
[
https://issues.apache.org/jira/browse/HBASE-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247590#comment-17247590
]
nilonealex commented on HBASE-25346:
This is PE test report on Hbase1.2( with default WAL setting)
[^test_for_randomWrite_hbase1.2.1.log]
in my code ,I have tried to change WAL setting
,(SKIP_WAL,SYNC_WAL,ASYNC_WAL），the Performance differences still exist
> hbase2.x the performance is lower than hbase 1.x ？
> ---
>
> Key: HBASE-25346
> URL: https://issues.apache.org/jira/browse/HBASE-25346
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.2
>Reporter: nilonealex
>Priority: Critical
> Attachments: hbase-pe-performace-test.log, hbase-site.xml,
> test_for_randomWrite.log, test_for_randomWrite_hbase1.2.1.log
>
>
> Recently we found that the newly built production hbase cluster is running a
> bit slow , the hadoop version is Hbase2.0.2 ( HDP3.1.1) and it has 100
> nodes.Then we begin to do load & query performance verification between
> Hbase2.0.2 ( HDP3.1.1) & Hbase1.2.0 ( CDH5.13.3 ) test environment （4nodes）,
> found that : put data based on hbase2.0 is much slower than hbase1.x (the
> former is almost half of the latter), I use BufferedMutator and
> BufferedMutatorParams term for batch put to improve efficiency. More
> confusing is the performance of the production environment is worse than my
> test environment
> Some of the codes are as follows:
> ---
> {color:#4C9AFF}List mutator = new ArrayList<>();
> BufferedMutator table = null;
> BufferedMutatorParams params = new
> BufferedMutatorParams(TableName.valueOf(fileHbRule.getHbaseTableName()));
> params.writeBufferSize(fileHbRule.getFlushBuffer().intValue()*1024*1024);
> table = connection.getBufferedMutator(params);
>
> mutator.add(p);
> if(totalCnts % 5000 == 0 ) {
> table.mutate(mutator);
> mutator.clear();
> }{color}
> ---
> The file to put is a text format file: 2 million rows comma-separated text
> file, each row records 110 columns, total size is about 1G. In addition to
> the main parameter configuration such as heap memory, I kept the default
> parameter values ??for most of the hbase services.
> The load program is designed for single thread.
> The following is the progress information :
> --- Hbase1.2.0 ( CDH5.13.3 )
>
> 2020-12-01 16:48:18 inserted: 10
> 2020-12-01 16:48:36 inserted: 20
> 2020-12-01 16:48:52 inserted: 30
> 2020-12-01 16:49:08 inserted: 40
> 2020-12-01 16:49:23 inserted: 50
> 2020-12-01 16:49:39 inserted: 60
> 2020-12-01 16:49:56 inserted: 70
> 2020-12-01 16:50:12 inserted: 80
> 2020-12-01 16:50:29 inserted: 90
> 2020-12-01 16:50:45 inserted: 100
> 2020-12-01 16:51:01 inserted: 110
> 2020-12-01 16:51:17 inserted: 120
> 2020-12-01 16:51:34 inserted: 130
> 2020-12-01 16:51:49 inserted: 140
> 2020-12-01 16:52:05 inserted: 150
> 2020-12-01 16:52:21 inserted: 160
> 2020-12-01 16:52:40 inserted: 170
> 2020-12-01 16:52:57 inserted: 180
> 2020-12-01 16:53:19 inserted: 190
> 2020-12-01 16:53:42 inserted: 200
> 2020-12-01 16:53:48 inserted: 200
> imp finished ok!
> --job finished--
> ---Hbase.2.0.2 (
> HDP3.1.1)-
> 2020-12-01 17:25:24 inserted: 10
> 2020-12-01 17:26:03 inserted: 20
> 2020-12-01 17:26:39 inserted: 30
> 2020-12-01 17:27:13 inserted: 40
> 2020-12-01 17:27:47 inserted: 50
> 2020-12-01 17:28:23 inserted: 60
> 2020-12-01 17:29:03 inserted: 70
> 2020-12-01 17:29:40 inserted: 80
> 2020-12-01 17:30:15 inserted: 90
> 2020-12-01 17:30:51 inserted: 100
> 2020-12-01 17:31:27 inserted: 110
> 2020-12-01 17:32:03 inserted: 120
> 2020-12-01 17:32:39 inserted: 130
> 2020-12-01 17:33:14 inserted: 140
> 2020-12-01 17:33:50 inserted: 150
> 2020-12-01 17:34:25 inserted: 160
> 2020-12-01 17:35:01 inserted: 170
> 2020-12-01 17:35:38 inserted: 180
> 2020-12-01 17:36:14 inserted: 190
> 2020-12-01 17:36:51 inserted: 200
> 2020-12-01 17:36:55 inserted: 200
> imp finished ok!
> --job finished--
> returnCode=0
> In addition, we also did some benchmark tests on the production cluster.The
> delay is seem to be a bit high. The detailed report is in the attachment.
> Are there any key points that I have not done configuration? or，, this
> version has performance defects ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25346) hbase2.x the performance is lower than hbase 1.x ？
[
https://issues.apache.org/jira/browse/HBASE-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247588#comment-17247588
]
nilonealex commented on HBASE-25346:
{color:#DE350B}Very very strange, I am running on the higher version of 2.1.2
on Hbase, and the performance is much worse than that on 2.0.2. I would rather
believe that my code is not written properly{color}
> hbase2.x the performance is lower than hbase 1.x ？
> ---
>
> Key: HBASE-25346
> URL: https://issues.apache.org/jira/browse/HBASE-25346
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.2
>Reporter: nilonealex
>Priority: Critical
> Attachments: hbase-pe-performace-test.log, hbase-site.xml,
> test_for_randomWrite.log
>
>
> Recently we found that the newly built production hbase cluster is running a
> bit slow , the hadoop version is Hbase2.0.2 ( HDP3.1.1) and it has 100
> nodes.Then we begin to do load & query performance verification between
> Hbase2.0.2 ( HDP3.1.1) & Hbase1.2.0 ( CDH5.13.3 ) test environment （4nodes）,
> found that : put data based on hbase2.0 is much slower than hbase1.x (the
> former is almost half of the latter), I use BufferedMutator and
> BufferedMutatorParams term for batch put to improve efficiency. More
> confusing is the performance of the production environment is worse than my
> test environment
> Some of the codes are as follows:
> ---
> {color:#4C9AFF}List mutator = new ArrayList<>();
> BufferedMutator table = null;
> BufferedMutatorParams params = new
> BufferedMutatorParams(TableName.valueOf(fileHbRule.getHbaseTableName()));
> params.writeBufferSize(fileHbRule.getFlushBuffer().intValue()*1024*1024);
> table = connection.getBufferedMutator(params);
>
> mutator.add(p);
> if(totalCnts % 5000 == 0 ) {
> table.mutate(mutator);
> mutator.clear();
> }{color}
> ---
> The file to put is a text format file: 2 million rows comma-separated text
> file, each row records 110 columns, total size is about 1G. In addition to
> the main parameter configuration such as heap memory, I kept the default
> parameter values ??for most of the hbase services.
> The load program is designed for single thread.
> The following is the progress information :
> --- Hbase1.2.0 ( CDH5.13.3 )
>
> 2020-12-01 16:48:18 inserted: 10
> 2020-12-01 16:48:36 inserted: 20
> 2020-12-01 16:48:52 inserted: 30
> 2020-12-01 16:49:08 inserted: 40
> 2020-12-01 16:49:23 inserted: 50
> 2020-12-01 16:49:39 inserted: 60
> 2020-12-01 16:49:56 inserted: 70
> 2020-12-01 16:50:12 inserted: 80
> 2020-12-01 16:50:29 inserted: 90
> 2020-12-01 16:50:45 inserted: 100
> 2020-12-01 16:51:01 inserted: 110
> 2020-12-01 16:51:17 inserted: 120
> 2020-12-01 16:51:34 inserted: 130
> 2020-12-01 16:51:49 inserted: 140
> 2020-12-01 16:52:05 inserted: 150
> 2020-12-01 16:52:21 inserted: 160
> 2020-12-01 16:52:40 inserted: 170
> 2020-12-01 16:52:57 inserted: 180
> 2020-12-01 16:53:19 inserted: 190
> 2020-12-01 16:53:42 inserted: 200
> 2020-12-01 16:53:48 inserted: 200
> imp finished ok!
> --job finished--
> ---Hbase.2.0.2 (
> HDP3.1.1)-
> 2020-12-01 17:25:24 inserted: 10
> 2020-12-01 17:26:03 inserted: 20
> 2020-12-01 17:26:39 inserted: 30
> 2020-12-01 17:27:13 inserted: 40
> 2020-12-01 17:27:47 inserted: 50
> 2020-12-01 17:28:23 inserted: 60
> 2020-12-01 17:29:03 inserted: 70
> 2020-12-01 17:29:40 inserted: 80
> 2020-12-01 17:30:15 inserted: 90
> 2020-12-01 17:30:51 inserted: 100
> 2020-12-01 17:31:27 inserted: 110
> 2020-12-01 17:32:03 inserted: 120
> 2020-12-01 17:32:39 inserted: 130
> 2020-12-01 17:33:14 inserted: 140
> 2020-12-01 17:33:50 inserted: 150
> 2020-12-01 17:34:25 inserted: 160
> 2020-12-01 17:35:01 inserted: 170
> 2020-12-01 17:35:38 inserted: 180
> 2020-12-01 17:36:14 inserted: 190
> 2020-12-01 17:36:51 inserted: 200
> 2020-12-01 17:36:55 inserted: 200
> imp finished ok!
> --job finished--
> returnCode=0
> In addition, we also did some benchmark tests on the production cluster.The
> delay is seem to be a bit high. The detailed report is in the attachment.
> Are there any key points that I have not done configuration? or，, this
> version has performance defects ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25346) hbase2.x the performance is lower than hbase 1.x ？
[
https://issues.apache.org/jira/browse/HBASE-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247586#comment-17247586
]
nilonealex commented on HBASE-25346:
The logic I used to write data is similar to the following：
public void testPutList() throws Exception {
TableName tableName = TableName.valueOf("baizhi:t_user");
BufferedMutator bufferedMutator = conn.getBufferedMutator(tableName);
Put put1 = new Put("4".getBytes());
put1.addColumn("cf1".getBytes(), "name".getBytes(),
"zhangsan".getBytes());
put1.addColumn("cf1".getBytes(), "age".getBytes(), "18".getBytes());
put1.addColumn("cf1".getBytes(), "sex".getBytes(), "false".getBytes());
Put put2 = new Put("5".getBytes());
put2.addColumn("cf1".getBytes(), "name".getBytes(),
"zhangsan".getBytes());
put2.addColumn("cf1".getBytes(), "age".getBytes(), "18".getBytes());
put2.addColumn("cf1".getBytes(), "sex".getBytes(), "false".getBytes());
Put put3 = new Put("6".getBytes());
put3.addColumn("cf1".getBytes(), "name".getBytes(),
"zhangsan".getBytes());
put3.addColumn("cf1".getBytes(), "age".getBytes(), "18".getBytes());
put3.addColumn("cf1".getBytes(), "sex".getBytes(), "false".getBytes());
ArrayList puts = new ArrayList();
puts.add(put1);
puts.add(put2);
puts.add(put3);
bufferedMutator.mutate(puts);
bufferedMutator.close();
}
> hbase2.x the performance is lower than hbase 1.x ？
> ---
>
> Key: HBASE-25346
> URL: https://issues.apache.org/jira/browse/HBASE-25346
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.2
>Reporter: nilonealex
>Priority: Critical
> Attachments: hbase-pe-performace-test.log, hbase-site.xml,
> test_for_randomWrite.log
>
>
> Recently we found that the newly built production hbase cluster is running a
> bit slow , the hadoop version is Hbase2.0.2 ( HDP3.1.1) and it has 100
> nodes.Then we begin to do load & query performance verification between
> Hbase2.0.2 ( HDP3.1.1) & Hbase1.2.0 ( CDH5.13.3 ) test environment （4nodes）,
> found that : put data based on hbase2.0 is much slower than hbase1.x (the
> former is almost half of the latter), I use BufferedMutator and
> BufferedMutatorParams term for batch put to improve efficiency. More
> confusing is the performance of the production environment is worse than my
> test environment
> Some of the codes are as follows:
> ---
> {color:#4C9AFF}List mutator = new ArrayList<>();
> BufferedMutator table = null;
> BufferedMutatorParams params = new
> BufferedMutatorParams(TableName.valueOf(fileHbRule.getHbaseTableName()));
> params.writeBufferSize(fileHbRule.getFlushBuffer().intValue()*1024*1024);
> table = connection.getBufferedMutator(params);
>
> mutator.add(p);
> if(totalCnts % 5000 == 0 ) {
> table.mutate(mutator);
> mutator.clear();
> }{color}
> ---
> The file to put is a text format file: 2 million rows comma-separated text
> file, each row records 110 columns, total size is about 1G. In addition to
> the main parameter configuration such as heap memory, I kept the default
> parameter values ??for most of the hbase services.
> The load program is designed for single thread.
> The following is the progress information :
> --- Hbase1.2.0 ( CDH5.13.3 )
>
> 2020-12-01 16:48:18 inserted: 10
> 2020-12-01 16:48:36 inserted: 20
> 2020-12-01 16:48:52 inserted: 30
> 2020-12-01 16:49:08 inserted: 40
> 2020-12-01 16:49:23 inserted: 50
> 2020-12-01 16:49:39 inserted: 60
> 2020-12-01 16:49:56 inserted: 70
> 2020-12-01 16:50:12 inserted: 80
> 2020-12-01 16:50:29 inserted: 90
> 2020-12-01 16:50:45 inserted: 100
> 2020-12-01 16:51:01 inserted: 110
> 2020-12-01 16:51:17 inserted: 120
> 2020-12-01 16:51:34 inserted: 130
> 2020-12-01 16:51:49 inserted: 140
> 2020-12-01 16:52:05 inserted: 150
> 2020-12-01 16:52:21 inserted: 160
> 2020-12-01 16:52:40 inserted: 170
> 2020-12-01 16:52:57 inserted: 180
> 2020-12-01 16:53:19 inserted: 190
> 2020-12-01 16:53:42 inserted: 200
> 2020-12-01 16:53:48 inserted: 200
> imp finished ok!
> --job finished--
> ---Hbase.2.0.2 (
> HDP3.1.1)-
> 2020-12-01 17:25:24 inserted: 10
> 2020-12-01 17:26:03 inserted: 20
> 2020-12-01 17:26:39 inserted: 30
> 2020-12-01 17:27:13 inserted: 40
> 2020-12-01 17:27:47 inserted: 50
> 2020-12-01 17:28:23 inserted:
[jira] [Commented] (HBASE-25346) hbase2.x the performance is lower than hbase 1.x ？
[
https://issues.apache.org/jira/browse/HBASE-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247133#comment-17247133
]
ramkrishna.s.vasudevan commented on HBASE-25346:
Here it is the random write report with 2.0.
What is the value u see with 1.x based hbase? The WAL system is AsyncFSWAL in
2.x and FileSystem in 1.x. BTW how many nodes are you testing?
In HBASE-24850 we have seen issues with the CellComparator performance when we
add more columns per row. The addition to memstore takes more time due to the
comparisons. Can you see if you can try that patch there to see if it improves
your write performance? I have a PR raised against branch-2.3.
> hbase2.x the performance is lower than hbase 1.x ？
> ---
>
> Key: HBASE-25346
> URL: https://issues.apache.org/jira/browse/HBASE-25346
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.2
>Reporter: nilonealex
>Priority: Critical
> Attachments: hbase-pe-performace-test.log, hbase-site.xml,
> test_for_randomWrite.log
>
>
> Recently we found that the newly built production hbase cluster is running a
> bit slow , the hadoop version is Hbase2.0.2 ( HDP3.1.1) and it has 100
> nodes.Then we begin to do load & query performance verification between
> Hbase2.0.2 ( HDP3.1.1) & Hbase1.2.0 ( CDH5.13.3 ) test environment （4nodes）,
> found that : put data based on hbase2.0 is much slower than hbase1.x (the
> former is almost half of the latter), I use BufferedMutator and
> BufferedMutatorParams term for batch put to improve efficiency. More
> confusing is the performance of the production environment is worse than my
> test environment
> Some of the codes are as follows:
> ---
> {color:#4C9AFF}List mutator = new ArrayList<>();
> BufferedMutator table = null;
> BufferedMutatorParams params = new
> BufferedMutatorParams(TableName.valueOf(fileHbRule.getHbaseTableName()));
> params.writeBufferSize(fileHbRule.getFlushBuffer().intValue()*1024*1024);
> table = connection.getBufferedMutator(params);
>
> mutator.add(p);
> if(totalCnts % 5000 == 0 ) {
> table.mutate(mutator);
> mutator.clear();
> }{color}
> ---
> The file to put is a text format file: 2 million rows comma-separated text
> file, each row records 110 columns, total size is about 1G. In addition to
> the main parameter configuration such as heap memory, I kept the default
> parameter values ??for most of the hbase services.
> The load program is designed for single thread.
> The following is the progress information :
> --- Hbase1.2.0 ( CDH5.13.3 )
>
> 2020-12-01 16:48:18 inserted: 10
> 2020-12-01 16:48:36 inserted: 20
> 2020-12-01 16:48:52 inserted: 30
> 2020-12-01 16:49:08 inserted: 40
> 2020-12-01 16:49:23 inserted: 50
> 2020-12-01 16:49:39 inserted: 60
> 2020-12-01 16:49:56 inserted: 70
> 2020-12-01 16:50:12 inserted: 80
> 2020-12-01 16:50:29 inserted: 90
> 2020-12-01 16:50:45 inserted: 100
> 2020-12-01 16:51:01 inserted: 110
> 2020-12-01 16:51:17 inserted: 120
> 2020-12-01 16:51:34 inserted: 130
> 2020-12-01 16:51:49 inserted: 140
> 2020-12-01 16:52:05 inserted: 150
> 2020-12-01 16:52:21 inserted: 160
> 2020-12-01 16:52:40 inserted: 170
> 2020-12-01 16:52:57 inserted: 180
> 2020-12-01 16:53:19 inserted: 190
> 2020-12-01 16:53:42 inserted: 200
> 2020-12-01 16:53:48 inserted: 200
> imp finished ok!
> --job finished--
> ---Hbase.2.0.2 (
> HDP3.1.1)-
> 2020-12-01 17:25:24 inserted: 10
> 2020-12-01 17:26:03 inserted: 20
> 2020-12-01 17:26:39 inserted: 30
> 2020-12-01 17:27:13 inserted: 40
> 2020-12-01 17:27:47 inserted: 50
> 2020-12-01 17:28:23 inserted: 60
> 2020-12-01 17:29:03 inserted: 70
> 2020-12-01 17:29:40 inserted: 80
> 2020-12-01 17:30:15 inserted: 90
> 2020-12-01 17:30:51 inserted: 100
> 2020-12-01 17:31:27 inserted: 110
> 2020-12-01 17:32:03 inserted: 120
> 2020-12-01 17:32:39 inserted: 130
> 2020-12-01 17:33:14 inserted: 140
> 2020-12-01 17:33:50 inserted: 150
> 2020-12-01 17:34:25 inserted: 160
> 2020-12-01 17:35:01 inserted: 170
> 2020-12-01 17:35:38 inserted: 180
> 2020-12-01 17:36:14 inserted: 190
> 2020-12-01 17:36:51 inserted: 200
> 2020-12-01 17:36:55 inserted: 200
> imp finished ok!
> --job finished--
> returnCode=0
> In addition, we also did some benchmark tests on the production cluster.The
> delay is seem to be a bit
[jira] [Commented] (HBASE-25346) hbase2.x the performance is lower than hbase 1.x ？
[
https://issues.apache.org/jira/browse/HBASE-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246984#comment-17246984
]
nilonealex commented on HBASE-25346:
[^test_for_randomWrite.log]
> hbase2.x the performance is lower than hbase 1.x ？
> ---
>
> Key: HBASE-25346
> URL: https://issues.apache.org/jira/browse/HBASE-25346
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.2
>Reporter: nilonealex
>Priority: Critical
> Attachments: hbase-pe-performace-test.log, hbase-site.xml,
> test_for_randomWrite.log
>
>
> Recently we found that the newly built production hbase cluster is running a
> bit slow , the hadoop version is Hbase2.0.2 ( HDP3.1.1) and it has 100
> nodes.Then we begin to do load & query performance verification between
> Hbase2.0.2 ( HDP3.1.1) & Hbase1.2.0 ( CDH5.13.3 ) test environment （4nodes）,
> found that : put data based on hbase2.0 is much slower than hbase1.x (the
> former is almost half of the latter), I use BufferedMutator and
> BufferedMutatorParams term for batch put to improve efficiency. More
> confusing is the performance of the production environment is worse than my
> test environment
> Some of the codes are as follows:
> ---
> {color:#4C9AFF}List mutator = new ArrayList<>();
> BufferedMutator table = null;
> BufferedMutatorParams params = new
> BufferedMutatorParams(TableName.valueOf(fileHbRule.getHbaseTableName()));
> params.writeBufferSize(fileHbRule.getFlushBuffer().intValue()*1024*1024);
> table = connection.getBufferedMutator(params);
>
> mutator.add(p);
> if(totalCnts % 5000 == 0 ) {
> table.mutate(mutator);
> mutator.clear();
> }{color}
> ---
> The file to put is a text format file: 2 million rows comma-separated text
> file, each row records 110 columns, total size is about 1G. In addition to
> the main parameter configuration such as heap memory, I kept the default
> parameter values ??for most of the hbase services.
> The load program is designed for single thread.
> The following is the progress information :
> --- Hbase1.2.0 ( CDH5.13.3 )
>
> 2020-12-01 16:48:18 inserted: 10
> 2020-12-01 16:48:36 inserted: 20
> 2020-12-01 16:48:52 inserted: 30
> 2020-12-01 16:49:08 inserted: 40
> 2020-12-01 16:49:23 inserted: 50
> 2020-12-01 16:49:39 inserted: 60
> 2020-12-01 16:49:56 inserted: 70
> 2020-12-01 16:50:12 inserted: 80
> 2020-12-01 16:50:29 inserted: 90
> 2020-12-01 16:50:45 inserted: 100
> 2020-12-01 16:51:01 inserted: 110
> 2020-12-01 16:51:17 inserted: 120
> 2020-12-01 16:51:34 inserted: 130
> 2020-12-01 16:51:49 inserted: 140
> 2020-12-01 16:52:05 inserted: 150
> 2020-12-01 16:52:21 inserted: 160
> 2020-12-01 16:52:40 inserted: 170
> 2020-12-01 16:52:57 inserted: 180
> 2020-12-01 16:53:19 inserted: 190
> 2020-12-01 16:53:42 inserted: 200
> 2020-12-01 16:53:48 inserted: 200
> imp finished ok!
> --job finished--
> ---Hbase.2.0.2 (
> HDP3.1.1)-
> 2020-12-01 17:25:24 inserted: 10
> 2020-12-01 17:26:03 inserted: 20
> 2020-12-01 17:26:39 inserted: 30
> 2020-12-01 17:27:13 inserted: 40
> 2020-12-01 17:27:47 inserted: 50
> 2020-12-01 17:28:23 inserted: 60
> 2020-12-01 17:29:03 inserted: 70
> 2020-12-01 17:29:40 inserted: 80
> 2020-12-01 17:30:15 inserted: 90
> 2020-12-01 17:30:51 inserted: 100
> 2020-12-01 17:31:27 inserted: 110
> 2020-12-01 17:32:03 inserted: 120
> 2020-12-01 17:32:39 inserted: 130
> 2020-12-01 17:33:14 inserted: 140
> 2020-12-01 17:33:50 inserted: 150
> 2020-12-01 17:34:25 inserted: 160
> 2020-12-01 17:35:01 inserted: 170
> 2020-12-01 17:35:38 inserted: 180
> 2020-12-01 17:36:14 inserted: 190
> 2020-12-01 17:36:51 inserted: 200
> 2020-12-01 17:36:55 inserted: 200
> imp finished ok!
> --job finished--
> returnCode=0
> In addition, we also did some benchmark tests on the production cluster.The
> delay is seem to be a bit high. The detailed report is in the attachment.
> Are there any key points that I have not done configuration? or，, this
> version has performance defects ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25346) hbase2.x the performance is lower than hbase 1.x ？
[
https://issues.apache.org/jira/browse/HBASE-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245779#comment-17245779
]
nilonealex commented on HBASE-25346:
the test is RandomWrite
> hbase2.x the performance is lower than hbase 1.x ？
> ---
>
> Key: HBASE-25346
> URL: https://issues.apache.org/jira/browse/HBASE-25346
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.2
>Reporter: nilonealex
>Priority: Critical
> Attachments: hbase-pe-performace-test.log, hbase-site.xml
>
>
> Recently we found that the newly built production hbase cluster is running a
> bit slow , the hadoop version is Hbase2.0.2 ( HDP3.1.1) and it has 100
> nodes.Then we begin to do load & query performance verification between
> Hbase2.0.2 ( HDP3.1.1) & Hbase1.2.0 ( CDH5.13.3 ) test environment （4nodes）,
> found that : put data based on hbase2.0 is much slower than hbase1.x (the
> former is almost half of the latter), I use BufferedMutator and
> BufferedMutatorParams term for batch put to improve efficiency. More
> confusing is the performance of the production environment is worse than my
> test environment
> Some of the codes are as follows:
> ---
> {color:#4C9AFF}List mutator = new ArrayList<>();
> BufferedMutator table = null;
> BufferedMutatorParams params = new
> BufferedMutatorParams(TableName.valueOf(fileHbRule.getHbaseTableName()));
> params.writeBufferSize(fileHbRule.getFlushBuffer().intValue()*1024*1024);
> table = connection.getBufferedMutator(params);
>
> mutator.add(p);
> if(totalCnts % 5000 == 0 ) {
> table.mutate(mutator);
> mutator.clear();
> }{color}
> ---
> The file to put is a text format file: 2 million rows comma-separated text
> file, each row records 110 columns, total size is about 1G. In addition to
> the main parameter configuration such as heap memory, I kept the default
> parameter values ??for most of the hbase services.
> The load program is designed for single thread.
> The following is the progress information :
> --- Hbase1.2.0 ( CDH5.13.3 )
>
> 2020-12-01 16:48:18 inserted: 10
> 2020-12-01 16:48:36 inserted: 20
> 2020-12-01 16:48:52 inserted: 30
> 2020-12-01 16:49:08 inserted: 40
> 2020-12-01 16:49:23 inserted: 50
> 2020-12-01 16:49:39 inserted: 60
> 2020-12-01 16:49:56 inserted: 70
> 2020-12-01 16:50:12 inserted: 80
> 2020-12-01 16:50:29 inserted: 90
> 2020-12-01 16:50:45 inserted: 100
> 2020-12-01 16:51:01 inserted: 110
> 2020-12-01 16:51:17 inserted: 120
> 2020-12-01 16:51:34 inserted: 130
> 2020-12-01 16:51:49 inserted: 140
> 2020-12-01 16:52:05 inserted: 150
> 2020-12-01 16:52:21 inserted: 160
> 2020-12-01 16:52:40 inserted: 170
> 2020-12-01 16:52:57 inserted: 180
> 2020-12-01 16:53:19 inserted: 190
> 2020-12-01 16:53:42 inserted: 200
> 2020-12-01 16:53:48 inserted: 200
> imp finished ok!
> --job finished--
> ---Hbase.2.0.2 (
> HDP3.1.1)-
> 2020-12-01 17:25:24 inserted: 10
> 2020-12-01 17:26:03 inserted: 20
> 2020-12-01 17:26:39 inserted: 30
> 2020-12-01 17:27:13 inserted: 40
> 2020-12-01 17:27:47 inserted: 50
> 2020-12-01 17:28:23 inserted: 60
> 2020-12-01 17:29:03 inserted: 70
> 2020-12-01 17:29:40 inserted: 80
> 2020-12-01 17:30:15 inserted: 90
> 2020-12-01 17:30:51 inserted: 100
> 2020-12-01 17:31:27 inserted: 110
> 2020-12-01 17:32:03 inserted: 120
> 2020-12-01 17:32:39 inserted: 130
> 2020-12-01 17:33:14 inserted: 140
> 2020-12-01 17:33:50 inserted: 150
> 2020-12-01 17:34:25 inserted: 160
> 2020-12-01 17:35:01 inserted: 170
> 2020-12-01 17:35:38 inserted: 180
> 2020-12-01 17:36:14 inserted: 190
> 2020-12-01 17:36:51 inserted: 200
> 2020-12-01 17:36:55 inserted: 200
> imp finished ok!
> --job finished--
> returnCode=0
> In addition, we also did some benchmark tests on the production cluster.The
> delay is seem to be a bit high. The detailed report is in the attachment.
> Are there any key points that I have not done configuration? or，, this
> version has performance defects ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25346) hbase2.x the performance is lower than hbase 1.x ？
[
https://issues.apache.org/jira/browse/HBASE-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245725#comment-17245725
]
ramkrishna.s.vasudevan commented on HBASE-25346:
[~nilone2]
>From the report you have attached am not sure whether it is the writes that is
>slower or reads also?
Is it possible to attach the TPS/latency numbers with writes and reads?
> hbase2.x the performance is lower than hbase 1.x ？
> ---
>
> Key: HBASE-25346
> URL: https://issues.apache.org/jira/browse/HBASE-25346
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.2
>Reporter: nilonealex
>Priority: Critical
> Attachments: hbase-pe-performace-test.log, hbase-site.xml
>
>
> Recently we found that the newly built production hbase cluster is running a
> bit slow , the hadoop version is Hbase2.0.2 ( HDP3.1.1) and it has 100
> nodes.Then we begin to do load & query performance verification between
> Hbase2.0.2 ( HDP3.1.1) & Hbase1.2.0 ( CDH5.13.3 ) test environment （4nodes）,
> found that : put data based on hbase2.0 is much slower than hbase1.x (the
> former is almost half of the latter), I use BufferedMutator and
> BufferedMutatorParams term for batch put to improve efficiency. More
> confusing is the performance of the production environment is worse than my
> test environment
> Some of the codes are as follows:
> ---
> {color:#4C9AFF}List mutator = new ArrayList<>();
> BufferedMutator table = null;
> BufferedMutatorParams params = new
> BufferedMutatorParams(TableName.valueOf(fileHbRule.getHbaseTableName()));
> params.writeBufferSize(fileHbRule.getFlushBuffer().intValue()*1024*1024);
> table = connection.getBufferedMutator(params);
>
> mutator.add(p);
> if(totalCnts % 5000 == 0 ) {
> table.mutate(mutator);
> mutator.clear();
> }{color}
> ---
> The file to put is a text format file: 2 million rows comma-separated text
> file, each row records 110 columns, total size is about 1G. In addition to
> the main parameter configuration such as heap memory, I kept the default
> parameter values ??for most of the hbase services.
> The load program is designed for single thread.
> The following is the progress information :
> --- Hbase1.2.0 ( CDH5.13.3 )
>
> 2020-12-01 16:48:18 inserted: 10
> 2020-12-01 16:48:36 inserted: 20
> 2020-12-01 16:48:52 inserted: 30
> 2020-12-01 16:49:08 inserted: 40
> 2020-12-01 16:49:23 inserted: 50
> 2020-12-01 16:49:39 inserted: 60
> 2020-12-01 16:49:56 inserted: 70
> 2020-12-01 16:50:12 inserted: 80
> 2020-12-01 16:50:29 inserted: 90
> 2020-12-01 16:50:45 inserted: 100
> 2020-12-01 16:51:01 inserted: 110
> 2020-12-01 16:51:17 inserted: 120
> 2020-12-01 16:51:34 inserted: 130
> 2020-12-01 16:51:49 inserted: 140
> 2020-12-01 16:52:05 inserted: 150
> 2020-12-01 16:52:21 inserted: 160
> 2020-12-01 16:52:40 inserted: 170
> 2020-12-01 16:52:57 inserted: 180
> 2020-12-01 16:53:19 inserted: 190
> 2020-12-01 16:53:42 inserted: 200
> 2020-12-01 16:53:48 inserted: 200
> imp finished ok!
> --job finished--
> ---Hbase.2.0.2 (
> HDP3.1.1)-
> 2020-12-01 17:25:24 inserted: 10
> 2020-12-01 17:26:03 inserted: 20
> 2020-12-01 17:26:39 inserted: 30
> 2020-12-01 17:27:13 inserted: 40
> 2020-12-01 17:27:47 inserted: 50
> 2020-12-01 17:28:23 inserted: 60
> 2020-12-01 17:29:03 inserted: 70
> 2020-12-01 17:29:40 inserted: 80
> 2020-12-01 17:30:15 inserted: 90
> 2020-12-01 17:30:51 inserted: 100
> 2020-12-01 17:31:27 inserted: 110
> 2020-12-01 17:32:03 inserted: 120
> 2020-12-01 17:32:39 inserted: 130
> 2020-12-01 17:33:14 inserted: 140
> 2020-12-01 17:33:50 inserted: 150
> 2020-12-01 17:34:25 inserted: 160
> 2020-12-01 17:35:01 inserted: 170
> 2020-12-01 17:35:38 inserted: 180
> 2020-12-01 17:36:14 inserted: 190
> 2020-12-01 17:36:51 inserted: 200
> 2020-12-01 17:36:55 inserted: 200
> imp finished ok!
> --job finished--
> returnCode=0
> In addition, we also did some benchmark tests on the production cluster.The
> delay is seem to be a bit high. The detailed report is in the attachment.
> Are there any key points that I have not done configuration? or，, this
> version has performance defects ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25346) hbase2.x the performance is lower than hbase 1.x ？
[
https://issues.apache.org/jira/browse/HBASE-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245713#comment-17245713
]
nilonealex commented on HBASE-25346:
yes ,the same
> hbase2.x the performance is lower than hbase 1.x ？
> ---
>
> Key: HBASE-25346
> URL: https://issues.apache.org/jira/browse/HBASE-25346
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.2
>Reporter: nilonealex
>Priority: Critical
> Attachments: hbase-pe-performace-test.log, hbase-site.xml
>
>
> Recently we found that the newly built production hbase cluster is running a
> bit slow , the hadoop version is Hbase2.0.2 ( HDP3.1.1) and it has 100
> nodes.Then we begin to do load & query performance verification between
> Hbase2.0.2 ( HDP3.1.1) & Hbase1.2.0 ( CDH5.13.3 ) test environment （4nodes）,
> found that : put data based on hbase2.0 is much slower than hbase1.x (the
> former is almost half of the latter), I use BufferedMutator and
> BufferedMutatorParams term for batch put to improve efficiency. More
> confusing is the performance of the production environment is worse than my
> test environment
> Some of the codes are as follows:
> ---
> {color:#4C9AFF}List mutator = new ArrayList<>();
> BufferedMutator table = null;
> BufferedMutatorParams params = new
> BufferedMutatorParams(TableName.valueOf(fileHbRule.getHbaseTableName()));
> params.writeBufferSize(fileHbRule.getFlushBuffer().intValue()*1024*1024);
> table = connection.getBufferedMutator(params);
>
> mutator.add(p);
> if(totalCnts % 5000 == 0 ) {
> table.mutate(mutator);
> mutator.clear();
> }{color}
> ---
> The file to put is a text format file: 2 million rows comma-separated text
> file, each row records 110 columns, total size is about 1G. In addition to
> the main parameter configuration such as heap memory, I kept the default
> parameter values ??for most of the hbase services.
> The load program is designed for single thread.
> The following is the progress information :
> --- Hbase1.2.0 ( CDH5.13.3 )
>
> 2020-12-01 16:48:18 inserted: 10
> 2020-12-01 16:48:36 inserted: 20
> 2020-12-01 16:48:52 inserted: 30
> 2020-12-01 16:49:08 inserted: 40
> 2020-12-01 16:49:23 inserted: 50
> 2020-12-01 16:49:39 inserted: 60
> 2020-12-01 16:49:56 inserted: 70
> 2020-12-01 16:50:12 inserted: 80
> 2020-12-01 16:50:29 inserted: 90
> 2020-12-01 16:50:45 inserted: 100
> 2020-12-01 16:51:01 inserted: 110
> 2020-12-01 16:51:17 inserted: 120
> 2020-12-01 16:51:34 inserted: 130
> 2020-12-01 16:51:49 inserted: 140
> 2020-12-01 16:52:05 inserted: 150
> 2020-12-01 16:52:21 inserted: 160
> 2020-12-01 16:52:40 inserted: 170
> 2020-12-01 16:52:57 inserted: 180
> 2020-12-01 16:53:19 inserted: 190
> 2020-12-01 16:53:42 inserted: 200
> 2020-12-01 16:53:48 inserted: 200
> imp finished ok!
> --job finished--
> ---Hbase.2.0.2 (
> HDP3.1.1)-
> 2020-12-01 17:25:24 inserted: 10
> 2020-12-01 17:26:03 inserted: 20
> 2020-12-01 17:26:39 inserted: 30
> 2020-12-01 17:27:13 inserted: 40
> 2020-12-01 17:27:47 inserted: 50
> 2020-12-01 17:28:23 inserted: 60
> 2020-12-01 17:29:03 inserted: 70
> 2020-12-01 17:29:40 inserted: 80
> 2020-12-01 17:30:15 inserted: 90
> 2020-12-01 17:30:51 inserted: 100
> 2020-12-01 17:31:27 inserted: 110
> 2020-12-01 17:32:03 inserted: 120
> 2020-12-01 17:32:39 inserted: 130
> 2020-12-01 17:33:14 inserted: 140
> 2020-12-01 17:33:50 inserted: 150
> 2020-12-01 17:34:25 inserted: 160
> 2020-12-01 17:35:01 inserted: 170
> 2020-12-01 17:35:38 inserted: 180
> 2020-12-01 17:36:14 inserted: 190
> 2020-12-01 17:36:51 inserted: 200
> 2020-12-01 17:36:55 inserted: 200
> imp finished ok!
> --job finished--
> returnCode=0
> In addition, we also did some benchmark tests on the production cluster.The
> delay is seem to be a bit high. The detailed report is in the attachment.
> Are there any key points that I have not done configuration? or，, this
> version has performance defects ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25346) hbase2.x the performance is lower than hbase 1.x ？
[
https://issues.apache.org/jira/browse/HBASE-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241422#comment-17241422
]
ramkrishna.s.vasudevan commented on HBASE-25346:
The WAL sits on HDFS and that is same in both the clusters ?
> hbase2.x the performance is lower than hbase 1.x ？
> ---
>
> Key: HBASE-25346
> URL: https://issues.apache.org/jira/browse/HBASE-25346
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.2
>Reporter: nilonealex
>Priority: Critical
> Attachments: hbase-site.xml
>
>
> Recently we found that the newly built production hbase cluster is running a
> bit slow , the hadoop version is Hbase2.0.2 ( HDP3.1.1) and it has 100
> nodes.Then we begin to do load & query performance verification between
> Hbase2.0.2 ( HDP3.1.1) & Hbase1.2.0 ( CDH5.13.3 ) test environment （4nodes）,
> found that : put data based on hbase2.0 is much slower than hbase1.x (the
> former is almost half of the latter), I use BufferedMutator and
> BufferedMutatorParams term for batch put to improve efficiency. More
> confusing is the performance of the production environment is worse than my
> test environment
> Some of the codes are as follows:
> ---
> {color:#4C9AFF}List mutator = new ArrayList<>();
> BufferedMutator table = null;
> BufferedMutatorParams params = new
> BufferedMutatorParams(TableName.valueOf(fileHbRule.getHbaseTableName()));
> params.writeBufferSize(fileHbRule.getFlushBuffer().intValue()*1024*1024);
> table = connection.getBufferedMutator(params);
>
> mutator.add(p);
> if(totalCnts % 5000 == 0 ) {
> table.mutate(mutator);
> mutator.clear();
> }{color}
> ---
> The file to put is a text format file: 2 million rows comma-separated text
> file, each row records 110 columns, total size is about 1G. In addition to
> the main parameter configuration such as heap memory, I kept the default
> parameter values ??for most of the hbase services.
> The load program is designed for single thread.
> The following is the progress information :
> --- Hbase1.2.0 ( CDH5.13.3 )
>
> 2020-12-01 16:48:18 inserted: 10
> 2020-12-01 16:48:36 inserted: 20
> 2020-12-01 16:48:52 inserted: 30
> 2020-12-01 16:49:08 inserted: 40
> 2020-12-01 16:49:23 inserted: 50
> 2020-12-01 16:49:39 inserted: 60
> 2020-12-01 16:49:56 inserted: 70
> 2020-12-01 16:50:12 inserted: 80
> 2020-12-01 16:50:29 inserted: 90
> 2020-12-01 16:50:45 inserted: 100
> 2020-12-01 16:51:01 inserted: 110
> 2020-12-01 16:51:17 inserted: 120
> 2020-12-01 16:51:34 inserted: 130
> 2020-12-01 16:51:49 inserted: 140
> 2020-12-01 16:52:05 inserted: 150
> 2020-12-01 16:52:21 inserted: 160
> 2020-12-01 16:52:40 inserted: 170
> 2020-12-01 16:52:57 inserted: 180
> 2020-12-01 16:53:19 inserted: 190
> 2020-12-01 16:53:42 inserted: 200
> 2020-12-01 16:53:48 inserted: 200
> imp finished ok!
> --job finished--
> ---Hbase.2.0.2 (
> HDP3.1.1)-
> 2020-12-01 17:25:24 inserted: 10
> 2020-12-01 17:26:03 inserted: 20
> 2020-12-01 17:26:39 inserted: 30
> 2020-12-01 17:27:13 inserted: 40
> 2020-12-01 17:27:47 inserted: 50
> 2020-12-01 17:28:23 inserted: 60
> 2020-12-01 17:29:03 inserted: 70
> 2020-12-01 17:29:40 inserted: 80
> 2020-12-01 17:30:15 inserted: 90
> 2020-12-01 17:30:51 inserted: 100
> 2020-12-01 17:31:27 inserted: 110
> 2020-12-01 17:32:03 inserted: 120
> 2020-12-01 17:32:39 inserted: 130
> 2020-12-01 17:33:14 inserted: 140
> 2020-12-01 17:33:50 inserted: 150
> 2020-12-01 17:34:25 inserted: 160
> 2020-12-01 17:35:01 inserted: 170
> 2020-12-01 17:35:38 inserted: 180
> 2020-12-01 17:36:14 inserted: 190
> 2020-12-01 17:36:51 inserted: 200
> 2020-12-01 17:36:55 inserted: 200
> imp finished ok!
> --job finished--
> returnCode=0
> In addition, we also did some benchmark tests on the production cluster.The
> delay is seem to be a bit high. The detailed report is in the attachment.
> Are there any key points that I have not done configuration? or，, this
> version has performance defects ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
