[jira] [Commented] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[ https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848388#comment-17848388 ] Bryan Beaudreault commented on HBASE-24791: --- I've updated the fixVersions to account for all of the backports. > Improve HFileOutputFormat2 to avoid always call getTableRelativePath method > --- > > Key: HBASE-24791 > URL: https://issues.apache.org/jira/browse/HBASE-24791 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 3.0.0-alpha-1 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Labels: HFileOutputFormat, bulkload, pull-request-available > Fix For: 3.0.0-alpha-1, 2.4.18, 2.7.0, 2.6.1, 2.5.9 > > > Bulkload use HFileOutputFormat2 to write HFile > In the HFileOutputFormat2.RecordWriter > in the write method always called the getTableRelativePath method each time > This is unnecessary -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[
https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846071#comment-17846071
]
Hudson commented on HBASE-24791:
Results for branch branch-2.5
[build #527 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/527/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/527/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/527/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/527/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/527/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
> ---
>
> Key: HBASE-24791
> URL: https://issues.apache.org/jira/browse/HBASE-24791
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Affects Versions: 3.0.0-alpha-1
>Reporter: Yechao Chen
>Assignee: Yechao Chen
>Priority: Critical
> Labels: HFileOutputFormat, bulkload, pull-request-available
> Fix For: 3.0.0-alpha-1
>
>
> Bulkload use HFileOutputFormat2 to write HFile
> In the HFileOutputFormat2.RecordWriter
> in the write method always called the getTableRelativePath method each time
> This is unnecessary
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[
https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845802#comment-17845802
]
Hudson commented on HBASE-24791:
Results for branch branch-2
[build #1054 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1054/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1054/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1054/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1054/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1054/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
> ---
>
> Key: HBASE-24791
> URL: https://issues.apache.org/jira/browse/HBASE-24791
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Affects Versions: 3.0.0-alpha-1
>Reporter: Yechao Chen
>Assignee: Yechao Chen
>Priority: Critical
> Labels: HFileOutputFormat, bulkload, pull-request-available
> Fix For: 3.0.0-alpha-1
>
>
> Bulkload use HFileOutputFormat2 to write HFile
> In the HFileOutputFormat2.RecordWriter
> in the write method always called the getTableRelativePath method each time
> This is unnecessary
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[
https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845784#comment-17845784
]
Hudson commented on HBASE-24791:
Results for branch branch-2.4
[build #735 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/735/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/735/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/735/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/735/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/735/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
> ---
>
> Key: HBASE-24791
> URL: https://issues.apache.org/jira/browse/HBASE-24791
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Affects Versions: 3.0.0-alpha-1
>Reporter: Yechao Chen
>Assignee: Yechao Chen
>Priority: Critical
> Labels: HFileOutputFormat, bulkload, pull-request-available
> Fix For: 3.0.0-alpha-1
>
>
> Bulkload use HFileOutputFormat2 to write HFile
> In the HFileOutputFormat2.RecordWriter
> in the write method always called the getTableRelativePath method each time
> This is unnecessary
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[
https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845779#comment-17845779
]
Hudson commented on HBASE-24791:
Results for branch branch-2.6
[build #116 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/116/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/116/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/116/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/116/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/116/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
> ---
>
> Key: HBASE-24791
> URL: https://issues.apache.org/jira/browse/HBASE-24791
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Affects Versions: 3.0.0-alpha-1
>Reporter: Yechao Chen
>Assignee: Yechao Chen
>Priority: Critical
> Labels: HFileOutputFormat, bulkload, pull-request-available
> Fix For: 3.0.0-alpha-1
>
>
> Bulkload use HFileOutputFormat2 to write HFile
> In the HFileOutputFormat2.RecordWriter
> in the write method always called the getTableRelativePath method each time
> This is unnecessary
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[ https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249172#comment-17249172 ] Huaxiang Sun commented on HBASE-24791: -- Sorry, was trying to watch and clicked "Assignee" by accident. > Improve HFileOutputFormat2 to avoid always call getTableRelativePath method > --- > > Key: HBASE-24791 > URL: https://issues.apache.org/jira/browse/HBASE-24791 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 3.0.0-alpha-1 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Labels: HFileOutputFormat, bulkload > Fix For: 3.0.0-alpha-1 > > > Bulkload use HFileOutputFormat2 to write HFile > In the HFileOutputFormat2.RecordWriter > in the write method always called the getTableRelativePath method each time > This is unnecessary -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[
https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170054#comment-17170054
]
Hudson commented on HBASE-24791:
Results for branch master
[build #1802 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1802/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1802/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1802/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1802/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/master/1802//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
> ---
>
> Key: HBASE-24791
> URL: https://issues.apache.org/jira/browse/HBASE-24791
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Affects Versions: 3.0.0-alpha-1
>Reporter: Yechao Chen
>Assignee: Yechao Chen
>Priority: Critical
> Labels: HFileOutputFormat, bulkload
> Fix For: 3.0.0-alpha-1
>
>
> Bulkload use HFileOutputFormat2 to write HFile
> In the HFileOutputFormat2.RecordWriter
> in the write method always called the getTableRelativePath method each time
> This is unnecessary
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[ https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167588#comment-17167588 ] Yechao Chen commented on HBASE-24791: - I check the code on branch-2.2, not related [~anoop.hbase] > Improve HFileOutputFormat2 to avoid always call getTableRelativePath method > --- > > Key: HBASE-24791 > URL: https://issues.apache.org/jira/browse/HBASE-24791 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 3.0.0-alpha-1 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Labels: HFileOutputFormat, bulkload > Fix For: 3.0.0-alpha-1 > > > Bulkload use HFileOutputFormat2 to write HFile > In the HFileOutputFormat2.RecordWriter > in the write method always called the getTableRelativePath method each time > This is unnecessary -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[
https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167581#comment-17167581
]
Yechao Chen commented on HBASE-24791:
-
{noformat}
How many columns in ur test? If that is large, the impact will be so huge which
u r seeing. I think this should be marked as a bugfix as it was a perf
regression in 2.0{noformat}
Test case 10 million columns with for each, and loop 10 times,
consider user usually use Bulkload import big data with many rows/columns to
hbase, I think this will be helpful.
> Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
> ---
>
> Key: HBASE-24791
> URL: https://issues.apache.org/jira/browse/HBASE-24791
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Affects Versions: 3.0.0-alpha-1
>Reporter: Yechao Chen
>Assignee: Yechao Chen
>Priority: Critical
> Labels: HFileOutputFormat, bulkload
> Fix For: 3.0.0-alpha-1
>
>
> Bulkload use HFileOutputFormat2 to write HFile
> In the HFileOutputFormat2.RecordWriter
> in the write method always called the getTableRelativePath method each time
> This is unnecessary
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[ https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167198#comment-17167198 ] Anoop Sam John commented on HBASE-24791: Checking further, HBASE-17825 added this code and so it is applicable only to master. Thanks [~pankajkumar] for checking 2.2 for this code path > Improve HFileOutputFormat2 to avoid always call getTableRelativePath method > --- > > Key: HBASE-24791 > URL: https://issues.apache.org/jira/browse/HBASE-24791 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 3.0.0-alpha-1 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Labels: HFileOutputFormat, bulkload > Fix For: 3.0.0-alpha-1 > > > Bulkload use HFileOutputFormat2 to write HFile > In the HFileOutputFormat2.RecordWriter > in the write method always called the getTableRelativePath method each time > This is unnecessary -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[ https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167050#comment-17167050 ] Anoop Sam John commented on HBASE-24791: How many columns in ur test? If that is large, the impact will be so huge which u r seeing. I think this should be marked as a bugfix as it was a perf regression in 2.0 > Improve HFileOutputFormat2 to avoid always call getTableRelativePath method > --- > > Key: HBASE-24791 > URL: https://issues.apache.org/jira/browse/HBASE-24791 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 2.0.0 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Labels: HFileOutputFormat, bulkload > Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.6 > > > Bulkload use HFileOutputFormat2 to write HFile > In the HFileOutputFormat2.RecordWriter > in the write method always called the getTableRelativePath method each time > This is unnecessary -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[ https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166944#comment-17166944 ] Yechao Chen commented on HBASE-24791: - This can imporve a lot A simple test case before improve cost: 8566118066 7954087533 7990385173 8121662923 8094065619 8107938198 8045528125 8116824329 8122912440 8108117395 after improve cost: 1536854 50156 1097601 33953 9907 10927 10445 13076 8596 10386 before all cost: 81227.639801 ms after all cost : 2.781901 ms before vs after: 29198.60908098455:1 The test code in https://github.com/utf7/hbase-client-example/blob/master/src/test/java/TestGetTableRelativePathImprove.java > Improve HFileOutputFormat2 to avoid always call getTableRelativePath method > --- > > Key: HBASE-24791 > URL: https://issues.apache.org/jira/browse/HBASE-24791 > Project: HBase > Issue Type: Improvement >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Major > > Bulkload use HFileOutputFormat2 to write HFile > In the HFileOutputFormat2.RecordWriter > in the write method always called the getTableRelativePath method each time > This is unnecessary -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[ https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166942#comment-17166942 ] Yechao Chen commented on HBASE-24791: - https://github.com/apache/hbase/pull/2167 don't know why auto create the link from github pr > Improve HFileOutputFormat2 to avoid always call getTableRelativePath method > --- > > Key: HBASE-24791 > URL: https://issues.apache.org/jira/browse/HBASE-24791 > Project: HBase > Issue Type: Improvement >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Major > > Bulkload use HFileOutputFormat2 to write HFile > In the HFileOutputFormat2.RecordWriter > in the write method always called the getTableRelativePath method each time > This is unnecessary -- This message was sent by Atlassian Jira (v8.3.4#803005)
