[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680322#comment-16680322
]
Sean Busbey commented on HBASE-21355:
-
awesome, thanks for the follow up.
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679158#comment-16679158
]
Zheng Hu commented on HBASE-21355:
--
bq. did this not impact branch-1.2 or was it just overlooked?
This bug was caused by HBASE-20940, HBASE-20940 was not applied to branch-1.2
, so branch-1.2 isn't impacted. Thanks.
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678681#comment-16678681
]
Sean Busbey commented on HBASE-21355:
-
did this not impact branch-1.2 or was it just overlooked?
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660700#comment-16660700
]
Hudson commented on HBASE-21355:
Results for branch master
[build #563 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/563/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/563//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/563//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/563//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660475#comment-16660475
]
Hudson commented on HBASE-21355:
Results for branch branch-1
[build #521 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/521/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/521//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/521//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/521//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659880#comment-16659880
]
Hudson commented on HBASE-21355:
Results for branch branch-1.3
[build #513 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/513/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/513//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/513//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/513//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659657#comment-16659657
]
Hudson commented on HBASE-21355:
Results for branch branch-1.4
[build #519 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/519/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/519//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/519//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/519//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659250#comment-16659250
]
Hudson commented on HBASE-21355:
Results for branch branch-2.1
[build #513 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/513/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/513//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/513//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/513//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659235#comment-16659235
]
Hudson commented on HBASE-21355:
Results for branch branch-2.0
[build #996 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/996/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/996//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/996//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/996//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659192#comment-16659192
]
Hudson commented on HBASE-21355:
Results for branch branch-2
[build #1428 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1428/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1428//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1428//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1428//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659152#comment-16659152
]
Hudson commented on HBASE-21355:
Results for branch master
[build #561 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/561/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/561//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/561//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/561//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/master/561//console].
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659016#comment-16659016
]
Hudson commented on HBASE-21355:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #494 (See
[https://builds.apache.org/job/HBase-1.3-IT/494/])
HBASE-21355 HStore's storeSize is calculated repeatedly which causing (openinx:
rev ab5e7c2b604c78f69b753a4712818ec56b07915e)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin2.java
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658991#comment-16658991
]
Zheng Hu commented on HBASE-21355:
--
Pushed to all branches. Thanks [~Apache9] & [~yuzhih...@gmail.com] for
reviewing.
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658927#comment-16658927
]
Hudson commented on HBASE-21355:
Results for branch branch-2
[build #1426 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1426/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1426//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1426//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1426//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9, 1.2.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658915#comment-16658915
]
Hadoop QA commented on HBASE-21355:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
51s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
41s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 6s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}138m 8s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}186m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21355 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944962/HBASE-21355.addendum.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 0c8745932656 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 77ac352a95 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/14796/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658905#comment-16658905
]
Hudson commented on HBASE-21355:
Results for branch branch-2.1
[build #511 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/511/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/511//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/511//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/511//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9, 1.2.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658907#comment-16658907
]
Hudson commented on HBASE-21355:
Results for branch branch-2.0
[build #994 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/994/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/994//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/994//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/994//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9, 1.2.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658897#comment-16658897
]
Hudson commented on HBASE-21355:
Results for branch branch-1
[build #519 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/519/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/519//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/519//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/519//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9, 1.2.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658886#comment-16658886
]
Hudson commented on HBASE-21355:
Results for branch branch-1.4
[build #518 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/518/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/518//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/518//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/518//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9, 1.2.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658876#comment-16658876
]
Duo Zhang commented on HBASE-21355:
---
+1 on the 3KB addendum... You'd better change the name if the content is
changed...
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9, 1.2.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch,
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
> Some comment here, I move the storeSize calculation out of loadStoreFiles()
> method, because the secondary read replica's refreshStoreFiles() will also
> use loadStoreFiles() to refresh its store files and update the storeSize in
> the completeCompaction(..) in the final (just like compaction.) , so no need
> calculate the storeSize twice..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658873#comment-16658873
]
Hadoop QA commented on HBASE-21355:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
51s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
0s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 44s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}144m
47s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}195m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21355 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944956/HBASE-21355.addendum.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d032a924fa6e 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 77ac352a95 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14795/testReport/ |
| Max. process+thread count | 4931 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658699#comment-16658699
]
Zheng Hu commented on HBASE-21355:
--
Thanks [~Apache9]'s advise. Upload an addendum for it, and the HBASE-20940
had a UT for this case,
TestEndToEndSplitTransaction#testCanSplitJustAfterASplit. I think it's
enough.
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9, 1.2.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.branch-1.patch,
> HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658698#comment-16658698
]
Duo Zhang commented on HBASE-21355:
---
I mean we do not need to change the way we calculate the total store file size
after the addendum? So let's revert the patch first?
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9, 1.2.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.branch-1.patch,
> HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658527#comment-16658527
]
Duo Zhang commented on HBASE-21355:
---
Talked with [~openinx] offline, we think that the problem is the code
introduced in HBASE-20940. We should not try to open the store files every time
when calling getRegionInfo, and then close them. It is too expensive. Instead,
I think we could also check the compacted files when testing whether a region
is mergable or splittable. And I think we should provide a UT for this, as
there is no test for HBASE-20940.
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9, 1.2.9
>
> Attachments: HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658502#comment-16658502
]
Hadoop QA commented on HBASE-21355:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HBASE-21355 does not apply to branch-1. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.8.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-21355 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944932/HBASE-21355.branch-1.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14791/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9, 1.2.9
>
> Attachments: HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658221#comment-16658221
]
Hadoop QA commented on HBASE-21355:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
21s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
26s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 13s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}201m
44s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}244m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21355 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944888/HBASE-21355.v1.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 2b6833769442 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ae5308ac4a |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14781/testReport/ |
| Max. process+thread count | 4869 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14781/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> HStore's storeSize is
[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split
[
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658125#comment-16658125
]
Zheng Hu commented on HBASE-21355:
--
bq. BTW, we seems forget to maintain the read replica's storeSize when refresh
the store files.
Oh, it's no ture, in the HStore#refreshStoreFilesInternal, we will execute
completeCompaction in the final, so the storeSize & totalUncompressedBytes will
also be refreshed..
> HStore's storeSize is calculated repeatedly which causing the confusing
> region split
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
>
> When testing the branch-2's write performance in our internal cluster, we
> found that the region will be inexplicably split.
> We use the default ConstantSizeRegionSplitPolicy and
> hbase.hregion.max.filesize=40G，but the region will be split even if its
> bytes size is less than 40G(only ~6G).
> Checked the code, I found that the following path will accumulate the
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
> -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh
> the store files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
