[jira] [Created] (HBASE-15062) IntegrationTestMTTR conditionally run some tests
Samir Ahmic created HBASE-15062: --- Summary: IntegrationTestMTTR conditionally run some tests Key: HBASE-15062 URL: https://issues.apache.org/jira/browse/HBASE-15062 Project: HBase Issue Type: Improvement Components: integration tests Affects Versions: 2.0.0 Reporter: Samir Ahmic Assignee: Samir Ahmic Priority: Minor On master branch master-rs collocation is on by default which cause that following tests in IntegrationTestMTTR will always fail: - testRestartRsHoldingTable - testKillRsHoldingMeta - testMoveMeta I will add condition so that if we have system tables on master tests above will be skipped else we will run them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15062) IntegrationTestMTTR conditionally run some tests
[ https://issues.apache.org/jira/browse/HBASE-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15062: Description: On master branch master-rs collocation is on by default which cause that following test in IntegrationTestMTTR is same test as restartMaster - testKillRsHoldingMeta I will add condition so that if we have system tables on master test above will be skipped else we will run it was: On master branch master-rs collocation is on by default which cause that following test in IntegrationTestMTTR are same test as restartMaster - testKillRsHoldingMeta I will add condition so that if we have system tables on master test above will be skipped else we will run it > IntegrationTestMTTR conditionally run some tests > - > > Key: HBASE-15062 > URL: https://issues.apache.org/jira/browse/HBASE-15062 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > > On master branch master-rs collocation is on by default which cause that > following test in IntegrationTestMTTR is same test as restartMaster > - testKillRsHoldingMeta > I will add condition so that if we have system tables on master test above > will be skipped else we will run it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15062) IntegrationTestMTTR conditionally run some tests
[ https://issues.apache.org/jira/browse/HBASE-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15062: Description: On master branch master-rs collocation is on by default which cause that following test in IntegrationTestMTTR are same test as restartMaster - testKillRsHoldingMeta I will add condition so that if we have system tables on master test above will be skipped else we will run it was: On master branch master-rs collocation is on by default which cause that following tests in IntegrationTestMTTR are same test as restartMaster - testRestartRsHoldingTable - testKillRsHoldingMeta I will add condition so that if we have system tables on master tests above will be skipped else we will run them. > IntegrationTestMTTR conditionally run some tests > - > > Key: HBASE-15062 > URL: https://issues.apache.org/jira/browse/HBASE-15062 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > > On master branch master-rs collocation is on by default which cause that > following test in IntegrationTestMTTR are same test as restartMaster > - testKillRsHoldingMeta > I will add condition so that if we have system tables on master test above > will be skipped else we will run it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15016) StoreServices facility in Region
[ https://issues.apache.org/jira/browse/HBASE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-15016: -- Attachment: Regioncounters.pdf The attached file (Region counters) show a view of the region counters before and after the patch, and can help understand how they affect the flush decisions. > StoreServices facility in Region > > > Key: HBASE-15016 > URL: https://issues.apache.org/jira/browse/HBASE-15016 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-15016-V01.patch, HBASE-15016-V02.patch, > HBASE-15016-V03.patch, Regioncounters.pdf > > > The default implementation of a memstore ensures that between two flushes the > memstore size increases monotonically. Supporting new memstores that store > data in different formats (specifically, compressed), or that allows to > eliminate data redundancies in memory (e.g., via compaction), means that the > size of the data stored in memory can decrease even between two flushes. This > requires memstores to have access to facilities that manipulate region > counters and synchronization. > This subtasks introduces a new region interface -- StoreServices, through > which store components can access these facilities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15058) AssignmentManager should account for unsuccessful split correctly which initially passes quota check
[
https://issues.apache.org/jira/browse/HBASE-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15058:
---
Description:
When region split doesn't pass quota check, we would see exception similar to
the following:
{code}
2015-12-29 16:07:33,653 INFO [RS:0;10.21.128.189:57449-splits-1451434041585]
regionserver.SplitRequest(97): Running rollback/cleanup of failed split of np2:
testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.;
Failed to get ok from master to split
np2:testRegionNormalizationSplitOnCluster,
z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
java.io.IOException: Failed to get ok from master to split
np2:testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
at
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:345)
at
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:262)
at
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:502)
at
org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
at
org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:155)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
{code}
However, region split may fail for subsequent SplitTransactionPhase's in
stepsBeforePONR().
Currently in branch-1, the distinction among the following TransitionCode's is
not clear in AssignmentManager#onRegionTransition():
{code}
case SPLIT_PONR:
case SPLIT:
case SPLIT_REVERTED:
errorMsg =
onRegionSplit(serverName, code, hri,
HRegionInfo.convert(transition.getRegionInfo(1)),
HRegionInfo.convert(transition.getRegionInfo(2)));
if (org.apache.commons.lang.StringUtils.isEmpty(errorMsg)) {
try {
regionStateListener.onRegionSplitReverted(hri);
{code}
onRegionSplit() handles the above 3 TransitionCode's. However, errorMsg is
normally null (onRegionSplit returns null at the end).
This would result in onRegionSplitReverted() being called for cases of
SPLIT_PONR and SPLIT.
When region split fails, AssignmentManager#onRegionTransition() should account
for the failure properly so that quota bookkeeping is consistent.
was:
When region split doesn't pass quota check, we would see exception similar to
the following:
{code}
2015-12-29 16:07:33,653 INFO [RS:0;10.21.128.189:57449-splits-1451434041585]
regionserver.SplitRequest(97): Running rollback/cleanup of failed split of np2:
testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.;
Failed to get ok from master to split
np2:testRegionNormalizationSplitOnCluster,
z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
java.io.IOException: Failed to get ok from master to split
np2:testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
at
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:345)
at
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:262)
at
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:502)
at
org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
at
org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:155)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
{code}
However, region split may fail for subsequent SplitTransactionPhase's in
stepsBeforePONR().
Currently in branch-1, the distinction among the following states is not clear
in AssignmentManager#onRegionTransition():
{code}
case SPLIT_PONR:
case SPLIT:
case SPLIT_REVERTED:
errorMsg =
onRegionSplit(serverName, code, hri,
HRegionInfo.convert(transition.getRegionInfo(1)),
HRegionInfo.convert(transition.getRegionInfo(2)));
if (org.apache.commons.lang.StringUtils.isEmpty(errorMsg)) {
try {
regionStateListener.onRegionSplitReverted(hri);
{code}
onRegionSplit() handles the above 3 TransitionCode's. However, errorMsg is
normally null (onRegionSplit returns null at the end).
This would result in onRegionSplitReverted() being called for cases of
SPLIT_PONR and SPLIT.
When region split fails, AssignmentManager#onRegionTransition() should account
for the failure properly so that quota bookkeeping is consistent.
> AssignmentManager should account for unsuccessful split correctly which
> initially passes quota check
>
>
> Key: HBASE-15058
> URL:
[jira] [Updated] (HBASE-15062) IntegrationTestMTTR conditionally run some tests
[ https://issues.apache.org/jira/browse/HBASE-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15062: Description: On master branch master-rs collocation is on by default which cause that following tests in IntegrationTestMTTR are same test as restartMaster - testRestartRsHoldingTable - testKillRsHoldingMeta I will add condition so that if we have system tables on master tests above will be skipped else we will run them. was: On master branch master-rs collocation is on by default which cause that following tests in IntegrationTestMTTR will always fail: - testRestartRsHoldingTable - testKillRsHoldingMeta - testMoveMeta I will add condition so that if we have system tables on master tests above will be skipped else we will run them. > IntegrationTestMTTR conditionally run some tests > - > > Key: HBASE-15062 > URL: https://issues.apache.org/jira/browse/HBASE-15062 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > > On master branch master-rs collocation is on by default which cause that > following tests in IntegrationTestMTTR are same test as restartMaster > - testRestartRsHoldingTable > - testKillRsHoldingMeta > I will add condition so that if we have system tables on master tests above > will be skipped else we will run them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15058) AssignmentManager should account for unsuccessful split correctly which initially passes quota check
[
https://issues.apache.org/jira/browse/HBASE-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15058:
---
Summary: AssignmentManager should account for unsuccessful split correctly
which initially passes quota check (was: NamespaceAuditor should account for
unsuccessful split which initially passes quota check)
> AssignmentManager should account for unsuccessful split correctly which
> initially passes quota check
>
>
> Key: HBASE-15058
> URL: https://issues.apache.org/jira/browse/HBASE-15058
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>
> When region split doesn't pass quota check, we would see exception similar to
> the following:
> {code}
> 2015-12-29 16:07:33,653 INFO [RS:0;10.21.128.189:57449-splits-1451434041585]
> regionserver.SplitRequest(97): Running rollback/cleanup of failed split of
> np2:
> testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.;
> Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,
> z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> java.io.IOException: Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:345)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:262)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:502)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:155)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> However, region split may fail for subsequent SplitTransactionPhase's in
> stepsBeforePONR().
> Currently there is no mechanism to rollback the update to namespace quota.
> When region split fails, NamespaceAuditor should account for the failure so
> that quota bookkeeping is consistent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15058) AssignmentManager should account for unsuccessful split correctly which initially passes quota check
[
https://issues.apache.org/jira/browse/HBASE-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15058:
---
Affects Version/s: 1.2.0
> AssignmentManager should account for unsuccessful split correctly which
> initially passes quota check
>
>
> Key: HBASE-15058
> URL: https://issues.apache.org/jira/browse/HBASE-15058
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Ted Yu
>
> When region split doesn't pass quota check, we would see exception similar to
> the following:
> {code}
> 2015-12-29 16:07:33,653 INFO [RS:0;10.21.128.189:57449-splits-1451434041585]
> regionserver.SplitRequest(97): Running rollback/cleanup of failed split of
> np2:
> testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.;
> Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,
> z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> java.io.IOException: Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:345)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:262)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:502)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:155)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> However, region split may fail for subsequent SplitTransactionPhase's in
> stepsBeforePONR().
> Currently in branch-1, the distinction among the following states is not
> clear in AssignmentManager#onRegionTransition():
> {code}
> case SPLIT_PONR:
> case SPLIT:
> case SPLIT_REVERTED:
> errorMsg =
> onRegionSplit(serverName, code, hri,
> HRegionInfo.convert(transition.getRegionInfo(1)),
> HRegionInfo.convert(transition.getRegionInfo(2)));
> if (org.apache.commons.lang.StringUtils.isEmpty(errorMsg)) {
> try {
> regionStateListener.onRegionSplitReverted(hri);
> {code}
> onRegionSplit() handles the above 3 TransitionCode's. However, errorMsg is
> normally null (onRegionSplit returns null at the end).
> This would result in onRegionSplitReverted() being called for cases of
> SPLIT_PONR and SPLIT.
> When region split fails, AssignmentManager#onRegionTransition() should
> account for the failure properly so that quota bookkeeping is consistent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15058) AssignmentManager should account for unsuccessful split correctly which initially passes quota check
[
https://issues.apache.org/jira/browse/HBASE-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15058:
---
Description:
When region split doesn't pass quota check, we would see exception similar to
the following:
{code}
2015-12-29 16:07:33,653 INFO [RS:0;10.21.128.189:57449-splits-1451434041585]
regionserver.SplitRequest(97): Running rollback/cleanup of failed split of np2:
testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.;
Failed to get ok from master to split
np2:testRegionNormalizationSplitOnCluster,
z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
java.io.IOException: Failed to get ok from master to split
np2:testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
at
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:345)
at
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:262)
at
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:502)
at
org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
at
org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:155)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
{code}
However, region split may fail for subsequent SplitTransactionPhase's in
stepsBeforePONR().
Currently in branch-1, the distinction among the following states is not clear
in AssignmentManager#onRegionTransition():
{code}
case SPLIT_PONR:
case SPLIT:
case SPLIT_REVERTED:
errorMsg =
onRegionSplit(serverName, code, hri,
HRegionInfo.convert(transition.getRegionInfo(1)),
HRegionInfo.convert(transition.getRegionInfo(2)));
if (org.apache.commons.lang.StringUtils.isEmpty(errorMsg)) {
try {
regionStateListener.onRegionSplitReverted(hri);
{code}
onRegionSplit() handles the above 3 TransitionCode's. However, errorMsg is
normally null (onRegionSplit returns null at the end).
This would result in onRegionSplitReverted() being called for cases of
SPLIT_PONR and SPLIT.
When region split fails, AssignmentManager#onRegionTransition() should account
for the failure properly so that quota bookkeeping is consistent.
was:
When region split doesn't pass quota check, we would see exception similar to
the following:
{code}
2015-12-29 16:07:33,653 INFO [RS:0;10.21.128.189:57449-splits-1451434041585]
regionserver.SplitRequest(97): Running rollback/cleanup of failed split of np2:
testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.;
Failed to get ok from master to split
np2:testRegionNormalizationSplitOnCluster,
z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
java.io.IOException: Failed to get ok from master to split
np2:testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
at
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:345)
at
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:262)
at
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:502)
at
org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
at
org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:155)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
{code}
However, region split may fail for subsequent SplitTransactionPhase's in
stepsBeforePONR().
Currently there is no mechanism to rollback the update to namespace quota.
When region split fails, NamespaceAuditor should account for the failure so
that quota bookkeeping is consistent.
> AssignmentManager should account for unsuccessful split correctly which
> initially passes quota check
>
>
> Key: HBASE-15058
> URL: https://issues.apache.org/jira/browse/HBASE-15058
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>
> When region split doesn't pass quota check, we would see exception similar to
> the following:
> {code}
> 2015-12-29 16:07:33,653 INFO [RS:0;10.21.128.189:57449-splits-1451434041585]
> regionserver.SplitRequest(97): Running rollback/cleanup of failed split of
> np2:
> testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.;
> Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,
> z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> java.io.IOException: Failed to get ok from
[jira] [Commented] (HBASE-15058) AssignmentManager should account for unsuccessful split correctly which initially passes quota check
[
https://issues.apache.org/jira/browse/HBASE-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075952#comment-15075952
]
Ted Yu commented on HBASE-15058:
Turns out that I was looking at branch-1 code.
Modified description to be specific.
> AssignmentManager should account for unsuccessful split correctly which
> initially passes quota check
>
>
> Key: HBASE-15058
> URL: https://issues.apache.org/jira/browse/HBASE-15058
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>
> When region split doesn't pass quota check, we would see exception similar to
> the following:
> {code}
> 2015-12-29 16:07:33,653 INFO [RS:0;10.21.128.189:57449-splits-1451434041585]
> regionserver.SplitRequest(97): Running rollback/cleanup of failed split of
> np2:
> testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.;
> Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,
> z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> java.io.IOException: Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:345)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:262)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:502)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:155)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> However, region split may fail for subsequent SplitTransactionPhase's in
> stepsBeforePONR().
> Currently in branch-1, the distinction among the following states is not
> clear in AssignmentManager#onRegionTransition():
> {code}
> case SPLIT_PONR:
> case SPLIT:
> case SPLIT_REVERTED:
> errorMsg =
> onRegionSplit(serverName, code, hri,
> HRegionInfo.convert(transition.getRegionInfo(1)),
> HRegionInfo.convert(transition.getRegionInfo(2)));
> if (org.apache.commons.lang.StringUtils.isEmpty(errorMsg)) {
> try {
> regionStateListener.onRegionSplitReverted(hri);
> {code}
> onRegionSplit() handles the above 3 TransitionCode's. However, errorMsg is
> normally null (onRegionSplit returns null at the end).
> This would result in onRegionSplitReverted() being called for cases of
> SPLIT_PONR and SPLIT.
> When region split fails, AssignmentManager#onRegionTransition() should
> account for the failure properly so that quota bookkeeping is consistent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15059) Allow 0.94 to compile against Hadoop 2.7.x
[ https://issues.apache.org/jira/browse/HBASE-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076053#comment-15076053 ] Lars Hofhansl commented on HBASE-15059: --- [~stack] you OK with committing V2? > Allow 0.94 to compile against Hadoop 2.7.x > -- > > Key: HBASE-15059 > URL: https://issues.apache.org/jira/browse/HBASE-15059 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15059-v2.txt, 15059.txt > > > Currently HBase 0.94 cannot be compiled against Hadoop 2.7. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15060) Cull TestHFileWriterV2 and HFileWriterFactory
[ https://issues.apache.org/jira/browse/HBASE-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076100#comment-15076100 ] Hudson commented on HBASE-15060: FAILURE: Integrated in HBase-Trunk_matrix #602 (See [https://builds.apache.org/job/HBase-Trunk_matrix/602/]) HBASE-15060 Cull TestHFileV2 and HFileWriterFactory (jmhsieh: rev 92abf8ac5743849d4c32d6abf8fd33f161bab249) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/RandomKeyValueUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLazyDataBlockDecompression.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekBeforeWithInlineBlocks.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterFactory.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV3.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileInlineToRootChunkConversion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestPrefetch.java > Cull TestHFileWriterV2 and HFileWriterFactory > - > > Key: HBASE-15060 > URL: https://issues.apache.org/jira/browse/HBASE-15060 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-15060.patch > > > 2.x removes hfile v2. This patch cleans up remnants from unit tests > associated with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15058) AssignmentManager should account for unsuccessful split correctly which initially passes quota check
[
https://issues.apache.org/jira/browse/HBASE-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15058:
---
Attachment: 15058-branch-1-v1.txt
Patch v1 separates the logic for SPLIT_REVERTED into onRegionSplitReverted()
Handling of the case of SPLIT_REVERTED is now not sharing the code path with
that for SPLIT_PONR and SPLIT cases.
> AssignmentManager should account for unsuccessful split correctly which
> initially passes quota check
>
>
> Key: HBASE-15058
> URL: https://issues.apache.org/jira/browse/HBASE-15058
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Ted Yu
> Attachments: 15058-branch-1-v1.txt
>
>
> When region split doesn't pass quota check, we would see exception similar to
> the following:
> {code}
> 2015-12-29 16:07:33,653 INFO [RS:0;10.21.128.189:57449-splits-1451434041585]
> regionserver.SplitRequest(97): Running rollback/cleanup of failed split of
> np2:
> testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.;
> Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,
> z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> java.io.IOException: Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:345)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:262)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:502)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:155)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> However, region split may fail for subsequent SplitTransactionPhase's in
> stepsBeforePONR().
> Currently in branch-1, the distinction among the following TransitionCode's
> is not clear in AssignmentManager#onRegionTransition():
> {code}
> case SPLIT_PONR:
> case SPLIT:
> case SPLIT_REVERTED:
> errorMsg =
> onRegionSplit(serverName, code, hri,
> HRegionInfo.convert(transition.getRegionInfo(1)),
> HRegionInfo.convert(transition.getRegionInfo(2)));
> if (org.apache.commons.lang.StringUtils.isEmpty(errorMsg)) {
> try {
> regionStateListener.onRegionSplitReverted(hri);
> {code}
> onRegionSplit() handles the above 3 TransitionCode's. However, errorMsg is
> normally null (onRegionSplit returns null at the end).
> This would result in onRegionSplitReverted() being called for cases of
> SPLIT_PONR and SPLIT.
> When region split fails, AssignmentManager#onRegionTransition() should
> account for the failure properly so that quota bookkeeping is consistent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15058) AssignmentManager should account for unsuccessful split correctly which initially passes quota check
[
https://issues.apache.org/jira/browse/HBASE-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15058:
---
Assignee: Ted Yu
Status: Patch Available (was: Open)
> AssignmentManager should account for unsuccessful split correctly which
> initially passes quota check
>
>
> Key: HBASE-15058
> URL: https://issues.apache.org/jira/browse/HBASE-15058
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15058-branch-1-v1.txt
>
>
> When region split doesn't pass quota check, we would see exception similar to
> the following:
> {code}
> 2015-12-29 16:07:33,653 INFO [RS:0;10.21.128.189:57449-splits-1451434041585]
> regionserver.SplitRequest(97): Running rollback/cleanup of failed split of
> np2:
> testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.;
> Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,
> z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> java.io.IOException: Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:345)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:262)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:502)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:155)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> However, region split may fail for subsequent SplitTransactionPhase's in
> stepsBeforePONR().
> Currently in branch-1, the distinction among the following TransitionCode's
> is not clear in AssignmentManager#onRegionTransition():
> {code}
> case SPLIT_PONR:
> case SPLIT:
> case SPLIT_REVERTED:
> errorMsg =
> onRegionSplit(serverName, code, hri,
> HRegionInfo.convert(transition.getRegionInfo(1)),
> HRegionInfo.convert(transition.getRegionInfo(2)));
> if (org.apache.commons.lang.StringUtils.isEmpty(errorMsg)) {
> try {
> regionStateListener.onRegionSplitReverted(hri);
> {code}
> onRegionSplit() handles the above 3 TransitionCode's. However, errorMsg is
> normally null (onRegionSplit returns null at the end).
> This would result in onRegionSplitReverted() being called for cases of
> SPLIT_PONR and SPLIT.
> When region split fails, AssignmentManager#onRegionTransition() should
> account for the failure properly so that quota bookkeeping is consistent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15062) IntegrationTestMTTR conditionally run some tests
[ https://issues.apache.org/jira/browse/HBASE-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15062: Attachment: HBASE-15062-v1.patch Here is new patch in v0 i have removed one comment accidentally, > IntegrationTestMTTR conditionally run some tests > - > > Key: HBASE-15062 > URL: https://issues.apache.org/jira/browse/HBASE-15062 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15062-v0.patch, HBASE-15062-v1.patch > > > On master branch master-rs collocation is on by default which cause that > following test in IntegrationTestMTTR is same test as restartMaster > - testKillRsHoldingMeta > I will add condition so that if we have system tables on master test above > will be skipped else we will run it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15062) IntegrationTestMTTR conditionally run some tests
[ https://issues.apache.org/jira/browse/HBASE-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15062: Fix Version/s: 2.0.0 Status: Patch Available (was: Open) > IntegrationTestMTTR conditionally run some tests > - > > Key: HBASE-15062 > URL: https://issues.apache.org/jira/browse/HBASE-15062 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15062-v0.patch, HBASE-15062-v1.patch > > > On master branch master-rs collocation is on by default which cause that > following test in IntegrationTestMTTR is same test as restartMaster > - testKillRsHoldingMeta > I will add condition so that if we have system tables on master test above > will be skipped else we will run it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15061) Refactor StoreFileScanner creation to builder pattern
[
https://issues.apache.org/jira/browse/HBASE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076044#comment-15076044
]
Jonathan Hsieh commented on HBASE-15061:
[~anoopsamjohn] Having to look up what the (..., false, true,false, false...)
means everytime by reading the chain of 3 flavors of getScannerForStoreFiles
makes it harder to read/review the code. I prefer going to one java doc to
find out which settings are on and off by default. If we say what the defaults
are there, the invocations highlight when we purposely are choosing different
settings instead of hiding them. When the next extension comes along the
series of booleans gets more difficult to grok.
I've chosen to require the list of files and readpoint in the constructor and
making them require arguments. Is your suggestion to potentially moving the
caching setting into the constructor to make it a required argument? At least
from the patch, that values is essentially always set false except for when an
override comes in from the scan.
Bigger picture -- related to HBASE-15051, there is more coming. I'm going
through and trying to regularize and cleanup all the argument handling and
instantiation around hfiles and their reader/writers. I'm working my way
through this and plan on getting to somewhere where all the flavors of hfile
reader/writer options (Conf, HTD, HCF, HFileContexts, StoreFileInfos,
CacheConfig, HalfStoreFile, References, HFileLinks) are a bit more unified.
[~larsh] Good point. I'll do a profiling run before considering a commit to
trunk.
> Refactor StoreFileScanner creation to builder pattern
> -
>
> Key: HBASE-15061
> URL: https://issues.apache.org/jira/browse/HBASE-15061
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-15061.patch, hbase-15061.v2.patch
>
>
> There are several falvors of calls that creates a list of StoreFileScanners,
> and new feature have been added to this recently. This patch converts the
> somewhat difficult to read (need to go to javadoc) call:
> {code}
> // which args are the most relevant to this?
> - List sfScanners =
> StoreFileScanner.getScannersForStoreFiles(sfs,
> -cacheMobBlocks, true, false, false, readPt);
> {code}
> into one that is more literate:
> {code}
> // ah, very clearly we are using defaults except for the caching settings
> + List sfScanners = new
> StoreFileScanner.ListBuilder(sfs, readPt)
> + .withCacheBlocks(cacheMobBlocks).build();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15058) AssignmentManager should account for unsuccessful split correctly which initially passes quota check
[
https://issues.apache.org/jira/browse/HBASE-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15058:
---
Attachment: (was: 15058-branch-1-v1.txt)
> AssignmentManager should account for unsuccessful split correctly which
> initially passes quota check
>
>
> Key: HBASE-15058
> URL: https://issues.apache.org/jira/browse/HBASE-15058
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15058-branch-1-v1.txt
>
>
> When region split doesn't pass quota check, we would see exception similar to
> the following:
> {code}
> 2015-12-29 16:07:33,653 INFO [RS:0;10.21.128.189:57449-splits-1451434041585]
> regionserver.SplitRequest(97): Running rollback/cleanup of failed split of
> np2:
> testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.;
> Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,
> z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> java.io.IOException: Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:345)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:262)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:502)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:155)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> However, region split may fail for subsequent SplitTransactionPhase's in
> stepsBeforePONR().
> Currently in branch-1, the distinction among the following TransitionCode's
> is not clear in AssignmentManager#onRegionTransition():
> {code}
> case SPLIT_PONR:
> case SPLIT:
> case SPLIT_REVERTED:
> errorMsg =
> onRegionSplit(serverName, code, hri,
> HRegionInfo.convert(transition.getRegionInfo(1)),
> HRegionInfo.convert(transition.getRegionInfo(2)));
> if (org.apache.commons.lang.StringUtils.isEmpty(errorMsg)) {
> try {
> regionStateListener.onRegionSplitReverted(hri);
> {code}
> onRegionSplit() handles the above 3 TransitionCode's. However, errorMsg is
> normally null (onRegionSplit returns null at the end).
> This would result in onRegionSplitReverted() being called for cases of
> SPLIT_PONR and SPLIT.
> When region split fails, AssignmentManager#onRegionTransition() should
> account for the failure properly so that quota bookkeeping is consistent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15058) AssignmentManager should account for unsuccessful split correctly which initially passes quota check
[
https://issues.apache.org/jira/browse/HBASE-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15058:
---
Attachment: 15058-branch-1-v1.txt
> AssignmentManager should account for unsuccessful split correctly which
> initially passes quota check
>
>
> Key: HBASE-15058
> URL: https://issues.apache.org/jira/browse/HBASE-15058
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15058-branch-1-v1.txt
>
>
> When region split doesn't pass quota check, we would see exception similar to
> the following:
> {code}
> 2015-12-29 16:07:33,653 INFO [RS:0;10.21.128.189:57449-splits-1451434041585]
> regionserver.SplitRequest(97): Running rollback/cleanup of failed split of
> np2:
> testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.;
> Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,
> z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> java.io.IOException: Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:345)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:262)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:502)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:155)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> However, region split may fail for subsequent SplitTransactionPhase's in
> stepsBeforePONR().
> Currently in branch-1, the distinction among the following TransitionCode's
> is not clear in AssignmentManager#onRegionTransition():
> {code}
> case SPLIT_PONR:
> case SPLIT:
> case SPLIT_REVERTED:
> errorMsg =
> onRegionSplit(serverName, code, hri,
> HRegionInfo.convert(transition.getRegionInfo(1)),
> HRegionInfo.convert(transition.getRegionInfo(2)));
> if (org.apache.commons.lang.StringUtils.isEmpty(errorMsg)) {
> try {
> regionStateListener.onRegionSplitReverted(hri);
> {code}
> onRegionSplit() handles the above 3 TransitionCode's. However, errorMsg is
> normally null (onRegionSplit returns null at the end).
> This would result in onRegionSplitReverted() being called for cases of
> SPLIT_PONR and SPLIT.
> When region split fails, AssignmentManager#onRegionTransition() should
> account for the failure properly so that quota bookkeeping is consistent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15062) IntegrationTestMTTR conditionally run some tests
[ https://issues.apache.org/jira/browse/HBASE-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15062: Attachment: HBASE-15062-v0.patch Here is patch. > IntegrationTestMTTR conditionally run some tests > - > > Key: HBASE-15062 > URL: https://issues.apache.org/jira/browse/HBASE-15062 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Attachments: HBASE-15062-v0.patch > > > On master branch master-rs collocation is on by default which cause that > following test in IntegrationTestMTTR is same test as restartMaster > - testKillRsHoldingMeta > I will add condition so that if we have system tables on master test above > will be skipped else we will run it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15060) Cull TestHFileWriterV2 and HFileWriterFactory
[ https://issues.apache.org/jira/browse/HBASE-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076045#comment-15076045 ] Jonathan Hsieh commented on HBASE-15060: committed to master/2.x. thanks for review stack. > Cull TestHFileWriterV2 and HFileWriterFactory > - > > Key: HBASE-15060 > URL: https://issues.apache.org/jira/browse/HBASE-15060 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-15060.patch > > > 2.x removes hfile v2. This patch cleans up remnants from unit tests > associated with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15060) Cull TestHFileWriterV2 and HFileWriterFactory
[ https://issues.apache.org/jira/browse/HBASE-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-15060: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Cull TestHFileWriterV2 and HFileWriterFactory > - > > Key: HBASE-15060 > URL: https://issues.apache.org/jira/browse/HBASE-15060 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-15060.patch > > > 2.x removes hfile v2. This patch cleans up remnants from unit tests > associated with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15058) AssignmentManager should account for unsuccessful split correctly which initially passes quota check
[
https://issues.apache.org/jira/browse/HBASE-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076134#comment-15076134
]
Hadoop QA commented on HBASE-15058:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12780135/15058-branch-1-v1.txt
against branch-1 branch at commit 92abf8ac5743849d4c32d6abf8fd33f161bab249.
ATTACHMENT ID: 12780135
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/17095//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/17095//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/17095//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17095//console
This message is automatically generated.
> AssignmentManager should account for unsuccessful split correctly which
> initially passes quota check
>
>
> Key: HBASE-15058
> URL: https://issues.apache.org/jira/browse/HBASE-15058
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15058-branch-1-v1.txt
>
>
> When region split doesn't pass quota check, we would see exception similar to
> the following:
> {code}
> 2015-12-29 16:07:33,653 INFO [RS:0;10.21.128.189:57449-splits-1451434041585]
> regionserver.SplitRequest(97): Running rollback/cleanup of failed split of
> np2:
> testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.;
> Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,
> z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> java.io.IOException: Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:345)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:262)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:502)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:155)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> However, region split may fail for subsequent SplitTransactionPhase's in
> stepsBeforePONR().
> Currently in branch-1, the distinction among the following TransitionCode's
> is not clear in AssignmentManager#onRegionTransition():
> {code}
> case SPLIT_PONR:
> case SPLIT:
> case SPLIT_REVERTED:
> errorMsg =
> onRegionSplit(serverName, code, hri,
> HRegionInfo.convert(transition.getRegionInfo(1)),
> HRegionInfo.convert(transition.getRegionInfo(2)));
> if (org.apache.commons.lang.StringUtils.isEmpty(errorMsg)) {
> try {
>
[jira] [Commented] (HBASE-15054) Allow 0.94 to compile with JDK8
[ https://issues.apache.org/jira/browse/HBASE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076163#comment-15076163 ] Hudson commented on HBASE-15054: SUCCESS: Integrated in HBase-0.94-security #589 (See [https://builds.apache.org/job/HBase-0.94-security/589/]) HBASE-15054 Allow 0.94 to compile with JDK8. (larsh: rev 30b0c98245b16a8d8e44b94d24aa3f5f35977b8e) * src/main/java/org/apache/hadoop/hbase/client/HTablePool.java * src/main/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/InputSampler.java * src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java * src/main/java/org/apache/hadoop/hbase/util/PoolMap.java * security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java > Allow 0.94 to compile with JDK8 > --- > > Key: HBASE-15054 > URL: https://issues.apache.org/jira/browse/HBASE-15054 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15054.txt > > > Fix the following two problems: > # PoolMap > # InputSampler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15058) AssignmentManager should account for unsuccessful split correctly which initially passes quota check
[
https://issues.apache.org/jira/browse/HBASE-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076137#comment-15076137
]
Hadoop QA commented on HBASE-15058:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12780162/15058-branch-1-v1.txt
against branch-1 branch at commit 92abf8ac5743849d4c32d6abf8fd33f161bab249.
ATTACHMENT ID: 12780162
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestShell
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/17096//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/17096//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/17096//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17096//console
This message is automatically generated.
> AssignmentManager should account for unsuccessful split correctly which
> initially passes quota check
>
>
> Key: HBASE-15058
> URL: https://issues.apache.org/jira/browse/HBASE-15058
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15058-branch-1-v1.txt
>
>
> When region split doesn't pass quota check, we would see exception similar to
> the following:
> {code}
> 2015-12-29 16:07:33,653 INFO [RS:0;10.21.128.189:57449-splits-1451434041585]
> regionserver.SplitRequest(97): Running rollback/cleanup of failed split of
> np2:
> testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.;
> Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,
> z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> java.io.IOException: Failed to get ok from master to split
> np2:testRegionNormalizationSplitOnCluster,z,1451434045065.27cccb3fae03002b8058beef61cb7c20.
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:345)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:262)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:502)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:155)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> However, region split may fail for subsequent SplitTransactionPhase's in
> stepsBeforePONR().
> Currently in branch-1, the distinction among the following TransitionCode's
> is not clear in AssignmentManager#onRegionTransition():
> {code}
> case SPLIT_PONR:
> case SPLIT:
> case SPLIT_REVERTED:
> errorMsg =
> onRegionSplit(serverName, code, hri,
> HRegionInfo.convert(transition.getRegionInfo(1)),
> HRegionInfo.convert(transition.getRegionInfo(2)));
> if
[jira] [Commented] (HBASE-15054) Allow 0.94 to compile with JDK8
[ https://issues.apache.org/jira/browse/HBASE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076166#comment-15076166 ] Hudson commented on HBASE-15054: FAILURE: Integrated in HBase-0.94-JDK7 #246 (See [https://builds.apache.org/job/HBase-0.94-JDK7/246/]) HBASE-15054 Allow 0.94 to compile with JDK8. (larsh: rev 30b0c98245b16a8d8e44b94d24aa3f5f35977b8e) * src/main/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/InputSampler.java * src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java * src/main/java/org/apache/hadoop/hbase/util/PoolMap.java * security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java * src/main/java/org/apache/hadoop/hbase/client/HTablePool.java > Allow 0.94 to compile with JDK8 > --- > > Key: HBASE-15054 > URL: https://issues.apache.org/jira/browse/HBASE-15054 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15054.txt > > > Fix the following two problems: > # PoolMap > # InputSampler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Open (was: Patch Available) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055-v3.patch Test cases are updated to verify that major compactions should be triggered only if some TTL expirations. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15061) Refactor StoreFileScanner creation to builder pattern
[
https://issues.apache.org/jira/browse/HBASE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076237#comment-15076237
]
Anoop Sam John commented on HBASE-15061:
bq.Is your suggestion to potentially moving the caching setting into the
constructor to make it a required argument?
No I dont mean that. Constructor to have the list of store files and readpnt
is enough. We have methods like withCacheEnabled(), withPread etc now right. I
mean call it at all create places even if the value to be used there is default
one there. So readers dont have to go to the builder to know what is the
default values. Am I making it clear?
> Refactor StoreFileScanner creation to builder pattern
> -
>
> Key: HBASE-15061
> URL: https://issues.apache.org/jira/browse/HBASE-15061
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-15061.patch, hbase-15061.v2.patch
>
>
> There are several falvors of calls that creates a list of StoreFileScanners,
> and new feature have been added to this recently. This patch converts the
> somewhat difficult to read (need to go to javadoc) call:
> {code}
> // which args are the most relevant to this?
> - List sfScanners =
> StoreFileScanner.getScannersForStoreFiles(sfs,
> -cacheMobBlocks, true, false, false, readPt);
> {code}
> into one that is more literate:
> {code}
> // ah, very clearly we are using defaults except for the caching settings
> + List sfScanners = new
> StoreFileScanner.ListBuilder(sfs, readPt)
> + .withCacheBlocks(cacheMobBlocks).build();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15061) Refactor StoreFileScanner creation to builder pattern
[
https://issues.apache.org/jira/browse/HBASE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075836#comment-15075836
]
Lars Hofhansl commented on HBASE-15061:
---
This is in a semi-hot code path (done for every Get request), so we should be
careful with creating more objects. (Probably OK, since we're seeking the
scanners later anyway, which is much more expensive; still something to
consider)
> Refactor StoreFileScanner creation to builder pattern
> -
>
> Key: HBASE-15061
> URL: https://issues.apache.org/jira/browse/HBASE-15061
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-15061.patch, hbase-15061.v2.patch
>
>
> There are several falvors of calls that creates a list of StoreFileScanners,
> and new feature have been added to this recently. This patch converts the
> somewhat difficult to read (need to go to javadoc) call:
> {code}
> // which args are the most relevant to this?
> - List sfScanners =
> StoreFileScanner.getScannersForStoreFiles(sfs,
> -cacheMobBlocks, true, false, false, readPt);
> {code}
> into one that is more literate:
> {code}
> // ah, very clearly we are using defaults except for the caching settings
> + List sfScanners = new
> StoreFileScanner.ListBuilder(sfs, readPt)
> + .withCacheBlocks(cacheMobBlocks).build();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
