[jira] [Updated] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17584: -- Hadoop Flags: Incompatible change,Reviewed (was: Reviewed) Status: Patch Available (was: Reopened) > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17584-branch-1.patch, HBASE-17584.patch, > HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17584: -- Attachment: HBASE-17584-branch-1.patch Patch for branch-1. And also move renewLease from AbstractClientScanner to ResultScanner as now we are allowed to add methods to interface at minor release. > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17584-branch-1.patch, HBASE-17584.patch, > HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17806) TestRSGroups#testMoveServersAndTables is flaky in master branch
[
https://issues.apache.org/jira/browse/HBASE-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932173#comment-15932173
]
Guangxu Cheng commented on HBASE-17806:
---
Sorry, it was normal to run on my machine many times.
Before executing the method, the default group should only have three tables,
including hbase:meta, hbase:namespace, hbase:rsgroup.
Then, we ceate a new table in the default grouping, just four tables on the
right.
The error message shows that there are five tables, but I do not know who they
are in the end.
>From the information, I can not figure out why the test failed.
> TestRSGroups#testMoveServersAndTables is flaky in master branch
> ---
>
> Key: HBASE-17806
> URL: https://issues.apache.org/jira/browse/HBASE-17806
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>
> According to https://builds.apache.org/job/HBASE-Flaky-Tests ,
> TestRSGroups#testMoveServersAndTables is the top flaky test.
> Recent test failure:
> https://builds.apache.org/job/HBASE-Flaky-Tests/14062/testReport/junit/org.apache.hadoop.hbase.rsgroup/TestRSGroups/testMoveServersAndTables/
> {code}
> java.lang.AssertionError: expected:<4> but was:<5>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.rsgroup.TestRSGroupsBase.testMoveServersAndTables(TestRSGroupsBase.java:747)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15583) Discuss mutable vs immutable HTableDescriptor
[ https://issues.apache.org/jira/browse/HBASE-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932171#comment-15932171 ] Chia-Ping Tsai commented on HBASE-15583: Shell we avoid the obscure immutable HTableDescriptor ? The UnsupportedOperationException is a runtime error. I'd think that we should introduce a "explicit" immutable HTableDescriptor. Two solutions are shown below. # a public interface called ImmutableHTableDescriptor, and HTableDescriptor implement it. The returned type of getTableDescriptor() is changed from HTableDescriptor to ImmutableHTableDescriptor (interface). The getTableDescriptor() return the HTD directly, but user can modify the HTD by casting. # fix the UnmodifyableHTableDescriptor. It will be renamed and don't inherit the HTableDescriptor to get rid of the "modify methods". The returned type of getTableDescriptor() is changed from HTableDescriptor to UnmodifyableHTableDescriptor. If we don't want to introduce an new public class into client library, a bit fix of UnmodifyableHTableDescriptor is another solution. > Discuss mutable vs immutable HTableDescriptor > - > > Key: HBASE-15583 > URL: https://issues.apache.org/jira/browse/HBASE-15583 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Gabor Liptak >Priority: Minor > Labels: beginner > Fix For: 2.0.0 > > > From [~enis] in https://issues.apache.org/jira/browse/HBASE-15505: > PS Should UnmodifyableHTableDescriptor be renamed to > UnmodifiableHTableDescriptor? > It should be named ImmutableHTableDescriptor to be consistent with > collections naming. Let's do this as a subtask of the parent jira, not here. > Thinking about it though, why would we return an Immutable HTD in > HTable.getTableDescriptor() versus a mutable HTD in > Admin.getTableDescriptor(). It does not make sense. Should we just get rid of > the Immutable ones? > We also have UnmodifyableHRegionInfo which is not used at the moment it > seems. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17805) We should remove BoundedByteBufferPool because it is replaced by ByteBufferPool
[ https://issues.apache.org/jira/browse/HBASE-17805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932162#comment-15932162 ] Chia-Ping Tsai commented on HBASE-17805: will commit it tomorrow. > We should remove BoundedByteBufferPool because it is replaced by > ByteBufferPool > --- > > Key: HBASE-17805 > URL: https://issues.apache.org/jira/browse/HBASE-17805 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17805.v0.patch, HBASE-17805.v1.patch > > > Follow the discussion in HBASE-15525. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17805) We should remove BoundedByteBufferPool because it is replaced by ByteBufferPool
[
https://issues.apache.org/jira/browse/HBASE-17805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932158#comment-15932158
]
Hadoop QA commented on HBASE-17805:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 47s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 109m 3s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 153m 40s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12859485/HBASE-17805.v1.patch |
| JIRA Issue | HBASE-17805 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux bfb8aaf6e739 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7c03a21 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6144/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6144/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> We should remove BoundedByteBufferPool because it
[jira] [Commented] (HBASE-13395) Remove HTableInterface
[
https://issues.apache.org/jira/browse/HBASE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932151#comment-15932151
]
Chia-Ping Tsai commented on HBASE-13395:
[~Jan Hentschel]
There are some comment which reference the "HTableInterface" in the
ConnectionImplementation. Please fix it. Thanks.
{noformat}
// thread executor shared by all HTableInterface instances created
// by this connection
private volatile ExecutorService batchPool = null;
// meta thread executor shared by all HTableInterface instances created
// by this connection
private volatile ExecutorService metaLookupPool = null;
{noformat}
> Remove HTableInterface
> --
>
> Key: HBASE-13395
> URL: https://issues.apache.org/jira/browse/HBASE-13395
> Project: HBase
> Issue Type: Sub-task
> Components: API
>Affects Versions: 2.0.0
>Reporter: Mikhail Antonov
>Assignee: Jan Hentschel
> Fix For: 2.0.0
>
> Attachments: HBASE-13395.master.001.patch
>
>
> This class is marked as deprecated, probably can remove it, and if any
> methods from this specific class are in active use, need to decide what to do
> on callers' side. Should be able to replace with just Table interface usage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15583) Discuss mutable vs immutable HTableDescriptor
[ https://issues.apache.org/jira/browse/HBASE-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932146#comment-15932146 ] stack commented on HBASE-15583: --- [~gliptak] makes a good point that we should be consistent. I'd think that any HTD we give out should be immutable. We do not want to give impression that modifying a HTD means the schema is changed throughout the system. I'd think you'd create a new HTD from the immutable one (I like the idea of naming it immutable rather than modifiable (or modifyable which I think I might have done mistakenly?) > Discuss mutable vs immutable HTableDescriptor > - > > Key: HBASE-15583 > URL: https://issues.apache.org/jira/browse/HBASE-15583 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Gabor Liptak >Priority: Minor > Labels: beginner > Fix For: 2.0.0 > > > From [~enis] in https://issues.apache.org/jira/browse/HBASE-15505: > PS Should UnmodifyableHTableDescriptor be renamed to > UnmodifiableHTableDescriptor? > It should be named ImmutableHTableDescriptor to be consistent with > collections naming. Let's do this as a subtask of the parent jira, not here. > Thinking about it though, why would we return an Immutable HTD in > HTable.getTableDescriptor() versus a mutable HTD in > Admin.getTableDescriptor(). It does not make sense. Should we just get rid of > the Immutable ones? > We also have UnmodifyableHRegionInfo which is not used at the moment it > seems. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-13395) Remove HTableInterface
[ https://issues.apache.org/jira/browse/HBASE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932142#comment-15932142 ] stack commented on HBASE-13395: --- +1 > Remove HTableInterface > -- > > Key: HBASE-13395 > URL: https://issues.apache.org/jira/browse/HBASE-13395 > Project: HBase > Issue Type: Sub-task > Components: API >Affects Versions: 2.0.0 >Reporter: Mikhail Antonov >Assignee: Jan Hentschel > Fix For: 2.0.0 > > Attachments: HBASE-13395.master.001.patch > > > This class is marked as deprecated, probably can remove it, and if any > methods from this specific class are in active use, need to decide what to do > on callers' side. Should be able to replace with just Table interface usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932139#comment-15932139 ] stack commented on HBASE-17765: --- Patch looks good. Where does the merging happen? This addition is useless. Don't bother with it if you making new patch 109* @param updateRegionSize > Reviving the merge possibility in the CompactingMemStore > > > Key: HBASE-17765 > URL: https://issues.apache.org/jira/browse/HBASE-17765 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-17765-V01.patch > > > According to the new performance results presented in the HBASE-16417 we see > that the read latency of the 90th percentile of the BASIC policy is too big > due to the need to traverse through too many segments in the pipeline. In > this JIRA we correct the bug in the merge sizing calculations and allow > pipeline size threshold to be a configurable parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17805) We should remove BoundedByteBufferPool because it is replaced by ByteBufferPool
[ https://issues.apache.org/jira/browse/HBASE-17805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932138#comment-15932138 ] stack commented on HBASE-17805: --- +1 > We should remove BoundedByteBufferPool because it is replaced by > ByteBufferPool > --- > > Key: HBASE-17805 > URL: https://issues.apache.org/jira/browse/HBASE-17805 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17805.v0.patch, HBASE-17805.v1.patch > > > Follow the discussion in HBASE-15525. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932137#comment-15932137 ] stack commented on HBASE-17584: --- Ok. +1 for backport to branch-1 Sir. > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17584.patch, HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932134#comment-15932134 ] Duo Zhang commented on HBASE-17584: --- No. There is a AbstractClientScanner but it is marked as @InterfaceAudience.Private. > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17584.patch, HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932132#comment-15932132 ] stack commented on HBASE-17584: --- is there a default implementation in this case? [~Apache9] > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17584.patch, HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16084) Clean up the stale references in javadoc
[
https://issues.apache.org/jira/browse/HBASE-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932123#comment-15932123
]
Ted Yu commented on HBASE-16084:
Check the new javadoc warnings.
Please look at failed tests as well.
> Clean up the stale references in javadoc
>
>
> Key: HBASE-16084
> URL: https://issues.apache.org/jira/browse/HBASE-16084
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Jan Hentschel
>Priority: Minor
> Attachments: HBASE-16084.master.001.patch
>
>
> From TestHFileOutputFormat2 , e.g.:
> {code}
> * Simple test for {@link CellSortReducer} and {@link HFileOutputFormat2}.
> {code}
> CellSortReducer doesn't exist.
> From TestSerialization.java :
> {code}
>* Create a table of name name with {@link COLUMNS} for
> {code}
> COLUMNS cannot be found.
> This issue is to clean up the stale references in javadoc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17801) Assigning dead region causing FAILED_OPEN permanent RIT that needs manual resolve
[
https://issues.apache.org/jira/browse/HBASE-17801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932120#comment-15932120
]
Allan Yang commented on HBASE-17801:
{quote}
Is there a way for DeleteTableProcedure to notify ServerShutdownHandler that
these regions are being offlined ?
{quote}
I think it is hard, and likely end up with some race conditions
Maybe the master can check whether the table of the region is already disabled
or deleted when assigning a region, if so, then giving up retry.
> Assigning dead region causing FAILED_OPEN permanent RIT that needs manual
> resolve
> --
>
> Key: HBASE-17801
> URL: https://issues.apache.org/jira/browse/HBASE-17801
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 1.1.2
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
>
> In Apache 1.x, there is a Assignment Manager bug when SSH and drop table
> happens at the same time. Here is the sequence:
> (1). The Region Server hosting the target region is dead, SSH (server
> shutdown handler) offlined all regions hosted by the RS:
> {noformat}
> 2017-02-20 20:39:25,022 ERROR
> org.apache.hadoop.hbase.master.MasterRpcServices: Region server
> rs01.foo.com,60020,1486760911253 reported a fatal error:
> ABORTING region server rs01.foo.com,60020,1486760911253:
> regionserver:60020-0x55a076071923f5f,
> quorum=zk01.foo.com:2181,zk02.foo.com:2181,zk3.foo.com:2181, baseZNode=/hbase
> regionserver:60020-0x1234567890abcdf received expired from ZooKeeper, aborting
> Cause:
> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode
> = Session expired
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:613)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:524)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> 2017-02-20 20:42:43,775 INFO
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs
> for rs01.foo.com,60020,1486760911253 before assignment; region count=999
> 2017-02-20 20:43:31,784 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {783a4814b862a6e23a3265a874c3048b state=OPEN, ts=1487568368296,
> server=rs01.foo.com,60020,1486760911253} to {783a4814b862a6e23a3265a874c3048b
> state=OFFLINE, ts=1487648611784, server=rs01.foo.com,60020,1486760911253}
> {noformat}
> (2). Now SSH goes through each region and check whether it should be
> re-assigned (at this time, SSH do check whether a table is disabled/deleted).
> If a region needs to be re-assigned, it would put into a list. Since at
> this time, the troubled region is still on the table that is enabled, it will
> be in the list.
> {noformat}
> 2017-02-20 20:43:31,795 INFO
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Reassigning 999
> region(s) that rs01.foo.com,60020,1486760911253 was carrying (and 0
> regions(s) that were opening on this server)
> {noformat}
> (3). Now, disable and delete table come in and also try to offline the
> region; since the region is already offlined, the deleted table just removes
> the region from meta and in-memory.
> {noformat}
> 2017-02-20 20:43:32,429 INFO org.apache.hadoop.hbase.master.HMaster:
> Client=b_kylin/null disable t1
> 2017-02-20 20:43:34,275 INFO
> org.apache.hadoop.hbase.zookeeper.ZKTableStateManager: Moving table t1 state
> from DISABLING to DISABLED
> 2017-02-20 20:43:34,276 INFO
> org.apache.hadoop.hbase.master.procedure.DisableTableProcedure: Disabled
> table, t1, is completed.
> 2017-02-20 20:43:35,624 INFO org.apache.hadoop.hbase.master.HMaster:
> Client=b_kylin/null delete t1
> 2017-02-20 20:43:36,011 INFO org.apache.hadoop.hbase.MetaTableAccessor:
> Deleted [{ENCODED => fbf9fda1381636aa5b3cd6e3fe0f6c1e, NAME =>
> 't1,,1487568367030.fbf9fda1381636aa5b3cd6e3fe0f6c1e.', STARTKEY => '', ENDKEY
> => '\x00\x01'}, {ENCODED => 783a4814b862a6e23a3265a874c3048b, NAME =>
> 't1,\x00\x01,1487568367030.783a4814b862a6e23a3265a874c3048b.', STARTKEY =>
> '\x00\x01', ENDKEY => ''}]
> {noformat}
> (4). However, SSH calls Assignment Manager to reassign the dead region (note
> that the dead region is in the re-assign list SSH collected and we don't
> re-check again)
> {noformat}
> 2017-02-20 20:43:52,725 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning but not in region
> states: {ENCODED => 783a4814b862a6e23a3265a874c3048b, NAME =>
> 't1,\x00\x01,1487568367030.783a4814b862a6e23a3265a874c3048b.', STARTKEY =>
> '\x00\x01', ENDKEY => ''}
> {noformat}
> (5). In the region
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: 14417-tbl-ext.v21.txt Patch v21 addresses findbugs warnings. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Blocker > Labels: backup > Fix For: 2.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v18.txt, 14417-tbl-ext.v19.txt, > 14417-tbl-ext.v20.txt, 14417-tbl-ext.v21.txt, 14417-tbl-ext.v9.txt, > 14417.v11.txt, 14417.v13.txt, 14417.v1.txt, 14417.v21.txt, 14417.v23.txt, > 14417.v24.txt, 14417.v25.txt, 14417.v2.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board (out of date): > https://reviews.apache.org/r/54258/ > In order not to miss the hfiles which are loaded into region directories in a > situation where postBulkLoadHFile() hook is not called (bulk load being > interrupted), we record hfile names thru preCommitStoreFile() hook. > At time of incremental backup, we check the presence of such hfiles. If they > are present, they become part of the incremental backup image. > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17806) TestRSGroups#testMoveServersAndTables is flaky in master branch
[
https://issues.apache.org/jira/browse/HBASE-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932114#comment-15932114
]
Andrew Purtell commented on HBASE-17806:
This would be due to HBASE-17758 ([RSGROUP] Add shell command to move servers
and tables at the same time)
[~andrewcheng] What do you think? Is there enough information on hand to debug
and fix the flaky test? Or should we revert HBASE-17758 for now?
> TestRSGroups#testMoveServersAndTables is flaky in master branch
> ---
>
> Key: HBASE-17806
> URL: https://issues.apache.org/jira/browse/HBASE-17806
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>
> According to https://builds.apache.org/job/HBASE-Flaky-Tests ,
> TestRSGroups#testMoveServersAndTables is the top flaky test.
> Recent test failure:
> https://builds.apache.org/job/HBASE-Flaky-Tests/14062/testReport/junit/org.apache.hadoop.hbase.rsgroup/TestRSGroups/testMoveServersAndTables/
> {code}
> java.lang.AssertionError: expected:<4> but was:<5>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.rsgroup.TestRSGroupsBase.testMoveServersAndTables(TestRSGroupsBase.java:747)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Reopened] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-17584: --- Reopen for committing the patch to branch-1 as HBASE-17802 is resolved. > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17584.patch, HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16084) Clean up the stale references in javadoc
[
https://issues.apache.org/jira/browse/HBASE-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932104#comment-15932104
]
Hadoop QA commented on HBASE-16084:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 18 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 19s
{color} | {color:red} hbase-prefix-tree in master has 1 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 51s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
40s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 16s
{color} | {color:red} hbase-client generated 36 new + 1 unchanged - 12 fixed =
37 total (was 13) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 10s
{color} | {color:red} hbase-prefix-tree generated 11 new + 0 unchanged - 5
fixed = 11 total (was 5) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 28s
{color} | {color:red} hbase-server generated 3 new + 0 unchanged - 1 fixed = 3
total (was 1) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 49s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 43s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hbase-prefix-tree in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 118m 56s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 185m 49s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.replication.regionserver.TestWALEntryStream |
| |
[jira] [Commented] (HBASE-17802) Add note that minor versions can add methods to Interfaces
[ https://issues.apache.org/jira/browse/HBASE-17802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932102#comment-15932102 ] Duo Zhang commented on HBASE-17802: --- Belated +1. So we can also commit the patch in HBASE-17584 to branch-1. Let me reopen it and prepare patch for branch-1. Thanks. > Add note that minor versions can add methods to Interfaces > -- > > Key: HBASE-17802 > URL: https://issues.apache.org/jira/browse/HBASE-17802 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-17802.master.001.patch > > > Clarify that adding methods to Interfaces in minor releases is allowed and > that we'll always try to do it in a backward compatible way. > Here is discussion from the list: > http://search-hadoop.com/m/HBase/YGbbQfpjp1kozD7?subj=About+adding+methods+to+an+interface+which+is+part+of+our+public+API+in+minor+release -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
[
https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932085#comment-15932085
]
Hadoop QA commented on HBASE-14417:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 5s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 4s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 12s
{color} | {color:red} hbase-server generated 3 new + 0 unchanged - 0 fixed = 3
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 119m 56s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 166m 4s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Using .equals to compare two byte[]'s, (equivalent to ==) in
org.apache.hadoop.hbase.backup.impl.BackupSystemTable.readBulkloadRows(List)
At BackupSystemTable.java:byte[]'s, (equivalent to ==) in
org.apache.hadoop.hbase.backup.impl.BackupSystemTable.readBulkloadRows(List)
At BackupSystemTable.java:[line 416] |
| | Boxing/unboxing to parse a primitive
org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.getTsFromBackupId(String)
At
RestoreTablesClient.java:org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.getTsFromBackupId(String)
At RestoreTablesClient.java:[line 263] |
| | Exception is caught when Exception is not thrown in
org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.restore(HashMap,
TableName[], TableName[], boolean) At RestoreTablesClient.java:is not thrown
in org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.restore(HashMap,
TableName[], TableName[], boolean) At RestoreTablesClient.java:[line 249] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-17802) Add note that minor versions can add methods to Interfaces
[ https://issues.apache.org/jira/browse/HBASE-17802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932082#comment-15932082 ] Hudson commented on HBASE-17802: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2703 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2703/]) HBASE-17802 Add note that minor versions can add methods to Interfaces (stack: rev 7c03a213ffc074c941333677065031a5c2c12d41) * (edit) src/main/asciidoc/_chapters/upgrading.adoc > Add note that minor versions can add methods to Interfaces > -- > > Key: HBASE-17802 > URL: https://issues.apache.org/jira/browse/HBASE-17802 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-17802.master.001.patch > > > Clarify that adding methods to Interfaces in minor releases is allowed and > that we'll always try to do it in a backward compatible way. > Here is discussion from the list: > http://search-hadoop.com/m/HBase/YGbbQfpjp1kozD7?subj=About+adding+methods+to+an+interface+which+is+part+of+our+public+API+in+minor+release -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17806) TestRSGroups#testMoveServersAndTables is flaky in master branch
Ted Yu created HBASE-17806:
--
Summary: TestRSGroups#testMoveServersAndTables is flaky in master
branch
Key: HBASE-17806
URL: https://issues.apache.org/jira/browse/HBASE-17806
Project: HBase
Issue Type: Test
Reporter: Ted Yu
According to https://builds.apache.org/job/HBASE-Flaky-Tests ,
TestRSGroups#testMoveServersAndTables is the top flaky test.
Recent test failure:
https://builds.apache.org/job/HBASE-Flaky-Tests/14062/testReport/junit/org.apache.hadoop.hbase.rsgroup/TestRSGroups/testMoveServersAndTables/
{code}
java.lang.AssertionError: expected:<4> but was:<5>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at
org.apache.hadoop.hbase.rsgroup.TestRSGroupsBase.testMoveServersAndTables(TestRSGroupsBase.java:747)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17805) We should remove BoundedByteBufferPool because it is replaced by ByteBufferPool
[ https://issues.apache.org/jira/browse/HBASE-17805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17805: --- Attachment: HBASE-17805.v1.patch call out the QA for hbase-server > We should remove BoundedByteBufferPool because it is replaced by > ByteBufferPool > --- > > Key: HBASE-17805 > URL: https://issues.apache.org/jira/browse/HBASE-17805 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17805.v0.patch, HBASE-17805.v1.patch > > > Follow the discussion in HBASE-15525. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17805) We should remove BoundedByteBufferPool because it is replaced by ByteBufferPool
[ https://issues.apache.org/jira/browse/HBASE-17805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17805: --- Status: Patch Available (was: Open) > We should remove BoundedByteBufferPool because it is replaced by > ByteBufferPool > --- > > Key: HBASE-17805 > URL: https://issues.apache.org/jira/browse/HBASE-17805 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17805.v0.patch, HBASE-17805.v1.patch > > > Follow the discussion in HBASE-15525. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17805) We should remove BoundedByteBufferPool because it is replaced by ByteBufferPool
[ https://issues.apache.org/jira/browse/HBASE-17805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17805: --- Status: Open (was: Patch Available) > We should remove BoundedByteBufferPool because it is replaced by > ByteBufferPool > --- > > Key: HBASE-17805 > URL: https://issues.apache.org/jira/browse/HBASE-17805 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17805.v0.patch > > > Follow the discussion in HBASE-15525. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17805) We should remove BoundedByteBufferPool because it is replaced by ByteBufferPool
[
https://issues.apache.org/jira/browse/HBASE-17805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932062#comment-15932062
]
Hadoop QA commented on HBASE-17805:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 35s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 47s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 13s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12859483/HBASE-17805.v0.patch |
| JIRA Issue | HBASE-17805 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 90dad305e25a 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7c03a21 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6143/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6143/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> We should remove BoundedByteBufferPool because it is replaced by
> ByteBufferPool
> ---
>
> Key: HBASE-17805
> URL: https://issues.apache.org/jira/browse/HBASE-17805
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix
[jira] [Commented] (HBASE-13395) Remove HTableInterface
[
https://issues.apache.org/jira/browse/HBASE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932060#comment-15932060
]
Hadoop QA commented on HBASE-13395:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 21s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 56s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839628/HBASE-13395.master.001.patch
|
| JIRA Issue | HBASE-13395 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 6d94badd3138 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7c03a21 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6142/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6142/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Remove HTableInterface
> --
>
> Key: HBASE-13395
> URL: https://issues.apache.org/jira/browse/HBASE-13395
> Project: HBase
> Issue Type: Sub-task
> Components: API
>Affects Versions: 2.0.0
>Reporter: Mikhail Antonov
>Assignee: Jan Hentschel
>
[jira] [Updated] (HBASE-17805) We should remove BoundedByteBufferPool because it is replaced by ByteBufferPool
[ https://issues.apache.org/jira/browse/HBASE-17805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17805: --- Status: Patch Available (was: Open) > We should remove BoundedByteBufferPool because it is replaced by > ByteBufferPool > --- > > Key: HBASE-17805 > URL: https://issues.apache.org/jira/browse/HBASE-17805 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17805.v0.patch > > > Follow the discussion in HBASE-15525. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17805) We should remove BoundedByteBufferPool because it is replaced by ByteBufferPool
[ https://issues.apache.org/jira/browse/HBASE-17805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17805: --- Attachment: HBASE-17805.v0.patch > We should remove BoundedByteBufferPool because it is replaced by > ByteBufferPool > --- > > Key: HBASE-17805 > URL: https://issues.apache.org/jira/browse/HBASE-17805 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17805.v0.patch > > > Follow the discussion in HBASE-15525. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15583) Discuss mutable vs immutable HTableDescriptor
[ https://issues.apache.org/jira/browse/HBASE-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932051#comment-15932051 ] Chia-Ping Tsai commented on HBASE-15583: What’s your opinion boss? [~stack] > Discuss mutable vs immutable HTableDescriptor > - > > Key: HBASE-15583 > URL: https://issues.apache.org/jira/browse/HBASE-15583 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Gabor Liptak >Priority: Minor > Labels: beginner > Fix For: 2.0.0 > > > From [~enis] in https://issues.apache.org/jira/browse/HBASE-15505: > PS Should UnmodifyableHTableDescriptor be renamed to > UnmodifiableHTableDescriptor? > It should be named ImmutableHTableDescriptor to be consistent with > collections naming. Let's do this as a subtask of the parent jira, not here. > Thinking about it though, why would we return an Immutable HTD in > HTable.getTableDescriptor() versus a mutable HTD in > Admin.getTableDescriptor(). It does not make sense. Should we just get rid of > the Immutable ones? > We also have UnmodifyableHRegionInfo which is not used at the moment it > seems. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-13395) Remove HTableInterface
[ https://issues.apache.org/jira/browse/HBASE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932049#comment-15932049 ] Chia-Ping Tsai commented on HBASE-13395: bq. It is deprecated in 1.0 I think. So we can remove it. +1 > Remove HTableInterface > -- > > Key: HBASE-13395 > URL: https://issues.apache.org/jira/browse/HBASE-13395 > Project: HBase > Issue Type: Sub-task > Components: API >Affects Versions: 2.0.0 >Reporter: Mikhail Antonov >Assignee: Jan Hentschel > Fix For: 2.0.0 > > Attachments: HBASE-13395.master.001.patch > > > This class is marked as deprecated, probably can remove it, and if any > methods from this specific class are in active use, need to decide what to do > on callers' side. Should be able to replace with just Table interface usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17805) We should remove BoundedByteBufferPool because it is replaced by ByteBufferPool
[ https://issues.apache.org/jira/browse/HBASE-17805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned HBASE-17805: -- Assignee: Chia-Ping Tsai > We should remove BoundedByteBufferPool because it is replaced by > ByteBufferPool > --- > > Key: HBASE-17805 > URL: https://issues.apache.org/jira/browse/HBASE-17805 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0 > > > Follow the discussion in HBASE-15525. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17805) We should remove BoundedByteBufferPool because it is replaced by ByteBufferPool
Chia-Ping Tsai created HBASE-17805: -- Summary: We should remove BoundedByteBufferPool because it is replaced by ByteBufferPool Key: HBASE-17805 URL: https://issues.apache.org/jira/browse/HBASE-17805 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: Chia-Ping Tsai Priority: Minor Fix For: 2.0.0 Follow the discussion in HBASE-15525. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932048#comment-15932048 ] Chia-Ping Tsai commented on HBASE-15525: see HBASE-17805 > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15525_V1.patch, HBASE-15525_V2.patch, > HBASE-15525_V3.patch, HBASE-15525_V4.patch, HBASE-15525_V5.patch, > HBASE-15525_V6.patch, HBASE-15525_WIP.patch, WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16014) Get and Put constructor argument lists are divergent
[
https://issues.apache.org/jira/browse/HBASE-16014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932046#comment-15932046
]
Chia-Ping Tsai commented on HBASE-16014:
Will commit it tomorrow if there is no objections.
> Get and Put constructor argument lists are divergent
>
>
> Key: HBASE-16014
> URL: https://issues.apache.org/jira/browse/HBASE-16014
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Nick Dimiduk
>Assignee: brandboat
> Fix For: 2.0.0
>
> Attachments: HBASE-16014.master.001.patch, HBASE-16014_v0.patch,
> HBASE-16014_v1.patch, HBASE-16014_v2.patch
>
>
> API for construing Get and Put objects for a specific rowkey is quite
> different.
> [Put|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html#constructor_summary]
> supports many more variations for specifying the target rowkey and timestamp
> compared to
> [Get|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html#constructor_summary].
> Notably lacking are {{Get(byte[], int, int)}} and {{Get(ByteBuffer)}}
> variations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932000#comment-15932000 ] stack commented on HBASE-16417: --- Thanks for the nice writeup. bq. . Basic improves the 50th percentile by 7% but the performance of 95th and 99th percentile degrade the performance by 15-30%. Adding merge fixes the above 99th percentile degradation? You say it does at the end of the paragraph. I don't get why reading from multiple indices ups our latency, unless the lookups are serial (i.e. why we need the merge). bq. Note that in sync wal mode all policies have the same number of wal files and the same volume of wal data. The number of wal file is smaller with async wal for all policies (in zipfian and uniform key distribution). When you get the answer to why thIS happens it might explain the number of wal files in eager policy. That is cool that async is faster for you (we were finding otherwise in our tests... but this was without a regionserver context ... maybe we need to look into this). Suggest you file an issue so we can answer the above [~eshcar] It is an interesting question. The charts show some nice, substantial dents in GC activity. Thats sweet. So, enable BASIC+MERGE as default with EAGER for the case where a user knows that they have a lot of duplicate data? > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-16417-benchmarkresults-20161101.pdf, > HBASE-16417-benchmarkresults-20161110.pdf, > HBASE-16417-benchmarkresults-20161123.pdf, > HBASE-16417-benchmarkresults-20161205.pdf, > HBASE-16417-benchmarkresults-20170309.pdf, > HBASE-16417-benchmarkresults-20170317.pdf > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931986#comment-15931986 ] stack commented on HBASE-15525: --- [~anoop.hbase] [~chia7712] New issue? > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15525_V1.patch, HBASE-15525_V2.patch, > HBASE-15525_V3.patch, HBASE-15525_V4.patch, HBASE-15525_V5.patch, > HBASE-15525_V6.patch, HBASE-15525_WIP.patch, WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16014) Get and Put constructor argument lists are divergent
[
https://issues.apache.org/jira/browse/HBASE-16014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931985#comment-15931985
]
stack commented on HBASE-16014:
---
+1
In general, I'd be in favor of less-variants but making the two types align is
good enough reason for this change.
> Get and Put constructor argument lists are divergent
>
>
> Key: HBASE-16014
> URL: https://issues.apache.org/jira/browse/HBASE-16014
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Nick Dimiduk
>Assignee: brandboat
> Fix For: 2.0.0
>
> Attachments: HBASE-16014.master.001.patch, HBASE-16014_v0.patch,
> HBASE-16014_v1.patch, HBASE-16014_v2.patch
>
>
> API for construing Get and Put objects for a specific rowkey is quite
> different.
> [Put|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html#constructor_summary]
> supports many more variations for specifying the target rowkey and timestamp
> compared to
> [Get|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html#constructor_summary].
> Notably lacking are {{Get(byte[], int, int)}} and {{Get(ByteBuffer)}}
> variations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-12579) Move obtainAuthTokenForJob() methods out of User
[
https://issues.apache.org/jira/browse/HBASE-12579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931983#comment-15931983
]
stack commented on HBASE-12579:
---
Yeah, I could not find obtainAuthTokenForJob in the code base either. My guess
though is it just got renamed and the root problem may still exist. Our
[~ghelmling] knows this stuff well... maybe when he has a moment he can make a
ruling here.
> Move obtainAuthTokenForJob() methods out of User
>
>
> Key: HBASE-12579
> URL: https://issues.apache.org/jira/browse/HBASE-12579
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Gary Helmling
>
> The {{User}} class currently contains some utility methods to obtain HBase
> authentication tokens for the given user. However, these methods initiate an
> RPC to the {{TokenProvider}} coprocessor endpoint, an action which should not
> be part of the User class' responsibilities.
> This leads to a couple of problems:
> # The way the methods are currently structured, it is impossible to integrate
> them with normal connection management for the cluster (the TokenUtil class
> constructs its own HTable instance internally).
> # The User class is logically part of the hbase-common module, but uses the
> TokenUtil class (part of hbase-server, though it should probably be moved to
> hbase-client) through reflection, leading to a hidden dependency.
> The {{obtainAuthTokenForJob()}} methods should be deprecated and the process
> of obtaining authentication tokens should be moved to use the normal
> connection lifecycle.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HBASE-17802) Add note that minor versions can add methods to Interfaces
[ https://issues.apache.org/jira/browse/HBASE-17802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-17802. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Release Note: Update our semver section to include a note on our allowing ourselves the right to add methods to an Interface over a minor version as agreed to up on the dev list: "If a Client implements an HBase Interface, a recompile MAY be required upgrading to a newer minor version (See release notes for warning about incompatible changes). All effort will be made to provide a default implementation so this case should not arise." Thanks for the reviews [~apurtell] and [~carp84] (Commit included Andrew's suggestion). > Add note that minor versions can add methods to Interfaces > -- > > Key: HBASE-17802 > URL: https://issues.apache.org/jira/browse/HBASE-17802 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-17802.master.001.patch > > > Clarify that adding methods to Interfaces in minor releases is allowed and > that we'll always try to do it in a backward compatible way. > Here is discussion from the list: > http://search-hadoop.com/m/HBase/YGbbQfpjp1kozD7?subj=About+adding+methods+to+an+interface+which+is+part+of+our+public+API+in+minor+release -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17447) Automatically delete quota when table is deleted
[
https://issues.apache.org/jira/browse/HBASE-17447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931965#comment-15931965
]
Hadoop QA commented on HBASE-17447:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 25s
{color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
26s {color} | {color:green} HBASE-16961 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 9s
{color} | {color:green} HBASE-16961 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 7s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 15s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 26s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 37s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 46s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.5.2. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m 8s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 102m 31s
{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 249m 58s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Fix Version/s: (was: HBASE-7912) 2.0 > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Blocker > Labels: backup > Fix For: 2.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v18.txt, 14417-tbl-ext.v19.txt, > 14417-tbl-ext.v20.txt, 14417-tbl-ext.v9.txt, 14417.v11.txt, 14417.v13.txt, > 14417.v1.txt, 14417.v21.txt, 14417.v23.txt, 14417.v24.txt, 14417.v25.txt, > 14417.v2.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board (out of date): > https://reviews.apache.org/r/54258/ > In order not to miss the hfiles which are loaded into region directories in a > situation where postBulkLoadHFile() hook is not called (bulk load being > interrupted), we record hfile names thru preCommitStoreFile() hook. > At time of incremental backup, we check the presence of such hfiles. If they > are present, they become part of the incremental backup image. > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Status: Patch Available (was: Open) > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Blocker > Labels: backup > Fix For: HBASE-7912 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v18.txt, 14417-tbl-ext.v19.txt, > 14417-tbl-ext.v20.txt, 14417-tbl-ext.v9.txt, 14417.v11.txt, 14417.v13.txt, > 14417.v1.txt, 14417.v21.txt, 14417.v23.txt, 14417.v24.txt, 14417.v25.txt, > 14417.v2.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board (out of date): > https://reviews.apache.org/r/54258/ > In order not to miss the hfiles which are loaded into region directories in a > situation where postBulkLoadHFile() hook is not called (bulk load being > interrupted), we record hfile names thru preCommitStoreFile() hook. > At time of incremental backup, we check the presence of such hfiles. If they > are present, they become part of the incremental backup image. > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-12579) Move obtainAuthTokenForJob() methods out of User
[
https://issues.apache.org/jira/browse/HBASE-12579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931922#comment-15931922
]
Jan Hentschel commented on HBASE-12579:
---
Is this still valid? I couldn't find the {{obtainAuthTokenForJob()}} method in
the current master.
> Move obtainAuthTokenForJob() methods out of User
>
>
> Key: HBASE-12579
> URL: https://issues.apache.org/jira/browse/HBASE-12579
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Gary Helmling
>
> The {{User}} class currently contains some utility methods to obtain HBase
> authentication tokens for the given user. However, these methods initiate an
> RPC to the {{TokenProvider}} coprocessor endpoint, an action which should not
> be part of the User class' responsibilities.
> This leads to a couple of problems:
> # The way the methods are currently structured, it is impossible to integrate
> them with normal connection management for the cluster (the TokenUtil class
> constructs its own HTable instance internally).
> # The User class is logically part of the hbase-common module, but uses the
> TokenUtil class (part of hbase-server, though it should probably be moved to
> hbase-client) through reflection, leading to a hidden dependency.
> The {{obtainAuthTokenForJob()}} methods should be deprecated and the process
> of obtaining authentication tokens should be moved to use the normal
> connection lifecycle.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-16084) Clean up the stale references in javadoc
[
https://issues.apache.org/jira/browse/HBASE-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Hentschel updated HBASE-16084:
--
Status: Patch Available (was: In Progress)
> Clean up the stale references in javadoc
>
>
> Key: HBASE-16084
> URL: https://issues.apache.org/jira/browse/HBASE-16084
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Jan Hentschel
>Priority: Minor
> Attachments: HBASE-16084.master.001.patch
>
>
> From TestHFileOutputFormat2 , e.g.:
> {code}
> * Simple test for {@link CellSortReducer} and {@link HFileOutputFormat2}.
> {code}
> CellSortReducer doesn't exist.
> From TestSerialization.java :
> {code}
>* Create a table of name name with {@link COLUMNS} for
> {code}
> COLUMNS cannot be found.
> This issue is to clean up the stale references in javadoc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-16084) Clean up the stale references in javadoc
[
https://issues.apache.org/jira/browse/HBASE-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Hentschel updated HBASE-16084:
--
Attachment: HBASE-16084.master.001.patch
> Clean up the stale references in javadoc
>
>
> Key: HBASE-16084
> URL: https://issues.apache.org/jira/browse/HBASE-16084
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Jan Hentschel
>Priority: Minor
> Attachments: HBASE-16084.master.001.patch
>
>
> From TestHFileOutputFormat2 , e.g.:
> {code}
> * Simple test for {@link CellSortReducer} and {@link HFileOutputFormat2}.
> {code}
> CellSortReducer doesn't exist.
> From TestSerialization.java :
> {code}
>* Create a table of name name with {@link COLUMNS} for
> {code}
> COLUMNS cannot be found.
> This issue is to clean up the stale references in javadoc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14123: --- Resolution: Fixed Fix Version/s: (was: HBASE-7912) 2.0.0 Status: Resolved (was: Patch Available) Thanks for the patch, Vlad. Thanks for the reviews. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v20.txt, 14123-master.v21.txt, > 14123-master.v24.txt, 14123-master.v25.txt, 14123-master.v27.txt, > 14123-master.v28.txt, 14123-master.v29.full.txt, 14123-master.v2.txt, > 14123-master.v30.txt, 14123-master.v31.txt, 14123-master.v32.txt, > 14123-master.v33.txt, 14123-master.v34.txt, 14123-master.v35.txt, > 14123-master.v36.txt, 14123-master.v37.txt, 14123-master.v38.txt, > 14123.master.v39.patch, 14123-master.v3.txt, 14123.master.v40.patch, > 14123.master.v41.patch, 14123.master.v42.patch, 14123.master.v44.patch, > 14123.master.v45.patch, 14123.master.v46.patch, 14123.master.v48.patch, > 14123.master.v49.patch, 14123.master.v50.patch, 14123.master.v51.patch, > 14123.master.v52.patch, 14123.master.v54.patch, 14123.master.v56.patch, > 14123.master.v57.patch, 14123.master.v58.patch, 14123.master.v59.patch, > 14123-master.v5.txt, 14123.master.v60.patch, 14123.master.v61.patch, > 14123-master.v6.txt, 14123-master.v7.txt, 14123-master.v8.txt, > 14123-master.v9.txt, 14123-v14.txt, Backup-restoreinHBase2.0 (1).pdf, > Backup-restoreinHBase2.0 (3).pdf, Backup-restoreinHBase2.0 (4).pdf, > Backup-restoreinHBase2.0.pdf, HBASE-14123-for-7912-v1.patch, > HBASE-14123-for-7912-v6.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v1.patch, HBASE-14123-v2.patch, > HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch, > HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931856#comment-15931856 ] Anoop Sam John commented on HBASE-15525: We use ByteBufferPool now which is serving fixed sized BBs. +1 for removing BBBP. > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15525_V1.patch, HBASE-15525_V2.patch, > HBASE-15525_V3.patch, HBASE-15525_V4.patch, HBASE-15525_V5.patch, > HBASE-15525_V6.patch, HBASE-15525_WIP.patch, WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931834#comment-15931834 ] Kahlil Oppenheimer edited comment on HBASE-17707 at 3/19/17 5:21 PM: - [~enis] [~tedyu] the new table skew cost function is actually guaranteed to be within the [0-1] range (and this behavior is even unit tested!). The cost function does not dominate over other cost functions because it is out of the [0-1] range. Instead, I debugged the breaking test and found that the issue is that the region replica host cost function can produce very small values when there are a lot of regions. In my testing, I found that for some medium-large cluster sizes, the cost function can produce values as small as 2.6 x 10^(-6). Sadly, this means that even with a weight of 5000 (which is what is set in the test), the "soft" requirement of having no two region replicas hosted on the same machine when it is avoidable is not met because the cost function has too small a contribution (even with this high weight). Instead, my latest patch updates the region replica cost function to give it a minimum value (.1) for any amount of co-hosted replicas. This makes it so that if two regions replicas are placed on the same host, the cost will be at least .1 (whether there are 5 or 1,000,000 regions in the cluster). This better enforces the "soft" constraint as it makes sure that no other cost functions can overpower the region replica host cost function. was (Author: kahliloppenheimer): [~enis] [~tedyu] the new table skew cost function is actually guaranteed to be within the [0-1] range (and this behavior is even unit tested!). The cost function does not dominate over other cost functions because it is out of the [0-1] range. Instead, I debugged the breaking test and found that the issue is that the region replica host cost function can produce very small values when there are a lot of regions. In my testing, I found that for some medium-large cluster sizes, the cost function can produce values as small as 2.6 x 10^(-6). Sadly, this means that even with a weight of 5000 (which is what is set in the test), the "soft" requirement of having no two region replicas hosted on the same machine when it is avoidable is not met because the cost function has too small a contribution (even with this high weight). Instead, my latest patch updates the region replica cost function to give it a minimum value (.1) for any amount of co-hosted replicas. This makes it so that if two regions replicas are placed on the same host, the cost will be at least .1 (whether or not there are 5 or 1,000,000 regions in the cluster). This better enforces the "soft" constraint as it makes sure that no other cost functions can overpower the region replica host cost function. > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, HBASE-17707-11.patch, > HBASE-17707-11.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931834#comment-15931834 ] Kahlil Oppenheimer commented on HBASE-17707: [~enis] [~tedyu] the new table skew cost function is actually guaranteed to be within the [0-1] range (and this behavior is even unit tested!). The cost function does not dominate over other cost functions because it is out of the [0-1] range. Instead, I debugged the breaking test and found that the issue is that the region replica host cost function can produce very small values when there are a lot of regions. In my testing, I found that for some medium-large cluster sizes, the cost function can produce values as small as 2.6 x 10^(-6). Sadly, this means that even with a weight of 5000 (which is what is set in the test), the "soft" requirement of having no two region replicas hosted on the same machine when it is avoidable is not met because the cost function has too small a contribution (even with this high weight). Instead, my latest patch updates the region replica cost function to give it a minimum value (.1) for any amount of co-hosted replicas. This makes it so that if two regions replicas are placed on the same host, the cost will be at least .1 (whether or not there are 5 or 1,000,000 regions in the cluster). This better enforces the "soft" constraint as it makes sure that no other cost functions can overpower the region replica host cost function. > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, HBASE-17707-11.patch, > HBASE-17707-11.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Work started] (HBASE-16084) Clean up the stale references in javadoc
[
https://issues.apache.org/jira/browse/HBASE-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-16084 started by Jan Hentschel.
-
> Clean up the stale references in javadoc
>
>
> Key: HBASE-16084
> URL: https://issues.apache.org/jira/browse/HBASE-16084
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Jan Hentschel
>Priority: Minor
>
> From TestHFileOutputFormat2 , e.g.:
> {code}
> * Simple test for {@link CellSortReducer} and {@link HFileOutputFormat2}.
> {code}
> CellSortReducer doesn't exist.
> From TestSerialization.java :
> {code}
>* Create a table of name name with {@link COLUMNS} for
> {code}
> COLUMNS cannot be found.
> This issue is to clean up the stale references in javadoc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HBASE-16084) Clean up the stale references in javadoc
[
https://issues.apache.org/jira/browse/HBASE-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Hentschel reassigned HBASE-16084:
-
Assignee: Jan Hentschel
> Clean up the stale references in javadoc
>
>
> Key: HBASE-16084
> URL: https://issues.apache.org/jira/browse/HBASE-16084
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Jan Hentschel
>Priority: Minor
>
> From TestHFileOutputFormat2 , e.g.:
> {code}
> * Simple test for {@link CellSortReducer} and {@link HFileOutputFormat2}.
> {code}
> CellSortReducer doesn't exist.
> From TestSerialization.java :
> {code}
>* Create a table of name name with {@link COLUMNS} for
> {code}
> COLUMNS cannot be found.
> This issue is to clean up the stale references in javadoc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17447) Automatically delete quota when table is deleted
[ https://issues.apache.org/jira/browse/HBASE-17447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-17447: --- Attachment: HBASE-17447.002.HBASE-16961.patch .002 Fixes trailing whitespace.. > Automatically delete quota when table is deleted > > > Key: HBASE-17447 > URL: https://issues.apache.org/jira/browse/HBASE-17447 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17447.001.HBASE-16961.patch, > HBASE-17447.002.HBASE-16961.patch > > > If a table has a space quota defined on it, we can delete that quota when the > table is deleted. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17803) PE always re-creates table when we specify the split policy
[
https://issues.apache.org/jira/browse/HBASE-17803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931763#comment-15931763
]
Hudson commented on HBASE-17803:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2701 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2701/])
HBASE-17803 PE always re-creates table when we specify the split policy
(chia7712: rev 23abc90068f0ea75f09c3eecf6ef758f1aee9219)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
HBASE-17803 Addendum fix NPE (chia7712: rev
261aa9445c3c52e09c10d06168a77d11d0c9b4b4)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
> PE always re-creates table when we specify the split policy
> ---
>
> Key: HBASE-17803
> URL: https://issues.apache.org/jira/browse/HBASE-17803
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17803.branch-1.v0.patch, HBASE-17803.v0.patch
>
>
> I find this bug when i run the tests for HBASE-17623
> The critical code is shown below.
> {noformat}
> if ((exists && opts.presplitRegions != DEFAULT_OPTS.presplitRegions)
> || (!isReadCmd && desc != null && desc.getRegionSplitPolicyClassName()
> != opts.splitPolicy)
> || (!isReadCmd && desc != null && desc.getRegionReplication() !=
> opts.replicas)) {
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17803) PE always re-creates table when we specify the split policy
[
https://issues.apache.org/jira/browse/HBASE-17803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931710#comment-15931710
]
Hudson commented on HBASE-17803:
FAILURE: Integrated in Jenkins build HBase-1.4 #676 (See
[https://builds.apache.org/job/HBase-1.4/676/])
HBASE-17803 PE always re-creates table when we specify the split policy
(chia7712: rev 02d9bf0c57b447ab0f4fa93a3858bd27c43e0cf9)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
> PE always re-creates table when we specify the split policy
> ---
>
> Key: HBASE-17803
> URL: https://issues.apache.org/jira/browse/HBASE-17803
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17803.branch-1.v0.patch, HBASE-17803.v0.patch
>
>
> I find this bug when i run the tests for HBASE-17623
> The critical code is shown below.
> {noformat}
> if ((exists && opts.presplitRegions != DEFAULT_OPTS.presplitRegions)
> || (!isReadCmd && desc != null && desc.getRegionSplitPolicyClassName()
> != opts.splitPolicy)
> || (!isReadCmd && desc != null && desc.getRegionReplication() !=
> opts.replicas)) {
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931696#comment-15931696 ] Chia-Ping Tsai commented on HBASE-15525: Should we remove the BoundedByteBufferPool ? It is useless for 2.0. > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15525_V1.patch, HBASE-15525_V2.patch, > HBASE-15525_V3.patch, HBASE-15525_V4.patch, HBASE-15525_V5.patch, > HBASE-15525_V6.patch, HBASE-15525_WIP.patch, WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-15583) Discuss mutable vs immutable HTableDescriptor
[ https://issues.apache.org/jira/browse/HBASE-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931688#comment-15931688 ] Chia-Ping Tsai edited comment on HBASE-15583 at 3/19/17 1:38 PM: - The immutable HTableDescriptor (UnmodifyableHTableDescriptor) is broken. It should override the setValue(final Bytes key, final Bytes value) and remove(Bytes key), but it doesn't. We should fix it, or just get rid of it. was (Author: chia7712): The immutable HTableDescriptor (UnmodifyableHTableDescriptor) is broken. It should override the setValue(final Bytes key, final Bytes value), remove(Bytes key), but it doesn't. We should fix it, or just get rid of it. > Discuss mutable vs immutable HTableDescriptor > - > > Key: HBASE-15583 > URL: https://issues.apache.org/jira/browse/HBASE-15583 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Gabor Liptak >Priority: Minor > Labels: beginner > Fix For: 2.0.0 > > > From [~enis] in https://issues.apache.org/jira/browse/HBASE-15505: > PS Should UnmodifyableHTableDescriptor be renamed to > UnmodifiableHTableDescriptor? > It should be named ImmutableHTableDescriptor to be consistent with > collections naming. Let's do this as a subtask of the parent jira, not here. > Thinking about it though, why would we return an Immutable HTD in > HTable.getTableDescriptor() versus a mutable HTD in > Admin.getTableDescriptor(). It does not make sense. Should we just get rid of > the Immutable ones? > We also have UnmodifyableHRegionInfo which is not used at the moment it > seems. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15583) Discuss mutable vs immutable HTableDescriptor
[ https://issues.apache.org/jira/browse/HBASE-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931688#comment-15931688 ] Chia-Ping Tsai commented on HBASE-15583: The immutable HTableDescriptor (UnmodifyableHTableDescriptor) is broken. It should override the setValue(final Bytes key, final Bytes value), remove(Bytes key), but it doesn't. We should fix it, or just get rid of it. > Discuss mutable vs immutable HTableDescriptor > - > > Key: HBASE-15583 > URL: https://issues.apache.org/jira/browse/HBASE-15583 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Gabor Liptak >Priority: Minor > Labels: beginner > Fix For: 2.0.0 > > > From [~enis] in https://issues.apache.org/jira/browse/HBASE-15505: > PS Should UnmodifyableHTableDescriptor be renamed to > UnmodifiableHTableDescriptor? > It should be named ImmutableHTableDescriptor to be consistent with > collections naming. Let's do this as a subtask of the parent jira, not here. > Thinking about it though, why would we return an Immutable HTD in > HTable.getTableDescriptor() versus a mutable HTD in > Admin.getTableDescriptor(). It does not make sense. Should we just get rid of > the Immutable ones? > We also have UnmodifyableHRegionInfo which is not used at the moment it > seems. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17803) PE always re-creates table when we specify the split policy
[
https://issues.apache.org/jira/browse/HBASE-17803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931649#comment-15931649
]
Chia-Ping Tsai commented on HBASE-17803:
Thanks for the reviews. [~jinghe]
> PE always re-creates table when we specify the split policy
> ---
>
> Key: HBASE-17803
> URL: https://issues.apache.org/jira/browse/HBASE-17803
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17803.branch-1.v0.patch, HBASE-17803.v0.patch
>
>
> I find this bug when i run the tests for HBASE-17623
> The critical code is shown below.
> {noformat}
> if ((exists && opts.presplitRegions != DEFAULT_OPTS.presplitRegions)
> || (!isReadCmd && desc != null && desc.getRegionSplitPolicyClassName()
> != opts.splitPolicy)
> || (!isReadCmd && desc != null && desc.getRegionReplication() !=
> opts.replicas)) {
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17803) PE always re-creates table when we specify the split policy
[
https://issues.apache.org/jira/browse/HBASE-17803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-17803:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> PE always re-creates table when we specify the split policy
> ---
>
> Key: HBASE-17803
> URL: https://issues.apache.org/jira/browse/HBASE-17803
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17803.branch-1.v0.patch, HBASE-17803.v0.patch
>
>
> I find this bug when i run the tests for HBASE-17623
> The critical code is shown below.
> {noformat}
> if ((exists && opts.presplitRegions != DEFAULT_OPTS.presplitRegions)
> || (!isReadCmd && desc != null && desc.getRegionSplitPolicyClassName()
> != opts.splitPolicy)
> || (!isReadCmd && desc != null && desc.getRegionReplication() !=
> opts.replicas)) {
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16014) Get and Put constructor argument lists are divergent
[
https://issues.apache.org/jira/browse/HBASE-16014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931640#comment-15931640
]
Chia-Ping Tsai commented on HBASE-16014:
Ping [~syuanjiang] [~ndimiduk]
> Get and Put constructor argument lists are divergent
>
>
> Key: HBASE-16014
> URL: https://issues.apache.org/jira/browse/HBASE-16014
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Nick Dimiduk
>Assignee: brandboat
> Fix For: 2.0.0
>
> Attachments: HBASE-16014.master.001.patch, HBASE-16014_v0.patch,
> HBASE-16014_v1.patch, HBASE-16014_v2.patch
>
>
> API for construing Get and Put objects for a specific rowkey is quite
> different.
> [Put|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html#constructor_summary]
> supports many more variations for specifying the target rowkey and timestamp
> compared to
> [Get|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html#constructor_summary].
> Notably lacking are {{Get(byte[], int, int)}} and {{Get(ByteBuffer)}}
> variations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931616#comment-15931616 ] Edward Bortnikov commented on HBASE-16417: -- [~eshcar], thanks for the thorough report, great stuff. Question to all - do these results suggest that we change the default to BASIC+MERGE? Seems that this method does not have any material overhead, even under the uniform workload. If the answer is "yes", we could take one of two ways: (1) say that BASIC+MERGE is a new BASIC (my favorite :)), or (2) introduce a new compaction level (MODERATE?). Let's converge fast - then we can update the documentation and finalize the code. This work notwithstanding, it is still appealing to come up with an automatic policy to tune handsfree (which was the original intent behind this JIRA). With the 2.0 release on our heels, we might not be able to make it until then. But let's have all the building blocks in place, at least (smile). > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-16417-benchmarkresults-20161101.pdf, > HBASE-16417-benchmarkresults-20161110.pdf, > HBASE-16417-benchmarkresults-20161123.pdf, > HBASE-16417-benchmarkresults-20161205.pdf, > HBASE-16417-benchmarkresults-20170309.pdf, > HBASE-16417-benchmarkresults-20170317.pdf > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
