[jira] [Updated] (HBASE-17338) Treat Cell data size under global memstore heap size only when that Cell can not be copied to MSLAB
[ https://issues.apache.org/jira/browse/HBASE-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-17338: --- Status: Patch Available (was: In Progress) > Treat Cell data size under global memstore heap size only when that Cell can > not be copied to MSLAB > --- > > Key: HBASE-17338 > URL: https://issues.apache.org/jira/browse/HBASE-17338 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-17338.patch > > > We have only data size and heap overhead being tracked globally. Off heap > memstore works with off heap backed MSLAB pool. But a cell, when added to > memstore, not always getting copied to MSLAB. Append/Increment ops doing an > upsert, dont use MSLAB. Also based on the Cell size, we sometimes avoid > MSLAB copy. But now we track these cell data size also under the global > memstore data size which indicated off heap size in case of off heap > memstore. For global checks for flushes (against lower/upper watermark > levels), we check this size against max off heap memstore size. We do check > heap overhead against global heap memstore size (Defaults to 40% of xmx) But > for such cells the data size also should be accounted under the heap overhead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17338) Treat Cell data size under global memstore heap size only when that Cell can not be copied to MSLAB
[ https://issues.apache.org/jira/browse/HBASE-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-17338: --- Attachment: HBASE-17338.patch > Treat Cell data size under global memstore heap size only when that Cell can > not be copied to MSLAB > --- > > Key: HBASE-17338 > URL: https://issues.apache.org/jira/browse/HBASE-17338 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-17338.patch > > > We have only data size and heap overhead being tracked globally. Off heap > memstore works with off heap backed MSLAB pool. But a cell, when added to > memstore, not always getting copied to MSLAB. Append/Increment ops doing an > upsert, dont use MSLAB. Also based on the Cell size, we sometimes avoid > MSLAB copy. But now we track these cell data size also under the global > memstore data size which indicated off heap size in case of off heap > memstore. For global checks for flushes (against lower/upper watermark > levels), we check this size against max off heap memstore size. We do check > heap overhead against global heap memstore size (Defaults to 40% of xmx) But > for such cells the data size also should be accounted under the heap overhead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HBASE-17338) Treat Cell data size under global memstore heap size only when that Cell can not be copied to MSLAB
[ https://issues.apache.org/jira/browse/HBASE-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-17338 started by Anoop Sam John. -- > Treat Cell data size under global memstore heap size only when that Cell can > not be copied to MSLAB > --- > > Key: HBASE-17338 > URL: https://issues.apache.org/jira/browse/HBASE-17338 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-17338.patch > > > We have only data size and heap overhead being tracked globally. Off heap > memstore works with off heap backed MSLAB pool. But a cell, when added to > memstore, not always getting copied to MSLAB. Append/Increment ops doing an > upsert, dont use MSLAB. Also based on the Cell size, we sometimes avoid > MSLAB copy. But now we track these cell data size also under the global > memstore data size which indicated off heap size in case of off heap > memstore. For global checks for flushes (against lower/upper watermark > levels), we check this size against max off heap memstore size. We do check > heap overhead against global heap memstore size (Defaults to 40% of xmx) But > for such cells the data size also should be accounted under the heap overhead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17355) Create a simplifed version of flush scanner
[
https://issues.apache.org/jira/browse/HBASE-17355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766392#comment-15766392
]
Hadoop QA commented on HBASE-17355:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 52s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 40s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 44s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestKeepDeletes |
| | hadoop.hbase.regionserver.TestMinVersions |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844187/HBASE-17354.patch |
| JIRA Issue | HBASE-17355 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 7391cd57a860 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e1f4aae |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5011/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5011/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5011/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5011/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was
[jira] [Commented] (HBASE-17107) FN info should be cleaned up on region/table cleanup
[ https://issues.apache.org/jira/browse/HBASE-17107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766387#comment-15766387 ] Hudson commented on HBASE-17107: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2170 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2170/]) HBASE-17107: FN info should be cleaned up on region/table cleanup (toffer: rev 3599716dffe33690a8f5446a2c22ef930d220402) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableFavoredNodes.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/favored/FavoredNodesManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java > FN info should be cleaned up on region/table cleanup > > > Key: HBASE-17107 > URL: https://issues.apache.org/jira/browse/HBASE-17107 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17107.master.001.patch, HBASE_17107.draft.patch > > > FN info should be cleaned up when table is deleted and when regions are GCed > (i.e. CatalogJanitor). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766386#comment-15766386
]
Hudson commented on HBASE-17328:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2170 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2170/])
HBASE-17328 Properly dispose of looped replication peers (apurtell: rev
f8474c8d4d3e722aa0129e085f6a5287c5e2be89)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24, 1.1.9
>
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch,
> HBASE-17328-master.v2.patch, HBASE-17328.0.98.v4.patch,
> HBASE-17328.branch-1.1.v2.patch, HBASE-17328.branch-1.1.v3.patch,
> HBASE-17328.branch-1.1.v4.patch, HBASE-17328.master.v4.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17314) Limit total buffered size for all replication sources
[ https://issues.apache.org/jira/browse/HBASE-17314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766388#comment-15766388 ] Hudson commented on HBASE-17314: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2170 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2170/]) HBASE-17314 Limit total buffered size for all replication sources (yangzhe1991: rev 3826e639672eea11d73da333e6c15f6b7c23a46c) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestGlobalThrottler.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > Limit total buffered size for all replication sources > - > > Key: HBASE-17314 > URL: https://issues.apache.org/jira/browse/HBASE-17314 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17314.branch-1.v01.patch, HBASE-17314.v01.patch, > HBASE-17314.v02.patch, HBASE-17314.v03.patch, HBASE-17314.v04.patch > > > If we have many peers or some servers have many recovered queues, we will > hold many entries in memory which will increase the pressure of GC, even > maybe OOM because we will read entries for 64MB to buffer in default for one > source. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15924) Enhance hbase services autorestart capability to hbase-daemon.sh
[ https://issues.apache.org/jira/browse/HBASE-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766385#comment-15766385 ] Hudson commented on HBASE-15924: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2170 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2170/]) HBASE-15924 Enhance hbase services autorestart capability to (apurtell: rev 06b67a632c47dd44150c8149a0120a6e1308b6d0) * (edit) bin/local-master-backup.sh * (edit) bin/regionservers.sh * (edit) bin/hbase-config.sh * (edit) bin/rolling-restart.sh * (edit) bin/hbase-daemon.sh * (edit) bin/hbase-daemons.sh * (edit) bin/local-regionservers.sh * (edit) bin/start-hbase.sh > Enhance hbase services autorestart capability to hbase-daemon.sh > - > > Key: HBASE-15924 > URL: https://issues.apache.org/jira/browse/HBASE-15924 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.19 >Reporter: Loknath Priyatham Teja Singamsetty >Assignee: Loknath Priyatham Teja Singamsetty > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15924.master.0001.patch, > HBASE-15924.master.0002.patch, HBASE-15924.master.0003.patch, > HBASE-15924.master.0004.patch > > > As part of HBASE-5939, the autorestart for hbase services has been added to > deal with scenarios where hbase services (master/regionserver/master-backup) > gets killed or goes down leading to unplanned outages. The changes were made > to hbase-daemon.sh to support autorestart option. > However, the autorestart implementation doesn't work in standalone mode and > other than that have few gaps with the implementation as per release notes of > HBASE-5939. Here is an attempt to re-design and fix the functionality > considering all possible usecases with hbase service operations. > Release Notes of HBASE-5939: > -- > When launched with autorestart, HBase processes will automatically restart if > they are not properly terminated, either by a "stop" command or by a cluster > stop. To ensure that it does not overload the system when the server itself > is corrupted and the process cannot be restarted, the server sleeps for 5 > minutes before restarting if it was already started 5 minutes ago previously. > To use it, launch the process with "bin/start-hbase autorestart". This option > is not fully compatible with the existing "restart" command: if you ask for a > restart on a server launched with autorestart, the server will restart but > the next server instance won't be automatically restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17353) Cache famLen in OffheapKeyValue
[
https://issues.apache.org/jira/browse/HBASE-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766364#comment-15766364
]
Hadoop QA commented on HBASE-17353:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 41s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 0s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844190/HBASE-17353.patch |
| JIRA Issue | HBASE-17353 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 69e8bc239f74 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e1f4aae |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5010/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5010/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Cache famLen in OffheapKeyValue
> ---
>
> Key: HBASE-17353
> URL: https://issues.apache.org/jira/browse/HBASE-17353
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>
[jira] [Commented] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766356#comment-15766356
]
Hudson commented on HBASE-17328:
SUCCESS: Integrated in Jenkins build HBase-1.1-JDK8 #1912 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1912/])
HBASE-17328 Properly dispose of looped replication peers (apurtell: rev
d3ffdf6e90a16cf250375469b2f04717b942ad94)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24, 1.1.9
>
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch,
> HBASE-17328-master.v2.patch, HBASE-17328.0.98.v4.patch,
> HBASE-17328.branch-1.1.v2.patch, HBASE-17328.branch-1.1.v3.patch,
> HBASE-17328.branch-1.1.v4.patch, HBASE-17328.master.v4.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17353) Cache famLen in OffheapKeyValue
[ https://issues.apache.org/jira/browse/HBASE-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17353: --- Attachment: HBASE-17353.patch Attaching patch for reference. Will wait for discussion and if really needed will take this. But in repeated runs now I can see that this OffheapKV#getFamilyLength() is coming at the top of the profiler tool. > Cache famLen in OffheapKeyValue > --- > > Key: HBASE-17353 > URL: https://issues.apache.org/jira/browse/HBASE-17353 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17353.patch, OffheapKV famLen cost.png > > > WE need to discuss here. Already we had a TODO here. But this comes again > after offheap memstore is committed in trunk. > Attaching a screenshot to show the impact. True that these changes won't have > a direct impact on the final perf but at micro level they would have. > In case of KeyValue it is just retrieving a byte from the byte[] (o(1) > access). > But here we need to access the memory to retrive that one byte though it is > Unsafe based. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17353) Cache famLen in OffheapKeyValue
[ https://issues.apache.org/jira/browse/HBASE-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17353: --- Status: Patch Available (was: Open) > Cache famLen in OffheapKeyValue > --- > > Key: HBASE-17353 > URL: https://issues.apache.org/jira/browse/HBASE-17353 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17353.patch, OffheapKV famLen cost.png > > > WE need to discuss here. Already we had a TODO here. But this comes again > after offheap memstore is committed in trunk. > Attaching a screenshot to show the impact. True that these changes won't have > a direct impact on the final perf but at micro level they would have. > In case of KeyValue it is just retrieving a byte from the byte[] (o(1) > access). > But here we need to access the memory to retrive that one byte though it is > Unsafe based. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17355) Create a simplifed version of flush scanner
[ https://issues.apache.org/jira/browse/HBASE-17355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766302#comment-15766302 ] ramkrishna.s.vasudevan commented on HBASE-17355: Note that in the attached screen shot, the PE tool was run with 50 cols per row. > Create a simplifed version of flush scanner > --- > > Key: HBASE-17355 > URL: https://issues.apache.org/jira/browse/HBASE-17355 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17354.patch, after patch.png, before patch.png > > > Currently we use StoreScanner for performing the flushes which actuallly goes > row by row. Probably that is not needed and we could always go ahead with a > simple loop in collecting the cells and writing them to the file. Inside > write path we have the required sanity check so it is not needed that the > store scanner does a sanity check. > Also the limit that could be retrieved in one next() call could be equivalent > to the block size configured as we do for compaction. > Are there any filters that we want to do (i mean any version check or > deletion) that we need to check in flush? If so then this simplified version > will not work. I may be missing something but if so we need to see what are > those and add it here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17355) Create a simplifed version of flush scanner
[ https://issues.apache.org/jira/browse/HBASE-17355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17355: --- Status: Patch Available (was: Open) > Create a simplifed version of flush scanner > --- > > Key: HBASE-17355 > URL: https://issues.apache.org/jira/browse/HBASE-17355 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17354.patch, after patch.png, before patch.png > > > Currently we use StoreScanner for performing the flushes which actuallly goes > row by row. Probably that is not needed and we could always go ahead with a > simple loop in collecting the cells and writing them to the file. Inside > write path we have the required sanity check so it is not needed that the > store scanner does a sanity check. > Also the limit that could be retrieved in one next() call could be equivalent > to the block size configured as we do for compaction. > Are there any filters that we want to do (i mean any version check or > deletion) that we need to check in flush? If so then this simplified version > will not work. I may be missing something but if so we need to see what are > those and add it here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17355) Create a simplifed version of flush scanner
[ https://issues.apache.org/jira/browse/HBASE-17355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17355: --- Attachment: after patch.png > Create a simplifed version of flush scanner > --- > > Key: HBASE-17355 > URL: https://issues.apache.org/jira/browse/HBASE-17355 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17354.patch, after patch.png, before patch.png > > > Currently we use StoreScanner for performing the flushes which actuallly goes > row by row. Probably that is not needed and we could always go ahead with a > simple loop in collecting the cells and writing them to the file. Inside > write path we have the required sanity check so it is not needed that the > store scanner does a sanity check. > Also the limit that could be retrieved in one next() call could be equivalent > to the block size configured as we do for compaction. > Are there any filters that we want to do (i mean any version check or > deletion) that we need to check in flush? If so then this simplified version > will not work. I may be missing something but if so we need to see what are > those and add it here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17355) Create a simplifed version of flush scanner
[ https://issues.apache.org/jira/browse/HBASE-17355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17355: --- Attachment: HBASE-17354.patch > Create a simplifed version of flush scanner > --- > > Key: HBASE-17355 > URL: https://issues.apache.org/jira/browse/HBASE-17355 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17354.patch, after patch.png, before patch.png > > > Currently we use StoreScanner for performing the flushes which actuallly goes > row by row. Probably that is not needed and we could always go ahead with a > simple loop in collecting the cells and writing them to the file. Inside > write path we have the required sanity check so it is not needed that the > store scanner does a sanity check. > Also the limit that could be retrieved in one next() call could be equivalent > to the block size configured as we do for compaction. > Are there any filters that we want to do (i mean any version check or > deletion) that we need to check in flush? If so then this simplified version > will not work. I may be missing something but if so we need to see what are > those and add it here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17355) Create a simplifed version of flush scanner
[ https://issues.apache.org/jira/browse/HBASE-17355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17355: --- Attachment: before patch.png > Create a simplifed version of flush scanner > --- > > Key: HBASE-17355 > URL: https://issues.apache.org/jira/browse/HBASE-17355 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: before patch.png > > > Currently we use StoreScanner for performing the flushes which actuallly goes > row by row. Probably that is not needed and we could always go ahead with a > simple loop in collecting the cells and writing them to the file. Inside > write path we have the required sanity check so it is not needed that the > store scanner does a sanity check. > Also the limit that could be retrieved in one next() call could be equivalent > to the block size configured as we do for compaction. > Are there any filters that we want to do (i mean any version check or > deletion) that we need to check in flush? If so then this simplified version > will not work. I may be missing something but if so we need to see what are > those and add it here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17355) Create a simplifed version of flush scanner
ramkrishna.s.vasudevan created HBASE-17355: -- Summary: Create a simplifed version of flush scanner Key: HBASE-17355 URL: https://issues.apache.org/jira/browse/HBASE-17355 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Currently we use StoreScanner for performing the flushes which actuallly goes row by row. Probably that is not needed and we could always go ahead with a simple loop in collecting the cells and writing them to the file. Inside write path we have the required sanity check so it is not needed that the store scanner does a sanity check. Also the limit that could be retrieved in one next() call could be equivalent to the block size configured as we do for compaction. Are there any filters that we want to do (i mean any version check or deletion) that we need to check in flush? If so then this simplified version will not work. I may be missing something but if so we need to see what are those and add it here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17354) Improve and optimize flush/compaction process
ramkrishna.s.vasudevan created HBASE-17354: -- Summary: Improve and optimize flush/compaction process Key: HBASE-17354 URL: https://issues.apache.org/jira/browse/HBASE-17354 Project: HBase Issue Type: Task Affects Versions: 2.0.0 Reporter: ramkrishna.s.vasudevan Fix For: 2.0.0 This could act as a parent JIRA for optimizing flush/compaction in terms of speed,algo and GC generated by these operations. Even if we have offheap memstore there are some places where flushes/compaction impacts GC and also there are some low hanging fruits which could be tweaked to overall improve flush/compaction performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766241#comment-15766241
]
Hudson commented on HBASE-17328:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #81 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/81/])
HBASE-17328 Properly dispose of looped replication peers (apurtell: rev
f276edfadbc5b598bdd7412809e76e8077207603)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24, 1.1.9
>
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch,
> HBASE-17328-master.v2.patch, HBASE-17328.0.98.v4.patch,
> HBASE-17328.branch-1.1.v2.patch, HBASE-17328.branch-1.1.v3.patch,
> HBASE-17328.branch-1.1.v4.patch, HBASE-17328.master.v4.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15924) Enhance hbase services autorestart capability to hbase-daemon.sh
[ https://issues.apache.org/jira/browse/HBASE-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766236#comment-15766236 ] Hudson commented on HBASE-15924: SUCCESS: Integrated in Jenkins build HBase-1.4 #574 (See [https://builds.apache.org/job/HBase-1.4/574/]) HBASE-15924 Enhance hbase services autorestart capability to (apurtell: rev 33002bd8e3df7a29e662fb369f62d18f73d0252e) * (edit) bin/local-master-backup.sh * (edit) bin/hbase-daemons.sh * (edit) bin/rolling-restart.sh * (edit) bin/local-regionservers.sh * (edit) bin/start-hbase.sh * (edit) bin/hbase-config.sh * (edit) bin/hbase-daemon.sh * (edit) bin/regionservers.sh > Enhance hbase services autorestart capability to hbase-daemon.sh > - > > Key: HBASE-15924 > URL: https://issues.apache.org/jira/browse/HBASE-15924 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.19 >Reporter: Loknath Priyatham Teja Singamsetty >Assignee: Loknath Priyatham Teja Singamsetty > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15924.master.0001.patch, > HBASE-15924.master.0002.patch, HBASE-15924.master.0003.patch, > HBASE-15924.master.0004.patch > > > As part of HBASE-5939, the autorestart for hbase services has been added to > deal with scenarios where hbase services (master/regionserver/master-backup) > gets killed or goes down leading to unplanned outages. The changes were made > to hbase-daemon.sh to support autorestart option. > However, the autorestart implementation doesn't work in standalone mode and > other than that have few gaps with the implementation as per release notes of > HBASE-5939. Here is an attempt to re-design and fix the functionality > considering all possible usecases with hbase service operations. > Release Notes of HBASE-5939: > -- > When launched with autorestart, HBase processes will automatically restart if > they are not properly terminated, either by a "stop" command or by a cluster > stop. To ensure that it does not overload the system when the server itself > is corrupted and the process cannot be restarted, the server sleeps for 5 > minutes before restarting if it was already started 5 minutes ago previously. > To use it, launch the process with "bin/start-hbase autorestart". This option > is not fully compatible with the existing "restart" command: if you ask for a > restart on a server launched with autorestart, the server will restart but > the next server instance won't be automatically restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766237#comment-15766237
]
Hudson commented on HBASE-17328:
SUCCESS: Integrated in Jenkins build HBase-1.4 #574 (See
[https://builds.apache.org/job/HBase-1.4/574/])
HBASE-17328 Properly dispose of looped replication peers (apurtell: rev
e79afbf0cbca95ed4dad67ef83d9755c86629a85)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24, 1.1.9
>
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch,
> HBASE-17328-master.v2.patch, HBASE-17328.0.98.v4.patch,
> HBASE-17328.branch-1.1.v2.patch, HBASE-17328.branch-1.1.v3.patch,
> HBASE-17328.branch-1.1.v4.patch, HBASE-17328.master.v4.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17334) Add locate row before/after support for AsyncRegionLocator
[ https://issues.apache.org/jira/browse/HBASE-17334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766230#comment-15766230 ] Duo Zhang commented on HBASE-17334: --- Any concerns? [~carp84] [~stack]. It is required for implementing HBASE-17320. Thanks. > Add locate row before/after support for AsyncRegionLocator > -- > > Key: HBASE-17334 > URL: https://issues.apache.org/jira/browse/HBASE-17334 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17334-v1.patch, HBASE-17334.patch > > > Now we only have a getPreviousRegionLocation method which is only used for > reverse scan, and it is not perfect as it can not deal with region merge. As > we want to add inclusive/exclusive support for start row and end row of a > scan, we need to implement general locate to row before/after method for > AsyncRegionLocator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11392) add/remove peer requests should be routed through master
[ https://issues.apache.org/jira/browse/HBASE-11392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-11392: --- Resolution: Fixed Status: Resolved (was: Patch Available) > add/remove peer requests should be routed through master > > > Key: HBASE-11392 > URL: https://issues.apache.org/jira/browse/HBASE-11392 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Guanghao Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-11392-v1.patch, HBASE-11392-v2.patch, > HBASE-11392-v3.patch, HBASE-11392-v4.patch, HBASE-11392-v5.patch, > HBASE-11392-v6.patch > > > ReplicationAdmin directly operates over the zookeeper data for replication > setup. We should move these operations to be routed through master for two > reasons: > - Replication implementation details are exposed to client. We should move > most of the replication related classes to hbase-server package. > - Routing the requests through master is the standard practice for all other > operations. It allows for decoupling implementation details from the client > and code. > Review board: https://reviews.apache.org/r/54730/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766191#comment-15766191
]
Hudson commented on HBASE-17328:
SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #83 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/83/])
HBASE-17328 Properly dispose of looped replication peers (apurtell: rev
7b3187c1a02eb875b2ba2daa49d43738f4dce8f8)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24, 1.1.9
>
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch,
> HBASE-17328-master.v2.patch, HBASE-17328.0.98.v4.patch,
> HBASE-17328.branch-1.1.v2.patch, HBASE-17328.branch-1.1.v3.patch,
> HBASE-17328.branch-1.1.v4.patch, HBASE-17328.master.v4.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11392) add/remove peer requests should be routed through master
[ https://issues.apache.org/jira/browse/HBASE-11392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766194#comment-15766194 ] Guanghao Zhang commented on HBASE-11392: Pushed to master branch.Thanks all for reviewing. > add/remove peer requests should be routed through master > > > Key: HBASE-11392 > URL: https://issues.apache.org/jira/browse/HBASE-11392 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Guanghao Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-11392-v1.patch, HBASE-11392-v2.patch, > HBASE-11392-v3.patch, HBASE-11392-v4.patch, HBASE-11392-v5.patch, > HBASE-11392-v6.patch > > > ReplicationAdmin directly operates over the zookeeper data for replication > setup. We should move these operations to be routed through master for two > reasons: > - Replication implementation details are exposed to client. We should move > most of the replication related classes to hbase-server package. > - Routing the requests through master is the standard practice for all other > operations. It allows for decoupling implementation details from the client > and code. > Review board: https://reviews.apache.org/r/54730/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17351) Unable to generate tar ball for master branch
[
https://issues.apache.org/jira/browse/HBASE-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766184#comment-15766184
]
stack commented on HBASE-17351:
---
Sounds good [~esteban] We don't inherit the maven-enforcer-plugin version from
the apache pom because it is just referenced as a dependency, it is not under
'plugindependencies'? Should we use version 1.4.1 rather than 1.4? Good one
[~esteban] +1
> Unable to generate tar ball for master branch
> -
>
> Key: HBASE-17351
> URL: https://issues.apache.org/jira/browse/HBASE-17351
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Attachments:
> 0001-HBASE-17351-Unable-to-generate-tar-ball-for-master-b.patch
>
>
> Using this command:
> mvn install -X -DskipTests assembly:single -Prelease
> I got:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce
> (min-maven-min-java-banned-xerces) on project hbase: Execution
> min-maven-min-java-banned-xerces of goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce failed.
> NullPointerException -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce
> (min-maven-min-java-banned-xerces) on project hbase: Execution
> min-maven-min-java-banned-xerces of goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce failed.
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.PluginExecutionException: Execution
> min-maven-min-java-banned-xerces of goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce failed.
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:145)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
> ... 20 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.maven.plugins.enforcer.EnforceBytecodeVersion.isBadArtifact(EnforceBytecodeVersion.java:221)
> at
> org.apache.maven.plugins.enforcer.EnforceBytecodeVersion.checkDependencies(EnforceBytecodeVersion.java:206)
> at
> org.apache.maven.plugins.enforcer.EnforceBytecodeVersion.handleArtifacts(EnforceBytecodeVersion.java:132)
> at
> org.apache.maven.plugins.enforcer.AbstractResolveDependencies.execute(AbstractResolveDependencies.java:77)
> at
> org.apache.maven.plugins.enforcer.EnforceMojo.execute(EnforceMojo.java:193)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
> ... 21 more
> {code}
> A brief search led to HBASE-16335:
> {code}
> commit 046d1a56b242d8586d8bcd81d26c11a859651972
> Author: Jan Hentschel
> Date: Thu Nov 17 21:03:55 2016 +0100
> HBASE-16335 Update to latest Apache parent pom
> Signed-off-by: Michael Stack
> commit
[jira] [Commented] (HBASE-11392) add/remove peer requests should be routed through master
[ https://issues.apache.org/jira/browse/HBASE-11392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766177#comment-15766177 ] Ashish Singhi commented on HBASE-11392: --- v6 patch is good to me. > add/remove peer requests should be routed through master > > > Key: HBASE-11392 > URL: https://issues.apache.org/jira/browse/HBASE-11392 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Guanghao Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-11392-v1.patch, HBASE-11392-v2.patch, > HBASE-11392-v3.patch, HBASE-11392-v4.patch, HBASE-11392-v5.patch, > HBASE-11392-v6.patch > > > ReplicationAdmin directly operates over the zookeeper data for replication > setup. We should move these operations to be routed through master for two > reasons: > - Replication implementation details are exposed to client. We should move > most of the replication related classes to hbase-server package. > - Routing the requests through master is the standard practice for all other > operations. It allows for decoupling implementation details from the client > and code. > Review board: https://reviews.apache.org/r/54730/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766173#comment-15766173
]
Hudson commented on HBASE-17328:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #75 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/75/])
HBASE-17328 Properly dispose of looped replication peers (apurtell: rev
f276edfadbc5b598bdd7412809e76e8077207603)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24, 1.1.9
>
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch,
> HBASE-17328-master.v2.patch, HBASE-17328.0.98.v4.patch,
> HBASE-17328.branch-1.1.v2.patch, HBASE-17328.branch-1.1.v3.patch,
> HBASE-17328.branch-1.1.v4.patch, HBASE-17328.master.v4.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17314) Limit total buffered size for all replication sources
[ https://issues.apache.org/jira/browse/HBASE-17314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17314: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.4.0 2.0.0 Release Note: Add a conf "replication.total.buffer.quota" to limit total size of buffered entries in all replication peers. It will prevent server getting OOM if there are many peers. Default value is 256MB. Status: Resolved (was: Patch Available) Pushed to master and branch-1 with typo fixed. Thanks [~tedyu] for review. > Limit total buffered size for all replication sources > - > > Key: HBASE-17314 > URL: https://issues.apache.org/jira/browse/HBASE-17314 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17314.branch-1.v01.patch, HBASE-17314.v01.patch, > HBASE-17314.v02.patch, HBASE-17314.v03.patch, HBASE-17314.v04.patch > > > If we have many peers or some servers have many recovered queues, we will > hold many entries in memory which will increase the pressure of GC, even > maybe OOM because we will read entries for 64MB to buffer in default for one > source. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17351) Unable to generate tar ball for master branch
[
https://issues.apache.org/jira/browse/HBASE-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766163#comment-15766163
]
Esteban Gutierrez commented on HBASE-17351:
---
1.4.1 is the latest and 1.4 is just 7 months behind. HBASE-16335 is the reason
was this version got pulled in:
https://repo1.maven.org/maven2/org/apache/apache/18/apache-18.pom
{code}
org.apache.maven.plugins
maven-enforcer-plugin
1.4.1
{code}
Before HBASE-16335 the version used of the enforcer used by the apache pom was
1.0.1:
{code}
org.apache.maven.plugins
maven-enforcer-plugin
1.0.1
{code}
> Unable to generate tar ball for master branch
> -
>
> Key: HBASE-17351
> URL: https://issues.apache.org/jira/browse/HBASE-17351
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Attachments:
> 0001-HBASE-17351-Unable-to-generate-tar-ball-for-master-b.patch
>
>
> Using this command:
> mvn install -X -DskipTests assembly:single -Prelease
> I got:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce
> (min-maven-min-java-banned-xerces) on project hbase: Execution
> min-maven-min-java-banned-xerces of goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce failed.
> NullPointerException -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce
> (min-maven-min-java-banned-xerces) on project hbase: Execution
> min-maven-min-java-banned-xerces of goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce failed.
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.PluginExecutionException: Execution
> min-maven-min-java-banned-xerces of goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce failed.
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:145)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
> ... 20 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.maven.plugins.enforcer.EnforceBytecodeVersion.isBadArtifact(EnforceBytecodeVersion.java:221)
> at
> org.apache.maven.plugins.enforcer.EnforceBytecodeVersion.checkDependencies(EnforceBytecodeVersion.java:206)
> at
> org.apache.maven.plugins.enforcer.EnforceBytecodeVersion.handleArtifacts(EnforceBytecodeVersion.java:132)
> at
> org.apache.maven.plugins.enforcer.AbstractResolveDependencies.execute(AbstractResolveDependencies.java:77)
> at
> org.apache.maven.plugins.enforcer.EnforceMojo.execute(EnforceMojo.java:193)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
> ... 21 more
> {code}
> A brief search led to HBASE-16335:
> {code}
> commit 046d1a56b242d8586d8bcd81d26c11a859651972
> Author: Jan Hentschel
[jira] [Commented] (HBASE-17291) Remove ImmutableSegment#getKeyValueScanner
[ https://issues.apache.org/jira/browse/HBASE-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766152#comment-15766152 ] ramkrishna.s.vasudevan commented on HBASE-17291: [~anastas] - want to have a look here? It will help me to do some other related JIRAs around this in terms of flushes. > Remove ImmutableSegment#getKeyValueScanner > -- > > Key: HBASE-17291 > URL: https://issues.apache.org/jira/browse/HBASE-17291 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17291.patch, HBASE-17291_1.patch, > HBASE-17291_2.patch > > > This is based on a discussion over [~anastas]'s patch. The MemstoreSnapshot > uses a KeyValueScanner which actually seems redundant considering we already > have a SegmentScanner. The idea is that the snapshot scanner should be a > simple iterator type of scanner but it lacks the capability to do the > reference counting on that segment that is now used in snapshot. With > snapshot having mulitple segments in the latest impl it is better we hold on > to the segment by doing ref counting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17353) Cache famLen in OffheapKeyValue
[ https://issues.apache.org/jira/browse/HBASE-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766148#comment-15766148 ] ramkrishna.s.vasudevan commented on HBASE-17353: We will have one more byte heap overhead for an offheap cell. We have now 47 bytes FIXED overhead for an offheap kv. Now if we cache famLen it will be 48 bytes. Not a big change but still when we have lot of small cells then we have a price for heap overhead though the cell data is offheap. > Cache famLen in OffheapKeyValue > --- > > Key: HBASE-17353 > URL: https://issues.apache.org/jira/browse/HBASE-17353 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: OffheapKV famLen cost.png > > > WE need to discuss here. Already we had a TODO here. But this comes again > after offheap memstore is committed in trunk. > Attaching a screenshot to show the impact. True that these changes won't have > a direct impact on the final perf but at micro level they would have. > In case of KeyValue it is just retrieving a byte from the byte[] (o(1) > access). > But here we need to access the memory to retrive that one byte though it is > Unsafe based. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17353) Cache famLen in OffheapKeyValue
[ https://issues.apache.org/jira/browse/HBASE-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17353: --- Description: WE need to discuss here. Already we had a TODO here. But this comes again after offheap memstore is committed in trunk. Attaching a screenshot to show the impact. True that these changes won't have a direct impact on the final perf but at micro level they would have. In case of KeyValue it is just retrieving a byte from the byte[] (o(1) access). But here we need to access the memory to retrive that one byte though it is Unsafe based. was: WE need to discuss here. Already we had a TODO here. But this comes again after offheap memstore is committed in trunk. Attaching a screenshot to show the impact. True that these changes won't have a direct impact on the final perf but at micro level they would have. In case of KeyValue it is just retrieving a byte from the byte[] (o(1) access). But here we need to access the memory to retrive that one byte thought it is Unsafe. > Cache famLen in OffheapKeyValue > --- > > Key: HBASE-17353 > URL: https://issues.apache.org/jira/browse/HBASE-17353 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: OffheapKV famLen cost.png > > > WE need to discuss here. Already we had a TODO here. But this comes again > after offheap memstore is committed in trunk. > Attaching a screenshot to show the impact. True that these changes won't have > a direct impact on the final perf but at micro level they would have. > In case of KeyValue it is just retrieving a byte from the byte[] (o(1) > access). > But here we need to access the memory to retrive that one byte though it is > Unsafe based. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17353) Cache famLen in OffheapKeyValue
[ https://issues.apache.org/jira/browse/HBASE-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17353: --- Attachment: OffheapKV famLen cost.png > Cache famLen in OffheapKeyValue > --- > > Key: HBASE-17353 > URL: https://issues.apache.org/jira/browse/HBASE-17353 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: OffheapKV famLen cost.png > > > WE need to discuss here. Already we had a TODO here. But this comes again > after offheap memstore is committed in trunk. > Attaching a screenshot to show the impact. True that these changes won't have > a direct impact on the final perf but at micro level they would have. > In case of KeyValue it is just retrieving a byte from the byte[] (o(1) > access). > But here we need to access the memory to retrive that one byte thought it is > Unsafe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17353) Cache famLen in OffheapKeyValue
ramkrishna.s.vasudevan created HBASE-17353: -- Summary: Cache famLen in OffheapKeyValue Key: HBASE-17353 URL: https://issues.apache.org/jira/browse/HBASE-17353 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 2.0.0 WE need to discuss here. Already we had a TODO here. But this comes again after offheap memstore is committed in trunk. Attaching a screenshot to show the impact. True that these changes won't have a direct impact on the final perf but at micro level they would have. In case of KeyValue it is just retrieving a byte from the byte[] (o(1) access). But here we need to access the memory to retrive that one byte thought it is Unsafe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16010) Put draining function through Admin API
[
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766128#comment-15766128
]
Jerry He commented on HBASE-16010:
--
Can fix on commit.
> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Matt Warhaftig
>Priority: Minor
> Attachments: hbase-16010-v1.patch, hbase-16010-v2.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to
> interact directly with Zookeeper draining node to add and remove draining
> servers.
> For example, in draining_servers.rb:
> {code}
> zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config,
> "draining_servers", nil)
> parentZnode = zkw.drainingZNode
> begin
> for server in servers
> node = ZKUtil.joinZNode(parentZnode, server)
> ZKUtil.createAndFailSilent(zkw, node)
> end
> ensure
> zkw.close()
> end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17288) Add warn log for huge Cell and huge row
[
https://issues.apache.org/jira/browse/HBASE-17288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766107#comment-15766107
]
Anoop Sam John commented on HBASE-17288:
bq.Nice, but the row is still needed? We need this to find the huge row.
Yep, may be the row alone we can add in the log?
> Add warn log for huge Cell and huge row
> ---
>
> Key: HBASE-17288
> URL: https://issues.apache.org/jira/browse/HBASE-17288
> Project: HBase
> Issue Type: Improvement
> Components: scan
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-17288-v1.patch, HBASE-17288-v2.patch,
> HBASE-17288.patch
>
>
> Some log examples from our production cluster.
> {code}
> 2016-12-10,17:08:11,478 WARN
> org.apache.hadoop.hbase.regionserver.StoreScanner: adding a HUGE KV into
> result list, kv size:1253360,
> kv:10567114001-1-c/R:r1/1481360887152/Put/vlen=1253245/ts=923099, from
> table X
> 2016-12-10,17:08:16,724 WARN
> org.apache.hadoop.hbase.regionserver.StoreScanner: adding a HUGE KV into
> result list, kv size:1048680,
> kv:0220459/I:i_0/1481360889551/Put/vlen=1048576/ts=13642, from table XX
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766102#comment-15766102
]
Hadoop QA commented on HBASE-17328:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 34s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
22s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} 0.98 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 39s
{color} | {color:red} hbase-server in 0.98 has 84 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 9m
11s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 125m 28s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 162m 8s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.util.TestHBaseFsck |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:ef91163 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844172/HBASE-17328.0.98.v4.patch
|
| JIRA Issue | HBASE-17328 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 1da274709fe8 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | 0.98 / f63b5a0 |
| Default Java | 1.7.0_80 |
| findbugs | v2.0.1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5008/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5008/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5008/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5008/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5008/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Properly dispose of looped replication peers
>
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766090#comment-15766090 ] ramkrishna.s.vasudevan commented on HBASE-16630: [~dvdreddy] Thanks for the info. We can review the fix again. [~saint@gmail.com] What do you think of [~dvdreddy] point? > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar >Priority: Critical > Attachments: 16630-v2-suggest.patch, 16630-v3-suggest.patch, > HBASE-16630-v2.patch, HBASE-16630-v3.patch, HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17352) Fix hbase-assembly build with bash 4
[
https://issues.apache.org/jira/browse/HBASE-17352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766084#comment-15766084
]
Ted Yu commented on HBASE-17352:
+1
> Fix hbase-assembly build with bash 4
>
>
> Key: HBASE-17352
> URL: https://issues.apache.org/jira/browse/HBASE-17352
> Project: HBase
> Issue Type: Bug
>Reporter: Junegunn Choi
>Assignee: Junegunn Choi
>Priority: Minor
> Attachments: HBASE-17352.patch
>
>
> hbase-assembly fails to build with bash 4.
> {noformat}
> [DEBUG] Executing command line: [env, bash, -c, cat
> maven-shared-archive-resources/META-INF/NOTICE \
> `find
> /Users/jg/github/hbase/hbase-assembly/target/dependency -iname NOTICE -or
> -iname NOTICE.txt` \]
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec
> (concat-NOTICE-files) on project hbase-assembly: Command execution failed.
> Process exited with an error: 1 (Exit value: 1) -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec (concat-NOTICE-files) on
> project hbase-assembly: Command execution failed.
> {noformat}
> The error is caused by the trailing backslash in the bash command for
> {{concat-NOTICE-files}}. You can see the behavioral difference between bash 3
> and 4 with the following snippet.
> {code}
> $ # Using bash 3
> $ /bin/bash -c 'cat <(echo foo) \' && echo good || echo bad
> foo
> good
> $ # Using bash 4
> $ /usr/local/bin/bash -c 'cat <(echo foo) \' && echo good || echo bad
> foo
> cat: \: No such file or directory
> bad
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17352) Fix hbase-assembly build with bash 4
[
https://issues.apache.org/jira/browse/HBASE-17352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766076#comment-15766076
]
Hadoop QA commented on HBASE-17352:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 12s
{color} | {color:green} hbase-assembly in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 56s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844179/HBASE-17352.patch |
| JIRA Issue | HBASE-17352 |
| Optional Tests | asflicense javac javadoc unit xml compile |
| uname | Linux 750e53058937 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3599716 |
| Default Java | 1.8.0_111 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5009/testReport/ |
| modules | C: hbase-assembly U: hbase-assembly |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5009/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fix hbase-assembly build with bash 4
>
>
> Key: HBASE-17352
> URL: https://issues.apache.org/jira/browse/HBASE-17352
> Project: HBase
> Issue Type: Bug
>Reporter: Junegunn Choi
>Assignee: Junegunn Choi
>Priority: Minor
> Attachments: HBASE-17352.patch
>
>
> hbase-assembly fails to build with bash 4.
> {noformat}
> [DEBUG] Executing command line: [env, bash, -c, cat
> maven-shared-archive-resources/META-INF/NOTICE \
> `find
> /Users/jg/github/hbase/hbase-assembly/target/dependency -iname NOTICE -or
> -iname NOTICE.txt` \]
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec
> (concat-NOTICE-files) on project hbase-assembly: Command execution failed.
> Process exited with an error: 1 (Exit value: 1) -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException:
[jira] [Commented] (HBASE-16010) Put draining function through Admin API
[
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766066#comment-15766066
]
Jerry He commented on HBASE-16010:
--
+1
Some nit.
In HBaseAdmin, two places:
{noformat}
+executeCallable(new MasterCallable(getConnection(),
getRpcControllerFactory()) {
+ @Override
+ public Void rpcCall() throws ServiceException {
+HBaseRpcController controller = rpcControllerFactory.newController();
{noformat}
Use getRpcController() instead of creating a new one.
In MasterRpcServices, two places:
{noformat}
+ for
(org.apache.hadoop.hbase.shaded.protobuf.generated.HBaseProtos.ServerName
pbServer : request
{noformat}
Let's use the short name in the code and add to import.
> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Matt Warhaftig
>Priority: Minor
> Attachments: hbase-16010-v1.patch, hbase-16010-v2.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to
> interact directly with Zookeeper draining node to add and remove draining
> servers.
> For example, in draining_servers.rb:
> {code}
> zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config,
> "draining_servers", nil)
> parentZnode = zkw.drainingZNode
> begin
> for server in servers
> node = ZKUtil.joinZNode(parentZnode, server)
> ZKUtil.createAndFailSilent(zkw, node)
> end
> ensure
> zkw.close()
> end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14932) bulkload fails because file not found
[
https://issues.apache.org/jira/browse/HBASE-14932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766051#comment-15766051
]
Hadoop QA commented on HBASE-14932:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 7s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
28s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} 0.98 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 37s
{color} | {color:red} hbase-server in 0.98 has 84 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 9m
12s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 124m 0s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 160m 15s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.util.TestHBaseFsck |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:ef91163 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12789142/HBASE-14932-0.98.patch
|
| JIRA Issue | HBASE-14932 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f6159b95edc1 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | 0.98 / f63b5a0 |
| Default Java | 1.7.0_80 |
| findbugs | v2.0.1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5002/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/5002/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5002/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5002/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5002/testReport/ |
| modules | C:
[jira] [Commented] (HBASE-17341) Add a timeout during replication endpoint termination
[
https://issues.apache.org/jira/browse/HBASE-17341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766019#comment-15766019
]
Hadoop QA commented on HBASE-17341:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} branch-1.1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} branch-1.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 48s
{color} | {color:red} hbase-server in branch-1.1 has 81 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 23s
{color} | {color:red} hbase-server in branch-1.1 failed with JDK v1.8.0_111.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 34s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 23s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_111.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 30s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 110m 16s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.handler.TestEnableTableHandler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8012383 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844174/HBASE-17341.branch-1.1.v2.patch
|
| JIRA Issue | HBASE-17341 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux e23b2d98316b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-5401) PerformanceEvaluation generates 10x the number of expected mappers
[
https://issues.apache.org/jira/browse/HBASE-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766016#comment-15766016
]
Hadoop QA commented on HBASE-5401:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 43s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 24s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 24s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 138m 28s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844170/HBASE-5401-V1.patch |
| JIRA Issue | HBASE-5401 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 9e7d8eac1e37 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 06b67a6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5006/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5006/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5006/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> PerformanceEvaluation generates 10x the number of expected mappers
> --
>
> Key: HBASE-5401
> URL: https://issues.apache.org/jira/browse/HBASE-5401
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.0
>Reporter:
[jira] [Commented] (HBASE-17341) Add a timeout during replication endpoint termination
[ https://issues.apache.org/jira/browse/HBASE-17341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766012#comment-15766012 ] Guanghao Zhang commented on HBASE-17341: +1 on v2. We met this problem on our cluster, too. The region server shutdown hanged when terminate ReplicationSource. > Add a timeout during replication endpoint termination > - > > Key: HBASE-17341 > URL: https://issues.apache.org/jira/browse/HBASE-17341 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Critical > Attachments: HBASE-17341.branch-1.1.v1.patch, > HBASE-17341.branch-1.1.v2.patch, HBASE-17341.master.v1.patch, > HBASE-17341.master.v2.patch > > > In ReplicationSource#terminate(), a Future is obtained from > ReplicationEndpoint#stop(). Future.get() is then called, but can potentially > hang there if something went wrong in the endpoint stop(). > Hanging there has serious implications, because the thread could potentially > be the ZK event thread (e.g. watcher calls > ReplicationSourceManager#removePeer() -> ReplicationSource#terminate() -> > blocked). This means no other events in the ZK event queue will get > processed, which for HBase means other ZK watches such as replication watch > notifications, snapshot watch notifications, even RegionServer shutdown will > all get blocked. > The short term fix addressed here is to simply add a timeout for > Future.get(). But the severe consequences seen here perhaps suggest a > broader refactoring of the ZKWatcher usage in HBase is in order, to protect > against situations like this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17352) Fix hbase-assembly build with bash 4
[
https://issues.apache.org/jira/browse/HBASE-17352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junegunn Choi updated HBASE-17352:
--
Status: Patch Available (was: Open)
> Fix hbase-assembly build with bash 4
>
>
> Key: HBASE-17352
> URL: https://issues.apache.org/jira/browse/HBASE-17352
> Project: HBase
> Issue Type: Bug
>Reporter: Junegunn Choi
>Assignee: Junegunn Choi
>Priority: Minor
> Attachments: HBASE-17352.patch
>
>
> hbase-assembly fails to build with bash 4.
> {noformat}
> [DEBUG] Executing command line: [env, bash, -c, cat
> maven-shared-archive-resources/META-INF/NOTICE \
> `find
> /Users/jg/github/hbase/hbase-assembly/target/dependency -iname NOTICE -or
> -iname NOTICE.txt` \]
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec
> (concat-NOTICE-files) on project hbase-assembly: Command execution failed.
> Process exited with an error: 1 (Exit value: 1) -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec (concat-NOTICE-files) on
> project hbase-assembly: Command execution failed.
> {noformat}
> The error is caused by the trailing backslash in the bash command for
> {{concat-NOTICE-files}}. You can see the behavioral difference between bash 3
> and 4 with the following snippet.
> {code}
> $ # Using bash 3
> $ /bin/bash -c 'cat <(echo foo) \' && echo good || echo bad
> foo
> good
> $ # Using bash 4
> $ /usr/local/bin/bash -c 'cat <(echo foo) \' && echo good || echo bad
> foo
> cat: \: No such file or directory
> bad
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17352) Fix hbase-assembly build with bash 4
[
https://issues.apache.org/jira/browse/HBASE-17352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junegunn Choi updated HBASE-17352:
--
Attachment: HBASE-17352.patch
> Fix hbase-assembly build with bash 4
>
>
> Key: HBASE-17352
> URL: https://issues.apache.org/jira/browse/HBASE-17352
> Project: HBase
> Issue Type: Bug
>Reporter: Junegunn Choi
>Assignee: Junegunn Choi
>Priority: Minor
> Attachments: HBASE-17352.patch
>
>
> hbase-assembly fails to build with bash 4.
> {noformat}
> [DEBUG] Executing command line: [env, bash, -c, cat
> maven-shared-archive-resources/META-INF/NOTICE \
> `find
> /Users/jg/github/hbase/hbase-assembly/target/dependency -iname NOTICE -or
> -iname NOTICE.txt` \]
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec
> (concat-NOTICE-files) on project hbase-assembly: Command execution failed.
> Process exited with an error: 1 (Exit value: 1) -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec (concat-NOTICE-files) on
> project hbase-assembly: Command execution failed.
> {noformat}
> The error is caused by the trailing backslash in the bash command for
> {{concat-NOTICE-files}}. You can see the behavioral difference between bash 3
> and 4 with the following snippet.
> {code}
> $ # Using bash 3
> $ /bin/bash -c 'cat <(echo foo) \' && echo good || echo bad
> foo
> good
> $ # Using bash 4
> $ /usr/local/bin/bash -c 'cat <(echo foo) \' && echo good || echo bad
> foo
> cat: \: No such file or directory
> bad
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-17352) Fix hbase-assembly build with bash 4
Junegunn Choi created HBASE-17352:
-
Summary: Fix hbase-assembly build with bash 4
Key: HBASE-17352
URL: https://issues.apache.org/jira/browse/HBASE-17352
Project: HBase
Issue Type: Bug
Reporter: Junegunn Choi
Assignee: Junegunn Choi
Priority: Minor
hbase-assembly fails to build with bash 4.
{noformat}
[DEBUG] Executing command line: [env, bash, -c, cat
maven-shared-archive-resources/META-INF/NOTICE \
`find /Users/jg/github/hbase/hbase-assembly/target/dependency
-iname NOTICE -or -iname NOTICE.txt` \]
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec
(concat-NOTICE-files) on project hbase-assembly: Command execution failed.
Process exited with an error: 1 (Exit value: 1) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal
org.codehaus.mojo:exec-maven-plugin:1.4.0:exec (concat-NOTICE-files) on project
hbase-assembly: Command execution failed.
{noformat}
The error is caused by the trailing backslash in the bash command for
{{concat-NOTICE-files}}. You can see the behavioral difference between bash 3
and 4 with the following snippet.
{code}
$ # Using bash 3
$ /bin/bash -c 'cat <(echo foo) \' && echo good || echo bad
foo
good
$ # Using bash 4
$ /usr/local/bin/bash -c 'cat <(echo foo) \' && echo good || echo bad
foo
cat: \: No such file or directory
bad
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765979#comment-15765979
]
Hadoop QA commented on HBASE-17328:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} branch-1.1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} branch-1.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 44s
{color} | {color:red} hbase-server in branch-1.1 has 81 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 31s
{color} | {color:red} hbase-server in branch-1.1 failed with JDK v1.8.0_111.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 31s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 22s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_111.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 82m 50s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 112m 48s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8012383 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844169/HBASE-17328.branch-1.1.v4.patch
|
| JIRA Issue | HBASE-17328 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 822eea1c3a8e 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1.1 / aa47da8 |
| Default Java | 1.7.0_80 |
| Multi-JDK
[jira] [Commented] (HBASE-17306) IntegrationTestRSGroup#testRegionMove may fail due to region server not online
[
https://issues.apache.org/jira/browse/HBASE-17306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765963#comment-15765963
]
Ted Yu commented on HBASE-17306:
[~toffer]:
>From our observation, at the end of testKillRS, the killed region server
>wasn't restored to online status, leading to subsequent test failure.
Have you ever seen the above problem ?
> IntegrationTestRSGroup#testRegionMove may fail due to region server not online
> --
>
> Key: HBASE-17306
> URL: https://issues.apache.org/jira/browse/HBASE-17306
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 17306.v1.txt
>
>
> {code}
> 2016-12-13 05:26:57,965|INFO|MainThread|machine.py:145 - run()|2)
> testRegionMove(org.apache.hadoop.hbase.rsgroup.IntegrationTestRSGroup)
> 2016-12-13 05:26:57,965|INFO|MainThread|machine.py:145 -
> run()|org.apache.hadoop.hbase.constraint.ConstraintException:
> org.apache.hadoop.hbase.constraint.ConstraintException:
> Server ctr-e77-1481596162056-0240-01-05.a.com:16020 is not an online
> server in default group.
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveServers(RSGroupAdminServer.java:135)
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint.moveServers(RSGroupAdminEndpoint.java:169)
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.
> callMethod(RSGroupAdminProtos.java:11136)
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:679)
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2
> {code}
> Shortly before the test failure, the server was shutdown:
> {code}
> 2016-12-13 05:21:25,428 INFO
> [MASTER_SERVER_OPERATIONS-ctr-e77-1481596162056-0240-01-08:2-4]
> handler.ServerShutdownHandler: Finished processing of shutdown of ctr-
> e77-1481596162056-0240-01-05.a.com,16020,1481606309159
> ...
> 2016-12-13 05:26:57,935 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=2]
> master.ServerManager: Registering
> server=ctr-e77-1481596162056-0240-01-05.hwx. site,16020,1481606803303
> 2016-12-13 05:27:06,219 DEBUG [main-EventThread]
> zookeeper.RegionServerTracker: Added tracking of RS
> /hbase-secure/rs/ctr-e77-1481596162056-0240-01-05.a.com,16020,
> 1481606803303
> {code}
> The registration of the new server (start code1481606803303) happened shortly
> after the test failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17306) IntegrationTestRSGroup#testRegionMove may fail due to region server not online
[
https://issues.apache.org/jira/browse/HBASE-17306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765953#comment-15765953
]
Francis Liu commented on HBASE-17306:
-
{quote}
Francis Liu:
Can you give us some background on the above requirement ?
{quote}
You can't move a regionserver that's not online in default group since
membership in default group is dynamic (all online regionservers that are not
members of any other group) there is no way to determine if a offline RS being
move is a valid RS or not which would just lead to more problems.
In any case it seems the problem here is more about stabilizing the test
itself. ie Avoiding the race of moving an RS that is still not online.
> IntegrationTestRSGroup#testRegionMove may fail due to region server not online
> --
>
> Key: HBASE-17306
> URL: https://issues.apache.org/jira/browse/HBASE-17306
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 17306.v1.txt
>
>
> {code}
> 2016-12-13 05:26:57,965|INFO|MainThread|machine.py:145 - run()|2)
> testRegionMove(org.apache.hadoop.hbase.rsgroup.IntegrationTestRSGroup)
> 2016-12-13 05:26:57,965|INFO|MainThread|machine.py:145 -
> run()|org.apache.hadoop.hbase.constraint.ConstraintException:
> org.apache.hadoop.hbase.constraint.ConstraintException:
> Server ctr-e77-1481596162056-0240-01-05.a.com:16020 is not an online
> server in default group.
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveServers(RSGroupAdminServer.java:135)
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint.moveServers(RSGroupAdminEndpoint.java:169)
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.
> callMethod(RSGroupAdminProtos.java:11136)
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:679)
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2
> {code}
> Shortly before the test failure, the server was shutdown:
> {code}
> 2016-12-13 05:21:25,428 INFO
> [MASTER_SERVER_OPERATIONS-ctr-e77-1481596162056-0240-01-08:2-4]
> handler.ServerShutdownHandler: Finished processing of shutdown of ctr-
> e77-1481596162056-0240-01-05.a.com,16020,1481606309159
> ...
> 2016-12-13 05:26:57,935 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=2]
> master.ServerManager: Registering
> server=ctr-e77-1481596162056-0240-01-05.hwx. site,16020,1481606803303
> 2016-12-13 05:27:06,219 DEBUG [main-EventThread]
> zookeeper.RegionServerTracker: Added tracking of RS
> /hbase-secure/rs/ctr-e77-1481596162056-0240-01-05.a.com,16020,
> 1481606803303
> {code}
> The registration of the new server (start code1481606803303) happened shortly
> after the test failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17107) FN info should be cleaned up on region/table cleanup
[ https://issues.apache.org/jira/browse/HBASE-17107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-17107: Resolution: Fixed Status: Resolved (was: Patch Available) Merged into master. Thanks [~thiruvel]! > FN info should be cleaned up on region/table cleanup > > > Key: HBASE-17107 > URL: https://issues.apache.org/jira/browse/HBASE-17107 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17107.master.001.patch, HBASE_17107.draft.patch > > > FN info should be cleaned up when table is deleted and when regions are GCed > (i.e. CatalogJanitor). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-17328:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I committed this to all active branches, picking and fixing up as necessary.
Thanks for the contribution [~vincentpoon] !
Confirmed the new unit in TestMasterReplication passed everywhere.
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.9, 0.98.24
>
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch,
> HBASE-17328-master.v2.patch, HBASE-17328.0.98.v4.patch,
> HBASE-17328.branch-1.1.v2.patch, HBASE-17328.branch-1.1.v3.patch,
> HBASE-17328.branch-1.1.v4.patch, HBASE-17328.master.v4.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17347) ExportSnapshot may write snapshot info file to wrong directory when specifying target name
[
https://issues.apache.org/jira/browse/HBASE-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765912#comment-15765912
]
Jianwei Cui commented on HBASE-17347:
-
Thanks for the review [~tedyu]
> ExportSnapshot may write snapshot info file to wrong directory when
> specifying target name
> --
>
> Key: HBASE-17347
> URL: https://issues.apache.org/jira/browse/HBASE-17347
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Jianwei Cui
>Assignee: Jianwei Cui
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17347-v1.patch
>
>
> Exportsnapshot will write a new snapshot info file when specifying the target
> name:
> {code}
> if (!targetName.equals(snapshotName)) {
> SnapshotDescription snapshotDesc =
> SnapshotDescriptionUtils.readSnapshotInfo(inputFs, snapshotDir)
> .toBuilder()
> .setName(targetName)
> .build();
> SnapshotDescriptionUtils.writeSnapshotInfo(snapshotDesc,
> snapshotTmpDir, outputFs);
> }
> {code}
> The snapshot info file will be written to the snapshot tmp directory,
> however, it should be directly written to the snapshot directory if
> {{snapshot.export.skip.tmp}} is true. In addition, owner and permission
> should be set for the new snapshot info file when needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17107) FN info should be cleaned up on region/table cleanup
[ https://issues.apache.org/jira/browse/HBASE-17107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765911#comment-15765911 ] Francis Liu commented on HBASE-17107: - +1 will merge in a bit > FN info should be cleaned up on region/table cleanup > > > Key: HBASE-17107 > URL: https://issues.apache.org/jira/browse/HBASE-17107 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17107.master.001.patch, HBASE_17107.draft.patch > > > FN info should be cleaned up when table is deleted and when regions are GCed > (i.e. CatalogJanitor). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17281) FN should use datanode port from hdfs configuration
[
https://issues.apache.org/jira/browse/HBASE-17281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765896#comment-15765896
]
Francis Liu commented on HBASE-17281:
-
Change looks fine to me. We can prolly move the logic of transforming and
sending the the fns on the master as part of sending the reigon to be opened.
Tho don't feel strongly about it as dn port changing is not likely and would
require a cluster restart anyway.
> FN should use datanode port from hdfs configuration
> ---
>
> Key: HBASE-17281
> URL: https://issues.apache.org/jira/browse/HBASE-17281
> Project: HBase
> Issue Type: Sub-task
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17281.master.001.patch,
> HBASE-17281.master.002.patch
>
>
> Currently we use the ServerName port for providing favored node hints. We
> should use the DN port from hdfs-site.xml instead to avoid warning messages
> in region server logs. The warnings will be from this section of HDFS code,
> it moves across classes.
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java#L1758
> {code}
> private boolean[] getPinnings(DatanodeInfo[] nodes) {
> if (favoredNodes == null) {
> return null;
> } else {
> boolean[] pinnings = new boolean[nodes.length];
> HashSet favoredSet = new HashSet<>(Arrays.asList(favoredNodes));
> for (int i = 0; i < nodes.length; i++) {
> pinnings[i] = favoredSet.remove(nodes[i].getXferAddrWithHostname());
> LOG.debug("{} was chosen by name node (favored={}).",
> nodes[i].getXferAddrWithHostname(), pinnings[i]);
> }
> if (!favoredSet.isEmpty()) {
> // There is one or more favored nodes that were not allocated.
> LOG.warn("These favored nodes were specified but not chosen: "
> + favoredSet + " Specified favored nodes: "
> + Arrays.toString(favoredNodes));
> }
> return pinnings;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17351) Unable to generate tar ball for master branch
[
https://issues.apache.org/jira/browse/HBASE-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765859#comment-15765859
]
Hadoop QA commented on HBASE-17351:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 19s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 12s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 138m 35s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.namespace.TestNamespaceAuditor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844153/0001-HBASE-17351-Unable-to-generate-tar-ball-for-master-b.patch
|
| JIRA Issue | HBASE-17351 |
| Optional Tests | asflicense javac javadoc unit xml compile |
| uname | Linux c671bf00bd78 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5ebb25d |
| Default Java | 1.8.0_111 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5000/artifact/patchprocess/patch-unit-root.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5000/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5000/testReport/ |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5000/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Unable to generate tar ball for master branch
> -
>
> Key: HBASE-17351
> URL: https://issues.apache.org/jira/browse/HBASE-17351
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Attachments:
> 0001-HBASE-17351-Unable-to-generate-tar-ball-for-master-b.patch
>
>
> Using this command:
> mvn install -X -DskipTests assembly:single -Prelease
> I got:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce
>
[jira] [Commented] (HBASE-17347) ExportSnapshot may write snapshot info file to wrong directory when specifying target name
[
https://issues.apache.org/jira/browse/HBASE-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765857#comment-15765857
]
Hudson commented on HBASE-17347:
FAILURE: Integrated in Jenkins build HBase-1.4 #573 (See
[https://builds.apache.org/job/HBase-1.4/573/])
HBASE-17347 ExportSnapshot may write snapshot info file to wrong (tedyu: rev
fa975fa3829fd7182896ece7f8a4951e7ca7bed4)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
> ExportSnapshot may write snapshot info file to wrong directory when
> specifying target name
> --
>
> Key: HBASE-17347
> URL: https://issues.apache.org/jira/browse/HBASE-17347
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Jianwei Cui
>Assignee: Jianwei Cui
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17347-v1.patch
>
>
> Exportsnapshot will write a new snapshot info file when specifying the target
> name:
> {code}
> if (!targetName.equals(snapshotName)) {
> SnapshotDescription snapshotDesc =
> SnapshotDescriptionUtils.readSnapshotInfo(inputFs, snapshotDir)
> .toBuilder()
> .setName(targetName)
> .build();
> SnapshotDescriptionUtils.writeSnapshotInfo(snapshotDesc,
> snapshotTmpDir, outputFs);
> }
> {code}
> The snapshot info file will be written to the snapshot tmp directory,
> however, it should be directly written to the snapshot directory if
> {{snapshot.export.skip.tmp}} is true. In addition, owner and permission
> should be set for the new snapshot info file when needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory
[ https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765856#comment-15765856 ] Hudson commented on HBASE-17292: FAILURE: Integrated in Jenkins build HBase-1.4 #573 (See [https://builds.apache.org/job/HBase-1.4/573/]) HBASE-17292 Add observer notification before bulk loaded hfile is moved (tedyu: rev 0b69f59133f46a1aee6294aa40b54b763fe4222d) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java > Add observer notification before bulk loaded hfile is moved to region > directory > --- > > Key: HBASE-17292 > URL: https://issues.apache.org/jira/browse/HBASE-17292 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.4.0 > > Attachments: 17292.addendum, 17292.branch-1.v1.txt, 17292.v1.txt, > 17292.v2.txt, 17292.v3.txt > > > Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded > hfiles. > However, if bulk load fails after hfile is moved to region directory but > before postBulkLoadHFile() hook is called, there is no way for pluggable > components (replication - see HBASE-17290, backup / restore) to know which > hfile(s) have been moved to region directory. > Even if postBulkLoadHFile() is called in finally block, the write (to backup > table or zookeeper) issued from postBulkLoadHFile() may fail, ending up with > same situation. > This issue adds a preCommitStoreFile() hook which notifies path of to be > committed hfile before bulk loaded hfile is moved to region directory. > With preCommitStoreFile() hook, write (to backup table or zookeeper) can be > issued before the movement of hfile. > If write fails, IOException would make bulk load fail, not leaving hfile in > region directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17341) Add a timeout during replication endpoint termination
[ https://issues.apache.org/jira/browse/HBASE-17341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765847#comment-15765847 ] Vincent Poon commented on HBASE-17341: -- Added some more info in log message > Add a timeout during replication endpoint termination > - > > Key: HBASE-17341 > URL: https://issues.apache.org/jira/browse/HBASE-17341 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Critical > Attachments: HBASE-17341.branch-1.1.v1.patch, > HBASE-17341.branch-1.1.v2.patch, HBASE-17341.master.v1.patch, > HBASE-17341.master.v2.patch > > > In ReplicationSource#terminate(), a Future is obtained from > ReplicationEndpoint#stop(). Future.get() is then called, but can potentially > hang there if something went wrong in the endpoint stop(). > Hanging there has serious implications, because the thread could potentially > be the ZK event thread (e.g. watcher calls > ReplicationSourceManager#removePeer() -> ReplicationSource#terminate() -> > blocked). This means no other events in the ZK event queue will get > processed, which for HBase means other ZK watches such as replication watch > notifications, snapshot watch notifications, even RegionServer shutdown will > all get blocked. > The short term fix addressed here is to simply add a timeout for > Future.get(). But the severe consequences seen here perhaps suggest a > broader refactoring of the ZKWatcher usage in HBase is in order, to protect > against situations like this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17341) Add a timeout during replication endpoint termination
[ https://issues.apache.org/jira/browse/HBASE-17341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Poon updated HBASE-17341: - Attachment: HBASE-17341.master.v2.patch > Add a timeout during replication endpoint termination > - > > Key: HBASE-17341 > URL: https://issues.apache.org/jira/browse/HBASE-17341 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Critical > Attachments: HBASE-17341.branch-1.1.v1.patch, > HBASE-17341.branch-1.1.v2.patch, HBASE-17341.master.v1.patch, > HBASE-17341.master.v2.patch > > > In ReplicationSource#terminate(), a Future is obtained from > ReplicationEndpoint#stop(). Future.get() is then called, but can potentially > hang there if something went wrong in the endpoint stop(). > Hanging there has serious implications, because the thread could potentially > be the ZK event thread (e.g. watcher calls > ReplicationSourceManager#removePeer() -> ReplicationSource#terminate() -> > blocked). This means no other events in the ZK event queue will get > processed, which for HBase means other ZK watches such as replication watch > notifications, snapshot watch notifications, even RegionServer shutdown will > all get blocked. > The short term fix addressed here is to simply add a timeout for > Future.get(). But the severe consequences seen here perhaps suggest a > broader refactoring of the ZKWatcher usage in HBase is in order, to protect > against situations like this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17341) Add a timeout during replication endpoint termination
[ https://issues.apache.org/jira/browse/HBASE-17341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Poon updated HBASE-17341: - Attachment: HBASE-17341.branch-1.1.v2.patch > Add a timeout during replication endpoint termination > - > > Key: HBASE-17341 > URL: https://issues.apache.org/jira/browse/HBASE-17341 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Critical > Attachments: HBASE-17341.branch-1.1.v1.patch, > HBASE-17341.branch-1.1.v2.patch, HBASE-17341.master.v1.patch, > HBASE-17341.master.v2.patch > > > In ReplicationSource#terminate(), a Future is obtained from > ReplicationEndpoint#stop(). Future.get() is then called, but can potentially > hang there if something went wrong in the endpoint stop(). > Hanging there has serious implications, because the thread could potentially > be the ZK event thread (e.g. watcher calls > ReplicationSourceManager#removePeer() -> ReplicationSource#terminate() -> > blocked). This means no other events in the ZK event queue will get > processed, which for HBase means other ZK watches such as replication watch > notifications, snapshot watch notifications, even RegionServer shutdown will > all get blocked. > The short term fix addressed here is to simply add a timeout for > Future.get(). But the severe consequences seen here perhaps suggest a > broader refactoring of the ZKWatcher usage in HBase is in order, to protect > against situations like this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17351) Unable to generate tar ball for master branch
[
https://issues.apache.org/jira/browse/HBASE-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765836#comment-15765836
]
stack commented on HBASE-17351:
---
I wonder why there was no version on the enforcer plugin up to this? Is 1.4 the
latest? Thanks.
> Unable to generate tar ball for master branch
> -
>
> Key: HBASE-17351
> URL: https://issues.apache.org/jira/browse/HBASE-17351
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Attachments:
> 0001-HBASE-17351-Unable-to-generate-tar-ball-for-master-b.patch
>
>
> Using this command:
> mvn install -X -DskipTests assembly:single -Prelease
> I got:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce
> (min-maven-min-java-banned-xerces) on project hbase: Execution
> min-maven-min-java-banned-xerces of goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce failed.
> NullPointerException -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce
> (min-maven-min-java-banned-xerces) on project hbase: Execution
> min-maven-min-java-banned-xerces of goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce failed.
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.PluginExecutionException: Execution
> min-maven-min-java-banned-xerces of goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce failed.
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:145)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
> ... 20 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.maven.plugins.enforcer.EnforceBytecodeVersion.isBadArtifact(EnforceBytecodeVersion.java:221)
> at
> org.apache.maven.plugins.enforcer.EnforceBytecodeVersion.checkDependencies(EnforceBytecodeVersion.java:206)
> at
> org.apache.maven.plugins.enforcer.EnforceBytecodeVersion.handleArtifacts(EnforceBytecodeVersion.java:132)
> at
> org.apache.maven.plugins.enforcer.AbstractResolveDependencies.execute(AbstractResolveDependencies.java:77)
> at
> org.apache.maven.plugins.enforcer.EnforceMojo.execute(EnforceMojo.java:193)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
> ... 21 more
> {code}
> A brief search led to HBASE-16335:
> {code}
> commit 046d1a56b242d8586d8bcd81d26c11a859651972
> Author: Jan Hentschel
> Date: Thu Nov 17 21:03:55 2016 +0100
> HBASE-16335 Update to latest Apache parent pom
> Signed-off-by: Michael Stack
> commit 7c6e839f6a98cf2c3ed37109318632db13b4a0df
> Author: Esteban Gutierrez
> Date: Thu Nov 17 11:11:30 2016 -0800
> HBASE-17058 Lower epsilon used
[jira] [Commented] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765831#comment-15765831
]
Vincent Poon commented on HBASE-17328:
--
[~apurtell] added a 0.98 patch
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24, 1.1.9
>
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch,
> HBASE-17328-master.v2.patch, HBASE-17328.0.98.v4.patch,
> HBASE-17328.branch-1.1.v2.patch, HBASE-17328.branch-1.1.v3.patch,
> HBASE-17328.branch-1.1.v4.patch, HBASE-17328.master.v4.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765828#comment-15765828
]
Andrew Purtell commented on HBASE-17328:
Thanks [~vincentpoon], v4 patch looks good. Committing
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24, 1.1.9
>
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch,
> HBASE-17328-master.v2.patch, HBASE-17328.0.98.v4.patch,
> HBASE-17328.branch-1.1.v2.patch, HBASE-17328.branch-1.1.v3.patch,
> HBASE-17328.branch-1.1.v4.patch, HBASE-17328.master.v4.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincent Poon updated HBASE-17328:
-
Attachment: HBASE-17328.0.98.v4.patch
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24, 1.1.9
>
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch,
> HBASE-17328-master.v2.patch, HBASE-17328.0.98.v4.patch,
> HBASE-17328.branch-1.1.v2.patch, HBASE-17328.branch-1.1.v3.patch,
> HBASE-17328.branch-1.1.v4.patch, HBASE-17328.master.v4.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17198) FN updates during region merge (follow up to Procedure v2 merge)
[ https://issues.apache.org/jira/browse/HBASE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765829#comment-15765829 ] Francis Liu edited comment on HBASE-17198 at 12/21/16 1:50 AM: --- +1 will commit after HBASE-17101 is in was (Author: toffer): +1 will merge after HBASE-17101 is in > FN updates during region merge (follow up to Procedure v2 merge) > > > Key: HBASE-17198 > URL: https://issues.apache.org/jira/browse/HBASE-17198 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17198.master.001.patch > > > As mentioned in https://reviews.apache.org/r/53242/ (HBASE-16941), since the > procedure v2 merge changes are in development, there is a follow up > optimization/cleanup that can be done for favored nodes during merge. This > jira will be taken up once HBASE-16119 is complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17198) FN updates during region merge (follow up to Procedure v2 merge)
[ https://issues.apache.org/jira/browse/HBASE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765829#comment-15765829 ] Francis Liu commented on HBASE-17198: - +1 will merge after HBASE-17101 is in > FN updates during region merge (follow up to Procedure v2 merge) > > > Key: HBASE-17198 > URL: https://issues.apache.org/jira/browse/HBASE-17198 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17198.master.001.patch > > > As mentioned in https://reviews.apache.org/r/53242/ (HBASE-16941), since the > procedure v2 merge changes are in development, there is a follow up > optimization/cleanup that can be done for favored nodes during merge. This > jira will be taken up once HBASE-16119 is complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17331) Avoid busy waiting in ThrottledInputStream
[
https://issues.apache.org/jira/browse/HBASE-17331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765815#comment-15765815
]
stack commented on HBASE-17331:
---
Thanks
> Avoid busy waiting in ThrottledInputStream
> --
>
> Key: HBASE-17331
> URL: https://issues.apache.org/jira/browse/HBASE-17331
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17331.addendum.patch, HBASE-17331.v0.patch,
> HBASE-17331.v1.patch, HBASE-17331.v2.patch, HBASE-17331.v2.patch
>
>
> {code:title=ThrottledInputStream.java|borderStyle=solid}
> // We can calculate the precise sleep time instead of busy waiting
> private void throttle() throws IOException {
> while (getBytesPerSec() > maxBytesPerSec) {
> try {
> Thread.sleep(SLEEP_DURATION_MS);
> totalSleepTime += SLEEP_DURATION_MS;
> } catch (InterruptedException e) {
> throw new IOException("Thread aborted", e);
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11392) add/remove peer requests should be routed through master
[ https://issues.apache.org/jira/browse/HBASE-11392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765810#comment-15765810 ] Guanghao Zhang commented on HBASE-11392: Failed ut is not related. [~enis] [~ashish singhi] Any more ideas about v6 patch? Thanks. > add/remove peer requests should be routed through master > > > Key: HBASE-11392 > URL: https://issues.apache.org/jira/browse/HBASE-11392 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Guanghao Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-11392-v1.patch, HBASE-11392-v2.patch, > HBASE-11392-v3.patch, HBASE-11392-v4.patch, HBASE-11392-v5.patch, > HBASE-11392-v6.patch > > > ReplicationAdmin directly operates over the zookeeper data for replication > setup. We should move these operations to be routed through master for two > reasons: > - Replication implementation details are exposed to client. We should move > most of the replication related classes to hbase-server package. > - Routing the requests through master is the standard practice for all other > operations. It allows for decoupling implementation details from the client > and code. > Review board: https://reviews.apache.org/r/54730/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15924) Enhance hbase services autorestart capability to hbase-daemon.sh
[ https://issues.apache.org/jira/browse/HBASE-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15924: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: (was: 0.98.24) 1.4.0 2.0.0 Status: Resolved (was: Patch Available) > Enhance hbase services autorestart capability to hbase-daemon.sh > - > > Key: HBASE-15924 > URL: https://issues.apache.org/jira/browse/HBASE-15924 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.19 >Reporter: Loknath Priyatham Teja Singamsetty >Assignee: Loknath Priyatham Teja Singamsetty > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15924.master.0001.patch, > HBASE-15924.master.0002.patch, HBASE-15924.master.0003.patch, > HBASE-15924.master.0004.patch > > > As part of HBASE-5939, the autorestart for hbase services has been added to > deal with scenarios where hbase services (master/regionserver/master-backup) > gets killed or goes down leading to unplanned outages. The changes were made > to hbase-daemon.sh to support autorestart option. > However, the autorestart implementation doesn't work in standalone mode and > other than that have few gaps with the implementation as per release notes of > HBASE-5939. Here is an attempt to re-design and fix the functionality > considering all possible usecases with hbase service operations. > Release Notes of HBASE-5939: > -- > When launched with autorestart, HBase processes will automatically restart if > they are not properly terminated, either by a "stop" command or by a cluster > stop. To ensure that it does not overload the system when the server itself > is corrupted and the process cannot be restarted, the server sleeps for 5 > minutes before restarting if it was already started 5 minutes ago previously. > To use it, launch the process with "bin/start-hbase autorestart". This option > is not fully compatible with the existing "restart" command: if you ask for a > restart on a server launched with autorestart, the server will restart but > the next server instance won't be automatically restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15924) Enhance hbase services autorestart capability to hbase-daemon.sh
[ https://issues.apache.org/jira/browse/HBASE-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765811#comment-15765811 ] Andrew Purtell commented on HBASE-15924: Pushed v4 patch to master and branch-1 > Enhance hbase services autorestart capability to hbase-daemon.sh > - > > Key: HBASE-15924 > URL: https://issues.apache.org/jira/browse/HBASE-15924 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.19 >Reporter: Loknath Priyatham Teja Singamsetty >Assignee: Loknath Priyatham Teja Singamsetty > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15924.master.0001.patch, > HBASE-15924.master.0002.patch, HBASE-15924.master.0003.patch, > HBASE-15924.master.0004.patch > > > As part of HBASE-5939, the autorestart for hbase services has been added to > deal with scenarios where hbase services (master/regionserver/master-backup) > gets killed or goes down leading to unplanned outages. The changes were made > to hbase-daemon.sh to support autorestart option. > However, the autorestart implementation doesn't work in standalone mode and > other than that have few gaps with the implementation as per release notes of > HBASE-5939. Here is an attempt to re-design and fix the functionality > considering all possible usecases with hbase service operations. > Release Notes of HBASE-5939: > -- > When launched with autorestart, HBase processes will automatically restart if > they are not properly terminated, either by a "stop" command or by a cluster > stop. To ensure that it does not overload the system when the server itself > is corrupted and the process cannot be restarted, the server sleeps for 5 > minutes before restarting if it was already started 5 minutes ago previously. > To use it, launch the process with "bin/start-hbase autorestart". This option > is not fully compatible with the existing "restart" command: if you ask for a > restart on a server launched with autorestart, the server will restart but > the next server instance won't be automatically restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17331) Avoid busy waiting in ThrottledInputStream
[
https://issues.apache.org/jira/browse/HBASE-17331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765801#comment-15765801
]
ChiaPing Tsai commented on HBASE-17331:
---
copy that
> Avoid busy waiting in ThrottledInputStream
> --
>
> Key: HBASE-17331
> URL: https://issues.apache.org/jira/browse/HBASE-17331
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17331.addendum.patch, HBASE-17331.v0.patch,
> HBASE-17331.v1.patch, HBASE-17331.v2.patch, HBASE-17331.v2.patch
>
>
> {code:title=ThrottledInputStream.java|borderStyle=solid}
> // We can calculate the precise sleep time instead of busy waiting
> private void throttle() throws IOException {
> while (getBytesPerSec() > maxBytesPerSec) {
> try {
> Thread.sleep(SLEEP_DURATION_MS);
> totalSleepTime += SLEEP_DURATION_MS;
> } catch (InterruptedException e) {
> throw new IOException("Thread aborted", e);
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17341) Add a timeout during replication endpoint termination
[
https://issues.apache.org/jira/browse/HBASE-17341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765804#comment-15765804
]
Guanghao Zhang commented on HBASE-17341:
+1 on this. One minor comment.
bq. LOG.warn("Got exception:", e);
Can you add more info to this log?
> Add a timeout during replication endpoint termination
> -
>
> Key: HBASE-17341
> URL: https://issues.apache.org/jira/browse/HBASE-17341
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Critical
> Attachments: HBASE-17341.branch-1.1.v1.patch,
> HBASE-17341.master.v1.patch
>
>
> In ReplicationSource#terminate(), a Future is obtained from
> ReplicationEndpoint#stop(). Future.get() is then called, but can potentially
> hang there if something went wrong in the endpoint stop().
> Hanging there has serious implications, because the thread could potentially
> be the ZK event thread (e.g. watcher calls
> ReplicationSourceManager#removePeer() -> ReplicationSource#terminate() ->
> blocked). This means no other events in the ZK event queue will get
> processed, which for HBase means other ZK watches such as replication watch
> notifications, snapshot watch notifications, even RegionServer shutdown will
> all get blocked.
> The short term fix addressed here is to simply add a timeout for
> Future.get(). But the severe consequences seen here perhaps suggest a
> broader refactoring of the ZKWatcher usage in HBase is in order, to protect
> against situations like this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17331) Avoid busy waiting in ThrottledInputStream
[
https://issues.apache.org/jira/browse/HBASE-17331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-17331:
--
Release Note: For each read(), old ThrottledInputStream sleeps/wakes/checks
for many times for controlling the throughput. After this patch,
ThrottledInputStream sleeps/wakes/checks only once. So we can reduce CPU usage.
> Avoid busy waiting in ThrottledInputStream
> --
>
> Key: HBASE-17331
> URL: https://issues.apache.org/jira/browse/HBASE-17331
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17331.addendum.patch, HBASE-17331.v0.patch,
> HBASE-17331.v1.patch, HBASE-17331.v2.patch, HBASE-17331.v2.patch
>
>
> {code:title=ThrottledInputStream.java|borderStyle=solid}
> // We can calculate the precise sleep time instead of busy waiting
> private void throttle() throws IOException {
> while (getBytesPerSec() > maxBytesPerSec) {
> try {
> Thread.sleep(SLEEP_DURATION_MS);
> totalSleepTime += SLEEP_DURATION_MS;
> } catch (InterruptedException e) {
> throw new IOException("Thread aborted", e);
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15454) Archive store files older than max age
[
https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765788#comment-15765788
]
Hadoop QA commented on HBASE-15454:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-15454 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12800944/HBASE-15454-v7.patch |
| JIRA Issue | HBASE-15454 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5004/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Archive store files older than max age
> --
>
> Key: HBASE-15454
> URL: https://issues.apache.org/jira/browse/HBASE-15454
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15454-v1.patch, HBASE-15454-v2.patch,
> HBASE-15454-v3.patch, HBASE-15454-v4.patch, HBASE-15454-v5.patch,
> HBASE-15454-v6.patch, HBASE-15454-v7.patch, HBASE-15454.patch
>
>
> In date tiered compaction, the store files older than max age are never
> touched by minor compactions. Here we introduce a 'freeze window' operation,
> which does the follow things:
> 1. Find all store files that contains cells whose timestamp are in the give
> window.
> 2. Compaction all these files and output one file for each window that these
> files covered.
> After the compaction, we will have only one in the give window, and all cells
> whose timestamp are in the give window are in the only file. And if you do
> not write new cells with an older timestamp in this window, the file will
> never be changed. This makes it easier to do erasure coding on the freezed
> file to reduce redundence. And also, it makes it possible to check
> consistency between master and peer cluster incrementally.
> And why use the word 'freeze'?
> Because there is already an 'HFileArchiver' class. I want to use a different
> word to prevent confusing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[
https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765787#comment-15765787
]
Hadoop QA commented on HBASE-15130:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s {color}
| {color:red} HBASE-15130 does not apply to 0.98. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12792592/HBASE-15130-0.98.v4.patch
|
| JIRA Issue | HBASE-15130 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5001/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Backport 0.98 Scan different TimeRange for each column family
> --
>
> Key: HBASE-15130
> URL: https://issues.apache.org/jira/browse/HBASE-15130
> Project: HBase
> Issue Type: Bug
> Components: Client, regionserver, Scanners
>Affects Versions: 0.98.17
>Reporter: churro morales
>Assignee: churro morales
> Fix For: 0.98.24
>
> Attachments: HBASE-15130-0.98.patch, HBASE-15130-0.98.v1.patch,
> HBASE-15130-0.98.v1.patch, HBASE-15130-0.98.v2.patch,
> HBASE-15130-0.98.v3.patch, HBASE-15130-0.98.v4.patch
>
>
> branch 98 version backport for HBASE-14355
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[
https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765789#comment-15765789
]
Hadoop QA commented on HBASE-12148:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 1s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s {color}
| {color:red} HBASE-12148 does not apply to branch-1. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805194/HBASE-12148.branch-1.v5.patch
|
| JIRA Issue | HBASE-12148 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5003/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Remove TimeRangeTracker as point of contention when many threads writing a
> Store
>
>
> Key: HBASE-12148
> URL: https://issues.apache.org/jira/browse/HBASE-12148
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Affects Versions: 2.0.0, 0.99.1
>Reporter: stack
>Assignee: Walter Koetke
> Fix For: 2.0.0, 1.4.0
>
> Attachments:
> 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch,
> 12148.addendum.txt, 12148.min_and_max_run_independent.patch, 12148.txt,
> 12148.txt, 12148v2.txt, 12148v2.txt, 12148v4.patch, HBASE-12148-V3.patch,
> HBASE-12148-V3.patch, HBASE-12148.branch-1.v5.patch,
> HBASE-12148.branch-1.v5.patch, HBASE-12148.txt, HBASE-12148V2.txt, Screen
> Shot 2014-10-01 at 3.39.46 PM.png, Screen Shot 2014-10-01 at 3.41.07 PM.png,
> Screen Shot 2016-04-13 at 1.49.30 PM.png, Screen Shot 2016-04-13 at 2.02.22
> PM.png, Screen Shot 2016-05-18 at 10.21.53 PM.png, TimeRangeTracker.tiff
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-5401) PerformanceEvaluation generates 10x the number of expected mappers
[ https://issues.apache.org/jira/browse/HBASE-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liang updated HBASE-5401: Assignee: Yi Liang Fix Version/s: 2.0.0 Affects Version/s: 2.0.0 Status: Patch Available (was: Open) > PerformanceEvaluation generates 10x the number of expected mappers > -- > > Key: HBASE-5401 > URL: https://issues.apache.org/jira/browse/HBASE-5401 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: Oliver Meyn >Assignee: Yi Liang > Fix For: 2.0.0 > > Attachments: HBASE-5401-V1.patch > > > With a command line like 'hbase org.apache.hadoop.hbase.PerformanceEvaluation > randomWrite 10' there are 100 mappers spawned, rather than the expected 10. > The culprit appears to be the outer loop in writeInputFile which sets up 10 > splits for every "asked-for client". I think the fix is just to remove that > outer loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5401) PerformanceEvaluation generates 10x the number of expected mappers
[
https://issues.apache.org/jira/browse/HBASE-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765781#comment-15765781
]
Yi Liang commented on HBASE-5401:
-
I have used this command and also encounter this issue, for example:
when I run hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=m
randomWrite n
if we use --nomapred, this will create n threads(clients) and each thread write
m/n rows into hbase
if we use default mapreduce, this will create 10*n mappers, and each mapper
will put m/(n*10) rows into hbase.
I think the static int {code}static int TASKS_PER_CLIENT = 10{code} here is
unnecessary,
1. If user want more mappers they can just change client numbers, however,
if *10 is here, user can only create 10, 20, 30... mappers for different number
of client, this is not flexible.
2. The TASKS_PER_CLIENT = 10 is hardcoded and invisible to user, sometime
may be user just want 5 mappers for their job, and current code will create 50
mappers.
3. when = 5, it means 5 threads and 50 mappers, which is a little
inconsistent, PS. I do not mean mapper is same as thread， but it is better to
keep them same.
What do you guys think?
> PerformanceEvaluation generates 10x the number of expected mappers
> --
>
> Key: HBASE-5401
> URL: https://issues.apache.org/jira/browse/HBASE-5401
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.0
>Reporter: Oliver Meyn
> Fix For: 2.0.0
>
> Attachments: HBASE-5401-V1.patch
>
>
> With a command line like 'hbase org.apache.hadoop.hbase.PerformanceEvaluation
> randomWrite 10' there are 100 mappers spawned, rather than the expected 10.
> The culprit appears to be the outer loop in writeInputFile which sets up 10
> splits for every "asked-for client". I think the fix is just to remove that
> outer loop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-5401) PerformanceEvaluation generates 10x the number of expected mappers
[ https://issues.apache.org/jira/browse/HBASE-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liang updated HBASE-5401: Attachment: HBASE-5401-V1.patch > PerformanceEvaluation generates 10x the number of expected mappers > -- > > Key: HBASE-5401 > URL: https://issues.apache.org/jira/browse/HBASE-5401 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Oliver Meyn > Attachments: HBASE-5401-V1.patch > > > With a command line like 'hbase org.apache.hadoop.hbase.PerformanceEvaluation > randomWrite 10' there are 100 mappers spawned, rather than the expected 10. > The culprit appears to be the outer loop in writeInputFile which sets up 10 > splits for every "asked-for client". I think the fix is just to remove that > outer loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincent Poon updated HBASE-17328:
-
Attachment: HBASE-17328.branch-1.1.v4.patch
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24, 1.1.9
>
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch,
> HBASE-17328-master.v2.patch, HBASE-17328.branch-1.1.v2.patch,
> HBASE-17328.branch-1.1.v3.patch, HBASE-17328.branch-1.1.v4.patch,
> HBASE-17328.master.v4.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincent Poon updated HBASE-17328:
-
Attachment: HBASE-17328.master.v4.patch
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24, 1.1.9
>
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch,
> HBASE-17328-master.v2.patch, HBASE-17328.branch-1.1.v2.patch,
> HBASE-17328.branch-1.1.v3.patch, HBASE-17328.branch-1.1.v4.patch,
> HBASE-17328.master.v4.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11290) Unlock RegionStates
[ https://issues.apache.org/jira/browse/HBASE-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11290: --- Fix Version/s: 0.98.25 > Unlock RegionStates > --- > > Key: HBASE-11290 > URL: https://issues.apache.org/jira/browse/HBASE-11290 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.2.0, 1.3.0 >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 2.0.0, 1.4.0, 0.98.25 > > Attachments: HBASE-11290-0.98.patch, HBASE-11290-0.98_v2.patch, > HBASE-11290.01.branch-1.patch, HBASE-11290.02.patch, HBASE-11290.03.patch, > HBASE-11290.draft.patch, HBASE-11290_trunk.patch > > > Even though RegionStates is a highly accessed data structure in HMaster. Most > of it's methods are synchronized. Which limits concurrency. Even simply > making some of the getters non-synchronized by using concurrent data > structures has helped with region assignments. We can go as simple as this > approach or create locks per region or a bucket lock per region bucket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14223) Meta WALs are not cleared if meta region was closed and RS aborts
[
https://issues.apache.org/jira/browse/HBASE-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-14223:
---
Fix Version/s: (was: 0.98.24)
> Meta WALs are not cleared if meta region was closed and RS aborts
> -
>
> Key: HBASE-14223
> URL: https://issues.apache.org/jira/browse/HBASE-14223
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.0.4, 1.3.1, 1.1.7, 1.2.5
>
> Attachments: HBASE-14223logs, hbase-14223_v0.patch,
> hbase-14223_v1-branch-1.patch, hbase-14223_v2-branch-1.patch,
> hbase-14223_v3-branch-1.patch, hbase-14223_v3-branch-1.patch,
> hbase-14223_v3-master.patch
>
>
> When an RS opens meta, and later closes it, the WAL(FSHlog) is not closed.
> The last WAL file just sits there in the RS WAL directory. If RS stops
> gracefully, the WAL file for meta is deleted. Otherwise if RS aborts, WAL for
> meta is not cleaned. It is also not split (which is correct) since master
> determines that the RS no longer hosts meta at the time of RS abort.
> From a cluster after running ITBLL with CM, I see a lot of {{-splitting}}
> directories left uncleaned:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs
> Found 31 items
> drwxr-xr-x - hbase hadoop 0 2015-06-05 01:14
> /apps/hbase/data/WALs/hregion-58203265
> drwxr-xr-x - hbase hadoop 0 2015-06-05 07:54
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433489308745-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 09:28
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433494382959-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 10:01
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433498252205-splitting
> ...
> {code}
> The directories contain WALs from meta:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting
> Found 2 items
> -rw-r--r-- 3 hbase hadoop 201608 2015-06-05 03:15
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> -rw-r--r-- 3 hbase hadoop 44420 2015-06-05 04:36
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> The RS hosted the meta region for some time:
> {code}
> 2015-06-05 03:14:28,692 INFO [PostOpenDeployTasks:1588230740]
> zookeeper.MetaTableLocator: Setting hbase:meta region location in ZooKeeper
> as os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285
> ...
> 2015-06-05 03:15:17,302 INFO
> [RS_CLOSE_META-os-enis-dal-test-jun-4-5:16020-0] regionserver.HRegion: Closed
> hbase:meta,,1.1588230740
> {code}
> In between, a WAL is created:
> {code}
> 2015-06-05 03:15:11,707 INFO
> [RS_OPEN_META-os-enis-dal-test-jun-4-5:16020-0-MetaLogRoller] wal.FSHLog:
> Rolled WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> with entries=385, filesize=196.88 KB; new WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> When CM killed the region server later master did not see these WAL files:
> {code}
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:46,075
> INFO [MASTER_SERVER_OPERATIONS-os-enis-dal-test-jun-4-3:16000-0]
> master.SplitLogManager: started splitting 2 logs in
> [hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting]
> for [os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285]
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:47,300
> INFO [main-EventThread] wal.WALSplitter: Archived processed log
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> to
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/oldWALs/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:50,497
> INFO [main-EventThread] wal.WALSplitter: Archived
[jira] [Updated] (HBASE-11290) Unlock RegionStates
[ https://issues.apache.org/jira/browse/HBASE-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11290: --- Fix Version/s: (was: 0.98.24) > Unlock RegionStates > --- > > Key: HBASE-11290 > URL: https://issues.apache.org/jira/browse/HBASE-11290 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.2.0, 1.3.0 >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-11290-0.98.patch, HBASE-11290-0.98_v2.patch, > HBASE-11290.01.branch-1.patch, HBASE-11290.02.patch, HBASE-11290.03.patch, > HBASE-11290.draft.patch, HBASE-11290_trunk.patch > > > Even though RegionStates is a highly accessed data structure in HMaster. Most > of it's methods are synchronized. Which limits concurrency. Even simply > making some of the getters non-synchronized by using concurrent data > structures has helped with region assignments. We can go as simple as this > approach or create locks per region or a bucket lock per region bucket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12148: --- Fix Version/s: (was: 0.98.24) (was: 1.3.1) 1.4.0 > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: Walter Koetke > Fix For: 2.0.0, 1.4.0 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.min_and_max_run_independent.patch, 12148.txt, > 12148.txt, 12148v2.txt, 12148v2.txt, 12148v4.patch, HBASE-12148-V3.patch, > HBASE-12148-V3.patch, HBASE-12148.branch-1.v5.patch, > HBASE-12148.branch-1.v5.patch, HBASE-12148.txt, HBASE-12148V2.txt, Screen > Shot 2014-10-01 at 3.39.46 PM.png, Screen Shot 2014-10-01 at 3.41.07 PM.png, > Screen Shot 2016-04-13 at 1.49.30 PM.png, Screen Shot 2016-04-13 at 2.02.22 > PM.png, Screen Shot 2016-05-18 at 10.21.53 PM.png, TimeRangeTracker.tiff > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14049) SnapshotHFileCleaner should optionally clean up after failed snapshots
[ https://issues.apache.org/jira/browse/HBASE-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14049: --- Fix Version/s: (was: 0.98.24) > SnapshotHFileCleaner should optionally clean up after failed snapshots > -- > > Key: HBASE-14049 > URL: https://issues.apache.org/jira/browse/HBASE-14049 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.13 >Reporter: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > > SnapshotHFileCleaner should optionally clean up after failed snapshots rather > than just complain. Add a configuration option that, if set to true > (defaulting to false), instructs SnapshotHFileCleaner to recursively remove > failed snapshot temporary directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14927) Backport HBASE-13014 and HBASE-14749 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14927: --- Summary: Backport HBASE-13014 and HBASE-14749 to branch-1 (was: Backport HBASE-13014 and HBASE-14749 to branch-1 and 0.98) > Backport HBASE-13014 and HBASE-14749 to branch-1 > > > Key: HBASE-14927 > URL: https://issues.apache.org/jira/browse/HBASE-14927 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Fix For: 1.4.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14932) bulkload fails because file not found
[ https://issues.apache.org/jira/browse/HBASE-14932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14932: --- Fix Version/s: (was: 0.98.24) > bulkload fails because file not found > - > > Key: HBASE-14932 > URL: https://issues.apache.org/jira/browse/HBASE-14932 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.98.10 >Reporter: Shuaifeng Zhou > Attachments: HBASE-14932-0.98.patch > > > When make a dobulkload call, one call may contain sevel hfiles to load, but > the call may timeout during regionserver load files, and client will retry to > load. > But when client doing retry call, regionserver may continue doing load > operation, if somefiles success, the retry call will throw filenotfound > exception, and this will cause client retry again and again until retry > exhausted, and bulkload fails. > When this happening, actually, some files are loaded successfully, that's a > inconsistent status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14927) Backport HBASE-13014 and HBASE-14749 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14927: --- Fix Version/s: (was: 0.98.24) > Backport HBASE-13014 and HBASE-14749 to branch-1 > > > Key: HBASE-14927 > URL: https://issues.apache.org/jira/browse/HBASE-14927 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Fix For: 1.4.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[ https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765772#comment-15765772 ] Andrew Purtell commented on HBASE-15130: [~churromorales] Last chance for 0.98.24 is tomorrow. After that, no further planned releases. Get this in or resolve as WontFix? > Backport 0.98 Scan different TimeRange for each column family > -- > > Key: HBASE-15130 > URL: https://issues.apache.org/jira/browse/HBASE-15130 > Project: HBase > Issue Type: Bug > Components: Client, regionserver, Scanners >Affects Versions: 0.98.17 >Reporter: churro morales >Assignee: churro morales > Fix For: 0.98.24 > > Attachments: HBASE-15130-0.98.patch, HBASE-15130-0.98.v1.patch, > HBASE-15130-0.98.v1.patch, HBASE-15130-0.98.v2.patch, > HBASE-15130-0.98.v3.patch, HBASE-15130-0.98.v4.patch > > > branch 98 version backport for HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15580) Tag coprocessor limitedprivate scope to StoreFile.Reader
[
https://issues.apache.org/jira/browse/HBASE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-15580:
---
Fix Version/s: (was: 0.98.24)
> Tag coprocessor limitedprivate scope to StoreFile.Reader
>
>
> Key: HBASE-15580
> URL: https://issues.apache.org/jira/browse/HBASE-15580
> Project: HBase
> Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 2.0.0, 1.0.4, 1.1.7, 1.2.5
>
> Attachments: HBASE-15580.patch, HBASE-15580_branch-1.0.patch
>
>
> For phoenix local indexing we need to have custom storefile reader
> constructor(IndexHalfStoreFileReader) to distinguish from other storefile
> readers. So wanted to mark StoreFile.Reader scope as
> InterfaceAudience.LimitedPrivate("Coprocessor")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15454) Archive store files older than max age
[ https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15454: --- Fix Version/s: (was: 0.98.24) > Archive store files older than max age > -- > > Key: HBASE-15454 > URL: https://issues.apache.org/jira/browse/HBASE-15454 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15454-v1.patch, HBASE-15454-v2.patch, > HBASE-15454-v3.patch, HBASE-15454-v4.patch, HBASE-15454-v5.patch, > HBASE-15454-v6.patch, HBASE-15454-v7.patch, HBASE-15454.patch > > > In date tiered compaction, the store files older than max age are never > touched by minor compactions. Here we introduce a 'freeze window' operation, > which does the follow things: > 1. Find all store files that contains cells whose timestamp are in the give > window. > 2. Compaction all these files and output one file for each window that these > files covered. > After the compaction, we will have only one in the give window, and all cells > whose timestamp are in the give window are in the only file. And if you do > not write new cells with an older timestamp in this window, the file will > never be changed. This makes it easier to do erasure coding on the freezed > file to reduce redundence. And also, it makes it possible to check > consistency between master and peer cluster incrementally. > And why use the word 'freeze'? > Because there is already an 'HFileArchiver' class. I want to use a different > word to prevent confusing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15809) Basic Replication WebUI
[ https://issues.apache.org/jira/browse/HBASE-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15809: --- Fix Version/s: (was: 0.98.24) > Basic Replication WebUI > --- > > Key: HBASE-15809 > URL: https://issues.apache.org/jira/browse/HBASE-15809 > Project: HBase > Issue Type: New Feature > Components: Replication, UI >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Appy >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15809-v0.patch, HBASE-15809-v0.png, > HBASE-15809-v1.patch > > > At the moment the only way to have some insight on replication from the webui > is looking at zkdump and metrics. > the basic information useful to get started debugging are: peer information > and the view of WALs offsets for each peer. > https://reviews.apache.org/r/47275/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16374) IT tests failure in 0.98.21
[
https://issues.apache.org/jira/browse/HBASE-16374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16374:
---
Fix Version/s: (was: 0.98.24)
0.98.25
> IT tests failure in 0.98.21
> ---
>
> Key: HBASE-16374
> URL: https://issues.apache.org/jira/browse/HBASE-16374
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.21
>Reporter: ramkrishna.s.vasudevan
>Assignee: Andrew Purtell
> Fix For: 0.98.25
>
>
> While testing the 0.98.21 RC found these failures
> {code}
> IntegrationTestIngestWithACL
> IntegrationTestMTTR.testKillRsHoldingMeta
> IntegrationTestMTTR.testRestartRsHoldingTable
> {code}
> The first one was a failure and the last 2 are errors.
> {code}
> java.util.concurrent.ExecutionException: java.io.IOException: did timeout
> 6ms waiting for region server to start: stobdtserver5
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR.run(IntegrationTestMTTR.java:317)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR.testKillRsHoldingMeta(IntegrationTestMTTR.java:276)
> Caused by: java.io.IOException: did timeout 6ms waiting for region server
> to start: stobdtserver5
> at
> org.apache.hadoop.hbase.HBaseCluster.waitForRegionServerToStart(HBaseCluster.java:173)
> at
> org.apache.hadoop.hbase.chaos.actions.Action.startRs(Action.java:142)
> at
> org.apache.hadoop.hbase.chaos.actions.RestartActionBaseAction.restartRs(RestartActionBaseAction.java:52)
> at
> org.apache.hadoop.hbase.chaos.actions.RestartRsHoldingMetaAction.perform(RestartRsHoldingMetaAction.java:38)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR$ActionCallable.call(IntegrationTestMTTR.java:585)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR$ActionCallable.call(IntegrationTestMTTR.java:576)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> testRestartRsHoldingTable(org.apache.hadoop.hbase.mttr.IntegrationTestMTTR)
> Time elapsed: 120.533 sec <<< ERROR!
> java.util.concurrent.ExecutionException: java.io.IOException: did timeout
> 6ms waiting for region server to start: stobdtserver5
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR.run(IntegrationTestMTTR.java:317)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR.testRestartRsHoldingTable(IntegrationTestMTTR.java:271)
> Caused by: java.io.IOException: did timeout 6ms waiting for region server
> to start: stobdtserver5
> at
> org.apache.hadoop.hbase.HBaseCluster.waitForRegionServerToStart(HBaseCluster.java:173)
> at
> org.apache.hadoop.hbase.chaos.actions.Action.startRs(Action.java:142)
> at
> org.apache.hadoop.hbase.chaos.actions.RestartActionBaseAction.restartRs(RestartActionBaseAction.java:52)
> at
> org.apache.hadoop.hbase.chaos.actions.RestartRsHoldingTableAction.perform(RestartRsHoldingTableAction.java:56)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR$ActionCallable.call(IntegrationTestMTTR.java:585)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR$ActionCallable.call(IntegrationTestMTTR.java:576)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15982) Interface ReplicationEndpoint extends Guava's Service
[
https://issues.apache.org/jira/browse/HBASE-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-15982:
---
Fix Version/s: (was: 0.98.24)
> Interface ReplicationEndpoint extends Guava's Service
> -
>
> Key: HBASE-15982
> URL: https://issues.apache.org/jira/browse/HBASE-15982
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Purtell
> Fix For: 2.0.0, 1.4.0
>
>
> We have Guava's Service leaking into the LimitedPrivate interface
> ReplicationEndpoint:
> {code}
> public interface ReplicationEndpoint extends Service,
> ReplicationPeerConfigListener
> {code}
> This required a private patch when I updated Guava for our internal
> deployments. This is going to be a problem for us for long term maintenance
> and implenters of pluggable replication endpoints. LP is only less than
> public by a degree. We shouldn't leak types from third part code into either
> Public or LP APIs in my opinion. Let's fix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765764#comment-15765764
]
Andrew Purtell commented on HBASE-16663:
[~ndimiduk] Can we resolve this again now?
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 0.98.24, 1.1.9
>
> Attachments: 16663-branch-1.1.00.patch, 16663.branch-1.1.patch,
> 16663.branch-1.1.patch, HBASE-16663-0.98-V4.patch, HBASE-16663-0.98.patch,
> HBASE-16663-V2.patch, HBASE-16663-V3.patch, HBASE-16663-V4.patch,
> HBASE-16663-branch-1.patch, HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21
