[jira] [Updated] (HBASE-20859) Backup and incremental load could fail in secure clusters
[
https://issues.apache.org/jira/browse/HBASE-20859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20859:
---
Affects Version/s: (was: 2.0.0)
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
> Backup and incremental load could fail in secure clusters
> -
>
> Key: HBASE-20859
> URL: https://issues.apache.org/jira/browse/HBASE-20859
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20859.master.001.patch,
> HBASE-20859.master.002.patch, HBASE-20859.master.003.patch
>
>
> HBase Backup and incremental load uses
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY for temporary path.
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY uses the Java runtime user name
> to generate a temporary path on HDFS. This can be a wrong assumption in a
> secure cluster where Kerberos principal name can be different from the system
> user name.
> {code:java}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This constant variable is used in BackupUtils.java and HFileOutputFormat2.java
> In such cases, you will not be able to write files to the temporary location
> on HDFS due to permission error, and therefore operations such as backup will
> fail.
> This bug is similar in nature to HDFS-12485.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20859) Backup and incremental load could fail in secure clusters
[
https://issues.apache.org/jira/browse/HBASE-20859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535977#comment-16535977
]
Ted Yu commented on HBASE-20859:
lgtm
> Backup and incremental load could fail in secure clusters
> -
>
> Key: HBASE-20859
> URL: https://issues.apache.org/jira/browse/HBASE-20859
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Affects Versions: 2.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-20859.master.001.patch,
> HBASE-20859.master.002.patch, HBASE-20859.master.003.patch
>
>
> HBase Backup and incremental load uses
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY for temporary path.
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY uses the Java runtime user name
> to generate a temporary path on HDFS. This can be a wrong assumption in a
> secure cluster where Kerberos principal name can be different from the system
> user name.
> {code:java}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This constant variable is used in BackupUtils.java and HFileOutputFormat2.java
> In such cases, you will not be able to write files to the temporary location
> on HDFS due to permission error, and therefore operations such as backup will
> fail.
> This bug is similar in nature to HDFS-12485.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20842) Infinite loop when replaying remote wals
[
https://issues.apache.org/jira/browse/HBASE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535955#comment-16535955
]
Duo Zhang commented on HBASE-20842:
---
Pushed to master to see if it helps.
> Infinite loop when replaying remote wals
>
>
> Key: HBASE-20842
> URL: https://issues.apache.org/jira/browse/HBASE-20842
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20842.master.001.patch,
> HBASE-20842.master.002.patch, HBASE-20842.master.002.patch,
> HBASE-20842.master.002.patch
>
>
> {noformat}
> 2018-07-03 12:25:11,375 WARN [RSProcedureDispatcher-pool13-t19]
> replication.SyncReplicationReplayWALRemoteProcedure(107): Replay wals
> [remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
> on asf916.gq1.ygridcore.net,33811,1530620636539 failed for peer id=1
> org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server
> asf916.gq1.ygridcore.net,33811,1530620636539 is not online
> at
> org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$DeadRSRemoteCall.call(RSProcedureDispatcher.java:285)
> at
> org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$DeadRSRemoteCall.call(RSProcedureDispatcher.java:276)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-07-03 12:25:11,440 DEBUG [Thread-2883]
> replication.TestSyncReplicationStandbyKillRS(111): Server
> [asf916.gq1.ygridcore.net,33811,1530620636539] marked as dead, waiting for it
> to finish dead processing
> 2018-07-03 12:25:11,441 DEBUG [Thread-2883]
> replication.TestSyncReplicationStandbyKillRS(114): Server
> [asf916.gq1.ygridcore.net,33811,1530620636539] still being processed, waiting
> 2018-07-03 12:25:11,456 WARN [RS:3;asf916:45751] wal.AbstractFSWAL(419):
> 'hbase.regionserver.maxlogs' was deprecated.
> 2018-07-03 12:25:11,457 INFO [RS:3;asf916:45751] wal.AbstractFSWAL(424): WAL
> configuration: blocksize=256 MB, rollsize=128 MB,
> prefix=asf916.gq1.ygridcore.net%2C45751%2C1530620709275, suffix=,
> logDir=hdfs://localhost:42624/user/jenkins/test-data/a86a805e-162f-5f22-7b9e-573dbf0f40fb/WALs/asf916.gq1.ygridcore.net,45751,1530620709275,
>
> archiveDir=hdfs://localhost:42624/user/jenkins/test-data/a86a805e-162f-5f22-7b9e-573dbf0f40fb/oldWALs
> 2018-07-03 12:25:11,467 DEBUG [RS-EventLoopGroup-14-4]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping
> handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1,
> datanodeId =
> DatanodeInfoWithStorage[127.0.0.1:38997,DS-6002160d-388b-4840-8538-e4c2255108be,DISK]
> 2018-07-03 12:25:11,467 DEBUG [RS-EventLoopGroup-14-5]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping
> handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1,
> datanodeId =
> DatanodeInfoWithStorage[127.0.0.1:45904,DS-e189e3c8-a1bd-475c-86c0-3891e541fc6e,DISK]
> 2018-07-03 12:25:11,467 DEBUG [RS-EventLoopGroup-14-3]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping
> handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1,
> datanodeId =
> DatanodeInfoWithStorage[127.0.0.1:39589,DS-62ced3f8-35c4-4904-80cc-4d514b8f4050,DISK]
> 2018-07-03 12:25:11,495 DEBUG [RegionServerTracker-0]
> procedure2.ProcedureExecutor(887): Stored pid=30,
> state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
> server=asf916.gq1.ygridcore.net,33811,1530620636539, splitWal=true, meta=true
> 2018-07-03 12:25:11,495 DEBUG [RegionServerTracker-0]
> assignment.AssignmentManager(1321):
> Added=asf916.gq1.ygridcore.net,33811,1530620636539 to dead servers, submitted
> shutdown handler to be executed meta=true
> 2018-07-03 12:25:11,498 INFO [PEWorker-6]
> procedure.ServerCrashProcedure(118): Start pid=30,
> state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
> server=asf916.gq1.ygridcore.net,33811,1530620636539, splitWal=true, meta=true
> 2018-07-03 12:25:11,500 WARN [RegionServerTracker-0]
> replication.SyncReplicationReplayWALRemoteProcedure(107): Replay wals
> [remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
> on asf916.gq1.ygridcore.net,33811,1530620636539 failed for peer id=1
> org.apache.hadoop.hbase.DoNotRetryIOException: server not online
> asf916.gq1.ygridcore.net,33811,1530620636539
> at
>
[jira] [Commented] (HBASE-20859) Backup and incremental load could fail in secure clusters
[
https://issues.apache.org/jira/browse/HBASE-20859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535904#comment-16535904
]
Hadoop QA commented on HBASE-20859:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
41s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 32s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
23s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
5s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
13s{color} | {color:green} hbase-backup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20859 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930651/HBASE-20859.master.003.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux dbecd0590c4d 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 361be53344 |
[jira] [Commented] (HBASE-20859) Backup and incremental load could fail in secure clusters
[
https://issues.apache.org/jira/browse/HBASE-20859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535882#comment-16535882
]
Wei-Chiu Chuang commented on HBASE-20859:
-
Thanks very much for comments, [~mdrob], [~reidchan] and [~yuzhih...@gmail.com]
All comments make sense to me so here's my rev003 to address the comments.
Added a test TestHFileOutputFormat2#TestConfigurePartitioner() for
tHFileOutputFormat2#configurePartitioner(). Updated license header and removed
the Deprecated annotation.
> Backup and incremental load could fail in secure clusters
> -
>
> Key: HBASE-20859
> URL: https://issues.apache.org/jira/browse/HBASE-20859
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Affects Versions: 2.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-20859.master.001.patch,
> HBASE-20859.master.002.patch, HBASE-20859.master.003.patch
>
>
> HBase Backup and incremental load uses
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY for temporary path.
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY uses the Java runtime user name
> to generate a temporary path on HDFS. This can be a wrong assumption in a
> secure cluster where Kerberos principal name can be different from the system
> user name.
> {code:java}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This constant variable is used in BackupUtils.java and HFileOutputFormat2.java
> In such cases, you will not be able to write files to the temporary location
> on HDFS due to permission error, and therefore operations such as backup will
> fail.
> This bug is similar in nature to HDFS-12485.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20859) Backup and incremental load could fail in secure clusters
[
https://issues.apache.org/jira/browse/HBASE-20859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HBASE-20859:
Attachment: HBASE-20859.master.003.patch
> Backup and incremental load could fail in secure clusters
> -
>
> Key: HBASE-20859
> URL: https://issues.apache.org/jira/browse/HBASE-20859
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Affects Versions: 2.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-20859.master.001.patch,
> HBASE-20859.master.002.patch, HBASE-20859.master.003.patch
>
>
> HBase Backup and incremental load uses
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY for temporary path.
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY uses the Java runtime user name
> to generate a temporary path on HDFS. This can be a wrong assumption in a
> secure cluster where Kerberos principal name can be different from the system
> user name.
> {code:java}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This constant variable is used in BackupUtils.java and HFileOutputFormat2.java
> In such cases, you will not be able to write files to the temporary location
> on HDFS due to permission error, and therefore operations such as backup will
> fail.
> This bug is similar in nature to HDFS-12485.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20832) Generate CHANGES.md and RELEASENOTES.md for 2.1.0
[
https://issues.apache.org/jira/browse/HBASE-20832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535837#comment-16535837
]
Hudson commented on HBASE-20832:
Results for branch branch-2.1
[build #34 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/34/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/34//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/34//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/34//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Generate CHANGES.md and RELEASENOTES.md for 2.1.0
> -
>
> Key: HBASE-20832
> URL: https://issues.apache.org/jira/browse/HBASE-20832
> Project: HBase
> Issue Type: Sub-task
> Components: documentation
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: HBASE-20832-branch-2.1-addendum-v1.patch,
> HBASE-20832-branch-2.1-addendum.patch, HBASE-20832-v1.patch, HBASE-20832.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18477) Umbrella JIRA for HBase Read Replica clusters
[
https://issues.apache.org/jira/browse/HBASE-18477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535821#comment-16535821
]
Hudson commented on HBASE-18477:
Results for branch HBASE-18477
[build #257 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/257/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/257//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/257//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/257//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/257//console].
> Umbrella JIRA for HBase Read Replica clusters
> -
>
> Key: HBASE-18477
> URL: https://issues.apache.org/jira/browse/HBASE-18477
> Project: HBase
> Issue Type: New Feature
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Attachments: HBase Read-Replica Clusters Scope doc.docx, HBase
> Read-Replica Clusters Scope doc.pdf, HBase Read-Replica Clusters Scope
> doc_v2.docx, HBase Read-Replica Clusters Scope doc_v2.pdf
>
>
> Recently, changes (such as HBASE-17437) have unblocked HBase to run with a
> root directory external to the cluster (such as in Amazon S3). This means
> that the data is stored outside of the cluster and can be accessible after
> the cluster has been terminated. One use case that is often asked about is
> pointing multiple clusters to one root directory (sharing the data) to have
> read resiliency in the case of a cluster failure.
>
> This JIRA is an umbrella JIRA to contain all the tasks necessary to create a
> read-replica HBase cluster that is pointed at the same root directory.
>
> This requires making the Read-Replica cluster Read-Only (no metadata
> operation or data operations).
> Separating the hbase:meta table for each cluster (Otherwise HBase gets
> confused with multiple clusters trying to update the meta table with their ip
> addresses)
> Adding refresh functionality for the meta table to ensure new metadata is
> picked up on the read replica cluster.
> Adding refresh functionality for HFiles for a given table to ensure new data
> is picked up on the read replica cluster.
>
> This can be used with any existing cluster that is backed by an external
> filesystem.
>
> Please note that this feature is still quite manual (with the potential for
> automation later).
>
> More information on this particular feature can be found here:
> https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20859) Backup and incremental load could fail in secure clusters
[
https://issues.apache.org/jira/browse/HBASE-20859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535766#comment-16535766
]
Ted Yu commented on HBASE-20859:
Without fix, the following fails:
{code}
Assert.assertTrue(bulkOutputDir.toString().startsWith(fooHomeDirectory.toString()));
{code}
Can you add some text to the assertion saying why the assertion fails ?
> Backup and incremental load could fail in secure clusters
> -
>
> Key: HBASE-20859
> URL: https://issues.apache.org/jira/browse/HBASE-20859
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Affects Versions: 2.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-20859.master.001.patch,
> HBASE-20859.master.002.patch
>
>
> HBase Backup and incremental load uses
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY for temporary path.
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY uses the Java runtime user name
> to generate a temporary path on HDFS. This can be a wrong assumption in a
> secure cluster where Kerberos principal name can be different from the system
> user name.
> {code:java}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This constant variable is used in BackupUtils.java and HFileOutputFormat2.java
> In such cases, you will not be able to write files to the temporary location
> on HDFS due to permission error, and therefore operations such as backup will
> fail.
> This bug is similar in nature to HDFS-12485.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20859) Backup and incremental load could fail in secure clusters
[
https://issues.apache.org/jira/browse/HBASE-20859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535758#comment-16535758
]
Ted Yu commented on HBASE-20859:
Nice discovery.
Please add license header to TestBackupUtils.
Thanks
> Backup and incremental load could fail in secure clusters
> -
>
> Key: HBASE-20859
> URL: https://issues.apache.org/jira/browse/HBASE-20859
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Affects Versions: 2.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-20859.master.001.patch,
> HBASE-20859.master.002.patch
>
>
> HBase Backup and incremental load uses
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY for temporary path.
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY uses the Java runtime user name
> to generate a temporary path on HDFS. This can be a wrong assumption in a
> secure cluster where Kerberos principal name can be different from the system
> user name.
> {code:java}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This constant variable is used in BackupUtils.java and HFileOutputFormat2.java
> In such cases, you will not be able to write files to the temporary location
> on HDFS due to permission error, and therefore operations such as backup will
> fail.
> This bug is similar in nature to HDFS-12485.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20808) Wrong shutdown order between Chores and ChoreService
[
https://issues.apache.org/jira/browse/HBASE-20808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535756#comment-16535756
]
Hudson commented on HBASE-20808:
Results for branch master
[build #389 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/389/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/389//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/389//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/389//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Wrong shutdown order between Chores and ChoreService
>
>
> Key: HBASE-20808
> URL: https://issues.apache.org/jira/browse/HBASE-20808
> Project: HBase
> Issue Type: Bug
>Reporter: Reid Chan
>Assignee: Nihal Jain
>Priority: Minor
> Labels: beginner
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.6
>
> Attachments: HBASE-20808.branch-1.001.patch,
> HBASE-20808.branch-1.002.patch, HBASE-20808.master.001.patch,
> HBASE-20808.master.002.patch, HBASE-20808.master.003.patch,
> HBASE-20808.master.004.patch, HBASE-20808.master.addendum.patch
>
>
> When stopping master, {{ChoreService}}, which serves all the chores, is
> stopped before canceling all running chores.
> It should cancel all running chores, then shutdown {{ChoreService}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20808) Wrong shutdown order between Chores and ChoreService
[
https://issues.apache.org/jira/browse/HBASE-20808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535720#comment-16535720
]
Hudson commented on HBASE-20808:
Results for branch branch-1
[build #373 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/373/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/373//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/373//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/373//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Wrong shutdown order between Chores and ChoreService
>
>
> Key: HBASE-20808
> URL: https://issues.apache.org/jira/browse/HBASE-20808
> Project: HBase
> Issue Type: Bug
>Reporter: Reid Chan
>Assignee: Nihal Jain
>Priority: Minor
> Labels: beginner
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.6
>
> Attachments: HBASE-20808.branch-1.001.patch,
> HBASE-20808.branch-1.002.patch, HBASE-20808.master.001.patch,
> HBASE-20808.master.002.patch, HBASE-20808.master.003.patch,
> HBASE-20808.master.004.patch, HBASE-20808.master.addendum.patch
>
>
> When stopping master, {{ChoreService}}, which serves all the chores, is
> stopped before canceling all running chores.
> It should cancel all running chores, then shutdown {{ChoreService}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20850) Put up 2.1.0RC0
[
https://issues.apache.org/jira/browse/HBASE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535670#comment-16535670
]
Duo Zhang commented on HBASE-20850:
---
Recreate 2.1.0RC0 tag at this commit
{noformat}
commit dd34297087181e42a4df5ca6f456a2dce04b7ddb
Author: zhangduo
Date: Sat Jul 7 15:39:15 2018 +0800
HBASE-20832 Addendum missed several issues
{noformat}
See
https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=tag;h=955ab56e71e8f5f15b68b54131984c4d9b79da64
> Put up 2.1.0RC0
> ---
>
> Key: HBASE-20850
> URL: https://issues.apache.org/jira/browse/HBASE-20850
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20832) Generate CHANGES.md and RELEASENOTES.md for 2.1.0
[ https://issues.apache.org/jira/browse/HBASE-20832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20832: -- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to branch-2.1. > Generate CHANGES.md and RELEASENOTES.md for 2.1.0 > - > > Key: HBASE-20832 > URL: https://issues.apache.org/jira/browse/HBASE-20832 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.1.0 > > Attachments: HBASE-20832-branch-2.1-addendum-v1.patch, > HBASE-20832-branch-2.1-addendum.patch, HBASE-20832-v1.patch, HBASE-20832.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20832) Generate CHANGES.md and RELEASENOTES.md for 2.1.0
[
https://issues.apache.org/jira/browse/HBASE-20832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535665#comment-16535665
]
Hadoop QA commented on HBASE-20832:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-20832 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930639/HBASE-20832-branch-2.1-addendum-v1.patch
|
| Optional Tests | asflicense |
| uname | Linux c5cba11d0945 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11
12:16:42 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2.1 / 927ac8228f |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Max. process+thread count | 47 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13537/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Generate CHANGES.md and RELEASENOTES.md for 2.1.0
> -
>
> Key: HBASE-20832
> URL: https://issues.apache.org/jira/browse/HBASE-20832
> Project: HBase
> Issue Type: Sub-task
> Components: documentation
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: HBASE-20832-branch-2.1-addendum-v1.patch,
> HBASE-20832-branch-2.1-addendum.patch, HBASE-20832-v1.patch, HBASE-20832.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20832) Generate CHANGES.md and RELEASENOTES.md for 2.1.0
[ https://issues.apache.org/jira/browse/HBASE-20832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20832: -- Attachment: HBASE-20832-branch-2.1-addendum-v1.patch > Generate CHANGES.md and RELEASENOTES.md for 2.1.0 > - > > Key: HBASE-20832 > URL: https://issues.apache.org/jira/browse/HBASE-20832 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.1.0 > > Attachments: HBASE-20832-branch-2.1-addendum-v1.patch, > HBASE-20832-branch-2.1-addendum.patch, HBASE-20832-v1.patch, HBASE-20832.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20639) Implement permission checking through AccessController instead of RSGroupAdminEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535662#comment-16535662
]
Duo Zhang commented on HBASE-20639:
---
OK, searched the commit log, it has been reverted already.
> Implement permission checking through AccessController instead of
> RSGroupAdminEndpoint
> --
>
> Key: HBASE-20639
> URL: https://issues.apache.org/jira/browse/HBASE-20639
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Nihal Jain
>Priority: Major
> Attachments: HBASE-20639.master.001.patch,
> HBASE-20639.master.002.patch, HBASE-20639.master.002.patch
>
>
> Currently permission checking for various RS group operations is done via
> RSGroupAdminEndpoint.
> e.g. in RSGroupAdminServiceImpl#moveServers() :
> {code}
> checkPermission("moveServers");
> groupAdminServer.moveServers(hostPorts, request.getTargetGroup());
> {code}
> The practice in remaining parts of hbase is to perform permission checking
> within AccessController.
> Now that observer hooks for RS group operations are in right place, we should
> follow best practice and move permission checking to AccessController.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20639) Implement permission checking through AccessController instead of RSGroupAdminEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535661#comment-16535661
]
Duo Zhang commented on HBASE-20639:
---
What's the status of this issue? It is reopened without any explanation? And
also no fix versions? For months?
> Implement permission checking through AccessController instead of
> RSGroupAdminEndpoint
> --
>
> Key: HBASE-20639
> URL: https://issues.apache.org/jira/browse/HBASE-20639
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Nihal Jain
>Priority: Major
> Attachments: HBASE-20639.master.001.patch,
> HBASE-20639.master.002.patch, HBASE-20639.master.002.patch
>
>
> Currently permission checking for various RS group operations is done via
> RSGroupAdminEndpoint.
> e.g. in RSGroupAdminServiceImpl#moveServers() :
> {code}
> checkPermission("moveServers");
> groupAdminServer.moveServers(hostPorts, request.getTargetGroup());
> {code}
> The practice in remaining parts of hbase is to perform permission checking
> within AccessController.
> Now that observer hooks for RS group operations are in right place, we should
> follow best practice and move permission checking to AccessController.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20625) refactor some WALCellCodec related code
[
https://issues.apache.org/jira/browse/HBASE-20625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-20625:
--
Fix Version/s: 2.2.0
2.1.0
> refactor some WALCellCodec related code
> ---
>
> Key: HBASE-20625
> URL: https://issues.apache.org/jira/browse/HBASE-20625
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 2.2.0
>
> Attachments: HBASE-20625.branch-2.001.patch,
> HBASE-20625.master.001.patch, HBASE-20625.master.002.patch,
> HBASE-20625.master.002.patch, HBASE-20625.master.003.patch,
> HBASE-20625.master.004.patch, HBASE-20625.master.005.patch,
> HBASE-20625.master.006.patch, HBASE-20625.master.007.patch,
> HBASE-20625.master.008.patch
>
>
> Currently I'm working on export HLog to another FileSystem, then I found the
> code of WALCellCodec and its related classes is not that clean. And there
> are several TODOs. Thus I tried to refactor the code based one these TODOs.
> e.g.
> {code}
> // TODO: it sucks that compression context is in WAL.Entry. It'd be nice if
> it was here.
> // Dictionary could be gotten by enum; initially, based on enum,
> context would create
> // an array of dictionaries.
> static class BaosAndCompressor extends ByteArrayOutputStream implements
> ByteStringCompressor {
> public ByteString toByteString() {
> // We need this copy to create the ByteString as the byte[] 'buf' is
> not immutable. We reuse
> // them.
> return ByteString.copyFrom(this.buf, 0, this.count);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20561) The way we stop a ReplicationSource may cause the RS down
[
https://issues.apache.org/jira/browse/HBASE-20561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-20561:
--
Fix Version/s: 2.2.0
2.1.0
3.0.0
> The way we stop a ReplicationSource may cause the RS down
> -
>
> Key: HBASE-20561
> URL: https://issues.apache.org/jira/browse/HBASE-20561
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.2.0
>
> Attachments: HBASE-20561.master.001.patch,
> HBASE-20561.master.002.patch, HBASE-20561.master.003.patch,
> HBASE-20561.master.004.patch, HBASE-20561.master.005.patch
>
>
> See this:
> https://builds.apache.org/job/HBASE-Flaky-Tests/31125/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.replication.multiwal.TestReplicationKillMasterRSCompressedWithMultipleAsyncWAL-output.txt
> {noformat}
> 2018-05-09 15:07:00,887 INFO [RS_REFRESH_PEER-regionserver/asf916:0-1]
> regionserver.RefreshPeerCallable(52): Received a peer change event, peerId=2,
> type=REMOVE_PEER
> 2018-05-09 15:07:00,890 INFO [RS_REFRESH_PEER-regionserver/asf916:0-1]
> regionserver.ReplicationSource(485): Closing source
> 2-asf916.gq1.ygridcore.net,36287,1525878368395 because: Replication stream
> was removed by a user
> 2018-05-09 15:07:00,892 DEBUG
> [ReplicationExecutor-0.replicationSource,2-asf916.gq1.ygridcore.net,36287,1525878368395.replicationSource.shipperasf916.gq1.ygridcore.net%2C36287%2C1525878368395.asf916.gq1.ygridcore.net%2C36287%2C1525878368395.regiongroup-0,2-asf916.gq1.ygridcore.net,36287,1525878368395]
> zookeeper.ZKWatcher(617): regionserver:34308-0x163456ff2490004,
> quorum=localhost:60149, baseZNode=/1 Received InterruptedException, will
> interrupt current thread and rethrow a SystemErrorException
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:871)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:166)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1231)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1220)
> at
> org.apache.hadoop.hbase.replication.ZKReplicationQueueStorage.removeWAL(ZKReplicationQueueStorage.java:198)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.lambda$cleanOldLogs$8(ReplicationSourceManager.java:526)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.abortWhenFail(ReplicationSourceManager.java:454)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.cleanOldLogs(ReplicationSourceManager.java:526)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.cleanOldLogs(ReplicationSourceManager.java:506)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:489)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.updateLogPosition(ReplicationSourceShipper.java:231)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:133)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:103)
> 2018-05-09 15:07:00,892 DEBUG
> [ReplicationExecutor-0.replicationSource,2-asf916.gq1.ygridcore.net,36287,1525878368395.replicationSource.shipperasf916.gq1.ygridcore.net%2C36287%2C1525878368395.asf916.gq1.ygridcore.net%2C36287%2C1525878368395.regiongroup-1,2-asf916.gq1.ygridcore.net,36287,1525878368395]
> zookeeper.ZKWatcher(617): regionserver:34308-0x163456ff2490004,
> quorum=localhost:60149, baseZNode=/1 Received InterruptedException, will
> interrupt current thread and rethrow a SystemErrorException
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:990)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:910)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:663)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1690)
> at
>
[jira] [Updated] (HBASE-20560) Revisit the TestReplicationDroppedTables ut
[ https://issues.apache.org/jira/browse/HBASE-20560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20560: -- Fix Version/s: 2.2.0 2.1.0 3.0.0 > Revisit the TestReplicationDroppedTables ut > --- > > Key: HBASE-20560 > URL: https://issues.apache.org/jira/browse/HBASE-20560 > Project: HBase > Issue Type: Bug >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.2.0 > > Attachments: HBASE-20560.patch, HBASE-20560.patch > > > After HBASE-20475, the ut TestReplicationDroppedTables has been more stable > now, but still failed in few times.. > Checked the code again, I found the problems: > 1. > https://issues.apache.org/jira/browse/HBASE-20475?focusedCommentId=16465759=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16465759 > 2. > https://issues.apache.org/jira/browse/HBASE-20475?focusedCommentId=16467225=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16467225 > So prepared a patch for the revisiting.. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19936) Introduce a new base class for replication peer procedure
[ https://issues.apache.org/jira/browse/HBASE-19936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19936: -- Fix Version/s: 2.2.0 > Introduce a new base class for replication peer procedure > - > > Key: HBASE-19936 > URL: https://issues.apache.org/jira/browse/HBASE-19936 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.2.0 > > Attachments: HBASE-19936.patch > > > As the sync replication peer state transition will have more steps than > normal replication peer, it will be good to have a common base class for them. > Since the peer id will be stored in this class, I tend to change the protobuf > message name from 'ModifyPeerStateData' to > 'ReplicationPeerProcedureStateData'. This will be committed to master and > HBASE-19397-branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20543) Fix the flaky TestThriftHttpServer
[
https://issues.apache.org/jira/browse/HBASE-20543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-20543:
--
Fix Version/s: 2.2.0
2.1.0
> Fix the flaky TestThriftHttpServer
> --
>
> Key: HBASE-20543
> URL: https://issues.apache.org/jira/browse/HBASE-20543
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.2.0
>
> Attachments: HBASE-20543.v1.patch
>
>
> The runThriftServer() in TestThriftHttpServer, we have the following :
> {code}
> private void runThriftServer(int customHeaderSize) throws Exception {
> //..
> startHttpServerThread(args.toArray(new String[args.size()]));
> // wait up to 10s for the server to start
> for (int i = 0; i < 100
> && (thriftServer.serverRunner == null ||
> thriftServer.serverRunner.httpServer ==
> null); i++) {
> Thread.sleep(100);
> }
> //..
> checkHttpMethods(url);
> //..
> }
> {code}
> The port may still not open even if the thriftServer.serverRunner != null
> and thriftServer.serverRunner.httpServer != null, so the checkHttpMethods
> will get a connection refused ...
> We should wait till the port is really listening
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20481) Replicate entries from same region serially in ReplicationEndpoint for serial replication
[ https://issues.apache.org/jira/browse/HBASE-20481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20481: -- Fix Version/s: 2.2.0 2.1.0 3.0.0 > Replicate entries from same region serially in ReplicationEndpoint for serial > replication > - > > Key: HBASE-20481 > URL: https://issues.apache.org/jira/browse/HBASE-20481 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.2.0 > > Attachments: HBASE-20481.v1.patch, HBASE-20481.v2.patch, > HBASE-20481.v3.patch, HBASE-20481.v3.patch, HBASE-20481.v4.patch > > > When debugging HBASE-20475, [~openinx] found that the > HBaseInterClusterReplicationEndpoint may send the entries for the same > regions concurrently, which breaks the serial replication. > As long as we can have multiple ReplicationEndpoint implementation, just fix > HBaseInterClusterReplicationEndpoint is not enough, we need to add a > setSerial method to ReplicationEndpoint, to tell the implementation that you > should keep the order of the entries from the same region. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19936) Introduce a new base class for replication peer procedure
[ https://issues.apache.org/jira/browse/HBASE-19936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19936: -- Fix Version/s: 2.1.0 > Introduce a new base class for replication peer procedure > - > > Key: HBASE-19936 > URL: https://issues.apache.org/jira/browse/HBASE-19936 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-19936.patch > > > As the sync replication peer state transition will have more steps than > normal replication peer, it will be good to have a common base class for them. > Since the peer id will be stored in this class, I tend to change the protobuf > message name from 'ModifyPeerStateData' to > 'ReplicationPeerProcedureStateData'. This will be committed to master and > HBASE-19397-branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20832) Generate CHANGES.md and RELEASENOTES.md for 2.1.0
[
https://issues.apache.org/jira/browse/HBASE-20832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535658#comment-16535658
]
Hadoop QA commented on HBASE-20832:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-20832 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930637/HBASE-20832-branch-2.1-addendum.patch
|
| Optional Tests | asflicense |
| uname | Linux 5a9efc091b7f 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11
12:16:42 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2.1 / 927ac8228f |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Max. process+thread count | 52 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13536/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Generate CHANGES.md and RELEASENOTES.md for 2.1.0
> -
>
> Key: HBASE-20832
> URL: https://issues.apache.org/jira/browse/HBASE-20832
> Project: HBase
> Issue Type: Sub-task
> Components: documentation
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: HBASE-20832-branch-2.1-addendum.patch,
> HBASE-20832-v1.patch, HBASE-20832.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20832) Generate CHANGES.md and RELEASENOTES.md for 2.1.0
[ https://issues.apache.org/jira/browse/HBASE-20832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20832: -- Attachment: HBASE-20832-branch-2.1-addendum.patch > Generate CHANGES.md and RELEASENOTES.md for 2.1.0 > - > > Key: HBASE-20832 > URL: https://issues.apache.org/jira/browse/HBASE-20832 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.1.0 > > Attachments: HBASE-20832-branch-2.1-addendum.patch, > HBASE-20832-v1.patch, HBASE-20832.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20832) Generate CHANGES.md and RELEASENOTES.md for 2.1.0
[ https://issues.apache.org/jira/browse/HBASE-20832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20832: -- Status: Patch Available (was: Reopened) > Generate CHANGES.md and RELEASENOTES.md for 2.1.0 > - > > Key: HBASE-20832 > URL: https://issues.apache.org/jira/browse/HBASE-20832 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.1.0 > > Attachments: HBASE-20832-branch-2.1-addendum.patch, > HBASE-20832-v1.patch, HBASE-20832.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20771) PUT operation fail with "No server address listed in hbase:meta for region xxxxx"
[
https://issues.apache.org/jira/browse/HBASE-20771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535656#comment-16535656
]
Pankaj Kumar commented on HBASE-20771:
--
[~apurtell] .. Please have a look on 002 patch.
> PUT operation fail with "No server address listed in hbase:meta for region
> x"
> -
>
> Key: HBASE-20771
> URL: https://issues.apache.org/jira/browse/HBASE-20771
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.5.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-20771.branch-1.001.patch,
> HBASE-20771.branch-1.002.patch
>
>
> 1) Create a table with 1 region
> 2) AM send RPC to RS to open the region,
> {code}
> // invoke assignment (async)
> ArrayList userRegionSet = new ArrayList(regions);
> for (Map.Entry> plan: bulkPlan.entrySet()) {
> if (!assign(plan.getKey(), plan.getValue())) {
> for (HRegionInfo region: plan.getValue()) {
> if (!regionStates.isRegionOnline(region)) {
> invokeAssign(region);
> if (!region.getTable().isSystemTable()) {
> userRegionSet.add(region);
> }
> }
> }
> }
> }
> {code}
> In above code if assignment fails (due to some problem) then AM will sumbit
> the assign request to the thread pool and wait for some duration (here it
> will wait for 20sec, region count is 1).
> 3) After 20sec, CreateTableProcedure will set table state as ENABLED in ZK
> and finish the procedure.
> {code}
> // Mark the table as Enabling
> assignmentManager.getTableStateManager().setTableState(tableName,
> ZooKeeperProtos.Table.State.ENABLING);
> // Trigger immediate assignment of the regions in round-robin fashion
> ModifyRegionUtils.assignRegions(assignmentManager, regions);
> // Enable table
> assignmentManager.getTableStateManager()
> .setTableState(tableName, ZooKeeperProtos.Table.State.ENABLED);
> {code}
> 4) At the client side, CreateTableFuture.waitProcedureResult(...) is waiting
> for the procedure to finish,
> {code}
> // If the procedure is no longer running, we should have a result
> if (response != null && response.getState() !=
> GetProcedureResultResponse.State.RUNNING) {
> procResultFound = response.getState() !=
> GetProcedureResultResponse.State.NOT_FOUND;
> return convertResult(response);
> }
> {code}
> Here we wait for operation result only when procedure result not found, but
> in this scenario it will be wrong because region assignment didnt complete,
> {code}
> // if we don't have a proc result, try the compatibility wait
> if (!procResultFound) {
> result = waitOperationResult(deadlineTs);
> }
> {code}
> Since HBaseAdmin didn't wait for operation result (successful region
> assignment), so client PUT operation will fail by the time region is
> successfully opened because "info:server" entry wont be there in meta.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20808) Wrong shutdown order between Chores and ChoreService
[
https://issues.apache.org/jira/browse/HBASE-20808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-20808:
--
Fix Version/s: (was: 2.1.1)
2.1.0
> Wrong shutdown order between Chores and ChoreService
>
>
> Key: HBASE-20808
> URL: https://issues.apache.org/jira/browse/HBASE-20808
> Project: HBase
> Issue Type: Bug
>Reporter: Reid Chan
>Assignee: Nihal Jain
>Priority: Minor
> Labels: beginner
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.6
>
> Attachments: HBASE-20808.branch-1.001.patch,
> HBASE-20808.branch-1.002.patch, HBASE-20808.master.001.patch,
> HBASE-20808.master.002.patch, HBASE-20808.master.003.patch,
> HBASE-20808.master.004.patch, HBASE-20808.master.addendum.patch
>
>
> When stopping master, {{ChoreService}}, which serves all the chores, is
> stopped before canceling all running chores.
> It should cancel all running chores, then shutdown {{ChoreService}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HBASE-20832) Generate CHANGES.md and RELEASENOTES.md for 2.1.0
[ https://issues.apache.org/jira/browse/HBASE-20832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-20832: --- Missed HBASE-20781 in CHANGES.md > Generate CHANGES.md and RELEASENOTES.md for 2.1.0 > - > > Key: HBASE-20832 > URL: https://issues.apache.org/jira/browse/HBASE-20832 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.1.0 > > Attachments: HBASE-20832-v1.patch, HBASE-20832.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20858) port HBASE-20695 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-20858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535643#comment-16535643
]
Hadoop QA commented on HBASE-20858:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
18s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} hbase-server: The patch generated 0 new + 19
unchanged - 2 fixed = 19 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
14s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 23s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}128m 5s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestLoadIncrementalHFiles |
| | hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint |
| | hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
| | hadoop.hbase.regionserver.TestEncryptionKeyRotation |
| | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:1f3957d |
| JIRA
