[jira] [Commented] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531877#comment-15531877
]
Hadoop QA commented on HBASE-16723:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 5s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 58s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 136m 48s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.regionserver.TestHRegionServerBulkLoadWithOldSecureEndpoint |
| | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| Timed out junit tests | org.apache.hadoop.hbase.TestHBaseTestingUtility |
| | org.apache.hadoop.hbase.filter.TestFilterWithScanLimits |
| | org.apache.hadoop.hbase.quotas.TestQuotaThrottle |
| | org.apache.hadoop.hbase.quotas.TestQuotaAdmin |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830847/HBASE-16723-V3.patch |
| JIRA Issue | HBASE-16723 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 2555bed4fdd3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 09a31bd |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3768/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
[jira] [Updated] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16372: --- Priority: Blocker (was: Critical) > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16372_1.patch, HBASE-16372_2.patch, > HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16725) Don't let flushThread hang in TestHRegion
[
https://issues.apache.org/jira/browse/HBASE-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531815#comment-15531815
]
stack commented on HBASE-16725:
---
+1
I don't see how the failures are related given a patch for a test. Are the
failing tests subclassing or reusing this part of TestHRegion?
> Don't let flushThread hang in TestHRegion
> -
>
> Key: HBASE-16725
> URL: https://issues.apache.org/jira/browse/HBASE-16725
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 16725.branch-1.v1.txt, 16725.branch-1.v2.txt,
> 16725.v2.txt
>
>
> I was running TestHRegion locally and observed the following in test output:
> {code}
> 2016-09-28 16:29:36,836 INFO [main] hbase.ResourceChecker(171): after:
> regionserver.TestHRegion#testFlushCacheWhileScanning Thread=50 (was 49)
> Potentially hanging thread: FlushThread
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
>
> org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3834)
> {code}
> Call to flushThread.done() etc should be placed in the finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16725) Don't let flushThread hang in TestHRegion
[
https://issues.apache.org/jira/browse/HBASE-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531801#comment-15531801
]
Hadoop QA commented on HBASE-16725:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 35s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 54s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 133m 41s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestIncrementFromClientSideWithCoprocessor
|
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830840/16725.v2.txt |
| JIRA Issue | HBASE-16725 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 92fca99b17f9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 09a31bd |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3767/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3767/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-16678) MapReduce jobs do not update counters from ScanMetrics
[
https://issues.apache.org/jira/browse/HBASE-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531759#comment-15531759
]
Jerry He commented on HBASE-16678:
--
+1
> MapReduce jobs do not update counters from ScanMetrics
> --
>
> Key: HBASE-16678
> URL: https://issues.apache.org/jira/browse/HBASE-16678
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16678_v1.patch
>
>
> Was inspecting a perf issue, where we needed the scanner metrics as counters
> for a MR job. Turns out that the HBase scan counters are no longer working in
> 1.0+. I think it got broken via HBASE-13030.
> These are the counters:
> {code}
> HBase Counters
> BYTES_IN_REMOTE_RESULTS=0
> BYTES_IN_RESULTS=280
> MILLIS_BETWEEN_NEXTS=11
> NOT_SERVING_REGION_EXCEPTION=0
> NUM_SCANNER_RESTARTS=0
> NUM_SCAN_RESULTS_STALE=0
> REGIONS_SCANNED=1
> REMOTE_RPC_CALLS=0
> REMOTE_RPC_RETRIES=0
> RPC_CALLS=3
> RPC_RETRIES=0
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531747#comment-15531747
]
Hadoop QA commented on HBASE-16721:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 11s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 11s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 128m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestTableSnapshotScanner |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830834/hbase-16721_v2.master.patch
|
| JIRA Issue | HBASE-16721 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux da73c8acacde 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 09a31bd |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3766/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3766/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3766/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3766/console |
| Powered
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531715#comment-15531715
]
Hadoop QA commented on HBASE-16721:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} branch-1 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 56s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} branch-1 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 21s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 21s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 116m 2s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-29 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830833/hbase-16721_v2.branch-1.patch
|
| JIRA Issue | HBASE-16721 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux abcba1203533 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh
[jira] [Updated] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-16663:
-
Description:
After HBASE-16284, unauthorized user will not able allowed to stop
HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
will be stopped before AccessController validation.
hbase-site.xml,
{noformat}
hbase.coprocessor.master.classes
org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
hbase.coprocessor.regionserver.classes
org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
{noformat}
HBaseAdmin.stopMaster(),
{noformat}
2016-09-20 21:12:26,796 INFO
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] hbase.JMXListener:
ConnectorServer stopped!
2016-09-20 21:13:55,380 WARN
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
user P72981
ExitCodeException exitCode=1: id: P72981: No such user
2016-09-20 21:14:00,495 ERROR
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
master.MasterRpcServices: Exception occurred while stopping master
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'P72981' (global, action=ADMIN)
at
org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
at
org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
at
org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
at
org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
{noformat}
HBaseAdmin.stopRegionServer(rs-host-port),
{noformat}
2016-09-20 20:59:01,234 INFO
[RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] hbase.JMXListener:
ConnectorServer stopped!
2016-09-20 20:59:01,250 WARN
[RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
user P72981
ExitCodeException exitCode=1: id: P72981: No such user
2016-09-20 20:59:01,253 WARN
[RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
regionserver.HRegionServer: The region server did not stop
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'P72981' (global, action=ADMIN)
at
org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
at
org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
at
org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
at
org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
at
org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
at
org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
{noformat}
HBaseAdmin.shutdown(),
{noformat}
2016-09-21 12:09:08,259 INFO
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
master.MasterRpcServices: Client=P72981//10.18.248.96 shutdown
2016-09-21 12:09:08,261 INFO
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] hbase.JMXListener:
ConnectorServer stopped!
2016-09-21 12:09:08,276 WARN
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
user P72981
ExitCodeException exitCode=1: id: P72981: No such user
2016-09-21 12:09:08,280 ERROR
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
master.MasterRpcServices: Exception occurred in HMaster.shutdown()
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'P72981' (global, action=ADMIN)
at
org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
at
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531677#comment-15531677
]
Hadoop QA commented on HBASE-14123:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 3s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 5s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 48 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 23s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 14m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
5s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 47s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 56s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 15m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch has 666 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 17s
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 44s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 30s
{color} | {color:red} hbase-client generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 45s
{color} | {color:red} hbase-server generated 2 new + 0 unchanged - 0 fixed = 2
total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 16s
{color} | {color:red} hbase-client generated 4 new + 14 unchanged - 0 fixed =
18 total (was 14) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 45s
{color} | {color:red} root generated 4 new + 20 unchanged - 0 fixed = 24 total
(was 20) {color} |
|
[jira] [Commented] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531665#comment-15531665
]
Pankaj Kumar commented on HBASE-16723:
--
Thanks for the review [~tedyu], addressed the FindBugs warningsin V3 patch,
test case failures are not relevant.
> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-16723-V2.patch, HBASE-16723-V3.patch,
> HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX
> Connector server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-16723:
-
Attachment: HBASE-16723-V3.patch
> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-16723-V2.patch, HBASE-16723-V3.patch,
> HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX
> Connector server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-16725) Don't let flushThread hang in TestHRegion
[
https://issues.apache.org/jira/browse/HBASE-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-16725:
--
Assignee: Ted Yu
> Don't let flushThread hang in TestHRegion
> -
>
> Key: HBASE-16725
> URL: https://issues.apache.org/jira/browse/HBASE-16725
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 16725.branch-1.v1.txt, 16725.branch-1.v2.txt,
> 16725.v2.txt
>
>
> I was running TestHRegion locally and observed the following in test output:
> {code}
> 2016-09-28 16:29:36,836 INFO [main] hbase.ResourceChecker(171): after:
> regionserver.TestHRegion#testFlushCacheWhileScanning Thread=50 (was 49)
> Potentially hanging thread: FlushThread
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
>
> org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3834)
> {code}
> Call to flushThread.done() etc should be placed in the finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16725) Don't let flushThread hang in TestHRegion
[
https://issues.apache.org/jira/browse/HBASE-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16725:
---
Attachment: 16725.v2.txt
Patch for master branch.
> Don't let flushThread hang in TestHRegion
> -
>
> Key: HBASE-16725
> URL: https://issues.apache.org/jira/browse/HBASE-16725
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 16725.branch-1.v1.txt, 16725.branch-1.v2.txt,
> 16725.v2.txt
>
>
> I was running TestHRegion locally and observed the following in test output:
> {code}
> 2016-09-28 16:29:36,836 INFO [main] hbase.ResourceChecker(171): after:
> regionserver.TestHRegion#testFlushCacheWhileScanning Thread=50 (was 49)
> Potentially hanging thread: FlushThread
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
>
> org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3834)
> {code}
> Call to flushThread.done() etc should be placed in the finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16678) MapReduce jobs do not update counters from ScanMetrics
[
https://issues.apache.org/jira/browse/HBASE-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531549#comment-15531549
]
Enis Soztutar commented on HBASE-16678:
---
Any quick reviews? The test failures are unrelated.
> MapReduce jobs do not update counters from ScanMetrics
> --
>
> Key: HBASE-16678
> URL: https://issues.apache.org/jira/browse/HBASE-16678
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16678_v1.patch
>
>
> Was inspecting a perf issue, where we needed the scanner metrics as counters
> for a MR job. Turns out that the HBase scan counters are no longer working in
> 1.0+. I think it got broken via HBASE-13030.
> These are the counters:
> {code}
> HBase Counters
> BYTES_IN_REMOTE_RESULTS=0
> BYTES_IN_RESULTS=280
> MILLIS_BETWEEN_NEXTS=11
> NOT_SERVING_REGION_EXCEPTION=0
> NUM_SCANNER_RESTARTS=0
> NUM_SCAN_RESULTS_STALE=0
> REGIONS_SCANNED=1
> REMOTE_RPC_CALLS=0
> REMOTE_RPC_RETRIES=0
> RPC_CALLS=3
> RPC_RETRIES=0
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-16721:
--
Attachment: hbase-16721_v2.master.patch
Attaching the master patch as well. On master, the test does not fail, since as
explained about we do not unlock the updatesLock until the transaction is fully
complete (WAL.append() went through). Hence the master patch is only the test
case.
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.0, 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16721_v1.branch-1.patch,
> hbase-16721_v2.branch-1.patch, hbase-16721_v2.master.patch
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531515#comment-15531515
]
Enis Soztutar edited comment on HBASE-16721 at 9/29/16 1:49 AM:
I was able to come up with a unit test for this. It is a bit involved, but
basically simulates the situation by holding the threads at appropriate times.
As any good unit test should, it fails on without the patch:
{code}
java.lang.AssertionError: Found seqId for the region which is already flushed
Expected :-1
Actual :3
{code}
Let me see whether we need the patch in master.
was (Author: enis):
I was able to come up with a unit test for this. It is a bit involved, but
basically simulates the situation with holding the threads at appropriate
times. As any good unit test should, it fails on without the patch:
{code}
java.lang.AssertionError: Found seqId for the region which is already flushed
Expected :-1
Actual :3
{code}
Let me see whether we need the patch in master.
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.0, 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16721_v1.branch-1.patch,
> hbase-16721_v2.branch-1.patch
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-16721:
--
Attachment: hbase-16721_v2.branch-1.patch
I was able to come up with a unit test for this. It is a bit involved, but
basically simulates the situation with holding the threads at appropriate
times. As any good unit test should, it fails on without the patch:
{code}
java.lang.AssertionError: Found seqId for the region which is already flushed
Expected :-1
Actual :3
{code}
Let me see whether we need the patch in master.
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.0, 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16721_v1.branch-1.patch,
> hbase-16721_v2.branch-1.patch
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16725) Don't let flushThread hang in TestHRegion
[
https://issues.apache.org/jira/browse/HBASE-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531511#comment-15531511
]
Appy commented on HBASE-16725:
--
+1
minor suggestion: Prefer using String.format in such cases.
{noformat}"toggle="+toggle+"i=" + i + " ts="+System.currentTimeMillis()
{noformat}
> Don't let flushThread hang in TestHRegion
> -
>
> Key: HBASE-16725
> URL: https://issues.apache.org/jira/browse/HBASE-16725
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 16725.branch-1.v1.txt, 16725.branch-1.v2.txt
>
>
> I was running TestHRegion locally and observed the following in test output:
> {code}
> 2016-09-28 16:29:36,836 INFO [main] hbase.ResourceChecker(171): after:
> regionserver.TestHRegion#testFlushCacheWhileScanning Thread=50 (was 49)
> Potentially hanging thread: FlushThread
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
>
> org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3834)
> {code}
> Call to flushThread.done() etc should be placed in the finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531345#comment-15531345
]
Hadoop QA commented on HBASE-16721:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 31s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
23s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} branch-1 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 8s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} branch-1 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 27s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 84m 42s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 6s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-28 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830795/hbase-16721_v1.branch-1.patch
|
| JIRA Issue | HBASE-16721 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f215a66d2783 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-16727) Backup refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531237#comment-15531237 ] stack commented on HBASE-16727: --- +1 from me. Thanks Vladimir. > Backup refactoring: move MR dependencies from HMaster > - > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16711) Fix hadoop-3.0 profile compile
[ https://issues.apache.org/jira/browse/HBASE-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531203#comment-15531203 ] Hudson commented on HBASE-16711: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1691 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1691/]) HBASE-16711 Fix hadoop-3.0 profile compile (jmhsieh: rev 09a31bd1e9476106d9c4700e850627f5b2822c18) * (edit) hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java > Fix hadoop-3.0 profile compile > -- > > Key: HBASE-16711 > URL: https://issues.apache.org/jira/browse/HBASE-16711 > Project: HBase > Issue Type: Bug > Components: build, hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-16711.v0.patch > > > The -Dhadoop.profile=3.0 build is failing currently due to code deprecated in > hadoop2 and removed in hadoop3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531200#comment-15531200
]
stack commented on HBASE-16721:
---
Good one. Rationale is reasonable. I like how 'no change', just reorder. Trying
to come up w/ a test would be hard. You know it intimately now though. Possible
to manufacture at all?
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.0, 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16721_v1.branch-1.patch
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531083#comment-15531083
]
Ted Yu commented on HBASE-16723:
Please address FindBugs warning.
> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-16723-V2.patch, HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX
> Connector server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531048#comment-15531048
]
Hadoop QA commented on HBASE-16179:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 21s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 57s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 1m
14s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 3m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 45s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
45s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 51s
{color} | {color:red} root generated 1 new + 19 unchanged - 1 fixed = 20 total
(was 20) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 9s
{color} | {color:red} hbase-spark generated 1 new + 17 unchanged - 1 fixed = 18
total (was 18) {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m 52s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m 25s
{color} | {color:red} hbase-spark in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 11s
{color} | {color:green} hbase-spark2.0-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 12s
{color} | {color:green} hbase-spark1.6-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 11s
{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 160m 34s {color}
| {color:black} {color} |
\\
\\
|| Reason ||
[jira] [Commented] (HBASE-16725) Don't let flushThread hang in TestHRegion
[
https://issues.apache.org/jira/browse/HBASE-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531037#comment-15531037
]
Hadoop QA commented on HBASE-16725:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} branch-1 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 1s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} branch-1 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 49s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 34s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 124m 47s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.replication.regionserver.TestReplicationSourceManager |
| | hadoop.hbase.mapred.TestMultiTableSnapshotInputFormat |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-28 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830775/16725.branch-1.v2.txt
|
| JIRA Issue | HBASE-16725 |
|
[jira] [Updated] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-16721:
--
Status: Patch Available (was: Open)
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.2.0, 1.1.0, 1.0.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16721_v1.branch-1.patch
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-16721:
--
Attachment: hbase-16721_v1.branch-1.patch
This should solve the problem for branch-1. On master, we may not have the same
problem since in master, we keep holding the region read lock until the
existing transaction is already sync'ed.
The patch makes it so that after getting the region write lock, we also wait
for the ongoing transactions before starting the FSHLog.startCacheFlush() call.
We already wait for all transactions to finish under the region write lock, so
semantically the behavior should not change (would be good if the reviewers
verify this independently).
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.0, 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16721_v1.branch-1.patch
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530997#comment-15530997
]
Jerry He commented on HBASE-16721:
--
bq. Do you know that whether these regions were opened before on this host.
I don't have access to the complete logs to confirm that now. My impression is
they were.
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.0, 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530809#comment-15530809 ] Ted Yu edited comment on HBASE-14123 at 9/28/16 9:13 PM: - Patch v27 sync's up with commit e35f7b920e80daa79631aa3c8c9846405658be21 over HBASE-7912 branch. was (Author: yuzhih...@gmail.com): Patch v20 sync's up with commit e35f7b920e80daa79631aa3c8c9846405658be21 over HBASE-7912 branch. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v3.txt, 14123-master.v5.txt, > 14123-master.v6.txt, 14123-master.v7.txt, 14123-master.v8.txt, > 14123-master.v9.txt, 14123-v14.txt, HBASE-14123-for-7912-v1.patch, > HBASE-14123-for-7912-v6.patch, HBASE-14123-v1.patch, HBASE-14123-v10.patch, > HBASE-14123-v11.patch, HBASE-14123-v12.patch, HBASE-14123-v13.patch, > HBASE-14123-v15.patch, HBASE-14123-v16.patch, HBASE-14123-v2.patch, > HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch, > HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14123: --- Attachment: 14123-master.v27.txt > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v3.txt, 14123-master.v5.txt, > 14123-master.v6.txt, 14123-master.v7.txt, 14123-master.v8.txt, > 14123-master.v9.txt, 14123-v14.txt, HBASE-14123-for-7912-v1.patch, > HBASE-14123-for-7912-v6.patch, HBASE-14123-v1.patch, HBASE-14123-v10.patch, > HBASE-14123-v11.patch, HBASE-14123-v12.patch, HBASE-14123-v13.patch, > HBASE-14123-v15.patch, HBASE-14123-v16.patch, HBASE-14123-v2.patch, > HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch, > HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9465) Push entries to peer clusters serially
[
https://issues.apache.org/jira/browse/HBASE-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530835#comment-15530835
]
Vincent Poon commented on HBASE-9465:
-
I think you need the code addition above 'continue" here - if readAllEntries...
returns true, then lastPositionsForSerialScope should always be empty?
{code}
if (readAllEntriesToReplicateOrNextFile(currentWALisBeingWrittenTo, entries,
lastPositionsForSerialScope)) {
for (Map.Entry entry :
lastPositionsForSerialScope.entrySet()) {
waitingUntilCanPush(entry);
}
try {
MetaTableAccessor
.updateReplicationPositions(manager.getConnection(),
actualPeerId,
lastPositionsForSerialScope);
} catch (IOException e) {
LOG.error("updateReplicationPositions fail", e);
stopper.stop("updateReplicationPositions fail");
}
continue;
}
{code}
> Push entries to peer clusters serially
> --
>
> Key: HBASE-9465
> URL: https://issues.apache.org/jira/browse/HBASE-9465
> Project: HBase
> Issue Type: New Feature
> Components: regionserver, Replication
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Honghua Feng
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-9465-branch-1-v1.patch,
> HBASE-9465-branch-1-v1.patch, HBASE-9465-branch-1-v2.patch,
> HBASE-9465-branch-1-v3.patch, HBASE-9465-branch-1-v4.patch,
> HBASE-9465-branch-1-v4.patch, HBASE-9465-v1.patch, HBASE-9465-v2.patch,
> HBASE-9465-v2.patch, HBASE-9465-v3.patch, HBASE-9465-v4.patch,
> HBASE-9465-v5.patch, HBASE-9465-v6.patch, HBASE-9465-v6.patch,
> HBASE-9465-v7.patch, HBASE-9465-v7.patch, HBASE-9465.pdf
>
>
> When region-move or RS failure occurs in master cluster, the hlog entries
> that are not pushed before region-move or RS-failure will be pushed by
> original RS(for region move) or another RS which takes over the remained hlog
> of dead RS(for RS failure), and the new entries for the same region(s) will
> be pushed by the RS which now serves the region(s), but they push the hlog
> entries of a same region concurrently without coordination.
> This treatment can possibly lead to data inconsistency between master and
> peer clusters:
> 1. there are put and then delete written to master cluster
> 2. due to region-move / RS-failure, they are pushed by different
> replication-source threads to peer cluster
> 3. if delete is pushed to peer cluster before put, and flush and
> major-compact occurs in peer cluster before put is pushed to peer cluster,
> the delete is collected and the put remains in peer cluster
> In this scenario, the put remains in peer cluster, but in master cluster the
> put is masked by the delete, hence data inconsistency between master and peer
> clusters
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530809#comment-15530809 ] Ted Yu commented on HBASE-14123: Patch v20 sync's up with commit e35f7b920e80daa79631aa3c8c9846405658be21 over HBASE-7912 branch. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v3.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530762#comment-15530762 ] Ben Manes commented on HBASE-15560: --- [~busbey], [~eshcar]: The remaining issues appear unrelated to my changes. > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16673) Enhance history command: history per backup set
[ https://issues.apache.org/jira/browse/HBASE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16673: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the patch, Vlad. > Enhance history command: history per backup set > --- > > Key: HBASE-16673 > URL: https://issues.apache.org/jira/browse/HBASE-16673 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16673-v1.patch, HBASE-16673-v2.patch > > > New command-line option: -set setName. Will retrieve backup history for a > particular backup set -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16727) Backup refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16727: -- Summary: Backup refactoring: move MR dependencies from HMaster (was: Refactoring: move MR dependencies from HMaster) > Backup refactoring: move MR dependencies from HMaster > - > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16711) Fix hadoop-3.0 profile compile
[ https://issues.apache.org/jira/browse/HBASE-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-16711: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for reviewing [~busbey] > Fix hadoop-3.0 profile compile > -- > > Key: HBASE-16711 > URL: https://issues.apache.org/jira/browse/HBASE-16711 > Project: HBase > Issue Type: Bug > Components: build, hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-16711.v0.patch > > > The -Dhadoop.profile=3.0 build is failing currently due to code deprecated in > hadoop2 and removed in hadoop3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16711) Fix hadoop-3.0 profile compile
[ https://issues.apache.org/jira/browse/HBASE-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530737#comment-15530737 ] Jonathan Hsieh commented on HBASE-16711: Agreed. filed HBASE-16728 > Fix hadoop-3.0 profile compile > -- > > Key: HBASE-16711 > URL: https://issues.apache.org/jira/browse/HBASE-16711 > Project: HBase > Issue Type: Bug > Components: build, hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-16711.v0.patch > > > The -Dhadoop.profile=3.0 build is failing currently due to code deprecated in > hadoop2 and removed in hadoop3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16728) Add hadoop.profile=3.0 pass to precommit checks.
Jonathan Hsieh created HBASE-16728: -- Summary: Add hadoop.profile=3.0 pass to precommit checks. Key: HBASE-16728 URL: https://issues.apache.org/jira/browse/HBASE-16728 Project: HBase Issue Type: Task Components: build, hadoop3 Affects Versions: 2.0.0 Reporter: Jonathan Hsieh Fix For: 2.0.0 once we get hadoop3 profile compiling (mvn test -DskipTests) and passing licensing muster (mvn install), we should add precommit checks to prevent regressions in these categories -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16725) Don't let flushThread hang in TestHRegion
[
https://issues.apache.org/jira/browse/HBASE-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16725:
---
Attachment: 16725.branch-1.v2.txt
Patch v2 addresses Appy's comments.
> Don't let flushThread hang in TestHRegion
> -
>
> Key: HBASE-16725
> URL: https://issues.apache.org/jira/browse/HBASE-16725
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 16725.branch-1.v1.txt, 16725.branch-1.v2.txt
>
>
> I was running TestHRegion locally and observed the following in test output:
> {code}
> 2016-09-28 16:29:36,836 INFO [main] hbase.ResourceChecker(171): after:
> regionserver.TestHRegion#testFlushCacheWhileScanning Thread=50 (was 49)
> Potentially hanging thread: FlushThread
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
>
> org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3834)
> {code}
> Call to flushThread.done() etc should be placed in the finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16727) Refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530706#comment-15530706 ] Matteo Bertozzi commented on HBASE-16727: - +1 for me > Refactoring: move MR dependencies from HMaster > -- > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16727) Refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530686#comment-15530686 ] Vladimir Rodionov commented on HBASE-16727: --- [~mbertozzi], [~saint@gmail.com], [~apurtell] what do you think? It is time now to give +-1 on this before we start refactoring. > Refactoring: move MR dependencies from HMaster > -- > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16673) Enhance history command: history per backup set
[ https://issues.apache.org/jira/browse/HBASE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530690#comment-15530690 ] Vladimir Rodionov commented on HBASE-16673: --- [~tedyu], ping. > Enhance history command: history per backup set > --- > > Key: HBASE-16673 > URL: https://issues.apache.org/jira/browse/HBASE-16673 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16673-v1.patch, HBASE-16673-v2.patch > > > New command-line option: -set setName. Will retrieve backup history for a > particular backup set -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16727) Refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530684#comment-15530684 ] Ted Yu commented on HBASE-16727: I am fine with creating HBASE-14123 branch based on the master branch which catches up to the tip of HBASE-7912 branch. > Refactoring: move MR dependencies from HMaster > -- > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16727) Refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530677#comment-15530677 ] Vladimir Rodionov commented on HBASE-16727: --- Yes. We pull MR stuff out from Master. But we have to keep it in server module due to dependencies on server internal API (HFile/WAL reader writer classes) > Refactoring: move MR dependencies from HMaster > -- > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16721:
Affects Version/s: 1.0.0
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.0, 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16721:
Priority: Critical (was: Major)
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16721:
Affects Version/s: 1.1.0
1.2.0
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16721:
Component/s: wal
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16727) Refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530653#comment-15530653 ] Devaraj Das commented on HBASE-16727: - Right [~busbey]... The title is probably a tad misleading. It's about removing the dependency on the MR runtime from the master that the code in the backup branch introduced. > Refactoring: move MR dependencies from HMaster > -- > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-16727) Refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-16727: Comment: was deleted (was: Right [~busbey]... The title is probably a tad misleading. It's about removing the dependency on the MR runtime from the master that the code in the backup branch introduced.) > Refactoring: move MR dependencies from HMaster > -- > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16727) Refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530652#comment-15530652 ] Devaraj Das commented on HBASE-16727: - Right [~busbey]... The title is probably a tad misleading. It's about removing the dependency on the MR runtime from the master that the code in the backup branch introduced. > Refactoring: move MR dependencies from HMaster > -- > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[
https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530647#comment-15530647
]
Sean Busbey commented on HBASE-16712:
-
{code}
+#if( ${dep.licenses.isEmpty()} )
+ERROR: This product includes ${dep.name} which has no licenses!
+Revert the change if invalid or if intentional add license info to
sumpplemental-models.xml
+groupId: ${dep.groupId}
+articfactId: ${dep.artifactId}
+packaging: ${dep.packaging}
+version: ${dep.version}
+url: ${dep.url}
+#end
{code}
I don't think this will actually error out the build. Could you include a
failure inducing error with a pointer for fixing, [like the out of bounds
access in
LICENSE.vm|https://github.com/apache/hbase/blob/e51fcdd7788271d33cf02240155429358d71627b/hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm#L1675]?
> fix hadoop-3.0 profile mvn install
> --
>
> Key: HBASE-16712
> URL: https://issues.apache.org/jira/browse/HBASE-16712
> Project: HBase
> Issue Type: Bug
> Components: build, hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16712.v0.patch, hbase-16712.v1.patch
>
>
> After the compile is fixed, mvn install fails due to transitive dependencies
> coming from hadoop3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16179: --- Attachment: 16179.v10.txt Patch v10 adopts Imran's comment above. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[
https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530644#comment-15530644
]
Sean Busbey commented on HBASE-16712:
-
{code}
+## See this FAQ link for justifications:
https://www.apache.org/legal/resolved.html
+## Note 'The Go license' is BSD 3 clause verbatim.
+#set($non_aggregate_fine = [ 'Public Domain', 'New BSD license', 'BSD
license', 'Mozilla Public License Version 2.0', 'Creative Commons Attribution
License, Version 2.5', 'The Go license', 'MPL 1.1'])
{code}
what's bringing in the go license, and can we update the uspplementation
information for it to properly name the BSD 3 clause instead? As-is, I think
this changeset will improperly include an entry in NOTICE for such a licensed
work, when it should only be in LICENSE.
> fix hadoop-3.0 profile mvn install
> --
>
> Key: HBASE-16712
> URL: https://issues.apache.org/jira/browse/HBASE-16712
> Project: HBase
> Issue Type: Bug
> Components: build, hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16712.v0.patch, hbase-16712.v1.patch
>
>
> After the compile is fixed, mvn install fails due to transitive dependencies
> coming from hadoop3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16711) Fix hadoop-3.0 profile compile
[ https://issues.apache.org/jira/browse/HBASE-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530635#comment-15530635 ] Sean Busbey commented on HBASE-16711: - +1 can we make sure there's a jira to track making precommit tests work against the hadoop-3 profile? I fear without that we're not going to maintain this compatibility long. > Fix hadoop-3.0 profile compile > -- > > Key: HBASE-16711 > URL: https://issues.apache.org/jira/browse/HBASE-16711 > Project: HBase > Issue Type: Bug > Components: build, hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-16711.v0.patch > > > The -Dhadoop.profile=3.0 build is failing currently due to code deprecated in > hadoop2 and removed in hadoop3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[ https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530625#comment-15530625 ] Sean Busbey commented on HBASE-16712: - another bit on that Xerces dependency, there's supposed to be an enforcer rule that barfs on xercesImpl; did it not correctly yell about the inclusion? I don't see it getting removed in the patch. looking at the brought in dependencies, do you happen to have a tree list handy of how they get brought in? can we exclude more of them (curator-test, for example, is extremely suspicious as something needed to talk to HDFS)? Is the hadoop 3 profile making use of a different Hadoop dependency than Hadoop 2 profile? A quick skim shows that it includes a dependency on hadoop-annotations, is that needed? > fix hadoop-3.0 profile mvn install > -- > > Key: HBASE-16712 > URL: https://issues.apache.org/jira/browse/HBASE-16712 > Project: HBase > Issue Type: Bug > Components: build, hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-16712.v0.patch, hbase-16712.v1.patch > > > After the compile is fixed, mvn install fails due to transitive dependencies > coming from hadoop3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16712) fix hadoop-3.0 profile mvn install
[
https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530603#comment-15530603
]
Sean Busbey edited comment on HBASE-16712 at 9/28/16 7:22 PM:
--
{code}
+
+
+
+ xerces
+ xercesImpl
+
+
+
+ Apache License, Version 2.0
+ http://www.apache.org/licenses/LICENSE-2.0.txt
+ repo
+
+
+
+
{code}
I've worked hard to keep Xerces out of our dependency set. Historically
Hadoop's use of the dependency has been limited to some tools that we don't
directly use (IIRC, the fsimage exporter). Is this dependency now required or
can we keep it out?
was (Author: busbey):
{quote}
+
+
+
+ xerces
+ xercesImpl
+
+
+
+ Apache License, Version 2.0
+ http://www.apache.org/licenses/LICENSE-2.0.txt
+ repo
+
+
+
+
{quote}
I've worked hard to keep Xerces out of our dependency set. Historically
Hadoop's use of the dependency has been limited to some tools that we don't
directly use (IIRC, the fsimage exporter). Is this dependency now required or
can we keep it out?
> fix hadoop-3.0 profile mvn install
> --
>
> Key: HBASE-16712
> URL: https://issues.apache.org/jira/browse/HBASE-16712
> Project: HBase
> Issue Type: Bug
> Components: build, hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16712.v0.patch, hbase-16712.v1.patch
>
>
> After the compile is fixed, mvn install fails due to transitive dependencies
> coming from hadoop3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[
https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530603#comment-15530603
]
Sean Busbey commented on HBASE-16712:
-
{quote}
+
+
+
+ xerces
+ xercesImpl
+
+
+
+ Apache License, Version 2.0
+ http://www.apache.org/licenses/LICENSE-2.0.txt
+ repo
+
+
+
+
{quote}
I've worked hard to keep Xerces out of our dependency set. Historically
Hadoop's use of the dependency has been limited to some tools that we don't
directly use (IIRC, the fsimage exporter). Is this dependency now required or
can we keep it out?
> fix hadoop-3.0 profile mvn install
> --
>
> Key: HBASE-16712
> URL: https://issues.apache.org/jira/browse/HBASE-16712
> Project: HBase
> Issue Type: Bug
> Components: build, hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16712.v0.patch, hbase-16712.v1.patch
>
>
> After the compile is fixed, mvn install fails due to transitive dependencies
> coming from hadoop3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530602#comment-15530602
]
Devaraj Das commented on HBASE-16721:
-
In one occurrence of this issue, the region was actually non-existent. It was a
region that was split into two regions, and the daughters were opened fine. But
the flush continued to be blocked.
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16727) Refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530594#comment-15530594 ] Sean Busbey commented on HBASE-16727: - this is just the MR stuff for backup/restore, right? > Refactoring: move MR dependencies from HMaster > -- > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16725) Don't let flushThread hang in TestHRegion
[
https://issues.apache.org/jira/browse/HBASE-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530592#comment-15530592
]
stack commented on HBASE-16725:
---
The potentially hanging thread is real helpful. Sometimes it is just a lagging
thread but no danger of zombies if no hanging thread at end of a test run.
> Don't let flushThread hang in TestHRegion
> -
>
> Key: HBASE-16725
> URL: https://issues.apache.org/jira/browse/HBASE-16725
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 16725.branch-1.v1.txt
>
>
> I was running TestHRegion locally and observed the following in test output:
> {code}
> 2016-09-28 16:29:36,836 INFO [main] hbase.ResourceChecker(171): after:
> regionserver.TestHRegion#testFlushCacheWhileScanning Thread=50 (was 49)
> Potentially hanging thread: FlushThread
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
>
> org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3834)
> {code}
> Call to flushThread.done() etc should be placed in the finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-16726) Refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-16726. - Resolution: Duplicate Assignee: (was: Vladimir Rodionov) > Refactoring: move MR dependencies from HMaster > -- > > Key: HBASE-16726 > URL: https://issues.apache.org/jira/browse/HBASE-16726 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16727) Refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530588#comment-15530588 ] Devaraj Das commented on HBASE-16727: - Yeah let's preserve the current hbase-7912 branch > Refactoring: move MR dependencies from HMaster > -- > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16725) Don't let flushThread hang in TestHRegion
[
https://issues.apache.org/jira/browse/HBASE-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530586#comment-15530586
]
Hadoop QA commented on HBASE-16725:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 3s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
16s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} branch-1 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 4s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} branch-1 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 56s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 48s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 116m 25s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestSerialReplication |
| | hadoop.hbase.replication.regionserver.TestReplicationSourceManager |
| | hadoop.hbase.regionserver.TestHRegion |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-28 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-16725) Don't let flushThread hang in TestHRegion
[
https://issues.apache.org/jira/browse/HBASE-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530547#comment-15530547
]
Appy commented on HBASE-16725:
--
Maybe put other thread joins also in finally?
{noformat}
putThread.join();
putThread.checkNoError();
{noformat}
Also, we should check for interrupted exception from join() otherwise we'll
fail to close region and wal.
As a side question, how accurate are "potentially hanging thread"
notifications? [~stack] your thoughts please.
> Don't let flushThread hang in TestHRegion
> -
>
> Key: HBASE-16725
> URL: https://issues.apache.org/jira/browse/HBASE-16725
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 16725.branch-1.v1.txt
>
>
> I was running TestHRegion locally and observed the following in test output:
> {code}
> 2016-09-28 16:29:36,836 INFO [main] hbase.ResourceChecker(171): after:
> regionserver.TestHRegion#testFlushCacheWhileScanning Thread=50 (was 49)
> Potentially hanging thread: FlushThread
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
>
> org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3834)
> {code}
> Call to flushThread.done() etc should be placed in the finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530546#comment-15530546
]
Imran Rashid commented on HBASE-16179:
--
ah, sorry I forgot that you were making JavaHBaseContext.scala work with both
versions of spark. I'd at least include a comment to that effect. As is, it
chooses the branch at runtime, as opposed to choosing which one at compile time
by having two different implementations which get chosen by the profile.
(Doesn't really matter to me which one you do, just noting the difference.) A
minor nit, if you are going to stick w/ doing it at runtime, I'd write it with
pattern matching (this may also fix the compile errors w/ 1.6):
{code}
def fn = (it: Iterator[T], conn: Connection) => {
asScalaIterator(
// the return type is different in spark 1.x & 2.x, we handle both cases
f.call(asJavaIterator(it), conn) match {
// spark 1.x
case iterable: Iterable[R] => iterable.iterator()
// spark 2.x
case iterator: Iterator[R] => iterator
}
)
}
{code}
on HBaseTableScanRDD -- I am surprised it can use the logging methods from
spark's internal Logging, thought that was against scala's scoping rules. But
I guess I was wrong if it works.
> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
> Issue Type: Bug
> Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt,
> 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16727) Refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530535#comment-15530535 ] Vladimir Rodionov commented on HBASE-16727: --- [~tedyu], [~devaraj], [~enis], your comments are welcomed. Should we create a separate branch for refactoring? A lot of code will be removed (BackupAdmin API, Master integration, procedures). I suggest freezing HBASE-7912 (we could recover old version there) and creating separate branch for the refactoring work. > Refactoring: move MR dependencies from HMaster > -- > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530523#comment-15530523
]
Enis Soztutar commented on HBASE-16721:
---
bq. It turned out that the two regions were opened and hosted on other region
servers, not on this region server.
Do you know that whether these regions were opened before on this host, then
closed. If it is, it maybe the same issue. At the time of the log roller and
log file cleanup, the region to force flush is already closed some time ago,
but WAL still thinks that there is unflushed entries from this region.
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16727) Refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530519#comment-15530519 ] Vladimir Rodionov commented on HBASE-16727: --- We will try to minimize work first. Backup code is currently split between client and server module. # We will not create a separate module for backup unless it will be requested by community. # Backup/restore code will be removed from Master # Backup / restore procedures will be converted to utility classes in a *server* module for command-line driven tools (BackupDriver and RestoreDriver) # We will need to remove completely BackupAdmin API, because after refactoring it will no longer be possible to trigger backup/restore from client module. # The only backup/ restore API we will provide to users are command-line tools. > Refactoring: move MR dependencies from HMaster > -- > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16673) Enhance history command: history per backup set
[
https://issues.apache.org/jira/browse/HBASE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530516#comment-15530516
]
Hadoop QA commented on HBASE-16673:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-16673 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830761/HBASE-16673-v2.patch |
| JIRA Issue | HBASE-16673 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3760/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Enhance history command: history per backup set
> ---
>
> Key: HBASE-16673
> URL: https://issues.apache.org/jira/browse/HBASE-16673
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16673-v1.patch, HBASE-16673-v2.patch
>
>
> New command-line option: -set setName. Will retrieve backup history for a
> particular backup set
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HBASE-16727) Refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-16727 started by Vladimir Rodionov. - > Refactoring: move MR dependencies from HMaster > -- > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16727) Refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16727: -- Description: * No MR jobs in HMaster * No proc2 implementation * client-driven backup-restore * basic security: only super user is allowed to run backup/restore > Refactoring: move MR dependencies from HMaster > -- > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16726) Refactoring: move MR dependencies from HMaster
Vladimir Rodionov created HBASE-16726: - Summary: Refactoring: move MR dependencies from HMaster Key: HBASE-16726 URL: https://issues.apache.org/jira/browse/HBASE-16726 Project: HBase Issue Type: Task Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16727) Refactoring: move MR dependencies from HMaster
Vladimir Rodionov created HBASE-16727: - Summary: Refactoring: move MR dependencies from HMaster Key: HBASE-16727 URL: https://issues.apache.org/jira/browse/HBASE-16727 Project: HBase Issue Type: Task Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16673) Enhance history command: history per backup set
[ https://issues.apache.org/jira/browse/HBASE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16673: -- Attachment: HBASE-16673-v2.patch updated patch v2. > Enhance history command: history per backup set > --- > > Key: HBASE-16673 > URL: https://issues.apache.org/jira/browse/HBASE-16673 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16673-v1.patch, HBASE-16673-v2.patch > > > New command-line option: -set setName. Will retrieve backup history for a > particular backup set -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530467#comment-15530467
]
Hadoop QA commented on HBASE-16179:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 10s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 1m
12s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 3m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 52s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
40s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 48s
{color} | {color:red} root generated 1 new + 19 unchanged - 1 fixed = 20 total
(was 20) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 9s
{color} | {color:red} hbase-spark generated 1 new + 17 unchanged - 1 fixed = 18
total (was 18) {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m 49s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m 23s
{color} | {color:red} hbase-spark in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 11s
{color} | {color:green} hbase-spark2.0-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 12s
{color} | {color:green} hbase-spark1.6-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 35s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 18s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 152m 26s {color}
| {color:black} {color} |
\\
\\
|| Reason ||
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530423#comment-15530423
]
Ted Yu commented on HBASE-16179:
For JavaHBaseContext.scala , the change is needed since Spark 1.6 returns
Iterable while Spark 2.0 returns Iterator.
HBaseTableScanRDD extends RDD which has:
{code}
) extends Serializable with Logging {
{code}
So HBaseTableScanRDD uses Spark's Logging.
w.r.t. the spark & scala version properties, since hbase-spark1.6-compat and
hbase-spark2.0-compat modules are built first, the versions need to be
specified.
I found one commented out import in Utils.scala. It is removed in the next
patch.
> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
> Issue Type: Bug
> Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt,
> 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530386#comment-15530386
]
Hadoop QA commented on HBASE-16179:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 5s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 41s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 46s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 22s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 1m
31s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 4m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
53s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch 5 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 7s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 35s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 2m
18s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 26s
{color} | {color:red} root generated 1 new + 19 unchanged - 1 fixed = 20 total
(was 20) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s
{color} | {color:red} hbase-spark generated 1 new + 17 unchanged - 1 fixed = 18
total (was 18) {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m 53s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m 23s
{color} | {color:red} hbase-spark in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 11s
{color} | {color:green} hbase-spark2.0-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 11s
{color} | {color:green} hbase-spark1.6-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 109m 9s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 12s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
48s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 178m 9s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530383#comment-15530383
]
Imran Rashid commented on HBASE-16179:
--
comments on v9:
* In JavaHBaseContext.scala
{code}
-def fn = (it: Iterator[T], conn: Connection) =>
- asScalaIterator(
-f.call((asJavaIterator(it), conn)).iterator()
- )
+def fn = (it: Iterator[T], conn: Connection) => {
+ val iter = f.call(asJavaIterator(it), conn)
+ if (iter.isInstanceOf[Iterable[R]]) {
+asScalaIterator(iter.asInstanceOf[Iterable[R]].iterator())
+ } else {
+asScalaIterator(iter)
+ }
+}
{code}
why do you need the if instanceOf? {{f.call}} will always return an iterator,
isn't that all you need? I thought you'd just remove the {{.iterator()}} in
the original version.
* In HBaseTableScanRDD.scala you are removing the mixed in {{Logging}} trait.
But there are still calls to {{logDebug}} etc. How does that work? Seems like
its using the logging from within RDD, but I thought the scala compiler would
complain about escaped visibility scope or something. I would think you'd need
to either remove the calls to {{logDebug}}, or do something like this to use
the hbase Logging trait:
{code}
class HBaseTableScanRDD ... {
val logger = new Object with Logging
...
logger.logDebug(...)
}
{code}
I'd be curious if it really does work like this, it would mean spark is leaking
something it doesn't intend to.
* I would think that the spark & scala version properties in
hbase-spark/pom.xml would be set inside the profiles, but I admit I don't spend
a ton of time following the exact pom setup, so maybe its fine.
* a couple of commented out bits that should be removed
> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
> Issue Type: Bug
> Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt,
> 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16673) Enhance history command: history per backup set
[
https://issues.apache.org/jira/browse/HBASE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530367#comment-15530367
]
Ted Yu commented on HBASE-16673:
{code}
+ * @param info: backup info
+ * @return true if info passes filter, false otherwise
+ */
+public boolean apply(BackupInfo image);
{code}
Correct parameter name of @param
{code}
+ if (value == null) return null;
+ return value;
{code}
Redundant check.
{code}
+if(!filters[i].apply(info)) {
+ passed = false; break;
{code}
Move break to next line.
Change to BackupClientUtil.java doesn't apply cleanly.
> Enhance history command: history per backup set
> ---
>
> Key: HBASE-16673
> URL: https://issues.apache.org/jira/browse/HBASE-16673
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16673-v1.patch
>
>
> New command-line option: -set setName. Will retrieve backup history for a
> particular backup set
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16660) ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
[
https://issues.apache.org/jira/browse/HBASE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530286#comment-15530286
]
Hudson commented on HBASE-16660:
FAILURE: Integrated in Jenkins build HBase-0.98-on-Hadoop-1.1 #1278 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1278/])
HBASE-16660 ArrayIndexOutOfBounds during the majorCompactionCheck in (apurtell:
rev 744b867c9b81d779605f3bf581d9b8db9be0e47c)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDateTieredCompactionPolicy.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DateTieredCompactionPolicy.java
> ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
> -
>
> Key: HBASE-16660
> URL: https://issues.apache.org/jira/browse/HBASE-16660
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 0.98.20
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.23
>
> Attachments: HBASE-16660-0.98.patch, HBASE-16660.master.001.patch,
> HBASE-16660.master.001.patch
>
>
> We get an ArrayIndexOutOfBoundsException during the major compaction check as
> follows
> {noformat}
> 2016-09-19 05:04:18,287 ERROR [20.compactionChecker]
> regionserver.HRegionServer$CompactionChecker - Caught exception
> java.lang.ArrayIndexOutOfBoundsException: -2
> at
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.shouldPerformMajorCompaction(DateTieredCompactionPolicy.java:159)
> at
> org.apache.hadoop.hbase.regionserver.HStore.isMajorCompaction(HStore.java:1412)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$CompactionChecker.chore(HRegionServer.java:1532)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This happens due to the following lines in
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.selectMajorCompaction
> {noformat}
> int lowerWindowIndex = Collections.binarySearch(boundaries,
> minTimestamp == null ? Long.MAX_VALUE : file.getMinimumTimestamp());
> int upperWindowIndex = Collections.binarySearch(boundaries,
> file.getMaximumTimestamp() == null ? Long.MAX_VALUE :
> file.getMaximumTimestamp());
> {noformat}
> These return negative values if the element is not found and in the case the
> values are equal we get the exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16725) Don't let flushThread hang in TestHRegion
[
https://issues.apache.org/jira/browse/HBASE-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16725:
---
Status: Patch Available (was: Open)
> Don't let flushThread hang in TestHRegion
> -
>
> Key: HBASE-16725
> URL: https://issues.apache.org/jira/browse/HBASE-16725
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 16725.branch-1.v1.txt
>
>
> I was running TestHRegion locally and observed the following in test output:
> {code}
> 2016-09-28 16:29:36,836 INFO [main] hbase.ResourceChecker(171): after:
> regionserver.TestHRegion#testFlushCacheWhileScanning Thread=50 (was 49)
> Potentially hanging thread: FlushThread
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
>
> org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3834)
> {code}
> Call to flushThread.done() etc should be placed in the finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16725) Don't let flushThread hang in TestHRegion
[
https://issues.apache.org/jira/browse/HBASE-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16725:
---
Attachment: 16725.branch-1.v1.txt
> Don't let flushThread hang in TestHRegion
> -
>
> Key: HBASE-16725
> URL: https://issues.apache.org/jira/browse/HBASE-16725
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 16725.branch-1.v1.txt
>
>
> I was running TestHRegion locally and observed the following in test output:
> {code}
> 2016-09-28 16:29:36,836 INFO [main] hbase.ResourceChecker(171): after:
> regionserver.TestHRegion#testFlushCacheWhileScanning Thread=50 (was 49)
> Potentially hanging thread: FlushThread
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
>
> org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3834)
> {code}
> Call to flushThread.done() etc should be placed in the finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16725) Don't let flushThread hang in TestHRegion
Ted Yu created HBASE-16725:
--
Summary: Don't let flushThread hang in TestHRegion
Key: HBASE-16725
URL: https://issues.apache.org/jira/browse/HBASE-16725
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
I was running TestHRegion locally and observed the following in test output:
{code}
2016-09-28 16:29:36,836 INFO [main] hbase.ResourceChecker(171): after:
regionserver.TestHRegion#testFlushCacheWhileScanning Thread=50 (was 49)
Potentially hanging thread: FlushThread
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3834)
{code}
Call to flushThread.done() etc should be placed in the finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529743#comment-15529743
]
Jonathan Hsieh commented on HBASE-16179:
Review on v7.
This fails for me:
{code}
mvn clean dependency:tree
mvn clean dependency:tree -Dspark.profile=1.6
{code}
This works
{code}
mvn clean dependency:tree -Dspark.profile=2.0
{code}
hbase-spark/pom.xml
- currently no profile is activated -- both say !spark.profile. You want
default with !spark.profile and the othres with actual values the values -- 1.6
and 2.0. Look at how hadoop.profile is handled.
- Based on sean's comments can you make the default scala 2.11? We don't have
to update as much going forward.
hbasetablescanrdd
- why is the "//with Logging" comment there? Remove?
pom.xml
- consider calling the modules hbase-spark1.6-compat and hbase-spark2.0-compat
instead of hbase-spark1.6-compat and hbase-spark-compat. This way when new
spark versions come out and we need do a ship, it will be obvious what to do.
DataTypeParserWrapper
- add comments about why this exists (e.g. necessary for spark2.0, not present
in 1.6 etc)
Logging
- add comments saying why this is present instead of using the spark1.6 Logging
class.
nits: fix space/tabs issues in hbase-spark/pom.xml
> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
> Issue Type: Bug
> Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt,
> 16179.v5.txt, 16179.v7.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529799#comment-15529799 ] Sean Busbey commented on HBASE-16179: - can we work it so both modules build all the time? if we have to build each individually we'll be making the build process for our RMs harder. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt, > 16179.v5.txt, 16179.v7.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529721#comment-15529721
]
Hadoop QA commented on HBASE-16723:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 55s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 8s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 24s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 132m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Write to static field org.apache.hadoop.hbase.JMXListener.RMI_REGISTRY
from instance method
org.apache.hadoop.hbase.JMXListener.startConnectorServer(int, int) At
JMXListener.java:from instance method
org.apache.hadoop.hbase.JMXListener.startConnectorServer(int, int) At
JMXListener.java:[line 134] |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestEnableTable |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830680/HBASE-16723.patch |
| JIRA Issue | HBASE-16723 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 18ece30c2471 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 47e12fb |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
[jira] [Commented] (HBASE-16724) Snapshot owner can't clone
[
https://issues.apache.org/jira/browse/HBASE-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529734#comment-15529734
]
Matteo Bertozzi commented on HBASE-16724:
-
what if we change cloneSnapshot to check for table admin?
{code}
requirePermission(getActiveUser(ctx), "cloneSnapshot " + snapshot.getName(),
hTableDescriptor.getTableName(), null, null,
Permission.Action.ADMIN);
{code}
snapshot and restore do the check for table admin.
so you can take a snapshot and by snapshot owner if you own the table.
clone should probably do the same thing, if you have permission on that table
you'll be able to create it.
this prevents for a user with a specific "table admin" permission to be able to
create other tables.
e.g. user1 is allowed to work/admin only table1. user1 can
snapshot/restore/clone snapshots for table1 as table1
> Snapshot owner can't clone
> --
>
> Key: HBASE-16724
> URL: https://issues.apache.org/jira/browse/HBASE-16724
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>
> Currently only Global admin has the access of cloning a snapshot.
> In AccessController,
> {code}
> @Override
> public void preCloneSnapshot(final
> ObserverContext ctx,
> final SnapshotDescription snapshot, final HTableDescriptor
> hTableDescriptor)
> throws IOException {
> requirePermission(getActiveUser(ctx), "cloneSnapshot " +
> snapshot.getName(), Action.ADMIN);
> }
> {code}
> Snapshot owner should be able to clone it, need to add a check like,
> {code}
> SnapshotDescriptionUtils.isSnapshotOwner(snapshot, user)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16656) BackupID must include backup set name
[ https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16656: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the patch, Vlad. > BackupID must include backup set name > - > > Key: HBASE-16656 > URL: https://issues.apache.org/jira/browse/HBASE-16656 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16656-v1.patch, HBASE-16656-v2.patch, > HBASE-16656-v3.patch, HBASE-16656-v4.patch > > > Default backup set name is "backup". If we do backup for a backup set > "SomeSetName", by default, backup id will be generated in a form: > *SomeSetName_timestamp*. > The goal is to separate backup images between different sets. > The history command will be updated and the new command - merge will use this > backup names (prefixes) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529709#comment-15529709
]
Ted Yu commented on HBASE-16723:
lgtm
> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-16723-V2.patch, HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX
> Connector server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16660) ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
[
https://issues.apache.org/jira/browse/HBASE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529521#comment-15529521
]
Hudson commented on HBASE-16660:
FAILURE: Integrated in Jenkins build HBase-0.98-matrix #405 (See
[https://builds.apache.org/job/HBase-0.98-matrix/405/])
HBASE-16660 ArrayIndexOutOfBounds during the majorCompactionCheck in (apurtell:
rev 744b867c9b81d779605f3bf581d9b8db9be0e47c)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DateTieredCompactionPolicy.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDateTieredCompactionPolicy.java
> ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
> -
>
> Key: HBASE-16660
> URL: https://issues.apache.org/jira/browse/HBASE-16660
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 0.98.20
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.23
>
> Attachments: HBASE-16660-0.98.patch, HBASE-16660.master.001.patch,
> HBASE-16660.master.001.patch
>
>
> We get an ArrayIndexOutOfBoundsException during the major compaction check as
> follows
> {noformat}
> 2016-09-19 05:04:18,287 ERROR [20.compactionChecker]
> regionserver.HRegionServer$CompactionChecker - Caught exception
> java.lang.ArrayIndexOutOfBoundsException: -2
> at
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.shouldPerformMajorCompaction(DateTieredCompactionPolicy.java:159)
> at
> org.apache.hadoop.hbase.regionserver.HStore.isMajorCompaction(HStore.java:1412)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$CompactionChecker.chore(HRegionServer.java:1532)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This happens due to the following lines in
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.selectMajorCompaction
> {noformat}
> int lowerWindowIndex = Collections.binarySearch(boundaries,
> minTimestamp == null ? Long.MAX_VALUE : file.getMinimumTimestamp());
> int upperWindowIndex = Collections.binarySearch(boundaries,
> file.getMaximumTimestamp() == null ? Long.MAX_VALUE :
> file.getMaximumTimestamp());
> {noformat}
> These return negative values if the element is not found and in the case the
> values are equal we get the exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529419#comment-15529419
]
Ashish Singhi commented on HBASE-16723:
---
+1
> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-16723-V2.patch, HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX
> Connector server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16724) Snapshot owner can't clone
[
https://issues.apache.org/jira/browse/HBASE-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529416#comment-15529416
]
Ashish Singhi commented on HBASE-16724:
---
Scenario:
user 'user1' (have admin and create rights on the namespace in which it create
the table below)
1. Create a table and takes the snapshot
2. Disable and drops the table
3. Performs restore_snapshot, which fails
Though there are sufficient permissions to the user 'user1' to execute
restore_snapshot as per [ACL
Matric|https://hbase.apache.org/book.html#appendix_acl_matrix] the access is
denied.
Currently when we restore a snapshot if the table that doesn't exist then
internally it will clone the snapshot which requires either superuser of global
admin permission only as per the ACL matrix and user1 lacks them, so it fails.
But the end user doesn't know about internal behavior.
So we think clone snapshot should also work with same permissions as
restore_snapshot.
WDYT [~mbertozzi]/Others ?
> Snapshot owner can't clone
> --
>
> Key: HBASE-16724
> URL: https://issues.apache.org/jira/browse/HBASE-16724
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>
> Currently only Global admin has the access of cloning a snapshot.
> In AccessController,
> {code}
> @Override
> public void preCloneSnapshot(final
> ObserverContext ctx,
> final SnapshotDescription snapshot, final HTableDescriptor
> hTableDescriptor)
> throws IOException {
> requirePermission(getActiveUser(ctx), "cloneSnapshot " +
> snapshot.getName(), Action.ADMIN);
> }
> {code}
> Snapshot owner should be able to clone it, need to add a check like,
> {code}
> SnapshotDescriptionUtils.isSnapshotOwner(snapshot, user)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529414#comment-15529414
]
Pankaj Kumar commented on HBASE-16723:
--
Thanks for the review [~ashish singhi]. Yes, you are right, addressed that in
V2 patch.
> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-16723-V2.patch, HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX
> Connector server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-16723:
-
Attachment: HBASE-16723-V2.patch
> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-16723-V2.patch, HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX
> Connector server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529382#comment-15529382
]
Ashish Singhi commented on HBASE-16723:
---
I think we should deregister RMI registry when there is an error to start
JMXConnectorServer also. WDYT ?
> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX
> Connector server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15871) Memstore flush doesn't finish because of backwardseek() in memstore scanner.
[
https://issues.apache.org/jira/browse/HBASE-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529381#comment-15529381
]
Hadoop QA commented on HBASE-15871:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 44s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 107m 49s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 145m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
| | hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestMobRestoreFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830656/HBASE-15871_6.patch |
| JIRA Issue | HBASE-15871 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux e4212b9c9cb3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / de7316b |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3753/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3753/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3753/testReport/ |
| modules | C: hbase-server U: hbase-server |
[jira] [Created] (HBASE-16724) Snapshot owner can't clone
Pankaj Kumar created HBASE-16724:
Summary: Snapshot owner can't clone
Key: HBASE-16724
URL: https://issues.apache.org/jira/browse/HBASE-16724
Project: HBase
Issue Type: Bug
Components: snapshots
Affects Versions: 2.0.0
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Currently only Global admin has the access of cloning a snapshot.
In AccessController,
{code}
@Override
public void preCloneSnapshot(final
ObserverContext ctx,
final SnapshotDescription snapshot, final HTableDescriptor
hTableDescriptor)
throws IOException {
requirePermission(getActiveUser(ctx), "cloneSnapshot " +
snapshot.getName(), Action.ADMIN);
}
{code}
Snapshot owner should be able to clone it, need to add a check like,
{code}
SnapshotDescriptionUtils.isSnapshotOwner(snapshot, user)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-16723:
-
Component/s: metrics
> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX
> Connector server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-16723:
-
Fix Version/s: 2.0.0
Status: Patch Available (was: Open)
Simple patch, please review.
> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.22, 1.2.3, 2.0.0, 1.4.0, 1.3.1
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX
> Connector server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
