[jira] [Comment Edited] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528433#comment-15528433
]
Jerry He edited comment on HBASE-16721 at 9/28/16 5:15 AM:
---
I had a similar problem on a customer production cluster recently.
The WALs for one of the region servers (server 11) kept on accumulating, and
these LOG info repeatedly showed up.
{code}
2016-09-03 14:37:15,989 INFO org.apache.hadoop.hbase.regionserver.wal.FSHLog:
Too many hlogs: logs=817, maxlogs=32; forcing flush of 2 regions(s):
1b86c057f80721d4fde43a303f63ebde, 32d36d4864259dc9d984326bf27dcc5e
2016-09-03 14:37:15,990 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 1b86c057f80721d4fde43a303f63ebde, region=null,
requester=null
2016-09-03 14:37:15,990 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 32d36d4864259dc9d984326bf27dcc5e, region=null,
requester=null
{code}
It turned out that the two regions were opened and hosted on other region
servers, not on this region server.
After manually moving the complaining regions from other region servers to
server 11, then server 11 was able to finish the "flush". The wal files for
server 11 came down right after that.
I didn't had a chance to look into what the root cause was. Some of the region
servers had crashed before that.
was (Author: jinghe):
I had a similar problem on a customer production cluster recently.
The WALs for one of the region servers (server 11) kept on accumulating, and
these LOG info repeatedly showed up.
{code}
2016-09-03 14:37:15,989 INFO org.apache.hadoop.hbase.regionserver.wal.FSHLog:
Too many hlogs: logs=817, maxlogs=32; forcing flush of 2 regions(s):
1b86c057f80721d4fde43a303f63ebde, 32d36d4864259dc9d984326bf27dcc5e
2016-09-03 14:37:15,990 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 1b86c057f80721d4fde43a303f63ebde, region=null,
requester=null
2016-09-03 14:37:15,990 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 32d36d4864259dc9d984326bf27dcc5e, region=null,
requester=null
{code}
It turned out that the two regions were opened and hosted on other region
servers, not on this region server.
After manually moving the complaining regions from other region servers to
server 11, then server 11 was able to finish the flush. The wal files for
server 11 came down right after that.
I didn't had a chance to look into what the root cause was. Some of the region
servers had crashed before that.
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528433#comment-15528433
]
Jerry He commented on HBASE-16721:
--
I had a similar problem on a customer production cluster recently.
The WALs for one of the region servers (server 11) kept on accumulating, and
these LOG info repeatedly showed up.
{code}
2016-09-03 14:37:15,989 INFO org.apache.hadoop.hbase.regionserver.wal.FSHLog:
Too many hlogs: logs=817, maxlogs=32; forcing flush of 2 regions(s):
1b86c057f80721d4fde43a303f63ebde, 32d36d4864259dc9d984326bf27dcc5e
2016-09-03 14:37:15,990 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 1b86c057f80721d4fde43a303f63ebde, region=null,
requester=null
2016-09-03 14:37:15,990 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 32d36d4864259dc9d984326bf27dcc5e, region=null,
requester=null
{code}
It turned out that the two regions were opened and hosted on other region
servers, not on this region server.
After manually moving the complaining regions from other region servers to
server 11, then server 11 was able to finish the flush. The wal files for
server 11 came down right after that.
I didn't had a chance to look into what the root cause was. Some of the region
servers had crashed before that.
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16696) After HBASE-16604 - does not release blocks in case of scanner exception
[ https://issues.apache.org/jira/browse/HBASE-16696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16696: --- Attachment: HBASE-16696_2.patch Patch that moves the shippedCallback to the try block and also handles the findbug warning by adding a null check before throwing the ScannerResetException. Will commit _2 after QA says fine. > After HBASE-16604 - does not release blocks in case of scanner exception > > > Key: HBASE-16696 > URL: https://issues.apache.org/jira/browse/HBASE-16696 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16696.patch, HBASE-16696_1.patch, > HBASE-16696_2.patch > > > TestBlockEvictionFromClient consistently fails in master branch. > From existing Jenkins builds, looks like this started with build 1639. > See attached Jenkins console logs. > Pls refer to this comment > https://issues.apache.org/jira/secure/EditComment!default.jspa?id=13004009=15522451 > for the reason for the test failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16696) After HBASE-16604 - does not release blocks in case of scanner exception
[ https://issues.apache.org/jira/browse/HBASE-16696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16696: --- Status: Patch Available (was: Open) > After HBASE-16604 - does not release blocks in case of scanner exception > > > Key: HBASE-16696 > URL: https://issues.apache.org/jira/browse/HBASE-16696 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16696.patch, HBASE-16696_1.patch, > HBASE-16696_2.patch > > > TestBlockEvictionFromClient consistently fails in master branch. > From existing Jenkins builds, looks like this started with build 1639. > See attached Jenkins console logs. > Pls refer to this comment > https://issues.apache.org/jira/secure/EditComment!default.jspa?id=13004009=15522451 > for the reason for the test failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16696) After HBASE-16604 - does not release blocks in case of scanner exception
[ https://issues.apache.org/jira/browse/HBASE-16696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16696: --- Status: Open (was: Patch Available) > After HBASE-16604 - does not release blocks in case of scanner exception > > > Key: HBASE-16696 > URL: https://issues.apache.org/jira/browse/HBASE-16696 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16696.patch, HBASE-16696_1.patch > > > TestBlockEvictionFromClient consistently fails in master branch. > From existing Jenkins builds, looks like this started with build 1639. > See attached Jenkins console logs. > Pls refer to this comment > https://issues.apache.org/jira/secure/EditComment!default.jspa?id=13004009=15522451 > for the reason for the test failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16574) Add backup / restore feature to refguide
[
https://issues.apache.org/jira/browse/HBASE-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528346#comment-15528346
]
Hadoop QA commented on HBASE-16574:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 57s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
12s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch 13 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 53s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 142m 22s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
| Timed out junit tests | org.apache.hadoop.hbase.TestZooKeeper |
| |
org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizerOnCluster |
| | org.apache.hadoop.hbase.master.TestRollingRestart |
| | org.apache.hadoop.hbase.master.TestDistributedLogSplitting |
| | org.apache.hadoop.hbase.replication.TestReplicationTableBase |
| | org.apache.hadoop.hbase.master.TestMasterStatusServlet |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830601/HBASE-16574.002.patch
|
| JIRA Issue | HBASE-16574 |
| Optional Tests | asflicense javac javadoc unit |
| uname | Linux d2a14afbc720 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / de7316b |
| Default Java | 1.8.0_101 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/3746/artifact/patchprocess/whitespace-tabs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3746/artifact/patchprocess/patch-unit-root.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3746/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3746/testReport/ |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3746/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add backup / restore feature to refguide
>
>
> Key: HBASE-16574
> URL: https://issues.apache.org/jira/browse/HBASE-16574
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Frank Welsch
> Labels: backup
> Fix For: 2.0.0
>
> Attachments:
[jira] [Commented] (HBASE-16656) BackupID must include backup set name
[
https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528341#comment-15528341
]
Hadoop QA commented on HBASE-16656:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-16656 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830625/HBASE-16656-v4.patch |
| JIRA Issue | HBASE-16656 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3750/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> BackupID must include backup set name
> -
>
> Key: HBASE-16656
> URL: https://issues.apache.org/jira/browse/HBASE-16656
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16656-v1.patch, HBASE-16656-v2.patch,
> HBASE-16656-v3.patch, HBASE-16656-v4.patch
>
>
> Default backup set name is "backup". If we do backup for a backup set
> "SomeSetName", by default, backup id will be generated in a form:
> *SomeSetName_timestamp*.
> The goal is to separate backup images between different sets.
> The history command will be updated and the new command - merge will use this
> backup names (prefixes)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16720) Sort build ids in flaky dashboard
[ https://issues.apache.org/jira/browse/HBASE-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528340#comment-15528340 ] Hudson commented on HBASE-16720: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1686 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1686/]) HBASE-16720 Sort build ids in flaky dashboard. (appy: rev de7316b11ec4ed369095510e8bc302e747e1bf21) * (edit) dev-support/flaky-dashboard-template.html * (edit) dev-support/report-flakies.py > Sort build ids in flaky dashboard > - > > Key: HBASE-16720 > URL: https://issues.apache.org/jira/browse/HBASE-16720 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16720.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528256#comment-15528256 ] Ben Manes commented on HBASE-15560: --- [~busbey], I can't seem to trigger a new build. Can you please take a look? > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16656) BackupID must include backup set name
[ https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16656: -- Attachment: HBASE-16656-v4.patch v4 > BackupID must include backup set name > - > > Key: HBASE-16656 > URL: https://issues.apache.org/jira/browse/HBASE-16656 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16656-v1.patch, HBASE-16656-v2.patch, > HBASE-16656-v3.patch, HBASE-16656-v4.patch > > > Default backup set name is "backup". If we do backup for a backup set > "SomeSetName", by default, backup id will be generated in a form: > *SomeSetName_timestamp*. > The goal is to separate backup images between different sets. > The history command will be updated and the new command - merge will use this > backup names (prefixes) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16661) Add last major compaction age to per-region metrics
[
https://issues.apache.org/jira/browse/HBASE-16661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528244#comment-15528244
]
Hadoop QA commented on HBASE-16661:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 41s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 119m 28s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 171m 21s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
| | hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
| Timed out junit tests | org.apache.hadoop.hbase.TestNamespace |
| | org.apache.hadoop.hbase.snapshot.TestMobRestoreFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.TestMetaTableAccessor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830597/HBASE-16661.003.patch
|
| JIRA Issue | HBASE-16661 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 63c25ef14822 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Manes updated HBASE-15560: -- Status: Patch Available (was: Open) > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Manes updated HBASE-15560: -- Status: Open (was: Patch Available) > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Manes updated HBASE-15560: -- Attachment: HBASE-15560.patch > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16179: --- Status: Patch Available (was: Open) > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt, > 16179.v5.txt, 16179.v7.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16722) Document: Broken link in CatalogJanitor
Aki Ariga created HBASE-16722: - Summary: Document: Broken link in CatalogJanitor Key: HBASE-16722 URL: https://issues.apache.org/jira/browse/HBASE-16722 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 2.0.0 Reporter: Aki Ariga Priority: Trivial A link in [CatalogJanitor|https://hbase.apache.org/book.html#master.processes.catalog] is broken. This should be linked to https://hbase.apache.org/book.html#arch.catalog.meta -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528202#comment-15528202 ] Ted Yu commented on HBASE-16179: To build against Scala 2.11 : mvn package -DskipTests -Dscala.version=2.11.8 > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt, > 16179.v5.txt, 16179.v7.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16179: --- Attachment: 16179.v7.txt Patch v7 adds two modules: hbase-spark-compat for Spark 2.0 (default) hbase-spark1.6-compat for Spark 1.6 You can build for Spark 1.6 by using: mvn package -DskipTests -Dspark.profile=1.6 > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt, > 16179.v5.txt, 16179.v7.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16710) Add ZStandard Codec to Compression.java
[
https://issues.apache.org/jira/browse/HBASE-16710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528153#comment-15528153
]
Hadoop QA commented on HBASE-16710:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m 0s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
47s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} 0.98 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} 0.98 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} 0.98 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 39s
{color} | {color:red} hbase-common in 0.98 has 6 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 36s
{color} | {color:red} hbase-server in 0.98 has 85 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 15s
{color} | {color:red} hbase-common in 0.98 failed with JDK v1.8.0_101. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 22s
{color} | {color:red} hbase-server in 0.98 failed with JDK v1.8.0_101. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} 0.98 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 22s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 15s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 18s
{color} | {color:red} hbase-common in the patch failed with JDK v1.8.0_101.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 25s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_101.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 5s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 122m 7s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense
[jira] [Commented] (HBASE-16656) BackupID must include backup set name
[
https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528115#comment-15528115
]
Hadoop QA commented on HBASE-16656:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s {color}
| {color:red} HBASE-16656 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830613/HBASE-16656-v3.patch |
| JIRA Issue | HBASE-16656 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3747/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> BackupID must include backup set name
> -
>
> Key: HBASE-16656
> URL: https://issues.apache.org/jira/browse/HBASE-16656
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16656-v1.patch, HBASE-16656-v2.patch,
> HBASE-16656-v3.patch
>
>
> Default backup set name is "backup". If we do backup for a backup set
> "SomeSetName", by default, backup id will be generated in a form:
> *SomeSetName_timestamp*.
> The goal is to separate backup images between different sets.
> The history command will be updated and the new command - merge will use this
> backup names (prefixes)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16660) ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
[
https://issues.apache.org/jira/browse/HBASE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528110#comment-15528110
]
Hudson commented on HBASE-16660:
FAILURE: Integrated in Jenkins build HBase-1.4 #434 (See
[https://builds.apache.org/job/HBase-1.4/434/])
HBASE-16660 ArrayIndexOutOfBounds during the majorCompactionCheck in (apurtell:
rev 96a8e8dce4ea65a12af6c14d22ef4d34b52ebb6c)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDateTieredCompactionPolicy.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DateTieredCompactionPolicy.java
> ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
> -
>
> Key: HBASE-16660
> URL: https://issues.apache.org/jira/browse/HBASE-16660
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 0.98.20
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.23
>
> Attachments: HBASE-16660-0.98.patch, HBASE-16660.master.001.patch,
> HBASE-16660.master.001.patch
>
>
> We get an ArrayIndexOutOfBoundsException during the major compaction check as
> follows
> {noformat}
> 2016-09-19 05:04:18,287 ERROR [20.compactionChecker]
> regionserver.HRegionServer$CompactionChecker - Caught exception
> java.lang.ArrayIndexOutOfBoundsException: -2
> at
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.shouldPerformMajorCompaction(DateTieredCompactionPolicy.java:159)
> at
> org.apache.hadoop.hbase.regionserver.HStore.isMajorCompaction(HStore.java:1412)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$CompactionChecker.chore(HRegionServer.java:1532)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This happens due to the following lines in
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.selectMajorCompaction
> {noformat}
> int lowerWindowIndex = Collections.binarySearch(boundaries,
> minTimestamp == null ? Long.MAX_VALUE : file.getMinimumTimestamp());
> int upperWindowIndex = Collections.binarySearch(boundaries,
> file.getMaximumTimestamp() == null ? Long.MAX_VALUE :
> file.getMaximumTimestamp());
> {noformat}
> These return negative values if the element is not found and in the case the
> values are equal we get the exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16656) BackupID must include backup set name
[ https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528056#comment-15528056 ] Ted Yu commented on HBASE-16656: There is BackupClientUtil.java.rej in patch v3. Please produce a clean patch. > BackupID must include backup set name > - > > Key: HBASE-16656 > URL: https://issues.apache.org/jira/browse/HBASE-16656 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16656-v1.patch, HBASE-16656-v2.patch, > HBASE-16656-v3.patch > > > Default backup set name is "backup". If we do backup for a backup set > "SomeSetName", by default, backup id will be generated in a form: > *SomeSetName_timestamp*. > The goal is to separate backup images between different sets. > The history command will be updated and the new command - merge will use this > backup names (prefixes) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16656) BackupID must include backup set name
[ https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16656: -- Attachment: HBASE-16656-v3.patch v3. > BackupID must include backup set name > - > > Key: HBASE-16656 > URL: https://issues.apache.org/jira/browse/HBASE-16656 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16656-v1.patch, HBASE-16656-v2.patch, > HBASE-16656-v3.patch > > > Default backup set name is "backup". If we do backup for a backup set > "SomeSetName", by default, backup id will be generated in a form: > *SomeSetName_timestamp*. > The goal is to separate backup images between different sets. > The history command will be updated and the new command - merge will use this > backup names (prefixes) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527972#comment-15527972
]
Enis Soztutar commented on HBASE-16721:
---
In branch-1 and earlier, FSHLog.findRegionsToForceFlush() is the method which
finds that region to be force-flushed. This means that the region is somehow
still in the oldestUnflushedStoreSequenceIds, although the region is closed
already and flushed.
I suspect the issue is a race between FSHLog.startCacheFlush() and
FSHLog.append() racing for the updates to the
oldestUnflushedStoreSequenceIdsOfRegion map.
FSHlog.append() is called from the ring buffer handler thread, and does this:
{code}
private void updateOldestUnflushedSequenceIds(byte[] encodedRegionName,
Set familyNameSet, Long lRegionSequenceId) {
ConcurrentMap oldestUnflushedStoreSequenceIdsOfRegion =
getOrCreateOldestUnflushedStoreSequenceIdsOfRegion(encodedRegionName);
for (byte[] familyName : familyNameSet) {
oldestUnflushedStoreSequenceIdsOfRegion.putIfAbsent(familyName,
lRegionSequenceId);
}
}
{code}
And FSHLog.startCacheFlush() does this:
{code}
synchronized (regionSequenceIdLock) {
ConcurrentMap oldestUnflushedStoreSequenceIdsOfRegion =
oldestUnflushedStoreSequenceIds.get(encodedRegionName);
if (oldestUnflushedStoreSequenceIdsOfRegion != null) {
for (byte[] familyName: flushedFamilyNames) {
Long seqId =
oldestUnflushedStoreSequenceIdsOfRegion.remove(familyName);
if (seqId != null) {
oldStoreSeqNum.put(familyName, seqId);
}
}
if (!oldStoreSeqNum.isEmpty()) {
Map oldValue = this.lowestFlushingStoreSequenceIds.put(
encodedRegionName, oldStoreSeqNum);
assert oldValue == null: "Flushing map not cleaned up for "
+ Bytes.toString(encodedRegionName);
}
if (oldestUnflushedStoreSequenceIdsOfRegion.isEmpty()) {
// Remove it otherwise it will be in oldestUnflushedStoreSequenceIds
for ever
// even if the region is already moved to other server.
// Do not worry about data racing, we held write lock of region when
calling
// startCacheFlush, so no one can add value to the map we removed.
oldestUnflushedStoreSequenceIds.remove(encodedRegionName);
} else {
oldestUnflushedSequenceId =
Collections.min(oldestUnflushedStoreSequenceIdsOfRegion.values());
}
}
{code}
The actual problem is that the sub-map for the region which keeps the per-store
unflushed edits gets removes from:
{code}
Long seqId =
oldestUnflushedStoreSequenceIdsOfRegion.remove(familyName);
{code}
but the actual region entry is removed some time later in:
{code}
if (oldestUnflushedStoreSequenceIdsOfRegion.isEmpty()) {
oldestUnflushedStoreSequenceIds.remove(encodedRegionName);
{code}
However, this comment is not correct:
{code}
// Do not worry about data racing, we held write lock of region when
calling
// startCacheFlush, so no one can add value to the map we removed.
{code}
and it is likely that an append() request is still executing and will re-add
the removed family to the map. Although startCacheFlush() happens under the
HRegion.updatesLock().writeLock() and doMiniBatchMutation() acquires the read
lock, we may give away the read lock before the ring buffer handler actually
does the FSHLog.append(). We release the region lock in step 6, but only wait
for the seqId assignment in step 8.
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
>
[jira] [Commented] (HBASE-16117) Fix Connection leak in mapred.TableOutputFormat
[
https://issues.apache.org/jira/browse/HBASE-16117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527930#comment-15527930
]
Hadoop QA commented on HBASE-16117:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 31s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 6s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 134m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestClientTimeouts |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830581/hbase-16117-part2.v5.patch
|
| JIRA Issue | HBASE-16117 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a94eb6062a73 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / d127d64 |
| Default Java | 1.8.0_101 |
|
[jira] [Created] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
Enis Soztutar created HBASE-16721:
-
Summary: Concurrency issue in WAL unflushed seqId tracking
Key: HBASE-16721
URL: https://issues.apache.org/jira/browse/HBASE-16721
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
I'm inspecting an interesting case where in a production cluster, some
regionservers ends up accumulating hundreds of WAL files, even with force
flushes going on due to max logs. This happened multiple times on the cluster,
but not on other clusters. The cluster has periodic memstore flusher disabled,
however, this still does not explain why the force flush of regions due to max
limit is not working. I think the periodic memstore flusher just masks the
underlying problem, which is why we do not see this in other clusters.
The problem starts like this:
{code}
2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1 regions(s):
d4cf39dc40ea79f5da4d0cf66d03cb1f
2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
regionserver.LogRoller: Failed to schedule flush of
d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
{code}
then, it continues until the RS is restarted:
{code}
2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1 regions(s):
d4cf39dc40ea79f5da4d0cf66d03cb1f
2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
regionserver.LogRoller: Failed to schedule flush of
d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
{code}
The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
split some time ago, and was able to flush its data and split without any
problems. However, the FSHLog still thinks that there is some unflushed data
for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Manes updated HBASE-15560: -- Status: Patch Available (was: Open) > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16574) Add backup / restore feature to refguide
[ https://issues.apache.org/jira/browse/HBASE-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-16574: --- Fix Version/s: 2.0.0 Status: Patch Available (was: Open) > Add backup / restore feature to refguide > > > Key: HBASE-16574 > URL: https://issues.apache.org/jira/browse/HBASE-16574 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Frank Welsch > Labels: backup > Fix For: 2.0.0 > > Attachments: Backup-and-Restore-Apache_19Sep2016.pdf, > HBASE-16574.001.patch, HBASE-16574.002.patch, hbase_reference_guide.v1.pdf > > > This issue is to add backup / restore feature description to hbase refguide. > The description should cover: > scenarios where backup / restore is used > backup / restore commands and sample usage > considerations in setup -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Manes updated HBASE-15560: -- Status: Open (was: Patch Available) Cancelling and re-submitting patch, hoping that Hadoop QA will pick it up this time. > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16574) Add backup / restore feature to refguide
[ https://issues.apache.org/jira/browse/HBASE-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-16574: --- Attachment: HBASE-16574.002.patch .002 is equivalent to .001 except for attribution to Frank (as my transcribing work was minor to the actual content). > Add backup / restore feature to refguide > > > Key: HBASE-16574 > URL: https://issues.apache.org/jira/browse/HBASE-16574 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Frank Welsch > Labels: backup > Fix For: 2.0.0 > > Attachments: Backup-and-Restore-Apache_19Sep2016.pdf, > HBASE-16574.001.patch, HBASE-16574.002.patch, hbase_reference_guide.v1.pdf > > > This issue is to add backup / restore feature description to hbase refguide. > The description should cover: > scenarios where backup / restore is used > backup / restore commands and sample usage > considerations in setup -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16574) Add backup / restore feature to refguide
[ https://issues.apache.org/jira/browse/HBASE-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-16574: --- Assignee: Frank Welsch > Add backup / restore feature to refguide > > > Key: HBASE-16574 > URL: https://issues.apache.org/jira/browse/HBASE-16574 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Frank Welsch > Labels: backup > Attachments: Backup-and-Restore-Apache_19Sep2016.pdf, > HBASE-16574.001.patch, HBASE-16574.002.patch, hbase_reference_guide.v1.pdf > > > This issue is to add backup / restore feature description to hbase refguide. > The description should cover: > scenarios where backup / restore is used > backup / restore commands and sample usage > considerations in setup -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16661) Add last major compaction age to per-region metrics
[ https://issues.apache.org/jira/browse/HBASE-16661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527834#comment-15527834 ] Dustin Pho edited comment on HBASE-16661 at 9/27/16 11:57 PM: -- Good call on the description! I missed that completely. was (Author: pho): Good call! > Add last major compaction age to per-region metrics > --- > > Key: HBASE-16661 > URL: https://issues.apache.org/jira/browse/HBASE-16661 > Project: HBase > Issue Type: Improvement >Reporter: Gary Helmling >Assignee: Dustin Pho >Priority: Minor > Attachments: HBASE-16661.001.patch, HBASE-16661.002.patch, > HBASE-16661.003.patch > > > After HBASE-12859, we can now track the last major compaction timestamp for > each region. However, this is only exposed through cluster status reporting > and the admin API. > We have similar per-region metrics around storefile age, but none that > filters on major compaction specifically. > Let's add a metric for last major compaction age. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16661) Add last major compaction age to per-region metrics
[ https://issues.apache.org/jira/browse/HBASE-16661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Pho updated HBASE-16661: --- Attachment: HBASE-16661.003.patch Good call! > Add last major compaction age to per-region metrics > --- > > Key: HBASE-16661 > URL: https://issues.apache.org/jira/browse/HBASE-16661 > Project: HBase > Issue Type: Improvement >Reporter: Gary Helmling >Assignee: Dustin Pho >Priority: Minor > Attachments: HBASE-16661.001.patch, HBASE-16661.002.patch, > HBASE-16661.003.patch > > > After HBASE-12859, we can now track the last major compaction timestamp for > each region. However, this is only exposed through cluster status reporting > and the admin API. > We have similar per-region metrics around storefile age, but none that > filters on major compaction specifically. > Let's add a metric for last major compaction age. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16661) Add last major compaction age to per-region metrics
[ https://issues.apache.org/jira/browse/HBASE-16661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527832#comment-15527832 ] Gary Helmling commented on HBASE-16661: --- The failing tests are unrelated -- all pass for me locally with the patch applied. > Add last major compaction age to per-region metrics > --- > > Key: HBASE-16661 > URL: https://issues.apache.org/jira/browse/HBASE-16661 > Project: HBase > Issue Type: Improvement >Reporter: Gary Helmling >Assignee: Dustin Pho >Priority: Minor > Attachments: HBASE-16661.001.patch, HBASE-16661.002.patch > > > After HBASE-12859, we can now track the last major compaction timestamp for > each region. However, this is only exposed through cluster status reporting > and the admin API. > We have similar per-region metrics around storefile age, but none that > filters on major compaction specifically. > Let's add a metric for last major compaction age. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16660) ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
[
https://issues.apache.org/jira/browse/HBASE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527782#comment-15527782
]
Hudson commented on HBASE-16660:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #24 (See
[https://builds.apache.org/job/HBase-1.3-JDK7/24/])
HBASE-16660 ArrayIndexOutOfBounds during the majorCompactionCheck in (apurtell:
rev e0066e713d6bd4f499e479ea173677e3fc14d2a2)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DateTieredCompactionPolicy.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDateTieredCompactionPolicy.java
> ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
> -
>
> Key: HBASE-16660
> URL: https://issues.apache.org/jira/browse/HBASE-16660
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 0.98.20
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.23
>
> Attachments: HBASE-16660-0.98.patch, HBASE-16660.master.001.patch,
> HBASE-16660.master.001.patch
>
>
> We get an ArrayIndexOutOfBoundsException during the major compaction check as
> follows
> {noformat}
> 2016-09-19 05:04:18,287 ERROR [20.compactionChecker]
> regionserver.HRegionServer$CompactionChecker - Caught exception
> java.lang.ArrayIndexOutOfBoundsException: -2
> at
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.shouldPerformMajorCompaction(DateTieredCompactionPolicy.java:159)
> at
> org.apache.hadoop.hbase.regionserver.HStore.isMajorCompaction(HStore.java:1412)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$CompactionChecker.chore(HRegionServer.java:1532)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This happens due to the following lines in
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.selectMajorCompaction
> {noformat}
> int lowerWindowIndex = Collections.binarySearch(boundaries,
> minTimestamp == null ? Long.MAX_VALUE : file.getMinimumTimestamp());
> int upperWindowIndex = Collections.binarySearch(boundaries,
> file.getMaximumTimestamp() == null ? Long.MAX_VALUE :
> file.getMaximumTimestamp());
> {noformat}
> These return negative values if the element is not found and in the case the
> values are equal we get the exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16720) Sort build ids in flaky dashboard
[ https://issues.apache.org/jira/browse/HBASE-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16720: - Resolution: Fixed Status: Resolved (was: Patch Available) > Sort build ids in flaky dashboard > - > > Key: HBASE-16720 > URL: https://issues.apache.org/jira/browse/HBASE-16720 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16720.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16720) Sort build ids in flaky dashboard
[ https://issues.apache.org/jira/browse/HBASE-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527767#comment-15527767 ] Appy commented on HBASE-16720: -- Thanks [~dspivak] and [~jmhsieh]. > Sort build ids in flaky dashboard > - > > Key: HBASE-16720 > URL: https://issues.apache.org/jira/browse/HBASE-16720 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16720.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16720) Sort build ids in flaky dashboard
[ https://issues.apache.org/jira/browse/HBASE-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16720: - Fix Version/s: 2.0.0 > Sort build ids in flaky dashboard > - > > Key: HBASE-16720 > URL: https://issues.apache.org/jira/browse/HBASE-16720 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16720.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527759#comment-15527759 ] Hudson commented on HBASE-16134: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1685 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1685/]) HBASE-16134 Introduce Cell extension for server side. (anoopsamjohn: rev b644e0fb8d725b9a09665d7b92544eb32155c689) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/NoTagsKeyValue.java * (delete) hbase-common/src/main/java/org/apache/hadoop/hbase/Streamable.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/codec/KeyValueCodecWithTags.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/SettableSequenceId.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/SettableTimestamp.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/codec/KeyValueCodec.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/SizeCachedNoTagsKeyValue.java * (add) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCell.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/OffheapKeyValue.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch, HBASE-16134_V3.patch, HBASE-16134_V3.patch, > HBASE-16134_V4.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16660) ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
[
https://issues.apache.org/jira/browse/HBASE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527758#comment-15527758
]
Hudson commented on HBASE-16660:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1685 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1685/])
HBASE-16660 ArrayIndexOutOfBounds during the majorCompactionCheck in (apurtell:
rev d127d64266baed275a15d39407585a3bca4f6f15)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DateTieredCompactionPolicy.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDateTieredCompactionPolicy.java
> ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
> -
>
> Key: HBASE-16660
> URL: https://issues.apache.org/jira/browse/HBASE-16660
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 0.98.20
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.23
>
> Attachments: HBASE-16660-0.98.patch, HBASE-16660.master.001.patch,
> HBASE-16660.master.001.patch
>
>
> We get an ArrayIndexOutOfBoundsException during the major compaction check as
> follows
> {noformat}
> 2016-09-19 05:04:18,287 ERROR [20.compactionChecker]
> regionserver.HRegionServer$CompactionChecker - Caught exception
> java.lang.ArrayIndexOutOfBoundsException: -2
> at
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.shouldPerformMajorCompaction(DateTieredCompactionPolicy.java:159)
> at
> org.apache.hadoop.hbase.regionserver.HStore.isMajorCompaction(HStore.java:1412)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$CompactionChecker.chore(HRegionServer.java:1532)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This happens due to the following lines in
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.selectMajorCompaction
> {noformat}
> int lowerWindowIndex = Collections.binarySearch(boundaries,
> minTimestamp == null ? Long.MAX_VALUE : file.getMinimumTimestamp());
> int upperWindowIndex = Collections.binarySearch(boundaries,
> file.getMaximumTimestamp() == null ? Long.MAX_VALUE :
> file.getMaximumTimestamp());
> {noformat}
> These return negative values if the element is not found and in the case the
> values are equal we get the exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16661) Add last major compaction age to per-region metrics
[ https://issues.apache.org/jira/browse/HBASE-16661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527716#comment-15527716 ] Gary Helmling commented on HBASE-16661: --- The 002 patch looks good to me. The only minor comment is to add a LAST_MAJOR_COMPACTION_DESC constant to use as a short description of the metric in MetricsRegionSourceImpl, instead of just reusing the metric name. > Add last major compaction age to per-region metrics > --- > > Key: HBASE-16661 > URL: https://issues.apache.org/jira/browse/HBASE-16661 > Project: HBase > Issue Type: Improvement >Reporter: Gary Helmling >Assignee: Dustin Pho >Priority: Minor > Attachments: HBASE-16661.001.patch, HBASE-16661.002.patch > > > After HBASE-12859, we can now track the last major compaction timestamp for > each region. However, this is only exposed through cluster status reporting > and the admin API. > We have similar per-region metrics around storefile age, but none that > filters on major compaction specifically. > Let's add a metric for last major compaction age. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16117) Fix Connection leak in mapred.TableOutputFormat
[
https://issues.apache.org/jira/browse/HBASE-16117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527704#comment-15527704
]
Hadoop QA commented on HBASE-16117:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 53s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} branch-1 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 59s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} branch-1 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 26s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 57s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 19s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 141m 33s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestClusterId |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-27 |
| JIRA Patch URL |
[jira] [Updated] (HBASE-16710) Add ZStandard Codec to Compression.java
[ https://issues.apache.org/jira/browse/HBASE-16710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] churro morales updated HBASE-16710: --- Status: Patch Available (was: Open) > Add ZStandard Codec to Compression.java > --- > > Key: HBASE-16710 > URL: https://issues.apache.org/jira/browse/HBASE-16710 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: churro morales >Assignee: churro morales >Priority: Minor > Attachments: HBASE-16710-0.98.patch, HBASE-16710-1.2.patch, > HBASE-16710.patch > > > HADOOP-13578 is adding the ZStandardCodec to hadoop. This is a placeholder > to ensure it gets added to hbase once this gets upstream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16715) Signing keys could not be imported
[
https://issues.apache.org/jira/browse/HBASE-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francis Chuang resolved HBASE-16715.
Resolution: Not A Problem
> Signing keys could not be imported
> --
>
> Key: HBASE-16715
> URL: https://issues.apache.org/jira/browse/HBASE-16715
> Project: HBase
> Issue Type: Bug
>Reporter: Francis Chuang
>
> I am trying to import the signing keys to verify downloaded hbase releases,
> but it appears to fail:
> {code}
> $ wget -O /tmp/KEYS https://www-us.apache.org/dist/hbase/KEYS
> Connecting to www-us.apache.org (140.211.11.105:443)
> KEYS 100% |***| 50537 0:00:00
> ETA
> $ gpg --import /tmp/KEYS
> gpg: directory '/root/.gnupg' created
> gpg: new configuration file '/root/.gnupg/dirmngr.conf' created
> gpg: new configuration file '/root/.gnupg/gpg.conf' created
> gpg: keybox '/root/.gnupg/pubring.kbx' created
> gpg: /root/.gnupg/trustdb.gpg: trustdb created
> gpg: key 945D66AF: public key "Jean-Daniel Cryans (ASF key)
> " imported
> gpg: key D34B98D6: public key "Michael Stack " imported
> gpg: key 30CD0996: public key "Michael Stack " imported
> gpg: key AEC77EAF: public key "Todd Lipcon " imported
> gpg: key F48B08A4: public key "Ted Yu (Apache Public Key)
> " imported
> gpg: key 867B57B8: public key "Ramkrishna S Vasudevan (for code checkin)
> " imported
> gpg: key 7CA45750: public key "Lars Hofhansl (CODE SIGNING KEY)
> " imported
> gpg: key A1AC25A9: public key "Lars Hofhansl (CODE SIGNING KEY)
> " imported
> gpg: key C7CFE328: public key "Lars Hofhansl (CODE SIGNING KEY)
> " imported
> gpg: key E964B5FF: public key "Enis Soztutar (CODE SIGNING KEY)
> " imported
> gpg: key 0D80DB7C: public key "Sean Busbey (CODE SIGNING KEY)
> " imported
> gpg: key 8644EEB6: public key "Nick Dimiduk " imported
> gpg: invalid radix64 character 3A skipped
> gpg: CRC error; E1B6C3 - DFECFB
> gpg: [don't know]: invalid packet (ctb=55)
> gpg: read_block: read error: Invalid packet
> gpg: import from '/tmp/KEYS' failed: Invalid keyring
> gpg: Total number processed: 12
> gpg: imported: 12
> gpg: no ultimately trusted keys found
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16715) Signing keys could not be imported
[
https://issues.apache.org/jira/browse/HBASE-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527601#comment-15527601
]
Francis Chuang commented on HBASE-16715:
I just tried it a few minutes ago and it appears to work properly where as it
was failing on my machine and docker hub yesterday. I think they key file is
refreshed daily and that might have fixed it.
> Signing keys could not be imported
> --
>
> Key: HBASE-16715
> URL: https://issues.apache.org/jira/browse/HBASE-16715
> Project: HBase
> Issue Type: Bug
>Reporter: Francis Chuang
>
> I am trying to import the signing keys to verify downloaded hbase releases,
> but it appears to fail:
> {code}
> $ wget -O /tmp/KEYS https://www-us.apache.org/dist/hbase/KEYS
> Connecting to www-us.apache.org (140.211.11.105:443)
> KEYS 100% |***| 50537 0:00:00
> ETA
> $ gpg --import /tmp/KEYS
> gpg: directory '/root/.gnupg' created
> gpg: new configuration file '/root/.gnupg/dirmngr.conf' created
> gpg: new configuration file '/root/.gnupg/gpg.conf' created
> gpg: keybox '/root/.gnupg/pubring.kbx' created
> gpg: /root/.gnupg/trustdb.gpg: trustdb created
> gpg: key 945D66AF: public key "Jean-Daniel Cryans (ASF key)
> " imported
> gpg: key D34B98D6: public key "Michael Stack " imported
> gpg: key 30CD0996: public key "Michael Stack " imported
> gpg: key AEC77EAF: public key "Todd Lipcon " imported
> gpg: key F48B08A4: public key "Ted Yu (Apache Public Key)
> " imported
> gpg: key 867B57B8: public key "Ramkrishna S Vasudevan (for code checkin)
> " imported
> gpg: key 7CA45750: public key "Lars Hofhansl (CODE SIGNING KEY)
> " imported
> gpg: key A1AC25A9: public key "Lars Hofhansl (CODE SIGNING KEY)
> " imported
> gpg: key C7CFE328: public key "Lars Hofhansl (CODE SIGNING KEY)
> " imported
> gpg: key E964B5FF: public key "Enis Soztutar (CODE SIGNING KEY)
> " imported
> gpg: key 0D80DB7C: public key "Sean Busbey (CODE SIGNING KEY)
> " imported
> gpg: key 8644EEB6: public key "Nick Dimiduk " imported
> gpg: invalid radix64 character 3A skipped
> gpg: CRC error; E1B6C3 - DFECFB
> gpg: [don't know]: invalid packet (ctb=55)
> gpg: read_block: read error: Invalid packet
> gpg: import from '/tmp/KEYS' failed: Invalid keyring
> gpg: Total number processed: 12
> gpg: imported: 12
> gpg: no ultimately trusted keys found
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16117) Fix Connection leak in mapred.TableOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-16117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-16117: --- Attachment: hbase-16117-part2.v5.patch This is an addendum that undoes some of v4 that was committed in june to make the patch resemble branch-1's more and also undoes the release note warning. I should really just close this issue as it and file a new jira to commit this patch and then branch-1 version. > Fix Connection leak in mapred.TableOutputFormat > > > Key: HBASE-16117 > URL: https://issues.apache.org/jira/browse/HBASE-16117 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 2.0.0, 1.3.0, 1.2.2, 1.1.6 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0, 1.3.1, 1.1.7, 1.2.4 > > Attachments: HBASE-16117.branch-1.001.patch, > hbase-16117-part2.v5.patch, hbase-16117.branch-1.002.patch, > hbase-16117.branch-1.patch, hbase-16117.patch, hbase-16117.v2.branch-1.patch, > hbase-16117.v2.patch, hbase-16117.v3.branch-1.patch, hbase-16117.v3.patch, > hbase-16117.v4.patch > > > Spark seems to instantiate multiple instances of output formats within a > single process. When mapred.TableOutputFormat (not > mapreduce.TableOutputFormat) is used, this may cause connection leaks that > slowly exhaust the cluster's zk connections. > This patch fixes that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16117) Fix Connection leak in mapred.TableOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-16117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527519#comment-15527519 ] Jonathan Hsieh edited comment on HBASE-16117 at 9/27/16 9:51 PM: - part2.v5 is an addendum that undoes some of v4 that was committed in june to make the patch resemble branch-1's more and also undoes the release note warning. I should really just close this issue as it and file a new jira to commit this patch and then branch-1 version. was (Author: jmhsieh): This is an addendum that undoes some of v4 that was committed in june to make the patch resemble branch-1's more and also undoes the release note warning. I should really just close this issue as it and file a new jira to commit this patch and then branch-1 version. > Fix Connection leak in mapred.TableOutputFormat > > > Key: HBASE-16117 > URL: https://issues.apache.org/jira/browse/HBASE-16117 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 2.0.0, 1.3.0, 1.2.2, 1.1.6 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0, 1.3.1, 1.1.7, 1.2.4 > > Attachments: HBASE-16117.branch-1.001.patch, > hbase-16117-part2.v5.patch, hbase-16117.branch-1.002.patch, > hbase-16117.branch-1.patch, hbase-16117.patch, hbase-16117.v2.branch-1.patch, > hbase-16117.v2.patch, hbase-16117.v3.branch-1.patch, hbase-16117.v3.patch, > hbase-16117.v4.patch > > > Spark seems to instantiate multiple instances of output formats within a > single process. When mapred.TableOutputFormat (not > mapreduce.TableOutputFormat) is used, this may cause connection leaks that > slowly exhaust the cluster's zk connections. > This patch fixes that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527475#comment-15527475 ] Sean Busbey commented on HBASE-16179: - So you'd like us to just mandate Scala 2.10 for both 1.6 and 2.0, right? Reading through the threads when dev@spark talks about scala versions I got the sense that Scala 2.10 won't be long for the 2.y line. When that time comes, what would we do? I'm not necessarily asking that we address this now, just want to know that we've thought of a general approach to supportability. We haven't taken a stance as a community yet on support timelines for Scala versions, but I'd expect we'd treat them similar to JDK versions, which would means we'd keep a Scala 2.10 based Spark 2 module for the remainder of HBase 2.y. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt, > 16179.v5.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527458#comment-15527458 ] Ted Yu commented on HBASE-16179: Since Scala 2.10 is supported by Spark 2.0, it is separate dimension. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt, > 16179.v5.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16660) ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
[
https://issues.apache.org/jira/browse/HBASE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527446#comment-15527446
]
Hudson commented on HBASE-16660:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #25 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/25/])
HBASE-16660 ArrayIndexOutOfBounds during the majorCompactionCheck in (apurtell:
rev e0066e713d6bd4f499e479ea173677e3fc14d2a2)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DateTieredCompactionPolicy.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDateTieredCompactionPolicy.java
> ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
> -
>
> Key: HBASE-16660
> URL: https://issues.apache.org/jira/browse/HBASE-16660
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 0.98.20
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.23
>
> Attachments: HBASE-16660-0.98.patch, HBASE-16660.master.001.patch,
> HBASE-16660.master.001.patch
>
>
> We get an ArrayIndexOutOfBoundsException during the major compaction check as
> follows
> {noformat}
> 2016-09-19 05:04:18,287 ERROR [20.compactionChecker]
> regionserver.HRegionServer$CompactionChecker - Caught exception
> java.lang.ArrayIndexOutOfBoundsException: -2
> at
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.shouldPerformMajorCompaction(DateTieredCompactionPolicy.java:159)
> at
> org.apache.hadoop.hbase.regionserver.HStore.isMajorCompaction(HStore.java:1412)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$CompactionChecker.chore(HRegionServer.java:1532)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This happens due to the following lines in
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.selectMajorCompaction
> {noformat}
> int lowerWindowIndex = Collections.binarySearch(boundaries,
> minTimestamp == null ? Long.MAX_VALUE : file.getMinimumTimestamp());
> int upperWindowIndex = Collections.binarySearch(boundaries,
> file.getMaximumTimestamp() == null ? Long.MAX_VALUE :
> file.getMaximumTimestamp());
> {noformat}
> These return negative values if the element is not found and in the case the
> values are equal we get the exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16179: --- Attachment: 16179.v5.txt Patch v5 adds DataTypeParser trait which can serve as the basis for multi-module support. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt, > 16179.v5.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16720) Sort build ids in flaky dashboard
[ https://issues.apache.org/jira/browse/HBASE-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527408#comment-15527408 ] Dima Spivak commented on HBASE-16720: - +1. > Sort build ids in flaky dashboard > - > > Key: HBASE-16720 > URL: https://issues.apache.org/jira/browse/HBASE-16720 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16720.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16117) Fix Connection leak in mapred.TableOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-16117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-16117: --- Attachment: hbase-16117.branch-1.002.patch New version of a patch for branch-1. I may be able to undo some of the interface changes in trunk -- will submit a new patch for trunk. > Fix Connection leak in mapred.TableOutputFormat > > > Key: HBASE-16117 > URL: https://issues.apache.org/jira/browse/HBASE-16117 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 2.0.0, 1.3.0, 1.2.2, 1.1.6 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0, 1.3.1, 1.1.7, 1.2.4 > > Attachments: HBASE-16117.branch-1.001.patch, > hbase-16117.branch-1.002.patch, hbase-16117.branch-1.patch, > hbase-16117.patch, hbase-16117.v2.branch-1.patch, hbase-16117.v2.patch, > hbase-16117.v3.branch-1.patch, hbase-16117.v3.patch, hbase-16117.v4.patch > > > Spark seems to instantiate multiple instances of output formats within a > single process. When mapred.TableOutputFormat (not > mapreduce.TableOutputFormat) is used, this may cause connection leaks that > slowly exhaust the cluster's zk connections. > This patch fixes that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16720) Sort build ids in flaky dashboard
[ https://issues.apache.org/jira/browse/HBASE-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527256#comment-15527256 ] Jonathan Hsieh commented on HBASE-16720: +1 lgtm. > Sort build ids in flaky dashboard > - > > Key: HBASE-16720 > URL: https://issues.apache.org/jira/browse/HBASE-16720 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16720.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16686) Add latency metrics for REST
[
https://issues.apache.org/jira/browse/HBASE-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527253#comment-15527253
]
Guang Yang commented on HBASE-16686:
Thanks [~apurtell] for the review.
{quote}
Convention across the code base is to use EnvironmentEdgeManager#getTime()
instead of System#currentTimeMillis so the time can be controlled in unit tests
{quote}
Thanks for pointing that out, will update the patch accordingly.
{quote}
While a simple count of exceptions could be useful, does it make sense to break
down the counts for common exceptions of interest?
{quote}
Yeah that would be helpful, let me take a look at the common exceptions that we
may report individually.
As for this patch, I will first focus on getting Jetty 9 work
(https://issues.apache.org/jira/browse/HBASE-12894), and then consolidate a
patch for metrics, since it would be helpful to have some metrics from Jetty as
well (e.g. queue length, busy/idle threads, etc.).
> Add latency metrics for REST
>
>
> Key: HBASE-16686
> URL: https://issues.apache.org/jira/browse/HBASE-16686
> Project: HBase
> Issue Type: New Feature
> Components: monitoring, REST
>Reporter: Guang Yang
>Priority: Minor
> Attachments: HBASE-16686_v0.patch
>
>
> It would be helpful to have the latency metrics for rest for various
> operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15721) Optimization in cloning cells into MSLAB
[
https://issues.apache.org/jira/browse/HBASE-15721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527239#comment-15527239
]
Hadoop QA commented on HBASE-15721:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 44s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 35s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 29s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 43s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 15s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 126m 32s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
| Timed out junit tests |
org.apache.hadoop.hbase.replication.TestReplicationStateHBaseImpl |
| | org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
|
| |
org.apache.hadoop.hbase.master.procedure.TestDispatchMergingRegionsProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestTruncateTableProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830534/HBASE-15721_V3.patch |
| JIRA Issue | HBASE-15721 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux c277dd7ff37d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-16574) Add backup / restore feature to refguide
[ https://issues.apache.org/jira/browse/HBASE-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527215#comment-15527215 ] Josh Elser commented on HBASE-16574: bq. commit message should list Frank Welsch as an additional author Good point -- let me fix that in the patch right now before it gets lost. Thanks in advance for finding the time to review, Sean. bq. pdf version of refguide corresponding to v1 patch. Thanks, Ted. I should have done that myself :) > Add backup / restore feature to refguide > > > Key: HBASE-16574 > URL: https://issues.apache.org/jira/browse/HBASE-16574 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu > Labels: backup > Attachments: Backup-and-Restore-Apache_19Sep2016.pdf, > HBASE-16574.001.patch, hbase_reference_guide.v1.pdf > > > This issue is to add backup / restore feature description to hbase refguide. > The description should cover: > scenarios where backup / restore is used > backup / restore commands and sample usage > considerations in setup -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16691) Optimize KeyOnlyFilter by utilizing KeyOnlyCell
[ https://issues.apache.org/jira/browse/HBASE-16691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527187#comment-15527187 ] Hudson commented on HBASE-16691: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1684 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1684/]) HBASE-16691 Optimize KeyOnlyFilter by utilizing KeyOnlyCell.(binlijin) (anoopsamjohn: rev b0fcca6d7bfb7a897d51537b7485acb6e2bb4714) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/filter/KeyOnlyFilter.java * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/filter/TestKeyOnlyFilter.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java > Optimize KeyOnlyFilter by utilizing KeyOnlyCell > --- > > Key: HBASE-16691 > URL: https://issues.apache.org/jira/browse/HBASE-16691 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0 > > Attachments: HBASE-16691-addendum.patch, HBASE-16691-master.patch, > HBASE-16691-master_v2.patch > > > In KeyOnlyFilter#transformCell will return a KeyOnlyCell that have no value > or has valueLength as value, current will copy all row keys into a new byte[] > and new a KeyValue, we can eliminate the copy and have a wrap KeyOnlyCell > that ignore the cell's value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16720) Sort build ids in flaky dashboard
[
https://issues.apache.org/jira/browse/HBASE-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527159#comment-15527159
]
Hadoop QA commented on HBASE-16720:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} pylint {color} | {color:green} 0m 3s
{color} | {color:green} There were no new pylint issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 16s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 8s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830550/HBASE-16720.master.001.patch
|
| JIRA Issue | HBASE-16720 |
| Optional Tests | asflicense pylint |
| uname | Linux ab8b3820e1bb 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / d127d64 |
| pylint | v1.6.4 |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3741/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Sort build ids in flaky dashboard
> -
>
> Key: HBASE-16720
> URL: https://issues.apache.org/jira/browse/HBASE-16720
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Minor
> Attachments: HBASE-16720.master.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527158#comment-15527158 ] Edward Bortnikov commented on HBASE-16608: -- Made one more pass on RB, seems we're good. Suggest to commit, and keep stress-testing the system (PE and YCSB) for correctness and performance. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15134) Add visibility into Flush and Compaction queues
[ https://issues.apache.org/jira/browse/HBASE-15134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527111#comment-15527111 ] Abhishek Singh Chouhan commented on HBASE-15134: [~busbey] [~anoop.hbase] Do have a look when you get time. Woukd appreciate any comments. Thanks. > Add visibility into Flush and Compaction queues > --- > > Key: HBASE-15134 > URL: https://issues.apache.org/jira/browse/HBASE-15134 > Project: HBase > Issue Type: New Feature > Components: Compaction, metrics, regionserver >Reporter: Elliott Clark >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-15134.master.001.patch, > HBASE-15134.master.002.patch, HBASE-15134.patch, HBASE-15134.patch > > > On busy spurts we can see regionservers start to see large queues for > compaction. It's really hard to tell if the server is queueing a lot of > compactions for the same region, lots of compactions for lots of regions, or > just falling behind. > For flushes much the same. There can be flushes in queue that aren't being > run because of delayed flushes. There's no way to know from the metrics how > many flushes are for each region, how many are delayed. Etc. > We should add either more metrics around this ( num per region, max per > region, min per region ) or add on a UI page that has the list of compactions > and flushes. > Or both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6581) Build with hadoop.profile=3.0
[
https://issues.apache.org/jira/browse/HBASE-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527096#comment-15527096
]
Hadoop QA commented on HBASE-6581:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hbase-assembly {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 39s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 18s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hbase-assembly {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 45s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 42s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 16s
{color} | {color:green} hbase-assembly in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 126m 22s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| |
[jira] [Updated] (HBASE-16682) Fix Shell tests failure. NoClassDefFoundError for MiniKdc
[
https://issues.apache.org/jira/browse/HBASE-16682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-16682:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix Shell tests failure. NoClassDefFoundError for MiniKdc
> -
>
> Key: HBASE-16682
> URL: https://issues.apache.org/jira/browse/HBASE-16682
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: HBASE-16682.master.001.patch,
> HBASE-16682.master.002.patch, HBASE-16682.master.003.patch
>
>
> Stacktrace
> {noformat}
> java.lang.NoClassDefFoundError: org/apache/hadoop/minikdc/MiniKdc
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.getDeclaredMethods(Class.java:1975)
> at org.jruby.javasupport.JavaClass.getMethods(JavaClass.java:2110)
> at org.jruby.javasupport.JavaClass.setupClassMethods(JavaClass.java:955)
> at org.jruby.javasupport.JavaClass.access$700(JavaClass.java:99)
> at
> org.jruby.javasupport.JavaClass$ClassInitializer.initialize(JavaClass.java:650)
> at org.jruby.javasupport.JavaClass.setupProxy(JavaClass.java:689)
> at org.jruby.javasupport.Java.createProxyClass(Java.java:526)
> at org.jruby.javasupport.Java.getProxyClass(Java.java:455)
> at org.jruby.javasupport.Java.getInstance(Java.java:364)
> at
> org.jruby.javasupport.JavaUtil.convertJavaToUsableRubyObject(JavaUtil.java:166)
> at
> org.jruby.javasupport.JavaEmbedUtils.javaToRuby(JavaEmbedUtils.java:291)
> at
> org.jruby.embed.variable.AbstractVariable.updateByJavaObject(AbstractVariable.java:81)
> at
> org.jruby.embed.variable.GlobalVariable.(GlobalVariable.java:69)
> at
> org.jruby.embed.variable.GlobalVariable.getInstance(GlobalVariable.java:60)
> at
> org.jruby.embed.variable.VariableInterceptor.getVariableInstance(VariableInterceptor.java:97)
> at org.jruby.embed.internal.BiVariableMap.put(BiVariableMap.java:321)
> at org.jruby.embed.ScriptingContainer.put(ScriptingContainer.java:1123)
> at
> org.apache.hadoop.hbase.client.AbstractTestShell.setUpBeforeClass(AbstractTestShell.java:61)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:108)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:78)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:144)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.minikdc.MiniKdc
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at
[jira] [Updated] (HBASE-16720) Sort build ids in flaky dashboard
[ https://issues.apache.org/jira/browse/HBASE-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16720: - Status: Patch Available (was: Open) > Sort build ids in flaky dashboard > - > > Key: HBASE-16720 > URL: https://issues.apache.org/jira/browse/HBASE-16720 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16720.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16720) Sort build ids in flaky dashboard
[ https://issues.apache.org/jira/browse/HBASE-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16720: - Attachment: HBASE-16720.master.001.patch > Sort build ids in flaky dashboard > - > > Key: HBASE-16720 > URL: https://issues.apache.org/jira/browse/HBASE-16720 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16720.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Manes updated HBASE-15560: -- Attachment: HBASE-15560.patch Fixed JavaDoc. I am unable to reproduce the test failure locally when I run, $ mvn test -Dtest=org.apache.hadoop.hbase.filter.TestFilter > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[
https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527056#comment-15527056
]
Hadoop QA commented on HBASE-16608:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 55s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 38s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 160m 55s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestAdmin2 |
| | org.apache.hadoop.hbase.client.replication.TestReplicationAdmin |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830515/HBASE-16608-V04.patch
|
| JIRA Issue | HBASE-16608 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 0e3c2b401650 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / aa4a678 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3736/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3736/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3736/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3736/console |
| Powered
[jira] [Created] (HBASE-16720) Sort build ids in flaky dashboard
Appy created HBASE-16720: Summary: Sort build ids in flaky dashboard Key: HBASE-16720 URL: https://issues.apache.org/jira/browse/HBASE-16720 Project: HBase Issue Type: Improvement Reporter: Appy Assignee: Appy Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16660) ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
[
https://issues.apache.org/jira/browse/HBASE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527054#comment-15527054
]
Hadoop QA commented on HBASE-16660:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 31s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 30s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 160m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestAdmin2 |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830511/HBASE-16660.master.001.patch
|
| JIRA Issue | HBASE-16660 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a3c109ec9c8f 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / aa4a678 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3737/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3737/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3737/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3737/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This
[jira] [Updated] (HBASE-16660) ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
[
https://issues.apache.org/jira/browse/HBASE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16660:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.3.0
Status: Resolved (was: Patch Available)
Pushed, thanks for the patch [~abhishek.chouhan]
> ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
> -
>
> Key: HBASE-16660
> URL: https://issues.apache.org/jira/browse/HBASE-16660
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 0.98.20
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.23
>
> Attachments: HBASE-16660-0.98.patch, HBASE-16660.master.001.patch,
> HBASE-16660.master.001.patch
>
>
> We get an ArrayIndexOutOfBoundsException during the major compaction check as
> follows
> {noformat}
> 2016-09-19 05:04:18,287 ERROR [20.compactionChecker]
> regionserver.HRegionServer$CompactionChecker - Caught exception
> java.lang.ArrayIndexOutOfBoundsException: -2
> at
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.shouldPerformMajorCompaction(DateTieredCompactionPolicy.java:159)
> at
> org.apache.hadoop.hbase.regionserver.HStore.isMajorCompaction(HStore.java:1412)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$CompactionChecker.chore(HRegionServer.java:1532)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This happens due to the following lines in
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.selectMajorCompaction
> {noformat}
> int lowerWindowIndex = Collections.binarySearch(boundaries,
> minTimestamp == null ? Long.MAX_VALUE : file.getMinimumTimestamp());
> int upperWindowIndex = Collections.binarySearch(boundaries,
> file.getMaximumTimestamp() == null ? Long.MAX_VALUE :
> file.getMaximumTimestamp());
> {noformat}
> These return negative values if the element is not found and in the case the
> values are equal we get the exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16710) Add ZStandard Codec to Compression.java
[ https://issues.apache.org/jira/browse/HBASE-16710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526984#comment-15526984 ] churro morales commented on HBASE-16710: It can be if you guys want it. All we use this for is CF property so I guess it is less typing. Let me know what you guys prefer and I will get the patches up. > Add ZStandard Codec to Compression.java > --- > > Key: HBASE-16710 > URL: https://issues.apache.org/jira/browse/HBASE-16710 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: churro morales >Assignee: churro morales >Priority: Minor > Attachments: HBASE-16710-0.98.patch, HBASE-16710-1.2.patch, > HBASE-16710.patch > > > HADOOP-13578 is adding the ZStandardCodec to hadoop. This is a placeholder > to ensure it gets added to hbase once this gets upstream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526947#comment-15526947 ] Sean Busbey commented on HBASE-16179: - what about the scala binary compat issue? Can we keep Scala out of the shim module and then use scala 2.10 for 1.6 and scala 2.11 for 2.0? > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16179: Fix Version/s: 2.0.0 > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526938#comment-15526938 ] Sean Busbey commented on HBASE-16179: - yeah that sounds good to me. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16715) Signing keys could not be imported
[
https://issues.apache.org/jira/browse/HBASE-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526926#comment-15526926
]
Josh Elser commented on HBASE-16715:
Hey [~francischuang] -- I just tried this myself and it worked fine. Any chance
you still have that KEYS file that you downloaded?
> Signing keys could not be imported
> --
>
> Key: HBASE-16715
> URL: https://issues.apache.org/jira/browse/HBASE-16715
> Project: HBase
> Issue Type: Bug
>Reporter: Francis Chuang
>
> I am trying to import the signing keys to verify downloaded hbase releases,
> but it appears to fail:
> {code}
> $ wget -O /tmp/KEYS https://www-us.apache.org/dist/hbase/KEYS
> Connecting to www-us.apache.org (140.211.11.105:443)
> KEYS 100% |***| 50537 0:00:00
> ETA
> $ gpg --import /tmp/KEYS
> gpg: directory '/root/.gnupg' created
> gpg: new configuration file '/root/.gnupg/dirmngr.conf' created
> gpg: new configuration file '/root/.gnupg/gpg.conf' created
> gpg: keybox '/root/.gnupg/pubring.kbx' created
> gpg: /root/.gnupg/trustdb.gpg: trustdb created
> gpg: key 945D66AF: public key "Jean-Daniel Cryans (ASF key)
> " imported
> gpg: key D34B98D6: public key "Michael Stack " imported
> gpg: key 30CD0996: public key "Michael Stack " imported
> gpg: key AEC77EAF: public key "Todd Lipcon " imported
> gpg: key F48B08A4: public key "Ted Yu (Apache Public Key)
> " imported
> gpg: key 867B57B8: public key "Ramkrishna S Vasudevan (for code checkin)
> " imported
> gpg: key 7CA45750: public key "Lars Hofhansl (CODE SIGNING KEY)
> " imported
> gpg: key A1AC25A9: public key "Lars Hofhansl (CODE SIGNING KEY)
> " imported
> gpg: key C7CFE328: public key "Lars Hofhansl (CODE SIGNING KEY)
> " imported
> gpg: key E964B5FF: public key "Enis Soztutar (CODE SIGNING KEY)
> " imported
> gpg: key 0D80DB7C: public key "Sean Busbey (CODE SIGNING KEY)
> " imported
> gpg: key 8644EEB6: public key "Nick Dimiduk " imported
> gpg: invalid radix64 character 3A skipped
> gpg: CRC error; E1B6C3 - DFECFB
> gpg: [don't know]: invalid packet (ctb=55)
> gpg: read_block: read error: Invalid packet
> gpg: import from '/tmp/KEYS' failed: Invalid keyring
> gpg: Total number processed: 12
> gpg: imported: 12
> gpg: no ultimately trusted keys found
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16710) Add ZStandard Codec to Compression.java
[
https://issues.apache.org/jira/browse/HBASE-16710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526898#comment-15526898
]
Dave Latham commented on HBASE-16710:
-
Should it be {{ZSTD}} instead of {{ZSTANDARD}}?
> Add ZStandard Codec to Compression.java
> ---
>
> Key: HBASE-16710
> URL: https://issues.apache.org/jira/browse/HBASE-16710
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: churro morales
>Assignee: churro morales
>Priority: Minor
> Attachments: HBASE-16710-0.98.patch, HBASE-16710-1.2.patch,
> HBASE-16710.patch
>
>
> HADOOP-13578 is adding the ZStandardCodec to hadoop. This is a placeholder
> to ensure it gets added to hbase once this gets upstream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526894#comment-15526894
]
Yu Li commented on HBASE-16698:
---
I don't have much benchmarking number at hand, but from our online cluster we
see no regression on put average time.
Notice that the problem exists even if there's only one region but many
parallel writes (yes, after a relook I think I stated something wrong, the
issue stands even if there's only one region), allow me to quote the existing
code flow on append handling:
{noformat}
RingBufferEventHandler grab one append
-> FSHLog#append is called
-> FSWALEntry#stampRegionSequenceId is called
-> One CountDownLatch is released
-> RingBufferEventHandler grab another append
-> Another CountDownLatch is released
-> Repeat
{noformat}
so the main problem is *sequential* appends and the logic that getting MVCC has
to wait for the relative append to finish.
I'll supply some perf number with YCSB, but should be days later because of
some headache online issues... Or it will be highly appreciated if anyone could
offer some help on the bench testing.
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.1.6, 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-16698.patch, HBASE-16698.v2.patch,
> hadoop0495.et2.jstack
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15721) Optimization in cloning cells into MSLAB
[
https://issues.apache.org/jira/browse/HBASE-15721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-15721:
---
Attachment: HBASE-15721_V3.patch
> Optimization in cloning cells into MSLAB
>
>
> Key: HBASE-15721
> URL: https://issues.apache.org/jira/browse/HBASE-15721
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15721.patch, HBASE-15721_V2.patch,
> HBASE-15721_V3.patch
>
>
> Before cells added to memstore CSLM, there is a clone of cell after copying
> it to MSLAB chunk area. This is done not in an efficient way.
> {code}
> public static int appendToByteArray(final Cell cell, final byte[] output,
> final int offset) {
> int pos = offset;
> pos = Bytes.putInt(output, pos, keyLength(cell));
> pos = Bytes.putInt(output, pos, cell.getValueLength());
> pos = appendKeyTo(cell, output, pos);
> pos = CellUtil.copyValueTo(cell, output, pos);
> if ((cell.getTagsLength() > 0)) {
> pos = Bytes.putAsShort(output, pos, cell.getTagsLength());
> pos = CellUtil.copyTagTo(cell, output, pos);
> }
> return pos;
> }
> {code}
> Copied in 9 steps and we end up parsing all lengths. When the cell
> implementation is backed by a single byte[] (Like KeyValue) this can be done
> in single step copy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16710) Add ZStandard Codec to Compression.java
[ https://issues.apache.org/jira/browse/HBASE-16710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] churro morales updated HBASE-16710: --- Attachment: HBASE-16710-0.98.patch HBASE-16710-1.2.patch HBASE-16710.patch HBASE-16710.patch works with [master / branch-1.3] The others list in the name which branch they work for. > Add ZStandard Codec to Compression.java > --- > > Key: HBASE-16710 > URL: https://issues.apache.org/jira/browse/HBASE-16710 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: churro morales >Assignee: churro morales >Priority: Minor > Attachments: HBASE-16710-0.98.patch, HBASE-16710-1.2.patch, > HBASE-16710.patch > > > HADOOP-13578 is adding the ZStandardCodec to hadoop. This is a placeholder > to ensure it gets added to hbase once this gets upstream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16134: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the reviews. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch, HBASE-16134_V3.patch, HBASE-16134_V3.patch, > HBASE-16134_V4.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11354) HConnectionImplementation#DelayedClosing does not start
[
https://issues.apache.org/jira/browse/HBASE-11354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526827#comment-15526827
]
Hudson commented on HBASE-11354:
FAILURE: Integrated in Jenkins build HBase-0.98-on-Hadoop-1.1 #1277 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1277/])
HBASE-11354 HConnectionImplementation#DelayedClosing does not start (apurtell:
rev 2b6da527b31c70c5ec764e09b109fc4d3ab6704f)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
> HConnectionImplementation#DelayedClosing does not start
> ---
>
> Key: HBASE-11354
> URL: https://issues.apache.org/jira/browse/HBASE-11354
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.99.0, 0.98.3
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
>Priority: Minor
> Fix For: 0.98.23
>
> Attachments: HBASE-11354-0.98.patch, HBASE_11354 (1).patch,
> HBASE_11354.patch, HBASE_11354.patch, HBASE_11354.patch
>
>
> The method "createAndStart" in class DelayedClosing only creates a instance,
> but forgets to start it. So thread delayedClosing is not running all the time.
> ConnectionManager#1623
> {code}
> static DelayedClosing createAndStart(HConnectionImplementation hci){
> Stoppable stoppable = new Stoppable() {
> private volatile boolean isStopped = false;
> @Override public void stop(String why) { isStopped = true;}
> @Override public boolean isStopped() {return isStopped;}
> };
> return new DelayedClosing(hci, stoppable);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16694) Reduce garbage for onDiskChecksum in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-16694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526828#comment-15526828 ] Hudson commented on HBASE-16694: FAILURE: Integrated in Jenkins build HBase-0.98-on-Hadoop-1.1 #1277 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1277/]) HBASE-16694 Reduce garbage for onDiskChecksum in HFileBlock (binlijin) (apurtell: rev 6903296a3c181958eda82d0e2cbe24aba986efda) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java > Reduce garbage for onDiskChecksum in HFileBlock > --- > > Key: HBASE-16694 > URL: https://issues.apache.org/jira/browse/HBASE-16694 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.. >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0, 1.4.0, 0.98.23 > > Attachments: HBASE-16694-master.patch > > > Current when finish a HFileBlock will create a new byte[] for onDiskChecksum, > we can reuse it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16649) Truncate table with splits preserved can cause both data loss and truncated data appeared again
[ https://issues.apache.org/jira/browse/HBASE-16649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526826#comment-15526826 ] Hudson commented on HBASE-16649: FAILURE: Integrated in Jenkins build HBase-0.98-on-Hadoop-1.1 #1277 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1277/]) HBASE-16649 Truncate table with splits preserved can cause both data (matteo.bertozzi: rev 1bdf05b1fa1c3d09b9e3c4fb43957c5ff32ac701) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/TruncateTableHandler.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java > Truncate table with splits preserved can cause both data loss and truncated > data appeared again > --- > > Key: HBASE-16649 > URL: https://issues.apache.org/jira/browse/HBASE-16649 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.3 >Reporter: Allan Yang >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.3.0, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-16649-v0.patch, HBASE-16649-v1.patch, > HBASE-16649-v2.patch > > > Since truncate table with splits preserved will delete hfiles and use the > previous regioninfo. It can cause odd behaviors > - Case 1: *Data appeared after truncate* > reproduce procedureļ¼ > 1. create a table, let's say 'test' > 2. write data to 'test', make sure memstore of 'test' is not empty > 3. truncate 'test' with splits preserved > 4. kill the regionserver hosting the region(s) of 'test' > 5. start the regionserver, now it is the time to witness the miracle! the > truncated data appeared in table 'test' > - Case 2: *Data loss* > reproduce procedure: > 1. create a table, let's say 'test' > 2. write some data to 'test', no matter how many > 3. truncate 'test' with splits preserved > 4. restart the regionserver to reset the seqid > 5. write some data, but less than 2 since we don't want the seqid to run over > the one in 2 > 6. kill the regionserver hosting the region(s) of 'test' > 7. restart the regionserver. Congratulations! the data writen in 4 is now all > lost > *Why?* > for case 1 > Since preserve splits in truncate table procedure will not change the > regioninfo, when log replay happens, the 'unflushed' data will restore back > to the region > for case 2 > since the flushedSequenceIdByRegion are stored in Master in a map with the > region's encodedName. Although the table is truncated, the region's name is > not changed since we chose to preserve the splits. So after truncate the > table, the region's sequenceid is reset in the regionserver, but not reset in > master. When flush comes and report to master, master will reject the update > of sequenceid since the new one is smaller than the old one. The same happens > in log replay, all the edits writen in 4 will be skipped since they have a > smaller seqid -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16705) Eliminate long to Long auto boxing in LongComparator
[
https://issues.apache.org/jira/browse/HBASE-16705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526825#comment-15526825
]
Hudson commented on HBASE-16705:
FAILURE: Integrated in Jenkins build HBase-0.98-on-Hadoop-1.1 #1277 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1277/])
HBASE-16705 Eliminate long to Long auto boxing in LongComparator. (apurtell:
rev 82187196686f7fda383e96bec91787b507c993a8)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/filter/LongComparator.java
> Eliminate long to Long auto boxing in LongComparator
>
>
> Key: HBASE-16705
> URL: https://issues.apache.org/jira/browse/HBASE-16705
> Project: HBase
> Issue Type: Improvement
> Components: Filters
>Affects Versions: 2.0..
>Reporter: binlijin
>Assignee: binlijin
>Priority: Minor
> Fix For: 2.0.0, 1.4.0, 0.98.23
>
> Attachments: HBASE-16705-master.patch
>
>
> LongComparator
> @Override
> public int compareTo(byte[] value, int offset, int length) {
> Long that = Bytes.toLong(value, offset, length);
> return this.longValue.compareTo(that);
> }
> Every time need to convert long to Long, this is not necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16691) Optimize KeyOnlyFilter by utilizing KeyOnlyCell
[ https://issues.apache.org/jira/browse/HBASE-16691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16691: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed V2 patch. Resolving again. > Optimize KeyOnlyFilter by utilizing KeyOnlyCell > --- > > Key: HBASE-16691 > URL: https://issues.apache.org/jira/browse/HBASE-16691 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0 > > Attachments: HBASE-16691-addendum.patch, HBASE-16691-master.patch, > HBASE-16691-master_v2.patch > > > In KeyOnlyFilter#transformCell will return a KeyOnlyCell that have no value > or has valueLength as value, current will copy all row keys into a new byte[] > and new a KeyValue, we can eliminate the copy and have a wrap KeyOnlyCell > that ignore the cell's value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6581) Build with hadoop.profile=3.0
[ https://issues.apache.org/jira/browse/HBASE-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526705#comment-15526705 ] Jonathan Hsieh commented on HBASE-6581: --- Linking related patches. > Build with hadoop.profile=3.0 > - > > Key: HBASE-6581 > URL: https://issues.apache.org/jira/browse/HBASE-6581 > Project: HBase > Issue Type: Bug >Reporter: Eric Charles >Assignee: Eric Charles > Fix For: 2.0.0, 1.4.0 > > Attachments: 6581-009.txt, 6581-010.txt, 6581.txt, 6581v2.txt, > HBASE-6581-1.patch, HBASE-6581-2.patch, HBASE-6581-20130821.patch, > HBASE-6581-3.patch, HBASE-6581-4.patch, HBASE-6581-5.patch, > HBASE-6581-6.patch, HBASE-6581-7.patch, HBASE-6581-8-pom.patch, > HBASE-6581.diff, HBASE-6581.diff > > > Building trunk with hadoop.profile=3.0 gives exceptions (see [1]) due to > change in the hadoop maven modules naming (and also usage of 3.0-SNAPSHOT > instead of 3.0.0-SNAPSHOT in hbase-common). > I can provide a patch that would move most of hadoop dependencies in their > respective profiles and will define the correct hadoop deps in the 3.0 > profile. > Please tell me if that's ok to go this way. > Thx, Eric > [1] > $ mvn clean install -Dhadoop.profile=3.0 > [INFO] Scanning for projects... > [ERROR] The build could not read 3 projects -> [Help 1] > [ERROR] > [ERROR] The project org.apache.hbase:hbase-server:0.95-SNAPSHOT > (/d/hbase.svn/hbase-server/pom.xml) has 3 errors > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-common:jar is missing. @ line 655, column 21 > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-annotations:jar is missing. @ line 659, column 21 > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 663, column 21 > [ERROR] > [ERROR] The project org.apache.hbase:hbase-common:0.95-SNAPSHOT > (/d/hbase.svn/hbase-common/pom.xml) has 3 errors > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-common:jar is missing. @ line 170, column 21 > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-annotations:jar is missing. @ line 174, column 21 > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 178, column 21 > [ERROR] > [ERROR] The project org.apache.hbase:hbase-it:0.95-SNAPSHOT > (/d/hbase.svn/hbase-it/pom.xml) has 3 errors > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-common:jar is missing. @ line 220, column 18 > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-annotations:jar is missing. @ line 224, column 21 > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 228, column 21 > [ERROR] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[
https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526617#comment-15526617
]
Hadoop QA commented on HBASE-16664:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 57s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 51s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 57s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 125m 44s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.TestClusterBootOrder |
| | org.apache.hadoop.hbase.client.TestHCM |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.client.TestTableSnapshotScanner |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830504/HBASE-16664-v2.patch |
| JIRA Issue | HBASE-16664 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 29e434d69858 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git
[jira] [Updated] (HBASE-16696) After HBASE-16604 - does not release blocks in case of scanner exception
[ https://issues.apache.org/jira/browse/HBASE-16696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16696: --- Attachment: (was: build-1639.out) > After HBASE-16604 - does not release blocks in case of scanner exception > > > Key: HBASE-16696 > URL: https://issues.apache.org/jira/browse/HBASE-16696 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16696.patch, HBASE-16696_1.patch > > > TestBlockEvictionFromClient consistently fails in master branch. > From existing Jenkins builds, looks like this started with build 1639. > See attached Jenkins console logs. > Pls refer to this comment > https://issues.apache.org/jira/secure/EditComment!default.jspa?id=13004009=15522451 > for the reason for the test failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16705) Eliminate long to Long auto boxing in LongComparator
[
https://issues.apache.org/jira/browse/HBASE-16705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526030#comment-15526030
]
Hudson commented on HBASE-16705:
FAILURE: Integrated in Jenkins build HBase-0.98-matrix #404 (See
[https://builds.apache.org/job/HBase-0.98-matrix/404/])
HBASE-16705 Eliminate long to Long auto boxing in LongComparator. (apurtell:
rev 82187196686f7fda383e96bec91787b507c993a8)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/filter/LongComparator.java
> Eliminate long to Long auto boxing in LongComparator
>
>
> Key: HBASE-16705
> URL: https://issues.apache.org/jira/browse/HBASE-16705
> Project: HBase
> Issue Type: Improvement
> Components: Filters
>Affects Versions: 2.0..
>Reporter: binlijin
>Assignee: binlijin
>Priority: Minor
> Fix For: 2.0.0, 1.4.0, 0.98.23
>
> Attachments: HBASE-16705-master.patch
>
>
> LongComparator
> @Override
> public int compareTo(byte[] value, int offset, int length) {
> Long that = Bytes.toLong(value, offset, length);
> return this.longValue.compareTo(that);
> }
> Every time need to convert long to Long, this is not necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11354) HConnectionImplementation#DelayedClosing does not start
[
https://issues.apache.org/jira/browse/HBASE-11354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526032#comment-15526032
]
Hudson commented on HBASE-11354:
FAILURE: Integrated in Jenkins build HBase-0.98-matrix #404 (See
[https://builds.apache.org/job/HBase-0.98-matrix/404/])
HBASE-11354 HConnectionImplementation#DelayedClosing does not start (apurtell:
rev 2b6da527b31c70c5ec764e09b109fc4d3ab6704f)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
> HConnectionImplementation#DelayedClosing does not start
> ---
>
> Key: HBASE-11354
> URL: https://issues.apache.org/jira/browse/HBASE-11354
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.99.0, 0.98.3
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
>Priority: Minor
> Fix For: 0.98.23
>
> Attachments: HBASE-11354-0.98.patch, HBASE_11354 (1).patch,
> HBASE_11354.patch, HBASE_11354.patch, HBASE_11354.patch
>
>
> The method "createAndStart" in class DelayedClosing only creates a instance,
> but forgets to start it. So thread delayedClosing is not running all the time.
> ConnectionManager#1623
> {code}
> static DelayedClosing createAndStart(HConnectionImplementation hci){
> Stoppable stoppable = new Stoppable() {
> private volatile boolean isStopped = false;
> @Override public void stop(String why) { isStopped = true;}
> @Override public boolean isStopped() {return isStopped;}
> };
> return new DelayedClosing(hci, stoppable);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16694) Reduce garbage for onDiskChecksum in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-16694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526033#comment-15526033 ] Hudson commented on HBASE-16694: FAILURE: Integrated in Jenkins build HBase-0.98-matrix #404 (See [https://builds.apache.org/job/HBase-0.98-matrix/404/]) HBASE-16694 Reduce garbage for onDiskChecksum in HFileBlock (binlijin) (apurtell: rev 6903296a3c181958eda82d0e2cbe24aba986efda) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java > Reduce garbage for onDiskChecksum in HFileBlock > --- > > Key: HBASE-16694 > URL: https://issues.apache.org/jira/browse/HBASE-16694 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.. >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0, 1.4.0, 0.98.23 > > Attachments: HBASE-16694-master.patch > > > Current when finish a HFileBlock will create a new byte[] for onDiskChecksum, > we can reuse it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16717) Reuse DataChecksum
[ https://issues.apache.org/jira/browse/HBASE-16717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin resolved HBASE-16717. -- Resolution: Invalid > Reuse DataChecksum > -- > > Key: HBASE-16717 > URL: https://issues.apache.org/jira/browse/HBASE-16717 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Priority: Minor > Attachments: HBASE-16717-master.patch > > > Every time to generateChecksums or validateChecksum will create a new > DataChecksum, and DataChecksum is not thread safe, so use ThreadLocal to > reuse it and keep thread safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16717) Reuse DataChecksum
[ https://issues.apache.org/jira/browse/HBASE-16717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525971#comment-15525971 ] binlijin commented on HBASE-16717: -- Do with a minor test, it is hard too say which is the best, so close this. > Reuse DataChecksum > -- > > Key: HBASE-16717 > URL: https://issues.apache.org/jira/browse/HBASE-16717 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Priority: Minor > Attachments: HBASE-16717-master.patch > > > Every time to generateChecksums or validateChecksum will create a new > DataChecksum, and DataChecksum is not thread safe, so use ThreadLocal to > reuse it and keep thread safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16691) Optimize KeyOnlyFilter by utilizing KeyOnlyCell
[ https://issues.apache.org/jira/browse/HBASE-16691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525928#comment-15525928 ] binlijin commented on HBASE-16691: -- Ok, done, see HBASE-16691-master_v2.patch. > Optimize KeyOnlyFilter by utilizing KeyOnlyCell > --- > > Key: HBASE-16691 > URL: https://issues.apache.org/jira/browse/HBASE-16691 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0 > > Attachments: HBASE-16691-addendum.patch, HBASE-16691-master.patch, > HBASE-16691-master_v2.patch > > > In KeyOnlyFilter#transformCell will return a KeyOnlyCell that have no value > or has valueLength as value, current will copy all row keys into a new byte[] > and new a KeyValue, we can eliminate the copy and have a wrap KeyOnlyCell > that ignore the cell's value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16691) Optimize KeyOnlyFilter by utilizing KeyOnlyCell
[ https://issues.apache.org/jira/browse/HBASE-16691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16691: - Status: Patch Available (was: Reopened) > Optimize KeyOnlyFilter by utilizing KeyOnlyCell > --- > > Key: HBASE-16691 > URL: https://issues.apache.org/jira/browse/HBASE-16691 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0 > > Attachments: HBASE-16691-addendum.patch, HBASE-16691-master.patch, > HBASE-16691-master_v2.patch > > > In KeyOnlyFilter#transformCell will return a KeyOnlyCell that have no value > or has valueLength as value, current will copy all row keys into a new byte[] > and new a KeyValue, we can eliminate the copy and have a wrap KeyOnlyCell > that ignore the cell's value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16691) Optimize KeyOnlyFilter by utilizing KeyOnlyCell
[ https://issues.apache.org/jira/browse/HBASE-16691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16691: - Attachment: HBASE-16691-master_v2.patch > Optimize KeyOnlyFilter by utilizing KeyOnlyCell > --- > > Key: HBASE-16691 > URL: https://issues.apache.org/jira/browse/HBASE-16691 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0 > > Attachments: HBASE-16691-addendum.patch, HBASE-16691-master.patch, > HBASE-16691-master_v2.patch > > > In KeyOnlyFilter#transformCell will return a KeyOnlyCell that have no value > or has valueLength as value, current will copy all row keys into a new byte[] > and new a KeyValue, we can eliminate the copy and have a wrap KeyOnlyCell > that ignore the cell's value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16653) Backport HBASE-11393 to all branches which support namespace
[ https://issues.apache.org/jira/browse/HBASE-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525691#comment-15525691 ] Guanghao Zhang commented on HBASE-16653: [~enis] Please help to review this. Thanks. This bug has been existed since we support Namespace. The fix is necessary for branch-1. But I don't know if it can be fixed in branch-1.3, branch-1.2, branch-1.1 and 0.98.. > Backport HBASE-11393 to all branches which support namespace > > > Key: HBASE-16653 > URL: https://issues.apache.org/jira/browse/HBASE-16653 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0, 1.0.5, 1.3.1, 0.98.22, 1.1.7, 1.2.4 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16653-branch-1-v1.patch, > HBASE-16653-branch-1-v2.patch > > > As HBASE-11386 mentioned, the parse code about replication table-cfs config > will be wrong when table name contains namespace and we can only config the > default namespace's tables in the peer. It is a bug for all branches which > support namespace. HBASE-11393 resolved this by use a pb object but it was > only merged to master branch. Other branches still have this problem. I > thought we should fix this bug in all branches which support namespace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16463) Improve transparent table/CF encryption with Commons Crypto
[
https://issues.apache.org/jira/browse/HBASE-16463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525679#comment-15525679
]
Hadoop QA commented on HBASE-16463:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 6s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
29s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 31s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 36s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
46s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 54s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 49s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 46s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 41s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.filter.TestFilter |
| | hadoop.hbase.filter.TestFilter |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830448/HBASE-16463.004.patch
|
| JIRA Issue | HBASE-16463 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile xml |
| uname | Linux bb69fe8b27da
[jira] [Updated] (HBASE-16696) After HBASE-16604 - does not release blocks in case of scanner exception
[ https://issues.apache.org/jira/browse/HBASE-16696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16696: --- Attachment: (was: build-1638.out) > After HBASE-16604 - does not release blocks in case of scanner exception > > > Key: HBASE-16696 > URL: https://issues.apache.org/jira/browse/HBASE-16696 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16696.patch, HBASE-16696_1.patch, build-1639.out > > > TestBlockEvictionFromClient consistently fails in master branch. > From existing Jenkins builds, looks like this started with build 1639. > See attached Jenkins console logs. > Pls refer to this comment > https://issues.apache.org/jira/secure/EditComment!default.jspa?id=13004009=15522451 > for the reason for the test failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525663#comment-15525663 ] Anoop Sam John commented on HBASE-16608: Left some comments in RB. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
