[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509217#comment-15509217 ] Phil Yang commented on HBASE-16664: --- Thanks [~chenheng] I think I find a bug. We use RpcRetryingCallerImpl.callWithoutRetries in AP and the retrying logic is in higher level. But in callWithoutRetries we only use callTimeout as the timeout of each RPC request. And callTimeout is AsyncRequestFutureImpl's currentCallTotalTimeout which is operation timeout. And in CancellableRegionServerCallable, we get the remaining time according to the start timestamp and the operation timeout. We don't considier rpc timeout any where... And in AP, we have a numTries but we never use it, I'm checking if we have retrying logic. > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang > Attachments: testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16463) Improve transparent table/CF encryption with Commons Crypto
[
https://issues.apache.org/jira/browse/HBASE-16463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509273#comment-15509273
]
ramkrishna.s.vasudevan commented on HBASE-16463:
I think this change looks harmless in terms of impl as it is implementing
existing interfaces. No new protocol needed like the other JIRA where we deal
with RPCClient and server.
Just wanted to know
{code}
1.0.0
{code}
This version is the updated stable release version available?
Things to be confirmed is that,
What is the procedure if the cluster has to be upgraded from AES to the new
commons cryto? Major compaction should be run before using the new algo?
{code}
public static final String RNG_ALGORITHM_KEY = "hbase.crypto.algorithm.rng";
60public static final String RNG_PROVIDER_KEY =
"hbase.crypto.algorithm.rng.provider";
{code}
these config keys can be moved to the Cipher abstract class if the existing AES
cipher also uses the same key. Same with IV_LENGTH, BLOCK_SIZE etc.
[~apurtell], [~apurt...@yahoo.com], [~ghelmling]
Want to have a look at this patch?
> Improve transparent table/CF encryption with Commons Crypto
> ---
>
> Key: HBASE-16463
> URL: https://issues.apache.org/jira/browse/HBASE-16463
> Project: HBase
> Issue Type: New Feature
> Components: encryption
>Affects Versions: 2.0.0
>Reporter: Dapeng Sun
> Attachments: HBASE-16463.001.patch, HBASE-16463.002.patch,
> HBASE-16463.003.patch
>
>
> Apache Commons Crypto
> (https://commons.apache.org/proper/commons-crypto/index.html) is a
> cryptographic library optimized with AES-NI.
> HBASE-7544 introduces a framework for transparent encryption feature for
> protecting HFile and WAL data at rest. Currently JCE cipher is used bu
> default, the improvement will use Commons Crypto to accelerate the
> transparent encryption of HBase. new crypto provider with Commons CRYPTO will
> be provided for Transparent encryption.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509656#comment-15509656
]
Hadoop QA commented on HBASE-16643:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 13s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 44s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 114m 30s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.regionserver.TestWalAndCompactingMemStoreFlush |
| | hadoop.hbase.regionserver.TestCompactingMemStore |
| | hadoop.hbase.regionserver.TestCompactingToCellArrayMapMemStore |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestFromClientSide3 |
| | org.apache.hadoop.hbase.client.TestEnableTable |
| | org.apache.hadoop.hbase.client.TestHCM |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829536/HBASE-16643_1.patch |
| JIRA Issue | HBASE-16643 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux ea4e785ecf54 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c67983e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/3627/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3627/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509299#comment-15509299
]
Xiang Li commented on HBASE-14882:
--
Hi [~anoop.hbase], I uploaded patch 002 for master. Please review.
1. There is a warning reported by findbugs, but it has nothing to do with the
patch 8-)
2. Regarding your comment
bq. Can u see all usage of these affected Put APIs.. (addImmutable)
Sorry that I did not get your question. Could you please elaborate more?
> Provide a Put API that adds the provided family, qualifier, value without
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-14882.master.000.patch,
> HBASE-14882.master.001.patch, HBASE-14882.master.002.patch
>
>
> In the Put API, we have addImmutable()
> {code}
> /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
> public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation, the family, qualifier and value are still being
> copied locally to create kv.
> Hopefully we should provide an API that truly uses immutable family,
> qualifier and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Attachment: HBASE-15968-v2.patch
Upload new patch, change the name to CombinedTracker. The CF's conf is not
changed and we can do this later. Support visibility labels in delete by extend
the CombinedTracker. And fix some bugs.
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch, HBASE-15968-v2.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16654) Better handle channelInactive and close for netty rpc client
[ https://issues.apache.org/jira/browse/HBASE-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509508#comment-15509508 ] Duo Zhang commented on HBASE-16654: --- Ping [~stack]. > Better handle channelInactive and close for netty rpc client > > > Key: HBASE-16654 > URL: https://issues.apache.org/jira/browse/HBASE-16654 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16654.patch > > > We should pass the event to the next handler in the pipeline as > channelInactive and close are usually used as the cleanup method of a > ChannelHandler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509525#comment-15509525 ] Duo Zhang commented on HBASE-15921: --- https://github.com/Apache9/hbase/blob/asynclocator/hbase-client/src/main/java/org/apache/hadoop/hbase/client/async/ZKAsyncRegistry.java The async zookeeper registry, based on curator. As the current locateRegionInMeta uses reversed scan, I think we need to find a slow but simple way to implement it first. Otherwise we need to implement asynchronous scan even if we only want to implement a AsyncTable with only get operation support... Thanks. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.demo.patch, HBASE-15921.patch, > HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Attachment: HBASE-15968-v3.patch
Add MediumTests
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch, HBASE-15968-v2.patch,
> HBASE-15968-v3.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509680#comment-15509680 ] Duo Zhang commented on HBASE-15921: --- I was misled by the current zh implementation in HBase... The common way of using zookeeper is cache the data at client side and use watcher to update the cached data. So we do not need to implement a client that read data from zookeeper asynchronously. Instead, just use the NodeCache or TreeCache or whatever cache in curator to keep the data in memory, and get the data directly, no Future needed. The cache will updated in background. This will greatly simplify the logic. Thanks. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.demo.patch, HBASE-15921.patch, > HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16643:
---
Summary: Reverse scanner heap creation may not allow MSLAB closure due to
improper ref counting of segments (was: Reverse scanner heap creation may not
allow MSLAB closure due to inproper ref counting of segments)
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16643:
---
Status: Open (was: Patch Available)
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Attachment: (was: HBASE-15968-v2.patch)
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Attachment: HBASE-15968-v2.patch
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch, HBASE-15968-v2.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509574#comment-15509574
]
Phil Yang commented on HBASE-15968:
---
RB: https://reviews.apache.org/r/52116/
thanks
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch, HBASE-15968-v2.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16665) Check whether KeyValueUtil.createXXX could be replaced by CellUtil without copy
[
https://issues.apache.org/jira/browse/HBASE-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509572#comment-15509572
]
Anoop Sam John commented on HBASE-16665:
AbstractMemStore#updateColumnValue change seems not needed at all as we are not
using this in code path. This API used by tests alone.. We should get rid of
this. The other place in AbstractMemStore though we must change. (The one u
mentioned). This can be changed also.. We might need to add new API in
CellUtil. Or else use CellUtil#createFirstOnRowColTS and pass the
LATEST_TIMESTAMP.
{code}
// Used only for testing purposes
static MatchCode checkColumn(ColumnTracker columnTracker, byte[] bytes, int
offset, int length,
long ttl, byte type, boolean ignoreCount) throws IOException {
KeyValue kv = KeyValueUtil.createFirstOnRow(HConstants.EMPTY_BYTE_ARRAY, 0,
0,
HConstants.EMPTY_BYTE_ARRAY, 0, 0, bytes, offset, length);
{code}
Used only by tests. We can just ignore and no need to change IMO.
> Check whether KeyValueUtil.createXXX could be replaced by CellUtil without
> copy
> ---
>
> Key: HBASE-16665
> URL: https://issues.apache.org/jira/browse/HBASE-16665
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-16665.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509308#comment-15509308
]
Hadoop QA commented on HBASE-16643:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 52s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 25s
{color} | {color:red} hbase-server generated 1 new + 1 unchanged - 0 fixed = 2
total (was 1) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 45s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 114m 40s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.regionserver.TestWalAndCompactingMemStoreFlush |
| | hadoop.hbase.regionserver.TestCompactingMemStore |
| | hadoop.hbase.regionserver.TestCompactingToCellArrayMapMemStore |
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
|
| | org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache |
| | org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829517/HBASE-16643.patch |
| JIRA Issue | HBASE-16643 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 6e02b8519a35 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 6624c67 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/3626/artifact/patchprocess/whitespace-eol.txt
|
| javadoc |
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509411#comment-15509411 ] ramkrishna.s.vasudevan commented on HBASE-16417: I think you need to configure your GC settings. Try with G1GC. That should be very important. Let us know what you observer. > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16666) Add append and remove peer namespaces cmds for replication
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-1: --- Attachment: HBASE-1.patch > Add append and remove peer namespaces cmds for replication > -- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Attachments: HBASE-1.patch > > > After HBASE-16447, we support replication by namespaces config in peer. Like > append_peer_tableCFs and remove_peer_tableCFs, I thought we need two new > shell cmd: append_peer_namespaces and remove_peer_namespaces. Then we can > easily change the namespaces config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509620#comment-15509620
]
Hadoop QA commented on HBASE-15968:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 48s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 9s
{color} | {color:red} hbase-server generated 3 new + 0 unchanged - 0 fixed = 3
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 43s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 19s
{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 28s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Switch statement found in
org.apache.hadoop.hbase.regionserver.querymatcher.CombinedTracker.add(Cell)
where default case is missing At CombinedTracker.java:where default case is
missing At CombinedTracker.java:[lines 184-200] |
| | Switch statement found in
org.apache.hadoop.hbase.security.visibility.VisibilityCombinedTracker.add(Cell)
where default case is missing At VisibilityCombinedTracker.java:where default
case is missing At VisibilityCombinedTracker.java:[lines 121-139] |
| | Should
org.apache.hadoop.hbase.security.visibility.VisibilityCombinedTracker$TagInfo
be a _static_ inner class? At VisibilityCombinedTracker.java:inner class? At
VisibilityCombinedTracker.java:[lines 54-65] |
| Failed junit tests | hadoop.hbase.TestCheckTestClasses |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829539/HBASE-15968-v2.patch |
| JIRA Issue | HBASE-15968 |
| Optional Tests |
[jira] [Commented] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to inproper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509182#comment-15509182
]
Eshcar Hillel commented on HBASE-16643:
---
Hi [~ram_krish]
Could you please add a link to review board so it is easier to review the
patch. Thanks.
> Reverse scanner heap creation may not allow MSLAB closure due to inproper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14734) TestGenerateDelegationToken fails with BindAddress clash
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509201#comment-15509201 ] Appy commented on HBASE-14734: -- I was thinking of a solution on lines of reserving some ports and then passing them to minikdc so it can use it. But seems like that ain't the way. http://unix.stackexchange.com/questions/15511/how-do-i-reserve-ports-for-my-application. Seems like the way stack suggested above might be the best i.e. check for exception and retry. > TestGenerateDelegationToken fails with BindAddress clash > > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack > > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[
https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509227#comment-15509227
]
Heng Chen commented on HBASE-16664:
---
{quote}
We don't considier rpc timeout any where...
{quote}
Yes, that is what i said above "I think we should respect the rules, pass
operationTimeout and do control for each RPC call with "rpcTimeout" and
remaining of "operationTimeout""
> Timeout logic in AsyncProcess is broken
> ---
>
> Key: HBASE-16664
> URL: https://issues.apache.org/jira/browse/HBASE-16664
> Project: HBase
> Issue Type: Bug
>Reporter: Phil Yang
> Attachments: testhcm.patch
>
>
> Have not checked the root cause, but I think timeout of all operations in
> AsyncProcess is broken
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16179: --- Attachment: 16179.v4.txt With patch v4, tests in hbase-spark pass against Spark 2.0 release. Still need to figure out how to support Spark 1.6 at the same time. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16666) Add append and remove peer namespaces cmds for replication
Guanghao Zhang created HBASE-1: -- Summary: Add append and remove peer namespaces cmds for replication Key: HBASE-1 URL: https://issues.apache.org/jira/browse/HBASE-1 Project: HBase Issue Type: Improvement Components: Replication Reporter: Guanghao Zhang Assignee: Guanghao Zhang Priority: Minor After HBASE-16447, we support replication by namespaces config in peer. Like append_peer_tableCFs and remove_peer_tableCFs, I thought we need two new shell cmd: append_peer_namespaces and remove_peer_namespaces. Then we can easily change the namespaces config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509582#comment-15509582 ] Anoop Sam John commented on HBASE-16417: What is the total memory available in ur RS node? What all other active process? I believe u have HM and RS and client app in same node. HM not needing 32 GB any way and client too > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16636) Regions in Transition counts wrong (zero) in HMaster /jmx, prevents detecting Regions Stuck in Transition
[
https://issues.apache.org/jira/browse/HBASE-16636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509629#comment-15509629
]
Hari Sekhon edited comment on HBASE-16636 at 9/21/16 11:30 AM:
---
I've also noticed that ritCount and ritOldestAge was zero as well. Are all 3 of
these metrics fixed in the other jira?
was (Author: harisekhon):
I've also noticed that ritOldestAge was zero as well. Are all 3 of these
metrics fixed in the other jira?
> Regions in Transition counts wrong (zero) in HMaster /jmx, prevents detecting
> Regions Stuck in Transition
> -
>
> Key: HBASE-16636
> URL: https://issues.apache.org/jira/browse/HBASE-16636
> Project: HBase
> Issue Type: Bug
> Components: UI
>Affects Versions: 1.1.2
> Environment: HDP 2.3.2
>Reporter: Hari Sekhon
>Priority: Minor
> Attachments: Regions_in_Transition_UI.png, ritCountOverThreshold.png
>
>
> I've discovered that the Region in Transition counts are wrong in the HMaster
> UI /jmx page.
> The /master-status page clearly shows 3 regions stuck in transition but the
> /jmx page I was monitoring reported 0 for ritCountOverThreshold.
> {code}
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManger",
> "modelerType" : "Master,sub=AssignmentManger",
> "tag.Context" : "master",
> ...
> "ritOldestAge" : 0,
> "ritCountOverThreshold" : 0,
> ...
> "ritCount" : 0,
> {code}
> I have a nagios plugin I wrote which was checking this which I've since had
> to rewrite to parse the /master-status page instead (the code is in
> check_hbase_regions_stuck_in_transition.py at
> https://github.com/harisekhon/nagios-plugins).
> I'm attaching screenshots of both /master-status and /jmx to show the
> difference in the 2 pages on the HMaster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[
https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509295#comment-15509295
]
Hadoop QA commented on HBASE-16264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 160 new or modified
test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 28m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
29s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 37s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 59s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 6s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 26s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 26s {color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 26s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 30m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
41s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2296 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 8s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 19s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s {color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s {color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s {color} | {color:green} hbase-common generated 0 new + 0 unchanged - 1
fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s {color} | {color:green} hbase-examples in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s {color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s {color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s {color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s {color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} hbase-server in the patch passed. {color} |
|
[jira] [Updated] (HBASE-16645) Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap
[ https://issues.apache.org/jira/browse/HBASE-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16645: --- Priority: Major (was: Minor) > Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap > -- > > Key: HBASE-16645 > URL: https://issues.apache.org/jira/browse/HBASE-16645 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai > Fix For: 2.0.0 > > Attachments: HBASE-16645.v0.patch, HBASE-16645.v1.patch > > > Two reasons are shown below: > 1) CellFlatMap#find doesn’t consider desc order array > 2) CellFlatMap#getValidIndex return the wrong upper bound -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16643:
---
Status: Patch Available (was: Open)
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16636) Regions in Transition counts wrong (zero) in HMaster /jmx, prevents detecting Regions Stuck in Transition
[
https://issues.apache.org/jira/browse/HBASE-16636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509629#comment-15509629
]
Hari Sekhon commented on HBASE-16636:
-
I've also noticed that ritOldestAge was zero as well. Are all 3 of these
metrics fixed in the other jira?
> Regions in Transition counts wrong (zero) in HMaster /jmx, prevents detecting
> Regions Stuck in Transition
> -
>
> Key: HBASE-16636
> URL: https://issues.apache.org/jira/browse/HBASE-16636
> Project: HBase
> Issue Type: Bug
> Components: UI
>Affects Versions: 1.1.2
> Environment: HDP 2.3.2
>Reporter: Hari Sekhon
>Priority: Minor
> Attachments: Regions_in_Transition_UI.png, ritCountOverThreshold.png
>
>
> I've discovered that the Region in Transition counts are wrong in the HMaster
> UI /jmx page.
> The /master-status page clearly shows 3 regions stuck in transition but the
> /jmx page I was monitoring reported 0 for ritCountOverThreshold.
> {code}
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManger",
> "modelerType" : "Master,sub=AssignmentManger",
> "tag.Context" : "master",
> ...
> "ritOldestAge" : 0,
> "ritCountOverThreshold" : 0,
> ...
> "ritCount" : 0,
> {code}
> I have a nagios plugin I wrote which was checking this which I've since had
> to rewrite to parse the /master-status page instead (the code is in
> check_hbase_regions_stuck_in_transition.py at
> https://github.com/harisekhon/nagios-plugins).
> I'm attaching screenshots of both /master-status and /jmx to show the
> difference in the 2 pages on the HMaster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509312#comment-15509312 ] Sudev Ambadi commented on HBASE-16179: -- Is there a way to compile HBase with Spark 2.0 without hitting this issue ? > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509363#comment-15509363 ] Anoop Sam John commented on HBASE-16417: Machine totally having ~150GB RAM RS process allocated with 32 GB of xmx. 50 regions. 42% as the global memstore max size. G1GC we used with initial heap occupancy factor as 50%. Waste percent configured as 10% Also the flusher thread count made to 10 and Blocking store files count 25 (Configs) And we have HDD only in the node. > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16643:
---
Attachment: HBASE-16643_1.patch
Updated patch. The failed test case seems to pass. Will check once again with
QA.
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509522#comment-15509522 ] Anoop Sam John commented on HBASE-16417: Are you seeing that a long pause GC happened and so RS got killed by master decision? You said RS got OOME.. That is strange. Because consider 32 GB heap and 50 regions and each region having 128 MB flush size. By default we have 4 as the blocking memstore size. Means this 128MB can become 512 MB. After that the memstore wont receive writes. 50* 128 * 4 = 25 GB. So am not getting how u got OOME. U might get long GC pause. > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509533#comment-15509533
]
Anoop Sam John commented on HBASE-14882:
I mean within HBase code base, where all put#addImmutable is being used? Some
where in the WAL read code flow it is getting used for sure. That is what the
failure u got when u have not made the new Cell to extend SettableSeqId
> Provide a Put API that adds the provided family, qualifier, value without
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-14882.master.000.patch,
> HBASE-14882.master.001.patch, HBASE-14882.master.002.patch
>
>
> In the Put API, we have addImmutable()
> {code}
> /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
> public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation, the family, qualifier and value are still being
> copied locally to create kv.
> Hopefully we should provide an API that truly uses immutable family,
> qualifier and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[
https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509576#comment-15509576
]
Eshcar Hillel commented on HBASE-16417:
---
The error message is by the GC
{\code}
Stack: [0x7f56a85d1000,0x7f56a86d2000], sp=0x7f56a86cfe10, free
space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xab97ea] VMError::report_and_die()+0x2ba
V [libjvm.so+0x4f9dab] report_vm_out_of_memory(char const*, int, unsigned
long, VMErrorType, char const*)+0x8b
V [libjvm.so+0x91a7c3] os::Linux::commit_memory_impl(char*, unsigned long,
bool)+0x103
V [libjvm.so+0x91ac65] os::pd_commit_memory_or_exit(char*, unsigned long,
unsigned long, bool, char const*)+0x35
V [libjvm.so+0x914d46] os::commit_memory_or_exit(char*, unsigned long,
unsigned long, bool, char const*)+0x26
V [libjvm.so+0x5c073f] G1PageBasedVirtualSpace::commit_internal(unsigned
long, unsigned long)+0xbf
V [libjvm.so+0x5c09cc] G1PageBasedVirtualSpace::commit(unsigned long,
unsigned long)+0x11c
V [libjvm.so+0x5c3610]
G1RegionsLargerThanCommitSizeMapper::commit_regions(unsigned int, unsigned
long)+0x40
V [libjvm.so+0x625157] HeapRegionManager::commit_regions(unsigned int,
unsigned long)+0x77
V [libjvm.so+0x6263f1] HeapRegionManager::make_regions_available(unsigned
int, unsigned int)+0x31
V [libjvm.so+0x626970] HeapRegionManager::expand_by(unsigned int)+0xb0
V [libjvm.so+0x597c29] G1CollectedHeap::expand(unsigned long)+0x199
V [libjvm.so+0x5a3b0d]
G1CollectedHeap::do_collection_pause_at_safepoint(double)+0xc6d
V [libjvm.so+0xac3c2a] VM_G1IncCollectionPause::doit()+0x9a
V [libjvm.so+0xac2c35] VM_Operation::evaluate()+0x55
V [libjvm.so+0xac100a] VMThread::evaluate_operation(VM_Operation*)+0xba
V [libjvm.so+0xac138e] VMThread::loop()+0x1ce
V [libjvm.so+0xac1800] VMThread::run()+0x70
V [libjvm.so+0x91cb88] java_start(Thread*)+0x108
{\code}
so the RS crashes, don't think it is killed by master.
Not writing to WAL.
> In-Memory MemStore Policy for Flattening and Compactions
>
>
> Key: HBASE-16417
> URL: https://issues.apache.org/jira/browse/HBASE-16417
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Fix For: 2.0.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16665) Check whether KeyValueUtil.createXXX could be replaced by CellUtil without copy
[ https://issues.apache.org/jira/browse/HBASE-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-16665: -- Attachment: HBASE-16665.patch The first patch deal with createFirstOnRow > Check whether KeyValueUtil.createXXX could be replaced by CellUtil without > copy > --- > > Key: HBASE-16665 > URL: https://issues.apache.org/jira/browse/HBASE-16665 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen > Attachments: HBASE-16665.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16665) Check whether KeyValueUtil.createXXX could be replaced by CellUtil without copy
[
https://issues.apache.org/jira/browse/HBASE-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509251#comment-15509251
]
Heng Chen commented on HBASE-16665:
---
Not sure for the two place whether KeyValueUtil.createFirstOnRow could be
replaced.
{code: title=AbstractMemStore.java}
Cell firstCell = KeyValueUtil.createFirstOnRow(
cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
cell.getFamilyArray(), cell.getFamilyOffset(), cell.getFamilyLength(),
cell.getQualifierArray(), cell.getQualifierOffset(),
cell.getQualifierLength());
SortedSet ss = active.tailSet(firstCell);
{code}
{code: title=ScanQueryMatcher.java}
KeyValue kv = KeyValueUtil.createFirstOnRow(HConstants.EMPTY_BYTE_ARRAY, 0,
0,
HConstants.EMPTY_BYTE_ARRAY, 0, 0, bytes, offset, length);
MatchCode matchCode = columnTracker.checkColumn(kv, type);
if (matchCode == MatchCode.INCLUDE) {
return columnTracker.checkVersions(kv, ttl, type, ignoreCount);
}
return matchCode;
{code}
> Check whether KeyValueUtil.createXXX could be replaced by CellUtil without
> copy
> ---
>
> Key: HBASE-16665
> URL: https://issues.apache.org/jira/browse/HBASE-16665
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-16665.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509291#comment-15509291 ] Eshcar Hillel commented on HBASE-16417: --- We've started working on the policy patch. We try to set a baseline for performance by running PE over the default memstore with the configuration settings [~ram_krish] suggested in HBASE-14921. *However* we are only able to complete the run with 2-4 threads, 10 regions (pre split) writing 10GB. Any attempt to either increase the number of threads or the amount of data results in an Out of Memory Error caused by the *GC*. (I'm running with the same settings for PE, hbase and GC as you posted.) We are running on a SSD 2.9PB disk (HDD wasn't even able to complete a run even with a single thread), with 48GB RAM. Some questions [~ram_krish]: 1. Did you run PE on an SSD or HDD? 2. How much memory do you have in your machine? 3. Can you think of any parameter that you forgot to mention that may have caused the difference in executions? I'll also try running PE with more space for the GC and will update on the result. > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16644) Errors when reading legit HFile' Trailer on branch 1.3
[
https://issues.apache.org/jira/browse/HBASE-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-16644:
Status: Patch Available (was: Open)
> Errors when reading legit HFile' Trailer on branch 1.3
> --
>
> Key: HBASE-16644
> URL: https://issues.apache.org/jira/browse/HBASE-16644
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Critical
> Fix For: 1.3.0
>
> Attachments: HBASE-16644.branch-1.3.patch
>
>
> There seems to be a regression in branch 1.3 where we can't read HFile
> trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on
> some HFiles that could be successfully read on 1.2.
> I've seen this error manifesting in two ways so far.
> {code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException:
> Problem reading HFile Trailer from file
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
> ... 6 more
> Caused by: java.io.IOException: Invalid HFile block magic:
> \x00\x00\x04\x00\x00\x00\x00\x00
> at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:344)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485)
> {code}
> and second
> {code}
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
> reading HFile Trailer from file
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
> at
> org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:104)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
> ... 6 more
> Caused by: java.io.IOException: Premature EOF from inputStream (read returned
> -1, was trying to read 10083 necessary bytes and 24 extra bytes, successfully
> read 1072
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:737)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1459)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1712)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
> at
>
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[
https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509464#comment-15509464
]
Eshcar Hillel commented on HBASE-16417:
---
I used the same GC settings as you mentioned
{\code}
export HBASE_OPTS="-XX:+UseG1GC -XX:MaxGCPauseMillis=200
-XX:InitiatingHeapOccupancyPercent=60 -XX:G1HeapWastePercent=20
-XX:G1MixedGCCountTarget=8"
{\code}
The last 2 parameters you mentioned seems critical.
Did you mean these two where changed?
{\code}
hbase.hstore.flusher.count
2
The number of flush threads. With fewer threads, the MemStore
flushes will be
queued. With more threads, the flushes will be executed in parallel,
increasing the load on
HDFS, and potentially causing more compactions.
hbase.hstore.blockingStoreFiles
10
If more than this number of StoreFiles exist in any one Store
(one StoreFile
is written per flush of MemStore), updates are blocked for this region
until a compaction is
completed, or until hbase.hstore.blockingWaitTime has been
exceeded.
{\code}
> In-Memory MemStore Policy for Flattening and Compactions
>
>
> Key: HBASE-16417
> URL: https://issues.apache.org/jira/browse/HBASE-16417
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Fix For: 2.0.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509515#comment-15509515 ] Anoop Sam John commented on HBASE-16417: Yep.. Changed to 10 and 25 respectively. > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509558#comment-15509558 ] ramkrishna.s.vasudevan commented on HBASE-16417: Sorry just seeing this ping. G1GC things looks fine. bq.Are you seeing that a long pause GC happened and so RS got killed by master decision? I think this is what will be happening. Also ensure you don't write to WAL. Hope you have that setting in your PE command. > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-12088) Remove un-used profiles in non-root poms
[ https://issues.apache.org/jira/browse/HBASE-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh reassigned HBASE-12088: -- Assignee: Jonathan Hsieh > Remove un-used profiles in non-root poms > > > Key: HBASE-12088 > URL: https://issues.apache.org/jira/browse/HBASE-12088 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Jonathan Hsieh > > Some of the poms still have hadoop 1 and hadoop 1.1 profiles even though the > root pom has them removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12088) Remove un-used profiles in non-root poms
[ https://issues.apache.org/jira/browse/HBASE-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509975#comment-15509975 ] Jonathan Hsieh commented on HBASE-12088: According to [1] we should apply this to all the live 1.x hbase branches [1] https://hbase.apache.org/book.html#hadoop > Remove un-used profiles in non-root poms > > > Key: HBASE-12088 > URL: https://issues.apache.org/jira/browse/HBASE-12088 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Jonathan Hsieh > Attachments: hbase-12088.v0.patch > > > Some of the poms still have hadoop 1 and hadoop 1.1 profiles even though the > root pom has them removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510045#comment-15510045
]
Hadoop QA commented on HBASE-15968:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 30s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 12s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 5s
{color} | {color:red} hbase-server generated 3 new + 0 unchanged - 0 fixed = 3
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 11s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 32s
{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 143m 8s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Switch statement found in
org.apache.hadoop.hbase.regionserver.querymatcher.CombinedTracker.add(Cell)
where default case is missing At CombinedTracker.java:where default case is
missing At CombinedTracker.java:[lines 184-200] |
| | Switch statement found in
org.apache.hadoop.hbase.security.visibility.VisibilityCombinedTracker.add(Cell)
where default case is missing At VisibilityCombinedTracker.java:where default
case is missing At VisibilityCombinedTracker.java:[lines 121-139] |
| | Should
org.apache.hadoop.hbase.security.visibility.VisibilityCombinedTracker$TagInfo
be a _static_ inner class? At VisibilityCombinedTracker.java:inner class? At
VisibilityCombinedTracker.java:[lines 54-65] |
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
| | hadoop.hbase.client.TestFromClientSide |
| | hadoop.hbase.client.TestFromClientSideWithCoprocessor |
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| |
[jira] [Created] (HBASE-16667) Building with JDK 8: ignoring option MaxPermSize=256m
Niels Basjes created HBASE-16667: Summary: Building with JDK 8: ignoring option MaxPermSize=256m Key: HBASE-16667 URL: https://issues.apache.org/jira/browse/HBASE-16667 Project: HBase Issue Type: Improvement Reporter: Niels Basjes Assignee: Niels Basjes In JDK 8 the permgen was removed. As a consequence the build shows this line a lot of times, cluttering the output. OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16667) Building with JDK 8: ignoring option MaxPermSize=256m
[ https://issues.apache.org/jira/browse/HBASE-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510096#comment-15510096 ] Niels Basjes commented on HBASE-16667: -- This patch increases the MaxPermSize to 512m for all places it is used because that was the highest value out there. I assume this is correct. > Building with JDK 8: ignoring option MaxPermSize=256m > - > > Key: HBASE-16667 > URL: https://issues.apache.org/jira/browse/HBASE-16667 > Project: HBase > Issue Type: Improvement >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-16667-01.patch > > > In JDK 8 the permgen was removed. > As a consequence the build shows this line a lot of times, cluttering the > output. > OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support > was removed in 8.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16654) Better handle channelInactive and close for netty rpc client
[ https://issues.apache.org/jira/browse/HBASE-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509699#comment-15509699 ] stack commented on HBASE-16654: --- +1 Cleanup, class renames, and the continuation of the call makes sense. Thanks. > Better handle channelInactive and close for netty rpc client > > > Key: HBASE-16654 > URL: https://issues.apache.org/jira/browse/HBASE-16654 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16654.patch > > > We should pass the event to the next handler in the pipeline as > channelInactive and close are usually used as the cleanup method of a > ChannelHandler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12088) Remove un-used profiles in non-root poms
[ https://issues.apache.org/jira/browse/HBASE-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-12088: --- Status: Patch Available (was: Open) > Remove un-used profiles in non-root poms > > > Key: HBASE-12088 > URL: https://issues.apache.org/jira/browse/HBASE-12088 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Jonathan Hsieh > Attachments: hbase-12088.v0.patch > > > Some of the poms still have hadoop 1 and hadoop 1.1 profiles even though the > root pom has them removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16659) Use CellUtil.createFirstOnRow instead of KeyValueUtil.createFirstOnRow in some places.
[ https://issues.apache.org/jira/browse/HBASE-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509710#comment-15509710 ] Hudson commented on HBASE-16659: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1646 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1646/]) HBASE-16659 Use CellUtil.createFirstOnRow instead of (chenheng: rev c67983ebf88d449a67bccd8b213237362a4093f6) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FuzzyRowFilter.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/filter/MultiRowRangeFilter.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedRegionScannerImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java > Use CellUtil.createFirstOnRow instead of KeyValueUtil.createFirstOnRow in > some places. > --- > > Key: HBASE-16659 > URL: https://issues.apache.org/jira/browse/HBASE-16659 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16659-master.patch, HBASE-16659-master_v2.patch, > HBASE-16659-master_v2.patch > > > CellUtil.createFirstOnRow is more effective than > KeyValueUtil.createFirstOnRow, because there is no byte[] copy. > Current in ReversedRegionScannerImpl#nextRow, to get the previous row to > seek, it use KeyValueUtil.createFirstOnRow which will have two copy, the > first copy is copy to a tmp byte[], the second copy is in KeyValue. We can > use CellUtil.createFirstOnRow which will have no copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509697#comment-15509697
]
ramkrishna.s.vasudevan commented on HBASE-16643:
Tests are failing due to assertion error I think. Let me check and upload a
patch correcting it.
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16643:
---
Status: Patch Available (was: Open)
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch,
> HBASE-16643_2.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16643:
---
Attachment: HBASE-16643_2.patch
REmoves the assertion change. It was my bad. Locally no problem as assertion
was not enabled so could not spot it.
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch,
> HBASE-16643_2.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15925) compat-module maven variable not evaluated
[
https://issues.apache.org/jira/browse/HBASE-15925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509980#comment-15509980
]
David Portabella commented on HBASE-15925:
--
I see that two new versions are published in maven central (1.2.2 and 1.2.3),
that's great!
one question:
this version 1.2.1 (and all previous versions) had this direct dependency's
artifactId on ${compat.module}, and we could see this here:
http://mvnrepository.com/artifact/org.apache.hbase/hbase-testing-util/1.2.1
however, I don't see this anymore. why?
Does this mean that you have fixed and replaced the 1.2.1 version (and all the
previous ones)?
If so, isn't that incorrect? a release version should stay immutable forever.
Compiling a code which depends on a release dependency should always produce
the same result.
Or maybe mvnrepository.com now resolves these variables, and it was not the
case when this ticket was open?
> compat-module maven variable not evaluated
> --
>
> Key: HBASE-15925
> URL: https://issues.apache.org/jira/browse/HBASE-15925
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.0.3, 1.1.5
>Reporter: Nick Dimiduk
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.2, 1.1.6
>
> Attachments: HBASE-15925.1.patch
>
>
> Looks like we've regressed on HBASE-8488. Have a look at the dependency
> artifacts list on
> http://mvnrepository.com/artifact/org.apache.hbase/hbase-testing-util/1.2.1.
> Notice the direct dependency's artifactId is {{$\{compat.module\}}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16667) Building with JDK 8: ignoring option MaxPermSize=256m
[
https://issues.apache.org/jira/browse/HBASE-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niels Basjes updated HBASE-16667:
-
Description:
In JDK 8 the permgen was removed.
As a consequence the build shows this line a lot of times, cluttering the
output.
{quote}
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was
removed in 8.0
{quote}
was:
In JDK 8 the permgen was removed.
As a consequence the build shows this line a lot of times, cluttering the
output.
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was
removed in 8.0
> Building with JDK 8: ignoring option MaxPermSize=256m
> -
>
> Key: HBASE-16667
> URL: https://issues.apache.org/jira/browse/HBASE-16667
> Project: HBase
> Issue Type: Improvement
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-16667-01.patch
>
>
> In JDK 8 the permgen was removed.
> As a consequence the build shows this line a lot of times, cluttering the
> output.
> {quote}
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support
> was removed in 8.0
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12088) Remove un-used profiles in non-root poms
[ https://issues.apache.org/jira/browse/HBASE-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-12088: --- Attachment: hbase-12088.v0.patch > Remove un-used profiles in non-root poms > > > Key: HBASE-12088 > URL: https://issues.apache.org/jira/browse/HBASE-12088 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Jonathan Hsieh > Attachments: hbase-12088.v0.patch > > > Some of the poms still have hadoop 1 and hadoop 1.1 profiles even though the > root pom has them removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16643:
---
Status: Open (was: Patch Available)
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16179: Component/s: spark > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16663:
Component/s: security
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> stopped will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21 12:09:08,259 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Client=P72981//10.18.248.96 shutdown
> 2016-09-21 12:09:08,261 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-21 12:09:08,276 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
>
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510129#comment-15510129 ] Duo Zhang commented on HBASE-15921: --- https://github.com/Apache9/hbase/blob/asynclocator/hbase-client/src/main/java/org/apache/hadoop/hbase/client/async/ZKClusterRegistry.java Much simpler... > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.demo.patch, HBASE-15921.patch, > HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16667) Building with JDK 8: ignoring option MaxPermSize=256m
[
https://issues.apache.org/jira/browse/HBASE-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16667:
Fix Version/s: (was: 1.1.7)
> Building with JDK 8: ignoring option MaxPermSize=256m
> -
>
> Key: HBASE-16667
> URL: https://issues.apache.org/jira/browse/HBASE-16667
> Project: HBase
> Issue Type: Improvement
> Components: build
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16667-01.patch
>
>
> In JDK 8 the permgen was removed.
> As a consequence the build shows this line a lot of times, cluttering the
> output.
> {quote}
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support
> was removed in 8.0
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510385#comment-15510385 ] Vladimir Rodionov commented on HBASE-16630: --- [~dvdreddy], memory requirements is very high. You efficiently add one more backingMap during defragmentation (majority of blocks will have zero refCount). backingMap itself takes 200+ bytes per block. For reasonably large caches, with 10M blocks, that is 2GB+ more heap to run defragmentation. Seems quite a high overhead. > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-16630-v2.patch, HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16654) Better handle channelInactive and close for netty rpc client
[ https://issues.apache.org/jira/browse/HBASE-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16654: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~stack] for reviewing. > Better handle channelInactive and close for netty rpc client > > > Key: HBASE-16654 > URL: https://issues.apache.org/jira/browse/HBASE-16654 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16654.patch > > > We should pass the event to the next handler in the pipeline as > channelInactive and close are usually used as the cleanup method of a > ChannelHandler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16423) Add re-compare option to VerifyReplication to avoid occasional inconsistent rows
[
https://issues.apache.org/jira/browse/HBASE-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510275#comment-15510275
]
Ted Yu commented on HBASE-16423:
{code}
125 sleepToReCompare = conf.getInt(NAME +".sleepToReCompare", 0);
{code}
How about naming the parameter sleepMsBeforeReCompare ?
{code}
+ return;
+} catch (Exception e) {
+ LOG.error("recompare fail!");
{code}
Can you log more information in the error (such as row key) ?
{code}
+System.err.println(" recomparesleep milliseconds to sleep before
recompare row.");
{code}
Please mention that value 0 disables recomparison.
> Add re-compare option to VerifyReplication to avoid occasional inconsistent
> rows
>
>
> Key: HBASE-16423
> URL: https://issues.apache.org/jira/browse/HBASE-16423
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.0.0
>Reporter: Jianwei Cui
>Priority: Minor
> Attachments: HBASE-16423-v1.patch
>
>
> Because replication keeps eventually consistency, VerifyReplication may
> report inconsistent rows if there are data being written to source or peer
> clusters during scanning. These occasionally inconsistent rows will have the
> same data if we do the comparison again after a short period. It is not easy
> to find the really inconsistent rows if VerifyReplication report a large
> number of such occasionally inconsistency. To avoid this case, we can add an
> option to make VerifyReplication read out the inconsistent rows again after
> sleeping a few seconds and re-compare the rows during scanning. This behavior
> follows the eventually consistency of hbase's replication. Suggestions and
> discussions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16667) Building with JDK 8: ignoring option MaxPermSize=256m
[
https://issues.apache.org/jira/browse/HBASE-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510467#comment-15510467
]
Sean Busbey commented on HBASE-16667:
-
In master we should just remove the option entirely, since it's jdk8+ only. The
attached patch looks good for branches-1, for those where we build with both
jdk7 and jdk8.
[~nielsbasjes], could you attach a patch for master that just removes the
permgen size entirely? once that goes through precommit please rename the
current patch so it gets tested against branch-1 (it would be something like
HBASE-16667-branch-1.v1.patch).
> Building with JDK 8: ignoring option MaxPermSize=256m
> -
>
> Key: HBASE-16667
> URL: https://issues.apache.org/jira/browse/HBASE-16667
> Project: HBase
> Issue Type: Improvement
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-16667-01.patch
>
>
> In JDK 8 the permgen was removed.
> As a consequence the build shows this line a lot of times, cluttering the
> output.
> {quote}
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support
> was removed in 8.0
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16667) Building with JDK 8: ignoring option MaxPermSize=256m
[
https://issues.apache.org/jira/browse/HBASE-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16667:
Priority: Minor (was: Major)
> Building with JDK 8: ignoring option MaxPermSize=256m
> -
>
> Key: HBASE-16667
> URL: https://issues.apache.org/jira/browse/HBASE-16667
> Project: HBase
> Issue Type: Improvement
> Components: build
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16667-01.patch
>
>
> In JDK 8 the permgen was removed.
> As a consequence the build shows this line a lot of times, cluttering the
> output.
> {quote}
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support
> was removed in 8.0
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16667) Building with JDK 8: ignoring option MaxPermSize=256m
[
https://issues.apache.org/jira/browse/HBASE-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16667:
Labels: beginner (was: )
> Building with JDK 8: ignoring option MaxPermSize=256m
> -
>
> Key: HBASE-16667
> URL: https://issues.apache.org/jira/browse/HBASE-16667
> Project: HBase
> Issue Type: Improvement
> Components: build
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16667-01.patch
>
>
> In JDK 8 the permgen was removed.
> As a consequence the build shows this line a lot of times, cluttering the
> output.
> {quote}
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support
> was removed in 8.0
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16667) Building with JDK 8: ignoring option MaxPermSize=256m
[
https://issues.apache.org/jira/browse/HBASE-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16667:
Component/s: build
> Building with JDK 8: ignoring option MaxPermSize=256m
> -
>
> Key: HBASE-16667
> URL: https://issues.apache.org/jira/browse/HBASE-16667
> Project: HBase
> Issue Type: Improvement
> Components: build
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16667-01.patch
>
>
> In JDK 8 the permgen was removed.
> As a consequence the build shows this line a lot of times, cluttering the
> output.
> {quote}
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support
> was removed in 8.0
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16667) Building with JDK 8: ignoring option MaxPermSize=256m
[
https://issues.apache.org/jira/browse/HBASE-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16667:
Fix Version/s: 1.2.4
1.1.7
1.4.0
1.3.0
2.0.0
> Building with JDK 8: ignoring option MaxPermSize=256m
> -
>
> Key: HBASE-16667
> URL: https://issues.apache.org/jira/browse/HBASE-16667
> Project: HBase
> Issue Type: Improvement
> Components: build
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16667-01.patch
>
>
> In JDK 8 the permgen was removed.
> As a consequence the build shows this line a lot of times, cluttering the
> output.
> {quote}
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support
> was removed in 8.0
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16663:
Priority: Critical (was: Major)
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> stopped will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21 12:09:08,259 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Client=P72981//10.18.248.96 shutdown
> 2016-09-21 12:09:08,261 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-21 12:09:08,276 WARN
>
[jira] [Updated] (HBASE-16536) Make the HBase minicluster easy to use for testing downstream applications.
[ https://issues.apache.org/jira/browse/HBASE-16536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niels Basjes updated HBASE-16536: - Attachment: HBASE-16536-03.patch This patch changes the way the registering is done to a manual choice. So you can run as many testing clusters from a JVM as you like. Yet you can only register one in the classpath once. This needs some more testing. > Make the HBase minicluster easy to use for testing downstream applications. > --- > > Key: HBASE-16536 > URL: https://issues.apache.org/jira/browse/HBASE-16536 > Project: HBase > Issue Type: Improvement >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-16536-01.patch, HBASE-16536-02.patch, > HBASE-16536-03.patch > > > In many applications I write I use HBase to store information. > A big problem is testing these applications. > I have seen several situations where people have written tests that create > tables in the development cluster and due to firewalls and such couldn't run > those tests from Jenkins. > A while ago I wrote the FilterTestingCluster class that makes unit testing > the client side filters a lot easier. With this ticket I propose to make this > more generic and make it so that user applications can easily incorporate it > into their own unit tests without any major modifications to their > application. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16667) Building with JDK 8: ignoring option MaxPermSize=256m
[ https://issues.apache.org/jira/browse/HBASE-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niels Basjes updated HBASE-16667: - Attachment: HBASE-16667-01.patch > Building with JDK 8: ignoring option MaxPermSize=256m > - > > Key: HBASE-16667 > URL: https://issues.apache.org/jira/browse/HBASE-16667 > Project: HBase > Issue Type: Improvement >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-16667-01.patch > > > In JDK 8 the permgen was removed. > As a consequence the build shows this line a lot of times, cluttering the > output. > OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support > was removed in 8.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
[ https://issues.apache.org/jira/browse/HBASE-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510284#comment-15510284 ] Robert Yokota commented on HBASE-16414: --- Ok, great. I created HADOOP-13635 so my company has a JIRA to refer to. I hope you don't mind. Thanks. > Improve performance for RPC encryption with Apache Common Crypto > > > Key: HBASE-16414 > URL: https://issues.apache.org/jira/browse/HBASE-16414 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HBASE-16414.001.patch, HBASE-16414.002.patch, > HBASE-16414.003.patch, HBASE-16414.004.patch, > HbaseRpcEncryptionWithCrypoto.docx > > > Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to > "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms > for secure authentication and data protection. For DIGEST-MD5, it uses DES, > 3DES or RC4 to do encryption and it is very slow, especially for Scan. This > will become the bottleneck of the RPC throughput. > Apache Commons Crypto is a cryptographic library optimized with AES-NI. It > provides Java API for both cipher level and Java stream level. Developers can > use it to implement high performance AES encryption/decryption with the > minimum code and effort. Compare with the current implementation of > org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher > and OpenSSL Cipher which is better performance than JCE Cipher. User can > configure the cipher type and the default is JCE Cipher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510294#comment-15510294 ] Ted Yu commented on HBASE-16630: If you have time, do you mind trying patch v2 on a (test) cluster ? Thanks > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-16630-v2.patch, HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16423) Add re-compare option to VerifyReplication to avoid occasional inconsistent rows
[ https://issues.apache.org/jira/browse/HBASE-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianwei Cui updated HBASE-16423: Attachment: HBASE-16423-v1.patch > Add re-compare option to VerifyReplication to avoid occasional inconsistent > rows > > > Key: HBASE-16423 > URL: https://issues.apache.org/jira/browse/HBASE-16423 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0 >Reporter: Jianwei Cui >Priority: Minor > Attachments: HBASE-16423-v1.patch > > > Because replication keeps eventually consistency, VerifyReplication may > report inconsistent rows if there are data being written to source or peer > clusters during scanning. These occasionally inconsistent rows will have the > same data if we do the comparison again after a short period. It is not easy > to find the really inconsistent rows if VerifyReplication report a large > number of such occasionally inconsistency. To avoid this case, we can add an > option to make VerifyReplication read out the inconsistent rows again after > sleeping a few seconds and re-compare the rows during scanning. This behavior > follows the eventually consistency of hbase's replication. Suggestions and > discussions are welcomed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510273#comment-15510273
]
Hadoop QA commented on HBASE-16643:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 22m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 29s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 115m 15s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 191m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestMasterReplication |
| Timed out junit tests |
org.apache.hadoop.hbase.namespace.TestNamespaceAuditor |
| | org.apache.hadoop.hbase.wal.TestWALSplitCompressed |
| |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes |
| | org.apache.hadoop.hbase.wal.TestBoundedRegionGroupingStrategy |
| | org.apache.hadoop.hbase.wal.TestWALSplit |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829558/HBASE-16643_2.patch |
| JIRA Issue | HBASE-16643 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 9f3c74c8d7ea 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c67983e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3630/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3630/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3630/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Commented] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510587#comment-15510587
]
Umesh Agashe commented on HBASE-16294:
--
Tested manually by following the steps given in this ticket.
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16668) Admin class should have a synchronous Admin#mergeRegions* method
Jonathan Hsieh created HBASE-16668:
--
Summary: Admin class should have a synchronous Admin#mergeRegions*
method
Key: HBASE-16668
URL: https://issues.apache.org/jira/browse/HBASE-16668
Project: HBase
Issue Type: Bug
Components: hbase
Affects Versions: 2.0.0
Reporter: Jonathan Hsieh
Fix For: 2.0.0
In trunk from HBASE-14552, we have deprecated {{void Admin#mergeRegions}} (in
1.x this was an asynchronous call) and replaced it with {{Future
Admin#mergeRegionsAsync}} which is clearly async.
This leaves us only with the async version.
We should have an easy way to make {{mergeRegions}} or an equivalant behave
synchronously.
For normal java Futures, we could just call the future's {{get()}} method.
Unforutnately, the future this method returns doesn't follow java Future
convention and throws Unimplemented operation when a plain {{get()}} is called
and makes the api harder to use and read. We could make this future act more
normally, and have the timeout throw an InterruptedException.
Alternately, we could expose a new method in {{Admin}} that behaves
synchronously such as {{HBaseAdmin#mergeRegionsSync}}. The caveat here is that
we shouldn't use the name {{#mergeRegions}} since it exists in 1.x with async
semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Agashe updated HBASE-16294:
-
Release Note: Fixed warning error message displayed for region directory
not found for non-default/ non-primary replicas in hbck
Status: Patch Available (was: Open)
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 1.2.2, 1.1.5, 1.0.3, 2.0.0, 1.3.0, 1.4.0
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16636) Regions in Transition counts wrong (zero) in HMaster /jmx, prevents detecting Regions Stuck in Transition
[
https://issues.apache.org/jira/browse/HBASE-16636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510622#comment-15510622
]
huaxiang sun commented on HBASE-16636:
--
I did not test ritOldestAge, so not sure about this one.
> Regions in Transition counts wrong (zero) in HMaster /jmx, prevents detecting
> Regions Stuck in Transition
> -
>
> Key: HBASE-16636
> URL: https://issues.apache.org/jira/browse/HBASE-16636
> Project: HBase
> Issue Type: Bug
> Components: UI
>Affects Versions: 1.1.2
> Environment: HDP 2.3.2
>Reporter: Hari Sekhon
>Priority: Minor
> Attachments: Regions_in_Transition_UI.png, ritCountOverThreshold.png
>
>
> I've discovered that the Region in Transition counts are wrong in the HMaster
> UI /jmx page.
> The /master-status page clearly shows 3 regions stuck in transition but the
> /jmx page I was monitoring reported 0 for ritCountOverThreshold.
> {code}
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManger",
> "modelerType" : "Master,sub=AssignmentManger",
> "tag.Context" : "master",
> ...
> "ritOldestAge" : 0,
> "ritCountOverThreshold" : 0,
> ...
> "ritCount" : 0,
> {code}
> I have a nagios plugin I wrote which was checking this which I've since had
> to rewrite to parse the /master-status page instead (the code is in
> check_hbase_regions_stuck_in_transition.py at
> https://github.com/harisekhon/nagios-plugins).
> I'm attaching screenshots of both /master-status and /jmx to show the
> difference in the 2 pages on the HMaster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16657) Expose per-region last major compaction timestamp in RegionServer UI
[ https://issues.apache.org/jira/browse/HBASE-16657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Pho updated HBASE-16657: --- Status: Patch Available (was: Open) > Expose per-region last major compaction timestamp in RegionServer UI > > > Key: HBASE-16657 > URL: https://issues.apache.org/jira/browse/HBASE-16657 > Project: HBase > Issue Type: Improvement > Components: regionserver, UI >Reporter: Gary Helmling >Assignee: Dustin Pho >Priority: Minor > Attachments: HBASE-16657.001.patch > > > HBASE-12859 added some tracking for the last major compaction completed for > each region. However, this is currently only exposed through the cluster > status reporting and the Admin API. Since the regionserver is already > reporting this information, it would be nice to fold it in somewhere to the > region listing in the regionserver UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Agashe updated HBASE-16294:
-
Attachment: HBASE-16294.v1.patch
Fixes warning error displayed for region directory not found for non-default/
non-primary replicas in hbck
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-16657) Expose per-region last major compaction timestamp in RegionServer UI
[ https://issues.apache.org/jira/browse/HBASE-16657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Pho reassigned HBASE-16657: -- Assignee: Dustin Pho > Expose per-region last major compaction timestamp in RegionServer UI > > > Key: HBASE-16657 > URL: https://issues.apache.org/jira/browse/HBASE-16657 > Project: HBase > Issue Type: Improvement > Components: regionserver, UI >Reporter: Gary Helmling >Assignee: Dustin Pho >Priority: Minor > > HBASE-12859 added some tracking for the last major compaction completed for > each region. However, this is currently only exposed through the cluster > status reporting and the Admin API. Since the regionserver is already > reporting this information, it would be nice to fold it in somewhere to the > region listing in the regionserver UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510693#comment-15510693
]
Matteo Bertozzi commented on HBASE-16294:
-
+1
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: 14417.v1.txt Patch v1 shows basic flow of bulk loaded hfiles support. To be added: cleanup of BulkLoadDescriptor at the completion of full backup BackupHFileCleaner which guards bulk loaded hfiles which are referenced by backup > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417.v1.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16657) Expose per-region last major compaction timestamp in RegionServer UI
[ https://issues.apache.org/jira/browse/HBASE-16657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Pho updated HBASE-16657: --- Attachment: HBASE-16657.001.patch Added a column for last compaction time under RegionServer UI > Regions > Compaction Metrics > Expose per-region last major compaction timestamp in RegionServer UI > > > Key: HBASE-16657 > URL: https://issues.apache.org/jira/browse/HBASE-16657 > Project: HBase > Issue Type: Improvement > Components: regionserver, UI >Reporter: Gary Helmling >Assignee: Dustin Pho >Priority: Minor > Attachments: HBASE-16657.001.patch > > > HBASE-12859 added some tracking for the last major compaction completed for > each region. However, this is currently only exposed through the cluster > status reporting and the Admin API. Since the regionserver is already > reporting this information, it would be nice to fold it in somewhere to the > region listing in the regionserver UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16668) Admin class should have a synchronous Admin#mergeRegions* method
[
https://issues.apache.org/jira/browse/HBASE-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510774#comment-15510774
]
Jonathan Hsieh commented on HBASE-16668:
[~mbertozzi], [~syuanjiang], this was introduced in HBASE-14552. What are your
opinions on this?
> Admin class should have a synchronous Admin#mergeRegions* method
> -
>
> Key: HBASE-16668
> URL: https://issues.apache.org/jira/browse/HBASE-16668
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
> Fix For: 2.0.0
>
>
> In trunk from HBASE-14552, we have deprecated {{void Admin#mergeRegions}} (in
> 1.x this was an asynchronous call) and replaced it with {{Future
> Admin#mergeRegionsAsync}} which is clearly async.
> This leaves us only with the async version.
> We should have an easy way to make {{mergeRegions}} or an equivalant behave
> synchronously.
> For normal java Futures, we could just call the future's {{get()}} method.
> Unforutnately, the future this method returns doesn't follow java Future
> convention and throws Unimplemented operation when a plain {{get()}} is
> called and makes the api harder to use and read. We could make this future
> act more normally, and have the timeout throw an InterruptedException.
> Alternately, we could expose a new method in {{Admin}} that behaves
> synchronously such as {{HBaseAdmin#mergeRegionsSync}}. The caveat here is
> that we shouldn't use the name {{#mergeRegions}} since it exists in 1.x with
> async semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12088) Remove un-used profiles in non-root poms
[
https://issues.apache.org/jira/browse/HBASE-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510524#comment-15510524
]
Hadoop QA commented on HBASE-12088:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 32s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
22s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 16s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 16s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 43s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 57s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 53s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s
{color} | {color:green} hbase-prefix-tree in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 4s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s
{color} | {color:green} hbase-testing-util in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s
{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 28s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hbase-examples in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hbase-external-blockcache in the patch
[jira] [Assigned] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-14417: -- Assignee: Ted Yu (was: Vladimir Rodionov) > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14734) BindException when setting up MiniKdc
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14734: - Description: Root cause : Port for kdc service gets selected in the constructor, but we bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() --> KdcServer.start(). In meantime, some other service can capture the port which results in BindException. The solution here is to catch the exception and retry. >From >https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ Error Message Address already in use Stacktrace java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:444) at sun.nio.ch.Net.bind(Net.java:436) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) at org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) at org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Can this utility be made to not fail if address taken? Try another? was: >From >https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ Error Message Address already in use Stacktrace java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:444) at sun.nio.ch.Net.bind(Net.java:436) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) at org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) at org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Can this utility be made to not fail if address taken? Try another? > BindException when setting up MiniKdc > - > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: Appy > Attachments: HBASE-14734.master.001.patch > > > Root cause : Port for kdc service gets selected in the constructor, but we > bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() --> > KdcServer.start(). In meantime, some other service can capture the port which > results in BindException. The solution here is to catch the exception and > retry. > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at
[jira] [Updated] (HBASE-16669) fix flaky TestAdmin#testmergeRegions
[
https://issues.apache.org/jira/browse/HBASE-16669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-16669:
---
Attachment: hbase-16669.v1.patch
The old version slept for a second and assumed that the merge operation
completed. This update waits until the future resolves or fails with a timeout.
Ran this 10x and it passed locally.
> fix flaky TestAdmin#testmergeRegions
>
>
> Key: HBASE-16669
> URL: https://issues.apache.org/jira/browse/HBASE-16669
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16669.v1.patch
>
>
> Recent test runs show that this test is failing with these error messages:
> {code}
> java.lang.AssertionError: expected:<2> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.client.TestAdmin1.testMergeRegions(TestAdmin1.java:1385)
> {code}
> or
> {code}
> java.lang.AssertionError: expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.client.TestAdmin1.testMergeRegions(TestAdmin1.java:1394)
> {code}
> looking at the code this indicates that merge operation did not complete or
> didn't work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510916#comment-15510916
]
Hadoop QA commented on HBASE-16294:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 54s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 13s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 121m 9s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.master.TestMasterFailover |
| | org.apache.hadoop.hbase.TestZooKeeper |
| | org.apache.hadoop.hbase.master.TestMasterFailoverBalancerPersistence |
| | org.apache.hadoop.hbase.master.TestTableLockManager |
| | org.apache.hadoop.hbase.master.TestMetaShutdownHandler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829624/HBASE-16294.v1.patch |
| JIRA Issue | HBASE-16294 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 4071fe70a611 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5568929 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3632/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3632/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3632/testReport/ |
| modules | C: hbase-server U:
[jira] [Updated] (HBASE-16669) fix flaky TestAdmin#testmergeRegions
[
https://issues.apache.org/jira/browse/HBASE-16669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-16669:
---
Status: Patch Available (was: Open)
> fix flaky TestAdmin#testmergeRegions
>
>
> Key: HBASE-16669
> URL: https://issues.apache.org/jira/browse/HBASE-16669
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16669.v1.patch
>
>
> Recent test runs show that this test is failing with these error messages:
> {code}
> java.lang.AssertionError: expected:<2> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.client.TestAdmin1.testMergeRegions(TestAdmin1.java:1385)
> {code}
> or
> {code}
> java.lang.AssertionError: expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.client.TestAdmin1.testMergeRegions(TestAdmin1.java:1394)
> {code}
> looking at the code this indicates that merge operation did not complete or
> didn't work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-16669) fix flaky TestAdmin#testmergeRegions
[
https://issues.apache.org/jira/browse/HBASE-16669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh reassigned HBASE-16669:
--
Assignee: Jonathan Hsieh
> fix flaky TestAdmin#testmergeRegions
>
>
> Key: HBASE-16669
> URL: https://issues.apache.org/jira/browse/HBASE-16669
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16669.v1.patch
>
>
> Recent test runs show that this test is failing with these error messages:
> {code}
> java.lang.AssertionError: expected:<2> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.client.TestAdmin1.testMergeRegions(TestAdmin1.java:1385)
> {code}
> or
> {code}
> java.lang.AssertionError: expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.client.TestAdmin1.testMergeRegions(TestAdmin1.java:1394)
> {code}
> looking at the code this indicates that merge operation did not complete or
> didn't work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16557) UI for backup / restore
[ https://issues.apache.org/jira/browse/HBASE-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510969#comment-15510969 ] Ted Yu commented on HBASE-16557: Once support for bulk loaded hfiles is added, the UI should also show number of rows (per table) in hbase:backup for BulkLoadDescriptor. > UI for backup / restore > --- > > Key: HBASE-16557 > URL: https://issues.apache.org/jira/browse/HBASE-16557 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu > Labels: backup, ui > > A new tab would be added to master UI, similar to procedures.jsp. > The following information would be shown (B means applicable to backup only): > Type: backup or restore > Id : backup Id (B) > Ancestors: dependent backups (B) > State: on-going, aborting or complete > Owner: owner's user name > Start time -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16657) Expose per-region last major compaction timestamp in RegionServer UI
[
https://issues.apache.org/jira/browse/HBASE-16657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510983#comment-15510983
]
Hadoop QA commented on HBASE-16657:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 30s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 3s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 114m 55s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829631/HBASE-16657.001.patch
|
| JIRA Issue | HBASE-16657 |
| Optional Tests | asflicense javac javadoc unit |
| uname | Linux dc090c8ca169 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5568929 |
| Default Java | 1.8.0_101 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3633/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3633/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3633/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3633/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Expose per-region last major compaction timestamp in RegionServer UI
>
>
> Key: HBASE-16657
> URL: https://issues.apache.org/jira/browse/HBASE-16657
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, UI
>Reporter: Gary Helmling
>Assignee: Dustin Pho
>Priority: Minor
> Attachments: HBASE-16657.001.patch
>
>
> HBASE-12859 added some tracking for the last major compaction completed for
> each region.
[jira] [Comment Edited] (HBASE-14135) HBase Backup/Restore Phase 3: Merge backup images
[ https://issues.apache.org/jira/browse/HBASE-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510810#comment-15510810 ] Vladimir Rodionov edited comment on HBASE-14135 at 9/21/16 6:47 PM: Two issues with incremental backup implementation: # Storage usage. WAL files are significantly larger than hfiles, especially when compression is enabled in hfiles # Restore operation time. The most significant is the first one. We can that storage reduction factor will be something between 3-5 once convert (merge) is implemented. Running conversion (merge) on a remote destination (cluster) is not feasible in many cases, so merge will require reading/writing data back and forth between source and destination. I will postpone this feature until filtering at the source during incremental backup is done (HBASE-14141). was (Author: vrodionov): Two issues with incremental backup implementation: # Storage usage. WAL files are significantly larger than hfiles, especially when compression is enabled in hfiles # Restore operation time. The most significant is the first one. We can that storage reduction factor will be something between 3-5 once convert (merge) is implemented. Running conversion (merge) on a remote destination (cluster) is not feasible in many cases, so merge will require reading/writing data back and forth between source and destination. I will postpone this feature until filtering at the source during incremental backup is done. > HBase Backup/Restore Phase 3: Merge backup images > - > > Key: HBASE-14135 > URL: https://issues.apache.org/jira/browse/HBASE-14135 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14734) TestGenerateDelegationToken fails with BindAddress clash
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14734: - Attachment: HBASE-14734.master.001.patch > TestGenerateDelegationToken fails with BindAddress clash > > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack > Attachments: HBASE-14734.master.001.patch > > > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14734) TestGenerateDelegationToken fails with BindAddress clash
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14734: - Status: Patch Available (was: Open) > TestGenerateDelegationToken fails with BindAddress clash > > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: Appy > Attachments: HBASE-14734.master.001.patch > > > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
