[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506958#comment-15506958 ] Yu Li commented on HBASE-15921: --- bq. And now, I prefer that we can implement a simple AsyncTable that only supports get, put and delete in this issue as what Jurriaan Mous proposed initially [~Apache9] This will be much easier, and the most complicated part lies in batch and scan. Will attach a demo patch soon. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506953#comment-15506953 ] stack commented on HBASE-16134: --- When do we ever want to emit tags? When debugging? WAL write time is 'write time' so when Cell comes in the front door, we use a write-time Cell that will include tags when we call write. Otherwise, when we pull from hfile, we use a Cell type that does not include tags? If debug, pass a flag and then use a different Cell type, one that will do debug things including emitting Tags? > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506978#comment-15506978 ] Anoop Sam John commented on HBASE-16134: 2 issues with this 1. We want to emit tags in any read time when the KVCodecWithTags is configured. Export tool when doing this read, we have to emit tags or else we will loose it. May be this area we need to see whether we have documented this correctly or not 2. WAL and write time go with diff Cell and read time diff.. The issue comes with compaction. Here there is read and write. We pass the read cells (From an HFile) to writer. May be we should decide based on isCompaction boolean. This means the Cells what we add to memstore (Those will go to writer finally) will need to have a logic which will write tags. So what abt this same cells when read. The scan can read cells from memstore also.. So we will end up in copying those to a new Cell of type which wont write tags? Sorry boss.. I fear this wont work out :-( > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507015#comment-15507015 ] ramkrishna.s.vasudevan commented on HBASE-16134: We need tags for compaction and also those tags are needed for visibility and access controller right. Ya we don't emit but still we need those details. Different cells for read and write wrt to tags won't work out in all cases. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input
[
https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507063#comment-15507063
]
Hadoop QA commented on HBASE-16646:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-16646 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829410/16646.addendum |
| JIRA Issue | HBASE-16646 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3620/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Enhance LoadIncrementalHFiles API to accept store file paths as input
> -
>
> Key: HBASE-16646
> URL: https://issues.apache.org/jira/browse/HBASE-16646
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16646.addendum, 16646.v0.txt, 16646.v1.txt,
> 16646.v2.txt, 16646.v3.txt
>
>
> Currently LoadIncrementalHFiles takes the directory (output path) as input
> parameter.
> In some scenarios (incremental restore of bulk loaded hfiles), the List of
> paths to hfiles is known.
> LoadIncrementalHFiles can take the List as input parameter and proceed with
> loading.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507134#comment-15507134 ] Anoop Sam John commented on HBASE-16134: #1 For replication we use this Codec. Also there is a config which can be used to specify the Codec to be used. So one can change to this Codec. #2 It is not Reader which create Cells by HFileScanner - Ya we will have new scanner object for the compaction read and that knows whether this is a compaction scan (Already it knows). That is kind of ok bq.Can MemSlab work here? Sorry did not get. How MemSLAB can help? bq.Just trying to have it so Tags are first-class. We just have to solve the issue that stops Cells being put on the RPC. A flag is easy. Just trying to explore if any other possibility. Understood.. Ya agree it is bit ugly. Will it be better if we have 2 methods in ExtendedCell ie. write(OS) and writeNoTags(OS)? > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507183#comment-15507183 ] Anoop Sam John commented on HBASE-16134: Replicator reads Cells from WAL. bq.Do we create a Cell from MemSLAB when we send Cell back over RPC? No. During write to memstore, we copy to MSLAB and create a new Cell object around it and keep in CSLM. When read, we return back the same object. Or else we will have to make many garbage. bq.Nah. This is just boolean in another form. Ya.. Just asking whether this will be a bit better. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506952#comment-15506952 ] Yu Li commented on HBASE-15921: --- Well, in our implementation, we use sync call to get region location and then issue the async call to the target RS with callback. We assume region locating is a low-cost step and for long running client like in streaming the location will be cached with a big chance. Makes sense? > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507049#comment-15507049 ] Anoop Sam John commented on HBASE-16134: If we have to avoid passing a boolean 'withTags' we may have to either have 2 methods (write, writeNoTags) or make the agreement of the write method such that it will write key and value but no tags. And the KVCodecWithTags will write the tags on its own. I dont feel the latter is good. Tags are part of Cell only > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16660) ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
[
https://issues.apache.org/jira/browse/HBASE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507077#comment-15507077
]
Hadoop QA commented on HBASE-16660:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 29s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} 0.98 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} 0.98 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
32s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} 0.98 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 58s
{color} | {color:red} hbase-server in 0.98 has 85 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 31s
{color} | {color:red} hbase-server in 0.98 failed with JDK v1.8.0_101. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} 0.98 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 26s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 16s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 31s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_101.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 126m 9s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 174m 45s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-20 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829392/HBASE-16660-0.98.patch
|
| JIRA Issue | HBASE-16660 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 7ac9912b5aa5 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | 0.98 / efd1ba1 |
| Default Java | 1.7.0_111 |
| Multi-JDK versions |
[jira] [Commented] (HBASE-16551) Cleanup SplitLogManager and CatalogJanitor
[ https://issues.apache.org/jira/browse/HBASE-16551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506980#comment-15506980 ] Stephen Yuan Jiang commented on HBASE-16551: +1 > Cleanup SplitLogManager and CatalogJanitor > -- > > Key: HBASE-16551 > URL: https://issues.apache.org/jira/browse/HBASE-16551 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16551-v0.patch > > > couple of cleanups around SplitLogManager and CatalogJanitor: > replace all copy-pasted cast in SplitLogManager with one call to an helper > method > remove Server, MasterServices, Stoppable since we call the class in only one > place and is called as master, master, master > reuse MockNoopMasterServices instead of creating the Server/MasterServices > classes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16646: --- Attachment: 16646.addendum Turns out I need to maintain the original cleanup logic since prepareHFileQueue() may throw IOE. Addendum extracts the cleanup code into its own method and moves the try / finally block into doBulkLoad(). > Enhance LoadIncrementalHFiles API to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16646.addendum, 16646.v0.txt, 16646.v1.txt, > 16646.v2.txt, 16646.v3.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507094#comment-15507094 ] stack commented on HBASE-16134: --- On #1, yeah, when is that Codec used? On #2, open a new Reader? One that includes Tags? On read from memstore, that is a problem. Can MemSlab work here? Just trying to have it so Tags are first-class. We just have to solve the issue that stops Cells being put on the RPC. A flag is easy. Just trying to explore if any other possibility. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507150#comment-15507150 ] stack commented on HBASE-16134: --- Replication opens a new Reader, right, each time it replicates? bq. How MemSLAB can help? Do we create a Cell from MemSLAB when we send Cell back over RPC? bq. Will it be better if we have 2 methods in ExtendedCell ie. write(OS) and writeNoTags(OS)? Nah. This is just boolean in another form. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[
https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505739#comment-15505739
]
Duo Zhang commented on HBASE-16648:
---
https://github.com/Apache9/microbench/tree/master/src/main/java/com/xiaomi/infra/microbench/compute
{noformat}
./bin/run.sh -f 1 .*IATest.*
Benchmark Mode Cnt ScoreError Units
CIATest.test thrpt 20 867.720 ± 0.233 ops/s
PIATest.test thrpt 20 1796.693 ± 39.908 ops/s
./bin/run.sh -f 1 -t 10 .*IATest.*
Benchmark Mode Cnt ScoreError Units
CIATest.test thrpt 20 3452.054 ± 44.537 ops/s
PIATest.test thrpt 20 12101.279 ± 7.215 ops/s
{noformat}
The putIfAbsent version of computeIfAbsent is much faster. So I think we should
use the computeIfAbsent in ConcurrentMap instead of ConcurrentHashMap as for us
the creation is usually cheap.
And in fact, it does not recommend to do a heavy operation in the
mappingFunction of computeIfAbsent.. Then I do not see the necessity of
reimplementing this method in ConcurrentHashMap...
> [JDK8] Use computeIfAbsent instead of get and putIfAbsent
> -
>
> Key: HBASE-16648
> URL: https://issues.apache.org/jira/browse/HBASE-16648
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-16648-v1.patch, HBASE-16648-v2.patch,
> HBASE-16648.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505717#comment-15505717 ] Jerry He commented on HBASE-16646: -- Is it possible that we don't have the added run method? > Enhance LoadIncrementalHFiles API to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16646.v0.txt, 16646.v1.txt, 16646.v2.txt, 16646.v3.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16578) Mob data loss after mob compaction and normal compcation
[ https://issues.apache.org/jira/browse/HBASE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du reassigned HBASE-16578: Assignee: Jingcheng Du (was: huaxiang sun) > Mob data loss after mob compaction and normal compcation > > > Key: HBASE-16578 > URL: https://issues.apache.org/jira/browse/HBASE-16578 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: Jingcheng Du > Attachments: TestMobCompaction.java, TestMobCompaction.java > > > I have a unittest case, which could explore the mob data loss issue. The root > cause is that with the following line: > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactor.java#L625 > It will make the old mob reference cell win during compaction. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505765#comment-15505765 ] Duo Zhang commented on HBASE-15921: --- [~chenheng] I found a big problem of your plan. How do you deal with scan? I do not think it is easy to unify scan into AsyncProcess... Thanks. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[
https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505973#comment-15505973
]
Phil Yang commented on HBASE-15921:
---
{quote}
As for 'async scan', i have not think about it carefully, maybe return one
Future Object for each call "next()" ?
{quote}
Talked with [~Apache9], we may have these options:
1: Use callback, when a Result is ready we run the callback provided by the
user. But if we run the callback in the main loop, the callback must be simple.
If users need complex logic, they must put the Result into their own
thread(pool). And we may guarantee if the previous callback is not done, we can
not run next row's callback.(It is simple if we have only one thread).
2: Like you said, for each next() we return a CompletableFuture. The first
future is the first Result, the second is the second... We should keep the
order if user call next for many times at first.
3: We return a future when the scanner init. And each time we get the Result
from future, we also return a new CompletableFuture belonging to the next
Result.
> Add first AsyncTable impl and create TableImpl based on it
> --
>
> Key: HBASE-15921
> URL: https://issues.apache.org/jira/browse/HBASE-15921
> Project: HBase
> Issue Type: Improvement
>Reporter: Jurriaan Mous
>Assignee: Jurriaan Mous
> Attachments: HBASE-15921.patch, HBASE-15921.v1.patch
>
>
> First we create an AsyncTable interface with implementation without the Scan
> functionality. Those will land in a separate patch since they need a refactor
> of existing scans.
> Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl
> internally and should be a bit faster because it does jump through less hoops
> to do ProtoBuf transportation. This way we can run all existing tests on the
> AsyncTableImpl to guarantee its quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505832#comment-15505832 ] stack commented on HBASE-16134: --- So, if no Tags, we don't serialize any out. We are talking about the case where a Cell has tags and it is asked to serialize itself. We want option of not serializing tags. When is this Cell created? It is created at read time only? Can we create a Cell at read-time that is of a type that never includes tags when it serializes? The only time the Cell will be asked to seriailze is when it is writing on the RPC to send back to the client? Are there other times? > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505928#comment-15505928 ] Phil Yang commented on HBASE-15921: --- And we may need AsyncResultScanner? > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16644) Errors when reading legit HFile' Trailer on branch 1.3
[
https://issues.apache.org/jira/browse/HBASE-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505801#comment-15505801
]
Mikhail Antonov commented on HBASE-16644:
-
[~stack] So we looked closer at offending files and found that they had HFile
version: major: 2, minor: 0 in them. (in HBASE-5074 we added notion of minor
version and checksums in blocks). Seems like really old files that never got
compacted away.
Looks like it's matter of format compatibility with old version.
> Errors when reading legit HFile' Trailer on branch 1.3
> --
>
> Key: HBASE-16644
> URL: https://issues.apache.org/jira/browse/HBASE-16644
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Critical
> Fix For: 1.3.0
>
> Attachments: HBASE-16644.branch-1.3.patch
>
>
> There seems to be a regression in branch 1.3 where we can't read HFile
> trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on
> some HFiles that could be successfully read on 1.2.
> I've seen this error manifesting in two ways so far.
> {code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException:
> Problem reading HFile Trailer from file
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
> ... 6 more
> Caused by: java.io.IOException: Invalid HFile block magic:
> \x00\x00\x04\x00\x00\x00\x00\x00
> at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:344)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485)
> {code}
> and second
> {code}
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
> reading HFile Trailer from file
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
> at
> org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:104)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
> ... 6 more
> Caused by: java.io.IOException: Premature EOF from inputStream (read returned
> -1, was trying to read 10083 necessary bytes and 24 extra bytes, successfully
> read 1072
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:737)
> at
>
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505857#comment-15505857 ] Duo Zhang commented on HBASE-15921: --- And now, I prefer that we can implement a simple AsyncTable that only supports get, put and delete in this issue as what [~jurmous] proposed initially. We can build the basic infrastructure of the callable and retry using the async stub at the same time. Then we could see how we can generalize it to all the operations, especially for scan. Thanks. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[
https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505739#comment-15505739
]
Duo Zhang edited comment on HBASE-16648 at 9/20/16 6:20 AM:
https://github.com/Apache9/microbench/tree/master/src/main/java/com/xiaomi/infra/microbench/compute
There is no contention for the first run, and may have some contentions for the
second 10 threads run.
{noformat}
./bin/run.sh -f 1 .*IATest.*
Benchmark Mode Cnt ScoreError Units
CIATest.test thrpt 20 867.720 ± 0.233 ops/s
PIATest.test thrpt 20 1796.693 ± 39.908 ops/s
./bin/run.sh -f 1 -t 10 .*IATest.*
Benchmark Mode Cnt ScoreError Units
CIATest.test thrpt 20 3452.054 ± 44.537 ops/s
PIATest.test thrpt 20 12101.279 ± 7.215 ops/s
{noformat}
The putIfAbsent version of computeIfAbsent is much faster. So I think we should
use the computeIfAbsent in ConcurrentMap instead of ConcurrentHashMap as for us
the creation is usually cheap.
And in fact, it does not recommend to do a heavy operation in the
mappingFunction of computeIfAbsent.. Then I do not see the necessity of
reimplementing this method in ConcurrentHashMap...
was (Author: apache9):
https://github.com/Apache9/microbench/tree/master/src/main/java/com/xiaomi/infra/microbench/compute
{noformat}
./bin/run.sh -f 1 .*IATest.*
Benchmark Mode Cnt ScoreError Units
CIATest.test thrpt 20 867.720 ± 0.233 ops/s
PIATest.test thrpt 20 1796.693 ± 39.908 ops/s
./bin/run.sh -f 1 -t 10 .*IATest.*
Benchmark Mode Cnt ScoreError Units
CIATest.test thrpt 20 3452.054 ± 44.537 ops/s
PIATest.test thrpt 20 12101.279 ± 7.215 ops/s
{noformat}
The putIfAbsent version of computeIfAbsent is much faster. So I think we should
use the computeIfAbsent in ConcurrentMap instead of ConcurrentHashMap as for us
the creation is usually cheap.
And in fact, it does not recommend to do a heavy operation in the
mappingFunction of computeIfAbsent.. Then I do not see the necessity of
reimplementing this method in ConcurrentHashMap...
> [JDK8] Use computeIfAbsent instead of get and putIfAbsent
> -
>
> Key: HBASE-16648
> URL: https://issues.apache.org/jira/browse/HBASE-16648
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-16648-v1.patch, HBASE-16648-v2.patch,
> HBASE-16648.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505820#comment-15505820 ] Ashish Singhi commented on HBASE-16646: --- bq. import java.util.HashMap; Unused import, not sure why checkstyle didn't report it. Can be addressed on commit if there is no other review comments. > Enhance LoadIncrementalHFiles API to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16646.v0.txt, 16646.v1.txt, 16646.v2.txt, 16646.v3.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505924#comment-15505924 ] Phil Yang commented on HBASE-15921: --- We have ClientAsyncPrefetchScanner which use a background thread loading cache again and again. Maybe it is enough for the first version of AsyncTable? > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505949#comment-15505949 ] Heng Chen commented on HBASE-15921: --- As for unified "sync scan" with AP, we need to pass AP into ClientScanner, and use AP to submit callable then wait until results come back. The most work is around unify ScannerCallableWithReplicas with AP i think. As for 'async scan', i have not think about it carefully, maybe return one Future Object for each call "next()" ? > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16658) Optimize UTF8 string/byte conversions
[ https://issues.apache.org/jira/browse/HBASE-16658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16658: - Attachment: HBASE-16658-master.patch > Optimize UTF8 string/byte conversions > - > > Key: HBASE-16658 > URL: https://issues.apache.org/jira/browse/HBASE-16658 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Priority: Minor > Attachments: HBASE-16658-master.patch > > > String/byte conversions may take either a Charset instance or its canonical > name. One might think a Charset instance would be faster due to avoiding a > lookup and instantiation of a Charset, but it's not. The canonical string > name variants will cache the string encoder/decoder (obtained from a Charset) > resulting in better performance. > See HDFS-10662 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[ https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506090#comment-15506090 ] Duo Zhang commented on HBASE-16648: --- Maybe I should use a new CHM instance for each iteration? Otherwise we will have all the keys in the map already after warm up. Let me see how to implement this logic with JMH. And yes the class will not be re-loaded and the static variable will not be initialized for each iteration. Thanks. > [JDK8] Use computeIfAbsent instead of get and putIfAbsent > - > > Key: HBASE-16648 > URL: https://issues.apache.org/jira/browse/HBASE-16648 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16648-v1.patch, HBASE-16648-v2.patch, > HBASE-16648.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505902#comment-15505902 ] Anoop Sam John commented on HBASE-16134: Yes as u said the case is when cell is having tags but the RPC layer dont want to serialize it. So it uses a Codec which tries to strip the tags. The Codec trying to call single write method in Cell to do the write to OS. So it has to have a way to instruct the method that dont write tags. bq.When is this Cell created? It is created at read time only? Can we create a Cell at read-time that is of a type that never includes tags when it serializes? I was exactly thinking in this direction from today morning. :-) But we cannot do that is what I got as ans Cell is created way down the read layer. ie. in HFileScanner. At that layer we have no idea of Codec/ RPC/ User and it should not also. So we can not create a special Cell at this level which knows it should do the Tags truncate while write to OS. The only time the Cell will be asked to seriailze is when it is writing on the RPC to send back to the client? Are there other times? It is at RPC side as well as WAL write time. Now we dont use but I believe we can use this Cell write method while we flush also. (HFileWriter flow). > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16658) Optimize UTF8 string/byte conversions
binlijin created HBASE-16658: Summary: Optimize UTF8 string/byte conversions Key: HBASE-16658 URL: https://issues.apache.org/jira/browse/HBASE-16658 Project: HBase Issue Type: Improvement Reporter: binlijin Priority: Minor String/byte conversions may take either a Charset instance or its canonical name. One might think a Charset instance would be faster due to avoiding a lookup and instantiation of a Charset, but it's not. The canonical string name variants will cache the string encoder/decoder (obtained from a Charset) resulting in better performance. See HDFS-10662 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16578) Mob data loss after mob compaction and normal compcation
[ https://issues.apache.org/jira/browse/HBASE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505742#comment-15505742 ] Jingcheng Du commented on HBASE-16578: -- Sure, I will assign this JIRA to myself. Thanks a lot for the finding! > Mob data loss after mob compaction and normal compcation > > > Key: HBASE-16578 > URL: https://issues.apache.org/jira/browse/HBASE-16578 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: TestMobCompaction.java, TestMobCompaction.java > > > I have a unittest case, which could explore the mob data loss issue. The root > cause is that with the following line: > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactor.java#L625 > It will make the old mob reference cell win during compaction. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16578) Mob data loss after mob compaction and normal compcation
[
https://issues.apache.org/jira/browse/HBASE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-16578:
-
Description:
StoreFileScanners on MOB cells rely on the scannerOrder to find the latest
cells after compaction. The value of scannerOrder is assigned by the order of
maxSeqId of StoreFile, and this maxSeqId is valued only after the reader of the
StoreFile is created.
In {{StoreFileScanner.getScannersForStoreFiles}}, the StoreFiles are sorted
before the readers are created and at that time the maxSeqId for each file is
-1 (the default value). This will lead to a chaos in scanners on MOB cells. We
need to create readers just before the sorting in the method
{{StoreFileScanner.getScannersForStoreFiles}}.
was:
I have a unittest case, which could explore the mob data loss issue. The root
cause is that with the following line:
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactor.java#L625
It will make the old mob reference cell win during compaction.
> Mob data loss after mob compaction and normal compcation
>
>
> Key: HBASE-16578
> URL: https://issues.apache.org/jira/browse/HBASE-16578
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Jingcheng Du
> Attachments: TestMobCompaction.java, TestMobCompaction.java
>
>
> StoreFileScanners on MOB cells rely on the scannerOrder to find the latest
> cells after compaction. The value of scannerOrder is assigned by the order of
> maxSeqId of StoreFile, and this maxSeqId is valued only after the reader of
> the StoreFile is created.
> In {{StoreFileScanner.getScannersForStoreFiles}}, the StoreFiles are sorted
> before the readers are created and at that time the maxSeqId for each file is
> -1 (the default value). This will lead to a chaos in scanners on MOB cells.
> We need to create readers just before the sorting in the method
> {{StoreFileScanner.getScannersForStoreFiles}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505932#comment-15505932 ] Duo Zhang commented on HBASE-15921: --- I think we need to design a new API for async scanner. Maybe something like grpc's StreamObserver? > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16654) Better handle channelInactive and close for netty rpc client
[ https://issues.apache.org/jira/browse/HBASE-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506030#comment-15506030 ] Duo Zhang commented on HBASE-16654: --- [~stack] Mind take a look? Polish on the netty rpc implementation. Thanks. > Better handle channelInactive and close for netty rpc client > > > Key: HBASE-16654 > URL: https://issues.apache.org/jira/browse/HBASE-16654 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16654.patch > > > We should pass the event to the next handler in the pipeline as > channelInactive and close are usually used as the cleanup method of a > ChannelHandler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[ https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506063#comment-15506063 ] Hiroshi Ikeda commented on HBASE-16648: --- I'm not sure about JMH, but I suspect that the classes are not re-loaded and the static variables are not initialized for each iteration, otherwise the classes cannot warm up. Sorry if I speak wide of the mark. > [JDK8] Use computeIfAbsent instead of get and putIfAbsent > - > > Key: HBASE-16648 > URL: https://issues.apache.org/jira/browse/HBASE-16648 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16648-v1.patch, HBASE-16648-v2.patch, > HBASE-16648.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506214#comment-15506214 ] Duo Zhang commented on HBASE-15921: --- OK, we need HTable to read meta table when implementing RegionLocator. Dead lock... And the location of meta table is fetched from zookeeper, so if we want a fully asynchronous client, we also need to use the asynchronous API of zookeeper. Seems the proper way is to write a completely new asynchronous client that share nothing with the current synchronous client? Thanks. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16659) Minor improvement in ReversedRegionScannerImpl
[ https://issues.apache.org/jira/browse/HBASE-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16659: - Attachment: HBASE-16659-master_v2.patch > Minor improvement in ReversedRegionScannerImpl > -- > > Key: HBASE-16659 > URL: https://issues.apache.org/jira/browse/HBASE-16659 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Attachments: HBASE-16659-master.patch, HBASE-16659-master_v2.patch > > > Current in ReversedRegionScannerImpl#nextRow, to get the previous row to > seek, it use KeyValueUtil.createFirstOnRow which will have two copy, the > first copy is copy to a tmp byte[], the second copy is in KeyValue. We can > use CellUtil.createFirstOnRow which will have no copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16658) Optimize UTF8 string/byte conversions
[
https://issues.apache.org/jira/browse/HBASE-16658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506249#comment-15506249
]
Hadoop QA commented on HBASE-16658:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 33s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 41s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 42s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 25s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829358/HBASE-16658-master.patch
|
| JIRA Issue | HBASE-16658 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 2b0857beccbc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b2eac0d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3617/artifact/patchprocess/branch-findbugs-hbase-common-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3617/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3617/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Optimize UTF8 string/byte conversions
> -
>
> Key: HBASE-16658
> URL: https://issues.apache.org/jira/browse/HBASE-16658
> Project:
[jira] [Commented] (HBASE-16652) Figure out performance difference between increment and append
[ https://issues.apache.org/jira/browse/HBASE-16652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506133#comment-15506133 ] Phil Yang commented on HBASE-16652: --- What are the sizes of two WAL.Entry? > Figure out performance difference between increment and append > -- > > Key: HBASE-16652 > URL: https://issues.apache.org/jira/browse/HBASE-16652 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen > Labels: performance > Attachments: 1.diff, HBASE-16652.patch, append.jfr, > append_get_profile.png, increment.jfr, increment_get_profile.png > > > When do performance test in HBASE-16625, i found it has the very big > difference between Append and Increment (append is about 37% faster than > increment). > As [~stack] mentioned in > https://issues.apache.org/jira/browse/HBASE-16610?focusedCommentId=15493166=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15493166, >append and increment has been unified in server-side, and they looks the > same in client-side. > This issue is to figure out why the performance looks different between them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16652) Figure out performance difference between increment and append
[ https://issues.apache.org/jira/browse/HBASE-16652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506121#comment-15506121 ] Heng Chen edited comment on HBASE-16652 at 9/20/16 9:17 AM: add time log for each step in append/increment path (1.diff), and do tests with PE 3 times. steps: 1. delete 'TestTable' 2. run pe for append 3 times 3. delete 'TestTable' 4. run pe for increment 3 times || type || client_time || reckonDeltas || doWALAppend || sync || || a_1 || 2.06ms || 0.06ms || 1.7ms || 1.6ms || || a_2 || 1.81ms || 0.05ms || 1.4ms || 1.3ms || || a_3 || 1.87ms || 0.05ms || 1.5ms || 1.4ms || || i_1 || 2.47ms || 0.05ms || 2.1ms || 2.0ms || || i_2 || 2.39ms || 0.06ms || 2.0ms || 1.9ms || || i_3 || 2.36ms || 0.06ms || 2.0ms || 1.9ms || Notes: it is one operation time in average. It seems the main difference is sync wal?? was (Author: chenheng): add time log for each step in append/increment path (1.diff), and do tests with PE 3 times. steps: 1. delete 'TestTable' 2. run pe for append 3 times 3. delete 'TestTable' 4. run pe for increment 3 times || type || client_time || reckonDeltas || doWALAppend || sync || || a_1 || 2.06ms || 0.06ms || 1.7ms || 1.6ms || || a_2 || 1.81ms || 0.05ms || 1.4ms || 1.3ms || || a_3 || 1.87ms || 0.05ms || 1.5ms || 1.4ms || || i_1 || 2.47ms || 0.05ms || 2.1ms || 2.0ms || || i_2 || 2.39ms || 0.06ms || 2.0ms || 1.9ms || || i_3 || 2.36ms || 0.06ms || 2.0ms || 1.9ms || It seems the main difference is sync wal?? > Figure out performance difference between increment and append > -- > > Key: HBASE-16652 > URL: https://issues.apache.org/jira/browse/HBASE-16652 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen > Labels: performance > Attachments: 1.diff, HBASE-16652.patch, append.jfr, > append_get_profile.png, increment.jfr, increment_get_profile.png > > > When do performance test in HBASE-16625, i found it has the very big > difference between Append and Increment (append is about 37% faster than > increment). > As [~stack] mentioned in > https://issues.apache.org/jira/browse/HBASE-16610?focusedCommentId=15493166=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15493166, >append and increment has been unified in server-side, and they looks the > same in client-side. > This issue is to figure out why the performance looks different between them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16659) Use CellUtil.createFirstOnRow instead of KeyValueUtil.createFirstOnRow in some places.
[ https://issues.apache.org/jira/browse/HBASE-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16659: - Summary: Use CellUtil.createFirstOnRow instead of KeyValueUtil.createFirstOnRow in some places. (was: Replace KeyValueUtil.createFirstOnRow by CellUtil.createFirstOnRow) > Use CellUtil.createFirstOnRow instead of KeyValueUtil.createFirstOnRow in > some places. > --- > > Key: HBASE-16659 > URL: https://issues.apache.org/jira/browse/HBASE-16659 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Attachments: HBASE-16659-master.patch, HBASE-16659-master_v2.patch > > > Current in ReversedRegionScannerImpl#nextRow, to get the previous row to > seek, it use KeyValueUtil.createFirstOnRow which will have two copy, the > first copy is copy to a tmp byte[], the second copy is in KeyValue. We can > use CellUtil.createFirstOnRow which will have no copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16659) Minor improvement in ReversedRegionScannerImpl
binlijin created HBASE-16659: Summary: Minor improvement in ReversedRegionScannerImpl Key: HBASE-16659 URL: https://issues.apache.org/jira/browse/HBASE-16659 Project: HBase Issue Type: Improvement Reporter: binlijin Priority: Minor Current in ReversedRegionScannerImpl#nextRow, to get the previous row to seek, it use KeyValueUtil.createFirstOnRow which will have two copy, the first copy is copy to a tmp byte[], the second copy is in KeyValue. We can use CellUtil.createFirstOnRow which will have no copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16659) Minor improvement in ReversedRegionScannerImpl
[ https://issues.apache.org/jira/browse/HBASE-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506161#comment-15506161 ] binlijin commented on HBASE-16659: -- I find some other places that we can use CellUtil.createFirstOnRow instead of KeyValueUtil.createFirstOnRow, which is more effective. > Minor improvement in ReversedRegionScannerImpl > -- > > Key: HBASE-16659 > URL: https://issues.apache.org/jira/browse/HBASE-16659 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Attachments: HBASE-16659-master.patch > > > Current in ReversedRegionScannerImpl#nextRow, to get the previous row to > seek, it use KeyValueUtil.createFirstOnRow which will have two copy, the > first copy is copy to a tmp byte[], the second copy is in KeyValue. We can > use CellUtil.createFirstOnRow which will have no copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16659) Use CellUtil.createFirstOnRow instead of KeyValueUtil.createFirstOnRow in some places.
[ https://issues.apache.org/jira/browse/HBASE-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16659: - Description: CellUtil.createFirstOnRow is more effective than KeyValueUtil.createFirstOnRow, because there is no byte[] copy. Current in ReversedRegionScannerImpl#nextRow, to get the previous row to seek, it use KeyValueUtil.createFirstOnRow which will have two copy, the first copy is copy to a tmp byte[], the second copy is in KeyValue. We can use CellUtil.createFirstOnRow which will have no copy. was:Current in ReversedRegionScannerImpl#nextRow, to get the previous row to seek, it use KeyValueUtil.createFirstOnRow which will have two copy, the first copy is copy to a tmp byte[], the second copy is in KeyValue. We can use CellUtil.createFirstOnRow which will have no copy. > Use CellUtil.createFirstOnRow instead of KeyValueUtil.createFirstOnRow in > some places. > --- > > Key: HBASE-16659 > URL: https://issues.apache.org/jira/browse/HBASE-16659 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Attachments: HBASE-16659-master.patch, HBASE-16659-master_v2.patch > > > CellUtil.createFirstOnRow is more effective than > KeyValueUtil.createFirstOnRow, because there is no byte[] copy. > Current in ReversedRegionScannerImpl#nextRow, to get the previous row to > seek, it use KeyValueUtil.createFirstOnRow which will have two copy, the > first copy is copy to a tmp byte[], the second copy is in KeyValue. We can > use CellUtil.createFirstOnRow which will have no copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16652) Figure out performance difference between increment and append
[ https://issues.apache.org/jira/browse/HBASE-16652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-16652: -- Attachment: 1.diff add time log for each step in append/increment path, and do tests with PE 3 times. || type || client_time || reckonDeltas || doWALAppend || sync || || a_1 || 2.06ms || 0.06ms || 1.7ms || 1.6ms || || a_2 || 1.81ms || 0.05ms || 1.4ms || 1.3ms || || a_3 || 1.87ms || 0.05ms || 1.5ms || 1.4ms || || i_1 || 2.47ms || 0.05ms || 2.1ms || 2.0ms || || i_2 || 2.39ms || 0.06ms || 2.0ms || 1.9ms || || i_3 || 2.36ms || 0.06ms || 2.0ms || 1.9ms || It seems the main difference is sync wal?? > Figure out performance difference between increment and append > -- > > Key: HBASE-16652 > URL: https://issues.apache.org/jira/browse/HBASE-16652 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen > Labels: performance > Attachments: 1.diff, HBASE-16652.patch, append.jfr, > append_get_profile.png, increment.jfr, increment_get_profile.png > > > When do performance test in HBASE-16625, i found it has the very big > difference between Append and Increment (append is about 37% faster than > increment). > As [~stack] mentioned in > https://issues.apache.org/jira/browse/HBASE-16610?focusedCommentId=15493166=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15493166, >append and increment has been unified in server-side, and they looks the > same in client-side. > This issue is to figure out why the performance looks different between them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16658) Optimize UTF8 string/byte conversions
[ https://issues.apache.org/jira/browse/HBASE-16658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16658: - Status: Patch Available (was: Open) > Optimize UTF8 string/byte conversions > - > > Key: HBASE-16658 > URL: https://issues.apache.org/jira/browse/HBASE-16658 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Attachments: HBASE-16658-master.patch > > > String/byte conversions may take either a Charset instance or its canonical > name. One might think a Charset instance would be faster due to avoiding a > lookup and instantiation of a Charset, but it's not. The canonical string > name variants will cache the string encoder/decoder (obtained from a Charset) > resulting in better performance. > See HDFS-10662 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16659) Use CellUtil.createFirstOnRow instead of KeyValueUtil.createFirstOnRow in some places.
[ https://issues.apache.org/jira/browse/HBASE-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506245#comment-15506245 ] Heng Chen commented on HBASE-16659: --- It seems KeyValueUtil.createFirstOnRow is used in many places, all should be replaced? > Use CellUtil.createFirstOnRow instead of KeyValueUtil.createFirstOnRow in > some places. > --- > > Key: HBASE-16659 > URL: https://issues.apache.org/jira/browse/HBASE-16659 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Attachments: HBASE-16659-master.patch, HBASE-16659-master_v2.patch > > > CellUtil.createFirstOnRow is more effective than > KeyValueUtil.createFirstOnRow, because there is no byte[] copy. > Current in ReversedRegionScannerImpl#nextRow, to get the previous row to > seek, it use KeyValueUtil.createFirstOnRow which will have two copy, the > first copy is copy to a tmp byte[], the second copy is in KeyValue. We can > use CellUtil.createFirstOnRow which will have no copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506124#comment-15506124 ] Duo Zhang commented on HBASE-15921: --- Ah I found that, we need to implement async region locator first... Let me try... > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16652) Figure out performance difference between increment and append
[ https://issues.apache.org/jira/browse/HBASE-16652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506121#comment-15506121 ] Heng Chen edited comment on HBASE-16652 at 9/20/16 9:14 AM: add time log for each step in append/increment path (1.diff), and do tests with PE 3 times. steps: 1. delete 'TestTable' 2. run pe for append 3 times 3. delete 'TestTable' 4. run pe for increment 3 times || type || client_time || reckonDeltas || doWALAppend || sync || || a_1 || 2.06ms || 0.06ms || 1.7ms || 1.6ms || || a_2 || 1.81ms || 0.05ms || 1.4ms || 1.3ms || || a_3 || 1.87ms || 0.05ms || 1.5ms || 1.4ms || || i_1 || 2.47ms || 0.05ms || 2.1ms || 2.0ms || || i_2 || 2.39ms || 0.06ms || 2.0ms || 1.9ms || || i_3 || 2.36ms || 0.06ms || 2.0ms || 1.9ms || It seems the main difference is sync wal?? was (Author: chenheng): add time log for each step in append/increment path, and do tests with PE 3 times. || type || client_time || reckonDeltas || doWALAppend || sync || || a_1 || 2.06ms || 0.06ms || 1.7ms || 1.6ms || || a_2 || 1.81ms || 0.05ms || 1.4ms || 1.3ms || || a_3 || 1.87ms || 0.05ms || 1.5ms || 1.4ms || || i_1 || 2.47ms || 0.05ms || 2.1ms || 2.0ms || || i_2 || 2.39ms || 0.06ms || 2.0ms || 1.9ms || || i_3 || 2.36ms || 0.06ms || 2.0ms || 1.9ms || It seems the main difference is sync wal?? > Figure out performance difference between increment and append > -- > > Key: HBASE-16652 > URL: https://issues.apache.org/jira/browse/HBASE-16652 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen > Labels: performance > Attachments: 1.diff, HBASE-16652.patch, append.jfr, > append_get_profile.png, increment.jfr, increment_get_profile.png > > > When do performance test in HBASE-16625, i found it has the very big > difference between Append and Increment (append is about 37% faster than > increment). > As [~stack] mentioned in > https://issues.apache.org/jira/browse/HBASE-16610?focusedCommentId=15493166=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15493166, >append and increment has been unified in server-side, and they looks the > same in client-side. > This issue is to figure out why the performance looks different between them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16658) Optimize UTF8 string/byte conversions
[ https://issues.apache.org/jira/browse/HBASE-16658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin reassigned HBASE-16658: Assignee: binlijin > Optimize UTF8 string/byte conversions > - > > Key: HBASE-16658 > URL: https://issues.apache.org/jira/browse/HBASE-16658 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Attachments: HBASE-16658-master.patch > > > String/byte conversions may take either a Charset instance or its canonical > name. One might think a Charset instance would be faster due to avoiding a > lookup and instantiation of a Charset, but it's not. The canonical string > name variants will cache the string encoder/decoder (obtained from a Charset) > resulting in better performance. > See HDFS-10662 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16659) Minor improvement in ReversedRegionScannerImpl
[ https://issues.apache.org/jira/browse/HBASE-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin reassigned HBASE-16659: Assignee: binlijin > Minor improvement in ReversedRegionScannerImpl > -- > > Key: HBASE-16659 > URL: https://issues.apache.org/jira/browse/HBASE-16659 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Attachments: HBASE-16659-master.patch > > > Current in ReversedRegionScannerImpl#nextRow, to get the previous row to > seek, it use KeyValueUtil.createFirstOnRow which will have two copy, the > first copy is copy to a tmp byte[], the second copy is in KeyValue. We can > use CellUtil.createFirstOnRow which will have no copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16659) Minor improvement in ReversedRegionScannerImpl
[ https://issues.apache.org/jira/browse/HBASE-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16659: - Status: Patch Available (was: Open) > Minor improvement in ReversedRegionScannerImpl > -- > > Key: HBASE-16659 > URL: https://issues.apache.org/jira/browse/HBASE-16659 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Attachments: HBASE-16659-master.patch > > > Current in ReversedRegionScannerImpl#nextRow, to get the previous row to > seek, it use KeyValueUtil.createFirstOnRow which will have two copy, the > first copy is copy to a tmp byte[], the second copy is in KeyValue. We can > use CellUtil.createFirstOnRow which will have no copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16659) Minor improvement in ReversedRegionScannerImpl
[ https://issues.apache.org/jira/browse/HBASE-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16659: - Attachment: HBASE-16659-master.patch > Minor improvement in ReversedRegionScannerImpl > -- > > Key: HBASE-16659 > URL: https://issues.apache.org/jira/browse/HBASE-16659 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Priority: Minor > Attachments: HBASE-16659-master.patch > > > Current in ReversedRegionScannerImpl#nextRow, to get the previous row to > seek, it use KeyValueUtil.createFirstOnRow which will have two copy, the > first copy is copy to a tmp byte[], the second copy is in KeyValue. We can > use CellUtil.createFirstOnRow which will have no copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16659) Replace KeyValueUtil.createFirstOnRow by CellUtil.createFirstOnRow
[ https://issues.apache.org/jira/browse/HBASE-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16659: - Summary: Replace KeyValueUtil.createFirstOnRow by CellUtil.createFirstOnRow (was: Minor improvement in ReversedRegionScannerImpl) > Replace KeyValueUtil.createFirstOnRow by CellUtil.createFirstOnRow > -- > > Key: HBASE-16659 > URL: https://issues.apache.org/jira/browse/HBASE-16659 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Attachments: HBASE-16659-master.patch, HBASE-16659-master_v2.patch > > > Current in ReversedRegionScannerImpl#nextRow, to get the previous row to > seek, it use KeyValueUtil.createFirstOnRow which will have two copy, the > first copy is copy to a tmp byte[], the second copy is in KeyValue. We can > use CellUtil.createFirstOnRow which will have no copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16660) ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
Abhishek Singh Chouhan created HBASE-16660:
--
Summary: ArrayIndexOutOfBounds during the majorCompactionCheck in
DateTieredCompaction
Key: HBASE-16660
URL: https://issues.apache.org/jira/browse/HBASE-16660
Project: HBase
Issue Type: Bug
Components: Compaction
Affects Versions: 0.98.20
Reporter: Abhishek Singh Chouhan
Assignee: Abhishek Singh Chouhan
Fix For: 2.0.0, 1.4.0, 0.98.23
We get an ArrayIndexOutOfBoundsException during the major compaction check as
follows
{noformat}
2016-09-19 05:04:18,287 ERROR [20.compactionChecker]
regionserver.HRegionServer$CompactionChecker - Caught exception
java.lang.ArrayIndexOutOfBoundsException: -2
at
org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.shouldPerformMajorCompaction(DateTieredCompactionPolicy.java:159)
at
org.apache.hadoop.hbase.regionserver.HStore.isMajorCompaction(HStore.java:1412)
at
org.apache.hadoop.hbase.regionserver.HRegionServer$CompactionChecker.chore(HRegionServer.java:1532)
at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
at java.lang.Thread.run(Thread.java:745)
{noformat}
This happens due to the following lines in
org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.selectMajorCompaction
{noformat}
int lowerWindowIndex = Collections.binarySearch(boundaries,
minTimestamp == null ? Long.MAX_VALUE : file.getMinimumTimestamp());
int upperWindowIndex = Collections.binarySearch(boundaries,
file.getMaximumTimestamp() == null ? Long.MAX_VALUE :
file.getMaximumTimestamp());
{noformat}
These return negative values if the element is not found and in the case the
values are equal we get the exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16660) ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
[
https://issues.apache.org/jira/browse/HBASE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506849#comment-15506849
]
Abhishek Singh Chouhan commented on HBASE-16660:
Will put up patch for other branches if the fix looks good. [~apurtell]
> ArrayIndexOutOfBounds during the majorCompactionCheck in DateTieredCompaction
> -
>
> Key: HBASE-16660
> URL: https://issues.apache.org/jira/browse/HBASE-16660
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 0.98.20
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0, 1.4.0, 0.98.23
>
> Attachments: HBASE-16660-0.98.patch
>
>
> We get an ArrayIndexOutOfBoundsException during the major compaction check as
> follows
> {noformat}
> 2016-09-19 05:04:18,287 ERROR [20.compactionChecker]
> regionserver.HRegionServer$CompactionChecker - Caught exception
> java.lang.ArrayIndexOutOfBoundsException: -2
> at
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.shouldPerformMajorCompaction(DateTieredCompactionPolicy.java:159)
> at
> org.apache.hadoop.hbase.regionserver.HStore.isMajorCompaction(HStore.java:1412)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$CompactionChecker.chore(HRegionServer.java:1532)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This happens due to the following lines in
> org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy.selectMajorCompaction
> {noformat}
> int lowerWindowIndex = Collections.binarySearch(boundaries,
> minTimestamp == null ? Long.MAX_VALUE : file.getMinimumTimestamp());
> int upperWindowIndex = Collections.binarySearch(boundaries,
> file.getMaximumTimestamp() == null ? Long.MAX_VALUE :
> file.getMaximumTimestamp());
> {noformat}
> These return negative values if the element is not found and in the case the
> values are equal we get the exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16551) Cleanup SplitLogManager and CatalogJanitor
[ https://issues.apache.org/jira/browse/HBASE-16551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16551: Resolution: Fixed Status: Resolved (was: Patch Available) > Cleanup SplitLogManager and CatalogJanitor > -- > > Key: HBASE-16551 > URL: https://issues.apache.org/jira/browse/HBASE-16551 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16551-v0.patch > > > couple of cleanups around SplitLogManager and CatalogJanitor: > replace all copy-pasted cast in SplitLogManager with one call to an helper > method > remove Server, MasterServices, Stoppable since we call the class in only one > place and is called as master, master, master > reuse MockNoopMasterServices instead of creating the Server/MasterServices > classes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16647) hbck should do offline reference repair before online repair
[
https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508018#comment-15508018
]
Hudson commented on HBASE-16647:
FAILURE: Integrated in Jenkins build HBase-1.4 #420 (See
[https://builds.apache.org/job/HBase-1.4/420/])
HBASE-16647 hbck should do offline reference repair before online repair
(jerryjch: rev abfb20206d59dbbe3d85b2d1ced44b61fd37d58c)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
> hbck should do offline reference repair before online repair
>
>
> Key: HBASE-16647
> URL: https://issues.apache.org/jira/browse/HBASE-16647
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16647-v2.patch, HBASE-16647.patch
>
>
> {noformat}
> hbck
> -fixReferenceFiles Try to offline lingering reference store files
> Metadata Repair shortcuts
> -repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles
> -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps
> -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes
> {noformat}
> Bad reference files prevent the region from coming online.
> If used in the shortcut combination, the reference files should be fixed
> before other online fix.
> I have seen repeated '-repair' did not work because bad reference files
> failed regions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16647) hbck should do offline reference repair before online repair
[
https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508030#comment-15508030
]
Hudson commented on HBASE-16647:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1643 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1643/])
HBASE-16647 hbck should do offline reference repair before online repair
(jerryjch: rev 66821206b85d85fba3012d6bb0d6b98c8ee23cb8)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
> hbck should do offline reference repair before online repair
>
>
> Key: HBASE-16647
> URL: https://issues.apache.org/jira/browse/HBASE-16647
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16647-v2.patch, HBASE-16647.patch
>
>
> {noformat}
> hbck
> -fixReferenceFiles Try to offline lingering reference store files
> Metadata Repair shortcuts
> -repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles
> -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps
> -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes
> {noformat}
> Bad reference files prevent the region from coming online.
> If used in the shortcut combination, the reference files should be fixed
> before other online fix.
> I have seen repeated '-repair' did not work because bad reference files
> failed regions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16658) Optimize UTF8 string/byte conversions
[ https://issues.apache.org/jira/browse/HBASE-16658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508031#comment-15508031 ] Hudson commented on HBASE-16658: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1643 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1643/]) HBASE-16658 Optimize UTF8 string/byte conversions (binlijin) (tedyu: rev 6624c676fe7043a95b47f84db4b21df0728ce359) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java > Optimize UTF8 string/byte conversions > - > > Key: HBASE-16658 > URL: https://issues.apache.org/jira/browse/HBASE-16658 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16658-master.patch > > > String/byte conversions may take either a Charset instance or its canonical > name. One might think a Charset instance would be faster due to avoiding a > lookup and instantiation of a Charset, but it's not. The canonical string > name variants will cache the string encoder/decoder (obtained from a Charset) > resulting in better performance. > See HDFS-10662 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11549) Create a Java client only assembly with dependencies from maven build
[ https://issues.apache.org/jira/browse/HBASE-11549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508039#comment-15508039 ] Jerry He commented on HBASE-11549: -- Given the sharded-client jar, I am closing this one as 'won't fix' > Create a Java client only assembly with dependencies from maven build > - > > Key: HBASE-11549 > URL: https://issues.apache.org/jira/browse/HBASE-11549 > Project: HBase > Issue Type: Task > Components: Client >Affects Versions: 0.99.0, 0.98.3 >Reporter: Jerry He >Assignee: Jerry He > Attachments: HBASE-11549-master.patch > > > The purpose is to create a hbase client only distribution with all its > dependencies. > It is will be something like hbase-client-2.0.0-SNAPSHOT-bin.tar.gz along > side with the overall hbase-2.0.0-SNAPSHOT-bin.tar.gz, but itscontent will be > a subset of the the overall hbase-2.0.0-SNAPSHOT-bin.tar.gz. > Client application of hbase can just build and run against this client > distribution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16540) Scan should do additional validation on start and stop row
[ https://issues.apache.org/jira/browse/HBASE-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-16540: -- Resolution: Fixed Release Note: Scan#setStartRow() and Scan#setStopRow() now validate the argument passed for each row key. If the length of the byte[] passed exceeds Short.MAX_VALUE, an IllegalArgumentException will be thrown. Status: Resolved (was: Patch Available) > Scan should do additional validation on start and stop row > -- > > Key: HBASE-16540 > URL: https://issues.apache.org/jira/browse/HBASE-16540 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Dustin Pho > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16540.002.patch, HBASE-16540.patch > > > Scan.setStartRow() and setStopRow() should validate the byte[] passed to > ensure it meets the criteria for a row key. If the byte[] length is greater > that Short.MAX_VALUE, we should throw an IllegalArgumentException in order to > fast fail and prevent server-side errors being thrown and retried. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11549) Create a Java client only assembly with dependencies from maven build
[ https://issues.apache.org/jira/browse/HBASE-11549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-11549: - Resolution: Won't Fix Status: Resolved (was: Patch Available) > Create a Java client only assembly with dependencies from maven build > - > > Key: HBASE-11549 > URL: https://issues.apache.org/jira/browse/HBASE-11549 > Project: HBase > Issue Type: Task > Components: Client >Affects Versions: 0.99.0, 0.98.3 >Reporter: Jerry He >Assignee: Jerry He > Attachments: HBASE-11549-master.patch > > > The purpose is to create a hbase client only distribution with all its > dependencies. > It is will be something like hbase-client-2.0.0-SNAPSHOT-bin.tar.gz along > side with the overall hbase-2.0.0-SNAPSHOT-bin.tar.gz, but itscontent will be > a subset of the the overall hbase-2.0.0-SNAPSHOT-bin.tar.gz. > Client application of hbase can just build and run against this client > distribution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15638) Shade protobuf
[
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508044#comment-15508044
]
stack commented on HBASE-15638:
---
Ok. Have a plan and patch for Endpoints in sub-task. Has downside of
duplicating .protos; once shaded for core and then non-shaded (for CPEPs).
Other downsides are temporary -- IDE usage is awkward until we go the next step
-- or minor (relatively); e.g. we do extra copy carrying CPEP request/response
in and out of the server.
Next steps are:
* Checkin of shaded protos; need to add build tooling so the generation and
shading are easy-to-do out-of-bound with the build.
* Checkin shaded protobuf source, preferably pb3
* Checkin our small patches to protobuf to support ByteBuffers; this latter is
probably our Ram and Anoop.
Let me make subtasks.
Should make a branch? The subtask issue for endpoints is 14.5MB in size.
Keeping it up is painful because it touches loads of the codebase. I should ask
out on dev list.
> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
> Issue Type: Bug
> Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch,
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch,
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch,
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch,
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch,
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch,
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch,
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs
> expect all buffers to be on-heap byte arrays. It does not have facility for
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact
> frustrates the off-heaping-of-the-write-path project as
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To
> ensure we pick up the patched protobuf always, we need to relocate/shade our
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints --
> which use protobuf Service to describe new API -- a blind relocation/shading
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP)
> in particular. For example, in the Table Interface, to invoke a method on a
> registered CPEP, we have:
> {code} Map
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey,
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while
> preserving our API as is so CPEPs continue to work on the new hbase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16661) Add last major compaction age to per-region metrics
Gary Helmling created HBASE-16661: - Summary: Add last major compaction age to per-region metrics Key: HBASE-16661 URL: https://issues.apache.org/jira/browse/HBASE-16661 Project: HBase Issue Type: Improvement Reporter: Gary Helmling Priority: Minor After HBASE-12859, we can now track the last major compaction timestamp for each region. However, this is only exposed through cluster status reporting and the admin API. We have similar per-region metrics around storefile age, but none that filters on major compaction specifically. Let's add a metric for last major compaction age. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16662) Fix open POODLE vulnerabilities
Ben Lau created HBASE-16662: --- Summary: Fix open POODLE vulnerabilities Key: HBASE-16662 URL: https://issues.apache.org/jira/browse/HBASE-16662 Project: HBase Issue Type: Bug Components: REST, Thrift Reporter: Ben Lau Assignee: Ben Lau We recently found a security issue in our HBase REST servers. The issue is a variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) and is present in the HBase Thrift server as well. It also appears to affect the JMXListener coprocessor. The vulnerabilities probably affect all versions of HBase that have the affected services. (If you don't use the affected services with SSL then this ticket probably doesn't affect you). Included is a patch to fix the known POODLE vulnerabilities in master. Let us know if we missed any. From our end we only personally encountered the HBase REST vulnerability. We do not use the Thrift server or JMXListener coprocessor but discovered those problems after discussing the issue with some of the HBase PMCs. Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure which is more or less the same as one of the fixes in this patch. Hadoop wasn't originally affected by the vulnerability in the SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 which does use that class, so they added a SslSelectChannelConnectorSecure class similar to this patch. Since this class is present in Hadoop 2.7.4+ which hasn't been released yet, we will for now just include our own version instead of depending on the Hadoop version. After the patch is approved for master we can backport as necessary to older versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Lau updated HBASE-16662: Attachment: HBASE-16662-master.patch > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[
https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507567#comment-15507567
]
stack commented on HBASE-16650:
---
Looking at the patch, can we update the description for the metric to explain
we DO NOT include evictions because an hfile was removed. Can do on commit.
Is it right putting code under this check: if
(evictedByEvictionProcess) {
We are setting some flags in this block that are used later. Do we need those
flags when an eviction because an hfile has been removed?
This is ugly but we are already keeping blockcache in a global when we fix
this, we can fix this new addition:
private static LruBlockCache GLOBAL_L1_CACHE_INSTANCE;
Otherwise, +1.
> Wrong usage of BlockCache eviction stat for heap memory tuning
> --
>
> Key: HBASE-16650
> URL: https://issues.apache.org/jira/browse/HBASE-16650
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 1.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16650.patch
>
>
> 1. We use the stat evictedBlocksCount - A block can get evicted because of
> eviction thread due to lack of space or because of removal of an HFile itself
> (After a compaction). We should not consider the latter in the tune decision
> at all. These are actually invalidation of blocks. Should the stat counter
> itself not use this count of evicted blocks? I think yes. This will give
> wrong message to users that there are lot of real eviction happening.
> 2. In case L1+ L2 combined block cache, what we use is the sum of evictions
> from both. But we will be tuning L1 size alone. Eviction count from L2 should
> not affect the tuning of L1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507509#comment-15507509 ] stack commented on HBASE-15921: --- Our current client is bloated, hard to understand, uses too many threads, and is slow. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.demo.patch, HBASE-15921.patch, > HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16264: -- Attachment: HBASE-16264.master.012.patch > Figure how to deal with endpoints and shaded pb > --- > > Key: HBASE-16264 > URL: https://issues.apache.org/jira/browse/HBASE-16264 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, Protobufs >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16264.tactic2.patch, HBASE-16264.master.001.patch, > HBASE-16264.master.002.patch, HBASE-16264.master.003.patch, > HBASE-16264.master.004.patch, HBASE-16264.master.005.patch, > HBASE-16264.master.006.patch, HBASE-16264.master.007.patch, > HBASE-16264.master.008.patch, HBASE-16264.master.009.patch, > HBASE-16264.master.010.patch, HBASE-16264.master.011.patch, > HBASE-16264.master.012.patch > > > Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. > Would be sweet if could make it so all just worked. At worst, come up w/ a > prescription for how to migrate existing CPs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16647) hbck should do offline reference repair before online repair
[
https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-16647:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
2.0.0
Status: Resolved (was: Patch Available)
> hbck should do offline reference repair before online repair
>
>
> Key: HBASE-16647
> URL: https://issues.apache.org/jira/browse/HBASE-16647
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16647-v2.patch, HBASE-16647.patch
>
>
> {noformat}
> hbck
> -fixReferenceFiles Try to offline lingering reference store files
> Metadata Repair shortcuts
> -repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles
> -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps
> -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes
> {noformat}
> Bad reference files prevent the region from coming online.
> If used in the shortcut combination, the reference files should be fixed
> before other online fix.
> I have seen repeated '-repair' did not work because bad reference files
> failed regions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507553#comment-15507553 ] stack commented on HBASE-16264: --- Chatting w/ Sean, the whitespace complaints against generated code cannot (currently) be skipped. Next patch fixes tests and findbugs, etc. > Figure how to deal with endpoints and shaded pb > --- > > Key: HBASE-16264 > URL: https://issues.apache.org/jira/browse/HBASE-16264 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, Protobufs >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16264.tactic2.patch, HBASE-16264.master.001.patch, > HBASE-16264.master.002.patch, HBASE-16264.master.003.patch, > HBASE-16264.master.004.patch, HBASE-16264.master.005.patch, > HBASE-16264.master.006.patch, HBASE-16264.master.007.patch, > HBASE-16264.master.008.patch, HBASE-16264.master.009.patch, > HBASE-16264.master.010.patch, HBASE-16264.master.011.patch > > > Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. > Would be sweet if could make it so all just worked. At worst, come up w/ a > prescription for how to migrate existing CPs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16551) Cleanup SplitLogManager and CatalogJanitor
[ https://issues.apache.org/jira/browse/HBASE-16551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507632#comment-15507632 ] Hudson commented on HBASE-16551: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1642 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1642/]) HBASE-16551 Cleanup SplitLogManager and CatalogJanitor (matteo.bertozzi: rev 181fed44508c9a9eb069b9ef60089c415d5e976a) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZKSplitLogManagerCoordination.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterWalManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/SplitLogManagerCoordination.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java > Cleanup SplitLogManager and CatalogJanitor > -- > > Key: HBASE-16551 > URL: https://issues.apache.org/jira/browse/HBASE-16551 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16551-v0.patch > > > couple of cleanups around SplitLogManager and CatalogJanitor: > replace all copy-pasted cast in SplitLogManager with one call to an helper > method > remove Server, MasterServices, Stoppable since we call the class in only one > place and is called as master, master, master > reuse MockNoopMasterServices instead of creating the Server/MasterServices > classes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507631#comment-15507631 ] Hudson commented on HBASE-16646: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1642 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1642/]) HBASE-16646 Enhance LoadIncrementalHFiles API to accept store file paths (tedyu: rev 22dcce571cbd6d06587f7e4b7fc7bb5787f9bcde) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java > Enhance LoadIncrementalHFiles API to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16646.addendum, 16646.v0.txt, 16646.v1.txt, > 16646.v2.txt, 16646.v3.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16658) Optimize UTF8 string/byte conversions
[ https://issues.apache.org/jira/browse/HBASE-16658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16658: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) Thanks for the patch, binlijin > Optimize UTF8 string/byte conversions > - > > Key: HBASE-16658 > URL: https://issues.apache.org/jira/browse/HBASE-16658 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16658-master.patch > > > String/byte conversions may take either a Charset instance or its canonical > name. One might think a Charset instance would be faster due to avoiding a > lookup and instantiation of a Charset, but it's not. The canonical string > name variants will cache the string encoder/decoder (obtained from a Charset) > resulting in better performance. > See HDFS-10662 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508116#comment-15508116 ] stack commented on HBASE-16264: --- Tried a branch-1.x client against a master server with this patch in place and at least the auth CPEP works. > Figure how to deal with endpoints and shaded pb > --- > > Key: HBASE-16264 > URL: https://issues.apache.org/jira/browse/HBASE-16264 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, Protobufs >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16264.tactic2.patch, HBASE-16264.master.001.patch, > HBASE-16264.master.002.patch, HBASE-16264.master.003.patch, > HBASE-16264.master.004.patch, HBASE-16264.master.005.patch, > HBASE-16264.master.006.patch, HBASE-16264.master.007.patch, > HBASE-16264.master.008.patch, HBASE-16264.master.009.patch, > HBASE-16264.master.010.patch, HBASE-16264.master.011.patch, > HBASE-16264.master.012.patch > > > Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. > Would be sweet if could make it so all just worked. At worst, come up w/ a > prescription for how to migrate existing CPs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508198#comment-15508198 ] stack commented on HBASE-15921: --- Demo patch is interesting. We don't want to build a separate AsyncTable Interface as per the [~jurmous] patch? The demo patch extends Table. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.demo.patch, HBASE-15921.patch, > HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16658) Optimize UTF8 string/byte conversions
[ https://issues.apache.org/jira/browse/HBASE-16658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508200#comment-15508200 ] binlijin commented on HBASE-16658: -- [~te...@apache.org] Thanks very much for the commit. > Optimize UTF8 string/byte conversions > - > > Key: HBASE-16658 > URL: https://issues.apache.org/jira/browse/HBASE-16658 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16658-master.patch > > > String/byte conversions may take either a Charset instance or its canonical > name. One might think a Charset instance would be faster due to avoiding a > lookup and instantiation of a Charset, but it's not. The canonical string > name variants will cache the string encoder/decoder (obtained from a Charset) > resulting in better performance. > See HDFS-10662 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16578) Mob data loss after mob compaction and normal compcation
[
https://issues.apache.org/jira/browse/HBASE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508209#comment-15508209
]
huaxiang sun commented on HBASE-16578:
--
Thanks [~jingcheng...@intel.com]! When I have time, I will probably debug more
about the original testing case to see why it randomly fails.
> Mob data loss after mob compaction and normal compcation
>
>
> Key: HBASE-16578
> URL: https://issues.apache.org/jira/browse/HBASE-16578
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Jingcheng Du
> Attachments: TestMobCompaction.java, TestMobCompaction.java
>
>
> StoreFileScanners on MOB cells rely on the scannerOrder to find the latest
> cells after compaction. The value of scannerOrder is assigned by the order of
> maxSeqId of StoreFile, and this maxSeqId is valued only after the reader of
> the StoreFile is created.
> In {{StoreFileScanner.getScannersForStoreFiles}}, the StoreFiles are sorted
> before the readers are created and at that time the maxSeqId for each file is
> -1 (the default value). This will lead to a chaos in scanners on MOB cells.
> We need to create readers just before the sorting in the method
> {{StoreFileScanner.getScannersForStoreFiles}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[
https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508333#comment-15508333
]
Hadoop QA commented on HBASE-16264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 45s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 160 new or modified
test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 40s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 11s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 30m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
30s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 36s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 58s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 22s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 11s
{color} | {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 14s
{color} | {color:red} hbase-it in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 13s
{color} | {color:red} hbase-rest in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 35s
{color} | {color:red} hbase-spark in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 32s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 32s {color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 32s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 32m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
51s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch has 2296 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 10s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
41m 18s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 51s
{color} | {color:red} hbase-common generated 1 new + 0 unchanged - 1 fixed = 1
total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 8s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[
https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508425#comment-15508425
]
Ted Yu commented on HBASE-16630:
{code}
+Set result = new HashSet<>(bucketCount);
+result.addAll(queue);
{code}
Result is a Set which doesn't preserve the order of items in queue.
Why is the MinMaxPriorityQueue needed ?
> Fragmentation in long running Bucket Cache
> --
>
> Key: HBASE-16630
> URL: https://issues.apache.org/jira/browse/HBASE-16630
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3
>Reporter: deepankar
>Assignee: deepankar
> Attachments: HBASE-16630-v2.patch, HBASE-16630.patch
>
>
> As we are running bucket cache for a long time in our system, we are
> observing cases where some nodes after some time does not fully utilize the
> bucket cache, in some cases it is even worse in the sense they get stuck at a
> value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables
> are configured in-memory for simplicity sake).
> We took a heap dump and analyzed what is happening and saw that is classic
> case of fragmentation, current implementation of BucketCache (mainly
> BucketAllocator) relies on the logic that fullyFreeBuckets are available for
> switching/adjusting cache usage between different bucketSizes . But once a
> compaction / bulkload happens and the blocks are evicted from a bucket size ,
> these are usually evicted from random places of the buckets of a bucketSize
> and thus locking the number of buckets associated with a bucketSize and in
> the worst case of the fragmentation we have seen some bucketSizes with
> occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to
> share with the other bucketSize.
> Currently the existing eviction logic helps in the cases where cache used is
> more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also
> done, the eviction (freeSpace function) will not evict anything and the cache
> utilization will be stuck at that value without any allocations for other
> required sizes.
> The fix for this we came up with is simple that we do deFragmentation (
> compaction) of the bucketSize and thus increasing the occupancy ratio and
> also freeing up the buckets to be fullyFree, this logic itself is not
> complicated as the bucketAllocator takes care of packing the blocks in the
> buckets, we need evict and re-allocate the blocks for all the BucketSizes
> that dont fit the criteria.
> I am attaching an initial patch just to give an idea of what we are thinking
> and I'll improve it based on the comments from the community.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16646: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the reviews. > Enhance LoadIncrementalHFiles API to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16646.addendum, 16646.v0.txt, 16646.v1.txt, > 16646.v2.txt, 16646.v3.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16661) Add last major compaction age to per-region metrics
[ https://issues.apache.org/jira/browse/HBASE-16661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling reassigned HBASE-16661: - Assignee: Dustin Pho > Add last major compaction age to per-region metrics > --- > > Key: HBASE-16661 > URL: https://issues.apache.org/jira/browse/HBASE-16661 > Project: HBase > Issue Type: Improvement >Reporter: Gary Helmling >Assignee: Dustin Pho >Priority: Minor > > After HBASE-12859, we can now track the last major compaction timestamp for > each region. However, this is only exposed through cluster status reporting > and the admin API. > We have similar per-region metrics around storefile age, but none that > filters on major compaction specifically. > Let's add a metric for last major compaction age. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508372#comment-15508372 ] Duo Zhang commented on HBASE-15921: --- If your goal is to implement the async client ASAP because you really need it then I think use sync call to fetch region location is acceptable. And I still think on master we should try to implement a completely asynchronous client on master. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.demo.patch, HBASE-15921.patch, > HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16578) Mob data loss after mob compaction and normal compcation
[
https://issues.apache.org/jira/browse/HBASE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508412#comment-15508412
]
Jingcheng Du commented on HBASE-16578:
--
This issue is for scanners on MOB cells, which includes the normal scans
before/after compaction, even in the compaction. It means the scanners used in
compaction have the same issue. The intermittent failures might be caused by
this.
> Mob data loss after mob compaction and normal compcation
>
>
> Key: HBASE-16578
> URL: https://issues.apache.org/jira/browse/HBASE-16578
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Jingcheng Du
> Attachments: TestMobCompaction.java, TestMobCompaction.java
>
>
> StoreFileScanners on MOB cells rely on the scannerOrder to find the latest
> cells after compaction. The value of scannerOrder is assigned by the order of
> maxSeqId of StoreFile, and this maxSeqId is valued only after the reader of
> the StoreFile is created.
> In {{StoreFileScanner.getScannersForStoreFiles}}, the StoreFiles are sorted
> before the readers are created and at that time the maxSeqId for each file is
> -1 (the default value). This will lead to a chaos in scanners on MOB cells.
> We need to create readers just before the sorting in the method
> {{StoreFileScanner.getScannersForStoreFiles}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] deepankar updated HBASE-16630: -- Attachment: HBASE-16630-v2.patch Attached v2, this patch changes the logic of defragmentation to the one suggested [~vrodionov] and the one used by MemCached where they just choose some slabs and evict them completely the code turned out to be much cleaner and smaller. The only heuristic is that we avoid the buckets that are the only buckets for a BucketSizeInfo and the ones that have blocks that are currently in use and order them by their occupancy ratio. Also added the change suggested [~zjushch] to add the memory type to the eviction. If this logic looks fine to the community I will go ahead and add a unit test to the freeEntireBlocks method > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-16630-v2.patch, HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[ https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508357#comment-15508357 ] Hiroshi Ikeda commented on HBASE-16648: --- That result seems natural. The atomicity of ConcurrentHashMap.computeIfAbsent is useful for, for example, establishing network connections, which are not easily excessively executed and immediately discarded. The atomicity apparently requires non-trivial logic and overhead but is indispensable for some cases, excluding just creating light objects. > [JDK8] Use computeIfAbsent instead of get and putIfAbsent > - > > Key: HBASE-16648 > URL: https://issues.apache.org/jira/browse/HBASE-16648 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16648-v1.patch, HBASE-16648-v2.patch, > HBASE-16648.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16661) Add last major compaction age to per-region metrics
[ https://issues.apache.org/jira/browse/HBASE-16661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Pho updated HBASE-16661: --- Status: Patch Available (was: Open) > Add last major compaction age to per-region metrics > --- > > Key: HBASE-16661 > URL: https://issues.apache.org/jira/browse/HBASE-16661 > Project: HBase > Issue Type: Improvement >Reporter: Gary Helmling >Assignee: Dustin Pho >Priority: Minor > Attachments: HBASE-16661.001.patch > > > After HBASE-12859, we can now track the last major compaction timestamp for > each region. However, this is only exposed through cluster status reporting > and the admin API. > We have similar per-region metrics around storefile age, but none that > filters on major compaction specifically. > Let's add a metric for last major compaction age. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16661) Add last major compaction age to per-region metrics
[ https://issues.apache.org/jira/browse/HBASE-16661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Pho updated HBASE-16661: --- Attachment: HBASE-16661.001.patch Added LastMajorCompactionAge to the per-region metrics > Add last major compaction age to per-region metrics > --- > > Key: HBASE-16661 > URL: https://issues.apache.org/jira/browse/HBASE-16661 > Project: HBase > Issue Type: Improvement >Reporter: Gary Helmling >Assignee: Dustin Pho >Priority: Minor > Attachments: HBASE-16661.001.patch > > > After HBASE-12859, we can now track the last major compaction timestamp for > each region. However, this is only exposed through cluster status reporting > and the admin API. > We have similar per-region metrics around storefile age, but none that > filters on major compaction specifically. > Let's add a metric for last major compaction age. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12949) Scanner can be stuck in infinite loop if the HFile is corrupted
[
https://issues.apache.org/jira/browse/HBASE-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-12949:
-
Attachment: HBASE-12949-master-v3.patch
v3 re-based with current master.
> Scanner can be stuck in infinite loop if the HFile is corrupted
> ---
>
> Key: HBASE-12949
> URL: https://issues.apache.org/jira/browse/HBASE-12949
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.3, 0.98.10
>Reporter: Jerry He
> Attachments: HBASE-12949-master-v2 (1).patch,
> HBASE-12949-master-v2.patch, HBASE-12949-master-v2.patch,
> HBASE-12949-master-v2.patch, HBASE-12949-master-v3.patch,
> HBASE-12949-master.patch
>
>
> We've encountered problem where compaction hangs and never completes.
> After looking into it further, we found that the compaction scanner was stuck
> in a infinite loop. See stack below.
> {noformat}
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:296)
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:257)
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:697)
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekToNextRow(StoreScanner.java:672)
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:529)
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:223)
> {noformat}
> We identified the hfile that seems to be corrupted. Using HFile tool shows
> the following:
> {noformat}
> [biadmin@hdtest009 bin]$ hbase org.apache.hadoop.hbase.io.hfile.HFile -v -k
> -m -f
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> 15/01/23 11:53:17 INFO Configuration.deprecation: hadoop.native.lib is
> deprecated. Instead, use io.native.lib.available
> 15/01/23 11:53:18 INFO util.ChecksumType: Checksum using
> org.apache.hadoop.util.PureJavaCrc32
> 15/01/23 11:53:18 INFO util.ChecksumType: Checksum can use
> org.apache.hadoop.util.PureJavaCrc32C
> 15/01/23 11:53:18 INFO Configuration.deprecation: fs.default.name is
> deprecated. Instead, use fs.defaultFS
> Scanning ->
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> WARNING, previous row is greater then current row
> filename ->
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> previous ->
> \x00/20110203-094231205-79442793-1410161293068203000\x0Aattributes16794406\x00\x00\x01\x00\x00\x00\x00\x00\x00
> current ->
> Exception in thread "main" java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:489)
> at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:347)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:856)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:768)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.scanKeysValues(HFilePrettyPrinter.java:362)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:262)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:220)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.main(HFilePrettyPrinter.java:539)
> at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:802)
> {noformat}
> Turning on Java Assert shows the following:
> {noformat}
> Exception in thread "main" java.lang.AssertionError: Key
> 20110203-094231205-79442793-1410161293068203000/attributes:16794406/1099511627776/Minimum/vlen=15/mvcc=0
> followed by a smaller key //0/Minimum/vlen=0/mvcc=0 in cf attributes
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.checkScanOrder(StoreScanner.java:672)
> {noformat}
> It shows that the hfile seems to be corrupted -- the keys don't seem to be
> right.
> But Scanner is not able to give a meaningful error, but stuck in an infinite
> loop in here:
> {code}
> KeyValueHeap.generalizedSeek()
> while ((scanner = heap.poll()) != null) {
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15984) Given failure to parse a given WAL that was closed cleanly, replay the WAL.
[ https://issues.apache.org/jira/browse/HBASE-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508433#comment-15508433 ] Nick Dimiduk commented on HBASE-15984: -- My read of the patch compat guidelines is in line with yours -- adding new metrics is considered compatible for a patch release. I've not reviewed the rest of the patch, but no objection from my on the grounds of metrics additions. > Given failure to parse a given WAL that was closed cleanly, replay the WAL. > --- > > Key: HBASE-15984 > URL: https://issues.apache.org/jira/browse/HBASE-15984 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.0.4, 1.4.0, 1.3.1, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-15984.1.patch, HBASE-15984.2.patch > > > subtask for a general work around for "underlying reader failed / is in a bad > state" just for the case where a WAL 1) was closed cleanly and 2) we can tell > that our current offset ought not be the end of parseable entries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[ https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508439#comment-15508439 ] Duo Zhang commented on HBASE-16648: --- Seems the critical operation is the get before putIfAbsent, we do not issue a get in the test method of computeIfAbsent. Will be back if I find something. Thanks. > [JDK8] Use computeIfAbsent instead of get and putIfAbsent > - > > Key: HBASE-16648 > URL: https://issues.apache.org/jira/browse/HBASE-16648 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16648-v1.patch, HBASE-16648-v2.patch, > HBASE-16648.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508442#comment-15508442 ] Heng Chen commented on HBASE-15921: --- The patch [~carp84] supplied missed some thing but i think it has been implemented in AP 1. How to deal with batch? For batch puts, how to control traffic? 2. How to deal with replica calls? 3. How to deal with retry and different Exception handling? > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.demo.patch, HBASE-15921.patch, > HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508469#comment-15508469 ] deepankar commented on HBASE-16630: --- I used it predominently for limiting the number of buckets we needed, if use a treeset we might still need to do the copy. > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-16630-v2.patch, HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508516#comment-15508516 ] Duo Zhang commented on HBASE-15921: --- As said above, with the non-blocking rpc client, the logic will be completely different as we do not need an extra thread pool to execute blocking rpc call. So even if it is another AsyncProcess, we still need to rewrite most of the code... > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.demo.patch, HBASE-15921.patch, > HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-7912) HBase Backup/Restore Based on HBase Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504659#comment-15504659 ] Frank Welsch edited comment on HBASE-7912 at 9/21/16 3:18 AM: -- I am attaching revised doc of the feature. The revisions are for the following: --correct the backupId argument spelling and format on the CLI --to remove the hbase backup cancel functionality from the doc because it's not yet supported --list limitations of the current HBase backup-and-restore functionality --added required property setting in the container-executor.cfg file was (Author: fwelsch): I am attaching revised doc of the feature. The revisions are for the following: --correct the backupId argument spelling and format on the CLI --to remove the hbase backup cancel functionality from the doc because it's not yet supported --list limitations of the current HBase backup-and-restore functionality > HBase Backup/Restore Based on HBase Snapshot > > > Key: HBASE-7912 > URL: https://issues.apache.org/jira/browse/HBASE-7912 > Project: HBase > Issue Type: Sub-task >Reporter: Richard Ding >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: Backup-and-Restore-Apache_19Sep2016.pdf, > Backup-and-Restore-Apache_9Sep2016.pdf, HBaseBackupAndRestore - v0.8.pdf, > HBaseBackupAndRestore -0.91.pdf, HBaseBackupAndRestore-v0.9.pdf, > HBaseBackupAndRestore.pdf, HBaseBackupRestore-Jira-7912-DesignDoc-v1.pdf, > HBaseBackupRestore-Jira-7912-DesignDoc-v2.pdf, > HBaseBackupRestore-Jira-7912-v4.pdf, HBaseBackupRestore-Jira-7912-v5 .pdf, > HBaseBackupRestore-Jira-7912-v6.pdf, HBase_BackupRestore-Jira-7912-CLI-v1.pdf > > > Finally, we completed the implementation of our backup/restore solution, and > would like to share with community through this jira. > We are leveraging existing hbase snapshot feature, and provide a general > solution to common users. Our full backup is using snapshot to capture > metadata locally and using exportsnapshot to move data to another cluster; > the incremental backup is using offline-WALplayer to backup HLogs; we also > leverage global distribution rolllog and flush to improve performance; other > added-on values such as convert, merge, progress report, and CLI commands. So > that a common user can backup hbase data without in-depth knowledge of hbase. > Our solution also contains some usability features for enterprise users. > The detail design document and CLI command will be attached in this jira. We > plan to use 10~12 subtasks to share each of the following features, and > document the detail implement in the subtasks: > * *Full Backup* : provide local and remote back/restore for a list of tables > * *offline-WALPlayer* to convert HLog to HFiles offline (for incremental > backup) > * *distributed* Logroll and distributed flush > * Backup *Manifest* and history > * *Incremental* backup: to build on top of full backup as daily/weekly backup > * *Convert* incremental backup WAL files into hfiles > * *Merge* several backup images into one(like merge weekly into monthly) > * *add and remove* table to and from Backup image > * *Cancel* a backup process > * backup progress *status* > * full backup based on *existing snapshot* > *-* > *Below is the original description, to keep here as the history for the > design and discussion back in 2013* > There have been attempts in the past to come up with a viable HBase > backup/restore solution (e.g., HBASE-4618). Recently, there are many > advancements and new features in HBase, for example, FileLink, Snapshot, and > Distributed Barrier Procedure. This is a proposal for a backup/restore > solution that utilizes these new features to achieve better performance and > consistency. > > A common practice of backup and restore in database is to first take full > baseline backup, and then periodically take incremental backup that capture > the changes since the full baseline backup. HBase cluster can store massive > amount data. Combination of full backups with incremental backups has > tremendous benefit for HBase as well. The following is a typical scenario > for full and incremental backup. > # The user takes a full backup of a table or a set of tables in HBase. > # The user schedules periodical incremental backups to capture the changes > from the full backup, or from last incremental backup. > # The user needs to restore table data to a past point of time. > # The full backup is restored to the table(s) or to different table name(s). > Then the incremental backups that are up to the desired point in time are > applied on top of the full
[jira] [Created] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
Pankaj Kumar created HBASE-16663:
Summary: JMX ConnectorServer stopped when unauthorized user try to
stop HM/RS/cluster
Key: HBASE-16663
URL: https://issues.apache.org/jira/browse/HBASE-16663
Project: HBase
Issue Type: Bug
Components: metrics
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
After HBASE-16284, unauthorized user will not able allowed to stop
HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
stopped will be stopped before AccessController validation.
{noformat}
hbase.coprocessor.master.classes
org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
hbase.coprocessor.regionserver.classes
org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
{noformat}
{noformat}
2016-09-20 21:12:26,796 INFO
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] hbase.JMXListener:
ConnectorServer stopped!
2016-09-20 21:13:55,380 WARN
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
user P72981
ExitCodeException exitCode=1: id: P72981: No such user
2016-09-20 21:14:00,495 ERROR
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
master.MasterRpcServices: Exception occurred while stopping master
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'P72981' (global, action=ADMIN)
at
org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
at
org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
at
org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
at
org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
{noformat}
{noformat}
2016-09-20 20:59:01,234 INFO
[RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] hbase.JMXListener:
ConnectorServer stopped!
2016-09-20 20:59:01,250 WARN
[RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
user P72981
ExitCodeException exitCode=1: id: P72981: No such user
2016-09-20 20:59:01,253 WARN
[RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
regionserver.HRegionServer: The region server did not stop
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'P72981' (global, action=ADMIN)
at
org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
at
org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
at
org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
at
org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
at
org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
at
org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
{noformat}
{noformat}
2016-09-21 12:09:08,259 INFO
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
master.MasterRpcServices: Client=P72981//10.18.248.96 shutdown
2016-09-21 12:09:08,261 INFO
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] hbase.JMXListener:
ConnectorServer stopped!
2016-09-21 12:09:08,276 WARN
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
user P72981
ExitCodeException exitCode=1: id: P72981: No such user
2016-09-21 12:09:08,280 ERROR
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
master.MasterRpcServices: Exception occurred in HMaster.shutdown()
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'P72981' (global, action=ADMIN)
at
[jira] [Updated] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-16663:
-
Description:
After HBASE-16284, unauthorized user will not able allowed to stop
HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
stopped will be stopped before AccessController validation.
hbase-site.xml,
{noformat}
hbase.coprocessor.master.classes
org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
hbase.coprocessor.regionserver.classes
org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
{noformat}
HBaseAdmin.stopMaster(),
{noformat}
2016-09-20 21:12:26,796 INFO
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] hbase.JMXListener:
ConnectorServer stopped!
2016-09-20 21:13:55,380 WARN
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
user P72981
ExitCodeException exitCode=1: id: P72981: No such user
2016-09-20 21:14:00,495 ERROR
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
master.MasterRpcServices: Exception occurred while stopping master
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'P72981' (global, action=ADMIN)
at
org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
at
org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
at
org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
at
org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
{noformat}
HBaseAdmin.stopRegionServer(rs-host-port),
{noformat}
2016-09-20 20:59:01,234 INFO
[RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] hbase.JMXListener:
ConnectorServer stopped!
2016-09-20 20:59:01,250 WARN
[RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
user P72981
ExitCodeException exitCode=1: id: P72981: No such user
2016-09-20 20:59:01,253 WARN
[RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
regionserver.HRegionServer: The region server did not stop
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'P72981' (global, action=ADMIN)
at
org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
at
org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
at
org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
at
org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
at
org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
at
org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
{noformat}
HBaseAdmin.shutdown(),
{noformat}
2016-09-21 12:09:08,259 INFO
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
master.MasterRpcServices: Client=P72981//10.18.248.96 shutdown
2016-09-21 12:09:08,261 INFO
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] hbase.JMXListener:
ConnectorServer stopped!
2016-09-21 12:09:08,276 WARN
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
user P72981
ExitCodeException exitCode=1: id: P72981: No such user
2016-09-21 12:09:08,280 ERROR
[RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
master.MasterRpcServices: Exception occurred in HMaster.shutdown()
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'P72981' (global, action=ADMIN)
at
org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
at
[jira] [Commented] (HBASE-8065) bulkload can load the hfile into hbase table,but this mechanism can't remove prior data
[ https://issues.apache.org/jira/browse/HBASE-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508710#comment-15508710 ] Rakesh Saini commented on HBASE-8065: - The lock free on callback mechanism will assure the removal of old data with every successful bulk load operation.It also take care of data contention problem raised during read operation. > bulkload can load the hfile into hbase table,but this mechanism can't remove > prior data > --- > > Key: HBASE-8065 > URL: https://issues.apache.org/jira/browse/HBASE-8065 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, mapreduce, regionserver >Affects Versions: 0.94.0 > Environment: hadoop-1.0.2、hbase-0.94.0 >Reporter: Yuan Kang >Assignee: Yuan Kang >Priority: Critical > Attachments: LoadIncrementalHFiles-bulkload-can-clean-olddata.patch > > > this patch can do bulkload for one more parameter ‘need to refresh’,when this > parameter is true.bulkload can clean the old date in the hbase table ,then do > the new date load -- This message was sent by Atlassian JIRA (v6.3.4#6332)
