[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610797#comment-15610797 ] ramkrishna.s.vasudevan commented on HBASE-16890: Just posting as per Stack request to what was done. bq.There is also a means of carrying unsync'd appends from one WAL to another if a WAL goes unresponsive, a feature we'd like to keep (Another 'feature' of asyncwal that we don't have in FSHLog is some size accounting so if N bytes of appends, sync regardlesswe should keep this too, something current FSHLog can't do because it intentionally lets appends and sync run independent of each other). Ok. Agree. bq. Any sense of how many syncs you were aggregating in your tests? Am not sure how to account for this in particular. The JFR reports says most of the time is spent in checksum as part of the sync call in case AsyncWAL but other than that I am not sure specifically how to see that. But I got your intention out here. And one reason what I see there is a speed up in my case is that because of the hack I intentionally come out of wait for some of the syncs than it was doing previously. But that should not be major the factor for the gain here. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: AsyncWAL_disruptor.patch, Screen Shot 2016-10-25 at > 7.34.47 PM.png, Screen Shot 2016-10-25 at 7.39.07 PM.png, Screen Shot > 2016-10-25 at 7.39.48 PM.png, async.svg, classic.svg, contention.png, > contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16947) Some improvements for DumpReplicationQueues tool
[
https://issues.apache.org/jira/browse/HBASE-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610786#comment-15610786
]
Hadoop QA commented on HBASE-16947:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 52s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 59s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 49s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 40s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestReplicaWithCluster |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835466/HBASE-16947-v1.patch |
| JIRA Issue | HBASE-16947 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 5b604931c2b0 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master /
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610784#comment-15610784 ] Duo Zhang commented on HBASE-16890: --- Oh, sorry. The motivation is to reduce the locking time. I think the heaviest operation when holding lock is offer and poll from queue, so I think we can use a lock-free queue and move the offer and poll operations outside the lock. And the ringbuffer sequence could also be used as txid. But I'm not sure if it could perform better than ram's patch as here we still have lots of threads that acquire the lock. In general, if the operation under lock could be run very fast, then the synchronization will perform like a spinlock which does not cause context switch. Thanks. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: AsyncWAL_disruptor.patch, Screen Shot 2016-10-25 at > 7.34.47 PM.png, Screen Shot 2016-10-25 at 7.39.07 PM.png, Screen Shot > 2016-10-25 at 7.39.48 PM.png, async.svg, classic.svg, contention.png, > contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16890: --- Attachment: AsyncWAL_disruptor.patch Creates a disruptor thread and hands over the Payload to the eventloop in the 'onEvent' method of the disruptor. A big hack is there in finishSync() because of the fact that the sequence id created for sync is unique from what was created for append. Corner cases and any even of failure - I need to check them but those are not there in this patch. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: AsyncWAL_disruptor.patch, Screen Shot 2016-10-25 at > 7.34.47 PM.png, Screen Shot 2016-10-25 at 7.39.07 PM.png, Screen Shot > 2016-10-25 at 7.39.48 PM.png, async.svg, classic.svg, contention.png, > contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610759#comment-15610759 ] ramkrishna.s.vasudevan commented on HBASE-16438: Am saying about the seqid discussion in this new cell. There was a discussion above saying if the seqid can be kept as state in the new cell or write in the byte[] itself to reduce heap overhead. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610751#comment-15610751 ] stack commented on HBASE-16890: --- Why keep us in suspense? What you thinking sir? > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: Screen Shot 2016-10-25 at 7.34.47 PM.png, Screen Shot > 2016-10-25 at 7.39.07 PM.png, Screen Shot 2016-10-25 at 7.39.48 PM.png, > async.svg, classic.svg, contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15809) Basic Replication WebUI
[ https://issues.apache.org/jira/browse/HBASE-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610748#comment-15610748 ] stack commented on HBASE-15809: --- @appy's tool is looking very nice [~zghaobac] What would you like to see in it though? No harm dumping it in here. > Basic Replication WebUI > --- > > Key: HBASE-15809 > URL: https://issues.apache.org/jira/browse/HBASE-15809 > Project: HBase > Issue Type: New Feature > Components: Replication, UI >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Appy >Priority: Minor > Fix For: 2.0.0, 1.4.0, 0.98.24 > > Attachments: HBASE-15809-v0.patch, HBASE-15809-v0.png, > HBASE-15809-v1.patch > > > At the moment the only way to have some insight on replication from the webui > is looking at zkdump and metrics. > the basic information useful to get started debugging are: peer information > and the view of WALs offsets for each peer. > https://reviews.apache.org/r/47275/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16947) Some improvements for DumpReplicationQueues tool
[ https://issues.apache.org/jira/browse/HBASE-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610746#comment-15610746 ] stack commented on HBASE-16947: --- You've used it on your cluster to your satisfaction? > Some improvements for DumpReplicationQueues tool > > > Key: HBASE-16947 > URL: https://issues.apache.org/jira/browse/HBASE-16947 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-16947-v1.patch, HBASE-16947.patch > > > Recently we met too many replication WALs problem in our production cluster. > We need the DumpReplicationQueues tool to analyze the replication queues info > in zookeeper. So I backport HBASE-16450 to our branch based 0.98 and did some > improvements for it. > 1. Show the dead regionservers under replication/rs znode. When there are too > many WALs under znode, it can't be atomic transferred to new rs znode. So the > dead rs znode will be leaved on zookeeper. > 2. Make a summary about all the queues that belong to peer has been deleted. > 3. Aggregate all regionservers' size of replication queue. Now the > regionserver report ReplicationLoad to master, but there were not a aggregate > metrics for replication. > 4. Show how many WALs which can not found on hdfs. But the reason (WAL Not > Found) need more time to dig. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16947) Some improvements for DumpReplicationQueues tool
[ https://issues.apache.org/jira/browse/HBASE-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610745#comment-15610745 ] stack commented on HBASE-16947: --- Excellent. Thank you [~zghaobac] for working on this. Add release note w/ your sample output in it. > Some improvements for DumpReplicationQueues tool > > > Key: HBASE-16947 > URL: https://issues.apache.org/jira/browse/HBASE-16947 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-16947-v1.patch, HBASE-16947.patch > > > Recently we met too many replication WALs problem in our production cluster. > We need the DumpReplicationQueues tool to analyze the replication queues info > in zookeeper. So I backport HBASE-16450 to our branch based 0.98 and did some > improvements for it. > 1. Show the dead regionservers under replication/rs znode. When there are too > many WALs under znode, it can't be atomic transferred to new rs znode. So the > dead rs znode will be leaved on zookeeper. > 2. Make a summary about all the queues that belong to peer has been deleted. > 3. Aggregate all regionservers' size of replication queue. Now the > regionserver report ReplicationLoad to master, but there were not a aggregate > metrics for replication. > 4. Show how many WALs which can not found on hdfs. But the reason (WAL Not > Found) need more time to dig. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16951) 1.3 assembly scripts don't package hbase-archetypes in the tarball
[ https://issues.apache.org/jira/browse/HBASE-16951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610737#comment-15610737 ] stack commented on HBASE-16951: --- Sorry Mikhail. Up on master, HBASE-16808 Included the generated, shaded java files from HBASE-15638 Shade protobuf in our src assembly added it. I should have backported the bit that adds in archetypes. > 1.3 assembly scripts don't package hbase-archetypes in the tarball > -- > > Key: HBASE-16951 > URL: https://issues.apache.org/jira/browse/HBASE-16951 > Project: HBase > Issue Type: Sub-task > Components: build >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 1.3.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16436) Add the CellChunkMap variant
[ https://issues.apache.org/jira/browse/HBASE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610733#comment-15610733 ] stack commented on HBASE-16436: --- bq. You think this would be costly? I think it will kill throughput but lets test. > Add the CellChunkMap variant > > > Key: HBASE-16436 > URL: https://issues.apache.org/jira/browse/HBASE-16436 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan > > This sub-task is specifically to add the CellChunkMap created by [~anastas] > and [~eshcar] with specific tests and integrate it with the in memory > flush/compaction flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610727#comment-15610727 ] stack commented on HBASE-16417: --- As is said elsewhere, lets see how bad lookup into a CellArrayMap using binary search is first. Agree, first, default policy should be flush all to disk. > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610715#comment-15610715 ] stack commented on HBASE-16438: --- bq. This is because tags parts comes after KL, VL, Key and Value. What would length do? Include tags+sequenceid? We have to serialize it all anyways so just use total length as next offset? Let the PoC on structure be done apart form regionservers and regions. Do it out in hbase-common. Keep it easy on yourselves. bq. So I think it is ok if we go with seqid as state only? What does this mean [~ram_krish]? > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610706#comment-15610706 ] Anoop Sam John commented on HBASE-16417: We need in memory merge for - Keep the tail of the pipeline with a bigger sized Segment. This is important to avoid small sized HFiles being created at flush. We are flushing only tail of the pipeline in any case now. - To help concurrent reads. When there is only active segment and it is a Map, one seek/read of a particular cell is just a map.get() op. When there are one more segment in pipeline (This is CellArrayMap), we will need binary search (Ya it is not linear search as in case of HFile blocks) to reach to the cell. When there are so many segments in pipeline, we will need more binary search and so compromise on the latency of the read op. Doing in btw merges of segments in pipeline reduce its number and so helps latency. Ya #2 seems valid and merge is the only way. But for #1 merge is not a mandatory one. So IMO flush only tail of whole of the segments (pipeline + active) is not directly related to merge. Even if we are having merge, there is no issue in flushing whole of the segments. Now we have merge with every in memory flush means we are trying keeping only one segment in pipeline. But the policy as discussed here, is going to change that. Doing merge every time with in memory flush is very costly. All agree to that. This means we will have 3+ segments in pipeline. There is still issue of we being flushing smaller sized files. So IMHO, we must flush whole to disk. In case of index merge, where we know there are no cell duplicates and so we avoid data compaction, there is no point at all to delay the flush of the other segments in pipeline. In case of data compaction ya it make sense. On the data compaction use case of Y, I have some Qs. Is it increment way? Or they are put ops but many duplicated cells comes in? > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610687#comment-15610687 ] ramkrishna.s.vasudevan commented on HBASE-16438: I think when I was doing the memstore chunk Cell I did not do the sequenceID in the byte array because in write path when the sequence id is assigned we need to write that to a byte[] and I think it is not like we could pass it along with the actual MSLAB copy. So that will be costly and again on read it will be a costly operation. Going with VLong (in case of saving byte[] then it is going to be much costlier). So I think it is ok if we go with seqid as state only? > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16436) Add the CellChunkMap variant
[ https://issues.apache.org/jira/browse/HBASE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610674#comment-15610674 ] Anoop Sam John commented on HBASE-16436: I feel like for scan (user scan) recreating it is ok. We can not avoid that any way. We have all SQM logic and CPs in the scan path which will expect Cells. So Cell object is a must create then. But in memory merge and even flush to disk paths expect Cells now. This we must solve. Every merge and flush bring back the Cell object to heap is some thing really bad. We need some new ways of merge/flush which can work on raw bytes. > Add the CellChunkMap variant > > > Key: HBASE-16436 > URL: https://issues.apache.org/jira/browse/HBASE-16436 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan > > This sub-task is specifically to add the CellChunkMap created by [~anastas] > and [~eshcar] with specific tests and integrate it with the in memory > flush/compaction flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610669#comment-15610669 ] Anoop Sam John commented on HBASE-16438: Describing diff possibilities at that time. We keep ref and offset means we can not keep whole of the index meta in off heap chunk. chunkid+ offset we can easily keep as both are ints and we can keep ints in offheap BB. But the ref means we need to have a ref array which is on heap. And we will need another data structure (chunk only) to keep offset + length. Yes we can avoid length when there are no tags. But when tags are there, we need to keep length :-(. This is because tags parts comes after KL, VL, Key and Value. Also missed one more thing at that time of discuss. ie. we need to keep the seqId 8 bytes).. When cells in CSLM, it is a long state on the object. Other way would be that when we copy Cell to MSLAB (or to a temp byte[]) at the time of addition to Memstore, we need to keep the seqId not as a state in obj. But that should be put as the last 8 bytes of the cell data bytes. (After key, value and tags). When MSLAB is off heap this will help us to keep more data off heap. We need a diff version of cell which can read seqId correctly from last 8 bytes. We will need decode it. Will need lots of PoC work around diff ideas. All these goes to other jira which says abt ChunkMap variant. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16436) Add the CellChunkMap variant
[ https://issues.apache.org/jira/browse/HBASE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610648#comment-15610648 ] ramkrishna.s.vasudevan commented on HBASE-16436: For scan what you say is true. We need to create instances of cells every time we recreate the cell from the index. You think this would be costly? In our tests we were focussing more on writes and not on reads. Plan is to do some benchmark with reads. Thanks Stack. > Add the CellChunkMap variant > > > Key: HBASE-16436 > URL: https://issues.apache.org/jira/browse/HBASE-16436 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan > > This sub-task is specifically to add the CellChunkMap created by [~anastas] > and [~eshcar] with specific tests and integrate it with the in memory > flush/compaction flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610643#comment-15610643 ] ramkrishna.s.vasudevan commented on HBASE-16438: bq.We'd write chunk id at the time we copy Cell from RPC Buffer to a MSLAB gotten from MSLAB pool? Yes. That is right. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16950) Print raw stats in the end of procedure performance tools for parsing results from scripts
[ https://issues.apache.org/jira/browse/HBASE-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610629#comment-15610629 ] Matteo Bertozzi commented on HBASE-16950: - +1 > Print raw stats in the end of procedure performance tools for parsing results > from scripts > -- > > Key: HBASE-16950 > URL: https://issues.apache.org/jira/browse/HBASE-16950 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Trivial > Attachments: HBASE-16950.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16931) Setting cell's seqId to zero in compaction flow might cause RS down.
[
https://issues.apache.org/jira/browse/HBASE-16931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610615#comment-15610615
]
Yu Li commented on HBASE-16931:
---
Ping [~mantonov], pending on your decision for branch-1.3 to close this JIRA
boss.
> Setting cell's seqId to zero in compaction flow might cause RS down.
>
>
> Key: HBASE-16931
> URL: https://issues.apache.org/jira/browse/HBASE-16931
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Fix For: 2.0.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: HBASE-16931-master.patch, HBASE-16931.branch-1.patch,
> HBASE-16931.branch-1.v2.patch, HBASE-16931_master_v2.patch,
> HBASE-16931_master_v3.patch, HBASE-16931_master_v4.patch,
> HBASE-16931_master_v5.patch
>
>
> Compactor#performCompaction
> do {
> hasMore = scanner.next(cells, scannerContext);
> // output to writer:
> for (Cell c : cells) {
> if (cleanSeqId && c.getSequenceId() <= smallestReadPoint) {
> CellUtil.setSequenceId(c, 0);
> }
> writer.append(c);
> }
> cells.clear();
> } while (hasMore);
> scanner.next will choose at most "hbase.hstore.compaction.kv.max" kvs, the
> last cell still reference by StoreScanner.prevCell, so if cleanSeqId is
> called when the scanner.next call StoreScanner.checkScanOrder may throw
> exception and cause regionserver down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16886) hbase-client: scanner with reversed=true and small=true gets no result
[
https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610612#comment-15610612
]
Hadoop QA commented on HBASE-16886:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 58s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
19m 12s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 57s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 44s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 32s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.TestGlobalMemStoreSize |
| | org.apache.hadoop.hbase.TestHBaseOnOtherDfsCluster |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
\\
\\
|| Subsystem || Report/Notes ||
|
[jira] [Commented] (HBASE-16949) Fix RAT License complaint about the hbase-protocol-shaded/src/main/patches content
[ https://issues.apache.org/jira/browse/HBASE-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610609#comment-15610609 ] Hudson commented on HBASE-16949: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1862 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1862/]) HBASE-16949 Fix RAT License complaint about the (stack: rev c776b3144de7a000c32230174a298eb08c4fef99) * (edit) hbase-protocol-shaded/README.txt * (edit) hbase-protocol/README.txt * (edit) pom.xml > Fix RAT License complaint about the hbase-protocol-shaded/src/main/patches > content > -- > > Key: HBASE-16949 > URL: https://issues.apache.org/jira/browse/HBASE-16949 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 16949.txt > > > Noticed by @duo zhang over on HBASE-16835. Let me exclude the patches dir > from rat check. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16948) Fix inconsistency between HRegion and Region javadoc on getRowLock
[ https://issues.apache.org/jira/browse/HBASE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610610#comment-15610610 ] Hudson commented on HBASE-16948: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1862 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1862/]) HBASE-16948 Fix inconsistency between HRegion and Region javadoc on (stack: rev 8d9b9dc6b731d84e8db5c7cde86edf39d95827ae) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Region.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Fix inconsistency between HRegion and Region javadoc on getRowLock > -- > > Key: HBASE-16948 > URL: https://issues.apache.org/jira/browse/HBASE-16948 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.4 > > Attachments: 16948.txt > > > [~giacomotaylor] noticed on the tail of HBASE-14465 that we forgot to update > the Region Interface #getLock javadoc so it agreed w/ the changed param > meaning in getRowLock. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16950) Print raw stats in the end of procedure performance tools for parsing results from scripts
[
https://issues.apache.org/jira/browse/HBASE-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610565#comment-15610565
]
Hadoop QA commented on HBASE-16950:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 16s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 21s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 10s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 59s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.util.TestHBaseFsckEncryption |
| | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestMobRestoreFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.util.TestConnectionCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835446/HBASE-16950.master.001.patch
|
| JIRA Issue | HBASE-16950 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux bcb3de8178a0 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 8d9b9dc |
| Default Java |
[jira] [Commented] (HBASE-16886) hbase-client: scanner with reversed=true and small=true gets no result
[
https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610558#comment-15610558
]
Hadoop QA commented on HBASE-16886:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 56s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 22s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 51s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 20s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 132m 21s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat
|
| Timed out junit tests | org.apache.hadoop.hbase.TestZooKeeper |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| |
[jira] [Created] (HBASE-16951) 1.3 assembly scripts don't package hbase-archetypes in the tarball
Mikhail Antonov created HBASE-16951: --- Summary: 1.3 assembly scripts don't package hbase-archetypes in the tarball Key: HBASE-16951 URL: https://issues.apache.org/jira/browse/HBASE-16951 Project: HBase Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Mikhail Antonov Assignee: Mikhail Antonov Fix For: 1.3.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16947) Some improvements for DumpReplicationQueues tool
[
https://issues.apache.org/jira/browse/HBASE-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610544#comment-15610544
]
Hadoop QA commented on HBASE-16947:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
36m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 19s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 141m 58s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| | org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache |
| | org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835444/HBASE-16947.patch |
| JIRA Issue | HBASE-16947 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 2cdf2086ff56 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision
[jira] [Updated] (HBASE-16947) Some improvements for DumpReplicationQueues tool
[ https://issues.apache.org/jira/browse/HBASE-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-16947: --- Attachment: HBASE-16947-v1.patch Attach v1 patch. Rename walNotFoundNum to numWalsNotFound. > Some improvements for DumpReplicationQueues tool > > > Key: HBASE-16947 > URL: https://issues.apache.org/jira/browse/HBASE-16947 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-16947-v1.patch, HBASE-16947.patch > > > Recently we met too many replication WALs problem in our production cluster. > We need the DumpReplicationQueues tool to analyze the replication queues info > in zookeeper. So I backport HBASE-16450 to our branch based 0.98 and did some > improvements for it. > 1. Show the dead regionservers under replication/rs znode. When there are too > many WALs under znode, it can't be atomic transferred to new rs znode. So the > dead rs znode will be leaved on zookeeper. > 2. Make a summary about all the queues that belong to peer has been deleted. > 3. Aggregate all regionservers' size of replication queue. Now the > regionserver report ReplicationLoad to master, but there were not a aggregate > metrics for replication. > 4. Show how many WALs which can not found on hdfs. But the reason (WAL Not > Found) need more time to dig. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15809) Basic Replication WebUI
[ https://issues.apache.org/jira/browse/HBASE-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610519#comment-15610519 ] Guanghao Zhang commented on HBASE-15809: Thanks for your reply. Look forward to your work. :-) > Basic Replication WebUI > --- > > Key: HBASE-15809 > URL: https://issues.apache.org/jira/browse/HBASE-15809 > Project: HBase > Issue Type: New Feature > Components: Replication, UI >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Appy >Priority: Minor > Fix For: 2.0.0, 1.4.0, 0.98.24 > > Attachments: HBASE-15809-v0.patch, HBASE-15809-v0.png, > HBASE-15809-v1.patch > > > At the moment the only way to have some insight on replication from the webui > is looking at zkdump and metrics. > the basic information useful to get started debugging are: peer information > and the view of WALs offsets for each peer. > https://reviews.apache.org/r/47275/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16947) Some improvements for DumpReplicationQueues tool
[
https://issues.apache.org/jira/browse/HBASE-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610499#comment-15610499
]
Ted Yu commented on HBASE-16947:
lgtm
{code}
65private long walNotFoundNum;
{code}
numWalsNotFound would be better name for the variable.
> Some improvements for DumpReplicationQueues tool
>
>
> Key: HBASE-16947
> URL: https://issues.apache.org/jira/browse/HBASE-16947
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-16947.patch
>
>
> Recently we met too many replication WALs problem in our production cluster.
> We need the DumpReplicationQueues tool to analyze the replication queues info
> in zookeeper. So I backport HBASE-16450 to our branch based 0.98 and did some
> improvements for it.
> 1. Show the dead regionservers under replication/rs znode. When there are too
> many WALs under znode, it can't be atomic transferred to new rs znode. So the
> dead rs znode will be leaved on zookeeper.
> 2. Make a summary about all the queues that belong to peer has been deleted.
> 3. Aggregate all regionservers' size of replication queue. Now the
> regionserver report ReplicationLoad to master, but there were not a aggregate
> metrics for replication.
> 4. Show how many WALs which can not found on hdfs. But the reason (WAL Not
> Found) need more time to dig.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15809) Basic Replication WebUI
[ https://issues.apache.org/jira/browse/HBASE-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610465#comment-15610465 ] Appy commented on HBASE-15809: -- Yes, still working on it. Worked on it about 2 weeks ago and finished the frontend. Handling backend wasn't trivial, so stack helped figure out best way. I have to complete that next. Will find sometime in next 2 weeks to get it done. > Basic Replication WebUI > --- > > Key: HBASE-15809 > URL: https://issues.apache.org/jira/browse/HBASE-15809 > Project: HBase > Issue Type: New Feature > Components: Replication, UI >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Appy >Priority: Minor > Fix For: 2.0.0, 1.4.0, 0.98.24 > > Attachments: HBASE-15809-v0.patch, HBASE-15809-v0.png, > HBASE-15809-v1.patch > > > At the moment the only way to have some insight on replication from the webui > is looking at zkdump and metrics. > the basic information useful to get started debugging are: peer information > and the view of WALs offsets for each peer. > https://reviews.apache.org/r/47275/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16948) Fix inconsistency between HRegion and Region javadoc on getRowLock
[ https://issues.apache.org/jira/browse/HBASE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610381#comment-15610381 ] Hudson commented on HBASE-16948: FAILURE: Integrated in Jenkins build HBase-1.4 #497 (See [https://builds.apache.org/job/HBase-1.4/497/]) HBASE-16948 Fix inconsistency between HRegion and Region javadoc on (stack: rev 24a92ed63a2e483d43cf66f220c666c581b33484) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Region.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Fix inconsistency between HRegion and Region javadoc on getRowLock > -- > > Key: HBASE-16948 > URL: https://issues.apache.org/jira/browse/HBASE-16948 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.4 > > Attachments: 16948.txt > > > [~giacomotaylor] noticed on the tail of HBASE-14465 that we forgot to update > the Region Interface #getLock javadoc so it agreed w/ the changed param > meaning in getRowLock. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16886) hbase-client: scanner with reversed=true and small=true gets no result
[ https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huzheng updated HBASE-16886: Attachment: HBASE-16886.v4.branch-1.patch > hbase-client: scanner with reversed=true and small=true gets no result > -- > > Key: HBASE-16886 > URL: https://issues.apache.org/jira/browse/HBASE-16886 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7, 0.98.23 >Reporter: huzheng > Labels: patch > Attachments: 16886.v4.branch-1.patch, HBASE-16886.v0.patch, > HBASE-16886.v1.patch, HBASE-16886.v2.patch, HBASE-16886.v3.patch, > HBASE-16886.v4.0.98.patch, HBASE-16886.v4.branch-1.patch, > HBASE-16886.v4.branch-1.patch, HBASE-16886.v4.branch-1.patch, > HBASE-16886.v4.master.patch, TestReversedSmallScan.java > > > Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all > rowKeys are located in region [d, +oo). using a Reversed Small Scanner will > get no result. > Attached file show this failed test case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610351#comment-15610351 ] Duo Zhang commented on HBASE-16890: --- I have some ideas on reducing the contentions in append and sync by using the RingBuffer in disruptor without introducing a new thread. Will prepare a patch when I have time. We can test it along with ram's patch to see which one is better. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: Screen Shot 2016-10-25 at 7.34.47 PM.png, Screen Shot > 2016-10-25 at 7.39.07 PM.png, Screen Shot 2016-10-25 at 7.39.48 PM.png, > async.svg, classic.svg, contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15809) Basic Replication WebUI
[ https://issues.apache.org/jira/browse/HBASE-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610320#comment-15610320 ] Guanghao Zhang commented on HBASE-15809: [~appy] Do you still work on this? If not, I can pick this up. Recently, we met too many replication WALs problem in our production cluster. This replication info is very useful for our use case. Thanks. > Basic Replication WebUI > --- > > Key: HBASE-15809 > URL: https://issues.apache.org/jira/browse/HBASE-15809 > Project: HBase > Issue Type: New Feature > Components: Replication, UI >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Appy >Priority: Minor > Fix For: 2.0.0, 1.4.0, 0.98.24 > > Attachments: HBASE-15809-v0.patch, HBASE-15809-v0.png, > HBASE-15809-v1.patch > > > At the moment the only way to have some insight on replication from the webui > is looking at zkdump and metrics. > the basic information useful to get started debugging are: peer information > and the view of WALs offsets for each peer. > https://reviews.apache.org/r/47275/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15347) Update CHANGES.txt for 1.3
[ https://issues.apache.org/jira/browse/HBASE-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610318#comment-15610318 ] Hudson commented on HBASE-15347: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #51 (See [https://builds.apache.org/job/HBase-1.3-JDK7/51/]) HBASE-15347 Update CHANGES.txt for 1.3 (antonov: rev b3fed047049418b327fbb1c69f7b42c42ac6e240) * (edit) CHANGES.txt HBASE-15347 update pom.xml files for 1.3 (antonov: rev 505d48ac2ce83bc850ec437d17ff2174aedb5068) * (edit) hbase-shaded/hbase-shaded-client/pom.xml * (edit) hbase-checkstyle/pom.xml * (edit) hbase-server/pom.xml * (edit) hbase-client/pom.xml * (edit) hbase-archetypes/hbase-archetype-builder/pom.xml * (edit) hbase-examples/pom.xml * (edit) hbase-archetypes/hbase-client-project/pom.xml * (edit) hbase-shaded/pom.xml * (edit) hbase-assembly/pom.xml * (edit) hbase-hadoop2-compat/pom.xml * (edit) hbase-thrift/pom.xml * (edit) pom.xml * (edit) hbase-annotations/pom.xml * (edit) hbase-testing-util/pom.xml * (edit) hbase-procedure/pom.xml * (edit) hbase-archetypes/hbase-shaded-client-project/pom.xml * (edit) hbase-prefix-tree/pom.xml * (edit) hbase-external-blockcache/pom.xml * (edit) hbase-it/pom.xml * (edit) hbase-archetypes/pom.xml * (edit) hbase-shell/pom.xml * (edit) hbase-shaded/hbase-shaded-server/pom.xml * (edit) hbase-resource-bundle/pom.xml * (edit) hbase-rest/pom.xml * (edit) hbase-common/pom.xml * (edit) hbase-hadoop-compat/pom.xml * (edit) hbase-protocol/pom.xml HBASE-15347 updated asciidoc for 1.3 (antonov: rev 6cb8a436c3584445f8e87e0251b8174be7d9dc21) * (edit) src/main/asciidoc/_chapters/ops_mgt.adoc * (edit) src/main/asciidoc/_chapters/hbck_in_depth.adoc * (add) src/main/site/resources/images/hbasecon2016-stacked.png * (edit) src/main/asciidoc/_chapters/case_studies.adoc * (edit) src/main/asciidoc/_chapters/thrift_filter_language.adoc * (edit) src/main/asciidoc/_chapters/upgrading.adoc * (edit) src/main/asciidoc/_chapters/appendix_hfile_format.adoc * (edit) src/main/asciidoc/_chapters/external_apis.adoc * (add) src/main/site/resources/images/hbase_logo_with_orca_large.png * (edit) src/main/asciidoc/_chapters/other_info.adoc * (add) src/main/asciidoc/_chapters/spark.adoc * (edit) src/main/asciidoc/_chapters/configuration.adoc * (edit) src/main/asciidoc/_chapters/developer.adoc * (edit) src/main/asciidoc/_chapters/troubleshooting.adoc * (edit) src/main/asciidoc/_chapters/architecture.adoc * (edit) src/main/asciidoc/_chapters/compression.adoc * (edit) src/main/asciidoc/_chapters/rpc.adoc * (edit) src/main/asciidoc/_chapters/cp.adoc * (edit) src/main/asciidoc/_chapters/hbase_history.adoc * (edit) src/main/asciidoc/_chapters/schema_design.adoc * (edit) src/main/asciidoc/_chapters/appendix_acl_matrix.adoc * (edit) src/main/asciidoc/_chapters/preface.adoc * (edit) src/main/asciidoc/_chapters/shell.adoc * (edit) src/main/asciidoc/_chapters/mapreduce.adoc * (edit) src/main/asciidoc/_chapters/getting_started.adoc * (edit) src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc * (edit) src/main/asciidoc/_chapters/faq.adoc * (edit) src/main/asciidoc/_chapters/tracing.adoc * (edit) src/main/asciidoc/_chapters/datamodel.adoc * (edit) src/main/asciidoc/_chapters/hbase_apis.adoc * (add) src/main/asciidoc/_chapters/hbase_mob.adoc * (edit) src/main/asciidoc/_chapters/performance.adoc * (edit) src/main/asciidoc/_chapters/asf.adoc * (add) src/main/asciidoc/_chapters/protobuf.adoc * (edit) src/main/asciidoc/_chapters/ycsb.adoc * (edit) src/main/asciidoc/book.adoc * (edit) src/main/asciidoc/_chapters/unit_testing.adoc * (add) src/main/site/resources/images/hbasecon2016-stack-logo.jpg * (edit) src/main/asciidoc/_chapters/hbase-default.adoc * (edit) src/main/asciidoc/_chapters/security.adoc * (edit) src/main/asciidoc/_chapters/zookeeper.adoc * (edit) src/main/asciidoc/_chapters/community.adoc > Update CHANGES.txt for 1.3 > -- > > Key: HBASE-15347 > URL: https://issues.apache.org/jira/browse/HBASE-15347 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 1.3.0 > > Attachments: HBASE-15347-branch-1.3.v1.patch, > HBASE-15347-v2.branch-1.3.patch > > > Going to post the steps in preparing changes file for 1.3 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610317#comment-15610317 ] Duo Zhang commented on HBASE-16835: --- But I do not think the precommit job could read our document and change the default protoc when building HBase... Is it possible to make use of [protobuf-maven-plugin|https://www.xolstice.org/protobuf-maven-plugin/] when building hbase-protocol-shaded? It can download the protoc artifact(starting from version 2.6.1) from the maven repo. And, any concerns of the patch here? [~stack] Thanks. > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15347) Update CHANGES.txt for 1.3
[ https://issues.apache.org/jira/browse/HBASE-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610300#comment-15610300 ] Hudson commented on HBASE-15347: FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #58 (See [https://builds.apache.org/job/HBase-1.3-JDK8/58/]) HBASE-15347 update pom.xml files for 1.3 (antonov: rev 505d48ac2ce83bc850ec437d17ff2174aedb5068) * (edit) hbase-resource-bundle/pom.xml * (edit) hbase-archetypes/hbase-archetype-builder/pom.xml * (edit) hbase-procedure/pom.xml * (edit) hbase-shaded/pom.xml * (edit) hbase-examples/pom.xml * (edit) hbase-hadoop2-compat/pom.xml * (edit) hbase-it/pom.xml * (edit) hbase-protocol/pom.xml * (edit) hbase-hadoop-compat/pom.xml * (edit) hbase-server/pom.xml * (edit) hbase-annotations/pom.xml * (edit) hbase-archetypes/hbase-client-project/pom.xml * (edit) hbase-testing-util/pom.xml * (edit) hbase-rest/pom.xml * (edit) pom.xml * (edit) hbase-archetypes/hbase-shaded-client-project/pom.xml * (edit) hbase-prefix-tree/pom.xml * (edit) hbase-assembly/pom.xml * (edit) hbase-client/pom.xml * (edit) hbase-shell/pom.xml * (edit) hbase-archetypes/pom.xml * (edit) hbase-checkstyle/pom.xml * (edit) hbase-external-blockcache/pom.xml * (edit) hbase-common/pom.xml * (edit) hbase-thrift/pom.xml * (edit) hbase-shaded/hbase-shaded-client/pom.xml * (edit) hbase-shaded/hbase-shaded-server/pom.xml HBASE-15347 updated asciidoc for 1.3 (antonov: rev 6cb8a436c3584445f8e87e0251b8174be7d9dc21) * (edit) src/main/asciidoc/_chapters/getting_started.adoc * (edit) src/main/asciidoc/_chapters/ops_mgt.adoc * (edit) src/main/asciidoc/_chapters/thrift_filter_language.adoc * (edit) src/main/asciidoc/_chapters/troubleshooting.adoc * (edit) src/main/asciidoc/_chapters/mapreduce.adoc * (edit) src/main/asciidoc/_chapters/faq.adoc * (edit) src/main/asciidoc/_chapters/schema_design.adoc * (edit) src/main/asciidoc/_chapters/ycsb.adoc * (edit) src/main/asciidoc/_chapters/zookeeper.adoc * (edit) src/main/asciidoc/_chapters/unit_testing.adoc * (edit) src/main/asciidoc/_chapters/community.adoc * (edit) src/main/asciidoc/_chapters/security.adoc * (edit) src/main/asciidoc/_chapters/architecture.adoc * (edit) src/main/asciidoc/book.adoc * (edit) src/main/asciidoc/_chapters/hbase_history.adoc * (edit) src/main/asciidoc/_chapters/case_studies.adoc * (edit) src/main/asciidoc/_chapters/hbck_in_depth.adoc * (edit) src/main/asciidoc/_chapters/cp.adoc * (edit) src/main/asciidoc/_chapters/external_apis.adoc * (edit) src/main/asciidoc/_chapters/tracing.adoc * (add) src/main/asciidoc/_chapters/hbase_mob.adoc * (edit) src/main/asciidoc/_chapters/preface.adoc * (edit) src/main/asciidoc/_chapters/developer.adoc * (add) src/main/asciidoc/_chapters/spark.adoc * (edit) src/main/asciidoc/_chapters/appendix_hfile_format.adoc * (add) src/main/site/resources/images/hbasecon2016-stack-logo.jpg * (edit) src/main/asciidoc/_chapters/asf.adoc * (edit) src/main/asciidoc/_chapters/hbase-default.adoc * (edit) src/main/asciidoc/_chapters/hbase_apis.adoc * (edit) src/main/asciidoc/_chapters/performance.adoc * (edit) src/main/asciidoc/_chapters/configuration.adoc * (edit) src/main/asciidoc/_chapters/other_info.adoc * (edit) src/main/asciidoc/_chapters/shell.adoc * (edit) src/main/asciidoc/_chapters/compression.adoc * (add) src/main/asciidoc/_chapters/protobuf.adoc * (edit) src/main/asciidoc/_chapters/rpc.adoc * (add) src/main/site/resources/images/hbasecon2016-stacked.png * (edit) src/main/asciidoc/_chapters/upgrading.adoc * (add) src/main/site/resources/images/hbase_logo_with_orca_large.png * (edit) src/main/asciidoc/_chapters/appendix_acl_matrix.adoc * (edit) src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc * (edit) src/main/asciidoc/_chapters/datamodel.adoc > Update CHANGES.txt for 1.3 > -- > > Key: HBASE-15347 > URL: https://issues.apache.org/jira/browse/HBASE-15347 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 1.3.0 > > Attachments: HBASE-15347-branch-1.3.v1.patch, > HBASE-15347-v2.branch-1.3.patch > > > Going to post the steps in preparing changes file for 1.3 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16886) hbase-client: scanner with reversed=true and small=true gets no result
[ https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16886: --- Attachment: 16886.v4.branch-1.patch Rerun branch-1 patch. > hbase-client: scanner with reversed=true and small=true gets no result > -- > > Key: HBASE-16886 > URL: https://issues.apache.org/jira/browse/HBASE-16886 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7, 0.98.23 >Reporter: huzheng > Labels: patch > Attachments: 16886.v4.branch-1.patch, HBASE-16886.v0.patch, > HBASE-16886.v1.patch, HBASE-16886.v2.patch, HBASE-16886.v3.patch, > HBASE-16886.v4.0.98.patch, HBASE-16886.v4.branch-1.patch, > HBASE-16886.v4.branch-1.patch, HBASE-16886.v4.master.patch, > TestReversedSmallScan.java > > > Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all > rowKeys are located in region [d, +oo). using a Reversed Small Scanner will > get no result. > Attached file show this failed test case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16950) Print raw stats in the end of procedure performance tools for parsing results from scripts
[ https://issues.apache.org/jira/browse/HBASE-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16950: - Summary: Print raw stats in the end of procedure performance tools for parsing results from scripts (was: Print raw stats in the end of proc performance tools for parsing results from scripts) > Print raw stats in the end of procedure performance tools for parsing results > from scripts > -- > > Key: HBASE-16950 > URL: https://issues.apache.org/jira/browse/HBASE-16950 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Trivial > Attachments: HBASE-16950.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16950) Print raw stats in the end of proc performance tools for parsing results from scripts
[ https://issues.apache.org/jira/browse/HBASE-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16950: - Status: Patch Available (was: Open) > Print raw stats in the end of proc performance tools for parsing results from > scripts > - > > Key: HBASE-16950 > URL: https://issues.apache.org/jira/browse/HBASE-16950 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Trivial > Attachments: HBASE-16950.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16950) Print raw stats in the end of proc performance tools for parsing results from scripts
[ https://issues.apache.org/jira/browse/HBASE-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16950: - Attachment: HBASE-16950.master.001.patch > Print raw stats in the end of proc performance tools for parsing results from > scripts > - > > Key: HBASE-16950 > URL: https://issues.apache.org/jira/browse/HBASE-16950 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Trivial > Attachments: HBASE-16950.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16950) Print raw stats in the end of proc performance tools for parsing results from scripts
Appy created HBASE-16950: Summary: Print raw stats in the end of proc performance tools for parsing results from scripts Key: HBASE-16950 URL: https://issues.apache.org/jira/browse/HBASE-16950 Project: HBase Issue Type: Improvement Reporter: Appy Assignee: Appy Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610268#comment-15610268 ] binlijin commented on HBASE-16398: -- We also run this patch on our production, and not have time to check the performance in production, and waiting Enis Soztutar to comment on this, if there are some other committers review it, i can continue finish it. > optimize HRegion computeHDFSBlocksDistribution > -- > > Key: HBASE-16398 > URL: https://issues.apache.org/jira/browse/HBASE-16398 > Project: HBase > Issue Type: Sub-task >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16398.patch > > > First i assume there is no reference and link in a region family's directory. > Without the patch to computeHDFSBlocksDistribution for a region family, there > is 1+2*N rpc call, N is hfile numbers, The first rpc call is to > DistributedFileSystem#listStatus to get hfiles, for every hfile there is two > rpc call DistributedFileSystem#getFileStatus(path) and then > DistributedFileSystem#getFileBlockLocations(status, start, length). > With the patch to computeHDFSBlocksDistribution for a region family, there is > 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and > DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter > filter). > So if there is at least one hfile, with the patch, the rpc call will less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16947) Some improvements for DumpReplicationQueues tool
[
https://issues.apache.org/jira/browse/HBASE-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610262#comment-15610262
]
Guanghao Zhang edited comment on HBASE-16947 at 10/27/16 1:15 AM:
--
A example for our cluster. Skip the host:port info.
{code}
Found 4 deleted queues, run hbck -fixReplication in order to remove the deleted
replication queues
"host","port",1476784747260/70-"host","port",,1476779010928
"host","port",1476784747260/70-"host","port",1470421093464-"host","port",1476773708939-"host","port",1476779010928
"host","port",1476784763605/70-"host","port",1470418208092-"host","port",1476773709589
"host","port",1476784763605/70-"host","port",1476773709589
Found 2 dead regionservers, restart one regionserver to transfer the queues of
dead regionservers
"host","port",1476784747260
"host","port",1476784763605
Dumping all peers's number of WALs in replication queue
PeerId: 80 , sizeOfLogQueue: 1352
PeerId: 30 , sizeOfLogQueue: 202
PeerId: 20 , sizeOfLogQueue: 202
PeerId: 10 , sizeOfLogQueue: 202
PeerId: 70 , sizeOfLogQueue: 31724
PeerId: 17000 , sizeOfLogQueue: 775
PeerId: 50 , sizeOfLogQueue: 4490
Total size of WALs on HDFS: 61.6G
ERROR: There are 120 WALs not found!!!
{code}
was (Author: zghaobac):
A example for our cluster. Skip the host:port info.
{format}
Found 4 deleted queues, run hbck -fixReplication in order to remove the deleted
replication queues
"host","port",1476784747260/70-"host","port",,1476779010928
"host","port",1476784747260/70-"host","port",1470421093464-"host","port",1476773708939-"host","port",1476779010928
"host","port",1476784763605/70-"host","port",1470418208092-"host","port",1476773709589
"host","port",1476784763605/70-"host","port",1476773709589
Found 2 dead regionservers, restart one regionserver to transfer the queues of
dead regionservers
"host","port",1476784747260
"host","port",1476784763605
Dumping all peers's number of WALs in replication queue
PeerId: 80 , sizeOfLogQueue: 1352
PeerId: 30 , sizeOfLogQueue: 202
PeerId: 20 , sizeOfLogQueue: 202
PeerId: 10 , sizeOfLogQueue: 202
PeerId: 70 , sizeOfLogQueue: 31724
PeerId: 17000 , sizeOfLogQueue: 775
PeerId: 50 , sizeOfLogQueue: 4490
Total size of WALs on HDFS: 61.6G
ERROR: There are 120 WALs not found!!!
{format}
> Some improvements for DumpReplicationQueues tool
>
>
> Key: HBASE-16947
> URL: https://issues.apache.org/jira/browse/HBASE-16947
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-16947.patch
>
>
> Recently we met too many replication WALs problem in our production cluster.
> We need the DumpReplicationQueues tool to analyze the replication queues info
> in zookeeper. So I backport HBASE-16450 to our branch based 0.98 and did some
> improvements for it.
> 1. Show the dead regionservers under replication/rs znode. When there are too
> many WALs under znode, it can't be atomic transferred to new rs znode. So the
> dead rs znode will be leaved on zookeeper.
> 2. Make a summary about all the queues that belong to peer has been deleted.
> 3. Aggregate all regionservers' size of replication queue. Now the
> regionserver report ReplicationLoad to master, but there were not a aggregate
> metrics for replication.
> 4. Show how many WALs which can not found on hdfs. But the reason (WAL Not
> Found) need more time to dig.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16947) Some improvements for DumpReplicationQueues tool
[
https://issues.apache.org/jira/browse/HBASE-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610262#comment-15610262
]
Guanghao Zhang commented on HBASE-16947:
A example for our cluster. Skip the host:port info.
{format}
Found 4 deleted queues, run hbck -fixReplication in order to remove the deleted
replication queues
"host","port",1476784747260/70-"host","port",,1476779010928
"host","port",1476784747260/70-"host","port",1470421093464-"host","port",1476773708939-"host","port",1476779010928
"host","port",1476784763605/70-"host","port",1470418208092-"host","port",1476773709589
"host","port",1476784763605/70-"host","port",1476773709589
Found 2 dead regionservers, restart one regionserver to transfer the queues of
dead regionservers
"host","port",1476784747260
"host","port",1476784763605
Dumping all peers's number of WALs in replication queue
PeerId: 80 , sizeOfLogQueue: 1352
PeerId: 30 , sizeOfLogQueue: 202
PeerId: 20 , sizeOfLogQueue: 202
PeerId: 10 , sizeOfLogQueue: 202
PeerId: 70 , sizeOfLogQueue: 31724
PeerId: 17000 , sizeOfLogQueue: 775
PeerId: 50 , sizeOfLogQueue: 4490
Total size of WALs on HDFS: 61.6G
ERROR: There are 120 WALs not found!!!
{format}
> Some improvements for DumpReplicationQueues tool
>
>
> Key: HBASE-16947
> URL: https://issues.apache.org/jira/browse/HBASE-16947
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-16947.patch
>
>
> Recently we met too many replication WALs problem in our production cluster.
> We need the DumpReplicationQueues tool to analyze the replication queues info
> in zookeeper. So I backport HBASE-16450 to our branch based 0.98 and did some
> improvements for it.
> 1. Show the dead regionservers under replication/rs znode. When there are too
> many WALs under znode, it can't be atomic transferred to new rs znode. So the
> dead rs znode will be leaved on zookeeper.
> 2. Make a summary about all the queues that belong to peer has been deleted.
> 3. Aggregate all regionservers' size of replication queue. Now the
> regionserver report ReplicationLoad to master, but there were not a aggregate
> metrics for replication.
> 4. Show how many WALs which can not found on hdfs. But the reason (WAL Not
> Found) need more time to dig.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16947) Some improvements for DumpReplicationQueues tool
[ https://issues.apache.org/jira/browse/HBASE-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-16947: --- Assignee: Guanghao Zhang Affects Version/s: 1.4.0 2.0.0 Status: Patch Available (was: Open) > Some improvements for DumpReplicationQueues tool > > > Key: HBASE-16947 > URL: https://issues.apache.org/jira/browse/HBASE-16947 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-16947.patch > > > Recently we met too many replication WALs problem in our production cluster. > We need the DumpReplicationQueues tool to analyze the replication queues info > in zookeeper. So I backport HBASE-16450 to our branch based 0.98 and did some > improvements for it. > 1. Show the dead regionservers under replication/rs znode. When there are too > many WALs under znode, it can't be atomic transferred to new rs znode. So the > dead rs znode will be leaved on zookeeper. > 2. Make a summary about all the queues that belong to peer has been deleted. > 3. Aggregate all regionservers' size of replication queue. Now the > regionserver report ReplicationLoad to master, but there were not a aggregate > metrics for replication. > 4. Show how many WALs which can not found on hdfs. But the reason (WAL Not > Found) need more time to dig. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16947) Some improvements for DumpReplicationQueues tool
[ https://issues.apache.org/jira/browse/HBASE-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-16947: --- Attachment: HBASE-16947.patch > Some improvements for DumpReplicationQueues tool > > > Key: HBASE-16947 > URL: https://issues.apache.org/jira/browse/HBASE-16947 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Guanghao Zhang > Attachments: HBASE-16947.patch > > > Recently we met too many replication WALs problem in our production cluster. > We need the DumpReplicationQueues tool to analyze the replication queues info > in zookeeper. So I backport HBASE-16450 to our branch based 0.98 and did some > improvements for it. > 1. Show the dead regionservers under replication/rs znode. When there are too > many WALs under znode, it can't be atomic transferred to new rs znode. So the > dead rs znode will be leaved on zookeeper. > 2. Make a summary about all the queues that belong to peer has been deleted. > 3. Aggregate all regionservers' size of replication queue. Now the > regionserver report ReplicationLoad to master, but there were not a aggregate > metrics for replication. > 4. Show how many WALs which can not found on hdfs. But the reason (WAL Not > Found) need more time to dig. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reopened HBASE-16608: --- [~anastas] Reopening. Please add a release note that describes what this issue adds, what the default action is, and what configuration I need to change to change behavior or turn it off. Thanks. You can resolve the issue again when done. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-Final.patch, HBASE-16608-Final.patch, > HBASE-16608-V01.patch, HBASE-16608-V03.patch, HBASE-16608-V04.patch, > HBASE-16608-V08.patch, HBASE-16608-V09.patch, HBASE-16608-V09.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610074#comment-15610074 ] stack commented on HBASE-16417: --- +1 on the @anoop sam john suggestion of first/default policy that flushes all in pipeline+active to the hfile as first cut. > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14921) Inmemory Compaction Optimizations; Segment Structure
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14921: -- Summary: Inmemory Compaction Optimizations; Segment Structure (was: Inmemory Compaction optimizations) > Inmemory Compaction Optimizations; Segment Structure > > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, > HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, > HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, > HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, > HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, > HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, > IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, > MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14921) Inmemory Compaction Optimizations; Segment Structure
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14921: -- Release Note: A long, working issue that discussed Segment formats introducing CellArrayMap (delivered as the patch attached to this issue) and CellChunkMap (to be delivered later in HBASE-16421 but see patch v02 for an embryonic form named CellBlockSerialized); when to copy Segment data (and when not too); and then what to include at flush time (the suffix Segment or all Segments). Designs that evolved as discussion went on are attached. Outstanding issues turned up here, not including a CellChunkMap implementation, are listed below but are to be addressed in follow-ons (See HBASE-16417): 1. The flattening without compaction is causing many small segments in pipeline, and they are not flushed all together. 2. The issue of compaction prediction cost. was: A long, working issue that discussed Segment formats introducing CellArrayMap (delivered as the patch attached to this issue) and CellChunkMap (to be delivered later in HBASE-16421 but see patch v02 for an embryonic form named CellBlockSerialized); when to copy Segment data (and when not too); and then what to include at flush time (the suffix Segment or all Segments). Designs that evolved as discussion went on are attached. Outstanding issues turned up here, not including a CellChunkMap implementation, are listed below but are to be addressed in follow-ons: 1. The flattening without compaction is causing many small segments in pipeline, and they are not flushed all together. 2. The issue of compaction prediction cost. > Inmemory Compaction Optimizations; Segment Structure > > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, > HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, > HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, > HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, > HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, > HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, > IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, > MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14921) Inmemory Compaction optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14921: -- Release Note: A long, working issue that discussed Segment formats introducing CellArrayMap (delivered as the patch attached to this issue) and CellChunkMap (to be delivered later in HBASE-16421 but see patch v02 for an embryonic form named CellBlockSerialized); when to copy Segment data (and when not too); and then what to include at flush time (the suffix Segment or all Segments). Designs that evolved as discussion went on are attached. Outstanding issues turned up here, not including a CellChunkMap implementation, are listed below but are to be addressed in follow-ons: 1. The flattening without compaction is causing many small segments in pipeline, and they are not flushed all together. 2. The issue of compaction prediction cost. > Inmemory Compaction optimizations > - > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, > HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, > HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, > HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, > HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, > HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, > IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, > MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610031#comment-15610031 ] stack commented on HBASE-16438: --- Pertinent [~anoop.hbase] comment copied from HBASE-14921: "Cells having ref to chunk data (byte[] now). Can we make the meta data here as ref + offset ( 8 = 4 = 12 bytes per Cell)..Ya it is 4 bytes more but its ok and better than 40 bytes per cell overhead. We need to mark the Cells created out of copy to MSLAB in a special way so as to retrieve the byte[] ref. I see that CellArrayMap is in under regionserver package in hbase-server. Would be cool to pull this all out and move under hbase-common along w/ tests. Would make dev easier. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16948) Fix inconsistency between HRegion and Region javadoc on getRowLock
[ https://issues.apache.org/jira/browse/HBASE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609996#comment-15609996 ] stack commented on HBASE-16948: --- Fixed white space on commit. Leaving issue open to backport to more branches after 1.3.0RC clears. > Fix inconsistency between HRegion and Region javadoc on getRowLock > -- > > Key: HBASE-16948 > URL: https://issues.apache.org/jira/browse/HBASE-16948 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.4 > > Attachments: 16948.txt > > > [~giacomotaylor] noticed on the tail of HBASE-14465 that we forgot to update > the Region Interface #getLock javadoc so it agreed w/ the changed param > meaning in getRowLock. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16948) Fix inconsistency between HRegion and Region javadoc on getRowLock
[ https://issues.apache.org/jira/browse/HBASE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16948: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.4.0 2.0.0 Status: Resolved (was: Patch Available) Pushed to master and branch-1. > Fix inconsistency between HRegion and Region javadoc on getRowLock > -- > > Key: HBASE-16948 > URL: https://issues.apache.org/jira/browse/HBASE-16948 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.4.0 > > Attachments: 16948.txt > > > [~giacomotaylor] noticed on the tail of HBASE-14465 that we forgot to update > the Region Interface #getLock javadoc so it agreed w/ the changed param > meaning in getRowLock. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16948) Fix inconsistency between HRegion and Region javadoc on getRowLock
[ https://issues.apache.org/jira/browse/HBASE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16948: -- Fix Version/s: 1.2.4 1.3.1 > Fix inconsistency between HRegion and Region javadoc on getRowLock > -- > > Key: HBASE-16948 > URL: https://issues.apache.org/jira/browse/HBASE-16948 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.4 > > Attachments: 16948.txt > > > [~giacomotaylor] noticed on the tail of HBASE-14465 that we forgot to update > the Region Interface #getLock javadoc so it agreed w/ the changed param > meaning in getRowLock. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609965#comment-15609965 ] Thiruvel Thirumoolan commented on HBASE-16398: -- [~aoxiang], Are you still working on this? Do you mind if I pick this up if you aren't actively working on it? Sorry, I ran into this jira only now. I was on a long break and didn't notice this jira. We ran into this problem sometime back at Yahoo! as balancer was taking a really long time to just calculate the block distribution. We pushed the changes described here to all our production clusters may be a quarter back and its much better now both in time taken and number of rpcs to the namenode. I can upload the client side code which lets someone calculate the time taken and rpcs and also verify the new and old implementation for correctness. On an avg with the new implementation took half the time of the old one. > optimize HRegion computeHDFSBlocksDistribution > -- > > Key: HBASE-16398 > URL: https://issues.apache.org/jira/browse/HBASE-16398 > Project: HBase > Issue Type: Sub-task >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16398.patch > > > First i assume there is no reference and link in a region family's directory. > Without the patch to computeHDFSBlocksDistribution for a region family, there > is 1+2*N rpc call, N is hfile numbers, The first rpc call is to > DistributedFileSystem#listStatus to get hfiles, for every hfile there is two > rpc call DistributedFileSystem#getFileStatus(path) and then > DistributedFileSystem#getFileBlockLocations(status, start, length). > With the patch to computeHDFSBlocksDistribution for a region family, there is > 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and > DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter > filter). > So if there is at least one hfile, with the patch, the rpc call will less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16948) Fix inconsistency between HRegion and Region javadoc on getRowLock
[
https://issues.apache.org/jira/browse/HBASE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609937#comment-15609937
]
Hadoop QA commented on HBASE-16948:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 17s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 32s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 123m 45s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestModifyTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835417/16948.txt |
| JIRA Issue | HBASE-16948 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 2698bb36ded4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / cd3dd6e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/4196/artifact/patchprocess/whitespace-eol.txt
|
| unit |
[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16438: -- Description: For CellChunkMap we may need a cell such that the chunk out of which it was created, the id of the chunk be embedded in it so that when doing flattening we can use the chunk id as a meta data. More details will follow once the initial tasks are completed. Why we need to embed the chunkid in the Cell is described by [~anastas] in this remark over in parent issue https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 was:For CellChunkMap we may need a cell such that the chunk out of which it was created, the id of the chunk be embedded in it so that when doing flattening we can use the chunk id as a meta data. More details will follow once the initial tasks are completed. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14465) Backport 'Allow rowlock to be reader/write' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-14465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609903#comment-15609903 ] Andrew Purtell commented on HBASE-14465: We could change the signature and deprecate the old with a warning to make it clear semantics changed. Anyway, I must have missed a comment because now I see above that James says they will address it in Phoenix. > Backport 'Allow rowlock to be reader/write' to branch-1 > --- > > Key: HBASE-14465 > URL: https://issues.apache.org/jira/browse/HBASE-14465 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: stack >Assignee: stack > Fix For: 1.2.0, 1.3.0 > > Attachments: 14465.branch-1.txt, 14465.branch-1.v2.txt, > 14465.branch-1.v2.txt, 14465.branch-1.v2.txt, 14465.branch-1.v3.txt, > 14465.branch-1.v3.txt, 14465.branch-1.v4.txt, 14465.branch-1.v4.txt, > 14465.branch-1.v4.txt, 14465.branch-1.v5.txt, 14465.branch-1.v5.txt, > 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.txt > > > Backport to branch-1. > You want this in 1.2 [~busbey]? Its cleanup and fixes a possible dataloss. On > other hand, its a bit of refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14465) Backport 'Allow rowlock to be reader/write' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-14465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609871#comment-15609871 ] stack commented on HBASE-14465: --- Or rather [~apurtell], looking closer, how we lock changed (support for read/write) so the old API no longer makes sense (you cannot respect the old boolean of whether to wait for the lock -- probably why it was changed). A restore looks awkward. > Backport 'Allow rowlock to be reader/write' to branch-1 > --- > > Key: HBASE-14465 > URL: https://issues.apache.org/jira/browse/HBASE-14465 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: stack >Assignee: stack > Fix For: 1.2.0, 1.3.0 > > Attachments: 14465.branch-1.txt, 14465.branch-1.v2.txt, > 14465.branch-1.v2.txt, 14465.branch-1.v2.txt, 14465.branch-1.v3.txt, > 14465.branch-1.v3.txt, 14465.branch-1.v4.txt, 14465.branch-1.v4.txt, > 14465.branch-1.v4.txt, 14465.branch-1.v5.txt, 14465.branch-1.v5.txt, > 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.txt > > > Backport to branch-1. > You want this in 1.2 [~busbey]? Its cleanup and fixes a possible dataloss. On > other hand, its a bit of refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15347) Update CHANGES.txt for 1.3
[ https://issues.apache.org/jira/browse/HBASE-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609866#comment-15609866 ] Hudson commented on HBASE-15347: FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #57 (See [https://builds.apache.org/job/HBase-1.3-JDK8/57/]) HBASE-15347 Update CHANGES.txt for 1.3 (antonov: rev b3fed047049418b327fbb1c69f7b42c42ac6e240) * (edit) CHANGES.txt > Update CHANGES.txt for 1.3 > -- > > Key: HBASE-15347 > URL: https://issues.apache.org/jira/browse/HBASE-15347 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 1.3.0 > > Attachments: HBASE-15347-branch-1.3.v1.patch, > HBASE-15347-v2.branch-1.3.patch > > > Going to post the steps in preparing changes file for 1.3 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609809#comment-15609809 ] stack commented on HBASE-16835: --- Pushed HBASE-16949 to fix the above. > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16949) Fix RAT License complaint about the hbase-protocol-shaded/src/main/patches content
[ https://issues.apache.org/jira/browse/HBASE-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-16949. --- Resolution: Fixed Assignee: stack Fix Version/s: 2.0.0 Pushed to master. > Fix RAT License complaint about the hbase-protocol-shaded/src/main/patches > content > -- > > Key: HBASE-16949 > URL: https://issues.apache.org/jira/browse/HBASE-16949 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 16949.txt > > > Noticed by @duo zhang over on HBASE-16835. Let me exclude the patches dir > from rat check. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16949) Fix RAT License complaint about the hbase-protocol-shaded/src/main/patches content
[ https://issues.apache.org/jira/browse/HBASE-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16949: -- Attachment: 16949.txt Add to the RAT exlcludes (Also included updates to README in the hbase-protocol* modules making it more clear on protoc versions to use also reported on over in HBASE-16835). > Fix RAT License complaint about the hbase-protocol-shaded/src/main/patches > content > -- > > Key: HBASE-16949 > URL: https://issues.apache.org/jira/browse/HBASE-16949 > Project: HBase > Issue Type: Bug >Reporter: stack > Fix For: 2.0.0 > > Attachments: 16949.txt > > > Noticed by @duo zhang over on HBASE-16835. Let me exclude the patches dir > from rat check. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14465) Backport 'Allow rowlock to be reader/write' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-14465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609728#comment-15609728 ] James Taylor commented on HBASE-14465: -- We'll fix it in Phoenix. And we better do a 4.8.2 too. > Backport 'Allow rowlock to be reader/write' to branch-1 > --- > > Key: HBASE-14465 > URL: https://issues.apache.org/jira/browse/HBASE-14465 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: stack >Assignee: stack > Fix For: 1.2.0, 1.3.0 > > Attachments: 14465.branch-1.txt, 14465.branch-1.v2.txt, > 14465.branch-1.v2.txt, 14465.branch-1.v2.txt, 14465.branch-1.v3.txt, > 14465.branch-1.v3.txt, 14465.branch-1.v4.txt, 14465.branch-1.v4.txt, > 14465.branch-1.v4.txt, 14465.branch-1.v5.txt, 14465.branch-1.v5.txt, > 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.txt > > > Backport to branch-1. > You want this in 1.2 [~busbey]? Its cleanup and fixes a possible dataloss. On > other hand, its a bit of refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14465) Backport 'Allow rowlock to be reader/write' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-14465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609721#comment-15609721 ] Lars Hofhansl edited comment on HBASE-14465 at 10/26/16 9:25 PM: - This is a coprocessor accessible API, so I'd say that we shouldn't have changed it. I didn't review the change so I can't complain, but I would've -1'd based on this. I'd also agree that coopting the same API to mean something _completely_ different is bad - the same boolean meaning taking a _readlock_ now instead of _waiting for a writelock_. In any case, this is water under the bridge now. Let's at least fix the Java Doc. Anything else we can do now? was (Author: lhofhansl): This is a coprocessor accessible API, so I'd say that we shouldn't have changed it. I didn't review the change so I can't complain, but I would've -1'd based on this. I'd also agree that coopting the same API mean something completely different is bad - the same boolean meaning taking a readlock now instead of waiting for a writelock. In any case, this is water under the bridge now. Let's at least fix the Java Doc. Anything else we can do now? > Backport 'Allow rowlock to be reader/write' to branch-1 > --- > > Key: HBASE-14465 > URL: https://issues.apache.org/jira/browse/HBASE-14465 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: stack >Assignee: stack > Fix For: 1.2.0, 1.3.0 > > Attachments: 14465.branch-1.txt, 14465.branch-1.v2.txt, > 14465.branch-1.v2.txt, 14465.branch-1.v2.txt, 14465.branch-1.v3.txt, > 14465.branch-1.v3.txt, 14465.branch-1.v4.txt, 14465.branch-1.v4.txt, > 14465.branch-1.v4.txt, 14465.branch-1.v5.txt, 14465.branch-1.v5.txt, > 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.txt > > > Backport to branch-1. > You want this in 1.2 [~busbey]? Its cleanup and fixes a possible dataloss. On > other hand, its a bit of refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14465) Backport 'Allow rowlock to be reader/write' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-14465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609721#comment-15609721 ] Lars Hofhansl commented on HBASE-14465: --- This is a coprocessor accessible API, so I'd say that we shouldn't have changed it. I didn't review the change so I can't complain, but I would've -1'd based on this. I'd also agree that coopting the same API mean something completely different is bad - the same boolean meaning taking a readlock now instead of waiting for a writelock. In any case, this is water under the bridge now. Let's at least fix the Java Doc. Anything else we can do now? > Backport 'Allow rowlock to be reader/write' to branch-1 > --- > > Key: HBASE-14465 > URL: https://issues.apache.org/jira/browse/HBASE-14465 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: stack >Assignee: stack > Fix For: 1.2.0, 1.3.0 > > Attachments: 14465.branch-1.txt, 14465.branch-1.v2.txt, > 14465.branch-1.v2.txt, 14465.branch-1.v2.txt, 14465.branch-1.v3.txt, > 14465.branch-1.v3.txt, 14465.branch-1.v4.txt, 14465.branch-1.v4.txt, > 14465.branch-1.v4.txt, 14465.branch-1.v5.txt, 14465.branch-1.v5.txt, > 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.txt > > > Backport to branch-1. > You want this in 1.2 [~busbey]? Its cleanup and fixes a possible dataloss. On > other hand, its a bit of refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609678#comment-15609678 ] stack commented on HBASE-16890: --- I think your ringbuffer hack 'proves' that the synchronization is the throughput killer; i.e. you did the sloppy test I was talking of doing above. Good. I suggest that now we take a step back. There are three Qs, a txid accounting and a netty eventloop that can't be blocked when running the asyncwal. There is also a means of carrying unsync'd appends from one WAL to another if a WAL goes unresponsive, a feature we'd like to keep (Another 'feature' of asyncwal that we don't have in FSHLog is some size accounting so if N bytes of appends, sync regardlesswe should keep this too, something current FSHLog can't do because it intentionally lets appends and sync run independent of each other). [~Apache9] has undone a bunch of moving parts. Lets be parsimonious about what we put back. Can you post your 'hack" I'd like to see it. On txid, its the ringbuffer txid in FSHLog -- so syncs get their own -- but in asyncwal, txid is run independently. Any sense of how many syncs you were aggregating in your tests? asyncwal seems to do about half of what ringbuffer was doing for whatever reason. More aggregating is better but not so much as to 'hold' tens of handlers from doing anything else (a problem that comes of Handler threads doing request start to finish rather than SEDA model -- a different issue). > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: Screen Shot 2016-10-25 at 7.34.47 PM.png, Screen Shot > 2016-10-25 at 7.39.07 PM.png, Screen Shot 2016-10-25 at 7.39.48 PM.png, > async.svg, classic.svg, contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14465) Backport 'Allow rowlock to be reader/write' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-14465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609675#comment-15609675 ] James Taylor commented on HBASE-14465: -- Did you consider adding a new method to Region instead of coopting this one? Especially a lock API which is typically used to protect against corruption on updates (as it does with Phoenix, to protect DDL operations from corrupting our system table)? Calling it an "inconvenience" is a bit of an understatement. > Backport 'Allow rowlock to be reader/write' to branch-1 > --- > > Key: HBASE-14465 > URL: https://issues.apache.org/jira/browse/HBASE-14465 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: stack >Assignee: stack > Fix For: 1.2.0, 1.3.0 > > Attachments: 14465.branch-1.txt, 14465.branch-1.v2.txt, > 14465.branch-1.v2.txt, 14465.branch-1.v2.txt, 14465.branch-1.v3.txt, > 14465.branch-1.v3.txt, 14465.branch-1.v4.txt, 14465.branch-1.v4.txt, > 14465.branch-1.v4.txt, 14465.branch-1.v5.txt, 14465.branch-1.v5.txt, > 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.txt > > > Backport to branch-1. > You want this in 1.2 [~busbey]? Its cleanup and fixes a possible dataloss. On > other hand, its a bit of refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16948) Fix inconsistency between HRegion and Region javadoc on getRowLock
[ https://issues.apache.org/jira/browse/HBASE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16948: -- Assignee: stack Status: Patch Available (was: Open) > Fix inconsistency between HRegion and Region javadoc on getRowLock > -- > > Key: HBASE-16948 > URL: https://issues.apache.org/jira/browse/HBASE-16948 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Attachments: 16948.txt > > > [~giacomotaylor] noticed on the tail of HBASE-14465 that we forgot to update > the Region Interface #getLock javadoc so it agreed w/ the changed param > meaning in getRowLock. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16948) Fix inconsistency between HRegion and Region javadoc on getRowLock
[ https://issues.apache.org/jira/browse/HBASE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16948: -- Attachment: 16948.txt Fix for Region#getRowLock javadoc. > Fix inconsistency between HRegion and Region javadoc on getRowLock > -- > > Key: HBASE-16948 > URL: https://issues.apache.org/jira/browse/HBASE-16948 > Project: HBase > Issue Type: Bug >Reporter: stack > Attachments: 16948.txt > > > [~giacomotaylor] noticed on the tail of HBASE-14465 that we forgot to update > the Region Interface #getLock javadoc so it agreed w/ the changed param > meaning in getRowLock. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14465) Backport 'Allow rowlock to be reader/write' to branch-1
[
https://issues.apache.org/jira/browse/HBASE-14465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609589#comment-15609589
]
stack commented on HBASE-14465:
---
Yeah, thats unfortunate that this backport didn't update both Region and
HRegion. Sorry for any inconvenience caused. It looks an ugly one to figure. I
can fix the javadoc issue over in HBASE-16948. This is in a shipping HBase
though so bit tough to retract at this stage [~giacomotaylor]. Region markings
would seem to allow us this change (though this little consolation):
{code}
@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)
@InterfaceStability.Evolving
{code}
> Backport 'Allow rowlock to be reader/write' to branch-1
> ---
>
> Key: HBASE-14465
> URL: https://issues.apache.org/jira/browse/HBASE-14465
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: stack
>Assignee: stack
> Fix For: 1.2.0, 1.3.0
>
> Attachments: 14465.branch-1.txt, 14465.branch-1.v2.txt,
> 14465.branch-1.v2.txt, 14465.branch-1.v2.txt, 14465.branch-1.v3.txt,
> 14465.branch-1.v3.txt, 14465.branch-1.v4.txt, 14465.branch-1.v4.txt,
> 14465.branch-1.v4.txt, 14465.branch-1.v5.txt, 14465.branch-1.v5.txt,
> 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.txt
>
>
> Backport to branch-1.
> You want this in 1.2 [~busbey]? Its cleanup and fixes a possible dataloss. On
> other hand, its a bit of refactoring.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16948) Fix inconsistency between HRegion and Region javadoc on getRowLock
stack created HBASE-16948: - Summary: Fix inconsistency between HRegion and Region javadoc on getRowLock Key: HBASE-16948 URL: https://issues.apache.org/jira/browse/HBASE-16948 Project: HBase Issue Type: Bug Reporter: stack [~giacomotaylor] noticed on the tail of HBASE-14465 that we forgot to update the Region Interface #getLock javadoc so it agreed w/ the changed param meaning in getRowLock. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16886) hbase-client: scanner with reversed=true and small=true gets no result
[ https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609555#comment-15609555 ] Mikhail Antonov commented on HBASE-16886: - (above is assuming this patch isn't considered a blocker, which judging by the priority setting on the jira it it not?) > hbase-client: scanner with reversed=true and small=true gets no result > -- > > Key: HBASE-16886 > URL: https://issues.apache.org/jira/browse/HBASE-16886 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7, 0.98.23 >Reporter: huzheng > Labels: patch > Attachments: HBASE-16886.v0.patch, HBASE-16886.v1.patch, > HBASE-16886.v2.patch, HBASE-16886.v3.patch, HBASE-16886.v4.0.98.patch, > HBASE-16886.v4.branch-1.patch, HBASE-16886.v4.branch-1.patch, > HBASE-16886.v4.master.patch, TestReversedSmallScan.java > > > Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all > rowKeys are located in region [d, +oo). using a Reversed Small Scanner will > get no result. > Attached file show this failed test case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16886) hbase-client: scanner with reversed=true and small=true gets no result
[ https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609551#comment-15609551 ] Mikhail Antonov commented on HBASE-16886: - [~te...@apache.org] I determined commit hash, updated pom.xml files and pushed RC0 tag and CHANGES.txt file. I assume the changes can go in now. If there's no urgency I think it'd be good to hold on on this patch going to 1.3 branch till today evening or tomorrow. We could cherry-pick it a bit later. > hbase-client: scanner with reversed=true and small=true gets no result > -- > > Key: HBASE-16886 > URL: https://issues.apache.org/jira/browse/HBASE-16886 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7, 0.98.23 >Reporter: huzheng > Labels: patch > Attachments: HBASE-16886.v0.patch, HBASE-16886.v1.patch, > HBASE-16886.v2.patch, HBASE-16886.v3.patch, HBASE-16886.v4.0.98.patch, > HBASE-16886.v4.branch-1.patch, HBASE-16886.v4.branch-1.patch, > HBASE-16886.v4.master.patch, TestReversedSmallScan.java > > > Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all > rowKeys are located in region [d, +oo). using a Reversed Small Scanner will > get no result. > Attached file show this failed test case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15347) Update CHANGES.txt for 1.3
[ https://issues.apache.org/jira/browse/HBASE-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609517#comment-15609517 ] Mikhail Antonov commented on HBASE-15347: - updated asciidoc > Update CHANGES.txt for 1.3 > -- > > Key: HBASE-15347 > URL: https://issues.apache.org/jira/browse/HBASE-15347 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 1.3.0 > > Attachments: HBASE-15347-branch-1.3.v1.patch, > HBASE-15347-v2.branch-1.3.patch > > > Going to post the steps in preparing changes file for 1.3 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16851) User-facing documentation for the In-Memory Compaction feature
[ https://issues.apache.org/jira/browse/HBASE-16851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609501#comment-15609501 ] Edward Bortnikov commented on HBASE-16851: -- Great idea, thanks [~stack]. In the meantime, the user doc is still in flux, until the next week. We'll publish a stable version next week. Parts of the doc that deal with the in-memory compaction policy (what segments/when/how to merge) are subject to experimentation in HBASE-16417, and will explicitly indicate that. All in all, the goal is to have a self-tuning policy that takes minimum configuration (if any). Once that task is complete, we'll re-publish the doc. > User-facing documentation for the In-Memory Compaction feature > -- > > Key: HBASE-16851 > URL: https://issues.apache.org/jira/browse/HBASE-16851 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Edward Bortnikov > Attachments: HBaseAcceleratedHbaseConf-final.pptx > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14921) Inmemory Compaction optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609436#comment-15609436 ] stack commented on HBASE-14921: --- For anyone trying to read along later, the CellBlockSerialized is in the second patch posted above only: HBASE-14921-V02.patch > Inmemory Compaction optimizations > - > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, > HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, > HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, > HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, > HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, > HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, > IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, > MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16436) Add the CellChunkMap variant
[ https://issues.apache.org/jira/browse/HBASE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609420#comment-15609420 ] stack commented on HBASE-16436: --- Forgot that Anoop points out length is not needed since can use offsets for that. Found a CellChunkMap implementation here in the SECOND! patch only as CellBlockSerialized: https://issues.apache.org/jira/secure/attachment/12794742/HBASE-14921-V02.patch Is there a reference to the actual Cell instance missing from the CellBlockSerialized implementation? The index item seems to only point at the serialized form of the Cell but for for reading/sorting/scanning the Cell, we'll need an instance (or a smarter scanner, one that can work w/ the serialized form). > Add the CellChunkMap variant > > > Key: HBASE-16436 > URL: https://issues.apache.org/jira/browse/HBASE-16436 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan > > This sub-task is specifically to add the CellChunkMap created by [~anastas] > and [~eshcar] with specific tests and integrate it with the in memory > flush/compaction flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16436) Add the CellChunkMap variant
[ https://issues.apache.org/jira/browse/HBASE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609361#comment-15609361 ] stack commented on HBASE-16436: --- How we going to do the CellChunkMap implementation. Is there a writeup/patch anywhere yet? (I'm digging through the issues and don't see one -- I may have missed a patch). Per Cell entry in the Index, we will have a reference to the Cell, then its chunkid, offset and length? If a Cell's backing array moves from one chunk to another, do we reinstantiate the Cell at this time or just do it lazily on first read? (Or could we Scan Cells in their flat form w/o having to instantiate instances?). Would be cool if CellChunkMap could be done out of hbase-common only easily testable standalone (Should make CellArrayMap do this too). > Add the CellChunkMap variant > > > Key: HBASE-16436 > URL: https://issues.apache.org/jira/browse/HBASE-16436 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan > > This sub-task is specifically to add the CellChunkMap created by [~anastas] > and [~eshcar] with specific tests and integrate it with the in memory > flush/compaction flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15347) Update CHANGES.txt for 1.3
[ https://issues.apache.org/jira/browse/HBASE-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609362#comment-15609362 ] Mikhail Antonov commented on HBASE-15347: - committed changex.txt, removed -SNAPSHOT from poms and pushed RC tag. > Update CHANGES.txt for 1.3 > -- > > Key: HBASE-15347 > URL: https://issues.apache.org/jira/browse/HBASE-15347 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 1.3.0 > > Attachments: HBASE-15347-branch-1.3.v1.patch, > HBASE-15347-v2.branch-1.3.patch > > > Going to post the steps in preparing changes file for 1.3 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609347#comment-15609347 ] stack commented on HBASE-16438: --- Thanks boys for the help [~ram_krish] and [~anoop.hbase] I see it is needed going from CSLS to CellChunkMap but also for CellArrayMap to CellChunkMap if that convertion is needed if we are NOT going to copy (which is a necessary transformation type). We'd write chunk id at the time we copy Cell from RPC Buffer to a MSLAB gotten from MSLAB pool? > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16747) Track memstore data size and heap overhead separately
[
https://issues.apache.org/jira/browse/HBASE-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609254#comment-15609254
]
Hadoop QA commented on HBASE-16747:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 13s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 38s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 12s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 136m 6s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestFastFail |
| | hadoop.hbase.client.TestClientPushback |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestReplicaWithCluster |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835374/HBASE-16747_V3.patch |
| JIRA Issue | HBASE-16747 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 2acb331113f4 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / cd3dd6e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: 14417.v21.txt > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417.v1.txt, 14417.v11.txt, 14417.v13.txt, > 14417.v2.txt, 14417.v21.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: (was: 14417.v21.txt) > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417.v1.txt, 14417.v11.txt, 14417.v13.txt, > 14417.v2.txt, 14417.v21.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609193#comment-15609193 ] Anoop Sam John edited comment on HBASE-16438 at 10/26/16 6:13 PM: -- Cells are added to active segment means it is added to CSLM. When it is in memory flushed and flattened to CellChunkMap, we need to create index from cell right? So the index needs to have the chunkid. How to know the chunkid of the chunk where cell data resides? was (Author: anoop.hbase): Cells are added to active segment means it is added to CSLM. When it is in memory flushed and flattened to CellChunkMap, we need to create index from chunk right? So the index needs to have the chunkid. How to know the chunkid of the chunk where cell data resides? > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609193#comment-15609193 ] Anoop Sam John commented on HBASE-16438: Cells are added to active segment means it is added to CSLM. When it is in memory flushed and flattened to CellChunkMap, we need to create index from chunk right? So the index needs to have the chunkid. How to know the chunkid of the chunk where cell data resides? > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609182#comment-15609182 ] stack commented on HBASE-16438: --- bq. I am just seeing Anoop's reply above. Yes as he says if we dont write the chunk id in the cell then we have to have keep references onheap. Man. I'm confused still. Why? We lookup stuff via the index. The index has chunkid+offset+length so we know how to find a Cell whether chunks are onheap or offheap. Index can be onheap or offheap, right? (Separate issue, doesn't this mean we have to instantiate a Cell everytime we need to 'look' at it? Or at least, when scanning, be able to scan Cell in its 'flat form'). > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609164#comment-15609164 ] Anoop Sam John commented on HBASE-16438: Oh ya. I forgot that in the doc, still we say abt copy to new chunk.. It was being discussed at that time how to avoid this copy just to get a chunk id. We might need to add that info to the design doc. After the PoC any way. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609126#comment-15609126 ] ramkrishna.s.vasudevan commented on HBASE-16438: Yes you are correct here. We now try to avoid that copy when we move from active to immutable segment. If we have chunk id we could directly use that as the pointer as to where the cell resides. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609117#comment-15609117 ] stack commented on HBASE-16438: --- So we are talking about going from active CSLS to an immutable segment flush? Just this case? And without a copy as described in step #7 in this doc https://issues.apache.org/jira/secure/attachment/12792492/CellBlocksSegmentinthecontextofMemStore%281%29.pdf Here, our index -- the CSLS -- does not include reference to the backing 'chunk'. We'd stamp a Cell with its containing chunk to save our having to copy from one chunk to another as part of the flush from active segment to immutable? > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16851) User-facing documentation for the In-Memory Compaction feature
[ https://issues.apache.org/jira/browse/HBASE-16851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609110#comment-15609110 ] stack commented on HBASE-16851: --- We should probably have doc for devs too. This little world of inmemory compaction has its own nomenclature, concepts and even musical instruments. I started one here: https://docs.google.com/document/d/1HmpqhZlVxttvJGivxJC-9zma6RTokz3UPRvYfJ-6GxM/edit?usp=sharing > User-facing documentation for the In-Memory Compaction feature > -- > > Key: HBASE-16851 > URL: https://issues.apache.org/jira/browse/HBASE-16851 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Edward Bortnikov > Attachments: HBaseAcceleratedHbaseConf-final.pptx > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: 14417.v21.txt > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417.v1.txt, 14417.v11.txt, 14417.v13.txt, > 14417.v2.txt, 14417.v21.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14921) Inmemory Compaction optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14921: -- Summary: Inmemory Compaction optimizations (was: Memory optimizations) > Inmemory Compaction optimizations > - > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, > HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, > HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, > HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, > HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, > HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, > IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, > MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609029#comment-15609029 ] ramkrishna.s.vasudevan commented on HBASE-16890: I profiled the code after this change and that heavy synchronization penalty is not there. It requires further tests in actual cluster and will report back. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: Screen Shot 2016-10-25 at 7.34.47 PM.png, Screen Shot > 2016-10-25 at 7.39.07 PM.png, Screen Shot 2016-10-25 at 7.39.48 PM.png, > async.svg, classic.svg, contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14465) Backport 'Allow rowlock to be reader/write' to branch-1
[
https://issues.apache.org/jira/browse/HBASE-14465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609025#comment-15609025
]
James Taylor commented on HBASE-14465:
--
Question on this one, as there appear to be inconsistencies (at least in the
Region javadoc and the HRegion implementation):
For HBase 1.2, here's the javadoc from the Region interface:
{code}
/**
* Tries to acquire a lock on the given row.
* @param waitForLock if true, will block until the lock is available.
*Otherwise, just tries to obtain the lock and returns
*false if unavailable.
* @return the row lock if acquired,
* null if waitForLock was false and the lock was not acquired
* @throws IOException if waitForLock was true and the lock could not be
acquired after waiting
*/
RowLock getRowLock(byte[] row, boolean waitForLock) throws IOException;
{code}
But it seems that the meaning of the boolean has changed, as here's the
implementation:
{code}
/**
*
* Get a row lock for the specified row. All locks are reentrant.
*
* Before calling this function make sure that a region operation has already
been
* started (the calling thread has already acquired the region-close-guard
lock).
* @param row The row actions will be performed against
* @param readLock is the lock reader or writer. True indicates that a
non-exlcusive
* lock is requested
*/
public RowLock getRowLock(byte[] row, boolean readLock) throws IOException {
{code}
Then in HBase 0.98, HRegion still works the old way:
{code}
/**
* Tries to acquire a lock on the given row.
* @param waitForLock if true, will block until the lock is available.
*Otherwise, just tries to obtain the lock and returns
*false if unavailable.
* @return the row lock if acquired,
* null if waitForLock was false and the lock was not acquired
* @throws IOException if waitForLock was true and the lock could not be
acquired after waiting
*/
public RowLock getRowLock(byte[] row, boolean waitForLock) throws IOException
{
{code}
Keeping the same method signature but changing the meaning is a dangerous
change IMHO. Perhaps creating a new method name would be prudent, [~stack],
[~busbey]? Phoenix is using this API.
> Backport 'Allow rowlock to be reader/write' to branch-1
> ---
>
> Key: HBASE-14465
> URL: https://issues.apache.org/jira/browse/HBASE-14465
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: stack
>Assignee: stack
> Fix For: 1.2.0, 1.3.0
>
> Attachments: 14465.branch-1.txt, 14465.branch-1.v2.txt,
> 14465.branch-1.v2.txt, 14465.branch-1.v2.txt, 14465.branch-1.v3.txt,
> 14465.branch-1.v3.txt, 14465.branch-1.v4.txt, 14465.branch-1.v4.txt,
> 14465.branch-1.v4.txt, 14465.branch-1.v5.txt, 14465.branch-1.v5.txt,
> 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.branch-1.v9.txt, 14465.txt
>
>
> Backport to branch-1.
> You want this in 1.2 [~busbey]? Its cleanup and fixes a possible dataloss. On
> other hand, its a bit of refactoring.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16747) Track memstore data size and heap overhead separately
[ https://issues.apache.org/jira/browse/HBASE-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16747: --- Attachment: HBASE-16747_V3.patch Reattach same patch for a QA run > Track memstore data size and heap overhead separately > -- > > Key: HBASE-16747 > URL: https://issues.apache.org/jira/browse/HBASE-16747 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16747.patch, HBASE-16747.patch, > HBASE-16747_V2.patch, HBASE-16747_V2.patch, HBASE-16747_V3.patch, > HBASE-16747_V3.patch, HBASE-16747_V3.patch, HBASE-16747_WIP.patch > > > We track the memstore size in 3 places. > 1. Global at RS level in RegionServerAccounting. This tracks all memstore's > size and used to calculate whether forced flushes needed because of global > heap pressure > 2. At region level in HRegion. This is sum of sizes of all memstores within > this region. This is used to decide whether region reaches flush size (128 MB) > 3. Segment level. This tracks the in memory flush/compaction decisions. > All these use the Cell's heap size which include the data bytes# as well as > Cell object heap overhead. Also we include the overhead because of addition > of Cells into Segment's data structures (Like CSLM). > Once we have off heap memstore, we will keep the cell data bytes in off heap > area. So we can not track both data size and heap overhead as one entity. We > need to separate them and track. > Proposal here is to track both cell data size and heap overhead separately at > global accounting layer. As of now we have only on heap memstore. So the > global memstore boundary checks will consider both (adds up and check against > global max memstore size) > Track cell data size alone (This can be on heap or off heap) in region level. > Region flushes use cell data size alone for the region flush decision. A > user configuring 128 MB as flush size, normally he will expect to get a 128MB > data flush size. But as we were including the heap overhead also, once the > flush happens, the actual data size getting flushed is way behind this 128 > MB. Now with this change we will behave more like what a user thinks. > Segment level in memory flush/compaction also considers cell data size alone. > But we will need to track the heap overhead also. (Once the in memory flush > or normal flush happens, we will have to adjust both cell data size and heap > overhead) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
