[jira] [Commented] (HBASE-16213) A new HFileBlock structure for fast random get
[
https://issues.apache.org/jira/browse/HBASE-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383619#comment-15383619
]
stack commented on HBASE-16213:
---
Nice. Seek in the row when random reading is one of the main consumers of CPU.
Why bother having two encoders? Why not just one that does row and column
family index?
Any idea on how much more work we are doing when this is enabled (CPU?). Is it
less with this feature on or more? Under what circumstances do you think?
Let me try this. Meantime here are some comments on the patch:
In class comment, either in encoder or decoder, describe how the encoding
works, what layout looks like with some advice on when to use it. Can then copy
paste as the release note on this issue.
For...
builder.write(cell);
Could the above return a length so you don't have to reget it on the next line
with:
int size = KeyValueUtil.length(cell);
The length parse costs.
Anywhere that you can get count of how many kvs in block that you can use here:
List kvs = new ArrayList();
Remove these...
// TODO Auto-generated method stub
Put these together?
102 LOG.trace("RowNumber: " + rowsOffset.size());
103 LOG.trace("onDiskSize: " + onDiskSize);
One line is easier to read than two...
Got half way through... will be back w/ more. Nice.
> A new HFileBlock structure for fast random get
> --
>
> Key: HBASE-16213
> URL: https://issues.apache.org/jira/browse/HBASE-16213
> Project: HBase
> Issue Type: New Feature
> Components: Performance
>Reporter: binlijin
>Assignee: binlijin
> Attachments: HBASE-16213-master_v1.patch, HBASE-16213.patch,
> HBASE-16213_v2.patch
>
>
> HFileBlock store cells sequential, current when to get a row from the block,
> it scan from the first cell until the row's cell.
> The new structure store every row's start offset with data, so it can find
> the exact row with binarySearch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-9465) Push entries to peer clusters serially
[ https://issues.apache.org/jira/browse/HBASE-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-9465: - Attachment: HBASE-9465-v1.patch Submit v1 patch for master. The implementation does not fully follow the design doc. I'll revise it and upload a pdf version soon > Push entries to peer clusters serially > -- > > Key: HBASE-9465 > URL: https://issues.apache.org/jira/browse/HBASE-9465 > Project: HBase > Issue Type: New Feature > Components: regionserver, Replication >Reporter: Honghua Feng >Assignee: Phil Yang > Attachments: HBASE-9465-v1.patch > > > When region-move or RS failure occurs in master cluster, the hlog entries > that are not pushed before region-move or RS-failure will be pushed by > original RS(for region move) or another RS which takes over the remained hlog > of dead RS(for RS failure), and the new entries for the same region(s) will > be pushed by the RS which now serves the region(s), but they push the hlog > entries of a same region concurrently without coordination. > This treatment can possibly lead to data inconsistency between master and > peer clusters: > 1. there are put and then delete written to master cluster > 2. due to region-move / RS-failure, they are pushed by different > replication-source threads to peer cluster > 3. if delete is pushed to peer cluster before put, and flush and > major-compact occurs in peer cluster before put is pushed to peer cluster, > the delete is collected and the put remains in peer cluster > In this scenario, the put remains in peer cluster, but in master cluster the > put is masked by the delete, hence data inconsistency between master and peer > clusters -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9465) Push entries to peer clusters serially
[ https://issues.apache.org/jira/browse/HBASE-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-9465: - Status: Patch Available (was: Open) > Push entries to peer clusters serially > -- > > Key: HBASE-9465 > URL: https://issues.apache.org/jira/browse/HBASE-9465 > Project: HBase > Issue Type: New Feature > Components: regionserver, Replication >Reporter: Honghua Feng >Assignee: Phil Yang > Attachments: HBASE-9465-v1.patch > > > When region-move or RS failure occurs in master cluster, the hlog entries > that are not pushed before region-move or RS-failure will be pushed by > original RS(for region move) or another RS which takes over the remained hlog > of dead RS(for RS failure), and the new entries for the same region(s) will > be pushed by the RS which now serves the region(s), but they push the hlog > entries of a same region concurrently without coordination. > This treatment can possibly lead to data inconsistency between master and > peer clusters: > 1. there are put and then delete written to master cluster > 2. due to region-move / RS-failure, they are pushed by different > replication-source threads to peer cluster > 3. if delete is pushed to peer cluster before put, and flush and > major-compact occurs in peer cluster before put is pushed to peer cluster, > the delete is collected and the put remains in peer cluster > In this scenario, the put remains in peer cluster, but in master cluster the > put is masked by the delete, hence data inconsistency between master and peer > clusters -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16213) A new HFileBlock structure for fast random get
[ https://issues.apache.org/jira/browse/HBASE-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16213: -- Assignee: binlijin > A new HFileBlock structure for fast random get > -- > > Key: HBASE-16213 > URL: https://issues.apache.org/jira/browse/HBASE-16213 > Project: HBase > Issue Type: New Feature > Components: Performance >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16213-master_v1.patch, HBASE-16213.patch, > HBASE-16213_v2.patch > > > HFileBlock store cells sequential, current when to get a row from the block, > it scan from the first cell until the row's cell. > The new structure store every row's start offset with data, so it can find > the exact row with binarySearch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16213) A new HFileBlock structure for fast random get
[ https://issues.apache.org/jira/browse/HBASE-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16213: -- Component/s: Performance > A new HFileBlock structure for fast random get > -- > > Key: HBASE-16213 > URL: https://issues.apache.org/jira/browse/HBASE-16213 > Project: HBase > Issue Type: New Feature > Components: Performance >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16213-master_v1.patch, HBASE-16213.patch, > HBASE-16213_v2.patch > > > HFileBlock store cells sequential, current when to get a row from the block, > it scan from the first cell until the row's cell. > The new structure store every row's start offset with data, so it can find > the exact row with binarySearch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14613) Remove MemStoreChunkPool?
[ https://issues.apache.org/jira/browse/HBASE-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383601#comment-15383601 ] ramkrishna.s.vasudevan commented on HBASE-14613: bq.What about the CMS vs G1GC discussion above? With pool we were able to tune the CMS and were able to see better performance. In case of G1GC, because we have offheap now and having a Buffer pool helps reduce the heap that is required and the tuning of G1GC helps to have a much predictable throughput. Will try out the same option with onheap and pooling. Will get back on that here. > Remove MemStoreChunkPool? > - > > Key: HBASE-14613 > URL: https://issues.apache.org/jira/browse/HBASE-14613 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 14613-0.98.txt, gc.png, writes.png > > > I just stumbled across MemStoreChunkPool. The idea behind is to reuse chunks > of allocations rather than letting the GC handle this. > Now, it's off by default, and it seems to me to be of dubious value. I'd > recommend just removing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16236) Typo in javadoc of InstancePending
[
https://issues.apache.org/jira/browse/HBASE-16236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16236:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: (was: 0.98.20)
(was: 1.3.0)
Status: Resolved (was: Patch Available)
Pushed to branch-1 and master branch... Thanks for the patch [~ikeda].
> Typo in javadoc of InstancePending
> --
>
> Key: HBASE-16236
> URL: https://issues.apache.org/jira/browse/HBASE-16236
> Project: HBase
> Issue Type: Sub-task
> Components: Zookeeper
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Trivial
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16236.patch
>
>
> {code}
> class InstancePending {
> ...
> /**
>* Returns the instance given by the method {@link #prepare}.
>* This is an interruptible blocking method
>* and the interruption flag will be set just before returning if any.
>*/
> T get() {
> {code}
> Sorry I just wanted to write "an *uninterruptible* blocking method".
> That's opposite and a seed of confusion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14921) Memory optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383595#comment-15383595 ] ramkrishna.s.vasudevan commented on HBASE-14921: bq.but I suggest first to proof this scan is causing the performance degradation. Any thoughts about that? The point is just for the default case. In default cases we are sure that we don't need to remove any duplicates. In such a doing a compaction is not needed. Yes for flattening we need to do a scan but that should be like an iterator mode not with the ScanQueryMatcher. > Memory optimizations > > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, > HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, > InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14921) Memory optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383594#comment-15383594 ] ramkrishna.s.vasudevan commented on HBASE-14921: I now got to which comment are you referring to wrt CELL_ARRAY_ENTRY. I checked the RB. bq.CELL_ARRAY_MAP_ENTRY = align(OBJECT + 2*REFERENCE + 2*Bytes.SIZEOF_INT); When you see the CELL_ARRAY_MAP_ENTRY it should only have the reference to the cell and one object right? The 2 ints and the byte[] will be accounted in the KeyValue.heapSize(). Now if you add the ref and int here then the KeyValue.heapSize will again add it. So the comment in RB is valid I feel. > Memory optimizations > > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, > HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, > InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16242) Upgrade Avro
[ https://issues.apache.org/jira/browse/HBASE-16242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383585#comment-15383585 ] stack commented on HBASE-16242: --- What version are you talking of [~b...@benmccann.com] There is no avro in master branch (I think). > Upgrade Avro > > > Key: HBASE-16242 > URL: https://issues.apache.org/jira/browse/HBASE-16242 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Ben McCann > > I'd like to see Avro upgraded to 1.8.1 or at the very least 1.7.7 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16244) LoclaHBaseCluster start timeout is not configurable
[
https://issues.apache.org/jira/browse/HBASE-16244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383584#comment-15383584
]
stack commented on HBASE-16244:
---
+1
> LoclaHBaseCluster start timeout is not configurable
> ---
>
> Key: HBASE-16244
> URL: https://issues.apache.org/jira/browse/HBASE-16244
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.0.1.1
>Reporter: Siddharth Wagle
> Fix For: 1.1.2
>
> Attachments: HBASE-16244.patch
>
>
> *Scenario*:
> - Ambari metrics service uses HBase in standalone mode
> - On restart of AMS HBase, the Master gives up in 30 seconds due to a
> hardcoded timeout in JVMClusterUtil
> {noformat}
> 2016-07-18 19:24:44,199 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Master not active after 30 seconds
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:194)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:445)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:227)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2526)
> {noformat}
> - On restart the current Master waits to become active and this leads to the
> timeout being triggered, waiting for a slightly longer time evades this issue.
> - The timeout it seems was meant for unit tests
> Attached patch allows the timeout to be configured via hbase-site as well as
> sets it to 5 minutes for clusters started through HMasterCommandLine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16212) Many connections to datanode are created when doing a large scan
[ https://issues.apache.org/jira/browse/HBASE-16212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383565#comment-15383565 ] Zhihua Deng edited comment on HBASE-16212 at 7/19/16 4:30 AM: -- Changing from threadlocal to a common field, yes there will be a potential synchronization bottleneck, but it better than io operation. So the question here is that how often the connection will be recreated for seeking + reading? The original threadlocal is declared as non static private field here, it means that the created fsreaderimpl instance will be reused later on, also a inputstream is initiated when fsreaderimpl created. Taken the case described in the attached log, The synchronization way is more better than threadlocal when acts as a sequential read . How about concurrent case? the worst case: Thread1.readBlockInternal -> Thread2.readBlockInternal -> Thread3.readBlockInternal -> Thread1.readBlockInternal -> In this case, the synchronization way is equal to threadlocal when taken how many connections will be created into consideration. was (Author: dengzh): Changing from threadlocal to a common field, yes there will be a potential synchronization bottleneck, but it better than io operation. So the question here is that how often the connection will be recreated for seeking + reading? The original threadlocal is declared as non static private field here, it means that the created fsreaderimpl instance will be reused later on, also a inputstream is initiated when fsreaderimpl created. Taken the case described in the attached log, The synchronization way is more better than threadlocal. How about concurrent case? the worst case: Thread1.readBlockInternal -> Thread2.readBlockInternal -> Thread3.readBlockInternal -> Thread1.readBlockInternal -> In this case, the synchronization way is equal to threadlocal when taken how many connections will be created into consideration. > Many connections to datanode are created when doing a large scan > - > > Key: HBASE-16212 > URL: https://issues.apache.org/jira/browse/HBASE-16212 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.1.2 >Reporter: Zhihua Deng > Attachments: HBASE-16212.patch, HBASE-16212.v2.patch, > regionserver-dfsinputstream.log > > > As described in https://issues.apache.org/jira/browse/HDFS-8659, the datanode > is suffering from logging the same repeatedly. Adding log to DFSInputStream, > it outputs as follows: > 2016-07-10 21:31:42,147 INFO > [B.defaultRpcServer.handler=22,queue=1,port=16020] hdfs.DFSClient: > DFSClient_NONMAPREDUCE_1984924661_1 seek > DatanodeInfoWithStorage[10.130.1.29:50010,DS-086bc494-d862-470c-86e8-9cb7929985c6,DISK] > for BP-360285305-10.130.1.11-1444619256876:blk_1109360829_35627143. pos: > 111506876, targetPos: 111506843 > ... > As the pos of this input stream is larger than targetPos(the pos trying to > seek), A new connection to the datanode will be created, the older one will > be closed as a consequence. When the wrong seeking ops are large, the > datanode's block scanner info message is spamming logs, as well as many > connections to the same datanode will be created. > hadoop version: 2.7.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16212) Many connections to datanode are created when doing a large scan
[ https://issues.apache.org/jira/browse/HBASE-16212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383565#comment-15383565 ] Zhihua Deng edited comment on HBASE-16212 at 7/19/16 4:22 AM: -- Changing from threadlocal to a common field, yes there will be a potential synchronization bottleneck, but it better than io operation. So the question here is that how often the connection will be recreated for seeking + reading? The original threadlocal is declared as non static private field here, it means that the created fsreaderimpl instance will be reused later on, also a inputstream is initiated when fsreaderimpl created. Taken the case described in the attached log, The synchronization way is more better than threadlocal. How about concurrent case? the worst case: Thread1.readBlockInternal -> Thread2.readBlockInternal -> Thread3.readBlockInternal -> Thread1.readBlockInternal -> In this case, the synchronization way is equal to threadlocal when taken how many connections will be created into consideration. was (Author: dengzh): Changing from threadlocal to a common field, yes there will be a potential synchronization bottleneck, but it better than io operation. So the question here is that how often the connection will be recreated for seeking + reading? The original threadlocal is declared as non static private field here, it means that the created fsreaderimpl instance will be reused later on, also a inputstream is initiated when fsreaderimpl created. Taken the case described in the attached log, The synchronization way is more better than threadlocal. How about concurrent case? the worst case: Thread1.readBlockInternal -> Thread2.readBlockInternal -> Thread3.readBlockInternal -> Thread1.readBlockInternal -> In this case, the synchronization way is equal to threadlocal when taken how many connections will be created into consideration. > Many connections to datanode are created when doing a large scan > - > > Key: HBASE-16212 > URL: https://issues.apache.org/jira/browse/HBASE-16212 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.1.2 >Reporter: Zhihua Deng > Attachments: HBASE-16212.patch, HBASE-16212.v2.patch, > regionserver-dfsinputstream.log > > > As described in https://issues.apache.org/jira/browse/HDFS-8659, the datanode > is suffering from logging the same repeatedly. Adding log to DFSInputStream, > it outputs as follows: > 2016-07-10 21:31:42,147 INFO > [B.defaultRpcServer.handler=22,queue=1,port=16020] hdfs.DFSClient: > DFSClient_NONMAPREDUCE_1984924661_1 seek > DatanodeInfoWithStorage[10.130.1.29:50010,DS-086bc494-d862-470c-86e8-9cb7929985c6,DISK] > for BP-360285305-10.130.1.11-1444619256876:blk_1109360829_35627143. pos: > 111506876, targetPos: 111506843 > ... > As the pos of this input stream is larger than targetPos(the pos trying to > seek), A new connection to the datanode will be created, the older one will > be closed as a consequence. When the wrong seeking ops are large, the > datanode's block scanner info message is spamming logs, as well as many > connections to the same datanode will be created. > hadoop version: 2.7.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14552) Procedure V2: Reimplement DispatchMergingRegionHandler
[ https://issues.apache.org/jira/browse/HBASE-14552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383566#comment-15383566 ] Matteo Bertozzi commented on HBASE-14552: - +1 > Procedure V2: Reimplement DispatchMergingRegionHandler > -- > > Key: HBASE-14552 > URL: https://issues.apache.org/jira/browse/HBASE-14552 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-14552-addendum.v1-master.patch, > HBASE-14552.v0-master.patch, HBASE-14552.v1-master.patch, > HBASE-14552.v2-master.patch, HBASE-14552.v3-master.patch > > > use the proc-v2 state machine for DispatchMergingRegionHandler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16212) Many connections to datanode are created when doing a large scan
[ https://issues.apache.org/jira/browse/HBASE-16212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383565#comment-15383565 ] Zhihua Deng commented on HBASE-16212: - Changing from threadlocal to a common field, yes there will be a potential synchronization bottleneck, but it better than io operation. So the question here is that how often the connection will be recreated for seeking + reading? The original threadlocal is declared as non static private field here, it means that the created fsreaderimpl instance will be reused later on, also a inputstream is initiated when fsreaderimpl created. Taken the case described in the attached log, The synchronization way is more better than threadlocal. How about concurrent case? the worst case: Thread1.readBlockInternal -> Thread2.readBlockInternal -> Thread3.readBlockInternal -> Thread1.readBlockInternal -> In this case, the synchronization way is equal to threadlocal when taken how many connections will be created into consideration. > Many connections to datanode are created when doing a large scan > - > > Key: HBASE-16212 > URL: https://issues.apache.org/jira/browse/HBASE-16212 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.1.2 >Reporter: Zhihua Deng > Attachments: HBASE-16212.patch, HBASE-16212.v2.patch, > regionserver-dfsinputstream.log > > > As described in https://issues.apache.org/jira/browse/HDFS-8659, the datanode > is suffering from logging the same repeatedly. Adding log to DFSInputStream, > it outputs as follows: > 2016-07-10 21:31:42,147 INFO > [B.defaultRpcServer.handler=22,queue=1,port=16020] hdfs.DFSClient: > DFSClient_NONMAPREDUCE_1984924661_1 seek > DatanodeInfoWithStorage[10.130.1.29:50010,DS-086bc494-d862-470c-86e8-9cb7929985c6,DISK] > for BP-360285305-10.130.1.11-1444619256876:blk_1109360829_35627143. pos: > 111506876, targetPos: 111506843 > ... > As the pos of this input stream is larger than targetPos(the pos trying to > seek), A new connection to the datanode will be created, the older one will > be closed as a consequence. When the wrong seeking ops are large, the > datanode's block scanner info message is spamming logs, as well as many > connections to the same datanode will be created. > hadoop version: 2.7.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14552) Procedure V2: Reimplement DispatchMergingRegionHandler
[ https://issues.apache.org/jira/browse/HBASE-14552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-14552: --- Status: Patch Available (was: Reopened) > Procedure V2: Reimplement DispatchMergingRegionHandler > -- > > Key: HBASE-14552 > URL: https://issues.apache.org/jira/browse/HBASE-14552 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-14552-addendum.v1-master.patch, > HBASE-14552.v0-master.patch, HBASE-14552.v1-master.patch, > HBASE-14552.v2-master.patch, HBASE-14552.v3-master.patch > > > use the proc-v2 state machine for DispatchMergingRegionHandler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14552) Procedure V2: Reimplement DispatchMergingRegionHandler
[ https://issues.apache.org/jira/browse/HBASE-14552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383560#comment-15383560 ] Stephen Yuan Jiang commented on HBASE-14552: The patch missed the nonce from hbase-client to deal with duplicate RPC call. This is not a regression (on par with 1.x releases), but with procedure support, we should do better than 1.x code base. Hence reopen and attach a addendum patch. FYI. [~mbertozzi] > Procedure V2: Reimplement DispatchMergingRegionHandler > -- > > Key: HBASE-14552 > URL: https://issues.apache.org/jira/browse/HBASE-14552 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-14552-addendum.v1-master.patch, > HBASE-14552.v0-master.patch, HBASE-14552.v1-master.patch, > HBASE-14552.v2-master.patch, HBASE-14552.v3-master.patch > > > use the proc-v2 state machine for DispatchMergingRegionHandler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-14552) Procedure V2: Reimplement DispatchMergingRegionHandler
[ https://issues.apache.org/jira/browse/HBASE-14552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang reopened HBASE-14552: > Procedure V2: Reimplement DispatchMergingRegionHandler > -- > > Key: HBASE-14552 > URL: https://issues.apache.org/jira/browse/HBASE-14552 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-14552-addendum.v1-master.patch, > HBASE-14552.v0-master.patch, HBASE-14552.v1-master.patch, > HBASE-14552.v2-master.patch, HBASE-14552.v3-master.patch > > > use the proc-v2 state machine for DispatchMergingRegionHandler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14552) Procedure V2: Reimplement DispatchMergingRegionHandler
[ https://issues.apache.org/jira/browse/HBASE-14552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-14552: --- Attachment: HBASE-14552-addendum.v1-master.patch > Procedure V2: Reimplement DispatchMergingRegionHandler > -- > > Key: HBASE-14552 > URL: https://issues.apache.org/jira/browse/HBASE-14552 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-14552-addendum.v1-master.patch, > HBASE-14552.v0-master.patch, HBASE-14552.v1-master.patch, > HBASE-14552.v2-master.patch, HBASE-14552.v3-master.patch > > > use the proc-v2 state machine for DispatchMergingRegionHandler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16197) Enchance backup delete command
[ https://issues.apache.org/jira/browse/HBASE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383555#comment-15383555 ] Vladimir Rodionov commented on HBASE-16197: --- cc: [~tedyu] > Enchance backup delete command > -- > > Key: HBASE-16197 > URL: https://issues.apache.org/jira/browse/HBASE-16197 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16197-v1.patch, HBASE-16197-v2.patch > > > Currently, backup delete command deletes only physical files/directories in > backup destination. It does not update backup system table (incremental > backup table set) and it does not delete related backup images (previous > ones), thus creating a hole in dependency chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16197) Enchance backup delete command
[ https://issues.apache.org/jira/browse/HBASE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16197: -- Attachment: HBASE-16197-v2.patch v2. Added one more unit test > Enchance backup delete command > -- > > Key: HBASE-16197 > URL: https://issues.apache.org/jira/browse/HBASE-16197 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16197-v1.patch, HBASE-16197-v2.patch > > > Currently, backup delete command deletes only physical files/directories in > backup destination. It does not update backup system table (incremental > backup table set) and it does not delete related backup images (previous > ones), thus creating a hole in dependency chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16096) Replication keeps accumulating znodes
[
https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383552#comment-15383552
]
Hadoop QA commented on HBASE-16096:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 28s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 47s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 137m 36s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12818721/HBASE-16096.patch |
| JIRA Issue | HBASE-16096 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c9925bc |
| Default Java | 1.7.0_80 |
| Multi-JDK versions | /home/jenkins/tools/java/jdk1.8.0:1.8.0
/home/jenkins/jenkins-slave/tools/hudson.model.JDK/JDK_1.7_latest_:1.7.0_80 |
| findbugs | v3.0.0 |
| unit |
[jira] [Updated] (HBASE-16244) LoclaHBaseCluster start timeout is not configurable
[
https://issues.apache.org/jira/browse/HBASE-16244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Wagle updated HBASE-16244:
Fix Version/s: 1.1.2
Status: Patch Available (was: Open)
> LoclaHBaseCluster start timeout is not configurable
> ---
>
> Key: HBASE-16244
> URL: https://issues.apache.org/jira/browse/HBASE-16244
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.0.1.1
>Reporter: Siddharth Wagle
> Fix For: 1.1.2
>
> Attachments: HBASE-16244.patch
>
>
> *Scenario*:
> - Ambari metrics service uses HBase in standalone mode
> - On restart of AMS HBase, the Master gives up in 30 seconds due to a
> hardcoded timeout in JVMClusterUtil
> {noformat}
> 2016-07-18 19:24:44,199 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Master not active after 30 seconds
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:194)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:445)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:227)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2526)
> {noformat}
> - On restart the current Master waits to become active and this leads to the
> timeout being triggered, waiting for a slightly longer time evades this issue.
> - The timeout it seems was meant for unit tests
> Attached patch allows the timeout to be configured via hbase-site as well as
> sets it to 5 minutes for clusters started through HMasterCommandLine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16244) LoclaHBaseCluster start timeout is not configurable
[
https://issues.apache.org/jira/browse/HBASE-16244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Wagle updated HBASE-16244:
Attachment: HBASE-16244.patch
> LoclaHBaseCluster start timeout is not configurable
> ---
>
> Key: HBASE-16244
> URL: https://issues.apache.org/jira/browse/HBASE-16244
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.0.1.1
>Reporter: Siddharth Wagle
> Attachments: HBASE-16244.patch
>
>
> *Scenario*:
> - Ambari metrics service uses HBase in standalone mode
> - On restart of AMS HBase, the Master gives up in 30 seconds due to a
> hardcoded timeout in JVMClusterUtil
> {noformat}
> 2016-07-18 19:24:44,199 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Master not active after 30 seconds
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:194)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:445)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:227)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2526)
> {noformat}
> - On restart the current Master waits to become active and this leads to the
> timeout being triggered, waiting for a slightly longer time evades this issue.
> - The timeout it seems was meant for unit tests
> Attached patch allows the timeout to be configured via hbase-site as well as
> sets it to 5 minutes for clusters started through HMasterCommandLine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16244) LoclaHBaseCluster start timeout is not configurable
Siddharth Wagle created HBASE-16244:
---
Summary: LoclaHBaseCluster start timeout is not configurable
Key: HBASE-16244
URL: https://issues.apache.org/jira/browse/HBASE-16244
Project: HBase
Issue Type: Bug
Components: hbase
Affects Versions: 1.0.1.1
Reporter: Siddharth Wagle
*Scenario*:
- Ambari metrics service uses HBase in standalone mode
- On restart of AMS HBase, the Master gives up in 30 seconds due to a hardcoded
timeout in JVMClusterUtil
{noformat}
2016-07-18 19:24:44,199 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Master not active after 30 seconds
at
org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:194)
at
org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:445)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:227)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2526)
{noformat}
- On restart the current Master waits to become active and this leads to the
timeout being triggered, waiting for a slightly longer time evades this issue.
- The timeout it seems was meant for unit tests
Attached patch allows the timeout to be configured via hbase-site as well as
sets it to 5 minutes for clusters started through HMasterCommandLine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16238) It's useless to catch SESSIONEXPIRED exception and retry in RecoverableZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-16238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383534#comment-15383534
]
Hudson commented on HBASE-16238:
FAILURE: Integrated in HBase-Trunk_matrix #1254 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1254/])
HBASE-16238 It's useless to catch SESSIONEXPIRED exception and retry in (stack:
rev c9925bcec1f5cd427c4a875dba64b92e3dafcc1e)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
> It's useless to catch SESSIONEXPIRED exception and retry in
> RecoverableZooKeeper
>
>
> Key: HBASE-16238
> URL: https://issues.apache.org/jira/browse/HBASE-16238
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Affects Versions: 1.1.5, 1.2.2, 0.98.20
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 1.3.0, 1.1.5, 1.2.2, 0.98.20
>
> Attachments: HBASE-16238.patch, HBASE-16238v2.patch
>
>
> After HBASE-5549, SESSIONEXPIRED exception was caught and retried with other
> zookeeper exceptions like ConnectionLoss. But it is useless to retry when a
> session expired happens, since the retry will never be successful. Though
> there is a config called "zookeeper.recovery.retry" to control retry times,
> in our cases, we set this config to a very big number like "9". When a
> session expired happens, the regionserver should kill itself, but because of
> the retrying, threads of regionserver stuck at trying to reconnect to
> zookeeper, and never properly shut down.
> {code}
> public Stat exists(String path, boolean watch)
> throws KeeperException, InterruptedException {
> TraceScope traceScope = null;
> try {
> traceScope = Trace.startSpan("RecoverableZookeeper.exists");
> RetryCounter retryCounter = retryCounterFactory.create();
> while (true) {
> try {
> return checkZk().exists(path, watch);
> } catch (KeeperException e) {
> switch (e.code()) {
> case CONNECTIONLOSS:
> case SESSIONEXPIRED: //we shouldn't catch this
> case OPERATIONTIMEOUT:
> retryOrThrow(retryCounter, e, "exists");
> break;
> default:
> throw e;
> }
> }
> retryCounter.sleepUntilNextRetry();
> }
> } finally {
> if (traceScope != null) traceScope.close();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16238) It's useless to catch SESSIONEXPIRED exception and retry in RecoverableZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-16238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383529#comment-15383529
]
Hudson commented on HBASE-16238:
FAILURE: Integrated in HBase-1.3 #789 (See
[https://builds.apache.org/job/HBase-1.3/789/])
HBASE-16238 It's useless to catch SESSIONEXPIRED exception and retry in (stack:
rev a76cede216a1bb7a68c3c96aeeccfcf98f0d8441)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
> It's useless to catch SESSIONEXPIRED exception and retry in
> RecoverableZooKeeper
>
>
> Key: HBASE-16238
> URL: https://issues.apache.org/jira/browse/HBASE-16238
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Affects Versions: 1.1.5, 1.2.2, 0.98.20
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 1.3.0, 1.1.5, 1.2.2, 0.98.20
>
> Attachments: HBASE-16238.patch, HBASE-16238v2.patch
>
>
> After HBASE-5549, SESSIONEXPIRED exception was caught and retried with other
> zookeeper exceptions like ConnectionLoss. But it is useless to retry when a
> session expired happens, since the retry will never be successful. Though
> there is a config called "zookeeper.recovery.retry" to control retry times,
> in our cases, we set this config to a very big number like "9". When a
> session expired happens, the regionserver should kill itself, but because of
> the retrying, threads of regionserver stuck at trying to reconnect to
> zookeeper, and never properly shut down.
> {code}
> public Stat exists(String path, boolean watch)
> throws KeeperException, InterruptedException {
> TraceScope traceScope = null;
> try {
> traceScope = Trace.startSpan("RecoverableZookeeper.exists");
> RetryCounter retryCounter = retryCounterFactory.create();
> while (true) {
> try {
> return checkZk().exists(path, watch);
> } catch (KeeperException e) {
> switch (e.code()) {
> case CONNECTIONLOSS:
> case SESSIONEXPIRED: //we shouldn't catch this
> case OPERATIONTIMEOUT:
> retryOrThrow(retryCounter, e, "exists");
> break;
> default:
> throw e;
> }
> }
> retryCounter.sleepUntilNextRetry();
> }
> } finally {
> if (traceScope != null) traceScope.close();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16238) It's useless to catch SESSIONEXPIRED exception and retry in RecoverableZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-16238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383520#comment-15383520
]
Hudson commented on HBASE-16238:
FAILURE: Integrated in HBase-1.4 #296 (See
[https://builds.apache.org/job/HBase-1.4/296/])
HBASE-16238 It's useless to catch SESSIONEXPIRED exception and retry in (stack:
rev f83c2345bb92e335eec4afe17f254580be5dc5dc)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
> It's useless to catch SESSIONEXPIRED exception and retry in
> RecoverableZooKeeper
>
>
> Key: HBASE-16238
> URL: https://issues.apache.org/jira/browse/HBASE-16238
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Affects Versions: 1.1.5, 1.2.2, 0.98.20
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 1.3.0, 1.1.5, 1.2.2, 0.98.20
>
> Attachments: HBASE-16238.patch, HBASE-16238v2.patch
>
>
> After HBASE-5549, SESSIONEXPIRED exception was caught and retried with other
> zookeeper exceptions like ConnectionLoss. But it is useless to retry when a
> session expired happens, since the retry will never be successful. Though
> there is a config called "zookeeper.recovery.retry" to control retry times,
> in our cases, we set this config to a very big number like "9". When a
> session expired happens, the regionserver should kill itself, but because of
> the retrying, threads of regionserver stuck at trying to reconnect to
> zookeeper, and never properly shut down.
> {code}
> public Stat exists(String path, boolean watch)
> throws KeeperException, InterruptedException {
> TraceScope traceScope = null;
> try {
> traceScope = Trace.startSpan("RecoverableZookeeper.exists");
> RetryCounter retryCounter = retryCounterFactory.create();
> while (true) {
> try {
> return checkZk().exists(path, watch);
> } catch (KeeperException e) {
> switch (e.code()) {
> case CONNECTIONLOSS:
> case SESSIONEXPIRED: //we shouldn't catch this
> case OPERATIONTIMEOUT:
> retryOrThrow(retryCounter, e, "exists");
> break;
> default:
> throw e;
> }
> }
> retryCounter.sleepUntilNextRetry();
> }
> } finally {
> if (traceScope != null) traceScope.close();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16212) Many connections to datanode are created when doing a large scan
[ https://issues.apache.org/jira/browse/HBASE-16212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383516#comment-15383516 ] Zhihua Deng edited comment on HBASE-16212 at 7/19/16 3:11 AM: -- Test it on my cluster, a hmaster with two regionservers based on hbase-1.1.2, it works. Further test should carry on the master brunch was (Author: dengzh): Test it on my cluster, a hmaster with two regionservers based on hbase-1.1.2, it works. Further test should carry on the master brunch > Many connections to datanode are created when doing a large scan > - > > Key: HBASE-16212 > URL: https://issues.apache.org/jira/browse/HBASE-16212 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.1.2 >Reporter: Zhihua Deng > Attachments: HBASE-16212.patch, HBASE-16212.v2.patch, > regionserver-dfsinputstream.log > > > As described in https://issues.apache.org/jira/browse/HDFS-8659, the datanode > is suffering from logging the same repeatedly. Adding log to DFSInputStream, > it outputs as follows: > 2016-07-10 21:31:42,147 INFO > [B.defaultRpcServer.handler=22,queue=1,port=16020] hdfs.DFSClient: > DFSClient_NONMAPREDUCE_1984924661_1 seek > DatanodeInfoWithStorage[10.130.1.29:50010,DS-086bc494-d862-470c-86e8-9cb7929985c6,DISK] > for BP-360285305-10.130.1.11-1444619256876:blk_1109360829_35627143. pos: > 111506876, targetPos: 111506843 > ... > As the pos of this input stream is larger than targetPos(the pos trying to > seek), A new connection to the datanode will be created, the older one will > be closed as a consequence. When the wrong seeking ops are large, the > datanode's block scanner info message is spamming logs, as well as many > connections to the same datanode will be created. > hadoop version: 2.7.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16212) Many connections to datanode are created when doing a large scan
[ https://issues.apache.org/jira/browse/HBASE-16212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383517#comment-15383517 ] Zhihua Deng commented on HBASE-16212: - Test it on my cluster, a hmaster with two regionservers based on hbase-1.1.2, it works. Further test should carry on the master brunch > Many connections to datanode are created when doing a large scan > - > > Key: HBASE-16212 > URL: https://issues.apache.org/jira/browse/HBASE-16212 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.1.2 >Reporter: Zhihua Deng > Attachments: HBASE-16212.patch, HBASE-16212.v2.patch, > regionserver-dfsinputstream.log > > > As described in https://issues.apache.org/jira/browse/HDFS-8659, the datanode > is suffering from logging the same repeatedly. Adding log to DFSInputStream, > it outputs as follows: > 2016-07-10 21:31:42,147 INFO > [B.defaultRpcServer.handler=22,queue=1,port=16020] hdfs.DFSClient: > DFSClient_NONMAPREDUCE_1984924661_1 seek > DatanodeInfoWithStorage[10.130.1.29:50010,DS-086bc494-d862-470c-86e8-9cb7929985c6,DISK] > for BP-360285305-10.130.1.11-1444619256876:blk_1109360829_35627143. pos: > 111506876, targetPos: 111506843 > ... > As the pos of this input stream is larger than targetPos(the pos trying to > seek), A new connection to the datanode will be created, the older one will > be closed as a consequence. When the wrong seeking ops are large, the > datanode's block scanner info message is spamming logs, as well as many > connections to the same datanode will be created. > hadoop version: 2.7.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16212) Many connections to datanode are created when doing a large scan
[ https://issues.apache.org/jira/browse/HBASE-16212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383516#comment-15383516 ] Zhihua Deng commented on HBASE-16212: - Test it on my cluster, a hmaster with two regionservers based on hbase-1.1.2, it works. Further test should carry on the master brunch > Many connections to datanode are created when doing a large scan > - > > Key: HBASE-16212 > URL: https://issues.apache.org/jira/browse/HBASE-16212 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.1.2 >Reporter: Zhihua Deng > Attachments: HBASE-16212.patch, HBASE-16212.v2.patch, > regionserver-dfsinputstream.log > > > As described in https://issues.apache.org/jira/browse/HDFS-8659, the datanode > is suffering from logging the same repeatedly. Adding log to DFSInputStream, > it outputs as follows: > 2016-07-10 21:31:42,147 INFO > [B.defaultRpcServer.handler=22,queue=1,port=16020] hdfs.DFSClient: > DFSClient_NONMAPREDUCE_1984924661_1 seek > DatanodeInfoWithStorage[10.130.1.29:50010,DS-086bc494-d862-470c-86e8-9cb7929985c6,DISK] > for BP-360285305-10.130.1.11-1444619256876:blk_1109360829_35627143. pos: > 111506876, targetPos: 111506843 > ... > As the pos of this input stream is larger than targetPos(the pos trying to > seek), A new connection to the datanode will be created, the older one will > be closed as a consequence. When the wrong seeking ops are large, the > datanode's block scanner info message is spamming logs, as well as many > connections to the same datanode will be created. > hadoop version: 2.7.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-16212) Many connections to datanode are created when doing a large scan
[ https://issues.apache.org/jira/browse/HBASE-16212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HBASE-16212: Comment: was deleted (was: Test it on my cluster, a hmaster with two regionservers based on hbase-1.1.2, it works. Further test should carry on the master brunch) > Many connections to datanode are created when doing a large scan > - > > Key: HBASE-16212 > URL: https://issues.apache.org/jira/browse/HBASE-16212 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.1.2 >Reporter: Zhihua Deng > Attachments: HBASE-16212.patch, HBASE-16212.v2.patch, > regionserver-dfsinputstream.log > > > As described in https://issues.apache.org/jira/browse/HDFS-8659, the datanode > is suffering from logging the same repeatedly. Adding log to DFSInputStream, > it outputs as follows: > 2016-07-10 21:31:42,147 INFO > [B.defaultRpcServer.handler=22,queue=1,port=16020] hdfs.DFSClient: > DFSClient_NONMAPREDUCE_1984924661_1 seek > DatanodeInfoWithStorage[10.130.1.29:50010,DS-086bc494-d862-470c-86e8-9cb7929985c6,DISK] > for BP-360285305-10.130.1.11-1444619256876:blk_1109360829_35627143. pos: > 111506876, targetPos: 111506843 > ... > As the pos of this input stream is larger than targetPos(the pos trying to > seek), A new connection to the datanode will be created, the older one will > be closed as a consequence. When the wrong seeking ops are large, the > datanode's block scanner info message is spamming logs, as well as many > connections to the same datanode will be created. > hadoop version: 2.7.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16212) Many connections to datanode are created when doing a large scan
[ https://issues.apache.org/jira/browse/HBASE-16212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383514#comment-15383514 ] Zhihua Deng commented on HBASE-16212: - Thanks [~stack]. From the logging, it implies that different threads share the same DFSInputStream instance, say 'defaultRpcServer.handler=7'(handler7) and 'defaultRpcServer.handler=4'(handler4), for example. The original will prefect the next block header and cache the header into thread. When defaultRpcServer.handler=4 comes, it first checks that the cached header offset is equal to the the block starting offset, unfortunately these two numbers are unequal(-1 != offset). The handler4 knows nothing about the block header, though the header has been prefected by handler7. The handler4 needs to seek the inputstream with the block starting offset for obtaining the header, while the inputstream has been over read by 33 bytes(the header size). So a new connection to datanode should be recreated, the elder one will be closed. When the datanode writes to a closed channel, an socket exception will be raised. When the same case happens frequently, the datanode will be suffered from logging the message described as it is. > Many connections to datanode are created when doing a large scan > - > > Key: HBASE-16212 > URL: https://issues.apache.org/jira/browse/HBASE-16212 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.1.2 >Reporter: Zhihua Deng > Attachments: HBASE-16212.patch, HBASE-16212.v2.patch, > regionserver-dfsinputstream.log > > > As described in https://issues.apache.org/jira/browse/HDFS-8659, the datanode > is suffering from logging the same repeatedly. Adding log to DFSInputStream, > it outputs as follows: > 2016-07-10 21:31:42,147 INFO > [B.defaultRpcServer.handler=22,queue=1,port=16020] hdfs.DFSClient: > DFSClient_NONMAPREDUCE_1984924661_1 seek > DatanodeInfoWithStorage[10.130.1.29:50010,DS-086bc494-d862-470c-86e8-9cb7929985c6,DISK] > for BP-360285305-10.130.1.11-1444619256876:blk_1109360829_35627143. pos: > 111506876, targetPos: 111506843 > ... > As the pos of this input stream is larger than targetPos(the pos trying to > seek), A new connection to the datanode will be created, the older one will > be closed as a consequence. When the wrong seeking ops are large, the > datanode's block scanner info message is spamming logs, as well as many > connections to the same datanode will be created. > hadoop version: 2.7.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16238) It's useless to catch SESSIONEXPIRED exception and retry in RecoverableZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-16238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383425#comment-15383425
]
stack commented on HBASE-16238:
---
[~ndimiduk], [~busbey], and [~mantonov] FYI
> It's useless to catch SESSIONEXPIRED exception and retry in
> RecoverableZooKeeper
>
>
> Key: HBASE-16238
> URL: https://issues.apache.org/jira/browse/HBASE-16238
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Affects Versions: 1.1.5, 1.2.2, 0.98.20
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 1.3.0, 1.1.5, 1.2.2, 0.98.20
>
> Attachments: HBASE-16238.patch, HBASE-16238v2.patch
>
>
> After HBASE-5549, SESSIONEXPIRED exception was caught and retried with other
> zookeeper exceptions like ConnectionLoss. But it is useless to retry when a
> session expired happens, since the retry will never be successful. Though
> there is a config called "zookeeper.recovery.retry" to control retry times,
> in our cases, we set this config to a very big number like "9". When a
> session expired happens, the regionserver should kill itself, but because of
> the retrying, threads of regionserver stuck at trying to reconnect to
> zookeeper, and never properly shut down.
> {code}
> public Stat exists(String path, boolean watch)
> throws KeeperException, InterruptedException {
> TraceScope traceScope = null;
> try {
> traceScope = Trace.startSpan("RecoverableZookeeper.exists");
> RetryCounter retryCounter = retryCounterFactory.create();
> while (true) {
> try {
> return checkZk().exists(path, watch);
> } catch (KeeperException e) {
> switch (e.code()) {
> case CONNECTIONLOSS:
> case SESSIONEXPIRED: //we shouldn't catch this
> case OPERATIONTIMEOUT:
> retryOrThrow(retryCounter, e, "exists");
> break;
> default:
> throw e;
> }
> }
> retryCounter.sleepUntilNextRetry();
> }
> } finally {
> if (traceScope != null) traceScope.close();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16096) Replication keeps accumulating znodes
[ https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16096: --- Status: Patch Available (was: Open) > Replication keeps accumulating znodes > - > > Key: HBASE-16096 > URL: https://issues.apache.org/jira/browse/HBASE-16096 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0, 2.0.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Joseph > Attachments: HBASE-16096.patch > > > If there is an error while creating the replication source on adding the > peer, the source if not added to the in memory list of sources but the > replication peer is. > However, in such a scenario, when you remove the peer, it is deleted from > zookeeper successfully but for removing the in memory list of peers, we wait > for the corresponding sources to get deleted (which as we said don't exist > because of error creating the source). > The problem here is the ordering of operations for adding/removing source and > peer. > Modifying the code to always remove queues from the underlying storage, even > if there exists no sources also requires a small refactoring of > TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty > queue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16241) Allow specification of annotations to use when running check_compatibility.sh
[
https://issues.apache.org/jira/browse/HBASE-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383414#comment-15383414
]
Hadoop QA commented on HBASE-16241:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s {color} | {color:green} The applied patch generated 0 new + 432 unchanged -
4 fixed = 432 total (was 436) {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
2s {color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
47m 55s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 52s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12818690/HBASE-16241_v2.patch |
| JIRA Issue | HBASE-16241 |
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux pomona.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT
Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c9925bc |
| shellcheck | v0.3.3 (This is an old version that has serious bugs. Consider
upgrading.) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/2672/console |
| Powered by | Apache Yetus 0.2.1 http://yetus.apache.org |
This message was automatically generated.
> Allow specification of annotations to use when running check_compatibility.sh
> -
>
> Key: HBASE-16241
> URL: https://issues.apache.org/jira/browse/HBASE-16241
> Project: HBase
> Issue Type: Improvement
> Components: scripts
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-16241.patch, HBASE-16241_v2.patch
>
>
> We already have a {{\-\-all}} option that doesn't include annotations at all,
> so we might as well have the ability to provide a list of annotations to
> override the defaults of {{InterfaceAudience.Public}} and
> {{InterfaceAudience.LimitedPrivate}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16096) Replication keeps accumulating znodes
[ https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16096: --- Status: Open (was: Patch Available) Uploaded the branch-1 patch on accident again > Replication keeps accumulating znodes > - > > Key: HBASE-16096 > URL: https://issues.apache.org/jira/browse/HBASE-16096 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0, 2.0.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Joseph > Attachments: HBASE-16096.patch > > > If there is an error while creating the replication source on adding the > peer, the source if not added to the in memory list of sources but the > replication peer is. > However, in such a scenario, when you remove the peer, it is deleted from > zookeeper successfully but for removing the in memory list of peers, we wait > for the corresponding sources to get deleted (which as we said don't exist > because of error creating the source). > The problem here is the ordering of operations for adding/removing source and > peer. > Modifying the code to always remove queues from the underlying storage, even > if there exists no sources also requires a small refactoring of > TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty > queue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16096) Replication keeps accumulating znodes
[ https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16096: --- Attachment: HBASE-16096.patch > Replication keeps accumulating znodes > - > > Key: HBASE-16096 > URL: https://issues.apache.org/jira/browse/HBASE-16096 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Joseph > Attachments: HBASE-16096.patch > > > If there is an error while creating the replication source on adding the > peer, the source if not added to the in memory list of sources but the > replication peer is. > However, in such a scenario, when you remove the peer, it is deleted from > zookeeper successfully but for removing the in memory list of peers, we wait > for the corresponding sources to get deleted (which as we said don't exist > because of error creating the source). > The problem here is the ordering of operations for adding/removing source and > peer. > Modifying the code to always remove queues from the underlying storage, even > if there exists no sources also requires a small refactoring of > TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty > queue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16096) Replication keeps accumulating znodes
[ https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16096: --- Attachment: (was: HBASE-16096.patch) > Replication keeps accumulating znodes > - > > Key: HBASE-16096 > URL: https://issues.apache.org/jira/browse/HBASE-16096 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Joseph > > If there is an error while creating the replication source on adding the > peer, the source if not added to the in memory list of sources but the > replication peer is. > However, in such a scenario, when you remove the peer, it is deleted from > zookeeper successfully but for removing the in memory list of peers, we wait > for the corresponding sources to get deleted (which as we said don't exist > because of error creating the source). > The problem here is the ordering of operations for adding/removing source and > peer. > Modifying the code to always remove queues from the underlying storage, even > if there exists no sources also requires a small refactoring of > TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty > queue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16210) Add Timestamp class to the hbase-common
[ https://issues.apache.org/jira/browse/HBASE-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383399#comment-15383399 ] Enis Soztutar commented on HBASE-16210: --- Agreed. See the analysis in the documentation. The hard coded values in the patch is carefully chosen. > Add Timestamp class to the hbase-common > --- > > Key: HBASE-16210 > URL: https://issues.apache.org/jira/browse/HBASE-16210 > Project: HBase > Issue Type: Sub-task >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: patch, testing > Attachments: HBASE-16210.master.1.patch, HBASE-16210.master.2.patch, > HBASE-16210.master.3.patch, HBASE-16210.master.4.patch, > HBASE-16210.master.5.patch, HBASE-16210.master.6.patch, > HBASE-16210.master.7.patch, HBASE-16210.master.8.1.patch, > HBASE-16210.master.8.patch > > > This is a sub-issue of > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070]. This JIRA is > a small step towards completely adding Hybrid Logical Clocks(HLC) to HBase. > The main idea of HLC is described in > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070] along with > the motivation of adding it to HBase. > This patch in this issue takes the code from the patch in the parent. > The parent patch is pretty big to review at once. So, plan is to get code > reviewed in smaller patches and > in the process take suggestions and change things if necessary. > What is this patch/issue about ? > This issue attempts to add a timestamp class to hbase-common and timestamp > type to HTable. > This is a part of the attempt to get HLC into HBase. This patch does not > interfere with the current working of HBase. > Why Timestamp Class ? > Timestamp class can be as an abstraction to represent time in Hbase in 64 > bits. > It is just used for manipulating with the 64 bits of the timestamp and is not > concerned about the actual time. > There are three types of timestamps. System time, Custom and HLC. Each one of > it has methods to manipulate the 64 bits of timestamp. > HTable changes: Added a timestamp type property to HTable. This will help > HBase exist in conjunction with old type of timestamp and also the HLC which > will be introduced. The default is set to custom timestamp(current way of > usage of timestamp). default unset timestamp is also custom timestamp as it > should be so. The default timestamp will be changed to HLC when HLC feature > is introduced completely in HBase. > Check HBASE-16210.master.6.patch. > Update: Based on the suggestions, made timestamp enum. Here is the > description of the new changes. > Check the HBASE-16210.master.8.1.patch > 1. Changed the Timestamp Implementation to Enum. > 2. Changed the Timestamp semantics. Instead of HLC, System monotonic and > custom, we now have Hybrid and Physical. System monotonic clock and Custom > clocks can map their timestamps to Physical. HLC clock can map its timestamp > to Hybrid. > 3. The HTableDescriptor will contain clock type(not implemented yet) instead > of timestamp type. As clocks convey the semantics of monotonic increasing and > non decreasing etc. TimestampType doesn't have those semantics enforced, it > just knows what to do with given 64 bits. Therefore, I removed the timestamp > type field in the HTableDescriptor. > Open for suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16238) It's useless to catch SESSIONEXPIRED exception and retry in RecoverableZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-16238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383383#comment-15383383
]
Hudson commented on HBASE-16238:
SUCCESS: Integrated in HBase-1.2-IT #557 (See
[https://builds.apache.org/job/HBase-1.2-IT/557/])
HBASE-16238 It's useless to catch SESSIONEXPIRED exception and retry in (stack:
rev 40980458fb457ade79c26a321946bba99081ae56)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
> It's useless to catch SESSIONEXPIRED exception and retry in
> RecoverableZooKeeper
>
>
> Key: HBASE-16238
> URL: https://issues.apache.org/jira/browse/HBASE-16238
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Affects Versions: 1.1.5, 1.2.2, 0.98.20
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 1.3.0, 1.1.5, 1.2.2, 0.98.20
>
> Attachments: HBASE-16238.patch, HBASE-16238v2.patch
>
>
> After HBASE-5549, SESSIONEXPIRED exception was caught and retried with other
> zookeeper exceptions like ConnectionLoss. But it is useless to retry when a
> session expired happens, since the retry will never be successful. Though
> there is a config called "zookeeper.recovery.retry" to control retry times,
> in our cases, we set this config to a very big number like "9". When a
> session expired happens, the regionserver should kill itself, but because of
> the retrying, threads of regionserver stuck at trying to reconnect to
> zookeeper, and never properly shut down.
> {code}
> public Stat exists(String path, boolean watch)
> throws KeeperException, InterruptedException {
> TraceScope traceScope = null;
> try {
> traceScope = Trace.startSpan("RecoverableZookeeper.exists");
> RetryCounter retryCounter = retryCounterFactory.create();
> while (true) {
> try {
> return checkZk().exists(path, watch);
> } catch (KeeperException e) {
> switch (e.code()) {
> case CONNECTIONLOSS:
> case SESSIONEXPIRED: //we shouldn't catch this
> case OPERATIONTIMEOUT:
> retryOrThrow(retryCounter, e, "exists");
> break;
> default:
> throw e;
> }
> }
> retryCounter.sleepUntilNextRetry();
> }
> } finally {
> if (traceScope != null) traceScope.close();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16096) Replication keeps accumulating znodes
[
https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383358#comment-15383358
]
Hadoop QA commented on HBASE-16096:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-16096 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.2.1/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12818693/HBASE-16096.patch |
| JIRA Issue | HBASE-16096 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/2673/console |
| Powered by | Apache Yetus 0.2.1 http://yetus.apache.org |
This message was automatically generated.
> Replication keeps accumulating znodes
> -
>
> Key: HBASE-16096
> URL: https://issues.apache.org/jira/browse/HBASE-16096
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Joseph
> Attachments: HBASE-16096.patch
>
>
> If there is an error while creating the replication source on adding the
> peer, the source if not added to the in memory list of sources but the
> replication peer is.
> However, in such a scenario, when you remove the peer, it is deleted from
> zookeeper successfully but for removing the in memory list of peers, we wait
> for the corresponding sources to get deleted (which as we said don't exist
> because of error creating the source).
> The problem here is the ordering of operations for adding/removing source and
> peer.
> Modifying the code to always remove queues from the underlying storage, even
> if there exists no sources also requires a small refactoring of
> TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty
> queue
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16237) Blocks for hbase:meta table are not cached in L1 cache
[ https://issues.apache.org/jira/browse/HBASE-16237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383350#comment-15383350 ] Hudson commented on HBASE-16237: FAILURE: Integrated in HBase-1.4 #295 (See [https://builds.apache.org/job/HBase-1.4/295/]) HBASE-16237 Blocks for hbase:meta table are not cached in L1 cache (Lars (tedyu: rev fd8460a4cf5526825af0344da65a84a4db4c80e2) * hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java > Blocks for hbase:meta table are not cached in L1 cache > -- > > Key: HBASE-16237 > URL: https://issues.apache.org/jira/browse/HBASE-16237 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.5 >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 1.3.0, 1.4.0, 1.2.3, 1.1.7 > > Attachments: 16237.branch-1.v1.txt > > > As discussed in this thread: > http://search-hadoop.com/m/YGbbAD1vnnzl5c2=hbase+meta+blocks+in+L1+ > blocks for hbase:meta table are not cached in L1 cache. > The missing setCacheDataInL1(true) call for the metaTableDescriptor() method > is the cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16238) It's useless to catch SESSIONEXPIRED exception and retry in RecoverableZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-16238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383341#comment-15383341
]
Hudson commented on HBASE-16238:
SUCCESS: Integrated in HBase-1.3-IT #760 (See
[https://builds.apache.org/job/HBase-1.3-IT/760/])
HBASE-16238 It's useless to catch SESSIONEXPIRED exception and retry in (stack:
rev a76cede216a1bb7a68c3c96aeeccfcf98f0d8441)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
> It's useless to catch SESSIONEXPIRED exception and retry in
> RecoverableZooKeeper
>
>
> Key: HBASE-16238
> URL: https://issues.apache.org/jira/browse/HBASE-16238
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Affects Versions: 1.1.5, 1.2.2, 0.98.20
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 1.3.0, 1.1.5, 1.2.2, 0.98.20
>
> Attachments: HBASE-16238.patch, HBASE-16238v2.patch
>
>
> After HBASE-5549, SESSIONEXPIRED exception was caught and retried with other
> zookeeper exceptions like ConnectionLoss. But it is useless to retry when a
> session expired happens, since the retry will never be successful. Though
> there is a config called "zookeeper.recovery.retry" to control retry times,
> in our cases, we set this config to a very big number like "9". When a
> session expired happens, the regionserver should kill itself, but because of
> the retrying, threads of regionserver stuck at trying to reconnect to
> zookeeper, and never properly shut down.
> {code}
> public Stat exists(String path, boolean watch)
> throws KeeperException, InterruptedException {
> TraceScope traceScope = null;
> try {
> traceScope = Trace.startSpan("RecoverableZookeeper.exists");
> RetryCounter retryCounter = retryCounterFactory.create();
> while (true) {
> try {
> return checkZk().exists(path, watch);
> } catch (KeeperException e) {
> switch (e.code()) {
> case CONNECTIONLOSS:
> case SESSIONEXPIRED: //we shouldn't catch this
> case OPERATIONTIMEOUT:
> retryOrThrow(retryCounter, e, "exists");
> break;
> default:
> throw e;
> }
> }
> retryCounter.sleepUntilNextRetry();
> }
> } finally {
> if (traceScope != null) traceScope.close();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16237) Blocks for hbase:meta table are not cached in L1 cache
[ https://issues.apache.org/jira/browse/HBASE-16237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383331#comment-15383331 ] Hudson commented on HBASE-16237: FAILURE: Integrated in HBase-1.3 #788 (See [https://builds.apache.org/job/HBase-1.3/788/]) HBASE-16237 Blocks for hbase:meta table are not cached in L1 cache (Lars (tedyu: rev 9467c9fe9b3aa2521fa4a1a6633c2cb8ceab45da) * hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java > Blocks for hbase:meta table are not cached in L1 cache > -- > > Key: HBASE-16237 > URL: https://issues.apache.org/jira/browse/HBASE-16237 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.5 >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 1.3.0, 1.4.0, 1.2.3, 1.1.7 > > Attachments: 16237.branch-1.v1.txt > > > As discussed in this thread: > http://search-hadoop.com/m/YGbbAD1vnnzl5c2=hbase+meta+blocks+in+L1+ > blocks for hbase:meta table are not cached in L1 cache. > The missing setCacheDataInL1(true) call for the metaTableDescriptor() method > is the cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16115) Missing security context in RegionObserver coprocessor when a compaction/split is triggered manually
[ https://issues.apache.org/jira/browse/HBASE-16115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383316#comment-15383316 ] Andrew Purtell commented on HBASE-16115: Yes, close and document > Missing security context in RegionObserver coprocessor when a > compaction/split is triggered manually > > > Key: HBASE-16115 > URL: https://issues.apache.org/jira/browse/HBASE-16115 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.20 >Reporter: Lars Hofhansl > > We ran into an interesting phenomenon which can easily render a cluster > unusable. > We loaded some tests data into a test table and forced a manual compaction > through the UI. We have some compaction hooks implemented in a region > observer, which writes back to another HBase table when the compaction > finishes. We noticed that this coprocessor is not setup correctly, it seems > the security context is missing. > The interesting part is that this _only_ happens when the compaction is > triggere through the UI. Automatic compactions (major or minor) or when > triggered via the HBase shell (folling a kinit) work fine. Only the > UI-triggered compactions cause this issues and lead to essentially > neverending compactions, immovable regions, etc. > Not sure what exactly the issue is, but I wanted to make sure I capture this. > [~apurtell], [~ghelmling], FYI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16237) Blocks for hbase:meta table are not cached in L1 cache
[ https://issues.apache.org/jira/browse/HBASE-16237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383296#comment-15383296 ] Hudson commented on HBASE-16237: SUCCESS: Integrated in HBase-1.2 #675 (See [https://builds.apache.org/job/HBase-1.2/675/]) HBASE-16237 Blocks for hbase:meta table are not cached in L1 cache (Lars (tedyu: rev 683c676b6b9fbde45d3713a1e4e768806454d900) * hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java > Blocks for hbase:meta table are not cached in L1 cache > -- > > Key: HBASE-16237 > URL: https://issues.apache.org/jira/browse/HBASE-16237 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.5 >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 1.3.0, 1.4.0, 1.2.3, 1.1.7 > > Attachments: 16237.branch-1.v1.txt > > > As discussed in this thread: > http://search-hadoop.com/m/YGbbAD1vnnzl5c2=hbase+meta+blocks+in+L1+ > blocks for hbase:meta table are not cached in L1 cache. > The missing setCacheDataInL1(true) call for the metaTableDescriptor() method > is the cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16243) Admin#takeSnapshotAsync should not expose Protos as the return type
[
https://issues.apache.org/jira/browse/HBASE-16243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383291#comment-15383291
]
Ted Yu commented on HBASE-16243:
Thanks for the reminder.
This issue was discovered when producing mega patch based in HBASE-7912 branch.
Looks like there is no need to change the Admin API signature in master branch.
Let me re-work the related part in mega patch.
> Admin#takeSnapshotAsync should not expose Protos as the return type
> ---
>
> Key: HBASE-16243
> URL: https://issues.apache.org/jira/browse/HBASE-16243
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16243.v1.txt
>
>
> In HBASE-7912 branch, Admin#takeSnapshotAsync would trigger the following
> assertion:
> {code}
> 2016-07-18 14:53:56,908 INFO [main]
> hbase.TestInterfaceAudienceAnnotations(340): These are the methods that have
> Protos as the return type
> 2016-07-18 14:53:56,909 INFO [main]
> hbase.TestInterfaceAudienceAnnotations(342):
> org.apache.hadoop.hbase.client.Admin takeSnapshotAsync
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$SnapshotResponse
> {code}
> Admin#takeSnapshotAsync should not expose Protos as the return type
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16198) Enhance backup history command
[
https://issues.apache.org/jira/browse/HBASE-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383287#comment-15383287
]
Hadoop QA commented on HBASE-16198:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s {color}
| {color:red} HBASE-16198 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.2.1/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12818673/HBASE-16198-v2.patch |
| JIRA Issue | HBASE-16198 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/2671/console |
| Powered by | Apache Yetus 0.2.1 http://yetus.apache.org |
This message was automatically generated.
> Enhance backup history command
> --
>
> Key: HBASE-16198
> URL: https://issues.apache.org/jira/browse/HBASE-16198
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-16198-v1.patch, HBASE-16198-v2.patch
>
>
> We need history per table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16232) ITBLL fails on branch-1.3, now loosing actual keys
[
https://issues.apache.org/jira/browse/HBASE-16232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383278#comment-15383278
]
Mikhail Antonov commented on HBASE-16232:
-
(I'm using config where I'm overriding num regions per RS to be way more than
the default of 3 regions - like 50 or 100 per RS)
> ITBLL fails on branch-1.3, now loosing actual keys
> --
>
> Key: HBASE-16232
> URL: https://issues.apache.org/jira/browse/HBASE-16232
> Project: HBase
> Issue Type: Bug
> Components: dataloss, integration tests
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
>
> So I'm running ITBLL off branch-1.3 on recent commit (after [~stack]'s fix
> for fake keys showing up in the scans) with increased number of regions per
> regionserver and seeing the following.
> {quote}
> $Verify​$Counts
> REFERENCED0 4,999,999,994 4,999,999,994
> UNDEFINED 0 3 3
> UNREFERENCED 0 3 3
> {quote}
> So we're loosing some keys. This time those aren't fake:
> {quote}
> undef
> \x89\x10\xE0\xBBx\xF1\xC4\xBAY`\xC4\xD77\x87\x84\x0F 0 1 1
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE30 1 1
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 0 1 1
> unref
> \x15\x1F*f\x92i6\x86\x1D\x8E\xB7\xE1\xC1=\x96\xEF 0 1 1
> \xF4G\xC6E\xD6\xF1\xAB\xB7\xDB\xC0\x94\xF2\xE7mN\xEC 0 1 1
> U\x0F'\x88\x106\x19\x1C\x87Y"\xF3\xE6\xC1\xC8\x15
> {quote}
> Re-running verify step with CM off still shows this issue. Search tool
> reports:
> {quote}
> Total
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE35 0 5
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 4 0 4
> CELL_WITH_MISSING_ROW 15 0 15
> {quote}
> Will post more as I dig into.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16096) Replication keeps accumulating znodes
[ https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16096: --- Status: Open (was: Patch Available) Fixing whitespace issue > Replication keeps accumulating znodes > - > > Key: HBASE-16096 > URL: https://issues.apache.org/jira/browse/HBASE-16096 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0, 2.0.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Joseph > Attachments: HBASE-16096.patch > > > If there is an error while creating the replication source on adding the > peer, the source if not added to the in memory list of sources but the > replication peer is. > However, in such a scenario, when you remove the peer, it is deleted from > zookeeper successfully but for removing the in memory list of peers, we wait > for the corresponding sources to get deleted (which as we said don't exist > because of error creating the source). > The problem here is the ordering of operations for adding/removing source and > peer. > Modifying the code to always remove queues from the underlying storage, even > if there exists no sources also requires a small refactoring of > TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty > queue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16096) Replication keeps accumulating znodes
[ https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16096: --- Status: Patch Available (was: Open) > Replication keeps accumulating znodes > - > > Key: HBASE-16096 > URL: https://issues.apache.org/jira/browse/HBASE-16096 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0, 2.0.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Joseph > Attachments: HBASE-16096.patch > > > If there is an error while creating the replication source on adding the > peer, the source if not added to the in memory list of sources but the > replication peer is. > However, in such a scenario, when you remove the peer, it is deleted from > zookeeper successfully but for removing the in memory list of peers, we wait > for the corresponding sources to get deleted (which as we said don't exist > because of error creating the source). > The problem here is the ordering of operations for adding/removing source and > peer. > Modifying the code to always remove queues from the underlying storage, even > if there exists no sources also requires a small refactoring of > TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty > queue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16096) Replication keeps accumulating znodes
[ https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16096: --- Attachment: HBASE-16096.patch > Replication keeps accumulating znodes > - > > Key: HBASE-16096 > URL: https://issues.apache.org/jira/browse/HBASE-16096 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Joseph > Attachments: HBASE-16096.patch > > > If there is an error while creating the replication source on adding the > peer, the source if not added to the in memory list of sources but the > replication peer is. > However, in such a scenario, when you remove the peer, it is deleted from > zookeeper successfully but for removing the in memory list of peers, we wait > for the corresponding sources to get deleted (which as we said don't exist > because of error creating the source). > The problem here is the ordering of operations for adding/removing source and > peer. > Modifying the code to always remove queues from the underlying storage, even > if there exists no sources also requires a small refactoring of > TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty > queue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16096) Replication keeps accumulating znodes
[ https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16096: --- Attachment: (was: HBASE-16096-v4.patch) > Replication keeps accumulating znodes > - > > Key: HBASE-16096 > URL: https://issues.apache.org/jira/browse/HBASE-16096 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Joseph > Attachments: HBASE-16096.patch > > > If there is an error while creating the replication source on adding the > peer, the source if not added to the in memory list of sources but the > replication peer is. > However, in such a scenario, when you remove the peer, it is deleted from > zookeeper successfully but for removing the in memory list of peers, we wait > for the corresponding sources to get deleted (which as we said don't exist > because of error creating the source). > The problem here is the ordering of operations for adding/removing source and > peer. > Modifying the code to always remove queues from the underlying storage, even > if there exists no sources also requires a small refactoring of > TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty > queue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16232) ITBLL fails on branch-1.3, now loosing actual keys
[
https://issues.apache.org/jira/browse/HBASE-16232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383273#comment-15383273
]
stack commented on HBASE-16232:
---
Trying tip of 1.3 now.
> ITBLL fails on branch-1.3, now loosing actual keys
> --
>
> Key: HBASE-16232
> URL: https://issues.apache.org/jira/browse/HBASE-16232
> Project: HBase
> Issue Type: Bug
> Components: dataloss, integration tests
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
>
> So I'm running ITBLL off branch-1.3 on recent commit (after [~stack]'s fix
> for fake keys showing up in the scans) with increased number of regions per
> regionserver and seeing the following.
> {quote}
> $Verify​$Counts
> REFERENCED0 4,999,999,994 4,999,999,994
> UNDEFINED 0 3 3
> UNREFERENCED 0 3 3
> {quote}
> So we're loosing some keys. This time those aren't fake:
> {quote}
> undef
> \x89\x10\xE0\xBBx\xF1\xC4\xBAY`\xC4\xD77\x87\x84\x0F 0 1 1
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE30 1 1
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 0 1 1
> unref
> \x15\x1F*f\x92i6\x86\x1D\x8E\xB7\xE1\xC1=\x96\xEF 0 1 1
> \xF4G\xC6E\xD6\xF1\xAB\xB7\xDB\xC0\x94\xF2\xE7mN\xEC 0 1 1
> U\x0F'\x88\x106\x19\x1C\x87Y"\xF3\xE6\xC1\xC8\x15
> {quote}
> Re-running verify step with CM off still shows this issue. Search tool
> reports:
> {quote}
> Total
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE35 0 5
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 4 0 4
> CELL_WITH_MISSING_ROW 15 0 15
> {quote}
> Will post more as I dig into.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16243) Admin#takeSnapshotAsync should not expose Protos as the return type
[
https://issues.apache.org/jira/browse/HBASE-16243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383254#comment-15383254
]
Stephen Yuan Jiang edited comment on HBASE-16243 at 7/18/16 11:04 PM:
--
In the master branch, Admin#takeSnapshotAsync returns void type.
{code}
@Override
public void takeSnapshotAsync(SnapshotDescription snapshotDesc) throws
IOException,
SnapshotCreationException {
{code}
I know MasterProtos$SnapshotResponse has a timeout value inside; that is logic
that you changed the return type to long. However, my question is that "is
there any reason that we want to return a different type as the master branch",
because eventually HBASE-7912 branch would merge with master branch?
was (Author: syuanjiang):
In the master branch, Admin#takeSnapshotAsync already return void?
{code}
@Override
public void takeSnapshotAsync(SnapshotDescription snapshotDesc) throws
IOException,
SnapshotCreationException {
{code}
I know MasterProtos$SnapshotResponse has a timeout value inside; that is logic
that you changed the return type to long. However, my question is that "is
there any reason that we want to return a different type as the master branch",
because eventually HBASE-7912 branch would merge with master branch?
> Admin#takeSnapshotAsync should not expose Protos as the return type
> ---
>
> Key: HBASE-16243
> URL: https://issues.apache.org/jira/browse/HBASE-16243
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16243.v1.txt
>
>
> In HBASE-7912 branch, Admin#takeSnapshotAsync would trigger the following
> assertion:
> {code}
> 2016-07-18 14:53:56,908 INFO [main]
> hbase.TestInterfaceAudienceAnnotations(340): These are the methods that have
> Protos as the return type
> 2016-07-18 14:53:56,909 INFO [main]
> hbase.TestInterfaceAudienceAnnotations(342):
> org.apache.hadoop.hbase.client.Admin takeSnapshotAsync
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$SnapshotResponse
> {code}
> Admin#takeSnapshotAsync should not expose Protos as the return type
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16243) Admin#takeSnapshotAsync should not expose Protos as the return type
[
https://issues.apache.org/jira/browse/HBASE-16243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383254#comment-15383254
]
Stephen Yuan Jiang commented on HBASE-16243:
In the master branch, Admin#takeSnapshotAsync already return void?
{code}
@Override
public void takeSnapshotAsync(SnapshotDescription snapshotDesc) throws
IOException,
SnapshotCreationException {
{code}
I know MasterProtos$SnapshotResponse has a timeout value inside; that is logic
that you changed the return type to long. However, my question is that "is
there any reason that we want to return a different type as the master branch",
because eventually HBASE-7912 branch would merge with master branch?
> Admin#takeSnapshotAsync should not expose Protos as the return type
> ---
>
> Key: HBASE-16243
> URL: https://issues.apache.org/jira/browse/HBASE-16243
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16243.v1.txt
>
>
> In HBASE-7912 branch, Admin#takeSnapshotAsync would trigger the following
> assertion:
> {code}
> 2016-07-18 14:53:56,908 INFO [main]
> hbase.TestInterfaceAudienceAnnotations(340): These are the methods that have
> Protos as the return type
> 2016-07-18 14:53:56,909 INFO [main]
> hbase.TestInterfaceAudienceAnnotations(342):
> org.apache.hadoop.hbase.client.Admin takeSnapshotAsync
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$SnapshotResponse
> {code}
> Admin#takeSnapshotAsync should not expose Protos as the return type
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16241) Allow specification of annotations to use when running check_compatibility.sh
[
https://issues.apache.org/jira/browse/HBASE-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383251#comment-15383251
]
stack commented on HBASE-16241:
---
Thanks [~dimaspivak] +1 if v2 comes back clean w/ shellcheck
> Allow specification of annotations to use when running check_compatibility.sh
> -
>
> Key: HBASE-16241
> URL: https://issues.apache.org/jira/browse/HBASE-16241
> Project: HBase
> Issue Type: Improvement
> Components: scripts
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-16241.patch, HBASE-16241_v2.patch
>
>
> We already have a {{\-\-all}} option that doesn't include annotations at all,
> so we might as well have the ability to provide a list of annotations to
> override the defaults of {{InterfaceAudience.Public}} and
> {{InterfaceAudience.LimitedPrivate}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16241) Allow specification of annotations to use when running check_compatibility.sh
[
https://issues.apache.org/jira/browse/HBASE-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak updated HBASE-16241:
Attachment: HBASE-16241_v2.patch
Attaching v2 of the patch, now with less shellcheck unhappiness!
> Allow specification of annotations to use when running check_compatibility.sh
> -
>
> Key: HBASE-16241
> URL: https://issues.apache.org/jira/browse/HBASE-16241
> Project: HBase
> Issue Type: Improvement
> Components: scripts
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-16241.patch, HBASE-16241_v2.patch
>
>
> We already have a {{\-\-all}} option that doesn't include annotations at all,
> so we might as well have the ability to provide a list of annotations to
> override the defaults of {{InterfaceAudience.Public}} and
> {{InterfaceAudience.LimitedPrivate}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16241) Allow specification of annotations to use when running check_compatibility.sh
[
https://issues.apache.org/jira/browse/HBASE-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383231#comment-15383231
]
Dima Spivak commented on HBASE-16241:
-
[~stack], skipping most of the output for brevity, running with just the
{{IA.Public}} classes being checked:
{noformat}
dspivak-MBP:hbase dspivak$ ./dev-support/check_compatibility.sh
--annotation-list=org.apache.hadoop.hbase.classification.InterfaceAudience.Public
branch-1.2 master
...
Running the Java API Compliance Checker...
using Java 1.7.0_71
reading classes branch-1.2/4098045 ...
WARNING: skip "impl" packages (use --keep-internal option to check them)
reading classes master/c9925bc ...
WARNING: skip "impl" packages
comparing classes ...
creating compatibility report ...
binary compatibility: 74.6%
source compatibility: 70.9%
total binary compatibility problems: 320, warnings: 37
total source compatibility problems: 342, warnings: 5
report:
./dev-support/target/compatibility/report/branch-1.2_master_compat_report.html
{noformat}
Rerunning, this time with {{IA.LimitedPrivate}} being checked:
{noformat}
dspivak-MBP:hbase dspivak$ ./dev-support/check_compatibility.sh
--annotation-list=org.apache.hadoop.hbase.classification.InterfaceAudience.LimitedPrivate
-n branch-1.2 master
...
Running the Java API Compliance Checker...
using Java 1.7.0_71
reading classes branch-1.2 ...
WARNING: skip "impl" packages (use --keep-internal option to check them)
reading classes master ...
WARNING: skip "impl" packages
comparing classes ...
creating compatibility report ...
binary compatibility: 57.5%
source compatibility: 52.9%
total binary compatibility problems: 181, warnings: 23
total source compatibility problems: 241, warnings: 2
report:
./dev-support/target/compatibility/report/branch-1.2_master_compat_report.html
{noformat}
Rerunning one last time, where I leave off the restriction of annotations (i.e.
defaults to IA.Public and IA.LimitedPrivate):
{noformat}
dspivak-MBP:hbase dspivak$ ./dev-support/check_compatibility.sh -n branch-1.2
master
...
Running the Java API Compliance Checker...
using Java 1.7.0_71
reading classes branch-1.2 ...
WARNING: skip "impl" packages (use --keep-internal option to check them)
reading classes master ...
WARNING: skip "impl" packages
comparing classes ...
creating compatibility report ...
binary compatibility: 68%
source compatibility: 64%
total binary compatibility problems: 492, warnings: 58
total source compatibility problems: 571, warnings: 7
report:
./dev-support/target/compatibility/report/branch-1.2_master_compat_report.html
{noformat}
Note that # of {{IA.Public}} problems + # of {{IA.LimitedPrivate}} problems
actually adds up to slightly more the number of problems seen if running the
report tool with {{"IA.Public,IA.LimitedPrivate"}}, which I'm assuming has to
do with the way we nest annotations.
As for the {{tr}}, it should be pretty portable. I've got it working on my Mac
with OS X El Capitan and have also confirmed it on a CentOS 6.4 and Ubuntu
14.04 box.
> Allow specification of annotations to use when running check_compatibility.sh
> -
>
> Key: HBASE-16241
> URL: https://issues.apache.org/jira/browse/HBASE-16241
> Project: HBase
> Issue Type: Improvement
> Components: scripts
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-16241.patch
>
>
> We already have a {{\-\-all}} option that doesn't include annotations at all,
> so we might as well have the ability to provide a list of annotations to
> override the defaults of {{InterfaceAudience.Public}} and
> {{InterfaceAudience.LimitedPrivate}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16241) Allow specification of annotations to use when running check_compatibility.sh
[
https://issues.apache.org/jira/browse/HBASE-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383227#comment-15383227
]
Hadoop QA commented on HBASE-16241:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m 5s
{color} | {color:red} The applied patch generated 2 new + 433 unchanged - 3
fixed = 435 total (was 436) {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
2s {color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
42m 5s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 47s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12818631/HBASE-16241.patch |
| JIRA Issue | HBASE-16241 |
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux priapus.apache.org 3.13.0-86-generic #131-Ubuntu SMP Thu May 12
23:33:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c9925bc |
| shellcheck | v0.3.3 (This is an old version that has serious bugs. Consider
upgrading.) |
| shellcheck |
https://builds.apache.org/job/PreCommit-HBASE-Build/2670/artifact/patchprocess/diff-patch-shellcheck.txt
|
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/2670/console |
| Powered by | Apache Yetus 0.2.1 http://yetus.apache.org |
This message was automatically generated.
> Allow specification of annotations to use when running check_compatibility.sh
> -
>
> Key: HBASE-16241
> URL: https://issues.apache.org/jira/browse/HBASE-16241
> Project: HBase
> Issue Type: Improvement
> Components: scripts
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-16241.patch
>
>
> We already have a {{\-\-all}} option that doesn't include annotations at all,
> so we might as well have the ability to provide a list of annotations to
> override the defaults of {{InterfaceAudience.Public}} and
> {{InterfaceAudience.LimitedPrivate}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16243) Admin#takeSnapshotAsync should not expose Protos as the return type
[
https://issues.apache.org/jira/browse/HBASE-16243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16243:
---
Attachment: 16243.v1.txt
> Admin#takeSnapshotAsync should not expose Protos as the return type
> ---
>
> Key: HBASE-16243
> URL: https://issues.apache.org/jira/browse/HBASE-16243
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16243.v1.txt
>
>
> In HBASE-7912 branch, Admin#takeSnapshotAsync would trigger the following
> assertion:
> {code}
> 2016-07-18 14:53:56,908 INFO [main]
> hbase.TestInterfaceAudienceAnnotations(340): These are the methods that have
> Protos as the return type
> 2016-07-18 14:53:56,909 INFO [main]
> hbase.TestInterfaceAudienceAnnotations(342):
> org.apache.hadoop.hbase.client.Admin takeSnapshotAsync
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$SnapshotResponse
> {code}
> Admin#takeSnapshotAsync should not expose Protos as the return type
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16243) Admin#takeSnapshotAsync should not expose Protos as the return type
[
https://issues.apache.org/jira/browse/HBASE-16243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383220#comment-15383220
]
Ted Yu commented on HBASE-16243:
[~enis] [~vrodionov]
Please take a look at the patch.
> Admin#takeSnapshotAsync should not expose Protos as the return type
> ---
>
> Key: HBASE-16243
> URL: https://issues.apache.org/jira/browse/HBASE-16243
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16243.v1.txt
>
>
> In HBASE-7912 branch, Admin#takeSnapshotAsync would trigger the following
> assertion:
> {code}
> 2016-07-18 14:53:56,908 INFO [main]
> hbase.TestInterfaceAudienceAnnotations(340): These are the methods that have
> Protos as the return type
> 2016-07-18 14:53:56,909 INFO [main]
> hbase.TestInterfaceAudienceAnnotations(342):
> org.apache.hadoop.hbase.client.Admin takeSnapshotAsync
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$SnapshotResponse
> {code}
> Admin#takeSnapshotAsync should not expose Protos as the return type
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16243) Admin#takeSnapshotAsync should not expose Protos as the return type
Ted Yu created HBASE-16243:
--
Summary: Admin#takeSnapshotAsync should not expose Protos as the
return type
Key: HBASE-16243
URL: https://issues.apache.org/jira/browse/HBASE-16243
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
In HBASE-7912 branch, Admin#takeSnapshotAsync would trigger the following
assertion:
{code}
2016-07-18 14:53:56,908 INFO [main]
hbase.TestInterfaceAudienceAnnotations(340): These are the methods that have
Protos as the return type
2016-07-18 14:53:56,909 INFO [main]
hbase.TestInterfaceAudienceAnnotations(342):
org.apache.hadoop.hbase.client.Admin takeSnapshotAsync
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$SnapshotResponse
{code}
Admin#takeSnapshotAsync should not expose Protos as the return type
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16242) Upgrade Avro
Ben McCann created HBASE-16242: -- Summary: Upgrade Avro Key: HBASE-16242 URL: https://issues.apache.org/jira/browse/HBASE-16242 Project: HBase Issue Type: Task Components: dependencies Reporter: Ben McCann I'd like to see Avro upgraded to 1.8.1 or at the very least 1.7.7 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16210) Add Timestamp class to the hbase-common
[ https://issues.apache.org/jira/browse/HBASE-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383215#comment-15383215 ] stack commented on HBASE-16210: --- Hardcoded as [~enis] had it. This ain't a config we'd want anyone playing with. Thanks [~saitejar] > Add Timestamp class to the hbase-common > --- > > Key: HBASE-16210 > URL: https://issues.apache.org/jira/browse/HBASE-16210 > Project: HBase > Issue Type: Sub-task >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: patch, testing > Attachments: HBASE-16210.master.1.patch, HBASE-16210.master.2.patch, > HBASE-16210.master.3.patch, HBASE-16210.master.4.patch, > HBASE-16210.master.5.patch, HBASE-16210.master.6.patch, > HBASE-16210.master.7.patch, HBASE-16210.master.8.1.patch, > HBASE-16210.master.8.patch > > > This is a sub-issue of > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070]. This JIRA is > a small step towards completely adding Hybrid Logical Clocks(HLC) to HBase. > The main idea of HLC is described in > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070] along with > the motivation of adding it to HBase. > This patch in this issue takes the code from the patch in the parent. > The parent patch is pretty big to review at once. So, plan is to get code > reviewed in smaller patches and > in the process take suggestions and change things if necessary. > What is this patch/issue about ? > This issue attempts to add a timestamp class to hbase-common and timestamp > type to HTable. > This is a part of the attempt to get HLC into HBase. This patch does not > interfere with the current working of HBase. > Why Timestamp Class ? > Timestamp class can be as an abstraction to represent time in Hbase in 64 > bits. > It is just used for manipulating with the 64 bits of the timestamp and is not > concerned about the actual time. > There are three types of timestamps. System time, Custom and HLC. Each one of > it has methods to manipulate the 64 bits of timestamp. > HTable changes: Added a timestamp type property to HTable. This will help > HBase exist in conjunction with old type of timestamp and also the HLC which > will be introduced. The default is set to custom timestamp(current way of > usage of timestamp). default unset timestamp is also custom timestamp as it > should be so. The default timestamp will be changed to HLC when HLC feature > is introduced completely in HBase. > Check HBASE-16210.master.6.patch. > Update: Based on the suggestions, made timestamp enum. Here is the > description of the new changes. > Check the HBASE-16210.master.8.1.patch > 1. Changed the Timestamp Implementation to Enum. > 2. Changed the Timestamp semantics. Instead of HLC, System monotonic and > custom, we now have Hybrid and Physical. System monotonic clock and Custom > clocks can map their timestamps to Physical. HLC clock can map its timestamp > to Hybrid. > 3. The HTableDescriptor will contain clock type(not implemented yet) instead > of timestamp type. As clocks convey the semantics of monotonic increasing and > non decreasing etc. TimestampType doesn't have those semantics enforced, it > just knows what to do with given 64 bits. Therefore, I removed the timestamp > type field in the HTableDescriptor. > Open for suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-10658) Show stats from different caches on RS web page
[ https://issues.apache.org/jira/browse/HBASE-10658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-10658. --- Resolution: Duplicate Resolving as duplicate of HBASE-4089 Reopen if I have it wrong. > Show stats from different caches on RS web page > --- > > Key: HBASE-10658 > URL: https://issues.apache.org/jira/browse/HBASE-10658 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Biju Nair > Labels: caching > > When deploying L1+L2 cache configurations using multiple cache > implementations (e.g. LRU + BucketCache), it would help users understand the > usage of caches separately on the RS page. This will help them to fine tune > the sizes based on the stats like the cache hit ratio. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16096) Replication keeps accumulating znodes
[
https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383185#comment-15383185
]
Hadoop QA commented on HBASE-16096:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 20s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 0s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 106m 9s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
54s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 158m 42s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12818607/HBASE-16096-v4.patch |
| JIRA Issue | HBASE-16096 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux
[jira] [Commented] (HBASE-16210) Add Timestamp class to the hbase-common
[ https://issues.apache.org/jira/browse/HBASE-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383183#comment-15383183 ] Sai Teja Ranuva commented on HBASE-16210: - [~stack] [~enis] Should the bits for Logical time and Physical time be flexible or hardcoded as it is now ? If we need it to be flexible, we cannot use an enum for timestamp. > Add Timestamp class to the hbase-common > --- > > Key: HBASE-16210 > URL: https://issues.apache.org/jira/browse/HBASE-16210 > Project: HBase > Issue Type: Sub-task >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: patch, testing > Attachments: HBASE-16210.master.1.patch, HBASE-16210.master.2.patch, > HBASE-16210.master.3.patch, HBASE-16210.master.4.patch, > HBASE-16210.master.5.patch, HBASE-16210.master.6.patch, > HBASE-16210.master.7.patch, HBASE-16210.master.8.1.patch, > HBASE-16210.master.8.patch > > > This is a sub-issue of > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070]. This JIRA is > a small step towards completely adding Hybrid Logical Clocks(HLC) to HBase. > The main idea of HLC is described in > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070] along with > the motivation of adding it to HBase. > This patch in this issue takes the code from the patch in the parent. > The parent patch is pretty big to review at once. So, plan is to get code > reviewed in smaller patches and > in the process take suggestions and change things if necessary. > What is this patch/issue about ? > This issue attempts to add a timestamp class to hbase-common and timestamp > type to HTable. > This is a part of the attempt to get HLC into HBase. This patch does not > interfere with the current working of HBase. > Why Timestamp Class ? > Timestamp class can be as an abstraction to represent time in Hbase in 64 > bits. > It is just used for manipulating with the 64 bits of the timestamp and is not > concerned about the actual time. > There are three types of timestamps. System time, Custom and HLC. Each one of > it has methods to manipulate the 64 bits of timestamp. > HTable changes: Added a timestamp type property to HTable. This will help > HBase exist in conjunction with old type of timestamp and also the HLC which > will be introduced. The default is set to custom timestamp(current way of > usage of timestamp). default unset timestamp is also custom timestamp as it > should be so. The default timestamp will be changed to HLC when HLC feature > is introduced completely in HBase. > Check HBASE-16210.master.6.patch. > Update: Based on the suggestions, made timestamp enum. Here is the > description of the new changes. > Check the HBASE-16210.master.8.1.patch > 1. Changed the Timestamp Implementation to Enum. > 2. Changed the Timestamp semantics. Instead of HLC, System monotonic and > custom, we now have Hybrid and Physical. System monotonic clock and Custom > clocks can map their timestamps to Physical. HLC clock can map its timestamp > to Hybrid. > 3. The HTableDescriptor will contain clock type(not implemented yet) instead > of timestamp type. As clocks convey the semantics of monotonic increasing and > non decreasing etc. TimestampType doesn't have those semantics enforced, it > just knows what to do with given 64 bits. Therefore, I removed the timestamp > type field in the HTableDescriptor. > Open for suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16052) Improve HBaseFsck Scalability
[
https://issues.apache.org/jira/browse/HBASE-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383179#comment-15383179
]
Hadoop QA commented on HBASE-16052:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
30s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} 0.98 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} 0.98 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} 0.98 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 40s
{color} | {color:red} hbase-server in 0.98 has 85 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 46s
{color} | {color:red} hbase-server in 0.98 failed with JDK v1.8.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} 0.98 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 24s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 33s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 122m 2s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 159m 37s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestZooKeeper |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12818610/HBASE-16052-0.98.v3.patch
|
| JIRA Issue | HBASE-16052 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | 0.98 / 8da8565 |
| Default Java | 1.7.0_80 |
|
[jira] [Updated] (HBASE-16198) Enhance backup history command
[ https://issues.apache.org/jira/browse/HBASE-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16198: -- Attachment: HBASE-16198-v2.patch Patch v2. cc: [~tedyu] > Enhance backup history command > -- > > Key: HBASE-16198 > URL: https://issues.apache.org/jira/browse/HBASE-16198 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16198-v1.patch, HBASE-16198-v2.patch > > > We need history per table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16210) Add Timestamp class to the hbase-common
[ https://issues.apache.org/jira/browse/HBASE-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383162#comment-15383162 ] stack commented on HBASE-16210: --- Ok. Good. > Add Timestamp class to the hbase-common > --- > > Key: HBASE-16210 > URL: https://issues.apache.org/jira/browse/HBASE-16210 > Project: HBase > Issue Type: Sub-task >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: patch, testing > Attachments: HBASE-16210.master.1.patch, HBASE-16210.master.2.patch, > HBASE-16210.master.3.patch, HBASE-16210.master.4.patch, > HBASE-16210.master.5.patch, HBASE-16210.master.6.patch, > HBASE-16210.master.7.patch, HBASE-16210.master.8.1.patch, > HBASE-16210.master.8.patch > > > This is a sub-issue of > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070]. This JIRA is > a small step towards completely adding Hybrid Logical Clocks(HLC) to HBase. > The main idea of HLC is described in > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070] along with > the motivation of adding it to HBase. > This patch in this issue takes the code from the patch in the parent. > The parent patch is pretty big to review at once. So, plan is to get code > reviewed in smaller patches and > in the process take suggestions and change things if necessary. > What is this patch/issue about ? > This issue attempts to add a timestamp class to hbase-common and timestamp > type to HTable. > This is a part of the attempt to get HLC into HBase. This patch does not > interfere with the current working of HBase. > Why Timestamp Class ? > Timestamp class can be as an abstraction to represent time in Hbase in 64 > bits. > It is just used for manipulating with the 64 bits of the timestamp and is not > concerned about the actual time. > There are three types of timestamps. System time, Custom and HLC. Each one of > it has methods to manipulate the 64 bits of timestamp. > HTable changes: Added a timestamp type property to HTable. This will help > HBase exist in conjunction with old type of timestamp and also the HLC which > will be introduced. The default is set to custom timestamp(current way of > usage of timestamp). default unset timestamp is also custom timestamp as it > should be so. The default timestamp will be changed to HLC when HLC feature > is introduced completely in HBase. > Check HBASE-16210.master.6.patch. > Update: Based on the suggestions, made timestamp enum. Here is the > description of the new changes. > Check the HBASE-16210.master.8.1.patch > 1. Changed the Timestamp Implementation to Enum. > 2. Changed the Timestamp semantics. Instead of HLC, System monotonic and > custom, we now have Hybrid and Physical. System monotonic clock and Custom > clocks can map their timestamps to Physical. HLC clock can map its timestamp > to Hybrid. > 3. The HTableDescriptor will contain clock type(not implemented yet) instead > of timestamp type. As clocks convey the semantics of monotonic increasing and > non decreasing etc. TimestampType doesn't have those semantics enforced, it > just knows what to do with given 64 bits. Therefore, I removed the timestamp > type field in the HTableDescriptor. > Open for suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16212) Many connections to datanode are created when doing a large scan
[ https://issues.apache.org/jira/browse/HBASE-16212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383158#comment-15383158 ] stack commented on HBASE-16212: --- Do you think we'll connect to the DN less freqently with this patch in place? Thanks. > Many connections to datanode are created when doing a large scan > - > > Key: HBASE-16212 > URL: https://issues.apache.org/jira/browse/HBASE-16212 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.1.2 >Reporter: Zhihua Deng > Attachments: HBASE-16212.patch, HBASE-16212.v2.patch, > regionserver-dfsinputstream.log > > > As described in https://issues.apache.org/jira/browse/HDFS-8659, the datanode > is suffering from logging the same repeatedly. Adding log to DFSInputStream, > it outputs as follows: > 2016-07-10 21:31:42,147 INFO > [B.defaultRpcServer.handler=22,queue=1,port=16020] hdfs.DFSClient: > DFSClient_NONMAPREDUCE_1984924661_1 seek > DatanodeInfoWithStorage[10.130.1.29:50010,DS-086bc494-d862-470c-86e8-9cb7929985c6,DISK] > for BP-360285305-10.130.1.11-1444619256876:blk_1109360829_35627143. pos: > 111506876, targetPos: 111506843 > ... > As the pos of this input stream is larger than targetPos(the pos trying to > seek), A new connection to the datanode will be created, the older one will > be closed as a consequence. When the wrong seeking ops are large, the > datanode's block scanner info message is spamming logs, as well as many > connections to the same datanode will be created. > hadoop version: 2.7.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16212) Many connections to datanode are created when doing a large scan
[ https://issues.apache.org/jira/browse/HBASE-16212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383156#comment-15383156 ] stack commented on HBASE-16212: --- Tell us more [~dengzh]? I think I get it. The thread local often held reference to a header from another file altogether and this was making for all the logging you were seeing? Looking at the patch you are making substantial changes removing the thread local that caches last header read by thread and instead doing the caching on the fsreaderimpl which is better in some ways but now we have a synchronization bottleneck for all threads to pass through. What you thinking here? You thinking it will be rare that more than one thread will be going against same file? Have you run with this patch? Is this patch for branch-1.1? Does master still have same issue (has same basic form but a bunch of refactoring has gone on in here). This patch looks like a nice one. Thanks. > Many connections to datanode are created when doing a large scan > - > > Key: HBASE-16212 > URL: https://issues.apache.org/jira/browse/HBASE-16212 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.1.2 >Reporter: Zhihua Deng > Attachments: HBASE-16212.patch, HBASE-16212.v2.patch, > regionserver-dfsinputstream.log > > > As described in https://issues.apache.org/jira/browse/HDFS-8659, the datanode > is suffering from logging the same repeatedly. Adding log to DFSInputStream, > it outputs as follows: > 2016-07-10 21:31:42,147 INFO > [B.defaultRpcServer.handler=22,queue=1,port=16020] hdfs.DFSClient: > DFSClient_NONMAPREDUCE_1984924661_1 seek > DatanodeInfoWithStorage[10.130.1.29:50010,DS-086bc494-d862-470c-86e8-9cb7929985c6,DISK] > for BP-360285305-10.130.1.11-1444619256876:blk_1109360829_35627143. pos: > 111506876, targetPos: 111506843 > ... > As the pos of this input stream is larger than targetPos(the pos trying to > seek), A new connection to the datanode will be created, the older one will > be closed as a consequence. When the wrong seeking ops are large, the > datanode's block scanner info message is spamming logs, as well as many > connections to the same datanode will be created. > hadoop version: 2.7.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383129#comment-15383129
]
Enis Soztutar commented on HBASE-14882:
---
Replicate what we have done for our C++ client in HBASE-15945?
Basically a {{CellImpl}} which looks like our PB cell object, individual row,
cf, column, value fields.
> Provide a Put API that adds the provided family, qualifier, value without
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Xiang Li
>
> In the Put API, we have addImmutable()
> {code}
> /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
> public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation the row, family. qualifier and value are still
> being copied locally to create kv.
> Hopefully we should provide an API that truely uses immutable family,
> qualifier and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16238) It's useless to catch SESSIONEXPIRED exception and retry in RecoverableZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-16238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16238:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.3.0
Status: Resolved (was: Patch Available)
Pushed to 0.98 and branch-1.1+ Thanks [~allan163] I pushed the first patch, not
the v2 with the findbugs breaking change. Thanks.
> It's useless to catch SESSIONEXPIRED exception and retry in
> RecoverableZooKeeper
>
>
> Key: HBASE-16238
> URL: https://issues.apache.org/jira/browse/HBASE-16238
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Affects Versions: 1.1.5, 1.2.2, 0.98.20
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 1.3.0, 0.98.20, 1.2.2, 1.1.5
>
> Attachments: HBASE-16238.patch, HBASE-16238v2.patch
>
>
> After HBASE-5549, SESSIONEXPIRED exception was caught and retried with other
> zookeeper exceptions like ConnectionLoss. But it is useless to retry when a
> session expired happens, since the retry will never be successful. Though
> there is a config called "zookeeper.recovery.retry" to control retry times,
> in our cases, we set this config to a very big number like "9". When a
> session expired happens, the regionserver should kill itself, but because of
> the retrying, threads of regionserver stuck at trying to reconnect to
> zookeeper, and never properly shut down.
> {code}
> public Stat exists(String path, boolean watch)
> throws KeeperException, InterruptedException {
> TraceScope traceScope = null;
> try {
> traceScope = Trace.startSpan("RecoverableZookeeper.exists");
> RetryCounter retryCounter = retryCounterFactory.create();
> while (true) {
> try {
> return checkZk().exists(path, watch);
> } catch (KeeperException e) {
> switch (e.code()) {
> case CONNECTIONLOSS:
> case SESSIONEXPIRED: //we shouldn't catch this
> case OPERATIONTIMEOUT:
> retryOrThrow(retryCounter, e, "exists");
> break;
> default:
> throw e;
> }
> }
> retryCounter.sleepUntilNextRetry();
> }
> } finally {
> if (traceScope != null) traceScope.close();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15453) [Performance] Considering reverting HBASE-10015 - reinstate synchronized in StoreScanner
[ https://issues.apache.org/jira/browse/HBASE-15453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383116#comment-15383116 ] stack commented on HBASE-15453: --- [~lhofhansl] yes > [Performance] Considering reverting HBASE-10015 - reinstate synchronized in > StoreScanner > > > Key: HBASE-15453 > URL: https://issues.apache.org/jira/browse/HBASE-15453 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Attachments: 15453-0.98.txt > > > In HBASE-10015 back then I found that intrinsic locks (synchronized) in > StoreScanner are slower that explicit locks. > I was surprised by this. To make sure I added a simple perf test and many > folks ran it on their machines. All found that explicit locks were faster. > Now... I just ran that test again. On the latest JDK8 I find that now the > intrinsic locks are significantly faster: > (OpenJDK Runtime Environment (build 1.8.0_72-b15)) > Explicit locks: > 10 runs mean:2223.6 sigma:72.29412147609237 > Intrinsic locks: > 10 runs mean:1865.3 sigma:32.63755505548784 > I confirmed the same with timing some Phoenix scans. We can save a bunch of > time by changing this back > Arrghhh... So maybe it's time to revert this now...? > (Note that in trunk due to [~ram_krish]'s work, we do not lock in > StoreScanner anymore) > I'll attach the perf test and a patch that changes lock to synchronized, if > some folks could run this on 0.98, that'd be great. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16210) Add Timestamp class to the hbase-common
[ https://issues.apache.org/jira/browse/HBASE-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383118#comment-15383118 ] Sai Teja Ranuva commented on HBASE-16210: - [~stack] [~enis] Timestamp will be internal and only clock class will talk to it. Clock type would drive the semantics of the time. So, user will have to know only which clock type to choose. > Add Timestamp class to the hbase-common > --- > > Key: HBASE-16210 > URL: https://issues.apache.org/jira/browse/HBASE-16210 > Project: HBase > Issue Type: Sub-task >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: patch, testing > Attachments: HBASE-16210.master.1.patch, HBASE-16210.master.2.patch, > HBASE-16210.master.3.patch, HBASE-16210.master.4.patch, > HBASE-16210.master.5.patch, HBASE-16210.master.6.patch, > HBASE-16210.master.7.patch, HBASE-16210.master.8.1.patch, > HBASE-16210.master.8.patch > > > This is a sub-issue of > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070]. This JIRA is > a small step towards completely adding Hybrid Logical Clocks(HLC) to HBase. > The main idea of HLC is described in > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070] along with > the motivation of adding it to HBase. > This patch in this issue takes the code from the patch in the parent. > The parent patch is pretty big to review at once. So, plan is to get code > reviewed in smaller patches and > in the process take suggestions and change things if necessary. > What is this patch/issue about ? > This issue attempts to add a timestamp class to hbase-common and timestamp > type to HTable. > This is a part of the attempt to get HLC into HBase. This patch does not > interfere with the current working of HBase. > Why Timestamp Class ? > Timestamp class can be as an abstraction to represent time in Hbase in 64 > bits. > It is just used for manipulating with the 64 bits of the timestamp and is not > concerned about the actual time. > There are three types of timestamps. System time, Custom and HLC. Each one of > it has methods to manipulate the 64 bits of timestamp. > HTable changes: Added a timestamp type property to HTable. This will help > HBase exist in conjunction with old type of timestamp and also the HLC which > will be introduced. The default is set to custom timestamp(current way of > usage of timestamp). default unset timestamp is also custom timestamp as it > should be so. The default timestamp will be changed to HLC when HLC feature > is introduced completely in HBase. > Check HBASE-16210.master.6.patch. > Update: Based on the suggestions, made timestamp enum. Here is the > description of the new changes. > Check the HBASE-16210.master.8.1.patch > 1. Changed the Timestamp Implementation to Enum. > 2. Changed the Timestamp semantics. Instead of HLC, System monotonic and > custom, we now have Hybrid and Physical. System monotonic clock and Custom > clocks can map their timestamps to Physical. HLC clock can map its timestamp > to Hybrid. > 3. The HTableDescriptor will contain clock type(not implemented yet) instead > of timestamp type. As clocks convey the semantics of monotonic increasing and > non decreasing etc. TimestampType doesn't have those semantics enforced, it > just knows what to do with given 64 bits. Therefore, I removed the timestamp > type field in the HTableDescriptor. > Open for suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14549) Simplify scanner stack reset logic
[ https://issues.apache.org/jira/browse/HBASE-14549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383114#comment-15383114 ] stack commented on HBASE-14549: --- So, this stuff is no longer complex? Nothing to do based off your findings [~lhofhansl]? > Simplify scanner stack reset logic > -- > > Key: HBASE-14549 > URL: https://issues.apache.org/jira/browse/HBASE-14549 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 14549-0.98.txt > > > Looking at the code, I find that the logic is unnecessarily complex. > We indicate in updateReaders that the scanner stack needs to be reset. Then > almost all store scanner (and derived classes) methods need to check and > actually reset the scanner stack. > Compaction are rare, we should reset the scanner stack in update readers, and > hence avoid needing to check in all methods. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383095#comment-15383095
]
stack commented on HBASE-14882:
---
Does it need to have a name? Can we not just make an anonymous Cell?
> Provide a Put API that adds the provided family, qualifier, value without
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Xiang Li
>
> In the Put API, we have addImmutable()
> {code}
> /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
> public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation the row, family. qualifier and value are still
> being copied locally to create kv.
> Hopefully we should provide an API that truely uses immutable family,
> qualifier and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14882:
--
Summary: Provide a Put API that adds the provided family, qualifier, value
without copying (was: Provide a Put API that adds the provided family,
qualifier, value without coping)
> Provide a Put API that adds the provided family, qualifier, value without
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Xiang Li
>
> In the Put API, we have addImmutable()
> {code}
> /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
> public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation the row, family. qualifier and value are still
> being copied locally to create kv.
> Hopefully we should provide an API that truely uses immutable family,
> qualifier and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16241) Allow specification of annotations to use when running check_compatibility.sh
[
https://issues.apache.org/jira/browse/HBASE-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383084#comment-15383084
]
Sean Busbey commented on HBASE-16241:
-
+1 will commit later this evening unless someone would like to give more
feedback.
> Allow specification of annotations to use when running check_compatibility.sh
> -
>
> Key: HBASE-16241
> URL: https://issues.apache.org/jira/browse/HBASE-16241
> Project: HBase
> Issue Type: Improvement
> Components: scripts
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-16241.patch
>
>
> We already have a {{\-\-all}} option that doesn't include annotations at all,
> so we might as well have the ability to provide a list of annotations to
> override the defaults of {{InterfaceAudience.Public}} and
> {{InterfaceAudience.LimitedPrivate}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16241) Allow specification of annotations to use when running check_compatibility.sh
[
https://issues.apache.org/jira/browse/HBASE-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383086#comment-15383086
]
Sean Busbey commented on HBASE-16241:
-
Just saw stack's feedback. :)
the precommit check should run it through shellcheck, which will complain if it
isn't portable.
> Allow specification of annotations to use when running check_compatibility.sh
> -
>
> Key: HBASE-16241
> URL: https://issues.apache.org/jira/browse/HBASE-16241
> Project: HBase
> Issue Type: Improvement
> Components: scripts
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-16241.patch
>
>
> We already have a {{\-\-all}} option that doesn't include annotations at all,
> so we might as well have the ability to provide a list of annotations to
> override the defaults of {{InterfaceAudience.Public}} and
> {{InterfaceAudience.LimitedPrivate}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16241) Allow specification of annotations to use when running check_compatibility.sh
[
https://issues.apache.org/jira/browse/HBASE-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383076#comment-15383076
]
stack commented on HBASE-16241:
---
Looks good. Does it work (paste in a bit of evidence...)
Is this portable? ANNOTATION_LIST=($(tr "," "\n" <<< "${2}"))
> Allow specification of annotations to use when running check_compatibility.sh
> -
>
> Key: HBASE-16241
> URL: https://issues.apache.org/jira/browse/HBASE-16241
> Project: HBase
> Issue Type: Improvement
> Components: scripts
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-16241.patch
>
>
> We already have a {{\-\-all}} option that doesn't include annotations at all,
> so we might as well have the ability to provide a list of annotations to
> override the defaults of {{InterfaceAudience.Public}} and
> {{InterfaceAudience.LimitedPrivate}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16210) Add Timestamp class to the hbase-common
[ https://issues.apache.org/jira/browse/HBASE-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383056#comment-15383056 ] stack commented on HBASE-16210: --- See the above [~saitejar] You were just talking of exposing clock as a config on table descriptor but Timestamp would then be an internal, single implmentation whose behavior would be shaped by the clock-type chosen? > Add Timestamp class to the hbase-common > --- > > Key: HBASE-16210 > URL: https://issues.apache.org/jira/browse/HBASE-16210 > Project: HBase > Issue Type: Sub-task >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: patch, testing > Attachments: HBASE-16210.master.1.patch, HBASE-16210.master.2.patch, > HBASE-16210.master.3.patch, HBASE-16210.master.4.patch, > HBASE-16210.master.5.patch, HBASE-16210.master.6.patch, > HBASE-16210.master.7.patch, HBASE-16210.master.8.1.patch, > HBASE-16210.master.8.patch > > > This is a sub-issue of > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070]. This JIRA is > a small step towards completely adding Hybrid Logical Clocks(HLC) to HBase. > The main idea of HLC is described in > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070] along with > the motivation of adding it to HBase. > This patch in this issue takes the code from the patch in the parent. > The parent patch is pretty big to review at once. So, plan is to get code > reviewed in smaller patches and > in the process take suggestions and change things if necessary. > What is this patch/issue about ? > This issue attempts to add a timestamp class to hbase-common and timestamp > type to HTable. > This is a part of the attempt to get HLC into HBase. This patch does not > interfere with the current working of HBase. > Why Timestamp Class ? > Timestamp class can be as an abstraction to represent time in Hbase in 64 > bits. > It is just used for manipulating with the 64 bits of the timestamp and is not > concerned about the actual time. > There are three types of timestamps. System time, Custom and HLC. Each one of > it has methods to manipulate the 64 bits of timestamp. > HTable changes: Added a timestamp type property to HTable. This will help > HBase exist in conjunction with old type of timestamp and also the HLC which > will be introduced. The default is set to custom timestamp(current way of > usage of timestamp). default unset timestamp is also custom timestamp as it > should be so. The default timestamp will be changed to HLC when HLC feature > is introduced completely in HBase. > Check HBASE-16210.master.6.patch. > Update: Based on the suggestions, made timestamp enum. Here is the > description of the new changes. > Check the HBASE-16210.master.8.1.patch > 1. Changed the Timestamp Implementation to Enum. > 2. Changed the Timestamp semantics. Instead of HLC, System monotonic and > custom, we now have Hybrid and Physical. System monotonic clock and Custom > clocks can map their timestamps to Physical. HLC clock can map its timestamp > to Hybrid. > 3. The HTableDescriptor will contain clock type(not implemented yet) instead > of timestamp type. As clocks convey the semantics of monotonic increasing and > non decreasing etc. TimestampType doesn't have those semantics enforced, it > just knows what to do with given 64 bits. Therefore, I removed the timestamp > type field in the HTableDescriptor. > Open for suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383041#comment-15383041 ] Ted Yu commented on HBASE-14123: Planning to re-use this JIRA for running test suite against master branch. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-v14.txt, HBASE-14123-for-7912-v1.patch, > HBASE-14123-for-7912-v6.patch, HBASE-14123-v1.patch, HBASE-14123-v10.patch, > HBASE-14123-v11.patch, HBASE-14123-v12.patch, HBASE-14123-v13.patch, > HBASE-14123-v15.patch, HBASE-14123-v16.patch, HBASE-14123-v2.patch, > HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch, > HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16237) Blocks for hbase:meta table are not cached in L1 cache
[ https://issues.apache.org/jira/browse/HBASE-16237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383004#comment-15383004 ] Hudson commented on HBASE-16237: SUCCESS: Integrated in HBase-1.2-IT #556 (See [https://builds.apache.org/job/HBase-1.2-IT/556/]) HBASE-16237 Blocks for hbase:meta table are not cached in L1 cache (Lars (tedyu: rev 683c676b6b9fbde45d3713a1e4e768806454d900) * hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java > Blocks for hbase:meta table are not cached in L1 cache > -- > > Key: HBASE-16237 > URL: https://issues.apache.org/jira/browse/HBASE-16237 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.5 >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 1.3.0, 1.4.0, 1.2.3, 1.1.7 > > Attachments: 16237.branch-1.v1.txt > > > As discussed in this thread: > http://search-hadoop.com/m/YGbbAD1vnnzl5c2=hbase+meta+blocks+in+L1+ > blocks for hbase:meta table are not cached in L1 cache. > The missing setCacheDataInL1(true) call for the metaTableDescriptor() method > is the cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14921) Memory optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382953#comment-15382953 ] Anastasia Braginsky commented on HBASE-14921: - Also, got a comment saying that scan (being done for deciding whether to compact or to flatten) is too costly. However I empirical results we truly do not see any performance issues due to this scan. We probably can add some sophisticated logic to predict the duplicate ratio, but I suggest first to proof this scan is causing the performance degradation. Any thoughts about that? > Memory optimizations > > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, > HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, > InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16241) Allow specification of annotations to use when running check_compatibility.sh
[
https://issues.apache.org/jira/browse/HBASE-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak updated HBASE-16241:
Status: Patch Available (was: Open)
> Allow specification of annotations to use when running check_compatibility.sh
> -
>
> Key: HBASE-16241
> URL: https://issues.apache.org/jira/browse/HBASE-16241
> Project: HBase
> Issue Type: Improvement
> Components: scripts
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-16241.patch
>
>
> We already have a {{\-\-all}} option that doesn't include annotations at all,
> so we might as well have the ability to provide a list of annotations to
> override the defaults of {{InterfaceAudience.Public}} and
> {{InterfaceAudience.LimitedPrivate}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16241) Allow specification of annotations to use when running check_compatibility.sh
[
https://issues.apache.org/jira/browse/HBASE-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak updated HBASE-16241:
Attachment: HBASE-16241.patch
Uploading patch that adds a {{\-l}} / {{\-\-annotation-list}} option to
{{check_compatibility.sh}}.
> Allow specification of annotations to use when running check_compatibility.sh
> -
>
> Key: HBASE-16241
> URL: https://issues.apache.org/jira/browse/HBASE-16241
> Project: HBase
> Issue Type: Improvement
> Components: scripts
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-16241.patch
>
>
> We already have a {{\-\-all}} option that doesn't include annotations at all,
> so we might as well have the ability to provide a list of annotations to
> override the defaults of {{InterfaceAudience.Public}} and
> {{InterfaceAudience.LimitedPrivate}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16210) Add Timestamp class to the hbase-common
[ https://issues.apache.org/jira/browse/HBASE-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382920#comment-15382920 ] Enis Soztutar commented on HBASE-16210: --- Clock algorithm (HLC, SYTEM_MONOTONIC, etc) and Timestamp are different but closely related concepts. For example the hybrid timestamp type is hard coded to encode/decode with 22bit PTs. It cannot work with anything else as is in the patch. Thus it is only suitable to be used with HL clock. The clock in the original patch is deliberately not exposed to the user at all. I fear doing clock type differently than timestamp type will make it so that the user has to know about two different concepts rather than one. The user should know about the clock type in the table as well as how it maps to the timestamp type. > Add Timestamp class to the hbase-common > --- > > Key: HBASE-16210 > URL: https://issues.apache.org/jira/browse/HBASE-16210 > Project: HBase > Issue Type: Sub-task >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: patch, testing > Attachments: HBASE-16210.master.1.patch, HBASE-16210.master.2.patch, > HBASE-16210.master.3.patch, HBASE-16210.master.4.patch, > HBASE-16210.master.5.patch, HBASE-16210.master.6.patch, > HBASE-16210.master.7.patch, HBASE-16210.master.8.1.patch, > HBASE-16210.master.8.patch > > > This is a sub-issue of > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070]. This JIRA is > a small step towards completely adding Hybrid Logical Clocks(HLC) to HBase. > The main idea of HLC is described in > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070] along with > the motivation of adding it to HBase. > This patch in this issue takes the code from the patch in the parent. > The parent patch is pretty big to review at once. So, plan is to get code > reviewed in smaller patches and > in the process take suggestions and change things if necessary. > What is this patch/issue about ? > This issue attempts to add a timestamp class to hbase-common and timestamp > type to HTable. > This is a part of the attempt to get HLC into HBase. This patch does not > interfere with the current working of HBase. > Why Timestamp Class ? > Timestamp class can be as an abstraction to represent time in Hbase in 64 > bits. > It is just used for manipulating with the 64 bits of the timestamp and is not > concerned about the actual time. > There are three types of timestamps. System time, Custom and HLC. Each one of > it has methods to manipulate the 64 bits of timestamp. > HTable changes: Added a timestamp type property to HTable. This will help > HBase exist in conjunction with old type of timestamp and also the HLC which > will be introduced. The default is set to custom timestamp(current way of > usage of timestamp). default unset timestamp is also custom timestamp as it > should be so. The default timestamp will be changed to HLC when HLC feature > is introduced completely in HBase. > Check HBASE-16210.master.6.patch. > Update: Based on the suggestions, made timestamp enum. Here is the > description of the new changes. > Check the HBASE-16210.master.8.1.patch > 1. Changed the Timestamp Implementation to Enum. > 2. Changed the Timestamp semantics. Instead of HLC, System monotonic and > custom, we now have Hybrid and Physical. System monotonic clock and Custom > clocks can map their timestamps to Physical. HLC clock can map its timestamp > to Hybrid. > 3. The HTableDescriptor will contain clock type(not implemented yet) instead > of timestamp type. As clocks convey the semantics of monotonic increasing and > non decreasing etc. TimestampType doesn't have those semantics enforced, it > just knows what to do with given 64 bits. Therefore, I removed the timestamp > type field in the HTableDescriptor. > Open for suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16210) Add Timestamp class to the hbase-common
[ https://issues.apache.org/jira/browse/HBASE-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382898#comment-15382898 ] Enis Soztutar commented on HBASE-16210: --- This is not for transactions per-se, but to fix some broken read-after-write semantics in places where we assume clocks never go back and clocks are roughly in-sync in the cluster. Design doc attached at parent: HBASE-14070. > Add Timestamp class to the hbase-common > --- > > Key: HBASE-16210 > URL: https://issues.apache.org/jira/browse/HBASE-16210 > Project: HBase > Issue Type: Sub-task >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: patch, testing > Attachments: HBASE-16210.master.1.patch, HBASE-16210.master.2.patch, > HBASE-16210.master.3.patch, HBASE-16210.master.4.patch, > HBASE-16210.master.5.patch, HBASE-16210.master.6.patch, > HBASE-16210.master.7.patch, HBASE-16210.master.8.1.patch, > HBASE-16210.master.8.patch > > > This is a sub-issue of > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070]. This JIRA is > a small step towards completely adding Hybrid Logical Clocks(HLC) to HBase. > The main idea of HLC is described in > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070] along with > the motivation of adding it to HBase. > This patch in this issue takes the code from the patch in the parent. > The parent patch is pretty big to review at once. So, plan is to get code > reviewed in smaller patches and > in the process take suggestions and change things if necessary. > What is this patch/issue about ? > This issue attempts to add a timestamp class to hbase-common and timestamp > type to HTable. > This is a part of the attempt to get HLC into HBase. This patch does not > interfere with the current working of HBase. > Why Timestamp Class ? > Timestamp class can be as an abstraction to represent time in Hbase in 64 > bits. > It is just used for manipulating with the 64 bits of the timestamp and is not > concerned about the actual time. > There are three types of timestamps. System time, Custom and HLC. Each one of > it has methods to manipulate the 64 bits of timestamp. > HTable changes: Added a timestamp type property to HTable. This will help > HBase exist in conjunction with old type of timestamp and also the HLC which > will be introduced. The default is set to custom timestamp(current way of > usage of timestamp). default unset timestamp is also custom timestamp as it > should be so. The default timestamp will be changed to HLC when HLC feature > is introduced completely in HBase. > Check HBASE-16210.master.6.patch. > Update: Based on the suggestions, made timestamp enum. Here is the > description of the new changes. > Check the HBASE-16210.master.8.1.patch > 1. Changed the Timestamp Implementation to Enum. > 2. Changed the Timestamp semantics. Instead of HLC, System monotonic and > custom, we now have Hybrid and Physical. System monotonic clock and Custom > clocks can map their timestamps to Physical. HLC clock can map its timestamp > to Hybrid. > 3. The HTableDescriptor will contain clock type(not implemented yet) instead > of timestamp type. As clocks convey the semantics of monotonic increasing and > non decreasing etc. TimestampType doesn't have those semantics enforced, it > just knows what to do with given 64 bits. Therefore, I removed the timestamp > type field in the HTableDescriptor. > Open for suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16210) Add Timestamp class to the hbase-common
[ https://issues.apache.org/jira/browse/HBASE-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382893#comment-15382893 ] Enis Soztutar commented on HBASE-16210: --- I am fine for doing an enum. Was just raising the case in case we want to follow that path. Seem that consensus is to leave it alone. > Add Timestamp class to the hbase-common > --- > > Key: HBASE-16210 > URL: https://issues.apache.org/jira/browse/HBASE-16210 > Project: HBase > Issue Type: Sub-task >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: patch, testing > Attachments: HBASE-16210.master.1.patch, HBASE-16210.master.2.patch, > HBASE-16210.master.3.patch, HBASE-16210.master.4.patch, > HBASE-16210.master.5.patch, HBASE-16210.master.6.patch, > HBASE-16210.master.7.patch, HBASE-16210.master.8.1.patch, > HBASE-16210.master.8.patch > > > This is a sub-issue of > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070]. This JIRA is > a small step towards completely adding Hybrid Logical Clocks(HLC) to HBase. > The main idea of HLC is described in > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070] along with > the motivation of adding it to HBase. > This patch in this issue takes the code from the patch in the parent. > The parent patch is pretty big to review at once. So, plan is to get code > reviewed in smaller patches and > in the process take suggestions and change things if necessary. > What is this patch/issue about ? > This issue attempts to add a timestamp class to hbase-common and timestamp > type to HTable. > This is a part of the attempt to get HLC into HBase. This patch does not > interfere with the current working of HBase. > Why Timestamp Class ? > Timestamp class can be as an abstraction to represent time in Hbase in 64 > bits. > It is just used for manipulating with the 64 bits of the timestamp and is not > concerned about the actual time. > There are three types of timestamps. System time, Custom and HLC. Each one of > it has methods to manipulate the 64 bits of timestamp. > HTable changes: Added a timestamp type property to HTable. This will help > HBase exist in conjunction with old type of timestamp and also the HLC which > will be introduced. The default is set to custom timestamp(current way of > usage of timestamp). default unset timestamp is also custom timestamp as it > should be so. The default timestamp will be changed to HLC when HLC feature > is introduced completely in HBase. > Check HBASE-16210.master.6.patch. > Update: Based on the suggestions, made timestamp enum. Here is the > description of the new changes. > Check the HBASE-16210.master.8.1.patch > 1. Changed the Timestamp Implementation to Enum. > 2. Changed the Timestamp semantics. Instead of HLC, System monotonic and > custom, we now have Hybrid and Physical. System monotonic clock and Custom > clocks can map their timestamps to Physical. HLC clock can map its timestamp > to Hybrid. > 3. The HTableDescriptor will contain clock type(not implemented yet) instead > of timestamp type. As clocks convey the semantics of monotonic increasing and > non decreasing etc. TimestampType doesn't have those semantics enforced, it > just knows what to do with given 64 bits. Therefore, I removed the timestamp > type field in the HTableDescriptor. > Open for suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16241) Allow specification of annotations to use when running check_compatibility.sh
Dima Spivak created HBASE-16241:
---
Summary: Allow specification of annotations to use when running
check_compatibility.sh
Key: HBASE-16241
URL: https://issues.apache.org/jira/browse/HBASE-16241
Project: HBase
Issue Type: Improvement
Components: scripts
Reporter: Dima Spivak
Assignee: Dima Spivak
We already have a {{\-\-all}} option that doesn't include annotations at all,
so we might as well have the ability to provide a list of annotations to
override the defaults of {{InterfaceAudience.Public}} and
{{InterfaceAudience.LimitedPrivate}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16172) Unify the retry logic in ScannerCallableWithReplicas and RpcRetryingCallerWithReadReplicas
[
https://issues.apache.org/jira/browse/HBASE-16172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382822#comment-15382822
]
Hudson commented on HBASE-16172:
SUCCESS: Integrated in HBase-1.4 #294 (See
[https://builds.apache.org/job/HBase-1.4/294/])
HBASE-16172 Unify the retry logic in ScannerCallableWithReplicas and (tedyu:
rev 630a1a41df5f95a06f17c5eeeaa41433955aeab8)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallableWithReplicas.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
> Unify the retry logic in ScannerCallableWithReplicas and
> RpcRetryingCallerWithReadReplicas
> --
>
> Key: HBASE-16172
> URL: https://issues.apache.org/jira/browse/HBASE-16172
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16172.branch-1.v4.txt, 16172.v1.txt, 16172.v2.txt,
> 16172.v2.txt, 16172.v3.txt, 16172.v4.txt, 16172.v4.txt, 16172.v4.txt
>
>
> The issue is pointed out by [~devaraj] in HBASE-16132 (Thanks D.D.), that in
> {{RpcRetryingCallerWithReadReplicas#call}} we will call
> {{ResultBoundedCompletionService#take}} instead of {{poll}} to dead-wait on
> the second one if the first replica timed out, while in
> {{ScannerCallableWithReplicas#call}} we still use
> {{ResultBoundedCompletionService#poll}} with some timeout for the 2nd replica.
> This JIRA aims at discussing whether to unify the logic in these two kinds of
> caller with region replica and taking action if necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15431) A bunch of methods are hot and too big to be inlined
[ https://issues.apache.org/jira/browse/HBASE-15431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382820#comment-15382820 ] Appy commented on HBASE-15431: -- I haven't done this kind of thing in past, so out of curiosity, i'd like to know what were the things you did and where did you hit the wall. Can you please enlighten us [~lhofhansl]. Thanks. > A bunch of methods are hot and too big to be inlined > > > Key: HBASE-15431 > URL: https://issues.apache.org/jira/browse/HBASE-15431 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: hotMethods.txt > > > I ran HBase with "-XX:+PrintCompilation -XX:+UnlockDiagnosticVMOptions > -XX:+PrintInlining" and then looked for "hot method too big" log lines. > I'll attach a log of those messages. > I tried to increase -XX:FreqInlineSize to 1010 to inline all these methods > (as long as they're hot, but actually didn't see any improvement). > In all cases I primed the JVM to make sure the JVM gets a chance to profile > the methods and decide whether they're hot or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16189) [Rolling Upgrade] 2.0 hfiles cannot be opened by 1.x servers
[
https://issues.apache.org/jira/browse/HBASE-16189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382821#comment-15382821
]
Hudson commented on HBASE-16189:
SUCCESS: Integrated in HBase-1.4 #294 (See
[https://builds.apache.org/job/HBase-1.4/294/])
HBASE-16189 [Rolling Upgrade] 2.0 hfiles cannot be opened by 1.x servers
(ramkrishna: rev cfc22ec1ecf5ccaba882c78daf46960f0b5c2510)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/util/CompoundBloomFilter.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
> [Rolling Upgrade] 2.0 hfiles cannot be opened by 1.x servers
>
>
> Key: HBASE-16189
> URL: https://issues.apache.org/jira/browse/HBASE-16189
> Project: HBase
> Issue Type: Sub-task
> Components: migration
>Reporter: Enis Soztutar
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 1.3.0, 1.4.0, 1.2.3, 1.1.7
>
> Attachments: HBASE-16189.branch-1.patch, HBASE-16189.patch,
> HBASE-16189_branch-1.patch
>
>
> HBASE-10800 added MetaCellComparator, which gets written to the HFile. 1.x
> code does not have the new class, hence fails to open the regions. I did not
> check whether this is only for meta or for regular tables as well.
> {code}
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
> reading HFile Trailer from file
> hdfs://cn017.l42scl.hortonworks.com:8020/apps/hbase/data/data/hbase/meta/1588230740/info/aa96e4ef463b4a82956330b236440437
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:483)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:511)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1123)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:267)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:409)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:512)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:687)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:130)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:554)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:551)
> ... 6 more
> Caused by: java.io.IOException: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.CellComparator$MetaCellComparator
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(FixedFileTrailer.java:581)
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.deserializeFromPB(FixedFileTrailer.java:300)
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.deserialize(FixedFileTrailer.java:242)
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:407)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:468)
> ... 15 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.CellComparator$MetaCellComparator
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(FixedFileTrailer.java:579)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16210) Add Timestamp class to the hbase-common
[ https://issues.apache.org/jira/browse/HBASE-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382814#comment-15382814 ] stack commented on HBASE-16210: --- Lets NOT make Timestamp pluggable. There is no call for it and there is enough going on here in this issue w/o allowing for custom implementations also. If someone wanted to change the timestamping in hbase, having to change the code will be a minor inconvenience comparatively (we could always move to make Timestamp pluggable at some later time if wanted). Thanks [~enis] > Add Timestamp class to the hbase-common > --- > > Key: HBASE-16210 > URL: https://issues.apache.org/jira/browse/HBASE-16210 > Project: HBase > Issue Type: Sub-task >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: patch, testing > Attachments: HBASE-16210.master.1.patch, HBASE-16210.master.2.patch, > HBASE-16210.master.3.patch, HBASE-16210.master.4.patch, > HBASE-16210.master.5.patch, HBASE-16210.master.6.patch, > HBASE-16210.master.7.patch, HBASE-16210.master.8.1.patch, > HBASE-16210.master.8.patch > > > This is a sub-issue of > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070]. This JIRA is > a small step towards completely adding Hybrid Logical Clocks(HLC) to HBase. > The main idea of HLC is described in > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070] along with > the motivation of adding it to HBase. > This patch in this issue takes the code from the patch in the parent. > The parent patch is pretty big to review at once. So, plan is to get code > reviewed in smaller patches and > in the process take suggestions and change things if necessary. > What is this patch/issue about ? > This issue attempts to add a timestamp class to hbase-common and timestamp > type to HTable. > This is a part of the attempt to get HLC into HBase. This patch does not > interfere with the current working of HBase. > Why Timestamp Class ? > Timestamp class can be as an abstraction to represent time in Hbase in 64 > bits. > It is just used for manipulating with the 64 bits of the timestamp and is not > concerned about the actual time. > There are three types of timestamps. System time, Custom and HLC. Each one of > it has methods to manipulate the 64 bits of timestamp. > HTable changes: Added a timestamp type property to HTable. This will help > HBase exist in conjunction with old type of timestamp and also the HLC which > will be introduced. The default is set to custom timestamp(current way of > usage of timestamp). default unset timestamp is also custom timestamp as it > should be so. The default timestamp will be changed to HLC when HLC feature > is introduced completely in HBase. > Check HBASE-16210.master.6.patch. > Update: Based on the suggestions, made timestamp enum. Here is the > description of the new changes. > Check the HBASE-16210.master.8.1.patch > 1. Changed the Timestamp Implementation to Enum. > 2. Changed the Timestamp semantics. Instead of HLC, System monotonic and > custom, we now have Hybrid and Physical. System monotonic clock and Custom > clocks can map their timestamps to Physical. HLC clock can map its timestamp > to Hybrid. > 3. The HTableDescriptor will contain clock type(not implemented yet) instead > of timestamp type. As clocks convey the semantics of monotonic increasing and > non decreasing etc. TimestampType doesn't have those semantics enforced, it > just knows what to do with given 64 bits. Therefore, I removed the timestamp > type field in the HTableDescriptor. > Open for suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382805#comment-15382805 ] Ted Yu commented on HBASE-14417: If bulk loading is scheduled at regular interval, one approach is to align incremental backup activity with the bulk load. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16052) Improve HBaseFsck Scalability
[ https://issues.apache.org/jira/browse/HBASE-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Lau updated HBASE-16052: Attachment: HBASE-16052-0.98.v3.patch Thanks [~tedyu] -- done. > Improve HBaseFsck Scalability > - > > Key: HBASE-16052 > URL: https://issues.apache.org/jira/browse/HBASE-16052 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Ben Lau >Assignee: Ben Lau > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16052-0.98.v3.patch, HBASE-16052-master.patch, > HBASE-16052-v3-0.98.patch, HBASE-16052-v3-branch-1.patch, > HBASE-16052-v3-master.patch > > > There are some problems with HBaseFsck that make it unnecessarily slow > especially for large tables or clusters with many regions. > This patch tries to fix the biggest bottlenecks and also include a couple of > bug fixes for some of the race conditions caused by gathering and holding > state about a live cluster that is no longer true by the time you use that > state in Fsck processing. These race conditions cause Fsck to crash and > become unusable on large clusters with lots of region splits/merges. > Here are some scalability/performance problems in HBaseFsck and the changes > the patch makes: > - Unnecessary I/O and RPCs caused by fetching an array of FileStatuses and > then discarding everything but the Paths, then passing the Paths to a > PathFilter, and then having the filter look up the (previously discarded) > FileStatuses of the paths again. This is actually worse than double I/O > because the first lookup obtains a batch of FileStatuses while all the other > lookups are individual RPCs performed sequentially. > -- Avoid this by adding a FileStatusFilter so that filtering can happen > directly on FileStatuses > -- This performance bug affects more than Fsck, but also to some extent > things like snapshots, hfile archival, etc. I didn't have time to look too > deep into other things affected and didn't want to increase the scope of this > ticket so I focus mostly on Fsck and make only a few improvements to other > codepaths. The changes in this patch though should make it fairly easy to > fix other code paths in later jiras if we feel there are some other features > strongly impacted by this problem. > - OfflineReferenceFileRepair is the most expensive part of Fsck (often 50% of > Fsck runtime) and the running time scales with the number of store files, yet > the function is completely serial > -- Make offlineReferenceFileRepair multithreaded > - LoadHdfsRegionDirs() uses table-level concurrency, which is a big > bottleneck if you have 1 large cluster with 1 very large table that has > nearly all the regions > -- Change loadHdfsRegionDirs() to region-level parallelism instead of > table-level parallelism for operations. > The changes benefit all clusters but are especially noticeable for large > clusters with a few very large tables. On our version of 0.98 with the > original patch we had a moderately sized production cluster with 2 (user) > tables and ~160k regions where HBaseFsck went from taking 18 min to 5 minutes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
