Re: HBase connection hangs

2014-01-14 Thread Samir Ahmic
Hi Mark, Base on this exception: 1. 1862 [pool-2-thread-1] WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly transient ZooKeeper, quorum=mark-7:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for

HBase Standalone Error

2014-01-14 Thread Jeff Zhang
I download hbase 0.96, and run it in standalone mode by following the instruction of here http://hbase.apache.org/book/quickstart.html But get the following error, could anyone help on this. 2014-01-14 20:51:25,243 WARN [main] zookeeper.ZKUtil: clean znode for master, quorum=localhost:2181,

Re: HBase Standalone Error

2014-01-14 Thread Stack
This is probably our most important bit of doc so I'd like to fix it. Please list the steps you followed. Did you have a previous instance of HBase up at any time? Did you restart server? Is there a zk process running at the time of the above start? Thank you, St.Ack On Tue, Jan 14, 2014 at

Re: Question on efficient, ordered composite keys

2014-01-14 Thread Ted Yu
Please take a look at HBASE-8089 which is an umbrella JIRA. Some of its subtasks are in 0.96 bq. claiming that short keys (as well as short column names) are relevant bq. Is that also true in 0.94.x? That is true in 0.94.x Cheers On Tue, Jan 14, 2014 at 6:56 AM, Henning Blohm

Rebuild HBASE Table to reduce Regions per RS

2014-01-14 Thread Upender Nimbekar
Hi, Does anyone have any experience rebuidling the HBASE table to reduce the number of regions. I am currently dealing with a situation where the no. of regions per RS have gone up quite significantly (500 per RS) and thereby causing some performance issues. This is how I am thinking of bringing

Question on efficient, ordered composite keys

2014-01-14 Thread Henning Blohm
Hi, for an application still running on Hbase 0.90.4 (but moving to 0.94.6) we are thinking about using more efficient composite row keys compared what we use today (fixed length strings with / separator). I ran into http://hbase.apache.org/book/rowkey.design.html claiming that short keys

Re: Rebuild HBASE Table to reduce Regions per RS

2014-01-14 Thread Dhaval Shah
If you can afford downtime for your table, there are ways to do it. You can: - Merge regions (requires table to be disabled atleast in some older versions and probably in newer ones too) - Go brute force by doing an export, truncate, import (this is a little more manageable when you have a large

Re: Rebuild HBASE Table to reduce Regions per RS

2014-01-14 Thread Vladimir Rodionov
I have never tried this before but I think the following should work: 1. Alter your table: habse alter 't1', METHOD = 'table_att', MAX_FILESIZE = '50' (place your own number here) 2. Merge regions: http://hbase.apache.org/book/ops.regionmgt.html On Tue, Jan 14, 2014 at 7:21 AM,

Re: Rebuild HBASE Table to reduce Regions per RS

2014-01-14 Thread Ted Yu
Upender: For 15.2.2 Merge, please note the following condition: LOG.info(Verifying that HBase is not running...); try { HBaseAdmin.checkHBaseAvailable(getConf()); LOG.fatal(HBase cluster must be off-line.); Cheers On Tue, Jan 14, 2014 at 10:40 AM, Vladimir Rodionov

Re: Rebuild HBASE Table to reduce Regions per RS

2014-01-14 Thread Vladimir Rodionov
Nice. Ted. Is there any reason why we can't do it online? On Tue, Jan 14, 2014 at 10:47 AM, Ted Yu yuzhih...@gmail.com wrote: Upender: For 15.2.2 Merge, please note the following condition: LOG.info(Verifying that HBase is not running...); try {

Re: Rebuild HBASE Table to reduce Regions per RS

2014-01-14 Thread Ted Yu
HBASE-7403 provided online merge capability. The usage is: HBaseAdmin#mergeRegions Note: online merge is in 0.96 and above releases. On Tue, Jan 14, 2014 at 11:00 AM, Vladimir Rodionov vladrodio...@gmail.comwrote: Nice. Ted. Is there any reason why we can't do it online? On Tue, Jan 14,

Re: Question on efficient, ordered composite keys

2014-01-14 Thread James Taylor
Hi Henning, My favorite implementation of efficient composite row keys is Phoenix. We support composite row keys whose byte representation sorts according to the natural sort order of the values (inspired by Lily). You can use our type system independent of querying/inserting data with Phoenix,

Interaction of SequenceID and timestamps

2014-01-14 Thread Ishan Chhabra
I am trying to understand the interaction of sequenceId and timestamps for KVs, and what was the real issue behind https://issues.apache.org/jira/browse/HBASE-6590 which says that bulkload can be used only to update only historical data and not current data. Taking an example: Lets say I have a

Re: Interaction of SequenceID and timestamps

2014-01-14 Thread Ted Yu
Please take a look at the following method in KeyValueHeap#KVScannerComparator : public int compare(KeyValueScanner left, KeyValueScanner right) { Cheers On Tue, Jan 14, 2014 at 3:26 PM, Ishan Chhabra ichha...@rocketfuel.comwrote: I am trying to understand the interaction of sequenceId

Re: Interaction of SequenceID and timestamps

2014-01-14 Thread Ishan Chhabra
Thanks for pointing out the code. My understanding is correct. Thanks! On Tue, Jan 14, 2014 at 3:40 PM, Ted Yu yuzhih...@gmail.com wrote: Please take a look at the following method in KeyValueHeap#KVScannerComparator : public int compare(KeyValueScanner left, KeyValueScanner right) {

Re: Question on efficient, ordered composite keys

2014-01-14 Thread Doug Meil
Hey there, re: efficient, correctly ordered, byte[] serialized composite row keys? I was the guy behind 7221 and that patch had the first part and the last part, but not the middle part (correctly ordered) because this patch relied on the HBase build-in implementations which have the

Re: The problems of power outages

2014-01-14 Thread Ted Yu
I went back from 01-13 all the way to hbase-lilaifeng-master-CentOS.log.2014-01-07 I saw: 2014-01-07 17:46:05,700 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is Slave2:60020 Do you have region server log for Slave2 ?

Corrupt HFile

2014-01-14 Thread 宾莉金
We use hbase-0.94.10 and encounter corrupt hfile. 2014-01-11 23:24:16,547 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating file=hdfs://dump002002.cm6:9000/hbase-0.90/cbu2/735414b148ed70e79f4c0406963bb0c9/.tmp/8a4869aafeae43ee8294bf7b65b92e63 with permission=rwxrwxrwx 2014-01-11 23:24:16,550

Re: Corrupt HFile

2014-01-14 Thread Ted Yu
Which hadoop release are you using ? How many HFiles were corrupted ? Does lzo work properly or lzo never worked ? Thanks On Tue, Jan 14, 2014 at 7:36 PM, 宾莉金 binli...@gmail.com wrote: We use hbase-0.94.10 and encounter corrupt hfile. 2014-01-11 23:24:16,547 DEBUG

Re: Corrupt HFile

2014-01-14 Thread lars hofhansl
Which version of Hadoop? If you get a data center wide power outage you can lose data. In Hadoop 1.1.1 or later you force a sync on block close, and thus you won't at least lose any old data (i.e. HFiles that were recently written due to compactions). I have blogged about that here:

Re: HBase Standalone Error

2014-01-14 Thread Jeff Zhang
Hi Stack, here's the step I use, 1. Download hbase 0.96 and untar it 2. Edit the hbase-site.xml as following configuration property namehbase.rootdir/name value/var/hbase/value /property property namehbase.zookeeper.property.dataDir/name

Fast scan with PrefixFilter?

2014-01-14 Thread Ramon Wang
Hi Folks We have a table with fixed pattern row key design, the format for the row key is YEAR_COUNTRY_randomNumber, for example: 20140101_EN_1 20140101_EN_2 20140101_EN_3 20140101_US_1 20140101_US_2 20140101_US_3 ... Is there a way i can quickly get the data for 20140101_EN_* by using Scan

Re: Fast scan with PrefixFilter?

2014-01-14 Thread Ted Yu
Please take a look at http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/filter/FuzzyRowFilter.html Cheers On Jan 14, 2014, at 10:16 PM, Ramon Wang ra...@appannie.com wrote: Hi Folks We have a table with fixed pattern row key design, the format for the row key is

Re: Fast scan with PrefixFilter?

2014-01-14 Thread Ramon Wang
Hi Ted Thanks for the quick reply. With this FuzzyRowFilter, do i still need to pass in startRow and stopRow like below when constructing a Scan object? Scan(byte [] startRow, byte [] stopRow) Will the FuzzyRowFilter provide us performance like a directly get by row when we pass something