date:20110223

I can't get many versions of the specified column,but only get the latest
version of the specified column. Is there anyone  help me?

//put data by version

 final Put p = new Put(key); // key
final long ts = System.currentTimeMillis();
   p.add(FAMILY, q1, ts,v1);
   p.add(FAMILY, q2, ts,v2);
   p.add(FAMILY, q3, ts,v3);

  table.put(p);


//get all versions of data

final Get get = new Get(key);
   get.addFamily(FAMILY);
   get.setMaxVersions(10);
   final Result result = htable.get(get);

   final NavigableMapbyte[], NavigableMapbyte[],
NavigableMapLong, byte[] map = result.getMap();
   final SetEntrybyte[], NavigableMapbyte[],
NavigableMapLong, byte[] entrySet = map.entrySet();
   final IteratorEntrybyte[], NavigableMapbyte[],
NavigableMapLong, byte[] iterator = entrySet
   .iterator();
   while (iterator.hasNext())
   {
   final Entrybyte[], NavigableMapbyte[],
NavigableMapLong, byte[] next = iterator.next();
   System.out.println(Bytes.toString(next.getKey())); //family
   for (final Entrybyte[], NavigableMapLong, byte[]
item : next.getValue().entrySet())
   {
   System.out.println(Bytes.toString(item.getKey()));
   for (final EntryLong, byte[] _item :
item.getValue().entrySet())
   {
   System.out.println(_item.getKey()); //q
   System.out.println(Bytes.toString(_item.getValue()));
//value
   }
   }

   }

-- 
Thanks  Best regards
jiajun

Re: Trying to contact region Some region

It could be due to slow splits, heavy GC, etc. Make sure your machines
don't swap at all, that HBase has plenty of memory, that you're not
trying to use more CPUs than your machines actually have (like setting
4 maps on a 4 cores machine when also using hbase), etc.

Also upgrading to 0.90.1 will help.

J-D

On Tue, Feb 22, 2011 at 10:18 PM, Hari Sreekumar
hsreeku...@clickable.com wrote:
 Thanks Ted, any way I can fix this in 0.20.6? How can a single Put refer to
 two rows? Is there any coding practice with which I can avoid this? This
 exception is not fatal in the sense that the process still gets completed, I
 just have a few failed tasks, but this leads to waste of time.

 Hari

 On Tue, Feb 22, 2011 at 9:05 PM, Ted Yu yuzhih...@gmail.com wrote:

 The put() call handles more than one row, destined for more than one region
 server.
 HConnectionManager wasn't able to find the region server which serves the
 row, hence the error.

 Please upgrade to 0.90.1

 On Tue, Feb 22, 2011 at 6:27 AM, Hari Sreekumar hsreeku...@clickable.com
 wrote:

  What does this exception signify:
 
  org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
 contact
  region server Some server, retryOnlyOne=true, index=0, islastrow=false,
  tries=9, numtries=10, i=0, listsize=405,
  region=NwKeywordTest,20927_57901_277247_8728141,1298383184948 for region
  KeywordTest,20927_57901_277247_8728141,1298383184948, row
  '20927_57902_277417_8744379', but failed after 10 attempts.
  Exceptions:
 
         at
 
 
 org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1157)
         at
 
 
 org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1238)
         at
  org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:666)
         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:510)
         at
 
 
 org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:94)
         at
 
 
 org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:55)
         at
 
 
 org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:498)
         at
 
 
 org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
         at
 
 
 com.clickable.dataengine.hbase.upload.BulkUploadtoHBase$BulkUploadMapper.map(Unknown
  Source)
         at
 
 
 com.clickable.dataengine.hbase.upload.BulkUploadtoHBase$BulkUploadMapper.map(Unknown
  Source)
         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
         at org.apache.hadoop.mapred.Child.main(Child.java:170)
 
  How can I avoid it?
 
  Thanks,
  Hari

Re: huge .oldlogs

2011-02-23 Thread Ted Yu

Please look for other exceptions.
I have been stress testing 0.90.1 and my .oldlogs folder is empty.


On Wed, Feb 23, 2011 at 11:18 AM, charan kumar charan.ku...@gmail.comwrote:

 Hi J-D,

  There are no NPE's in the log.

 Thanks,
 Charan

 On Wed, Feb 23, 2011 at 11:04 AM, Jean-Daniel Cryans jdcry...@apache.org
 wrote:

  Check you master log, if you see a lot of NPEs then it means you have
  an old hbase-default.xml lying around.
 
  J-D
 
  On Wed, Feb 23, 2011 at 10:58 AM, charan kumar charan.ku...@gmail.com
  wrote:
   Hello,
  
 I was wondering, if I can safely remove the .oldlogs folder. The
 table
   data I have has 1 TB , where as 2.5 TB (with replication 7.5 TB)  for
   .oldlogs folder.
  
I am using hbase-0.90.0 hadoop-append.
  
   Thanks,
   Charan

Re: huge .oldlogs

I'll have to trust you on that :)

The other possible situation is that you are inserting a ton of data
and logs are generated faster than they get cleaned. 0.90.0 has a
limiter that was later removed in 0.90.1 by
https://issues.apache.org/jira/browse/HBASE-3501 so you should upgrade
and see if it fixes it.

J-D

On Wed, Feb 23, 2011 at 11:18 AM, charan kumar charan.ku...@gmail.com wrote:
 Hi J-D,

  There are no NPE's in the log.

 Thanks,
 Charan

 On Wed, Feb 23, 2011 at 11:04 AM, Jean-Daniel Cryans 
 jdcry...@apache.orgwrote:

 Check you master log, if you see a lot of NPEs then it means you have
 an old hbase-default.xml lying around.

 J-D

 On Wed, Feb 23, 2011 at 10:58 AM, charan kumar charan.ku...@gmail.com
 wrote:
  Hello,
 
    I was wondering, if I can safely remove the .oldlogs folder. The table
  data I have has 1 TB , where as 2.5 TB (with replication 7.5 TB)  for
  .oldlogs folder.
 
   I am using hbase-0.90.0 hadoop-append.
 
  Thanks,
  Charan

Re: Trying to contact region Some region

We fixed a lot of the exception handling in 0.90.  The exception text
is much better. Check it out!

-ryan

On Wed, Feb 23, 2011 at 11:18 AM, Jean-Daniel Cryans
jdcry...@apache.org wrote:
 It could be due to slow splits, heavy GC, etc. Make sure your machines
 don't swap at all, that HBase has plenty of memory, that you're not
 trying to use more CPUs than your machines actually have (like setting
 4 maps on a 4 cores machine when also using hbase), etc.

 Also upgrading to 0.90.1 will help.

 J-D

 On Tue, Feb 22, 2011 at 10:18 PM, Hari Sreekumar
 hsreeku...@clickable.com wrote:
 Thanks Ted, any way I can fix this in 0.20.6? How can a single Put refer to
 two rows? Is there any coding practice with which I can avoid this? This
 exception is not fatal in the sense that the process still gets completed, I
 just have a few failed tasks, but this leads to waste of time.

 Hari

 On Tue, Feb 22, 2011 at 9:05 PM, Ted Yu yuzhih...@gmail.com wrote:

 The put() call handles more than one row, destined for more than one region
 server.
 HConnectionManager wasn't able to find the region server which serves the
 row, hence the error.

 Please upgrade to 0.90.1

 On Tue, Feb 22, 2011 at 6:27 AM, Hari Sreekumar hsreeku...@clickable.com
 wrote:

  What does this exception signify:
 
  org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
 contact
  region server Some server, retryOnlyOne=true, index=0, islastrow=false,
  tries=9, numtries=10, i=0, listsize=405,
  region=NwKeywordTest,20927_57901_277247_8728141,1298383184948 for region
  KeywordTest,20927_57901_277247_8728141,1298383184948, row
  '20927_57902_277417_8744379', but failed after 10 attempts.
  Exceptions:
 
         at
 
 
 org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1157)
         at
 
 
 org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1238)
         at
  org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:666)
         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:510)
         at
 
 
 org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:94)
         at
 
 
 org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:55)
         at
 
 
 org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:498)
         at
 
 
 org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
         at
 
 
 com.clickable.dataengine.hbase.upload.BulkUploadtoHBase$BulkUploadMapper.map(Unknown
  Source)
         at
 
 
 com.clickable.dataengine.hbase.upload.BulkUploadtoHBase$BulkUploadMapper.map(Unknown
  Source)
         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
         at org.apache.hadoop.mapred.Child.main(Child.java:170)
 
  How can I avoid it?
 
  Thanks,
  Hari

Re: huge .oldlogs

Yes, you can delete the content of the folder (not the folder itself) safely.

J-D

On Wed, Feb 23, 2011 at 11:37 AM, charan kumar charan.ku...@gmail.com wrote:
 I have been inserting a ton of data for the past few days. This looks like
 the issue.

 If the issue is related to that, can I delete the .oldlogs folder without
 causing any issues?

 I will also look into upgrading..


 On Wed, Feb 23, 2011 at 11:23 AM, Jean-Daniel Cryans 
 jdcry...@apache.orgwrote:

 I'll have to trust you on that :)

 The other possible situation is that you are inserting a ton of data
 and logs are generated faster than they get cleaned. 0.90.0 has a
 limiter that was later removed in 0.90.1 by
 https://issues.apache.org/jira/browse/HBASE-3501 so you should upgrade
 and see if it fixes it.

 J-D

 On Wed, Feb 23, 2011 at 11:18 AM, charan kumar charan.ku...@gmail.com
 wrote:
  Hi J-D,
 
   There are no NPE's in the log.
 
  Thanks,
  Charan
 
  On Wed, Feb 23, 2011 at 11:04 AM, Jean-Daniel Cryans 
 jdcry...@apache.orgwrote:
 
  Check you master log, if you see a lot of NPEs then it means you have
  an old hbase-default.xml lying around.
 
  J-D
 
  On Wed, Feb 23, 2011 at 10:58 AM, charan kumar charan.ku...@gmail.com
  wrote:
   Hello,
  
     I was wondering, if I can safely remove the .oldlogs folder. The
 table
   data I have has 1 TB , where as 2.5 TB (with replication 7.5 TB)  for
   .oldlogs folder.
  
    I am using hbase-0.90.0 hadoop-append.
  
   Thanks,
   Charan

Re: huge .oldlogs

2011-02-23 Thread charan kumar

Excellent! Thank you J-D

On Wed, Feb 23, 2011 at 11:45 AM, Jean-Daniel Cryans jdcry...@apache.orgwrote:

 Yes, you can delete the content of the folder (not the folder itself)
 safely.

 J-D

 On Wed, Feb 23, 2011 at 11:37 AM, charan kumar charan.ku...@gmail.com
 wrote:
  I have been inserting a ton of data for the past few days. This looks
 like
  the issue.
 
  If the issue is related to that, can I delete the .oldlogs folder without
  causing any issues?
 
  I will also look into upgrading..
 
 
  On Wed, Feb 23, 2011 at 11:23 AM, Jean-Daniel Cryans 
 jdcry...@apache.orgwrote:
 
  I'll have to trust you on that :)
 
  The other possible situation is that you are inserting a ton of data
  and logs are generated faster than they get cleaned. 0.90.0 has a
  limiter that was later removed in 0.90.1 by
  https://issues.apache.org/jira/browse/HBASE-3501 so you should upgrade
  and see if it fixes it.
 
  J-D
 
  On Wed, Feb 23, 2011 at 11:18 AM, charan kumar charan.ku...@gmail.com
  wrote:
   Hi J-D,
  
There are no NPE's in the log.
  
   Thanks,
   Charan
  
   On Wed, Feb 23, 2011 at 11:04 AM, Jean-Daniel Cryans 
  jdcry...@apache.orgwrote:
  
   Check you master log, if you see a lot of NPEs then it means you have
   an old hbase-default.xml lying around.
  
   J-D
  
   On Wed, Feb 23, 2011 at 10:58 AM, charan kumar 
 charan.ku...@gmail.com
   wrote:
Hello,
   
  I was wondering, if I can safely remove the .oldlogs folder. The
  table
data I have has 1 TB , where as 2.5 TB (with replication 7.5 TB)
  for
.oldlogs folder.
   
 I am using hbase-0.90.0 hadoop-append.
   
Thanks,
Charan

table creation is failing now and then (CDH3b3)

2011-02-23 Thread Dmitriy Lyubimov

Hi all,

from time to time we come to a sitation where .META. table seems to be stuck
in some corrupted state.
In particular, attempts to create more tables cause

ERROR: org.apache.hadoop.hbase.client.NoServerForRegionException: No server
address listed in .META. for region
LEAD_DATA,,1298484698945.8226d7a44286402e80c9b991d2c00cc3.

The server is not even under any load worth mentioning at the moment.

Table manipulations are failing with both shell and HBAdmin, so it's not an
isolated shell issue (there seems to be similar issue related to shell only,
872, but it's pretty old one).

I scanned for solutions for this particular error, and it seems to be
revolving around flushing .META. , major compact it and restarting the
master(s).

In our case, only restarting the master helped to remove the condition.

logs seem to be stacktrace-free.

My questions are --

-- is there a way to figure out what causes those inconsistences in .META.?
-- is there a way to fix that other than restarting master? Is restarting
master in this case implies a downtime?

Thanks.
-Dmitriy

Re: table creation is failing now and then (CDH3b3)

You should consider upgrading to hbase 0.90.1, a lot of these kinds of
issues were fixed.

-ryan

On Wed, Feb 23, 2011 at 12:02 PM, Dmitriy Lyubimov dlie...@gmail.com wrote:
 Hi all,

 from time to time we come to a sitation where .META. table seems to be stuck
 in some corrupted state.
 In particular, attempts to create more tables cause

 ERROR: org.apache.hadoop.hbase.client.NoServerForRegionException: No server
 address listed in .META. for region
 LEAD_DATA,,1298484698945.8226d7a44286402e80c9b991d2c00cc3.

 The server is not even under any load worth mentioning at the moment.

 Table manipulations are failing with both shell and HBAdmin, so it's not an
 isolated shell issue (there seems to be similar issue related to shell only,
 872, but it's pretty old one).

 I scanned for solutions for this particular error, and it seems to be
 revolving around flushing .META. , major compact it and restarting the
 master(s).

 In our case, only restarting the master helped to remove the condition.

 logs seem to be stacktrace-free.

 My questions are --

 -- is there a way to figure out what causes those inconsistences in .META.?
 -- is there a way to fix that other than restarting master? Is restarting
 master in this case implies a downtime?

 Thanks.
 -Dmitriy

Re: TableInputFormat configuration problems with 0.90

How do you create the configuration object Dan? Are you doing:

Configuration conf = HBaseConfiguration.create();
Job job = new Job(conf, somename);

or are you just creating a normal Configuration?

BTW the code I wrote is what I expect people do and what I'm doing myself.

J-D

On Wed, Feb 23, 2011 at 7:19 AM, Dan danharve...@gmail.com wrote:
Or the other way would be adding the HBase configs to the Hadoop config,
which I think maybe what is intended.

If I do it whilst I'm setting up the job
with HBaseConfiguration.addHbaseResources() it works fine, should
the TableMapReduceUtil.initTableMapperJob do this for you?

I think this was the change that remove the old HBaseConfiguration from
being used https://issues.apache.org/jira/browse/HBASE-2036, but no other
way to add the HBase configs for a TableInputFormat seems to have been
added?

Sorry if I'm completely missing something here!

On Wed, Feb 23, 2011 at 1:57 PM, Dan danharve...@gmail.com wrote:

Hey,

I'm just testing our code to move over to 0.90 and I'm finding some issues
with the map/reduce jobs we've written using TableInputFormat.

We setup the jobs using TableMapReduceUtil.initTableMapperJob(..); which
worked fine in 0.20.6 but now throws the following errors when I try to run
them :-

11/02/23 12:38:44 ERROR zookeeper.ZKConfig: no clientPort found in zoo.cfg
11/02/23 12:38:44 ERROR mapreduce.TableInputFormat:
org.apache.hadoop.hbase.ZooKeeperConnectionException: java.io.IOException:
Unable to determine ZooKeeper ensemble
...
Exception in thread main java.io.IOException: No table was provided.
at
org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:130)
...

Which I think basically means it can't find the quorum/port for zookeeper
in the hbase configuration object from the hbase-*.xml files.
Looking into this a bit further it seems the TableInputFormat's setConf()
method creates the HTable using the Hadoop configuration object which
doesn't contain any of the HBase configuration...

So am I setting the job up incorrectly? If not it doesn't seem to make
sense to me that the HTable should be constructed with the hadoop
configuration so I guess there might be a bug in the TableInputFormat.
Modifying it to not take the hadoop configuration object allows the job to
work correctly and picks up the hbase configuration I've put on the class
path, https://gist.github.com/840403. If that's the case I'll file a bug
report and add the patch.

Thanks,

when does put return to the caller?

2011-02-23 Thread Hiller, Dean (Contractor)

I was wonder if put returns after writing the data into memory on two
out of the three nodes letting my client continue so we don't have to
wait for the memory to then go to disk.  After all, if it is replicated,
probably don't need to wait for it to be written to disk(ie. Kind of
like the in-memory data grids that exist out there).

 

Also, is there an asynchronous request/response for PutAll so I can slam
the grid with batches of 1000 entries kind of like

 

PutAll(ListPutOps puts, AsynchCallback cb); such that cb would be
called with the failure or success response after the put?

 

Thanks,

Dean


This message and any attachments are intended only for the use of the addressee 
and
may contain information that is privileged and confidential. If the reader of 
the 
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.

Re: when does put return to the caller?

There is a batch put call, should be trivial to use some kind of
background thread to invoke callbacks when it returns.

Check out the HTable API, javadoc, etc.  All available via http://hbase.org !

-ryan

On Wed, Feb 23, 2011 at 1:25 PM, Hiller, Dean  (Contractor)
dean.hil...@broadridge.com wrote:
 I was wonder if put returns after writing the data into memory on two
 out of the three nodes letting my client continue so we don't have to
 wait for the memory to then go to disk.  After all, if it is replicated,
 probably don't need to wait for it to be written to disk(ie. Kind of
 like the in-memory data grids that exist out there).



 Also, is there an asynchronous request/response for PutAll so I can slam
 the grid with batches of 1000 entries kind of like



 PutAll(ListPutOps puts, AsynchCallback cb); such that cb would be
 called with the failure or success response after the put?



 Thanks,

 Dean


 This message and any attachments are intended only for the use of the 
 addressee and
 may contain information that is privileged and confidential. If the reader of 
 the
 message is not the intended recipient or an authorized representative of the
 intended recipient, you are hereby notified that any dissemination of this
 communication is strictly prohibited. If you have received this communication 
 in
 error, please notify us immediately by e-mail and delete the message and any
 attachments from your system.

Re: which hadoop and zookeeper version should I use with hbase 0.90.1

2011-02-23 Thread Mike Spreitzer

I have now installed the recommended way --- build Hadoop 
branch-0.20-append, install and config it, then smash its Hadoop core jar 
into the HBase lib/.  Very light testing revealed no problems.  But the 
testing is still so little that I do not recommend drawing any conclusions 
about reliability.

BTW, Cloudera seems to pre-req specifically the Sun (now Oracle) JRE, so 
anyone who has a problem with that can not use Cloudera.

Regards,
Mike Spreitzer

async table updates?

2011-02-23 Thread Vishal Kapoor

I have two tables called LIVE and MASTER.
LIVE reports about the MASTER activity and I need to process records in LIVE
almost real time( some business logic)

if I need to store the activity of entities reported by LIVE rows in MASTER
say in ACTIVITY:LAST_REPORTED

I could process my data in LIVE and at the same time update the MASTER's
LIVE REPORTED attribute with value of current timestamp.
but I was wondering if I can update the MASTER table in a async call so that
I don't compromise on the speed.
was wondering if anyone tried to achieve this or even if it is advisable...


thanks,
Vishal Kapoor

Re: async table updates?

In thrift there is a 'oneway' or 'async' or 'fire and forget' call
type.  I cant recommend those kinds of approaches, since once your
system runs into problems you have no feedback.  So if you are asking
for a one shot, no reply assume it worked call, we don't have one
(nor would I wish that hell on anyone).

So it boils down to your application architecture.  I'm sure you can
figure out how to get notifications via callbacks (with background
threads or threadpools).

Or did you mean something else?
-ryan

On Wed, Feb 23, 2011 at 2:31 PM, Vishal Kapoor
vishal.kapoor...@gmail.com wrote:
 I have two tables called LIVE and MASTER.
 LIVE reports about the MASTER activity and I need to process records in LIVE
 almost real time( some business logic)

 if I need to store the activity of entities reported by LIVE rows in MASTER
 say in ACTIVITY:LAST_REPORTED

 I could process my data in LIVE and at the same time update the MASTER's
 LIVE REPORTED attribute with value of current timestamp.
 but I was wondering if I can update the MASTER table in a async call so that
 I don't compromise on the speed.
 was wondering if anyone tried to achieve this or even if it is advisable...


 thanks,
 Vishal Kapoor

Multiple scans vs single scan with filters

2011-02-23 Thread Alex Baranau

Hello,

Would be great if somebody can share thoughts/ideas/some numbers on the
following problem.

We have a reporting app. To fetch data for some chart/report we currently
use multiple scans, usually 10-50. We fetch about 100 records with each scan
which we use to construct a report.

I've revised data we store and code logic and see that we could really fetch
same data with single scan by specifying filters to filter out data which
doesn't fit the report params. In this case the scan range will be about
100-200K records from which after filtering we'd get the same records as we
do currently fetch with multiple scans.

So the question is: given these numbers (10-50 scans fetching 100 records
each VS 1 scan + filters on range of 100-200K records) will the optimization
I have in mind really improve performance? Unfortunately we don't have good
volume of data currently to perform tests on. May be someone can share
thoughts based solely on these numbers? Record size is about 1Kb.

Thank you!
Alex Baranau

Re: TableInputFormat configuration problems with 0.90

2011-02-23 Thread Dan Harvey

Ah ok, most of the time we were using the default Hadoop configuration object
and not the HBase one.

I guess that's a change between 0.20 and 0.90? Would it not make sense for the
TableMapReduceUtil class to do that for you? As you'll need it in every HBase
map reduce job.

Anyway, I guess we should just change our job configs to include the HBase
configuration too then it would work fine.

Thanks,

On 23 Feb 2011, at 19:44, Jean-Daniel Cryans jdcry...@apache.org wrote:

How do you create the configuration object Dan? Are you doing:

Configuration conf = HBaseConfiguration.create();
Job job = new Job(conf, somename);

or are you just creating a normal Configuration?

BTW the code I wrote is what I expect people do and what I'm doing myself.

J-D

On Wed, Feb 23, 2011 at 7:19 AM, Dan danharve...@gmail.com wrote:
Or the other way would be adding the HBase configs to the Hadoop config,
which I think maybe what is intended.

If I do it whilst I'm setting up the job
with HBaseConfiguration.addHbaseResources() it works fine, should
the TableMapReduceUtil.initTableMapperJob do this for you?

Sorry if I'm completely missing something here!

On Wed, Feb 23, 2011 at 1:57 PM, Dan danharve...@gmail.com wrote:

Hey,

I'm just testing our code to move over to 0.90 and I'm finding some issues
with the map/reduce jobs we've written using TableInputFormat.

We setup the jobs using TableMapReduceUtil.initTableMapperJob(..); which
worked fine in 0.20.6 but now throws the following errors when I try to run
them :-

Thanks,

Re: TableInputFormat configuration problems with 0.90

Yeah it should, also I'm pretty sure you're right to say that this
regression comes from HBASE-2036... would you mind opening a jira?

Thanks for the report and the digging Dan!

J-D

On Wed, Feb 23, 2011 at 3:30 PM, Dan Harvey danharve...@gmail.com wrote:
Ah ok, most of the time we were using the default Hadoop configuration object
and not the HBase one.

I guess that's a change between 0.20 and 0.90? Would it not make sense for
the TableMapReduceUtil class to do that for you? As you'll need it in every
HBase map reduce job.

Anyway, I guess we should just change our job configs to include the HBase
configuration too then it would work fine.

Thanks,

On 23 Feb 2011, at 19:44, Jean-Daniel Cryans jdcry...@apache.org wrote:

How do you create the configuration object Dan? Are you doing:

Configuration conf = HBaseConfiguration.create();
Job job = new Job(conf, somename);

or are you just creating a normal Configuration?

BTW the code I wrote is what I expect people do and what I'm doing myself.

J-D

On Wed, Feb 23, 2011 at 7:19 AM, Dan danharve...@gmail.com wrote:
Or the other way would be adding the HBase configs to the Hadoop config,
which I think maybe what is intended.

If I do it whilst I'm setting up the job
with HBaseConfiguration.addHbaseResources() it works fine, should
the TableMapReduceUtil.initTableMapperJob do this for you?

Sorry if I'm completely missing something here!

On Wed, Feb 23, 2011 at 1:57 PM, Dan danharve...@gmail.com wrote:

Hey,

I'm just testing our code to move over to 0.90 and I'm finding some issues
with the map/reduce jobs we've written using TableInputFormat.

We setup the jobs using TableMapReduceUtil.initTableMapperJob(..); which
worked fine in 0.20.6 but now throws the following errors when I try to run
them :-

Thanks,

Re: I can't get many versions of the specified column,but only get the latest version of the specified column

I execute it five times at diffrent time.

//put data by version

 final Put p = new Put(key); // key
final long ts = System.currentTimeMillis();
   p.add(FAMILY, q1, ts,v1);
   p.add(FAMILY, q2, ts,v2);
   p.add(FAMILY, q3, ts,v3);
  table.put(p);

So I can get five versions ,right?

On Thu, Feb 24, 2011 at 2:06 AM, Stack st...@duboce.net wrote:

 What do you get for a result?

 You are only entering a single version of each column, a single
 version of FAMILY:q1, a single version FAMILY:q2, and a FAMILY:q3.

 St.Ack

 On Wed, Feb 23, 2011 at 2:54 AM, 陈加俊 cjjvict...@gmail.com wrote:
  I can't get many versions of the specified column,but only get the latest
  version of the specified column. Is there anyone  help me?
 
  //put data by version
 
   final Put p = new Put(key); // key
  final long ts = System.currentTimeMillis();
p.add(FAMILY, q1, ts,v1);
p.add(FAMILY, q2, ts,v2);
p.add(FAMILY, q3, ts,v3);
 
   table.put(p);
 
 
  //get all versions of data
 
  final Get get = new Get(key);
get.addFamily(FAMILY);
get.setMaxVersions(10);
final Result result = htable.get(get);
 
final NavigableMapbyte[], NavigableMapbyte[],
  NavigableMapLong, byte[] map = result.getMap();
final SetEntrybyte[], NavigableMapbyte[],
  NavigableMapLong, byte[] entrySet = map.entrySet();
final IteratorEntrybyte[], NavigableMapbyte[],
  NavigableMapLong, byte[] iterator = entrySet
.iterator();
while (iterator.hasNext())
{
final Entrybyte[], NavigableMapbyte[],
  NavigableMapLong, byte[] next = iterator.next();
System.out.println(Bytes.toString(next.getKey())); //family
for (final Entrybyte[], NavigableMapLong, byte[]
  item : next.getValue().entrySet())
{
System.out.println(Bytes.toString(item.getKey()));
for (final EntryLong, byte[] _item :
  item.getValue().entrySet())
{
System.out.println(_item.getKey()); //q
 
 System.out.println(Bytes.toString(_item.getValue()));
  //value
}
}
 
}
 
  --
  Thanks  Best regards
  jiajun
 




-- 
Thanks  Best regards
jiajun

RE: I can't get many versions of the specified column,but only get the latest version of the specified column

2011-02-23 Thread Buttler, David

What is your table schema set to?  By default it holds 3 versions.
Also, you might iterating over KeyValues instead of using the Map since you 
don't really care about the organization, just the time.

Dave

-Original Message-
From: 陈加俊 [mailto:cjjvict...@gmail.com] 
Sent: Wednesday, February 23, 2011 5:22 PM
To: user@hbase.apache.org
Cc: Stack
Subject: Re: I can't get many versions of the specified column,but only get the 
latest version of the specified column

I execute it five times at diffrent time.

//put data by version

 final Put p = new Put(key); // key
final long ts = System.currentTimeMillis();
   p.add(FAMILY, q1, ts,v1);
   p.add(FAMILY, q2, ts,v2);
   p.add(FAMILY, q3, ts,v3);
  table.put(p);

So I can get five versions ,right?

On Thu, Feb 24, 2011 at 2:06 AM, Stack st...@duboce.net wrote:

 What do you get for a result?

 You are only entering a single version of each column, a single
 version of FAMILY:q1, a single version FAMILY:q2, and a FAMILY:q3.

 St.Ack

 On Wed, Feb 23, 2011 at 2:54 AM, 陈加俊 cjjvict...@gmail.com wrote:
  I can't get many versions of the specified column,but only get the latest
  version of the specified column. Is there anyone  help me?
 
  //put data by version
 
   final Put p = new Put(key); // key
  final long ts = System.currentTimeMillis();
p.add(FAMILY, q1, ts,v1);
p.add(FAMILY, q2, ts,v2);
p.add(FAMILY, q3, ts,v3);
 
   table.put(p);
 
 
  //get all versions of data
 
  final Get get = new Get(key);
get.addFamily(FAMILY);
get.setMaxVersions(10);
final Result result = htable.get(get);
 
final NavigableMapbyte[], NavigableMapbyte[],
  NavigableMapLong, byte[] map = result.getMap();
final SetEntrybyte[], NavigableMapbyte[],
  NavigableMapLong, byte[] entrySet = map.entrySet();
final IteratorEntrybyte[], NavigableMapbyte[],
  NavigableMapLong, byte[] iterator = entrySet
.iterator();
while (iterator.hasNext())
{
final Entrybyte[], NavigableMapbyte[],
  NavigableMapLong, byte[] next = iterator.next();
System.out.println(Bytes.toString(next.getKey())); //family
for (final Entrybyte[], NavigableMapLong, byte[]
  item : next.getValue().entrySet())
{
System.out.println(Bytes.toString(item.getKey()));
for (final EntryLong, byte[] _item :
  item.getValue().entrySet())
{
System.out.println(_item.getKey()); //q
 
 System.out.println(Bytes.toString(_item.getValue()));
  //value
}
}
 
}
 
  --
  Thanks  Best regards
  jiajun
 




-- 
Thanks  Best regards
jiajun

Stack Overflow?

2011-02-23 Thread Buttler, David

Hi all,
It seems that we are getting a lot of repeated questions now.  Perhaps it would 
be useful to start migrating the simple questions off to stackoverflow (or 
whichever stack exchange website is most appropriate), and just pointing people 
there?  Obviously there are still a lot of questions that can't be migrated 
over since they require detailed examination of logs, but some solid answers 
over there may help keep the signal high on this list.

Dave

Re: I can't get many versions of the specified column,but only get the latest version of the specified column

Thank you David !

I alter the table schema as follow:

 alter 'cjjIndexPageModify', {NAME = 'log' , VERSIONS = 5 , METHOD =
'add'}

How to iterate over KeyValues?  which method in Result?

On Thu, Feb 24, 2011 at 9:27 AM, Buttler, David buttl...@llnl.gov wrote:

 What is your table schema set to?  By default it holds 3 versions.
 Also, you might iterating over KeyValues instead of using the Map since you
 don't really care about the organization, just the time.

 Dave

 -Original Message-
 From: 陈加俊 [mailto:cjjvict...@gmail.com]
 Sent: Wednesday, February 23, 2011 5:22 PM
 To: user@hbase.apache.org
 Cc: Stack
 Subject: Re: I can't get many versions of the specified column,but only get
 the latest version of the specified column

 I execute it five times at diffrent time.

 //put data by version

  final Put p = new Put(key); // key
 final long ts = System.currentTimeMillis();
   p.add(FAMILY, q1, ts,v1);
   p.add(FAMILY, q2, ts,v2);
   p.add(FAMILY, q3, ts,v3);
  table.put(p);

 So I can get five versions ,right?

 On Thu, Feb 24, 2011 at 2:06 AM, Stack st...@duboce.net wrote:

  What do you get for a result?
 
  You are only entering a single version of each column, a single
  version of FAMILY:q1, a single version FAMILY:q2, and a FAMILY:q3.
 
  St.Ack
 
  On Wed, Feb 23, 2011 at 2:54 AM, 陈加俊 cjjvict...@gmail.com wrote:
   I can't get many versions of the specified column,but only get the
 latest
   version of the specified column. Is there anyone  help me?
  
   //put data by version
  
final Put p = new Put(key); // key
   final long ts = System.currentTimeMillis();
 p.add(FAMILY, q1, ts,v1);
 p.add(FAMILY, q2, ts,v2);
 p.add(FAMILY, q3, ts,v3);
  
table.put(p);
  
  
   //get all versions of data
  
   final Get get = new Get(key);
 get.addFamily(FAMILY);
 get.setMaxVersions(10);
 final Result result = htable.get(get);
  
 final NavigableMapbyte[], NavigableMapbyte[],
   NavigableMapLong, byte[] map = result.getMap();
 final SetEntrybyte[], NavigableMapbyte[],
   NavigableMapLong, byte[] entrySet = map.entrySet();
 final IteratorEntrybyte[], NavigableMapbyte[],
   NavigableMapLong, byte[] iterator = entrySet
 .iterator();
 while (iterator.hasNext())
 {
 final Entrybyte[], NavigableMapbyte[],
   NavigableMapLong, byte[] next = iterator.next();
 System.out.println(Bytes.toString(next.getKey()));
 //family
 for (final Entrybyte[], NavigableMapLong, byte[]
   item : next.getValue().entrySet())
 {
 System.out.println(Bytes.toString(item.getKey()));
 for (final EntryLong, byte[] _item :
   item.getValue().entrySet())
 {
 System.out.println(_item.getKey()); //q
  
  System.out.println(Bytes.toString(_item.getValue()));
   //value
 }
 }
  
 }
  
   --
   Thanks  Best regards
   jiajun
  
 



 --
 Thanks  Best regards
 jiajun




-- 
Thanks  Best regards
jiajun

RE: I can't get many versions of the specified column,but only get the latest version of the specified column

2011-02-23 Thread Buttler, David

Result.list() ?
Putting the hbase source into your IDE of choice (yay Eclipse!) is really 
helpful

Dave


-Original Message-
From: 陈加俊 [mailto:cjjvict...@gmail.com] 
Sent: Wednesday, February 23, 2011 5:42 PM
To: user@hbase.apache.org
Cc: Buttler, David
Subject: Re: I can't get many versions of the specified column,but only get the 
latest version of the specified column

Thank you David !

I alter the table schema as follow:

 alter 'cjjIndexPageModify', {NAME = 'log' , VERSIONS = 5 , METHOD =
'add'}

How to iterate over KeyValues?  which method in Result?

On Thu, Feb 24, 2011 at 9:27 AM, Buttler, David buttl...@llnl.gov wrote:

 What is your table schema set to?  By default it holds 3 versions.
 Also, you might iterating over KeyValues instead of using the Map since you
 don't really care about the organization, just the time.

 Dave

 -Original Message-
 From: 陈加俊 [mailto:cjjvict...@gmail.com]
 Sent: Wednesday, February 23, 2011 5:22 PM
 To: user@hbase.apache.org
 Cc: Stack
 Subject: Re: I can't get many versions of the specified column,but only get
 the latest version of the specified column

 I execute it five times at diffrent time.

 //put data by version

  final Put p = new Put(key); // key
 final long ts = System.currentTimeMillis();
   p.add(FAMILY, q1, ts,v1);
   p.add(FAMILY, q2, ts,v2);
   p.add(FAMILY, q3, ts,v3);
  table.put(p);

 So I can get five versions ,right?

 On Thu, Feb 24, 2011 at 2:06 AM, Stack st...@duboce.net wrote:

  What do you get for a result?
 
  You are only entering a single version of each column, a single
  version of FAMILY:q1, a single version FAMILY:q2, and a FAMILY:q3.
 
  St.Ack
 
  On Wed, Feb 23, 2011 at 2:54 AM, 陈加俊 cjjvict...@gmail.com wrote:
   I can't get many versions of the specified column,but only get the
 latest
   version of the specified column. Is there anyone  help me?
  
   //put data by version
  
final Put p = new Put(key); // key
   final long ts = System.currentTimeMillis();
 p.add(FAMILY, q1, ts,v1);
 p.add(FAMILY, q2, ts,v2);
 p.add(FAMILY, q3, ts,v3);
  
table.put(p);
  
  
   //get all versions of data
  
   final Get get = new Get(key);
 get.addFamily(FAMILY);
 get.setMaxVersions(10);
 final Result result = htable.get(get);
  
 final NavigableMapbyte[], NavigableMapbyte[],
   NavigableMapLong, byte[] map = result.getMap();
 final SetEntrybyte[], NavigableMapbyte[],
   NavigableMapLong, byte[] entrySet = map.entrySet();
 final IteratorEntrybyte[], NavigableMapbyte[],
   NavigableMapLong, byte[] iterator = entrySet
 .iterator();
 while (iterator.hasNext())
 {
 final Entrybyte[], NavigableMapbyte[],
   NavigableMapLong, byte[] next = iterator.next();
 System.out.println(Bytes.toString(next.getKey()));
 //family
 for (final Entrybyte[], NavigableMapLong, byte[]
   item : next.getValue().entrySet())
 {
 System.out.println(Bytes.toString(item.getKey()));
 for (final EntryLong, byte[] _item :
   item.getValue().entrySet())
 {
 System.out.println(_item.getKey()); //q
  
  System.out.println(Bytes.toString(_item.getValue()));
   //value
 }
 }
  
 }
  
   --
   Thanks  Best regards
   jiajun
  
 



 --
 Thanks  Best regards
 jiajun




-- 
Thanks  Best regards
jiajun

Re: I can't get many versions of the specified column,but only get the latest version of the specified column

   final ListKeyValue list = result.list();
for (final KeyValue it : list)
{
System.out.println(Bytes.toString(it.getKey()));
System.out.println(Bytes.toString(it.getValue()));
}
I can only get the last version!  why ? Is there any testcase in Hbase?

On Thu, Feb 24, 2011 at 9:56 AM, 陈加俊 cjjvict...@gmail.com wrote:

  /**
* Create a sorted list of the KeyValue's in this result.
*
* @return The sorted list of KeyValue's.
*/
   public ListKeyValue list() {
 if(this.kvs == null) {
   readFields();
 }
 return isEmpty()? null: Arrays.asList(sorted());
   }

 I will try it . Thank you very much!

 On Thu, Feb 24, 2011 at 9:45 AM, Buttler, David buttl...@llnl.gov wrote:

 Result.list() ?
 Putting the hbase source into your IDE of choice (yay Eclipse!) is really
 helpful

 Dave


 -Original Message-
 From: 陈加俊 [mailto:cjjvict...@gmail.com]
 Sent: Wednesday, February 23, 2011 5:42 PM
 To: user@hbase.apache.org
 Cc: Buttler, David
 Subject: Re: I can't get many versions of the specified column,but only
 get the latest version of the specified column

 Thank you David !

 I alter the table schema as follow:

  alter 'cjjIndexPageModify', {NAME = 'log' , VERSIONS = 5 , METHOD =
 'add'}

 How to iterate over KeyValues?  which method in Result?

 On Thu, Feb 24, 2011 at 9:27 AM, Buttler, David buttl...@llnl.gov
 wrote:

  What is your table schema set to?  By default it holds 3 versions.
  Also, you might iterating over KeyValues instead of using the Map since
 you
  don't really care about the organization, just the time.
 
  Dave
 
  -Original Message-
  From: 陈加俊 [mailto:cjjvict...@gmail.com]
  Sent: Wednesday, February 23, 2011 5:22 PM
  To: user@hbase.apache.org
  Cc: Stack
  Subject: Re: I can't get many versions of the specified column,but only
 get
  the latest version of the specified column
 
  I execute it five times at diffrent time.
 
  //put data by version
 
   final Put p = new Put(key); // key
  final long ts = System.currentTimeMillis();
p.add(FAMILY, q1, ts,v1);
p.add(FAMILY, q2, ts,v2);
p.add(FAMILY, q3, ts,v3);
   table.put(p);
 
  So I can get five versions ,right?
 
  On Thu, Feb 24, 2011 at 2:06 AM, Stack st...@duboce.net wrote:
 
   What do you get for a result?
  
   You are only entering a single version of each column, a single
   version of FAMILY:q1, a single version FAMILY:q2, and a FAMILY:q3.
  
   St.Ack
  
   On Wed, Feb 23, 2011 at 2:54 AM, 陈加俊 cjjvict...@gmail.com wrote:
I can't get many versions of the specified column,but only get the
  latest
version of the specified column. Is there anyone  help me?
   
//put data by version
   
 final Put p = new Put(key); // key
final long ts = System.currentTimeMillis();
  p.add(FAMILY, q1, ts,v1);
  p.add(FAMILY, q2, ts,v2);
  p.add(FAMILY, q3, ts,v3);
   
 table.put(p);
   
   
//get all versions of data
   
final Get get = new Get(key);
  get.addFamily(FAMILY);
  get.setMaxVersions(10);
  final Result result = htable.get(get);
   
  final NavigableMapbyte[], NavigableMapbyte[],
NavigableMapLong, byte[] map = result.getMap();
  final SetEntrybyte[], NavigableMapbyte[],
NavigableMapLong, byte[] entrySet = map.entrySet();
  final IteratorEntrybyte[], NavigableMapbyte[],
NavigableMapLong, byte[] iterator = entrySet
  .iterator();
  while (iterator.hasNext())
  {
  final Entrybyte[], NavigableMapbyte[],
NavigableMapLong, byte[] next = iterator.next();
  System.out.println(Bytes.toString(next.getKey()));
  //family
  for (final Entrybyte[], NavigableMapLong, byte[]
item : next.getValue().entrySet())
  {
  System.out.println(Bytes.toString(item.getKey()));
  for (final EntryLong, byte[] _item :
item.getValue().entrySet())
  {
  System.out.println(_item.getKey()); //q
   
   System.out.println(Bytes.toString(_item.getValue()));
//value
  }
  }
   
  }
   
--
Thanks  Best regards
jiajun
   
  
 
 
 
  --
  Thanks  Best regards
  jiajun
 



 --
 Thanks  Best regards
 jiajun




 --
 Thanks  Best regards
 jiajun




-- 
Thanks  Best regards
jiajun

Re: I can't get many versions of the specified column,but only get the latest version of the specified column

There are test cases for this, the functionality DOES work, something is up...

Without full code and full descriptions of your tables, debugging is
harder than it needs to be.  It's probably a simple typo or something,
check your code and table descriptions again. Many people rely on the
multi version query capabilities and it is very unlikely to be broken
in a released version of hbase.

On Wed, Feb 23, 2011 at 6:27 PM, 陈加俊 cjjvict...@gmail.com wrote:
   final ListKeyValue list = result.list();
for (final KeyValue it : list)
{
System.out.println(Bytes.toString(it.getKey()));
System.out.println(Bytes.toString(it.getValue()));
}
 I can only get the last version!  why ? Is there any testcase in Hbase?

 On Thu, Feb 24, 2011 at 9:56 AM, 陈加俊 cjjvict...@gmail.com wrote:

  /**
* Create a sorted list of the KeyValue's in this result.
*
* @return The sorted list of KeyValue's.
*/
   public ListKeyValue list() {
 if(this.kvs == null) {
   readFields();
 }
 return isEmpty()? null: Arrays.asList(sorted());
   }

 I will try it . Thank you very much!

 On Thu, Feb 24, 2011 at 9:45 AM, Buttler, David buttl...@llnl.gov wrote:

 Result.list() ?
 Putting the hbase source into your IDE of choice (yay Eclipse!) is really
 helpful

 Dave


 -Original Message-
 From: 陈加俊 [mailto:cjjvict...@gmail.com]
 Sent: Wednesday, February 23, 2011 5:42 PM
 To: user@hbase.apache.org
 Cc: Buttler, David
 Subject: Re: I can't get many versions of the specified column,but only
 get the latest version of the specified column

 Thank you David !

 I alter the table schema as follow:

  alter 'cjjIndexPageModify', {NAME = 'log' , VERSIONS = 5 , METHOD =
 'add'}

 How to iterate over KeyValues?  which method in Result?

 On Thu, Feb 24, 2011 at 9:27 AM, Buttler, David buttl...@llnl.gov
 wrote:

  What is your table schema set to?  By default it holds 3 versions.
  Also, you might iterating over KeyValues instead of using the Map since
 you
  don't really care about the organization, just the time.
 
  Dave
 
  -Original Message-
  From: 陈加俊 [mailto:cjjvict...@gmail.com]
  Sent: Wednesday, February 23, 2011 5:22 PM
  To: user@hbase.apache.org
  Cc: Stack
  Subject: Re: I can't get many versions of the specified column,but only
 get
  the latest version of the specified column
 
  I execute it five times at diffrent time.
 
  //put data by version
 
   final Put p = new Put(key); // key
  final long ts = System.currentTimeMillis();
p.add(FAMILY, q1, ts,v1);
p.add(FAMILY, q2, ts,v2);
p.add(FAMILY, q3, ts,v3);
   table.put(p);
 
  So I can get five versions ,right?
 
  On Thu, Feb 24, 2011 at 2:06 AM, Stack st...@duboce.net wrote:
 
   What do you get for a result?
  
   You are only entering a single version of each column, a single
   version of FAMILY:q1, a single version FAMILY:q2, and a FAMILY:q3.
  
   St.Ack
  
   On Wed, Feb 23, 2011 at 2:54 AM, 陈加俊 cjjvict...@gmail.com wrote:
I can't get many versions of the specified column,but only get the
  latest
version of the specified column. Is there anyone  help me?
   
//put data by version
   
 final Put p = new Put(key); // key
final long ts = System.currentTimeMillis();
  p.add(FAMILY, q1, ts,v1);
  p.add(FAMILY, q2, ts,v2);
  p.add(FAMILY, q3, ts,v3);
   
 table.put(p);
   
   
//get all versions of data
   
final Get get = new Get(key);
  get.addFamily(FAMILY);
  get.setMaxVersions(10);
  final Result result = htable.get(get);
   
  final NavigableMapbyte[], NavigableMapbyte[],
NavigableMapLong, byte[] map = result.getMap();
  final SetEntrybyte[], NavigableMapbyte[],
NavigableMapLong, byte[] entrySet = map.entrySet();
  final IteratorEntrybyte[], NavigableMapbyte[],
NavigableMapLong, byte[] iterator = entrySet
  .iterator();
  while (iterator.hasNext())
  {
  final Entrybyte[], NavigableMapbyte[],
NavigableMapLong, byte[] next = iterator.next();
  System.out.println(Bytes.toString(next.getKey()));
  //family
  for (final Entrybyte[], NavigableMapLong, byte[]
item : next.getValue().entrySet())
  {
  System.out.println(Bytes.toString(item.getKey()));
  for (final EntryLong, byte[] _item :
item.getValue().entrySet())
  {
  System.out.println(_item.getKey()); //q
   
   System.out.println(Bytes.toString(_item.getValue()));
//value
  }
  }
   
  }
   
--
Thanks  Best regards
jiajun
   
  
 
 
 
  --
  Thanks  Best regards
  jiajun
 



 --
 Thanks  Best regards
 jiajun




 --
 Thanks  Best regards
 jiajun




 --
 Thanks  Best regards
 jiajun

Number of regions

2011-02-23 Thread Nanheng Wu

What are some of the trade-offs of using larger region files and less
regions vs the other way round? Currently each of my host has ~700
regions with the default hfile size, is this an acceptable number?
(hosts have 16GB of RAM). Another totally unrelated question: I have
Gzip enabled on the hfile and in my region file's log file all I see
are messages like got a brand new decompressor, they are filling up
my screen and making it very hard to see what's happening with my
servers. Is there a way to disable that? Thanks so much.

Re: Number of regions

There have been threads about this lately, check out the search box on
hbase.org which searches the list archives.
On Feb 23, 2011 6:56 PM, Nanheng Wu nanhen...@gmail.com wrote:
 What are some of the trade-offs of using larger region files and less
 regions vs the other way round? Currently each of my host has ~700
 regions with the default hfile size, is this an acceptable number?
 (hosts have 16GB of RAM). Another totally unrelated question: I have
 Gzip enabled on the hfile and in my region file's log file all I see
 are messages like got a brand new decompressor, they are filling up
 my screen and making it very hard to see what's happening with my
 servers. Is there a way to disable that? Thanks so much.

Re: I can't get many versions of the specified column,but only get the latest version of the specified column

I will check my code and table descriptions again. And the test case
is TestGetRowVersions. I
believe that I made a mistake.

2011/2/24 Ryan Rawson ryano...@gmail.com

 There are test cases for this, the functionality DOES work, something is
 up...

 Without full code and full descriptions of your tables, debugging is
 harder than it needs to be.  It's probably a simple typo or something,
 check your code and table descriptions again. Many people rely on the
 multi version query capabilities and it is very unlikely to be broken
 in a released version of hbase.

 On Wed, Feb 23, 2011 at 6:27 PM, 陈加俊 cjjvict...@gmail.com wrote:
final ListKeyValue list = result.list();
 for (final KeyValue it : list)
 {
 System.out.println(Bytes.toString(it.getKey()));
 System.out.println(Bytes.toString(it.getValue()));
 }
  I can only get the last version!  why ? Is there any testcase in Hbase?
 
  On Thu, Feb 24, 2011 at 9:56 AM, 陈加俊 cjjvict...@gmail.com wrote:
 
   /**
 * Create a sorted list of the KeyValue's in this result.
 *
 * @return The sorted list of KeyValue's.
 */
public ListKeyValue list() {
  if(this.kvs == null) {
readFields();
  }
  return isEmpty()? null: Arrays.asList(sorted());
}
 
  I will try it . Thank you very much!
 
  On Thu, Feb 24, 2011 at 9:45 AM, Buttler, David buttl...@llnl.gov
 wrote:
 
  Result.list() ?
  Putting the hbase source into your IDE of choice (yay Eclipse!) is
 really
  helpful
 
  Dave
 
 
  -Original Message-
  From: 陈加俊 [mailto:cjjvict...@gmail.com]
  Sent: Wednesday, February 23, 2011 5:42 PM
  To: user@hbase.apache.org
  Cc: Buttler, David
  Subject: Re: I can't get many versions of the specified column,but only
  get the latest version of the specified column
 
  Thank you David !
 
  I alter the table schema as follow:
 
   alter 'cjjIndexPageModify', {NAME = 'log' , VERSIONS = 5 , METHOD
 =
  'add'}
 
  How to iterate over KeyValues?  which method in Result?
 
  On Thu, Feb 24, 2011 at 9:27 AM, Buttler, David buttl...@llnl.gov
  wrote:
 
   What is your table schema set to?  By default it holds 3 versions.
   Also, you might iterating over KeyValues instead of using the Map
 since
  you
   don't really care about the organization, just the time.
  
   Dave
  
   -Original Message-
   From: 陈加俊 [mailto:cjjvict...@gmail.com]
   Sent: Wednesday, February 23, 2011 5:22 PM
   To: user@hbase.apache.org
   Cc: Stack
   Subject: Re: I can't get many versions of the specified column,but
 only
  get
   the latest version of the specified column
  
   I execute it five times at diffrent time.
  
   //put data by version
  
final Put p = new Put(key); // key
   final long ts = System.currentTimeMillis();
 p.add(FAMILY, q1, ts,v1);
 p.add(FAMILY, q2, ts,v2);
 p.add(FAMILY, q3, ts,v3);
table.put(p);
  
   So I can get five versions ,right?
  
   On Thu, Feb 24, 2011 at 2:06 AM, Stack st...@duboce.net wrote:
  
What do you get for a result?
   
You are only entering a single version of each column, a single
version of FAMILY:q1, a single version FAMILY:q2, and a FAMILY:q3.
   
St.Ack
   
On Wed, Feb 23, 2011 at 2:54 AM, 陈加俊 cjjvict...@gmail.com wrote:
 I can't get many versions of the specified column,but only get
 the
   latest
 version of the specified column. Is there anyone  help me?

 //put data by version

  final Put p = new Put(key); // key
 final long ts = System.currentTimeMillis();
   p.add(FAMILY, q1, ts,v1);
   p.add(FAMILY, q2, ts,v2);
   p.add(FAMILY, q3, ts,v3);

  table.put(p);


 //get all versions of data

 final Get get = new Get(key);
   get.addFamily(FAMILY);
   get.setMaxVersions(10);
   final Result result = htable.get(get);

   final NavigableMapbyte[], NavigableMapbyte[],
 NavigableMapLong, byte[] map = result.getMap();
   final SetEntrybyte[], NavigableMapbyte[],
 NavigableMapLong, byte[] entrySet = map.entrySet();
   final IteratorEntrybyte[], NavigableMapbyte[],
 NavigableMapLong, byte[] iterator = entrySet
   .iterator();
   while (iterator.hasNext())
   {
   final Entrybyte[], NavigableMapbyte[],
 NavigableMapLong, byte[] next = iterator.next();
   System.out.println(Bytes.toString(next.getKey()));
   //family
   for (final Entrybyte[], NavigableMapLong,
 byte[]
 item : next.getValue().entrySet())
   {

 System.out.println(Bytes.toString(item.getKey()));
   for (final EntryLong, byte[] _item :
 item.getValue().entrySet())
   {
   System.out.println(_item.getKey()); //q

Re: I can't get many versions of the specified column,but only get the latest version of the specified column

2011-02-23 Thread Tatsuya Kawano

Hi Jiajun, 

Make sure you don't have the same timestamp on every versions of puts; try to 
put Thread.sleep() in your test(?) codes when necessary. 

You might not want to specify the timestamp by yourself but want to let HBase 
to store appropriate ones. 

--
Tatsuya Kawano (Mr.)
Tokyo, Japan


On Feb 24, 2011, at 11:30 AM, Ryan Rawson ryano...@gmail.com wrote:

 There are test cases for this, the functionality DOES work, something is up...
 
 Without full code and full descriptions of your tables, debugging is
 harder than it needs to be.  It's probably a simple typo or something,
 check your code and table descriptions again. Many people rely on the
 multi version query capabilities and it is very unlikely to be broken
 in a released version of hbase.
 
 On Wed, Feb 23, 2011 at 6:27 PM, 陈加俊 cjjvict...@gmail.com wrote:
  final ListKeyValue list = result.list();
   for (final KeyValue it : list)
   {
   System.out.println(Bytes.toString(it.getKey()));
   System.out.println(Bytes.toString(it.getValue()));
   }
 I can only get the last version!  why ? Is there any testcase in Hbase?
 
 On Thu, Feb 24, 2011 at 9:56 AM, 陈加俊 cjjvict...@gmail.com wrote:
 
 /**
   * Create a sorted list of the KeyValue's in this result.
   *
   * @return The sorted list of KeyValue's.
   */
  public ListKeyValue list() {
if(this.kvs == null) {
  readFields();
}
return isEmpty()? null: Arrays.asList(sorted());
  }
 
 I will try it . Thank you very much!
 
 On Thu, Feb 24, 2011 at 9:45 AM, Buttler, David buttl...@llnl.gov wrote:
 
 Result.list() ?
 Putting the hbase source into your IDE of choice (yay Eclipse!) is really
 helpful
 
 Dave
 
 
 -Original Message-
 From: 陈加俊 [mailto:cjjvict...@gmail.com]
 Sent: Wednesday, February 23, 2011 5:42 PM
 To: user@hbase.apache.org
 Cc: Buttler, David
 Subject: Re: I can't get many versions of the specified column,but only
 get the latest version of the specified column
 
 Thank you David !
 
 I alter the table schema as follow:
 
 alter 'cjjIndexPageModify', {NAME = 'log' , VERSIONS = 5 , METHOD =
 'add'}
 
 How to iterate over KeyValues?  which method in Result?
 
 On Thu, Feb 24, 2011 at 9:27 AM, Buttler, David buttl...@llnl.gov
 wrote:
 
 What is your table schema set to?  By default it holds 3 versions.
 Also, you might iterating over KeyValues instead of using the Map since
 you
 don't really care about the organization, just the time.
 
 Dave
 
 -Original Message-
 From: 陈加俊 [mailto:cjjvict...@gmail.com]
 Sent: Wednesday, February 23, 2011 5:22 PM
 To: user@hbase.apache.org
 Cc: Stack
 Subject: Re: I can't get many versions of the specified column,but only
 get
 the latest version of the specified column
 
 I execute it five times at diffrent time.
 
 //put data by version
 
 final Put p = new Put(key); // key
 final long ts = System.currentTimeMillis();
  p.add(FAMILY, q1, ts,v1);
  p.add(FAMILY, q2, ts,v2);
  p.add(FAMILY, q3, ts,v3);
 table.put(p);
 
 So I can get five versions ,right?
 
 On Thu, Feb 24, 2011 at 2:06 AM, Stack st...@duboce.net wrote:
 
 What do you get for a result?
 
 You are only entering a single version of each column, a single
 version of FAMILY:q1, a single version FAMILY:q2, and a FAMILY:q3.
 
 St.Ack
 
 On Wed, Feb 23, 2011 at 2:54 AM, 陈加俊 cjjvict...@gmail.com wrote:
 I can't get many versions of the specified column,but only get the
 latest
 version of the specified column. Is there anyone  help me?
 
 //put data by version
 
 final Put p = new Put(key); // key
 final long ts = System.currentTimeMillis();
  p.add(FAMILY, q1, ts,v1);
  p.add(FAMILY, q2, ts,v2);
  p.add(FAMILY, q3, ts,v3);
 
 table.put(p);
 
 
 //get all versions of data
 
 final Get get = new Get(key);
  get.addFamily(FAMILY);
  get.setMaxVersions(10);
  final Result result = htable.get(get);
 
  final NavigableMapbyte[], NavigableMapbyte[],
 NavigableMapLong, byte[] map = result.getMap();
  final SetEntrybyte[], NavigableMapbyte[],
 NavigableMapLong, byte[] entrySet = map.entrySet();
  final IteratorEntrybyte[], NavigableMapbyte[],
 NavigableMapLong, byte[] iterator = entrySet
  .iterator();
  while (iterator.hasNext())
  {
  final Entrybyte[], NavigableMapbyte[],
 NavigableMapLong, byte[] next = iterator.next();
  System.out.println(Bytes.toString(next.getKey()));
 //family
  for (final Entrybyte[], NavigableMapLong, byte[]
 item : next.getValue().entrySet())
  {
  System.out.println(Bytes.toString(item.getKey()));
  for (final EntryLong, byte[] _item :
 item.getValue().entrySet())
  {
  System.out.println(_item.getKey()); //q
 
 System.out.println(Bytes.toString(_item.getValue()));
 //value
  }
  }
 
  }
 
 --
 Thanks  Best regards
 jiajun

Re: Stack Overflow?

2011-02-23 Thread Otis Gospodnetic

Hi David,

When I see people asking questions that others have asked before (and received 
answers) I tend to point them to those questions/answers via a tool, so they 
become aware of the tool, hopefully start using it, and thus check before 
asking 
next time around.  For Lucene, Solr, etc. I point people to appropriate search 
results on http://search-lucene.com and for HBase and friends I'd point people 
to search results or specific ML threads over on http://search-hadoop.com/ .  
That way a parallel knowledge base doesn't have to exist.

Otis


Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/



- Original Message 
 From: Buttler, David buttl...@llnl.gov
 To: user@hbase.apache.org user@hbase.apache.org
 Sent: Wed, February 23, 2011 8:31:32 PM
 Subject: Stack Overflow?
 
 Hi all,
 It seems that we are getting a lot of repeated questions now.   Perhaps it 
would be useful to start migrating the simple questions off to  stackoverflow 
(or whichever stack exchange website is most appropriate), and  just pointing 
people there?  Obviously there are still a lot of questions  that can't be 
migrated over since they require detailed examination of logs, but  some solid 
answers over there may help keep the signal high on this  list.
 
 Dave

Re: HBase 0.90.0 cannot be put more data after running hours

2011-02-23 Thread Anty

1) when there are only 2 client threads, vmstat output is
procs ---memory-- ---swap-- -io --system--
-cpu--
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 29236 161 557488 12027072 0 0 19 53 0 0 3 1 97 0 0
0 0 29236 1610152 557488 12027076 0 0 0 0 4630 23177 2 1 97 0 0
0 0 29236 1610484 557488 12027076 0 0 0 603 4604 22758 2 1 97 0 0
0 0 29236 1610544 557488 12027076 0 0 0 11 4643 23009 2 1 97 0 0
0 0 29236 1610912 557488 12027080 0 0 0 5 4578 22900 2 1 97 0 0
3 0 29236 1610600 557488 12027080 0 0 0 21 4627 22922 2 1 97 0 0
1 0 29236 1610616 557488 12027080 0 0 0 0 4552 23077 2 1 97 0 0
0 0 29236 1610992 557488 12027084 0 0 0 608 4608 22995 2 1 97 0 0
0 0 29236 1611900 557488 12027084 0 0 0 144 4641 23000 2 1 97 0 0
0 0 29236 1612040 557488 12027084 0 0 0 0 4621 22952 2 1 97 0 0
0 0 29236 1611428 557488 12027088 0 0 0 593 4668 23133 2 1 96 0 0
1 0 29236 1611812 557488 12027088 0 0 0 0 4623 23673 2 1 97 0 0
0 0 29236 1612068 557488 12027088 0 0 0 145 4627 23245 2 1 97 0 0
1 0 29236 1612440 557488 12027092 0 0 0 1 4654 23351 2 1 97 0 0
1 0 29236 1612120 557488 12027092 0 0 0 131 4695 23137 4 1 95 0 0
2 0 29236 1612492 557492 12027096 0 0 0 39 4647 22818 2 1 97 0 0
1 0 29236 1610164 557492 12027096 0 0 0 633 4666 22914 2 1 97 0 0
0 0 29236 1610380 557492 12027096 0 0 0 73 4668 22957 2 1 97 0 0
1 0 29236 1610420 557492 12027100 0 0 0 47 4642 22907 2 1 97 0 0
0 0 29236 1611596 557492 12027100 0 0 0 0 4664 22999 2 1 97 0 0
1 0 29236 1612236 557492 12027084 0 0 0 657 4665 22552 2 1 97 0 0
1 0 29236 1612732 557492 12027088 0 0 0 0 4586 22640 2 1 97 0 0
0 0 29236 1612504 557492 12027088 0 0 0 33 4605 22670 2 1 97 0 0
2 0 29236 1611500 557492 12027092 0 0 0 161 4575 22763 2 1 97 0 0
0 0 29236 1612160 557500 12027092 0 0 0 5 4570 23265 2 1 97 0 0
0 0 29236 1612568 557500 12027092 0 0 0 592 4616 23285 2 1 97 0 0
0 0 29236 1612700 557500 12027096 0 0 0 0 4688 23754 3 1 96 0 0
1 0 29236 1612192 557504 12027092 0 0 0 27 4649 23501 2 1 97 0 0
1 0 29236 1611748 557504 12027104 0 0 0 12 4612 23664 2 1 97 0 0
0 0 29236 1611932 557504 12027116 0 0 0 768 4606 22910 2 1 97 0 0
2 0 29236 1611788 557504 12027116 0 0 0 25 4545 22991 2 1 97 0 0
1 0 29236 1611916 557504 12027120 0 0 0 0 4615 23138 2 1 97 0 0
0 0 29236 1612176 557504 12027120 0 0 0 139 4604 23231 2 1 97 0 0
0 0 29236 1612848 557504 12027108 0 0 0 0 4624 23673 2 1 97 0 0
0 0 29236 1612820 557504 12027108 0 0 0 616 4657 23247 2 1 97 0 0
0 0 29236 1613196 557508 12027108 0 0 0 107 4581 23193 2 1 97 0 0
1 0 29236 1611772 557508 12027112 0 0 0 185 4594 22807 2 1 97 0 0
2 0 29236 1610432 557508 12027116 0 0 0 1 4603 23395 2 1 97 0 0
2 0 29236 1610348 557512 12027116 0 0 0 653 4724 23562 2 1 97 0 0
0 0 29236 1610612 557512 12027132 0 0 0 0 4621 23533 2 1 97 0 0
1 0 29236 1612368 557512 12027116 0 0 0 77 4609 23223 2 1 97 0 0
0 0 29236 1612628 557512 12027120 0 0 0 0 4571 22697 2 1 97 0 0
1 0 29236 1610908 557512 12027120 0 0 0 19 4585 22614 2 1 97 0 0
1 0 29236 1610020 557512 12027124 0 0 0 629 4656 23382 2 1 97 0 0


2) up client threads number to 8, vmstat output is


procs ---memory-- ---swap-- -io --system--
-cpu--
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 29236 161 557488 12027072 0 0 19 53 0 0 3 1 97 0 0
0 0 29236 1610152 557488 12027076 0 0 0 0 4630 23177 2 1 97 0 0
0 0 29236 1610484 557488 12027076 0 0 0 603 4604 22758 2 1 97 0 0
0 0 29236 1610544 557488 12027076 0 0 0 11 4643 23009 2 1 97 0 0
0 0 29236 1610912 557488 12027080 0 0 0 5 4578 22900 2 1 97 0 0
3 0 29236 1610600 557488 12027080 0 0 0 21 4627 22922 2 1 97 0 0
1 0 29236 1610616 557488 12027080 0 0 0 0 4552 23077 2 1 97 0 0
0 0 29236 1610992 557488 12027084 0 0 0 608 4608 22995 2 1 97 0 0
0 0 29236 1611900 557488 12027084 0 0 0 144 4641 23000 2 1 97 0 0
0 0 29236 1612040 557488 12027084 0 0 0 0 4621 22952 2 1 97 0 0
0 0 29236 1611428 557488 12027088 0 0 0 593 4668 23133 2 1 96 0 0
1 0 29236 1611812 557488 12027088 0 0 0 0 4623 23673 2 1 97 0 0
0 0 29236 1612068 557488 12027088 0 0 0 145 4627 23245 2 1 97 0 0
1 0 29236 1612440 557488 12027092 0 0 0 1 4654 23351 2 1 97 0 0
1 0 29236 1612120 557488 12027092 0 0 0 131 4695 23137 4 1 95 0 0
2 0 29236 1612492 557492 12027096 0 0 0 39 4647 22818 2 1 97 0 0
1 0 29236 1610164 557492 12027096 0 0 0 633 4666 22914 2 1 97 0 0
0 0 29236 1610380 557492 12027096 0 0 0 73 4668 22957 2 1 97 0 0
1 0 29236 1610420 557492 12027100 0 0 0 47 4642 22907 2 1 97 0 0
0 0 29236 1611596 557492 12027100 0 0 0 0 4664 22999 2 1 97 0 0
1 0 29236 1612236 557492 12027084 0 0 0 657 4665 22552 2 1 97 0 0
1 0 29236 1612732 557492 12027088 0 0 0 0 4586 22640 2 1 97 0 0
0 0 29236 1612504 557492 12027088 0 0 0 33 4605 22670 2 1 97 0 0
2 0 29236 1611500 557492 12027092 0 0 0 161 4575 22763 2 1 97 0 0
0 0 29236 1612160 557500 12027092 0 0 0 5 4570 23265 2 1 97 0 0
0 0 29236 1612568 557500 12027092 0 0 0 592 4616 23285 2 1 97 0 0
0 0 29236 1612700 557500 12027096 0 0 0 0 4688 23754 3 1 96 0 0

Re: HBase 0.90.0 cannot be put more data after running hours

2011-02-23 Thread Anty

Sorry, the vmstat output for 2) is wrong.
1) when there are only 2 client threads, vmstat output is

procs ---memory-- ---swap-- -io --system--
-cpu--
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 29236 161 557488 12027072 0 0 19 53 0 0 3 1 97 0 0
0 0 29236 1610152 557488 12027076 0 0 0 0 4630 23177 2 1 97 0 0
0 0 29236 1610484 557488 12027076 0 0 0 603 4604 22758 2 1 97 0 0
0 0 29236 1610544 557488 12027076 0 0 0 11 4643 23009 2 1 97 0 0
0 0 29236 1610912 557488 12027080 0 0 0 5 4578 22900 2 1 97 0 0
3 0 29236 1610600 557488 12027080 0 0 0 21 4627 22922 2 1 97 0 0
1 0 29236 1610616 557488 12027080 0 0 0 0 4552 23077 2 1 97 0 0
0 0 29236 1610992 557488 12027084 0 0 0 608 4608 22995 2 1 97 0 0
0 0 29236 1611900 557488 12027084 0 0 0 144 4641 23000 2 1 97 0 0
0 0 29236 1612040 557488 12027084 0 0 0 0 4621 22952 2 1 97 0 0
0 0 29236 1611428 557488 12027088 0 0 0 593 4668 23133 2 1 96 0 0
1 0 29236 1611812 557488 12027088 0 0 0 0 4623 23673 2 1 97 0 0
0 0 29236 1612068 557488 12027088 0 0 0 145 4627 23245 2 1 97 0 0
1 0 29236 1612440 557488 12027092 0 0 0 1 4654 23351 2 1 97 0 0
1 0 29236 1612120 557488 12027092 0 0 0 131 4695 23137 4 1 95 0 0
2 0 29236 1612492 557492 12027096 0 0 0 39 4647 22818 2 1 97 0 0
1 0 29236 1610164 557492 12027096 0 0 0 633 4666 22914 2 1 97 0 0
0 0 29236 1610380 557492 12027096 0 0 0 73 4668 22957 2 1 97 0 0
1 0 29236 1610420 557492 12027100 0 0 0 47 4642 22907 2 1 97 0 0
0 0 29236 1611596 557492 12027100 0 0 0 0 4664 22999 2 1 97 0 0
1 0 29236 1612236 557492 12027084 0 0 0 657 4665 22552 2 1 97 0 0
1 0 29236 1612732 557492 12027088 0 0 0 0 4586 22640 2 1 97 0 0
0 0 29236 1612504 557492 12027088 0 0 0 33 4605 22670 2 1 97 0 0
2 0 29236 1611500 557492 12027092 0 0 0 161 4575 22763 2 1 97 0 0
0 0 29236 1612160 557500 12027092 0 0 0 5 4570 23265 2 1 97 0 0
0 0 29236 1612568 557500 12027092 0 0 0 592 4616 23285 2 1 97 0 0
0 0 29236 1612700 557500 12027096 0 0 0 0 4688 23754 3 1 96 0 0
1 0 29236 1612192 557504 12027092 0 0 0 27 4649 23501 2 1 97 0 0
1 0 29236 1611748 557504 12027104 0 0 0 12 4612 23664 2 1 97 0 0
0 0 29236 1611932 557504 12027116 0 0 0 768 4606 22910 2 1 97 0 0
2 0 29236 1611788 557504 12027116 0 0 0 25 4545 22991 2 1 97 0 0
1 0 29236 1611916 557504 12027120 0 0 0 0 4615 23138 2 1 97 0 0
0 0 29236 1612176 557504 12027120 0 0 0 139 4604 23231 2 1 97 0 0
0 0 29236 1612848 557504 12027108 0 0 0 0 4624 23673 2 1 97 0 0
0 0 29236 1612820 557504 12027108 0 0 0 616 4657 23247 2 1 97 0 0
0 0 29236 1613196 557508 12027108 0 0 0 107 4581 23193 2 1 97 0 0
1 0 29236 1611772 557508 12027112 0 0 0 185 4594 22807 2 1 97 0 0
2 0 29236 1610432 557508 12027116 0 0 0 1 4603 23395 2 1 97 0 0
2 0 29236 1610348 557512 12027116 0 0 0 653 4724 23562 2 1 97 0 0
0 0 29236 1610612 557512 12027132 0 0 0 0 4621 23533 2 1 97 0 0
1 0 29236 1612368 557512 12027116 0 0 0 77 4609 23223 2 1 97 0 0
0 0 29236 1612628 557512 12027120 0 0 0 0 4571 22697 2 1 97 0 0
1 0 29236 1610908 557512 12027120 0 0 0 19 4585 22614 2 1 97 0 0
1 0 29236 1610020 557512 12027124 0 0 0 629 4656 23382 2 1 97 0 0
On Thu, Feb 24, 2011 at 11:39 AM, Anty anty@gmail.com wrote:


2) up client threads number to 8, vmstat output is

procs ---memory-- ---swap-- -io --system--
-cpu--
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 29236 1442908 557276 12026508 0 0 19 53 0 0 3 1 97 0 0
1 0 29236 1443652 557276 12026508 0 0 0 0 1362 4740 8 0 91 0 0
1 0 29236 1443676 557276 12026508 0 0 0 36 1530 7361 10 0 90 0 0
0 0 29236 1443384 557284 12026512 0 0 0 728 1327 4565 8 0 92 0 0
1 0 29236 1442864 557284 12026512 0 0 0 5 1558 4889 10 0 90 0 0
1 0 29236 1443028 557284 12026516 0 0 0 24 1321 5751 9 0 91 0 0
2 0 29236 1440228 557296 12026512 0 0 0 61 1399 5529 9 0 90 0 0
2 0 29236 1439756 557296 12026528 0 0 0 91 1450 3485 8 0 92 0 0
1 0 29236 1440164 557296 12026528 0 0 0 217 1433 7146 10 0 90 0 0
1 0 29236 1439604 557296 12026528 0 0 0 4 1387 3420 8 0 92 0 0
4 0 29236 1439604 557296 12026532 0 0 0 593 1438 6291 9 0 90 0 0
1 0 29236 1439624 557296 12026532 0 0 0 41 1296 7312 9 0 91 0 0
1 0 29236 1439632 557296 12026532 0 0 0 1 1417 6495 9 0 91 0 0
1 0 29236 1439820 557296 12026536 0 0 0 59 1379 5482 10 0 90 0 0
1 0 29236 1439112 557296 12026540 0 0 0 12 1409 3470 8 0 92 0 0
1 0 29236 1438024 557296 12026556 0 0 0 93 1384 4741 8 0 91 0 0
1 0 29236 1437672 557296 12026556 0 0 0 627 1298 6614 10 0 90 0 0
1 0 29236 1438288 557296 12026556 0 0 0 4 1482 4857 8 0 91 0 0
1 0 29236 1438636 557296 12026560 0 0 0 169 1421 6282 10 0 90 0 0
1 0 29236 1438576 557296 12026560 0 0 0 431 1309 6078 9 0 91 0 0
1 0 29236 1438744 557296 12026560 0 0 0 728 1357 7340 9 0 91 0 0
1 0 29236 1439008 557296 12026564 0 0 0 12 1446 6676 10 0 90 0 0
4 0 29236 1438384 557296 12026564 0 0 0 3 1350 4046 9 0 91 0 0
2 0 29236 1438644 557296 12026568 0 0 0 144 1429 3761 8 0 92 0 0
1 0 29236 1438776 557296 12026568 0 0 0 1 1285 5099 8 0 91 0 0
1 0 29236 1438760 557296 12026568 0 0 0

Re: HBase 0.90.0 cannot be put more data after running hours

2011-02-23 Thread Schubert Zhang

On Sat, Jan 29, 2011 at 1:02 AM, Stack st...@duboce.net wrote:

 On Thu, Jan 27, 2011 at 10:33 PM, Schubert Zhang zson...@gmail.com
 wrote:
  1. The .META. table seems ok
  I can read my data table (one thread for reading).
  I can use hbase shell to scan my data table.
  And I can use 1~4 threads to put more data into my data table.
 

 Good.  This would seem to say that .META. is not locked out (You are
 doing these scans while your 8+client process is hung?).


Yes, write and read at the same time.



 Before this issue happen, about 800 millions entities (column) have
 been
  put into the table successfully, and there are 253 regions for this
 table.
 


 So, you were running fine with 8+ clients until you hit the 800million
 entries?


Yes, it ware running fine before this issue happen.




  3. All clients use HBaseConfiguration.create() for a new Configuration
  instance.
 

 Do you do this for each new instance of HTable or do you pass them all
 the same Configuration instance?


Every client thread use HBaseConfiguration.create() to create a new
Configuration and use it to new HTable.



  4. The 8+ client threads running on a single machine and a single JVM.
 

 How many instances of this process?  One or many?



In this process, 8 threads each with a instance of HTable (i.e. totally 8
HTable instances)




  5. Seems all 8+ threads are blocked in same location waiting on call to
  return.
 

 If you want to paste a thread dump of your client, some one of us will
 give it a gander.

 St.Ack

Re: HBase 0.90.0 cannot be put more data after running hours

2011-02-23 Thread Schubert Zhang

Currently, with 0.90.1, this issue happen when there is only 8 regions in
each RS, and totally 64 regions in all totally 8 RS.

Ths CPU% of the client is very high.

On Thu, Feb 24, 2011 at 10:55 AM, Schubert Zhang zson...@gmail.com wrote:

 Now, I am trying the 0.90.1, but this issue is still there.

 I attach the jstack output. Coud you please help me analyze it.

 Seems all the 8 client threads are doing metaScan!

   On Sat, Jan 29, 2011 at 1:02 AM, Stack st...@duboce.net wrote:

 On Thu, Jan 27, 2011 at 10:33 PM, Schubert Zhang zson...@gmail.com
 wrote:
  1. The .META. table seems ok
  I can read my data table (one thread for reading).
  I can use hbase shell to scan my data table.
  And I can use 1~4 threads to put more data into my data table.
 

 Good.  This would seem to say that .META. is not locked out (You are
 doing these scans while your 8+client process is hung?).


 Before this issue happen, about 800 millions entities (column) have
 been
  put into the table successfully, and there are 253 regions for this
 table.
 


 So, you were running fine with 8+ clients until you hit the 800million
 entries?


  3. All clients use HBaseConfiguration.create() for a new Configuration
  instance.
 

 Do you do this for each new instance of HTable or do you pass them all
 the same Configuration instance?


  4. The 8+ client threads running on a single machine and a single JVM.
 

 How many instances of this process?  One or many?


  5. Seems all 8+ threads are blocked in same location waiting on call to
  return.
 

 If you want to paste a thread dump of your client, some one of us will
 give it a gander.

 St.Ack

Install problem - HBase 0.90.1 cannot connect to zookeeper

2011-02-23 Thread sun sf

I have installed HBase0.20.6 successfully but I met the following
problem when try to install HBase0.90.1.

It always says zookeepee cannot be connected when we use the same
configuration
as HBase0.20.6.

At last, I reinstalled the CentOS5.5, and start HBase0.90.1 in Stand alone,
the following errors shows in log file

org.apache.haddop.hase.zookeeper.MiniZookeeperCluster
Server localhost:2181 not up java.net.NoRouteToHostException
No route to host MinizookeeperCluster.java 1221


I have noticed that
*If downloading HBase 0.90.0*, make sure your reverse DNS on Master node is
in sync with what the RegionServer reports as its hostname; see
HBase-3431https://issues.apache.org/jira/browse/HBASE-3431.

In
http://ftp.kddilabs.jp/infosystems/apache//hbase/

Is there any connection?
Thanks in advance.

Re: Stack Overflow?

2011-02-23 Thread Stack

Hey David:

Yeah, a few of us have started to refer to the 'two week cycle' where
it seems the same questions come around again.

Karl Fogels' Producing Open Source Software,
http://producingoss.com/en/producingoss.pdf, has a good section on
this topic.  In it he advocates 'Conspicuous Use of Archives':

Use those archives as much as possible, and as conspicuously as
possible. Even when you know the
answer to some question off the top of your head, if you think there's
a reference in the archives that
contains the answer, spend the time to dig it up and present it. Every
time you do that in a publicly
visible way, some people learn for the first time that the archives
are there, and that searching in them
can produce answers. Also, by referring to the archives instead of
rewriting the advice, you reinforce
the social norm against duplicating information. Why have the same
answer in two different places?
[Pg 105]

The new search-hadoop.com search box that is at the top right hand
corner of hbase.apache.org since the new maven-generated 0.90.0 hbase
website went up, makes the digging in archives quite a bit easier.

Going forward I for one was going to try and mine our archives more at
least for dealing with the repeats.

Thanks for bringing up this topic David,
St.Ack



On Wed, Feb 23, 2011 at 5:31 PM, Buttler, David buttl...@llnl.gov wrote:
 Hi all,
 It seems that we are getting a lot of repeated questions now.  Perhaps it 
 would be useful to start migrating the simple questions off to stackoverflow 
 (or whichever stack exchange website is most appropriate), and just pointing 
 people there?  Obviously there are still a lot of questions that can't be 
 migrated over since they require detailed examination of logs, but some solid 
 answers over there may help keep the signal high on this list.

 Dave

Re: Install problem - HBase 0.90.1 cannot connect to zookeeper

2011-02-23 Thread sun sf

Thank you for your quick reply.

I know there are several different default configurations
between HBase0.90.1 and HBase0.20.6.

And so I tried pseudo and standalone install,
it seems both of them had the same zookeeper error.

In the standalone,I only added the root.dir to the hbase-site.xml
and not install hadoop.

I tried again and the error message turned to be
the following.

I attached the log file to make it more clear.
---
2011-02-23 15:09:46,980 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server localhost/192.168.70.5:2181
2011-02-23 15:09:46,981 WARN org.apache.zookeeper.ClientCnxn: Session
0x12e54c8de7f0001 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
2011-02-23 15:09:46,986 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server localhost/192.168.70.5:2181
2011-02-23 15:09:46,987 WARN org.apache.zookeeper.ClientCnxn: Session
0x12e54c8de7f0002 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
---


On Thu, Feb 24, 2011 at 2:33 PM, Stack st...@duboce.net wrote:

 On Wed, Feb 23, 2011 at 9:08 PM, sun sf revlet...@gmail.com wrote:
 
 
  org.apache.haddop.hase.zookeeper.MiniZookeeperCluster
  Server localhost:2181 not up java.net.NoRouteToHostException
  No route to host MinizookeeperCluster.java 1221

 Your networking on that machine is borked when it says
 NoRouteToHostException.  Poke around in the logs.  Its probably trying
 to connect to localhost or whatever its finding for hostname on this
 machine.  Trying doing same with telnet.  You'll probably run into the
 NoRouteToHost message.  Fix it first.

 Be careful using same config. for 0.90.1 as you used for 0.20.6.  The
 default values have changed as has how you do config. for zookeeper.
 Start with 0.90.1 defaults and change one at a time.   Go by the
 documentation, here http://hbase.apache.org/notsoquick.html, to be
 sure.

 Hope this helps,
 St.Ack

Re: Install problem - HBase 0.90.1 cannot connect to zookeeper

2011-02-23 Thread Stack

Zookeeper ensemble is not running.  See logs for why.
St.Ack

On Wed, Feb 23, 2011 at 9:46 PM, sun sf revlet...@gmail.com wrote:
 Thank you for your quick reply.

 I know there are several different default configurations
 between HBase0.90.1 and HBase0.20.6.

 And so I tried pseudo and standalone install,
 it seems both of them had the same zookeeper error.

 In the standalone,I only added the root.dir to the hbase-site.xml
 and not install hadoop.

 I tried again and the error message turned to be
 the following.

 I attached the log file to make it more clear.
 ---
 2011-02-23 15:09:46,980 INFO org.apache.zookeeper.ClientCnxn: Opening socket
 connection to server localhost/192.168.70.5:2181
 2011-02-23 15:09:46,981 WARN org.apache.zookeeper.ClientCnxn: Session
 0x12e54c8de7f0001 for server null, unexpected error, closing socket
 connection and attempting reconnect
 java.net.ConnectException: Connection refused
     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
     at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
     at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
 2011-02-23 15:09:46,986 INFO org.apache.zookeeper.ClientCnxn: Opening socket
 connection to server localhost/192.168.70.5:2181
 2011-02-23 15:09:46,987 WARN org.apache.zookeeper.ClientCnxn: Session
 0x12e54c8de7f0002 for server null, unexpected error, closing socket
 connection and attempting reconnect
 java.net.ConnectException: Connection refused
     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
     at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
     at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
 ---


 On Thu, Feb 24, 2011 at 2:33 PM, Stack st...@duboce.net wrote:

 On Wed, Feb 23, 2011 at 9:08 PM, sun sf revlet...@gmail.com wrote:
 
  
  org.apache.haddop.hase.zookeeper.MiniZookeeperCluster
  Server localhost:2181 not up java.net.NoRouteToHostException
  No route to host MinizookeeperCluster.java 1221

 Your networking on that machine is borked when it says
 NoRouteToHostException.  Poke around in the logs.  Its probably trying
 to connect to localhost or whatever its finding for hostname on this
 machine.  Trying doing same with telnet.  You'll probably run into the
 NoRouteToHost message.  Fix it first.

 Be careful using same config. for 0.90.1 as you used for 0.20.6.  The
 default values have changed as has how you do config. for zookeeper.
 Start with 0.90.1 defaults and change one at a time.   Go by the
 documentation, here http://hbase.apache.org/notsoquick.html, to be
 sure.

 Hope this helps,
 St.Ack

Re: Install problem - HBase 0.90.1 cannot connect to zookeeper

2011-02-23 Thread sun sf

St.Ack

I found the past question you have answerd.

I have checked the out log file and it gives the same errors

-
http://hbase.apache.org/docs/r0.20.6/cygwin.html
Stack
stack@...http://gmane.org/get-address.php?address=stack%2d%2bc050i%2b%2bPmesTnJN9%2bBGXg%40public.gmane.org

2011-01-16 01:44:52 GMT

It looks like the zookeeper ensemble has not started. Is that possible?

2011-01-14 19:42:03,773 WARN org.apache.zookeeper.ClientCnxn:
Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl at 76e8a7^M
java.net.ConnectException: Connection refused: no further information^M
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)^M
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)^M
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)^M

Is this standalone or pseudo-distributed or what? Unless you tampered
with settings in hbase-env.sh, ZK should be started for you. Check
out the .out file or if a zookeeper log beside this master log, check
that.

I see that windows would seem to set ulimit for files to 256 by
default. Once you get things going, try and up that if you can.

St.Ack

---

On Thu, Feb 24, 2011 at 3:31 PM, sun sf revlet...@gmail.com wrote:

I only use the default configuration to start.

I check the log but I do not know how to where the
Zookeeper ensemble error lies.

Can you tell me how to check Zookeeper ensemble status
and how to fix the problem?

Thanks in advance.

On Thu, Feb 24, 2011 at 3:01 PM, Stack st...@duboce.net wrote:

Zookeeper ensemble is not running. See logs for why.
St.Ack

On Wed, Feb 23, 2011 at 9:46 PM, sun sf revlet...@gmail.com wrote:
Thank you for your quick reply.

I know there are several different default configurations
between HBase0.90.1 and HBase0.20.6.

And so I tried pseudo and standalone install,
it seems both of them had the same zookeeper error.

In the standalone,I only added the root.dir to the hbase-site.xml
and not install hadoop.

I tried again and the error message turned to be
the following.

I attached the log file to make it more clear.

---
2011-02-23 15:09:46,980 INFO org.apache.zookeeper.ClientCnxn: Opening
socket
connection to server localhost/192.168.70.5:2181
2011-02-23 15:09:46,981 WARN org.apache.zookeeper.ClientCnxn: Session
0x12e54c8de7f0001 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
2011-02-23 15:09:46,986 INFO org.apache.zookeeper.ClientCnxn: Opening
socket
connection to server localhost/192.168.70.5:2181
2011-02-23 15:09:46,987 WARN org.apache.zookeeper.ClientCnxn: Session
0x12e54c8de7f0002 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)

---

On Thu, Feb 24, 2011 at 2:33 PM, Stack st...@duboce.net wrote:

On Wed, Feb 23, 2011 at 9:08 PM, sun sf revlet...@gmail.com wrote:

org.apache.haddop.hase.zookeeper.MiniZookeeperCluster
Server localhost:2181 not up java.net.NoRouteToHostException
No route to host MinizookeeperCluster.java 1221

Your networking on that machine is borked when it says
NoRouteToHostException. Poke around in the logs. Its probably trying
to connect to localhost or whatever its finding for hostname on this
machine. Trying doing same with telnet. You'll probably run into the
NoRouteToHost message. Fix it first.

Be careful using same config. for 0.90.1 as you used for 0.20.6. The
default values have changed as has how you do config. for zookeeper.
Start with 0.90.1 defaults and change one at a time. Go by the
documentation, here http://hbase.apache.org/notsoquick.html, to be
sure.

Hope this helps,
St.Ack

Re: Stargate

2011-02-23 Thread Lars George

Hi Mike,

The values are Base64 encoded, so you need to use a decoder. HBase
ships with one in the REST package that you can use for example.

Lars

On Wed, Feb 23, 2011 at 7:22 PM, Mike mi...@yesmail.com wrote:
 I'm having some issues converting the results of a restful call through
 stargate.  I'm returning the data as a json representation which appears to 
 work
 fine as it returns the desired fields:

 JsonRepresentation jr = new
 JsonRepresentation(resource.get(MediaType.APPLICATION_JSON));

 When I parse through the Json I can break down the values and print them out:

 time: 1298398387275, Value: MTEzMzQz, Column: ZGF0YTpjaWQ=
 time: 1298398387275, Value:
 NTUwIGFubmRhdmlzQGVhcnRobGluay5uZXQuLi5Vc2VyIGFjY291bnQgaXMgdW5hdmFpbGFibGU=,
 Column: ZGF0YTpkZXRhaWxz
 time: 1298398387275, Value: ZWFydGhsaW5rLm5ldA==, Column: ZGF0YTpkb21haW4=
 time: 1298398387275, Value: MTMwNjA2Mg==, Column: ZGF0YTptaWQ=

 But I cannot seem to convert the values to a string.  I've tried using the
 Bytes.toString and recieve the following Error:

 xception in thread main java.lang.NoClassDefFoundError:
 org/apache/hadoop/io/RawComparator

 Any help is appreciated.

Re: I can't get many versions of the specified column,but only get the latest version of the specified column

Which line is line 89?

Also it's preferable to do:
assertEquals(3, versionMap.size());
vs:
assertTrue(versionMap.size() == 3);

since the error messages from the former are more descriptive
expected 3 was 2.

looking at the code it looks like it should work...

On Wed, Feb 23, 2011 at 11:07 PM, 陈加俊 cjjvict...@gmail.com wrote:
  This is my test case ,but I get NPE some times .

 java.lang.NullPointerException
 at
 com.uuwatch.idm.hbase.GetRowVersionsTest.testGetRowMultipleVersions(GetRowVersionsTest.java:89)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
 at
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
 at
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
 at
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
 at
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
 at
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
 at
 org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49)
 at
 org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
 at
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
 at
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
 at
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
 at
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

 public class GetRowVersionsTest extends TestCase
 {
    private final byte[] family    = Bytes.toBytes(log);

    private final byte[] qualifier = Bytes.toBytes(siteUrl);

    private final byte[] rowKey    = Bytes.toBytes(1);

    private final HTable table     =
 IDMHBaseConfiguration.getTable(BigTableName.getSite());

    private final long   ts1       = 1298529542218L;

    private final long   ts2       = ts1 + 100;

    private final long   ts3       = ts1 + 100;

    private final byte[] value1    = Bytes.toBytes(value1);

    private final byte[] value2    = Bytes.toBytes(value2);

    private final byte[] value3    = Bytes.toBytes(value3);

    private void insert(final long ts, final byte[] value) throws
 IOException
    {
        final Put put = new Put(rowKey);
        put.add(family, qualifier, ts, value);
        table.put(put);
    }

    private void sleep()
    {
        try
        {
            Thread.sleep(1000);
        }
        catch (final InterruptedException e)
        {
            e.printStackTrace();
        }
    }

    @Test
    public void testGetRowMultipleVersions() throws Exception
    {
        insert(ts1, value1);
        sleep();
        insert(ts2, value2);
        sleep();
        insert(ts3, value3);
        sleep();

        // check getRow with multiple versions
        final Get get = new Get(rowKey);
        get.setMaxVersions();
        final Result r = table.get(get);
        final NavigableMapbyte[], NavigableMapbyte[], NavigableMapLong,
 byte[] map = r.getMap();
        final NavigableMapbyte[], NavigableMapLong, byte[] familyMap =
 map.get(family);
        final NavigableMapLong, byte[] versionMap =
 familyMap.get(qualifier);
        for (final Map.EntryLong, byte[] entry : versionMap.entrySet())
        {
            System.err.println(entry.getKey());
            System.err.println(Bytes.toString(entry.getValue()));
        }
        //        assertTrue(versionMap.size() == 3);

        //        assertTrue(value1 == versionMap.get(ts1));
        //        assertTrue(value2 == versionMap.get(ts2));
        //        assertTrue(value3 == versionMap.get(ts3));

        //        table.delete(new Delete(rowKey));
        //        assertTrue(table.get(get).size() == 0);
        //        table.close();
    }
 }


 On Thu, Feb 24, 2011 at 11:07 AM, Tatsuya Kawano tatsuya6...@gmail.comwrote:

 Hi Jiajun,

 Make sure you don't have the same timestamp on every versions of puts; try
 to put Thread.sleep() in your test(?) codes when necessary.

 You might not want to specify the timestamp by yourself but want to let
 HBase to store appropriate ones.

 --
 Tatsuya Kawano (Mr.)
 Tokyo, Japan


 On Feb 24, 2011, at

Re: I can't get many versions of the specified column,but only get the latest version of the specified column

line 89:final NavigableMapbyte[], NavigableMapLong, byte[]
familyMap = map.get(family);
map is null ,
and strangely  I use r.list() instead,
final ListKeyValue list = r.list();
r is null !



2011/2/24 Ryan Rawson ryano...@gmail.com

 Which line is line 89?

 Also it's preferable to do:
 assertEquals(3, versionMap.size());
 vs:
 assertTrue(versionMap.size() == 3);

 since the error messages from the former are more descriptive
 expected 3 was 2.

 looking at the code it looks like it should work...

 On Wed, Feb 23, 2011 at 11:07 PM, 陈加俊 cjjvict...@gmail.com wrote:
   This is my test case ,but I get NPE some times .
 
  java.lang.NullPointerException
  at
 
 com.uuwatch.idm.hbase.GetRowVersionsTest.testGetRowMultipleVersions(GetRowVersionsTest.java:89)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at
 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  at
 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at
 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
  at
 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
  at
 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
  at
 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
  at
 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
  at
 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
  at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
  at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
  at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
  at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
  at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
  at
 
 org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49)
  at
 
 org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
  at
 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
  at
 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
  at
 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
  at
 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
 
  public class GetRowVersionsTest extends TestCase
  {
 private final byte[] family= Bytes.toBytes(log);
 
 private final byte[] qualifier = Bytes.toBytes(siteUrl);
 
 private final byte[] rowKey= Bytes.toBytes(1);
 
 private final HTable table =
  IDMHBaseConfiguration.getTable(BigTableName.getSite());
 
 private final long   ts1   = 1298529542218L;
 
 private final long   ts2   = ts1 + 100;
 
 private final long   ts3   = ts1 + 100;
 
 private final byte[] value1= Bytes.toBytes(value1);
 
 private final byte[] value2= Bytes.toBytes(value2);
 
 private final byte[] value3= Bytes.toBytes(value3);
 
 private void insert(final long ts, final byte[] value) throws
  IOException
 {
 final Put put = new Put(rowKey);
 put.add(family, qualifier, ts, value);
 table.put(put);
 }
 
 private void sleep()
 {
 try
 {
 Thread.sleep(1000);
 }
 catch (final InterruptedException e)
 {
 e.printStackTrace();
 }
 }
 
 @Test
 public void testGetRowMultipleVersions() throws Exception
 {
 insert(ts1, value1);
 sleep();
 insert(ts2, value2);
 sleep();
 insert(ts3, value3);
 sleep();
 
 // check getRow with multiple versions
 final Get get = new Get(rowKey);
 get.setMaxVersions();
 final Result r = table.get(get);
 final NavigableMapbyte[], NavigableMapbyte[], NavigableMapLong,
  byte[] map = r.getMap();
 final NavigableMapbyte[], NavigableMapLong, byte[] familyMap =
  map.get(family);
 final NavigableMapLong, byte[] versionMap =
  familyMap.get(qualifier);
 for (final Map.EntryLong, byte[] entry : versionMap.entrySet())
 {
 System.err.println(entry.getKey());
 System.err.println(Bytes.toString(entry.getValue()));
 }
 //assertTrue(versionMap.size() == 3);
 
 //assertTrue(value1 == versionMap.get(ts1));
 //assertTrue(value2 == versionMap.get(ts2));
 //assertTrue(value3 == versionMap.get(ts3));
 
 //table.delete(new Delete(rowKey));
 //assertTrue(table.get(get).size() == 0);
 //table.close();
 }
  }
 
 
  On Thu, Feb

Re: How to limit the number of logs that producted by DailyRollingFileAppender

I uncomment MaxBackupIndex and restart regionserver but warn message as
follows:

starting regionserver, logging to
/app/cloud/hbase/bin/../logs/hbase-uuwatch-regionserver-gentoo_uuwatch_183.out
log4j:WARN No such property [maxBackupIndex] in
org.apache.log4j.DailyRollingFileAppender.


On Thu, Feb 17, 2011 at 9:30 AM, Tatsuya Kawano tatsuya6...@gmail.comwrote:


 Did you try MaxBackupIndex? Didn't it work?

 I found this in Log4j wiki, but I don't know if this class has been merged
 into log4j trunk.

 http://wiki.apache.org/logging-log4j/DailyRollingFileAppender
 http://wiki.apache.org/logging-log4j/DailyRollingFileAppender

 I've change the DailyRollingFileAppender to support the 
 MaxBackupIndexhttp://wiki.apache.org/logging-log4j/MaxBackupIndex,
 this is the class to add to the jar that contains the log4j library:

 Since this is a log4j question, you could get better answer if you ask your
 question at log4j-user mailing list.

 Thanks,

 --
 Tatsuya Kawano
 Tokyo, Japan


 On Feb 17, 2011, at 10:17 AM, 陈加俊  cjjvict...@gmail.com
 cjjvict...@gmail.com wrote:

 I want to limit the number of log files for a DailyRollingFileAppender. I
 search for a parameter like maxBackupIndex (in RollingFileAppender) but can
 not find one. Is there really no way to limit the number of log-Files?

 What is the strategy to ensure that the (number) of log files do not
 overrun. Delete them manually?

 On Wed, Feb 16, 2011 at 5:56 PM, Tatsuya Kawano  
 tatsuya6...@gmail.comtatsuya6...@gmail.com
 tatsuya6...@gmail.com wrote:

 Hi,

 On 02/16/2011, at 4:51 PM, 陈加俊 wrote:
  How to limit the number of logs that producted by
 DailyRollingFileAppender ?
 
  I find the logs are exceeding  disk apace limit.


 If you're using log4j.properties coming with HBase, uncomment
 MaxBackupIndex property ant change its value.

 # 30-day backup
 #log4j.appender.DRFA.MaxBackupIndex=30


 Also, if it's for production environment, make sure you change the logging
 level to INFO, so you can reduce the amount of the logs.

 log4j.logger.org.apache.hadoop.hbase=INFO


 Thanks,

 --
 Tatsuya Kawano (Mr.)
 Tokyo, Japan

  http://twitter.com/#!/tatsuya6502 http://twitter.com/#!/tatsuya6502
 http://twitter.com/#!/tatsuya6502









 --
 Thanks  Best regards
 jiajun




-- 
Thanks  Best regards
jiajun

Re: I can't get many versions of the specified column,but only get the latest version of the specified column

Does the HTable object have setAutoFlush(false) turned on by any chance?

On Wed, Feb 23, 2011 at 11:22 PM, 陈加俊 cjjvict...@gmail.com wrote:
 line 89:        final NavigableMapbyte[], NavigableMapLong, byte[]
 familyMap = map.get(family);
 map is null ,
 and strangely  I use r.list() instead,
 final ListKeyValue list = r.list();
 r is null !


 2011/2/24 Ryan Rawson ryano...@gmail.com

 Which line is line 89?

 Also it's preferable to do:
 assertEquals(3, versionMap.size());
 vs:
 assertTrue(versionMap.size() == 3);

 since the error messages from the former are more descriptive
 expected 3 was 2.

 looking at the code it looks like it should work...

 On Wed, Feb 23, 2011 at 11:07 PM, 陈加俊 cjjvict...@gmail.com wrote:
   This is my test case ,but I get NPE some times .
 
  java.lang.NullPointerException
  at
 
  com.uuwatch.idm.hbase.GetRowVersionsTest.testGetRowMultipleVersions(GetRowVersionsTest.java:89)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at
 
  sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  at
 
  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at
 
  org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
  at
 
  org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
  at
 
  org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
  at
 
  org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
  at
 
  org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
  at
 
  org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
  at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
  at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
  at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
  at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
  at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
  at
 
  org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49)
  at
 
  org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
  at
 
  org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
  at
 
  org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
  at
 
  org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
  at
 
  org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
 
  public class GetRowVersionsTest extends TestCase
  {
     private final byte[] family    = Bytes.toBytes(log);
 
     private final byte[] qualifier = Bytes.toBytes(siteUrl);
 
     private final byte[] rowKey    = Bytes.toBytes(1);
 
     private final HTable table     =
  IDMHBaseConfiguration.getTable(BigTableName.getSite());
 
     private final long   ts1       = 1298529542218L;
 
     private final long   ts2       = ts1 + 100;
 
     private final long   ts3       = ts1 + 100;
 
     private final byte[] value1    = Bytes.toBytes(value1);
 
     private final byte[] value2    = Bytes.toBytes(value2);
 
     private final byte[] value3    = Bytes.toBytes(value3);
 
     private void insert(final long ts, final byte[] value) throws
  IOException
     {
         final Put put = new Put(rowKey);
         put.add(family, qualifier, ts, value);
         table.put(put);
     }
 
     private void sleep()
     {
         try
         {
             Thread.sleep(1000);
         }
         catch (final InterruptedException e)
         {
             e.printStackTrace();
         }
     }
 
     @Test
     public void testGetRowMultipleVersions() throws Exception
     {
         insert(ts1, value1);
         sleep();
         insert(ts2, value2);
         sleep();
         insert(ts3, value3);
         sleep();
 
         // check getRow with multiple versions
         final Get get = new Get(rowKey);
         get.setMaxVersions();
         final Result r = table.get(get);
         final NavigableMapbyte[], NavigableMapbyte[],
  NavigableMapLong,
  byte[] map = r.getMap();
         final NavigableMapbyte[], NavigableMapLong, byte[] familyMap
  =
  map.get(family);
         final NavigableMapLong, byte[] versionMap =
  familyMap.get(qualifier);
         for (final Map.EntryLong, byte[] entry : versionMap.entrySet())
         {
             System.err.println(entry.getKey());
             System.err.println(Bytes.toString(entry.getValue()));
         }
         //        assertTrue(versionMap.size() == 3);
 
         //        assertTrue(value1 == versionMap.get(ts1));
         //        assertTrue(value2 == versionMap.get(ts2));
         //        assertTrue(value3 == versionMap.get(ts3));

Re: I can't get many versions of the specified column,but only get the latest version of the specified column