Using HBase's export/import function...

2010-08-06 Thread Michael Segel

Ok,

Silly question...

Inside the /usr/lib/hbase/*.jar (base jar for HBase) There's an export/import 
tool.

If you supply the #versions, and the start time and end time, you can timebox 
your scan so your map/reduce job will let you do daily, weekly, etc type of 
incremental backups. 

So here's my questions:


1) Is anyone using this.
2) There isn't any documentation, I'm assuming that the start time and end 
times are timestamps (long values representing the number of miliseconds since 
the epoch which are what is being stored in hbase).
3) Is there an easy way to convert a date in to a time stamp? (not in ksh, and 
I'm struggling on finding a way to reverse the datetime object in python.

Thx

-Mike

  

Re: Using HBase's export/import function...

2010-08-06 Thread Stack
On Fri, Aug 6, 2010 at 11:13 AM, Michael Segel
michael_se...@hotmail.com wrote:
 2) There isn't any documentation, I'm assuming that the start time and end 
 times are timestamps (long values representing the number of miliseconds 
 since the epoch which are what is being stored in hbase).

Yes.

What kinda doc. do you need?  The javadoc on the class is minimal:
http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/mapreduce/Export.html


 3) Is there an easy way to convert a date in to a time stamp? (not in ksh, 
 and I'm struggling on finding a way to reverse the datetime object in python.


On the end of this page it shows you how to do date convertions inside
in the hbase shell: http://wiki.apache.org/hadoop/Hbase/Shell

St.Ack


 Thx

 -Mike




RE: Using HBase's export/import function...

2010-08-06 Thread Michael Segel

StAck...

LOL...

The idea is to automate the use of the export function to be run within a cron 
job. 
(And yes, there are some use cases where we want to actually back data up.. ;-)
I originally wanted to do this in ksh (yeah I'm that old. :-) but ended up 
looking at Python because I couldn't figure out how to create the time stamp in 
ksh.

As to documentation... just something which tells us what is meant by start 
time and end time. (Like that its in ms from the epoch instead of making us 
assume that.)
[And you know what they say about assumptions.]

As to converting the date / time to a timestamp...

In Python:
You build up a date object then you can do the following:
mytime = datetime.datetime(year,month,day,hour,min,sec) *where hour,min,sec are 
optional
mytimestamp = time.mktime(mytime.timetuple())

I'm in the process of testing this... I think it will work.

Thx

-Mike




 Date: Fri, 6 Aug 2010 11:28:57 -0700
 Subject: Re: Using HBase's export/import function...
 From: st...@duboce.net
 To: user@hbase.apache.org
 
 On Fri, Aug 6, 2010 at 11:13 AM, Michael Segel
 michael_se...@hotmail.com wrote:
  2) There isn't any documentation, I'm assuming that the start time and end 
  times are timestamps (long values representing the number of miliseconds 
  since the epoch which are what is being stored in hbase).
 
 Yes.
 
 What kinda doc. do you need?  The javadoc on the class is minimal:
 http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/mapreduce/Export.html
 
 
  3) Is there an easy way to convert a date in to a time stamp? (not in ksh, 
  and I'm struggling on finding a way to reverse the datetime object in 
  python.
 
 
 On the end of this page it shows you how to do date convertions inside
 in the hbase shell: http://wiki.apache.org/hadoop/Hbase/Shell
 
 St.Ack
 
 
  Thx
 
  -Mike