Thanks, Jean. In the end we've gone for testing things out on a couple of relatively unimportant/easy to regenerate tables first. And the whole experience confirms that our planned upgrade to 0.90 will be worth the pain.
Regards, James On Fri, Jun 10, 2011 at 6:33 PM, Jean-Daniel Cryans <[email protected]>wrote: > The ruby scripts have a tendency to rot since they aren't tested as > part of our unit test suite... we should probably just delete > copy_table.rb since IIRC it was a hacked up script that Stack needed > for Powerset in 2008. > > There's no practical way of copying a table, in the most recent > version there's the CopyTable mapreduce job that you could use but in > 0.20 I think you could distcp the table's folder to another cluster > and try it there or have another hbase instance using the same HDFS > but root to a different folder. In both cases you'll have to use the > add_table.rb script. > > J-D > > On Fri, Jun 10, 2011 at 10:40 AM, James Hammerton > <[email protected]> wrote: > > Hi, > > > > I did some digging around in the HBase source and found that HLogEdit > last > > existed in version 0.18 (we're on 0.20.6) and > > HConstants.COL_REGIONINFO_ARRAY last existed in version 0.2. > > > > After this digging around and making some educated guesses I made a new > > version of the script which I've attached to this message. > > > > I've tried running this script on a small 525M (on disc) table in a test > > instance of HBase and it's been running for over 12 mins, and is > repeatedly > > outputting messages like the following: > > > > 11/06/10 11:27:05 DEBUG hfile.LruBlockCache: Cache Stats: Sizes: > > Total=1.6478577MB (1727904), Free=197.52715MB (207122224), Max=199.175MB > > (208850128), Counts: Blocks=3, Access=3, Hit=0, Miss=3, Evictions=0, > > Evicted=0, Ratios: Hit Ratio=0.0%, Miss Ratio=100.0%, Evicted/Run=NaN > > 11/06/10 11:28:05 DEBUG hfile.LruBlockCache: Cache Stats: Sizes: > > Total=1.6478577MB (1727904), Free=197.52715MB (207122224), Max=199.175MB > > (208850128), Counts: Blocks=3, Access=3, Hit=0, Miss=3, Evictions=0, > > Evicted=0, Ratios: Hit Ratio=0.0%, Miss Ratio=100.0%, Evicted/Run=NaN > > > > The du command tells me that the copied table's directory so far takes up > > just 4k on disk. Is my hacked script actually doing anything useful? > > > > James > > > > On Fri, Jun 10, 2011 at 11:19 AM, James Hammerton > > <[email protected]> wrote: > >> > >> [Re-sending as I'm not sure this got through] > >> > >> Hi, > >> > >> Before trying to merge regions on a table in our live database we > decided > >> to copy the table and merge the regions on the copy first to test the > >> merging code works before risking our live data. > >> > >> However when we try to run the copy_table.rb script it fails due to the > >> following error (this is actually from a test HBase instance but the > error > >> from the live instance is the same, modulo different paths): > >> > >> > >> > file:/home/james/hbase-0.20.6/lib/jruby-complete-1.2.0.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.8/builtin/javasupport/core_ext/object.rb:33:in > >> `get_proxy_or_package_under_package': cannot load Java class > >> org.apache.hadoop.hbase.regionserver.HLogEdit (NameError) > >> from > >> > file:/home/james/hbase-0.20.6/lib/jruby-complete-1.2.0.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.8/builtin/javasupport/java.rb:51:in > >> `method_missing' > >> from /home/james/hbase/bin/copy_table.rb:40 > >> > >> Looking through the copy_table.rb file I could so no reference to > HLogEdit > >> other than the import, so I tried removing it. Now running the script > >> produces the error below: > >> > >> > >> > file:/home/james/hbase-0.20.6/lib/jruby-complete-1.2.0.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.8/builtin/java/ast.rb:80:in > >> `const_missing': uninitialized constant > >> Java::OrgApacheHadoopHbase::HConstants::COL_REGIONINFO_ARRAY (NameError) > >> from /home/james/hbase/bin/copy_table.rb:124 > >> from > >> > file:/home/james/hbase-0.20.6/lib/jruby-complete-1.2.0.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.8/builtin/java/collections.rb:47:in > >> `call' > >> from > >> > file:/home/james/hbase-0.20.6/lib/jruby-complete-1.2.0.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.8/builtin/java/collections.rb:47:in > >> `each' > >> from /home/james/hbase/bin/copy_table.rb:118 > >> > >> I looked at the latest version of the copy_table.rb script and it also > >> references both HLogEdit and HConstants::COL_REGIONINFO_ > >> ARRAY... > >> Any ideas how to fix this? > >> > >> Regards, > >> > >> James > >> > >> -- > >> James Hammerton | Senior Data Mining Engineer > >> www.mendeley.com/profiles/james-hammerton > >> > >> Mendeley Limited | London, UK | www.mendeley.com > >> Registered in England and Wales | Company Number 6419015 > >> > >> > >> > > > > > > > > -- > > James Hammerton | Senior Data Mining Engineer > > www.mendeley.com/profiles/james-hammerton > > > > Mendeley Limited | London, UK | www.mendeley.com > > Registered in England and Wales | Company Number 6419015 > > > > > > > > > -- James Hammerton | Senior Data Mining Engineer www.mendeley.com/profiles/james-hammerton Mendeley Limited | London, UK | www.mendeley.com Registered in England and Wales | Company Number 6419015
