I have added some logs on the TableOutputFormat class and it seems to receive the right parameters. However, it's still not able to connect to the other cluster.
13/11/14 20:38:42 mapreduce.TableOutputFormat: address=hbasetest1.distparser.com:2181:/hbase 13/11/14 20:38:42 mapreduce.TableOutputFormat: zkClientPort=0 13/11/14 20:38:42 mapreduce.TableOutputFormat: serverClass=null 13/11/14 20:38:42 mapreduce.TableOutputFormat: serverImpl=null Basically, it's trying to connect from a 0.94.12 client to a 0.96.0 cluster. 0.96.0 talk PB. But I guess 0.94.12 not... Can this be why it's not working? Has anyone been able to copytable from 0.94 to 0.96? 2013/11/14 Jean-Marc Spaggiari <jean-m...@spaggiari.org> > Thanks for looking a it St.Ack. > > Here are my regions on the source table: > Table RegionsNameRegion ServerStart Key End Key Requests > dns,,1379202070789.bb65f685cdefc4f2491d246f376fc1f0. node3:60030 > gestion-v.com 0 dns,gestion-v.com > ,1379202070789.d02ce8e3fa1a200c7f034b349acf8cc8. buldo:60030 gestion-v.com > rfkcenter.org 0 dns,rfkcenter.org > ,1379202071281.c6569a72cc3c2750d14976ab85f02315. node5:60030 rfkcenter.org > theafronews.ca 0 dns,theafronews.ca > ,1379202071281.d5b74aaa2853b00b0ad0f20f60c74398. node7:60030 > theafronews.ca > 0 > Sound like "normal". > > I tried to scan the source table from the shell and it's working fine. > > > Now, since the stack show TableOutputFormat does it mean it's on the > destination cluster? If I put a wrong server name or port, I get a > connection exception, which mean server name is right. > > So on the target side, I can also scan the table: > > hbase(main):001:0> scan 'dns' > 2013-11-14 07:58:20,597 WARN [main] util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > ROW > COLUMN+CELL > > 0 row(s) in 0.0620 seconds > > hbase(main):002:0> > > > Also put is working fine: > hbase(main):003:0> put 'dns', 'test', '@:@', 'TestValue' > 0 row(s) in 0.2090 seconds > > hbase(main):004:0> scan 'dns' > ROW > COLUMN+CELL > > test column=@:@, > timestamp=1384433996320, > value=TestValue > > 1 row(s) in 0.0570 seconds > > hbase(main):005:0> > > > I have 3 ZK servers on the source and one on the target. > > ZK shell is working fine on target cluster. However, there is this: > [zk: localhost:2181(CONNECTED) 6] ls /hbase/master > [] > > It's the same on 0.94. The master ZNode is empty. Is that normal? Can this > be the issue? > > JM > > > 2013/11/14 Stack <st...@duboce.net> > >> On Thu, Nov 14, 2013 at 9:23 PM, Jean-Marc Spaggiari < >> jean-m...@spaggiari.org> wrote: >> >> > Hum. I let is run over night and got that: >> > 13/11/13 22:24:17 INFO zookeeper.ClientCnxn: Session establishment >> complete >> > on server hbasetest1/192.168.23.51:2181, sessionid = 0x1423ef50f7d0241, >> > negotiated timeout = 40000 >> > 13/11/13 23:24:41 ERROR mapreduce.TableOutputFormat: >> > org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to >> find >> > region for dns,,99999999999999 after 14 tries. >> > >> >> dns,,99999999999999 is an artificial key made in the client doing get >> closest or before; it is how we find start key for a region (the first >> region in the table in this case). It should not be coming up and out as >> an exception. Was there no region whose key was '',,' in the dns table? >> >> St.Ack >> >> >> > 13/11/13 23:24:41 INFO mapred.JobClient: Cleaning up the staging area >> > >> > >> file:/tmp/hadoop-hbase/mapred/staging/hbase-368698754/.staging/job_local_0001 >> > Exception in thread "main" java.lang.RuntimeException: >> > org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to >> find >> > region for dns,,99999999999999 after 14 tries. >> > at >> > >> > >> org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:206) >> > at >> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) >> > at >> > >> > >> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) >> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:884) >> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at javax.security.auth.Subject.doAs(Subject.java:415) >> > at >> > >> > >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) >> > at >> > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850) >> > at org.apache.hadoop.mapreduce.Job.submit(Job.java:500) >> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530) >> > at >> org.apache.hadoop.hbase.mapreduce.CopyTable.run(CopyTable.java:274) >> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >> > at >> org.apache.hadoop.hbase.mapreduce.CopyTable.main(CopyTable.java:265) >> > Caused by: org.apache.hadoop.hbase.client.NoServerForRegionException: >> > Unable to find region for dns,,99999999999999 after 14 tries. >> > at >> > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1095) >> > at >> > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1000) >> > at >> > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1102) >> > at >> > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1004) >> > at >> > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961) >> > at >> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:227) >> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:170) >> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:129) >> > at >> > >> > >> org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:201) >> > ... 14 more >> > >> > >> > Looking at the destination server I can see that my table and regions >> are >> > online nut not with the same name: >> > dns,,1384397901249.cfcf94a7ec8186baf462b2627d138481 >> > >> > I don't have anywhere (nor source, neither destination) a region called >> > dns,,99999999999999. >> > >> > Strange. >> > >> > JM >> > >> > >> > 2013/11/13 Jean-Marc Spaggiari <jean-m...@spaggiari.org> >> > >> > > Hi, >> > > >> > > Is there anything I missed for my copyTable command? >> > > >> > > hbase@node3:~/hbase-0.94.12$ bin/hbase >> > > org.apache.hadoop.hbase.mapreduce.CopyTable --endtime=1384395265000 >> > > --peer.adr=hbasetest1:2181:/hbase dns >> > > 13/11/13 21:28:37 INFO zookeeper.RecoverableZooKeeper: The identifier >> of >> > > this process is 31738@node3 >> > > >> > > I killed it after 30 minutes. There was nothing done. I have my 2 >> > clusters >> > > running. DistCP between the 2 is working fine, and my 2 HBase are also >> > > working fine. If I stop HBase on the target, distcp, upgrade and >> start, I >> > > have all my data. I dropped all the tables and tried CopyTable and I >> see >> > > nothing going. >> > > >> > > I tried with the short name (hbasetest1) or with FQDN, same result. I >> > > tried to run rowcount on the target cluster on an existing table, it's >> > > working fine. >> > > >> > > Any idea what I might be missing? I see nothing on the masters logs >> (nor >> > > in the source, neither in the target). >> > > >> > > Thanks, >> > > >> > > JM >> > > >> > >> > >
