Thanks for looking a it St.Ack. Here are my regions on the source table: Table RegionsNameRegion ServerStart KeyEnd Key Requests dns,,1379202070789.bb65f685cdefc4f2491d246f376fc1f0. node3:60030 gestion-v.com 0 dns,gestion-v.com ,1379202070789.d02ce8e3fa1a200c7f034b349acf8cc8. buldo:60030 gestion-v.com rfkcenter.org 0 dns,rfkcenter.org ,1379202071281.c6569a72cc3c2750d14976ab85f02315. node5:60030 rfkcenter.org theafronews.ca 0 dns,theafronews.ca ,1379202071281.d5b74aaa2853b00b0ad0f20f60c74398. node7:60030 theafronews.ca 0 Sound like "normal".
I tried to scan the source table from the shell and it's working fine. Now, since the stack show TableOutputFormat does it mean it's on the destination cluster? If I put a wrong server name or port, I get a connection exception, which mean server name is right. So on the target side, I can also scan the table: hbase(main):001:0> scan 'dns' 2013-11-14 07:58:20,597 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ROW COLUMN+CELL 0 row(s) in 0.0620 seconds hbase(main):002:0> Also put is working fine: hbase(main):003:0> put 'dns', 'test', '@:@', 'TestValue' 0 row(s) in 0.2090 seconds hbase(main):004:0> scan 'dns' ROW COLUMN+CELL test column=@:@, timestamp=1384433996320, value=TestValue 1 row(s) in 0.0570 seconds hbase(main):005:0> I have 3 ZK servers on the source and one on the target. ZK shell is working fine on target cluster. However, there is this: [zk: localhost:2181(CONNECTED) 6] ls /hbase/master [] It's the same on 0.94. The master ZNode is empty. Is that normal? Can this be the issue? JM 2013/11/14 Stack <st...@duboce.net> > On Thu, Nov 14, 2013 at 9:23 PM, Jean-Marc Spaggiari < > jean-m...@spaggiari.org> wrote: > > > Hum. I let is run over night and got that: > > 13/11/13 22:24:17 INFO zookeeper.ClientCnxn: Session establishment > complete > > on server hbasetest1/192.168.23.51:2181, sessionid = 0x1423ef50f7d0241, > > negotiated timeout = 40000 > > 13/11/13 23:24:41 ERROR mapreduce.TableOutputFormat: > > org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find > > region for dns,,99999999999999 after 14 tries. > > > > dns,,99999999999999 is an artificial key made in the client doing get > closest or before; it is how we find start key for a region (the first > region in the table in this case). It should not be coming up and out as > an exception. Was there no region whose key was '',,' in the dns table? > > St.Ack > > > > 13/11/13 23:24:41 INFO mapred.JobClient: Cleaning up the staging area > > > > > file:/tmp/hadoop-hbase/mapred/staging/hbase-368698754/.staging/job_local_0001 > > Exception in thread "main" java.lang.RuntimeException: > > org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find > > region for dns,,99999999999999 after 14 tries. > > at > > > > > org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:206) > > at > > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > > at > > > > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:884) > > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > > > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) > > at > > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850) > > at org.apache.hadoop.mapreduce.Job.submit(Job.java:500) > > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530) > > at > org.apache.hadoop.hbase.mapreduce.CopyTable.run(CopyTable.java:274) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > > at > org.apache.hadoop.hbase.mapreduce.CopyTable.main(CopyTable.java:265) > > Caused by: org.apache.hadoop.hbase.client.NoServerForRegionException: > > Unable to find region for dns,,99999999999999 after 14 tries. > > at > > > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1095) > > at > > > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1000) > > at > > > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1102) > > at > > > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1004) > > at > > > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961) > > at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:227) > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:170) > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:129) > > at > > > > > org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:201) > > ... 14 more > > > > > > Looking at the destination server I can see that my table and regions are > > online nut not with the same name: > > dns,,1384397901249.cfcf94a7ec8186baf462b2627d138481 > > > > I don't have anywhere (nor source, neither destination) a region called > > dns,,99999999999999. > > > > Strange. > > > > JM > > > > > > 2013/11/13 Jean-Marc Spaggiari <jean-m...@spaggiari.org> > > > > > Hi, > > > > > > Is there anything I missed for my copyTable command? > > > > > > hbase@node3:~/hbase-0.94.12$ bin/hbase > > > org.apache.hadoop.hbase.mapreduce.CopyTable --endtime=1384395265000 > > > --peer.adr=hbasetest1:2181:/hbase dns > > > 13/11/13 21:28:37 INFO zookeeper.RecoverableZooKeeper: The identifier > of > > > this process is 31738@node3 > > > > > > I killed it after 30 minutes. There was nothing done. I have my 2 > > clusters > > > running. DistCP between the 2 is working fine, and my 2 HBase are also > > > working fine. If I stop HBase on the target, distcp, upgrade and > start, I > > > have all my data. I dropped all the tables and tried CopyTable and I > see > > > nothing going. > > > > > > I tried with the short name (hbasetest1) or with FQDN, same result. I > > > tried to run rowcount on the target cluster on an existing table, it's > > > working fine. > > > > > > Any idea what I might be missing? I see nothing on the masters logs > (nor > > > in the source, neither in the target). > > > > > > Thanks, > > > > > > JM > > > > > >
