Thanks Esteban for replying. The Kerberos realm is shared between the two clusters.
I searched zookeeper config and couldn't find the rule, so where it is set? Having said that, I looked at parameters passed to getData call, and it doesn't look like security related. PS. I am using hbase 1.1.2. Here is the log: com.merkleinc.cr.hbase_maintenance.tableexport.CopyTableTest,testCopyTable Connected to the target VM, address: '127.0.0.1:50669', transport: 'socket' 0 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil - hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 247 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 1492 [main] INFO org.apache.hadoop.security.UserGroupInformation - Login successful for user [email protected] using keytab file \Users\jluo\tidsvc_exphbase.keytab 11462 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil - hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 164493 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil - hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 167720 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil - hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 172108 [main] WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows. 224735 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil - hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 225165 [main] INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process identifier=hconnection-0x4dd5d6bb connecting to ZooKeeper ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181 225190 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=JLUOWIN7.pclc0.merkle.local 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.7.0_51 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=C:\work\jdk1.7.0_51\jre 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=C:\Users\jluo\AppData\Local\Temp\ 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Windows 7 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=6.1 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=jluo 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=C:\Users\jluo 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=C:\work\project\c2\trunk\misc\hbase_maintenance 225197 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181 sessionTimeout=120000 watcher=hconnection-0x4dd5d6bb0x0, quorum=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181, baseZNode=/hbase-secure 225442 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181. Will not attempt to authenticate using SASL (unknown error) 225490 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, initiating session 225544 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, sessionid = 0x357067c5442410e, negotiated timeout = 120000 231758 [main] INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181 231758 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181 sessionTimeout=120000 watcher=TokenUtil-getAuthToken0x0, quorum=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181, baseZNode=/hbase-secure 231876 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181. Will not attempt to authenticate using SASL (unknown error) 253405 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, initiating session 257914 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, sessionid = 0x357067c54424116, negotiated timeout = 120000 374820 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 96993ms for sessionid 0x357067c5442410e, closing socket connection and attempting reconnect 374923 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x357067c54424116 closed 374924 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down 375053 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181. Will not attempt to authenticate using SASL (unknown error) 375095 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181, initiating session 375136 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181, sessionid = 0x357067c5442410e, negotiated timeout = 120000 376538 [main] INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation - Closing zookeeper sessionid=0x357067c5442410e 376581 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x357067c5442410e closed 376581 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down 376746 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil - hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 376749 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 376839 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil - hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 376841 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 376843 [main] INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process identifier=hconnection-0x613cf3c5 connecting to ZooKeeper ensemble=hqetghdpadm91.pclc0.merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181 376843 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=hqetghdpadm91.pclc0.merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181 sessionTimeout=120000 watcher=hconnection-0x613cf3c50x0, quorum=hqetghdpadm91.pclc0.merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181, baseZNode=/hbase 377008 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181. Will not attempt to authenticate using SASL (unknown error) 377057 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181, initiating session 377098 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181, sessionid = 0x2565c552d296982, negotiated timeout = 40000 377140 [main] INFO org.apache.hadoop.hbase.client.ZooKeeperRegistry - ClusterId read in ZooKeeper is null 377141 [main] INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper ensemble=hqetghdpadm91.pclc0.merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181 377141 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=hqetghdpadm91.pclc0.merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181 sessionTimeout=120000 watcher=TokenUtil-getAuthToken0x0, quorum=hqetghdpadm91.pclc0.merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181, baseZNode=/hbase 377189 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181. Will not attempt to authenticate using SASL (unknown error) 399494 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, initiating session 399541 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, sessionid = 0x1565c552cfd6b03, negotiated timeout = 40000 1036441 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 636835ms for sessionid 0x1565c552cfd6b03, closing socket connection and attempting reconnect 1036441 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 636900ms for sessionid 0x2565c552d296982, closing socket connection and attempting reconnect 1236709 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181. Will not attempt to authenticate using SASL (unknown error) 1236761 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, initiating session 1236762 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1565c552cfd6b03 closed 1236762 [main] INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation - Closing zookeeper sessionid=0x2565c552d296982 1236762 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down 1236908 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x2565c552d296982 closed 1236909 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down java.io.IOException: Failed to get cluster ID at org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:365) at org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:327) at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:451) at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableReducerJob(TableMapReduceUtil.java:658) at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableReducerJob(TableMapReduceUtil.java:591) at org.apache.hadoop.hbase.mapreduce.CopyTable.createSubmittableJob(CopyTable.java:168) at org.apache.hadoop.hbase.mapreduce.CopyTable.run(CopyTable.java:348) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.hbase.mapreduce.CopyTable.main(CopyTable.java:341) at com.merkleinc.cr.hbase_maintenance.tableexport.CopyTableTest.export(CopyTableTest.java:44) at com.merkleinc.cr.hbase_maintenance.tableexport.CopyTableTest.testCopyTable(CopyTableTest.java:25) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at com.intellij.junit4.JUnit4TestRunnerUtil$IgnoreIgnoredTestJUnit4ClassRunner.runChild(JUnit4TestRunnerUtil.java:349) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74) Disconnected from the target VM, address: '127.0.0.1:50669', transport: 'socket' Process finished with exit code -1 -----Original Message----- From: Esteban Gutierrez [mailto:[email protected]] Sent: Thursday, September 08, 2016 12:37 PM To: [email protected] Subject: Re: CopyTable fails on copying between two secured clusters Frank, Do you have a cross realm setup? if yes, do you have ZK to use the proper rules via the -Dzookeeper.security.auth_to_local flag? If you could share additional logs that would be helpful for us. Thanks! esteban. -- Cloudera, Inc. On Thu, Sep 8, 2016 at 10:32 AM, Frank Luo <[email protected]> wrote: > I couldn’t manage to get the CopyTable to work between two secured > clusters and hope someone can shed some light. > > I have a table created on both clusters, and I am running the > CopyTable command from the source cluster. > > The parameters are the following: > "--peer.adr=zookeeper1, zookeeper2:2181:/hbase", > "--new.name=TargetTable", "SourceTable" > > I have ensured that the user has all permissions on both tables. When > I run it, I got: > > ===================== > java.io.IOException: Failed to get cluster ID > > at org.apache.hadoop.hbase.security.token.TokenUtil. > getAuthToken(TokenUtil.java:365) > at org.apache.hadoop.hbase.security.token.TokenUtil. > addTokenForJob(TokenUtil.java:327) > ===================== > > Debugging into it, I found that it is trying to get the clusterId from > my target cluster using address of zookeeper1/2, and the code below > returns > null: > > > ===================== > > ZKUtil.getData(watcher, watcher.clusterIdZNode); > ===================== > > Any hints please? > > > > Access the Q2 2016 Digital Marketing Report for a fresh set of trends > and benchmarks in digital > marketing<http://www2.merkleinc.com/l/47252/2016-07- > 26/47gt7c> > > Download our latest report titled “The Case for Change: Exploring the > Myths of Customer-Centric Transformation” > <http://www2.merkleinc.com/l/ 47252/2016-08-04/4b9p7c> > > This email and any attachments transmitted with it are intended for > use by the intended recipient(s) only. If you have received this email > in error, please notify the sender immediately and then delete it. If > you are not the intended recipient, you must not keep, use, disclose, > copy or distribute this email without the author’s prior permission. > We take precautions to minimize the risk of transmitting software > viruses, but we advise you to perform your own virus checks on any > attachment to this message. We cannot accept liability for any loss or > damage caused by software viruses. The information contained in this > communication may be confidential and may be subject to the attorney-client > privilege. > Access the Q2 2016 Digital Marketing Report for a fresh set of trends and benchmarks in digital marketing<http://www2.merkleinc.com/l/47252/2016-07-26/47gt7c> Download our latest report titled “The Case for Change: Exploring the Myths of Customer-Centric Transformation” <http://www2.merkleinc.com/l/47252/2016-08-04/4b9p7c> This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
