Hi James, I think I've seen this few times after stopping and starting HBase and the hostname/ip of my laptop change when hopping different DHCP servers. Ideally "hbase hbck -fixMeta" have you tried that?
cheers, esteban. -- Cloudera, Inc. On Tue, Mar 31, 2015 at 12:09 AM, James Teng <[email protected]> wrote: > Hi all, > i have encountered the problem connecting to hbase via phoenix client, > below is the debug log i got from the client side.and the most notable > error should be > > > > > > > > retrying after sleep of 10013 because: No server address listed in > hbase:meta for region > SYSTEM.CATALOG,,1427426385032.7e46009a444329769d8e851cb5900006. containing > row > but i totally have no idea how this happens on my laptop.any comment on > this is appreciated.thanks > uknow. > > > > > > > > > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > INFO [main] (Configuration.java:840) - dfs.df.interval is deprecated. > Instead, use fs.df.interval > INFO [main] (Configuration.java:840) - hadoop.native.lib is deprecated. > Instead, use io.native.lib.available > INFO [main] (Configuration.java:840) - fs.default.name is deprecated. > Instead, use fs.defaultFS > INFO [main] (Configuration.java:840) - topology.script.number.args is > deprecated. Instead, use net.topology.script.number.args > INFO [main] (Configuration.java:840) - dfs.umaskmode is deprecated. > Instead, use fs.permissions.umask-mode > INFO [main] (Configuration.java:840) - topology.node.switch.mapping.impl > is deprecated. Instead, use net.topology.node.switch.mapping.impl > DEBUG [main] (MutableMetricsFactory.java:42) - field > org.apache.hadoop.metrics2.lib.MutableRate > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess > with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, > value=[Rate of successful kerberos logins and latency (milliseconds)], > valueName=Time, always=false, type=DEFAULT, sampleName=Ops) > DEBUG [main] (MutableMetricsFactory.java:42) - field > org.apache.hadoop.metrics2.lib.MutableRate > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure > with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, > value=[Rate of failed kerberos logins and latency (milliseconds)], > valueName=Time, always=false, type=DEFAULT, sampleName=Ops) > DEBUG [main] (MetricsSystemImpl.java:220) - UgiMetrics, User and group > related metrics > DEBUG [main] (Groups.java:180) - Creating new Groups object > DEBUG [main] (NativeCodeLoader.java:46) - Trying to load the custom-built > native-hadoop library... > DEBUG [main] (NativeCodeLoader.java:55) - Failed to load native-hadoop > with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path > DEBUG [main] (NativeCodeLoader.java:56) - > java.library.path=/Users/tenglinxiao/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. > WARN [main] (NativeCodeLoader.java:62) - Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > DEBUG [main] (JniBasedUnixGroupsMappingWithFallback.java:40) - Falling > back to shell based > DEBUG [main] (JniBasedUnixGroupsMappingWithFallback.java:44) - Group > mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping > DEBUG [main] (Groups.java:66) - Group mapping > impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; > cacheTimeout=300000 > DEBUG [main] (UserGroupInformation.java:177) - hadoop login > DEBUG [main] (UserGroupInformation.java:126) - hadoop login commit > DEBUG [main] (UserGroupInformation.java:156) - using local > user:UnixPrincipal: tenglinxiao > DEBUG [main] (UserGroupInformation.java:703) - UGI loginUser:tenglinxiao > (auth:SIMPLE) > DEBUG [main] (Shell.java:322) - setsid is not available on this machine. > So not using it. > DEBUG [main] (Shell.java:326) - setsid exited with exit code 0 > INFO [main] (RecoverableZooKeeper.java:121) - Process > identifier=hconnection-0x57a83d23 connecting to ZooKeeper > ensemble=localhost:2181 > DEBUG [main-EventThread] (ZooKeeperWatcher.java:312) - > hconnection-0x57a83d23, quorum=localhost:2181, baseZNode=/hbase Received > ZooKeeper Event, type=None, state=SyncConnected, path=null > DEBUG [main-EventThread] (ZooKeeperWatcher.java:389) - > hconnection-0x57a83d23-0x14c6e88b4620010 connected > DEBUG [main] (RpcClient.java:1283) - > Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@277cd3d6, > compressor=null, tcpKeepAlive=true, tcpNoDelay=true, maxIdleTime=10000, > maxRetries=0, fallbackAllowed=false, ping interval=60000ms, bind > address=null > INFO [main] (Metrics.java:44) - Initializing metrics system: phoenix > DEBUG [main] (MetricsSystemImpl.java:581) - from system property: null > DEBUG [main] (MetricsSystemImpl.java:582) - from environment variable: null > DEBUG [main] (ConfigurationUtils.java:447) - ConfigurationUtils.locate(): > base is null, name is hadoop-metrics2-phoenix.properties > DEBUG [main] (ConfigurationUtils.java:447) - ConfigurationUtils.locate(): > base is null, name is hadoop-metrics2.properties > WARN [main] (MetricsConfig.java:124) - Cannot locate configuration: tried > hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties > DEBUG [main] (MetricsConfig.java:179) - poking parent > 'PropertiesConfiguration' for key: period > DEBUG [main] (MutableMetricsFactory.java:42) - field > org.apache.hadoop.metrics2.lib.MutableStat > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.snapshotStat with > annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, > value=[Snapshot, Snapshot stats], valueName=Time, always=false, > type=DEFAULT, sampleName=Ops) > DEBUG [main] (MutableMetricsFactory.java:42) - field > org.apache.hadoop.metrics2.lib.MutableStat > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.publishStat with > annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, > value=[Publish, Publishing stats], valueName=Time, always=false, > type=DEFAULT, sampleName=Ops) > DEBUG [main] (MutableMetricsFactory.java:42) - field > org.apache.hadoop.metrics2.lib.MutableCounterLong > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.droppedPubAll with > annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, > value=[Dropped updates by all sinks], valueName=Time, always=false, > type=DEFAULT, sampleName=Ops) > DEBUG [main] (MetricsConfig.java:179) - poking parent > 'PropertiesConfiguration' for key: source.source.start_mbeans > DEBUG [main] (MetricsConfig.java:179) - poking parent 'MetricsConfig' for > key: source.start_mbeans > DEBUG [main] (MetricsConfig.java:179) - poking parent > 'PropertiesConfiguration' for key: *.source.start_mbeans > DEBUG [main] (MetricsSourceAdapter.java:238) - Updating attr cache... > DEBUG [main] (MetricsSourceAdapter.java:252) - Done. # tags & metrics=10 > DEBUG [main] (MetricsSourceAdapter.java:232) - Updating info cache... > DEBUG [main] (MBeanInfoBuilder.java:109) - > [javax.management.MBeanAttributeInfo[description=Metrics context, > name=tag.Context, type=java.lang.String, read-only, descriptor={}], > javax.management.MBeanAttributeInfo[description=Number of active metrics > sources, name=NumActiveSources, type=java.lang.Integer, read-only, > descriptor={}], javax.management.MBeanAttributeInfo[description=Number of > all registered metrics sources, name=NumAllSources, type=java.lang.Integer, > read-only, descriptor={}], > javax.management.MBeanAttributeInfo[description=Number of active metrics > sinks, name=NumActiveSinks, type=java.lang.Integer, read-only, > descriptor={}], javax.management.MBeanAttributeInfo[description=Number of > all registered metrics sinks, name=NumAllSinks, type=java.lang.Integer, > read-only, descriptor={}], > javax.management.MBeanAttributeInfo[description=Number of ops for snapshot > stats, name=SnapshotNumOps, type=java.lang.Long, read-only, descriptor={}], > javax.management.MBeanAttributeInfo[description=Average time for snapshot > stats, name=SnapshotAvgTime, type=java.lang.Double, read-only, > descriptor={}], javax.management.MBeanAttributeInfo[description=Number of > ops for publishing stats, name=PublishNumOps, type=java.lang.Long, > read-only, descriptor={}], > javax.management.MBeanAttributeInfo[description=Average time for publishing > stats, name=PublishAvgTime, type=java.lang.Double, read-only, > descriptor={}], javax.management.MBeanAttributeInfo[description=Dropped > updates by all sinks, name=DroppedPubAll, type=java.lang.Long, read-only, > descriptor={}]] > DEBUG [main] (MetricsSourceAdapter.java:234) - Done > DEBUG [main] (MBeans.java:58) - Registered > Hadoop:service=phoenix,name=MetricsSystem,sub=Stats > DEBUG [main] (MetricsSourceAdapter.java:221) - MBean for source > MetricsSystem,sub=Stats registered. > INFO [main] (MetricsSystemImpl.java:344) - Scheduled snapshot period at > 10 second(s). > INFO [main] (MetricsSystemImpl.java:183) - phoenix metrics system started > DEBUG [main] (MetricsConfig.java:179) - poking parent > 'PropertiesConfiguration' for key: source.source.start_mbeans > DEBUG [main] (MetricsConfig.java:179) - poking parent 'MetricsConfig' for > key: source.start_mbeans > DEBUG [main] (MetricsConfig.java:179) - poking parent > 'PropertiesConfiguration' for key: *.source.start_mbeans > DEBUG [main] (MetricsSourceAdapter.java:238) - Updating attr cache... > DEBUG [main] (MetricsSourceAdapter.java:252) - Done. # tags & metrics=6 > DEBUG [main] (MetricsSourceAdapter.java:232) - Updating info cache... > DEBUG [main] (MBeanInfoBuilder.java:109) - > [javax.management.MBeanAttributeInfo[description=Metrics context, > name=tag.Context, type=java.lang.String, read-only, descriptor={}], > javax.management.MBeanAttributeInfo[description=Local hostname, > name=tag.Hostname, type=java.lang.String, read-only, descriptor={}], > javax.management.MBeanAttributeInfo[description=Number of ops for rate of > successful kerberos logins and latency (milliseconds), > name=LoginSuccessNumOps, type=java.lang.Long, read-only, descriptor={}], > javax.management.MBeanAttributeInfo[description=Average time for rate of > successful kerberos logins and latency (milliseconds), > name=LoginSuccessAvgTime, type=java.lang.Double, read-only, descriptor={}], > javax.management.MBeanAttributeInfo[description=Number of ops for rate of > failed kerberos logins and latency (milliseconds), name=LoginFailureNumOps, > type=java.lang.Long, read-only, descriptor={}], > javax.management.MBeanAttributeInfo[description=Average time for rate of > failed kerberos logins and latency (milliseconds), > name=LoginFailureAvgTime, type=java.lang.Double, read-only, descriptor={}]] > DEBUG [main] (MetricsSourceAdapter.java:234) - Done > DEBUG [main] (MBeans.java:58) - Registered > Hadoop:service=phoenix,name=UgiMetrics > DEBUG [main] (MetricsSourceAdapter.java:221) - MBean for source UgiMetrics > registered. > DEBUG [main] (MetricsSystemImpl.java:245) - Registered source UgiMetrics > DEBUG [main] (MBeans.java:58) - Registered > Hadoop:service=phoenix,name=MetricsSystem,sub=Control > DEBUG [main] (MetricsSystemImpl.java:220) - tracing, Phoenix call tracing > DEBUG [main] (MetricsConfig.java:179) - poking parent > 'PropertiesConfiguration' for key: source.source.start_mbeans > DEBUG [main] (MetricsConfig.java:179) - poking parent 'MetricsConfig' for > key: source.start_mbeans > DEBUG [main] (MetricsConfig.java:179) - poking parent > 'PropertiesConfiguration' for key: *.source.start_mbeans > DEBUG [main] (MetricsSourceAdapter.java:238) - Updating attr cache... > DEBUG [main] (MetricsSourceAdapter.java:252) - Done. # tags & metrics=2 > DEBUG [main] (MetricsSourceAdapter.java:232) - Updating info cache... > DEBUG [main] (MBeanInfoBuilder.java:109) - > [javax.management.MBeanAttributeInfo[description=num spans, name=tag.stat, > type=java.lang.String, read-only, descriptor={}], > javax.management.MBeanAttributeInfo[description=Local hostname, > name=tag.Hostname, type=java.lang.String, read-only, descriptor={}]] > DEBUG [main] (MetricsSourceAdapter.java:234) - Done > DEBUG [main] (MBeans.java:58) - Registered > Hadoop:service=phoenix,name=tracing > DEBUG [main] (MetricsSourceAdapter.java:221) - MBean for source tracing > registered. > DEBUG [main] (MetricsSystemImpl.java:245) - Registered source tracing > INFO [main] (RecoverableZooKeeper.java:121) - Process > identifier=hconnection-0x73a6068c connecting to ZooKeeper > ensemble=localhost:2181 > DEBUG [main-EventThread] (ZooKeeperWatcher.java:312) - > hconnection-0x73a6068c, quorum=localhost:2181, baseZNode=/hbase Received > ZooKeeper Event, type=None, state=SyncConnected, path=null > DEBUG [main-EventThread] (ZooKeeperWatcher.java:389) - > hconnection-0x73a6068c-0x14c6e88b4620011 connected > DEBUG [main] (RpcClient.java:1283) - > Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4be2d842, > compressor=null, tcpKeepAlive=true, tcpNoDelay=true, maxIdleTime=10000, > maxRetries=0, fallbackAllowed=false, ping interval=60000ms, bind > address=null > DEBUG [main] (RpcClient.java:428) - Use SIMPLE authentication for service > MasterService, sasl=false > DEBUG [main] (RpcClient.java:862) - Connecting to /10.128.122.77:16020 > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16020 from > tenglinxiao] (RpcClient.java:722) - IPC Client (920483469) connection to / > 10.128.122.77:16020 from tenglinxiao: starting, connections 1 > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16020 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16020 from tenglinxiao: got response header call_id: 0, > totalSize: 6 bytes > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16020 from tenglinxiao: wrote request header call_id: 0 > method_name: "IsMasterRunning" request_param: true > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16020 from tenglinxiao: wrote request header call_id: 1 > method_name: "GetTableDescriptors" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16020 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16020 from tenglinxiao: got response header call_id: 1, > totalSize: 821 bytes > DEBUG [main] (RpcClient.java:428) - Use SIMPLE authentication for service > ClientService, sasl=false > DEBUG [main] (RpcClient.java:862) - Connecting to /10.128.122.77:16201 > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:722) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: starting, connections 1 > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 0 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 0, > totalSize: 155 bytes > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 1 > method_name: "Scan" request_param: true priority: 100 > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 1 > cell_block_meta { length: 1084 }, totalSize: 1105 bytes > DEBUG [main] (ClientSmallScanner.java:136) - Finished with small scan at > {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => > ''} > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 2 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 2, > totalSize: 155 bytes > DEBUG [main] (HConnectionManager.java:1339) - locateRegionInMeta > parentTable=hbase:meta, metaLocation={region=hbase:meta,,1.1588230740, > hostname=10.128.122.77,16201,1427783593678, seqNum=0}, attempt=0 of 35 > failed; retrying after sleep of 100 because: No server address listed in > hbase:meta for region > SYSTEM.CATALOG,,1427426385032.7e46009a444329769d8e851cb5900006. containing > row > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 3 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 3, > totalSize: 155 bytes > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 4 > method_name: "Scan" request_param: true priority: 100 > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 4 > cell_block_meta { length: 1084 }, totalSize: 1105 bytes > DEBUG [main] (ClientSmallScanner.java:136) - Finished with small scan at > {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => > ''} > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 5 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 5, > totalSize: 155 bytes > DEBUG [main] (HConnectionManager.java:1339) - locateRegionInMeta > parentTable=hbase:meta, metaLocation={region=hbase:meta,,1.1588230740, > hostname=10.128.122.77,16201,1427783593678, seqNum=0}, attempt=1 of 35 > failed; retrying after sleep of 200 because: No server address listed in > hbase:meta for region > SYSTEM.CATALOG,,1427426385032.7e46009a444329769d8e851cb5900006. containing > row > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 6 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 6, > totalSize: 155 bytes > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 7 > method_name: "Scan" request_param: true priority: 100 > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 7 > cell_block_meta { length: 1084 }, totalSize: 1105 bytes > DEBUG [main] (ClientSmallScanner.java:136) - Finished with small scan at > {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => > ''} > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 8 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 8, > totalSize: 155 bytes > DEBUG [main] (HConnectionManager.java:1339) - locateRegionInMeta > parentTable=hbase:meta, metaLocation={region=hbase:meta,,1.1588230740, > hostname=10.128.122.77,16201,1427783593678, seqNum=0}, attempt=2 of 35 > failed; retrying after sleep of 301 because: No server address listed in > hbase:meta for region > SYSTEM.CATALOG,,1427426385032.7e46009a444329769d8e851cb5900006. containing > row > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 9 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 9, > totalSize: 155 bytes > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 10 > method_name: "Scan" request_param: true priority: 100 > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 10 > cell_block_meta { length: 1084 }, totalSize: 1105 bytes > DEBUG [main] (ClientSmallScanner.java:136) - Finished with small scan at > {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => > ''} > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 11 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 11, > totalSize: 155 bytes > DEBUG [main] (HConnectionManager.java:1339) - locateRegionInMeta > parentTable=hbase:meta, metaLocation={region=hbase:meta,,1.1588230740, > hostname=10.128.122.77,16201,1427783593678, seqNum=0}, attempt=3 of 35 > failed; retrying after sleep of 504 because: No server address listed in > hbase:meta for region > SYSTEM.CATALOG,,1427426385032.7e46009a444329769d8e851cb5900006. containing > row > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 12 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 12, > totalSize: 155 bytes > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 13 > cell_block_meta { length: 1084 }, totalSize: 1105 bytes > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 13 > method_name: "Scan" request_param: true priority: 100 > DEBUG [main] (ClientSmallScanner.java:136) - Finished with small scan at > {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => > ''} > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 14 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 14, > totalSize: 155 bytes > DEBUG [main] (HConnectionManager.java:1339) - locateRegionInMeta > parentTable=hbase:meta, metaLocation={region=hbase:meta,,1.1588230740, > hostname=10.128.122.77,16201,1427783593678, seqNum=0}, attempt=4 of 35 > failed; retrying after sleep of 1001 because: No server address listed in > hbase:meta for region > SYSTEM.CATALOG,,1427426385032.7e46009a444329769d8e851cb5900006. containing > row > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 15 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 15, > totalSize: 155 bytes > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 16 > method_name: "Scan" request_param: true priority: 100 > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 16 > cell_block_meta { length: 1084 }, totalSize: 1105 bytes > DEBUG [main] (ClientSmallScanner.java:136) - Finished with small scan at > {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => > ''} > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 17 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 17, > totalSize: 155 bytes > DEBUG [main] (HConnectionManager.java:1339) - locateRegionInMeta > parentTable=hbase:meta, metaLocation={region=hbase:meta,,1.1588230740, > hostname=10.128.122.77,16201,1427783593678, seqNum=0}, attempt=5 of 35 > failed; retrying after sleep of 2017 because: No server address listed in > hbase:meta for region > SYSTEM.CATALOG,,1427426385032.7e46009a444329769d8e851cb5900006. containing > row > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 18 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 18, > totalSize: 155 bytes > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 19 > method_name: "Scan" request_param: true priority: 100 > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 19 > cell_block_meta { length: 1084 }, totalSize: 1105 bytes > DEBUG [main] (ClientSmallScanner.java:136) - Finished with small scan at > {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => > ''} > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 20 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 20, > totalSize: 155 bytes > DEBUG [main] (HConnectionManager.java:1339) - locateRegionInMeta > parentTable=hbase:meta, metaLocation={region=hbase:meta,,1.1588230740, > hostname=10.128.122.77,16201,1427783593678, seqNum=0}, attempt=6 of 35 > failed; retrying after sleep of 4039 because: No server address listed in > hbase:meta for region > SYSTEM.CATALOG,,1427426385032.7e46009a444329769d8e851cb5900006. containing > row > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 21 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 21, > totalSize: 155 bytes > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 22 > method_name: "Scan" request_param: true priority: 100 > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 22 > cell_block_meta { length: 1084 }, totalSize: 1105 bytes > DEBUG [main] (ClientSmallScanner.java:136) - Finished with small scan at > {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => > ''} > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 23 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 23, > totalSize: 155 bytes > DEBUG [main] (HConnectionManager.java:1339) - locateRegionInMeta > parentTable=hbase:meta, metaLocation={region=hbase:meta,,1.1588230740, > hostname=10.128.122.77,16201,1427783593678, seqNum=0}, attempt=7 of 35 > failed; retrying after sleep of 10042 because: No server address listed in > hbase:meta for region > SYSTEM.CATALOG,,1427426385032.7e46009a444329769d8e851cb5900006. containing > row > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16020 from > tenglinxiao] (RpcClient.java:1017) - IPC Client (920483469) connection to / > 10.128.122.77:16020 from tenglinxiao: closed > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16020 from > tenglinxiao] (RpcClient.java:737) - IPC Client (920483469) connection to / > 10.128.122.77:16020 from tenglinxiao: stopped, connections 0 > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1017) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: closed > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:737) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: stopped, connections 0 > DEBUG [main] (RpcClient.java:428) - Use SIMPLE authentication for service > ClientService, sasl=false > DEBUG [main] (RpcClient.java:862) - Connecting to /10.128.122.77:16201 > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:722) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: starting, connections 1 > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 24 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 24, > totalSize: 155 bytes > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 25 > method_name: "Scan" request_param: true priority: 100 > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 25 > cell_block_meta { length: 1084 }, totalSize: 1105 bytes > DEBUG [main] (ClientSmallScanner.java:136) - Finished with small scan at > {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => > ''} > DEBUG [main] (RpcClient.java:1054) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: wrote request header call_id: 26 > method_name: "Get" request_param: true > DEBUG [IPC Client (920483469) connection to /10.128.122.77:16201 from > tenglinxiao] (RpcClient.java:1082) - IPC Client (920483469) connection to / > 10.128.122.77:16201 from tenglinxiao: got response header call_id: 26, > totalSize: 155 bytes > DEBUG [main] (HConnectionManager.java:1339) - locateRegionInMeta > parentTable=hbase:meta, metaLocation={region=hbase:meta,,1.1588230740, > hostname=10.128.122.77,16201,1427783593678, seqNum=0}, attempt=8 of 35 > failed; retrying after sleep of 10013 because: No server address listed in > hbase:meta for region > SYSTEM.CATALOG,,1427426385032.7e46009a444329769d8e851cb5900006. containing > row
