That’s quite an old version of 0.98 and we no longer support Hadoop 1. Would it be possible for you to upgrade your cluster and use Hadoop 2 instead?
On Sat, Nov 18, 2017 at 1:16 PM Eisenhut, Roman <roman.eisen...@tum.de> wrote: > Dear Phoenix community, > > > > I’m trying to implement apache-phoenix-4.13.0-HBase-0.98-bin on > Hadoop-1.2.1 and HBase-0.98.6.1 on a 4 node cluster, running the following > services: > > - roman-master: > - NameNode > - SecondaryNamenode > - HMaster > - HQuorumPeer > - roman-rs-1: > - DataNode > - HRegionServer > - roman-rs-2: > - DataNode > - HRegionServer > - roman-rs-3: > - DataNode > - HRegionServer > > > > I have copied the corresponding server.jar into the lib directory of my > HBase installation on every server. > > > > *ubuntu@roman-master:~$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib | grep > phoenix* > > *-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:15 > phoenix-4.13.0-HBase-0.98-server.jar* > > > > *ubuntu@roman-rs-1:~$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib | grep > phoenix* > > *-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:15 > phoenix-4.13.0-HBase-0.98-server.jar* > > > > *ubuntu@roman-rs-2:/usr/local$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib > | grep phoenix* > > *-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:16 > phoenix-4.13.0-HBase-0.98-server.jar* > > > > *ubuntu@roman-rs-3:/usr/local$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib > | grep phoenix* > > *-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:16 > phoenix-4.13.0-HBase-0.98-server.jar* > > > > I can: > > - Start Hadoop > - Start HBase > - Create a table in HBase > - Put data > - Get data > - Browse > - http://roman-master:50070/dfshealth.jsp > - http://roman-master:60010/master-status > > > > jps shows all the processes are up and running on each machine and the > http interfaces show everything is running properly. It seems like the > system is in perfect health. However, as soon as I try to connect to > phoenix via */usr/local/ > apache-phoenix-4.13.0-HBase-0.98-bin/bin/sqlline.py roman-master:2181* on > roman-master, where roman-maste is my zookeeper quorum, it kills every > regionservers on the entire cluster. After a long time, I don’t know > exactly how long, sqlline will return an output (can be found under [3]). > > > > *Connection call* > > ubuntu@roman-master:/usr/local/apache-phoenix-4.13.0-HBase-0.98-bin/bin$ > ./sqlline.py roman-master:2181 > > Setting property: [incremental, false] > > Setting property: [isolation, TRANSACTION_READ_COMMITTED] > > issuing: !connect jdbc:phoenix:roman-master:2181 none none > org.apache.phoenix.jdbc.PhoenixDriver > > Connecting to jdbc:phoenix:roman-master:2181 > > 17/11/18 19:24:33 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > > > > Underneath you’ll find the following files > > - Conf files for > - Phoenix > - env.sh > - Hadoop > - hadoop-env.sh > - core-site.xml > - hdfs-site.xml > - HBase > - hbase-site.xml > - hbase-env.sh > - [1] master log on roman-master > - [2] regionserver log roman-rs-1 > - [3] zookeeper log on roman-master > - [4] output of sqlline.py > > > > I’m pretty sure there’s a reasonable explanation for this behavior. > Nevertheless, I’ve been trying to get phoenix up and running for days now > and I can’t figure out what I’m doing wrong. I hope someone can point me > into the right direction. > > > > Best regards, > > Roman > > > > > > > > > > > > *Phoenix conf:* > > > ubuntu@roman-master:/usr/local/apache-phoenix-4.13.0-HBase-0.98-bin/bin/config$ > cat env.sh > > JAVA_HOME=/usr/lib/jvm/java-8-oracle/ > > HBASE_PATH=/usr/local/hbase-0.98.6.1-hadoop1/ > > > > *Hadoop conf:* > > *Hadoop-env.sh* > > > > export JAVA_HOME=/usr/lib/jvm/java-8-oracle > > # The maximum amount of heap to use, in MB. Default is 1000. > > # export HADOOP_HEAPSIZE=2000 > > # Command specific options appended to HADOOP_OPTS when specified > > export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote > $HADOOP_NAMENODE_OPTS" > > export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote > $HADOOP_SECONDARYNAMENODE_OPTS" > > export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote > $HADOOP_DATANODE_OPTS" > > export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote > $HADOOP_BALANCER_OPTS" > > export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote > $HADOOP_JOBTRACKER_OPTS" > > _EOF_ > > > > <?xml version="1.0"?> > > <!-- core-site.xml --> > > > > <configuration> > > <property> > > <name>fs.default.name</name> > > <value>hdfs://roman-master/</value> > > <final>true</final> > > </property> > > <property> > > <name>hadoop.tmp.dir</name> > > <value>/usr/local/hadoop_tmp_dir</value> > > </property> > > </configuration> > > > > <?xml version="1.0"?> > > <!-- hdfs-site.xml --> > > <configuration> > > <property> > > <name>dfs.name.dir</name> > > <value>/usr/local/dfs/namenode</value> > > <final>true</final> > > </property> > > <property> > > <name>dfs.data.dir</name> > > <value>/usr/local/dfs/datanode</value> > > <final>true</final> > > </property> > > <property> > > <name>fs.checkpoint.dir</name> > > <value>/usr/local/dfs/checkpoint</value> > > <final>true</final> > > </property> > > </configuration> > > > > *HBase conf* > > > > *hbase.env.sh <http://hbase.env.sh>* > > export HBASE_HEAPSIZE=4096 > > export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64/ > > export HBASE_OPTS="-XX:+UseConcMarkSweepGC" > > > > > > *hbase-site.xml* > > <?xml version="1.0"?> > > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > > > > <configuration> > > <property> > > <name>hbase.zookeeper.quorum</name> > > <value>roman-master</value> > > </property> > > <!-- > > <property> > > <name>hbase.zookeeper.property.clientPort</name> > > <value>5000</value> > > </property> > > --> > > <property> > > > <name>hbase.zookeeper.property.dataDir</name> > > <value>/usr/local/zookeeper</value> > > </property> > > <property> > > <name>hbase.rootdir</name> > > <value>hdfs://roman-master/hbase</value> > > </property> > > <property> > > <name>hbase.cluster.distributed</name> > > <value>true</value> > > </property> > > </configuration> > > > > > > [1] > > 2017-11-18 19:42:08,789 DEBUG [RegionOpenAndInitThread-SYSTEM.CATALOG-1] > regionserver.HRegion: Instantiated > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. > > 2017-11-18 19:42:08,789 DEBUG [RegionOpenAndInitThread-SYSTEM.CATALOG-1] > regionserver.HRegion: Closing > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.: disabling > compactions & flushes > > 2017-11-18 19:42:08,790 DEBUG [RegionOpenAndInitThread-SYSTEM.CATALOG-1] > regionserver.HRegion: Updates disabled for region > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. > > 2017-11-18 19:42:08,790 INFO [RegionOpenAndInitThread-SYSTEM.CATALOG-1] > regionserver.HRegion: Closed > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. > > 2017-11-18 19:42:08,801 INFO > [MASTER_TABLE_OPERATIONS-roman-master:60000-0] catalog.MetaEditor: Added 1 > > 2017-11-18 19:42:08,801 INFO > [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.AssignmentManager: > Bulk assigning 1 region(s) across 3 server(s), round-robin=true > > 2017-11-18 19:42:08,802 DEBUG > [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.GeneralBulkAssigner: > Timeout-on-RIT=151000 > > 2017-11-18 19:42:08,802 DEBUG > [roman-master,60000,1511033853960-GeneralBulkAssigner-2] > master.AssignmentManager: Assigning 1 region(s) to > roman-rs-3,60020,1511033854738 > > 2017-11-18 19:42:08,803 DEBUG > [roman-master,60000,1511033853960-GeneralBulkAssigner-2] > zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000, > quorum=roman-master:2181, baseZNode=/hbase Async create of unassigned node > 6608080647cc1ca0c6610b92de5ae0c4 with OFFLINE state > > 2017-11-18 19:42:08,809 DEBUG [main-EventThread] master.OfflineCallback: > rs={6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034128801, > server=null}, server=roman-rs-3,60020,1511033854738 > > 2017-11-18 19:42:08,810 DEBUG [main-EventThread] > master.OfflineCallback$ExistCallback: rs={6608080647cc1ca0c6610b92de5ae0c4 > state=OFFLINE, ts=1511034128801, server=null}, > server=roman-rs-3,60020,1511033854738 > > 2017-11-18 19:42:08,813 INFO > [roman-master,60000,1511033853960-GeneralBulkAssigner-2] > master.AssignmentManager: roman-rs-3,60020,1511033854738 unassigned > znodes=1 of total=1 > > 2017-11-18 19:42:08,813 INFO > [roman-master,60000,1511033853960-GeneralBulkAssigner-2] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=OFFLINE, ts=1511034128803, server=null} to > {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034128813, > server=roman-rs-3,60020,1511033854738} > > 2017-11-18 19:42:08,826 DEBUG > [roman-master,60000,1511033853960-GeneralBulkAssigner-2] > master.AssignmentManager: Bulk assigning done for > roman-rs-3,60020,1511033854738 > > 2017-11-18 19:42:08,826 DEBUG > [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.GeneralBulkAssigner: > bulk assigning total 1 regions to 3 servers, took 24ms, with 1 regions > still in transition > > 2017-11-18 19:42:08,826 INFO > [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.AssignmentManager: > Bulk assigning done > > 2017-11-18 19:42:08,841 DEBUG > [MASTER_TABLE_OPERATIONS-roman-master:60000-0] lock.ZKInterProcessLockBase: > Released /hbase/table-lock/SYSTEM.CATALOG/write-master:600000000000000 > > 2017-11-18 19:42:08,841 INFO > [MASTER_TABLE_OPERATIONS-roman-master:60000-0] handler.CreateTableHandler: > failed. null > > 2017-11-18 19:42:08,848 DEBUG [AM.ZK.Worker-pool2-t10] > master.AssignmentManager: Handling RS_ZK_REGION_OPENING, > server=roman-rs-3,60020,1511033854738, > region=6608080647cc1ca0c6610b92de5ae0c4, > current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, > ts=1511034128813, server=roman-rs-3,60020,1511033854738} > > 2017-11-18 19:42:08,848 INFO [AM.ZK.Worker-pool2-t10] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=PENDING_OPEN, ts=1511034128813, > server=roman-rs-3,60020,1511033854738} to {6608080647cc1ca0c6610b92de5ae0c4 > state=OPENING, ts=1511034128848, server=roman-rs-3,60020,1511033854738} > > 2017-11-18 19:42:08,946 ERROR [FifoRpcScheduler.handler1-thread-12] > master.HMaster: Region server roman-rs-3,60020,1511033854738 reported a > fatal error: > > ABORTING region server roman-rs-3,60020,1511033854738: The coprocessor > org.apache.phoenix.coprocessor.MetaDataEndpointImpl threw > java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo > > Cause: > > java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo > > at > org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:252) > > at > org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:484) > > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:673) > > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:265) > > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:218) > > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:227) > > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:183) > > at > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:623) > > at > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:530) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > at > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4160) > > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4471) > > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4444) > > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4400) > > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4351) > > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:482) > > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:145) > > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.MetricsInfo > > at > java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > ... 24 more > > > > 2017-11-18 19:42:11,245 DEBUG [AM.ZK.Worker-pool2-t11] > master.AssignmentManager: Handling RS_ZK_REGION_FAILED_OPEN, > server=roman-rs-3,60020,1511033854738, > region=6608080647cc1ca0c6610b92de5ae0c4, > current_state={6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, > ts=1511034128848, server=roman-rs-3,60020,1511033854738} > > 2017-11-18 19:42:11,245 INFO [AM.ZK.Worker-pool2-t11] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=OPENING, ts=1511034128848, server=roman-rs-3,60020,1511033854738} to > {6608080647cc1ca0c6610b92de5ae0c4 state=CLOSED, ts=1511034131245, > server=roman-rs-3,60020,1511033854738} > > 2017-11-18 19:42:11,245 DEBUG [AM.ZK.Worker-pool2-t11] > master.AssignmentManager: Found an existing plan for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination > server is roman-rs-3,60020,1511033854738 accepted as a dest server = false > > 2017-11-18 19:42:11,245 DEBUG [AM.ZK.Worker-pool2-t11] > master.AssignmentManager: No previous transition plan found (or ignoring an > existing plan) for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated > random > plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., > src=, dest=roman-rs-2,60020,1511033854813; 3 (online=3, available=2) > available servers, forceNewPlan=true > > 2017-11-18 19:42:11,246 DEBUG [AM.ZK.Worker-pool2-t11] > handler.ClosedRegionHandler: Handling CLOSED event for > 6608080647cc1ca0c6610b92de5ae0c4 > > 2017-11-18 19:42:11,247 DEBUG [AM.ZK.Worker-pool2-t11] > master.AssignmentManager: Found an existing plan for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination > server is roman-rs-2,60020,1511033854813 accepted as a dest server = true > > 2017-11-18 19:42:11,247 DEBUG [AM.ZK.Worker-pool2-t11] > master.AssignmentManager: Using pre-existing plan for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; > plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., > src=, dest=roman-rs-2,60020,1511033854813 > > 2017-11-18 19:42:11,247 INFO [AM.ZK.Worker-pool2-t11] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=CLOSED, ts=1511034131246, server=roman-rs-3,60020,1511033854738} to > {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034131247, > server=roman-rs-3,60020,1511033854738} > > 2017-11-18 19:42:11,247 DEBUG [AM.ZK.Worker-pool2-t11] zookeeper.ZKAssign: > master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase > Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 > with OFFLINE state > > 2017-11-18 19:42:11,256 INFO [AM.ZK.Worker-pool2-t11] > master.AssignmentManager: Assigning > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to > roman-rs-2,60020,1511033854813 > > 2017-11-18 19:42:11,257 INFO [AM.ZK.Worker-pool2-t11] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=OFFLINE, ts=1511034131247, server=roman-rs-3,60020,1511033854738} to > {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034131256, > server=roman-rs-2,60020,1511033854813} > > 2017-11-18 19:42:11,257 DEBUG [AM.ZK.Worker-pool2-t11] > master.ServerManager: New admin connection to roman-rs-2,60020,1511033854813 > > 2017-11-18 19:42:11,424 DEBUG [AM.ZK.Worker-pool2-t12] > master.AssignmentManager: Handling M_ZK_REGION_OFFLINE, > server=roman-rs-2,60020,1511033854813, > region=6608080647cc1ca0c6610b92de5ae0c4, > current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, > ts=1511034131256, server=roman-rs-2,60020,1511033854813} > > 2017-11-18 19:42:11,444 DEBUG [AM.ZK.Worker-pool2-t13] > master.AssignmentManager: Handling RS_ZK_REGION_OPENING, > server=roman-rs-2,60020,1511033854813, > region=6608080647cc1ca0c6610b92de5ae0c4, > current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, > ts=1511034131256, server=roman-rs-2,60020,1511033854813} > > 2017-11-18 19:42:11,444 INFO [AM.ZK.Worker-pool2-t13] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=PENDING_OPEN, ts=1511034131256, > server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4 > state=OPENING, ts=1511034131444, server=roman-rs-2,60020,1511033854813} > > 2017-11-18 19:42:11,591 ERROR [FifoRpcScheduler.handler1-thread-14] > master.HMaster: Region server roman-rs-2,60020,1511033854813 reported a > fatal error: > > ABORTING region server roman-rs-2,60020,1511033854813: The coprocessor > org.apache.phoenix.coprocessor.MetaDataEndpointImpl threw > java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo > > Cause: > > java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo > > at > org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:252) > > at > org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:484) > > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:673) > > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:265) > > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:218) > > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:227) > > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:183) > > at > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:623) > > at > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:530) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > at > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4160) > > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4471) > > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4444) > > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4400) > > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4351) > > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:482) > > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:145) > > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.MetricsInfo > > at > java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > ... 24 more > > > > 2017-11-18 19:42:13,874 DEBUG [AM.ZK.Worker-pool2-t14] > master.AssignmentManager: Handling RS_ZK_REGION_FAILED_OPEN, > server=roman-rs-2,60020,1511033854813, > region=6608080647cc1ca0c6610b92de5ae0c4, > current_state={6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, > ts=1511034131444, server=roman-rs-2,60020,1511033854813} > > 2017-11-18 19:42:13,874 INFO [AM.ZK.Worker-pool2-t14] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=OPENING, ts=1511034131444, server=roman-rs-2,60020,1511033854813} to > {6608080647cc1ca0c6610b92de5ae0c4 state=CLOSED, ts=1511034133874, > server=roman-rs-2,60020,1511033854813} > > 2017-11-18 19:42:13,887 DEBUG [AM.ZK.Worker-pool2-t14] > master.AssignmentManager: Found an existing plan for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination > server is roman-rs-2,60020,1511033854813 accepted as a dest server = false > > 2017-11-18 19:42:13,887 DEBUG [AM.ZK.Worker-pool2-t14] > master.AssignmentManager: No previous transition plan found (or ignoring an > existing plan) for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated > random > plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., > src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=2) > available servers, forceNewPlan=true > > 2017-11-18 19:42:13,887 DEBUG [AM.ZK.Worker-pool2-t14] > handler.ClosedRegionHandler: Handling CLOSED event for > 6608080647cc1ca0c6610b92de5ae0c4 > > 2017-11-18 19:42:13,888 DEBUG [AM.ZK.Worker-pool2-t14] > master.AssignmentManager: Found an existing plan for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination > server is roman-rs-1,60020,1511033854754 accepted as a dest server = true > > 2017-11-18 19:42:13,888 DEBUG [AM.ZK.Worker-pool2-t14] > master.AssignmentManager: Using pre-existing plan for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; > plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., > src=, dest=roman-rs-1,60020,1511033854754 > > 2017-11-18 19:42:13,888 INFO [AM.ZK.Worker-pool2-t14] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=CLOSED, ts=1511034133888, server=roman-rs-2,60020,1511033854813} to > {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034133888, > server=roman-rs-2,60020,1511033854813} > > 2017-11-18 19:42:13,888 DEBUG [AM.ZK.Worker-pool2-t14] zookeeper.ZKAssign: > master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase > Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 > with OFFLINE state > > 2017-11-18 19:42:13,896 INFO [AM.ZK.Worker-pool2-t14] > master.AssignmentManager: Assigning > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to > roman-rs-1,60020,1511033854754 > > 2017-11-18 19:42:13,896 INFO [AM.ZK.Worker-pool2-t14] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=OFFLINE, ts=1511034133888, server=roman-rs-2,60020,1511033854813} to > {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034133896, > server=roman-rs-1,60020,1511033854754} > > 2017-11-18 19:42:13,896 DEBUG [AM.ZK.Worker-pool2-t14] > master.ServerManager: New admin connection to roman-rs-1,60020,1511033854754 > > 2017-11-18 19:42:14,055 DEBUG [AM.ZK.Worker-pool2-t15] > master.AssignmentManager: Handling M_ZK_REGION_OFFLINE, > server=roman-rs-1,60020,1511033854754, > region=6608080647cc1ca0c6610b92de5ae0c4, > current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, > ts=1511034133896, server=roman-rs-1,60020,1511033854754} > > 2017-11-18 19:42:14,073 DEBUG [AM.ZK.Worker-pool2-t16] > master.AssignmentManager: Handling RS_ZK_REGION_OPENING, > server=roman-rs-1,60020,1511033854754, > region=6608080647cc1ca0c6610b92de5ae0c4, > current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, > ts=1511034133896, server=roman-rs-1,60020,1511033854754} > > 2017-11-18 19:42:14,073 INFO [AM.ZK.Worker-pool2-t16] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=PENDING_OPEN, ts=1511034133896, > server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4 > state=OPENING, ts=1511034134073, server=roman-rs-1,60020,1511033854754} > > 2017-11-18 19:42:14,171 ERROR [FifoRpcScheduler.handler1-thread-16] > master.HMaster: Region server roman-rs-1,60020,1511033854754 reported a > fatal error: > > ABORTING region server roman-rs-1,60020,1511033854754: The coprocessor > org.apache.phoenix.coprocessor.MetaDataEndpointImpl threw > java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo > > Cause: > > java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo > > at > org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:252) > > at > org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:484) > > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:673) > > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:265) > > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:218) > > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:227) > > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:183) > > at > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:623) > > at > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:530) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > at > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4160) > > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4471) > > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4444) > > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4400) > > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4351) > > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:482) > > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:145) > > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.MetricsInfo > > at > java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > ... 24 more > > > > 2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Handling RS_ZK_REGION_FAILED_OPEN, > server=roman-rs-1,60020,1511033854754, > region=6608080647cc1ca0c6610b92de5ae0c4, > current_state={6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, > ts=1511034134073, server=roman-rs-1,60020,1511033854754} > > 2017-11-18 19:42:16,457 INFO [AM.ZK.Worker-pool2-t17] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=OPENING, ts=1511034134073, server=roman-rs-1,60020,1511033854754} to > {6608080647cc1ca0c6610b92de5ae0c4 state=CLOSED, ts=1511034136457, > server=roman-rs-1,60020,1511033854754} > > 2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Found an existing plan for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination > server is roman-rs-1,60020,1511033854754 accepted as a dest server = false > > 2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: No previous transition plan found (or ignoring an > existing plan) for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated > random > plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., > src=, dest=roman-rs-2,60020,1511033854813; 3 (online=3, available=2) > available servers, forceNewPlan=true > > 2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] > handler.ClosedRegionHandler: Handling CLOSED event for > 6608080647cc1ca0c6610b92de5ae0c4 > > 2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Found an existing plan for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination > server is roman-rs-2,60020,1511033854813 accepted as a dest server = true > > 2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Using pre-existing plan for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; > plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., > src=, dest=roman-rs-2,60020,1511033854813 > > 2017-11-18 19:42:16,478 INFO [AM.ZK.Worker-pool2-t17] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=CLOSED, ts=1511034136457, server=roman-rs-1,60020,1511033854754} to > {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136478, > server=roman-rs-1,60020,1511033854754} > > 2017-11-18 19:42:16,478 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: > master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase > Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 > with OFFLINE state > > 2017-11-18 19:42:16,488 INFO [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Assigning > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to > roman-rs-2,60020,1511033854813 > > 2017-11-18 19:42:16,488 INFO [AM.ZK.Worker-pool2-t17] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=OFFLINE, ts=1511034136478, server=roman-rs-1,60020,1511033854754} to > {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136488, > server=roman-rs-2,60020,1511033854813} > > 2017-11-18 19:42:16,491 WARN [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Failed assignment of > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to > roman-rs-2,60020,1511033854813, trying to assign elsewhere instead; try=1 > of 10 > > java.io.IOException: Unexpected closed connection > > at > org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1450) > > at > org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) > > at > org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) > > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964) > > at > org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671) > > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097) > > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577) > > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550) > > at > org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104) > > at > org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032) > > at > org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447) > > at > org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > 2017-11-18 19:42:16,494 DEBUG [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Found an existing plan for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination > server is roman-rs-2,60020,1511033854813 accepted as a dest server = true > > 2017-11-18 19:42:16,494 DEBUG [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: No previous transition plan found (or ignoring an > existing plan) for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated > random > plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., > src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=3) > available servers, forceNewPlan=true > > 2017-11-18 19:42:16,494 INFO [AM.ZK.Worker-pool2-t17] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=PENDING_OPEN, ts=1511034136488, > server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4 > state=OFFLINE, ts=1511034136494, server=roman-rs-2,60020,1511033854813} > > 2017-11-18 19:42:16,494 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: > master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase > Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 > with OFFLINE state > > 2017-11-18 19:42:16,502 INFO [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Assigning > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to > roman-rs-1,60020,1511033854754 > > 2017-11-18 19:42:16,502 INFO [AM.ZK.Worker-pool2-t17] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=OFFLINE, ts=1511034136494, server=roman-rs-2,60020,1511033854813} to > {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136502, > server=roman-rs-1,60020,1511033854754} > > 2017-11-18 19:42:16,503 WARN [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Failed assignment of > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to > roman-rs-1,60020,1511033854754, trying to assign elsewhere instead; try=2 > of 10 > > java.io.IOException: Call to roman-rs-1/172.24.62.221:60020 failed on > local exception: java.io.EOFException > > at > org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1489) > > at > org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1461) > > at > org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) > > at > org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) > > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964) > > at > org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671) > > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097) > > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577) > > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550) > > at > org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104) > > at > org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032) > > at > org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447) > > at > org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > Caused by: java.io.EOFException > > at > java.io.DataInputStream.readInt(DataInputStream.java:392) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1076) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:727) > > 2017-11-18 19:42:16,504 DEBUG [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Found an existing plan for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination > server is roman-rs-1,60020,1511033854754 accepted as a dest server = true > > 2017-11-18 19:42:16,504 DEBUG [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: No previous transition plan found (or ignoring an > existing plan) for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated > random > plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., > src=, dest=roman-rs-2,60020,1511033854813; 3 (online=3, available=3) > available servers, forceNewPlan=true > > 2017-11-18 19:42:16,504 INFO [AM.ZK.Worker-pool2-t17] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=PENDING_OPEN, ts=1511034136502, > server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4 > state=OFFLINE, ts=1511034136504, server=roman-rs-1,60020,1511033854754} > > 2017-11-18 19:42:16,504 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: > master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase > Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 > with OFFLINE state > > 2017-11-18 19:42:16,513 INFO [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Assigning > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to > roman-rs-2,60020,1511033854813 > > 2017-11-18 19:42:16,513 INFO [AM.ZK.Worker-pool2-t17] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=OFFLINE, ts=1511034136504, server=roman-rs-1,60020,1511033854754} to > {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136513, > server=roman-rs-2,60020,1511033854813} > > 2017-11-18 19:42:16,515 WARN [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Failed assignment of > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to > roman-rs-2,60020,1511033854813, trying to assign elsewhere instead; try=3 > of 10 > > java.net.ConnectException: Connection refused > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) > > at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > > at > org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511) > > at > org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) > > at > org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) > > at > org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) > > at > org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) > > at > org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) > > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964) > > at > org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671) > > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097) > > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577) > > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550) > > at > org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104) > > at > org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032) > > at > org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447) > > at > org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > 2017-11-18 19:42:16,515 DEBUG [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Found an existing plan for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination > server is roman-rs-2,60020,1511033854813 accepted as a dest server = true > > 2017-11-18 19:42:16,515 DEBUG [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: No previous transition plan found (or ignoring an > existing plan) for > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated > random > plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., > src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=3) > available servers, forceNewPlan=true > > 2017-11-18 19:42:16,515 INFO [AM.ZK.Worker-pool2-t17] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=PENDING_OPEN, ts=1511034136513, > server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4 > state=OFFLINE, ts=1511034136515, server=roman-rs-2,60020,1511033854813} > > 2017-11-18 19:42:16,515 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: > master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase > Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 > with OFFLINE state > > 2017-11-18 19:42:16,538 INFO [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Assigning > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to > roman-rs-1,60020,1511033854754 > > 2017-11-18 19:42:16,538 INFO [AM.ZK.Worker-pool2-t17] > master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 > state=OFFLINE, ts=1511034136515, server=roman-rs-2,60020,1511033854813} to > {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136538, > server=roman-rs-1,60020,1511033854754} > > 2017-11-18 19:42:16,539 WARN [AM.ZK.Worker-pool2-t17] > master.AssignmentManager: Failed assignment of > SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to > roman-rs-1,60020,1511033854754, trying to assign elsewhere instead; try=4 > of 10 > > java.net.ConnectException: Connection refused > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) > > at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > > at > org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511) > > at > org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) > > at org.apache.hadoop.hba >
