jcl: I can see that DEBUG log wasn't turned on. Can you set log4j to DEBUG level and see if there is more information ?
Cheers On Tue, Aug 14, 2018 at 6:56 AM Allan Yang <[email protected]> wrote: > Those log is not enough to locate the problem. > Best Regards > Allan Yang > > > jcl <[email protected]> 于2018年8月14日周二 下午9:18写道： > > > Power off and restart(Hadoop and HBase), Master is initializing - Hbase > > ServerManager: but crash processing already in progress > > > > command jps, HMaster and HRegionServer is live > > > > > > WARN [Thread-14] master.ServerManager: Expiration called on > > hbase-115,16020,1534248994825 but crash processing already in progress > > WARN [Thread-14] master.ServerManager: Expiration called on > > hbase-116,16020,1534248590107 but crash processing already in progress > > WARN [Thread-14] master.ServerManager: Expiration called on > > hbase-115,16020,1534249077856 but crash processing already in progress > > WARN [Thread-14] master.ServerManager: Expiration called on > > hbase-116,16020,1534248994045 but crash processing already in progress > > WARN [Thread-14] master.ServerManager: Expiration called on > > hbase-115,16020,1534248708149 but crash processing already in progress > > WARN [Thread-14] master.ServerManager: Expiration called on > > hbase-116,16020,1534248707381 but crash processing already in progress > > > > > > > > LOG: > > > > core file size (blocks, -c) 0 > > data seg size (kbytes, -d) unlimited > > scheduling priority (-e) 0 > > file size (blocks, -f) unlimited > > pending signals (-i) 64091 > > max locked memory (kbytes, -l) 64 > > max memory size (kbytes, -m) unlimited > > open files (-n) 1024 > > pipe size (512 bytes, -p) 8 > > POSIX message queues (bytes, -q) 819200 > > real-time priority (-r) 0 > > stack size (kbytes, -s) 8192 > > cpu time (seconds, -t) unlimited > > max user processes (-u) 64091 > > virtual memory (kbytes, -v) unlimited > > file locks (-x) unlimited > > 2018-08-14 17:25:00,173 INFO [main] master.HMaster: STARTING service > > HMaster > > 2018-08-14 17:25:00,174 INFO [main] util.VersionInfo: HBase 2.1.0 > > 2018-08-14 17:25:00,174 INFO [main] util.VersionInfo: Source code > > repository revision=4531d1c947a25b28a9a994b60c791a112c12a2b4 > > 2018-08-14 17:25:00,174 INFO [main] util.VersionInfo: Compiled by hbase > on > > Wed Aug 1 11:25:59 2018 > > 2018-08-14 17:25:00,174 INFO [main] util.VersionInfo: From source with > > checksum fc32566f7e030ff71458fbf6dc77bce9 > > 2018-08-14 17:25:00,516 INFO [main] util.ServerCommandLine: > hbase.tmp.dir: > > /tmp/hbase-root > > 2018-08-14 17:25:00,516 INFO [main] util.ServerCommandLine: > hbase.rootdir: > > hdfs://192.168.101.114:9000/hbase > > 2018-08-14 <http://192.168.101.114:9000/hbase2018-08-14> 17:25:00,516 > > INFO [main] util.ServerCommandLine: hbase.cluster.distributed: true > > 2018-08-14 17:25:00,516 INFO [main] util.ServerCommandLine: > > hbase.zookeeper.quorum: 192.168.101.114:2181 > > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: > > > env:PATH=/opt/apache-phoenix-5.0.0-HBase-2.0-bin/bin:/opt/hbase-2.1.0/bin:/opt/hadoop-2.8.4/bin:/opt/jdk1.8.0_172/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin > > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: > > env:HADOOP_CONF_DIR=/opt/hadoop-2.8.4/etc/hadoop > > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: > > env:HISTCONTROL=ignoredups > > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: > > > env:JAVA_LIBRARY_PATH=/opt/hadoop-2.8.4/lib/native::/opt/hadoop-2.8.4/lib/native: > > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: > > env:HBASE_REGIONSERVER_OPTS= -Xdebug -Xnoagent -Djava.compiler=NONE > > -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071 > > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: > > env:HBASE_CONF_DIR=/opt/hbase-2.1.0/conf > > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: > > env:HDFS_DATANODE_SECURE_USER=root > > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: > > env:MAIL=/var/spool/mail/root > > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: > > env:PHOENIX_HOME=/opt/apache-phoenix-5.0.0-HBase-2.0-bin > > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: > > > env:LD_LIBRARY_PATH=:/opt/hadoop-2.8.4/lib/native::/opt/hadoop-2.8.4/lib/native: > > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine: > > env:LOGNAME=root > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:HBASE_REST_OPTS= > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:PWD=/opt/hbase-2.1.0/bin > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:HADOOP_PREFIX=/opt/hadoop-2.8.4 > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:HADOOP_INSTALL=/opt/hadoop-2.8.4 > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:HBASE_ROOT_LOGGER=INFO,RFA > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:LESSOPEN=||/usr/bin/lesspipe.sh %s > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:SHELL=/bin/bash > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:SELINUX_USE_CURRENT_RANGE= > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:YARN_CONF_DIR=/opt/hadoop-2.8.4/etc/hadoop > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:HBASE_ENV_INIT=true > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:HADOOP_YARN_HOME=/opt/hadoop-2.8.4 > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:HBASE_MASTER_OPTS= -Xdebug > > -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070 > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:HBASE_MANAGES_ZK=false > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:HADOOP_HOME=/opt/hadoop-2.8.4 > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:HBASE_NICENESS=0 > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:HBASE_OPTS= -XX:+UseConcMarkSweepGC -Xdebug > > -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070 > > -Dhbase.log.dir=/opt/hbase-2.1.0/logs > > -Dhbase.log.file=hbase-root-master-hbase-114.log > > -Dhbase.home.dir=/opt/hbase-2.1.0 -Dhbase.id.str=root > > -Dhbase.root.logger=INFO,RFA > > > -Djava.library.path=/opt/hadoop-2.8.4/lib/native::/opt/hadoop-2.8.4/lib/native: > > -Dhbase.security.logger=INFO,RFAS > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:HBASE_SECURITY_LOGGER=INFO,RFAS > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:HDFS_DATANODE_USER=root > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > > 1:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=01;36:.au=01;36:.flac=01;36:.mid=01;36:.midi=01;36:.mka=01;36:.mp3=01;36:.mpc=01;36:.ogg=01;36:.ra=01;36:.wav=01;36:.axa=01;36:.oga=01;36:.spx=01;36:*.xspf=01;36: > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: env:SHLVL=4 > > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine: > > env:HBASE_LOGFILE=hbase-root-master-hbase-114.log > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:HISTSIZE=1000 > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:JAVA_HOME=/opt/jdk1.8.0_172 > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > env:TERM=xterm > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:LANG=zh_CN.UTF-8 > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:XDG_SESSION_ID=1 > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:SELINUX_LEVEL_REQUESTED= > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:HADOOP_LIBEXEC_DIR=/opt/hadoop-2.8.4/libexec > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:SELINUX_ROLE_REQUESTED= > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:HADOOP_HDFS_HOME=/opt/hadoop-2.8.4 > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:HADOOP_MAPRED_HOME=/opt/hadoop-2.8.4 > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:HADOOP_COMMON_HOME=/opt/hadoop-2.8.4 > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > > env:HADOOP_OPTS=-Djava.library.path=/opt/hadoop-2.8.4/lib:/opt/hadoop-2.8.4/lib/native > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:HBASE_IDENT_STRING=root > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:HBASE_ZNODE_FILE=/tmp/hbase-root-master.znode > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:SSH_TTY=/dev/pts/0 > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:SSH_CLIENT=192.168.98.129 35604 22 > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:HBASE_LOG_PREFIX=hbase-root-master-hbase-114 > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:HBASE_LOG_DIR=/opt/hbase-2.1.0/logs > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: env:USER=root > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > > dparty/commons-logging-1.2.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/findbugs-annotations-1.3.9-1.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/log4j-1.2.17.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/phoenix-5.0.0-HBase-2.0-server.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/phoenix-core-5.0.0-HBase-2.0.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/slf4j-api-1.7.25.jar:/opt/hadoop-2.8.4/etc/hadoop:/opt/hadoop-2.8.4/share/hadoop/common/lib/:/opt/hadoop-2.8.4/share/hadoop/common/:/opt/hadoop-2.8.4/share/hadoop/hdfs:/opt/hadoop-2.8.4/share/hadoop/hdfs/lib/:/opt/hadoop-2.8.4/share/hadoop/hdfs/:/opt/hadoop-2.8.4/share/hadoop/yarn/lib/:/opt/hadoop-2.8.4/share/hadoop/yarn/:/opt/hadoop-2.8.4/share/hadoop/mapreduce/lib/:/opt/hadoop-2.8.4/share/hadoop/mapreduce/:/opt/hadoop-2.8.4/contrib/capacity-scheduler/*.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:HDFS_NAMENODE_USER=root > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:SSH_CONNECTION=192.168.98.129 35604 192.168.101.114 22 > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:HBASE_AUTOSTART_FILE=/tmp/hbase-root-master.autostart > > 2018-08-14 17:25:00,519 INFO [main] util.ServerCommandLine: > > env:HOSTNAME=hbase-114 > > 2018-08-14 17:25:00,520 INFO [main] util.ServerCommandLine: > > env:HADOOP_COMMON_LIB_NATIVE_DIR=/opt/hadoop-2.8.4/lib/native > > 2018-08-14 17:25:00,520 INFO [main] util.ServerCommandLine: > > env:XDG_RUNTIME_DIR=/run/user/0 > > 2018-08-14 17:25:00,520 INFO [main] util.ServerCommandLine: > > env:HDFS_SECONDARYNAMENODE_USER=root > > 2018-08-14 17:25:00,520 INFO [main] util.ServerCommandLine: > > env:HBASE_THRIFT_OPTS= > > 2018-08-14 17:25:00,520 INFO [main] util.ServerCommandLine: > > env:HBASE_HOME=/opt/hbase-2.1.0 > > 2018-08-14 17:25:00,520 INFO [main] util.ServerCommandLine: > env:HOME=/root > > 2018-08-14 17:25:00,520 INFO [main] util.ServerCommandLine: > > env:MALLOC_ARENA_MAX=4 > > 2018-08-14 17:25:00,521 INFO [main] util.ServerCommandLine: vmName=Java > > HotSpot(TM) 64-Bit Server VM, vmVendor=Oracle Corporation, > > vmVersion=25.172-b11 > > 2018-08-14 17:25:00,521 INFO [main] util.ServerCommandLine: > > vmInputArguments=[-Dproc_master, -XX:OnOutOfMemoryError=kill -9 %p, > > -XX:+UseConcMarkSweepGC, -Xdebug, > > -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070, > > -Dhbase.log.dir=/opt/hbase-2.1.0/logs, > > -Dhbase.log.file=hbase-root-master-hbase-114.log, > > -Dhbase.home.dir=/opt/hbase-2.1.0, -Dhbase.id.str=root, > > -Dhbase.root.logger=INFO,RFA, > > > -Djava.library.path=/opt/hadoop-2.8.4/lib/native::/opt/hadoop-2.8.4/lib/native:, > > -Dhbase.security.logger=INFO,RFAS] > > 2018-08-14 17:25:00,886 INFO [main] metrics.MetricRegistries: Loaded > > MetricRegistries class > > org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl > > 2018-08-14 17:25:01,258 INFO [main] regionserver.RSRpcServices: > > master/hbase-114:16000 server-side Connection retries=3 > > 2018-08-14 17:25:01,278 INFO [main] ipc.RpcExecutor: Instantiated > > default.FPBQ.Fifo with queueClass=class > > java.util.concurrent.LinkedBlockingQueue; numCallQueues=3, > > maxQueueLength=300, handlerCount=30 > > 2018-08-14 17:25:01,280 INFO [main] ipc.RpcExecutor: Instantiated > > priority.FPBQ.Fifo with queueClass=class > > java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, > > maxQueueLength=300, handlerCount=20 > > 2018-08-14 17:25:01,280 INFO [main] ipc.RpcExecutor: Instantiated > > replication.FPBQ.Fifo with queueClass=class > > java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, > > maxQueueLength=300, handlerCount=3 > > 2018-08-14 17:25:01,418 INFO [main] ipc.RpcServerFactory: Creating > > org.apache.hadoop.hbase.ipc.NettyRpcServer hosting > hbase.pb.MasterService, > > hbase.pb.RegionServerStatusService, hbase.pb.LockService, > > hbase.pb.ClientService, hbase.pb.AdminService > > 2018-08-14 17:25:01,632 INFO [main] ipc.NettyRpcServer: Bind to / > > 192.168.101.114:16000 > > 2018-08-14 17:25:01,688 INFO [main] hfile.CacheConfig: Allocating onheap > > LruBlockCache size=1.55 GB, blockSize=64 KB > > 2018-08-14 17:25:01,694 INFO [main] hfile.CacheConfig: Created > > cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1.16 MB, > > freeSize=1.55 GB, maxSize=1.55 GB, heapSize=1.16 MB, minSize=1.47 GB, > > minFactor=0.95, multiSize=752.80 MB, multiFactor=0.5, singleSize=376.40 > MB, > > singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, > > cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, > > cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false > > 2018-08-14 17:25:01,695 INFO [main] hfile.CacheConfig: Created > > cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1.16 MB, > > freeSize=1.55 GB, maxSize=1.55 GB, heapSize=1.16 MB, minSize=1.47 GB, > > minFactor=0.95, multiSize=752.80 MB, multiFactor=0.5, singleSize=376.40 > MB, > > singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, > > cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, > > cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false > > 2018-08-14 17:25:02,160 INFO [main] fs.HFileSystem: Added intercepting > > call to namenode#getBlockLocations so can do block reordering using class > > org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks > > 2018-08-14 17:25:02,163 INFO [main] fs.HFileSystem: Added intercepting > > call to namenode#getBlockLocations so can do block reordering using class > > org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks > > 2018-08-14 17:25:02,233 INFO [main] zookeeper.RecoverableZooKeeper: > > Process identifier=master:16000 connecting to ZooKeeper ensemble= > > 192.168.101.114:2181 > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client > > > environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, > > built on 03/23/2017 10:13 GMT > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client > > environment:host.name=hbase-114 > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client > > environment:java.version=1.8.0_172 > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client > > environment:java.vendor=Oracle Corporation > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client > > environment:java.home=/opt/jdk1.8.0_172/jre > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: > > > o-2.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.8.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.8.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.8.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.8.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.4.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.4-tests.jar:/opt/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.4.jar:/opt/hadoop-2.8.4/contrib/capacity-scheduler/*.jar:/opt/hbase-2.1.0/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client > > > environment:java.library.path=/opt/hadoop-2.8.4/lib/native::/opt/hadoop-2.8.4/lib/native: > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client > > environment:java.io.tmpdir=/tmp > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client > > environment:java.compiler=<NA> > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client > > environment:os.name=Linux > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client > > environment:os.arch=amd64 > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client > > environment:os.version=3.10.0-862.el7.x86_64 > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client > > environment:user.name=root > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client > > environment:user.home=/root > > 2018-08-14 17:25:02,239 INFO [main] zookeeper.ZooKeeper: Client > > environment:user.dir=/opt/hbase-2.1.0/bin > > 2018-08-14 17:25:02,240 INFO [main] zookeeper.ZooKeeper: Initiating > client > > connection, connectString=192.168.101.114:2181 sessionTimeout=90000 > > watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@4ae2e781 > > 2018-08-14 17:25:02,256 INFO [main-SendThread(192.168.101.114:2181)] > > zookeeper.ClientCnxn: Opening socket connection to server > > 192.168.101.114/192.168.101.114:2181. Will not attempt to authenticate > > using SASL (unknown error) > > 2018-08-14 17:25:02,264 INFO [main-SendThread(192.168.101.114:2181)] > > zookeeper.ClientCnxn: Socket connection established to > > 192.168.101.114/192.168.101.114:2181, initiating session > > 2018-08-14 17:25:02,282 INFO [main-SendThread(192.168.101.114:2181)] > > zookeeper.ClientCnxn: Session establishment complete on server > > 192.168.101.114/192.168.101.114:2181, sessionid = 0x10004dac1970000, > > negotiated timeout = 40000 > > 2018-08-14 17:25:02,352 INFO [main] util.log: Logging initialized @2552ms > > 2018-08-14 17:25:02,413 INFO [main] http.HttpRequestLog: Http request log > > for http.requests.master is not defined > > 2018-08-14 17:25:02,426 INFO [main] http.HttpServer: Added global filter > > 'safety' > (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) > > 2018-08-14 17:25:02,426 INFO [main] http.HttpServer: Added global filter > > 'clickjackingprevention' > > (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) > > 2018-08-14 17:25:02,428 INFO [main] http.HttpServer: Added filter > > static_user_filter > > > (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) > > to context master > > 2018-08-14 17:25:02,428 INFO [main] http.HttpServer: Added filter > > static_user_filter > > > (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) > > to context static > > 2018-08-14 17:25:02,428 INFO [main] http.HttpServer: Added filter > > static_user_filter > > > (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) > > to context logs > > 2018-08-14 17:25:02,455 INFO [main] http.HttpServer: Jetty bound to port > > 16010 > > 2018-08-14 17:25:02,456 INFO [main] server.Server: jetty-9.3.19.v20170502 > > 2018-08-14 17:25:02,489 INFO [main] handler.ContextHandler: Started > > o.e.j.s.ServletContextHandler@49232c6f > > {/logs,file:///opt/hbase-2.1.0/logs/,AVAILABLE} > > 2018-08-14 17:25:02,490 INFO [main] handler.ContextHandler: Started > > o.e.j.s.ServletContextHandler@279126f5 > > {/static,file:///opt/hbase-2.1.0/hbase-webapps/static/,AVAILABLE} > > 2018-08-14 17:25:02,582 INFO [main] handler.ContextHandler: Started > > o.e.j.w.WebAppContext@537b3b2e > > {/,file:///opt/hbase-2.1.0/hbase-webapps/master/,AVAILABLE} > > > > {file:/opt/hbase-2.1.0/hbase-webapps/master} > > > > 2018-08-14 17:25:02,587 INFO [main] server.AbstractConnector: Started > > ServerConnector@550c973e{HTTP/1.1,[http/1.1]} > > > > {0.0.0.0:16010} > > > > 2018-08-14 17:25:02,587 INFO [main] server.Server: Started @2787ms > > 2018-08-14 17:25:02,590 INFO [main] master.HMaster: hbase.rootdir=hdfs:// > > 192.168.101.114:9000/hbase, hbase.cluster.distributed=true > > 2018-08-14 17:25:02,606 INFO [Thread-14] master.HMaster: Adding backup > > master ZNode /hbase/backup-masters/hbase-114,16000,1534238700547 > > 2018-08-14 17:25:02,685 INFO [Thread-14] master.ActiveMasterManager: > > Deleting ZNode for /hbase/backup-masters/hbase-114,16000,1534238700547 > from > > backup master directory > > 2018-08-14 17:25:02,691 INFO [Thread-14] master.ActiveMasterManager: > > Registered as active master=hbase-114,16000,1534238700547 > > 2018-08-14 17:25:02,697 INFO [Thread-14] regionserver.ChunkCreator: > > Allocating data MemStoreChunkPool with chunk size 2 MB, max count 713, > > initial count 0 > > 2018-08-14 17:25:02,698 INFO [Thread-14] regionserver.ChunkCreator: > > Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count > > 792, initial count 0 > > 2018-08-14 17:25:02,992 INFO [Thread-14] fs.HFileSystem: Added > > intercepting call to namenode#getBlockLocations so can do block > reordering > > using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks > > 2018-08-14 17:25:03,001 INFO [Thread-14] > > coordination.SplitLogManagerCoordination: Found 0 orphan tasks and 0 > rescan > > nodes > > 2018-08-14 17:25:03,094 INFO [Thread-14] zookeeper.ReadOnlyZKClient: > > Connect 0x66461af1 to 192.168.101.114:2181 with session timeout=90000ms, > > retries 30, retry interval 1000ms, keepAlive=60000ms > > 2018-08-14 17:25:03,100 INFO > > [ReadOnlyZKClient-192.168.101.114:2181@0x66461af1] zookeeper.ZooKeeper: > > Initiating client connection, connectString=192.168.101.114:2181 > > sessionTimeout=90000 > > > watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$52/1619197561@6e451c19 > > 2018-08-14 17:25:03,101 INFO > > [ReadOnlyZKClient-192.168.101.114:2181@0x66461af1-SendThread( > > 192.168.101.114:2181)] zookeeper.ClientCnxn: Opening socket connection > to > > server 192.168.101.114/192.168.101.114:2181. Will not attempt to > > authenticate using SASL (unknown error) > > 2018-08-14 17:25:03,101 INFO > > [ReadOnlyZKClient-192.168.101.114:2181@0x66461af1-SendThread( > > 192.168.101.114:2181)] zookeeper.ClientCnxn: Socket connection > > established to 192.168.101.114/192.168.101.114:2181, initiating session > > 2018-08-14 17:25:03,104 INFO > > [ReadOnlyZKClient-192.168.101.114:2181@0x66461af1-SendThread( > > 192.168.101.114:2181)] zookeeper.ClientCnxn: Session establishment > > complete on server 192.168.101.114/192.168.101.114:2181, sessionid = > > 0x10004dac1970001, negotiated timeout = 40000 > > 2018-08-14 17:25:03,145 INFO [Thread-14] procedure2.ProcedureExecutor: > > Starting 16 core workers (bigger of cpus/4 or 16) with max (burst) worker > > count=160 > > 2018-08-14 17:25:03,149 INFO [Thread-14] util.FSHDFSUtils: Recover lease > > on dfs file hdfs:// > > 192.168.101.114:9000/hbase/MasterProcWALs/pv2-00000000000000000004.log > > 2018-08-14 > > < > http://192.168.101.114:9000/hbase/MasterProcWALs/pv2-00000000000000000004.log2018-08-14 > > > > 17:25:03,153 INFO [Thread-14] util.FSHDFSUtils: Recovered lease, > attempt=0 > > on file=hdfs:// > > 192.168.101.114:9000/hbase/MasterProcWALs/pv2-00000000000000000004.log > > after 4ms > > 2018-08-14 17:25:03,188 WARN [Thread-14] util.CommonFSUtils: Your Hadoop > > installation does not include the StreamCapabilities class from > HDFS-11644, > > so we will skip checking if any FSDataOutputStreams actually support > > hflush/hsync. If you are running on top of HDFS this probably just means > > you have an older version and this can be ignored. If you are running on > > top of an alternate FileSystem implementation you should manually verify > > that hflush and hsync are implemented; otherwise you risk data loss and > > hard to diagnose errors when our assumptions are violated. > > 2018-08-14 17:25:03,189 INFO [Thread-14] wal.WALProcedureStore: Rolled > new > > Procedure Store WAL, id=5 > > 2018-08-14 17:25:03,190 INFO [Thread-14] procedure2.ProcedureExecutor: > > Recovered WALProcedureStore lease in 42msec > > 2018-08-14 17:25:03,224 INFO [Thread-14] procedure2.ProcedureExecutor: > > Loaded WALProcedureStore in 33msec > > 2018-08-14 17:25:03,224 INFO [Thread-14] > > procedure2.RemoteProcedureDispatcher: Instantiated, coreThreads=128 > > (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 > > 2018-08-14 17:25:03,261 WARN [Thread-14] master.ServerManager: Expiration > > of hbase-116,16020,1534237430655 but server not online > > 2018-08-14 17:25:03,261 INFO [Thread-14] master.ServerManager: Processing > > expiration of hbase-116,16020,1534237430655 on > hbase-114,16000,1534238700547 > > 2018-08-14 17:25:03,481 WARN [Thread-14] master.ServerManager: Expiration > > of hbase-115,16020,1534237425729 but server not online > > 2018-08-14 17:25:03,481 INFO [Thread-14] master.ServerManager: Processing > > expiration of hbase-115,16020,1534237425729 on > hbase-114,16000,1534238700547 > > 2018-08-14 17:25:03,622 INFO [Thread-14] balancer.BaseLoadBalancer: > > slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false > > 2018-08-14 17:25:03,629 INFO [Thread-14] balancer.StochasticLoadBalancer: > > Loaded config; maxSteps=1000000, stepsPerRegion=800, > maxRunningTime=30000, > > isByTable=false, etc. > > 2018-08-14 17:25:03,669 INFO [Thread-14] master.HMaster: Active/primary > > master=hbase-114,16000,1534238700547, sessionid=0x10004dac1970000, > setting > > cluster-up flag (Was=false) > > 2018-08-14 17:25:03,771 INFO [PEWorker-4] procedure.ServerCrashProcedure: > > Start pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure > > server=hbase-115,16020,1534237425729, splitWal=true, meta=false > > 2018-08-14 17:25:03,772 INFO [Thread-14] > procedure2.TimeoutExecutorThread: > > ADDED pid=-1, state=WAITING_TIMEOUT; > > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; > > timeout=30000, timestamp=1534238733772 > > 2018-08-14 17:25:03,774 INFO [PEWorker-3] procedure.ServerCrashProcedure: > > Start pid=11, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure > > server=hbase-116,16020,1534237430655, splitWal=true, meta=false > > 2018-08-14 17:25:03,775 INFO [Thread-14] cleaner.CleanerChore: Cleaner > > pool size is 1 > > 2018-08-14 17:25:03,776 INFO [Thread-14] zookeeper.RecoverableZooKeeper: > > Process identifier=replicationLogCleaner connecting to ZooKeeper > ensemble= > > 192.168.101.114:2181 > > 2018-08-14 17:25:03,776 INFO [Thread-14] zookeeper.ZooKeeper: Initiating > > client connection, connectString=192.168.101.114:2181 > > sessionTimeout=90000 > > watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@46bb7405 > > 2018-08-14 17:25:03,777 INFO [Thread-14-SendThread(192.168.101.114:2181 > )] > > zookeeper.ClientCnxn: Opening socket connection to server > > 192.168.101.114/192.168.101.114:2181. Will not attempt to authenticate > > using SASL (unknown error) > > 2018-08-14 17:25:03,777 INFO [Thread-14-SendThread(192.168.101.114:2181 > )] > > zookeeper.ClientCnxn: Socket connection established to > > 192.168.101.114/192.168.101.114:2181, initiating session > > 2018-08-14 17:25:03,777 INFO [Thread-14] cleaner.LogCleaner: Creating > > OldWALs cleaners with size=2 > > 2018-08-14 17:25:03,780 INFO [Thread-14-SendThread(192.168.101.114:2181 > )] > > zookeeper.ClientCnxn: Session establishment complete on server > > 192.168.101.114/192.168.101.114:2181, sessionid = 0x10004dac1970006, > > negotiated timeout = 40000 > > 2018-08-14 17:25:03,967 INFO > > [RpcServer.default.FPBQ.Fifo.handler=28,queue=1,port=16000] > > master.ServerManager: Registering > regionserver=hbase-116,16020,1534238701517 > > 2018-08-14 17:25:03,967 INFO > > [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000] > > master.ServerManager: Registering > regionserver=hbase-115,16020,1534238702258 > > 2018-08-14 17:25:04,022 INFO [RegionServerTracker-0] > > master.RegionServerTracker: RegionServer ephemeral node created, adding > > [hbase-116,16020,1534238701517] > > 2018-08-14 17:25:04,023 INFO [RegionServerTracker-0] > > master.RegionServerTracker: RegionServer ephemeral node created, adding > > [hbase-115,16020,1534238702258] > > 2018-08-14 17:25:33,877 INFO [WALProcedureStoreSyncThread] > > wal.ProcedureWALFile: Archiving hdfs:// > > 192.168.101.114:9000/hbase/MasterProcWALs/pv2-00000000000000000004.log > to > > hdfs://192.168.101.114:9000/hbase/oldWALs/pv2-00000000000000000004.log > > 2018-08-14 > > < > http://192.168.101.114:9000/hbase/oldWALs/pv2-00000000000000000004.log2018-08-14 > > > > 17:26:59,875 WARN [qtp1304765785-87] servlet.ServletHandler: > /master-status > > org.apache.hadoop.hbase.PleaseHoldException: Master is initializing > > at > > > org.apache.hadoop.hbase.master.HMaster.isInMaintenanceMode(HMaster.java:2890) > > at > > > org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:277) > > at > > > org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:395) > > at > > > org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:386) > > at > > > org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:81) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) > > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) > > at > > > org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:112) > > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) > > at > > > org.apache.hadoop.hbase.http.ClickjackingPreventionFilter.doFilter(ClickjackingPreventionFilter.java:48) > > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) > > at > > > org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1374) > > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) > > at > > > org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49) > > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) > > at > > > org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49) > > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) > > at > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > > at > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > > at > > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > > at > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > > at > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > > at > > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > > at > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > > at > > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > > at > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > > at org.eclipse.jetty.server.Server.handle(Server.java:534) > > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > > at > > > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > > at org.eclipse.jetty.io > > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) > > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) > > at org.eclipse.jetty.io > > .SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > > at > > > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > > at > > > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > > at > > > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > > at java.lang.Thread.run(Thread.java:748) >
