I didn't create ` hbase` user, but because I installed the hadoop-client
from HortonWorks's apt-repo, it created the `hdfs` user, so I thought I
might just use that.
What is really interesting with the issue I am seeing is that there is
no info in the logs besides the following in the .out file:
Thu Dec 17 02:37:26 UTC 2015 Starting master on hmaster2
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 23668
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1048576
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Is there anyway I can make it show more information to debug the system?
On 17/12/2015 3:15 PM, Ted Yu wrote:
Interesting, looks like you launched hbase daemon as user hdfs.
Don't you have user 'hbase' ?
Below is an example of region server process (non-docker). You can see
that HRegionServer was run by bin/java
hbase 13878 0.2 4.3 3963016 703116 ? Sl Dec15 5:26
/usr/jdk64/jdk1.8.0_60/bin/java -Dproc_regionserver
-XX:OnOutOfMemoryError=kill -9 %p -Dhdp.version=2.3.4.0-3471
-XX:+UseConcMarkSweepGC -XX:ErrorFile=/grid/0/log/hbase/hs_err_pid%p.log
-Djava.security.auth.login.config=/usr/hdp/current/hbase-regionserver/conf/hbase_client_jaas.conf
-Djava.io.tmpdir=/tmp -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:/grid/0/log/hbase/gc.log-201512152150
-Xmn408m -XX:CMSInitiatingOccupancyFraction=70 -Xms2048m -Xmx2048m
-Djava.security.auth.login.config=/usr/hdp/current/hbase-regionserver/conf/hbase_regionserver_jaas.conf
-Dhbase.log.dir=/grid/0/log/hbase
-Dhbase.log.file=hbase-hbase-regionserver-ted-hbase-devenv-4.log
-Dhbase.home.dir=/usr/hdp/current/hbase-regionserver/bin/..
-Dhbase.id.str=hbase -Dhbase.root.logger=INFO,RFA
-Djava.library.path=:/usr/hdp/2.3.4.0-3471/hadoop/lib/native/Linux-amd64-64:/grid/0/hdp/2.3.4.0-3471/hadoop/lib/native
-Dhbase.security.logger=INFO,RFAS
org.apache.hadoop.hbase.regionserver.HRegionServer start
FYI
On Wed, Dec 16, 2015 at 8:05 PM, F21 <[email protected]> wrote:
Thanks.
Here are the results of jstack:
$ ps aux | grep master
root 1 0.0 0.0 18140 3040 ? Ss 02:37 0:01 bash
/opt/hbase/bin/run-master.sh
hdfs 130 0.0 0.0 9644 2716 ? S 02:40 0:00 bash
/opt/hbase/bin/hbase-daemon.sh --config /opt/hbase/conf foreground_start
master
hdfs 144 0.0 0.0 9640 2720 ? S 02:40 0:00 bash
/opt/hbase/bin/../bin/hbase --config /opt/hbase/conf master start
hdfs 155 0.0 0.0 9640 1900 ? S 02:40 0:00 bash
/opt/hbase/bin/../bin/hbase --config /opt/hbase/conf master start
root 5195 0.0 0.0 8868 776 ? S+ 04:02 0:00 grep
--color=auto master
$ jps
5321 Jps
$ jstack 130
130: Unable to open socket file: target process not responding or HotSpot
VM not loaded
The -F option can be used when the target process is not responding
$ jstack 144
144: Unable to open socket file: target process not responding or HotSpot
VM not loaded
The -F option can be used when the target process is not responding
$ jstack 155
155: Unable to open socket file: target process not responding or HotSpot
VM not loaded
The -F option can be used when the target process is not responding
$ jstack -F 130
Attaching to process ID 130, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException:
cannot open binary file
sun.jvm.hotspot.debugger.DebuggerException:
sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:278)
at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:671)
at
sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611)
at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337)
at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304)
at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
at sun.tools.jstack.JStack.main(JStack.java:106)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: cannot open binary
file
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native
Method)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$100(LinuxDebuggerLocal.java:62)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1AttachTask.doit(LinuxDebuggerLocal.java:269)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:138)
$ jstack -F 144
Attaching to process ID 144, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException:
cannot open binary file
sun.jvm.hotspot.debugger.DebuggerException:
sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:278)
at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:671)
at
sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611)
at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337)
at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304)
at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
at sun.tools.jstack.JStack.main(JStack.java:106)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: cannot open binary
file
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native
Method)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$100(LinuxDebuggerLocal.java:62)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1AttachTask.doit(LinuxDebuggerLocal.java:269)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:138)
$ jstack -F 155
Attaching to process ID 155, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException:
cannot open binary file
sun.jvm.hotspot.debugger.DebuggerException:
sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:278)
at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:671)
at
sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611)
at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337)
at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304)
at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
at sun.tools.jstack.JStack.main(JStack.java:106)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: cannot open binary
file
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native
Method)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$100(LinuxDebuggerLocal.java:62)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1AttachTask.doit(LinuxDebuggerLocal.java:269)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:138)
On 17/12/2015 3:01 PM, Ted Yu wrote:
ps aux | grep aster
