Problem disappeared. I haven't done any particular test, but no more CPU
at 100%.
Here the 2 logs and 2 stack traces.
http://pastebin.com/FVWP56b6
http://pastebin.com/gXnnwriw
http://pastebin.com/GFhmg4nu
http://pastebin.com/QdreZLXF
I also tried to use stack from jdk1.8 to get a stack trace of hbase
running on 1.7 but of course it didn't work, also because of 32 bit
(1.7) vs 64 bit (1.8).
At least now Google will help all the people with the same issue in the
future.
Thanks for the help and let me know if you need more info.
Olaf
On 14/11/2016 17:38, Ted Yu wrote:
Olaf:
After switching away from OmniOS Java, did you still encounter similar
problem ?
If so, mind updating the thread with log / stack trace ?
Thanks
On Sun, Nov 13, 2016 at 8:40 AM, Olaf Marzocchi <[email protected]> wrote:
I checked and they are located in the same folder.
I found that java and jstack come from two different packages (java from
java and jstack from jdk), but they are both preinstalled with OmniOS,
maintained by OmniTI and compiled at the same time as the operating system.
They should be 100% compatible.
I will ask the OmniOS mailing list.
Olaf
On 13/11/2016 17:28, Ted Yu wrote:
bq. Caused by: java.lang.UnsatisfiedLinkError: no saproc in
java.library.path
Was the jstack of version 1.7.0_101 ?
I used google and found:
http://umchee.blogspot.com/p/i-recently-needed-to-investigate.html
but it was about Java 1.5
You can use this command to find the commandline for hbase process
(replace 16059
with the new process Id):
ps aux | grep 16059
Then use the jstack from same path as java to capture stack trace:
Thanks
On Sun, Nov 13, 2016 at 8:16 AM, Olaf Marzocchi <[email protected]>
wrote:
I'm not 100% sure about the stack trace (not a developer myself), but I
searched online and used jstack:
http://pastebin.com/AT6pScBv
Additional info: after turning on DEBUG, the java process did not
immediately jump to 100% cpu as it used to do, it ramped up and it took 3
minutes to get there. It may however be a coincidence.
Hope this can help pinpointing the issue.
Olaf
On 13/11/2016 01:18, Ted Yu wrote:
Can you turn on DEBUG logging ?
In log4j.properties :
log4j.logger.org.apache.hadoop.hbase=DEBUG
Since this is reproducible, can you capture one or two stack traces
before
killing the server ?
Thanks
On Sat, Nov 12, 2016 at 3:09 PM, Olaf Marzocchi <[email protected]>
wrote:
Hello,
I tested HBase with OpenTSDB some time ago on OS X locally and the
quickstart guide worked perfectly, merged with
http://opentsdb.net/setup-hbase.html where necessary.
I tried yesterday to do the same on my OmniOS/illumos server and I
encountered issues with HBase.
First of all I set "JAVA_HOME=/usr/java" in "conf/hbase-env.sh".
$ /usr/java/bin/java -version
openjdk version "1.7.0_101"
OpenJDK Runtime Environment (build 1.7.0_101-b00)
OpenJDK Server VM (build 24.101-b00, mixed mode)
I edited also "hbase-site.xml":
<configuration>
<property>
<name>hbase.rootdir</name>
<value>file:///export/home/olaf/hbase</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/export/home/olaf/zookeeper</value>
</property>
</configuration>
I then launched "bin/start-hbase.sh" and it quits apparently normally.
A
directory "/export/home/olaf/zookeeper" appeared, but no
/export/home/olaf/hbase.
$ ls -l /export/home/olaf/
drwxrwxr-x 3 olaf olaf 3 Nov 12 01:16 zookeeper
I also checked with prstat ("top") and I see that java is taking a full
CPU. After some minutes it's still crunching something.
If I launch "./bin/hbase shell" it never gets to "hbase(main):001:0>".
At this point the only way is to kill -9 both java processes: the stop
script doesn't work, it outputs dots forever.
I uploaded the two log files (.log and .out) to Pastebin:
http://pastebin.com/aFMXmWYU
I'm not sure where to begin with the debugging, since I did no
customisation at all and I don't see any error logged.
I would be grateful for some help.
Thanks
Olaf
