Timothy Stewart created AMQ-5228:
------------------------------------
Summary: java.lang.NoClassDefFoundError:
org/fusesource/leveldbjni/internal/JniDB error during cleanup
Key: AMQ-5228
URL: https://issues.apache.org/jira/browse/AMQ-5228
Project: ActiveMQ
Issue Type: Bug
Components: activemq-leveldb-store
Affects Versions: 5.10.0, 5.9.1
Environment: Linux 2.6.32-279.5.2.el6.x86_64 #1 SMP Thu Aug 23
12:05:59 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
JDK 1.7.0_51
Apache Karaf 2.3.5 with activemq-osgi, activemq-blueprint, activemq-client,
activemq-webconsole, activemq-camel, activemq features installed. Using
version 5.9.1 (and tried 5.10.0)
Reporter: Timothy Stewart
In our production environment, our storage folder runs out of disk space every
few days (75 Gb). Restarting the container addresses the issue, it cleans it
up. I noticed a stack trace coming on system.err in the wrapper.log (Karaf
starts with a wrapper service). It may or may not be related to our disk space
issue:
INFO | jvm 1 | 2014/06/13 03:22:36 | Exception in thread "Thread-117"
java.lang.NoClassDefFoundError: org/fusesource/leveldbjni/internal/JniDB
INFO | jvm 1 | 2014/06/13 03:22:36 | at
org.apache.activemq.leveldb.LevelDBClient$RichDB.compact(LevelDBClient.scala:377)
INFO | jvm 1 | 2014/06/13 03:22:36 | at
org.apache.activemq.leveldb.LevelDBClient.gc(LevelDBClient.scala:1647)
INFO | jvm 1 | 2014/06/13 03:22:36 | at
org.apache.activemq.leveldb.DBManager$$anonfun$pollGc$1$$anonfun$apply$mcV$sp$2.apply$mcV$sp(DBManager.scala:648)
INFO | jvm 1 | 2014/06/13 03:22:36 | at
org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
INFO | jvm 1 | 2014/06/13 03:22:36 | at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
INFO | jvm 1 | 2014/06/13 03:22:36 | at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
INFO | jvm 1 | 2014/06/13 03:22:36 | at
java.lang.Thread.run(Thread.java:744)
INFO | jvm 1 | 2014/06/13 03:22:36 | Caused by:
java.lang.ClassNotFoundException: org.fusesource.leveldbjni.internal.JniDB not
found by org.apache.activemq.activemq-osgi [105]
INFO | jvm 1 | 2014/06/13 03:22:36 | at
org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1460)
INFO | jvm 1 | 2014/06/13 03:22:36 | at
org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
INFO | jvm 1 | 2014/06/13 03:22:36 | at
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
INFO | jvm 1 | 2014/06/13 03:22:36 | at
java.lang.ClassLoader.loadClass(ClassLoader.java:358)
INFO | jvm 1 | 2014/06/13 03:22:36 | ... 7 more
I see the same problem in our dev environment but could not replicate it. I
was finally able to replicate by using the hawtio console to execute the
compact operation. Everytime I do this, the same stack trace outputs and the
operation cycles endlessly (well as long as I've waited anyhow). The 5.10.0
stack trace when I execute the operation is:
INFO | jvm 2 | 2014/06/14 21:00:44 | Exception in thread "Thread-106"
java.lang.NoClassDefFoundError: org/fusesource/leveldbjni/internal/JniDB
INFO | jvm 2 | 2014/06/14 21:00:44 | at
org.apache.activemq.leveldb.LevelDBClient$RichDB.compact(LevelDBClient.scala:378)
INFO | jvm 2 | 2014/06/14 21:00:44 | at
org.apache.activemq.leveldb.LevelDBClient.gc(LevelDBClient.scala:1654)
INFO | jvm 2 | 2014/06/14 21:00:44 | at
org.apache.activemq.leveldb.LevelDBStoreView$$anonfun$compact$1.apply$mcV$sp(LevelDBStore.scala:126)
INFO | jvm 2 | 2014/06/14 21:00:44 | at
org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)
INFO | jvm 2 | 2014/06/14 21:00:44 | at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
INFO | jvm 2 | 2014/06/14 21:00:44 | at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
INFO | jvm 2 | 2014/06/14 21:00:44 | at
java.lang.Thread.run(Thread.java:744)
INFO | jvm 2 | 2014/06/14 21:00:44 | Caused by:
java.lang.ClassNotFoundException: org.fusesource.leveldbjni.internal.JniDB not
found by org.apache.activemq.activemq-osgi [390]
INFO | jvm 2 | 2014/06/14 21:00:44 | at
org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1460)
INFO | jvm 2 | 2014/06/14 21:00:44 | at
org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
INFO | jvm 2 | 2014/06/14 21:00:44 | at
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
INFO | jvm 2 | 2014/06/14 21:00:44 | at
java.lang.ClassLoader.loadClass(ClassLoader.java:358)
INFO | jvm 2 | 2014/06/14 21:00:44 | ... 7 more
I started looking at the activemq-osgi and activemq-leveldb project poms, and
noticed that the leveldbjni projects are commented out with a note that we want
to focus on hardening the pure java version. With that in mind, I switched the
indexFactory on my leveldb config to
indexFactory="org.iq80.leveldb.impl.Iq80DBFactory". I restarted, and tried the
compact operation again, but got the same stacktrace.
My leveldb config now looks like:
<amq:persistenceAdapter>
<amq:levelDB directory="[[karaf.storage]]/activemq/default/leveldb"
logSize="107374182" logCompression="snappy"
indexFactory="org.iq80.leveldb.impl.Iq80DBFactory" />
</amq:persistenceAdapter>
The logCompression and the indexFactory were both added as an attempt to
mitigate the problem, but the issue exists either way. I also saw a note on
the replicated levedb store about scheduled jobs, which are used a lot, but
when I looked at those folders in our storage folder it seems to use a
completely different kahadb store, so I threw that out as a reason.
I did attempt to add leveldbjni-linux64 or leveldbjni-all to the osgi
deployment, but it does not export the internal package which activemq-osgi
wants to import. I could build a new package of these, or rebuild
activemq-osgi with the commented dependencies removed.. perhaps that will work,
but it seems that as it is with the dependencies commented out that there is
something broken within the compact routine.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
