On 12/13/2013 01:02 PM, Andy Boothe [WCG] wrote:
Timothy,
Thank you for the quick response!
I had read that, but didn't see anything useful in it. Having re-read it,
I realize that my DLQs (which contain 100k messages) were a likely suspect
in keeping the log files "live." I cleared the DLQs, and ActiveMQ
diligently deleted most of the log files a couple minutes later:
$ cd apache-activemq-5.8.0/data/kahadb/
$ ls -lht | head
$ ls -lht | head
total 4.9G
-rw-rw-r-- 1 ubuntu ubuntu 32M Dec 13 18:00 db-1564.log
-rw-rw-r-- 1 ubuntu ubuntu 36M Dec 13 18:00 db.data
-rw-rw-r-- 1 ubuntu ubuntu 2.6M Dec 13 18:00 db.redo
-rw-rw-r-- 1 ubuntu ubuntu 33M Dec 13 17:44 db-1563.log
-rw-rw-r-- 1 ubuntu ubuntu 33M Dec 13 16:14 db-1562.log
-rw-rw-r-- 1 ubuntu ubuntu 33M Dec 13 14:47 db-1561.log
-rw-rw-r-- 1 ubuntu ubuntu 33M Dec 13 13:28 db-1560.log
-rw-rw-r-- 1 ubuntu ubuntu 33M Dec 13 12:09 db-1559.log
-rw-rw-r-- 1 ubuntu ubuntu 33M Dec 13 10:55 db-1558.log
Thank you for the hint, Timothy!
Andy Boothe
Director of Data Engineering, Analytics
WCG
60 Francisco St, San Francisco, 94133
direct 512-961-3993 cell 979-574-1089
twitter @sigpwned <http://twitter.com/sigpwned> linkedin aboothe
<http://www.linkedin.com/in/aboothe>
github sigpwned <https://github.com/sigpwned> stack overflow
sigpwned
<http://stackoverflow.com/users/2103602/sigpwned>
€ € € € € € € Go. Ahead.
ŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠ
On 12/13/13 10:59 AM, "Timothy Bish" <tabish...@gmail.com> wrote:
On 12/13/2013 11:27 AM, Andy Boothe [WCG] wrote:
Hello,
I have an ActiveMQ instance that has been running nicely for a couple
of months, but doesn't seem to clean up its KahaDB logs like I expect
it to. (More specifically, I expect to have like 1 log file, but in
reality I have ~1500 log files with 49G of data.)
ActiveMQ has processed about 12M messages over the past couple of
months, and has roughly 106K messages currently pending (in DLQs):
Given the number of messages currently pending, I would not expect the
KahaDB logs to be very big. In reality, they're about 49G:
$ cd apache-activemq-5.8.0/data/kahadb/
$ ls -lht | head
total 49G
-rw-rw-r-- 1 ubuntu ubuntu 32M Dec 13 16:00 db-1562.log
-rw-rw-r-- 1 ubuntu ubuntu 36M Dec 13 16:00 db.data
-rw-rw-r-- 1 ubuntu ubuntu 173K Dec 13 16:00 db.redo
-rw-rw-r-- 1 ubuntu ubuntu 33M Dec 13 14:47 db-1561.log
-rw-rw-r-- 1 ubuntu ubuntu 33M Dec 13 13:28 db-1560.log
-rw-rw-r-- 1 ubuntu ubuntu 33M Dec 13 12:09 db-1559.log
-rw-rw-r-- 1 ubuntu ubuntu 33M Dec 13 10:55 db-1558.log
-rw-rw-r-- 1 ubuntu ubuntu 33M Dec 13 09:48 db-1557.log
-rw-rw-r-- 1 ubuntu ubuntu 33M Dec 13 08:40 db-1556.log
Based on the maximum log ID, almost none of the log files appear to
have been cleaned up:
$ ls db-*.log | wc
1540 1540 17421
I'm using ActiveMQ 5.8.0. The only change I've made to the vanilla
configuration is Java heap size, which shouldn't affect whether or not
DB log files get cleaned up (AFAIK). I'm "connecting" to ActiveMQ with
Camel. Obviously from the number of queues, I have several different
workflows in flight, but here's a representative example of how I'm
using ActiveMQ in my Java code:
from("activemq:statuslinks")
.bean(new LinkExtract())
.split(body(String.class).tokenize("\n"))
.filter(new LinkHandled(jdbcConnectionString))
.to("activemq:linkunwind");
The only problems I'm seeing in my activemq.log file appear to be
"normal." I get rashes of transport exceptions every few weeks, which
I have associated (rightly or wrongly) with hitting the web admin
interface. Given the infrequency and nature of the errors, I don't
think they explain why the logs wouldn't be deleted. In case it's
useful, though, here are a couple of representative lines from the log
file:
2013-12-13 09:46:52,269 | WARN | Transport Connection to:
tcp://127.0.0.1:39604 failed:
org.apache.activemq.transport.InactivityIOException: Channel was
inactive for too (>30000) long: tcp://127.0.0.1:39604 |
org.apache.activemq.broker.TransportConnection.Transport |
ActiveMQ InactivityMonitor Worker
2013-12-13 14:43:41,215 | WARN | Transport Connection to:
tcp://127.0.0.1:35711 failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport |
ActiveMQ Transport: tcp:///127.0.0.1:35711@61616
Given the use case ‹ very simple message processing using Camel ‹ and
ActiveMQ's vanilla configuration, I would expect the log files to be
tiny. Clearly, though they're not. Does anyone have any idea why those
files aren't getting deleted? I'm sure I'm doing (or not doing)
something very silly, but after pouring over the logs and
documentation, I just can't figure out what I'm doing wrong. Does
anyone have an idea why my logs wouldn't be getting cleaned up?
Thanks everyone in advance for your attention and help! And thank you
ActiveMQ devs for such a wonderful product! :)
*Andy Boothe*
****
Have you referenced this documentation:
http://cp.mcafee.com/d/FZsS93gs839J5BVMQsLInjhdTdFEIzDxRQTxNJd5AsYZtZCVJd5
AsYeKCYYrjhp7ff8T7c6Z8lHcDUacm8CaNfgVsSoIhclyuxOVJxd1AxGr_nVdBVVBPHTbETpd7
fc9L9CzBXBHFShhlLtzBgY-F6lK1FJ4SMrLOtXTLuZXTdTdw0ZJehDRhrJmSNf-00Uo9ByW5iV
niuE42VZi9tSh_QbNZnpgbWXfUBX7_I6RK1NJKDNEnzYSUMbIzCnXBZ8lHcDVNyN4Nm9W7bwHr
jApZkmXlJIj_w09Is--yr1vF6y02J9W739EwfGhd43Iz_Eq88lCq82fPDa14Qgmf_ouwq86Rzh
0nbCy01ipoArKruq0ctEi81xx
--
Tim Bish
Sr Software Engineer | RedHat Inc.
tim.b...@redhat.com |
http://cp.mcafee.com/d/k-Kr6xEp3xESyOYUqenSbFECXCQQmhPMWWrMUSCyOeuuK-PsSCy
Oeu7njuudFEIzDDArzC3uAaRCjY56b4j5oDEsKrcm8CaNfgVsSMCwOgRd_HYCOYYOVRXBQrICz
DC4TAPhOZORQX8EGTKNOEuvkzaT0QSCrodTVeZXTLuZXCXCODeJQ-BmUS_BPrtfzgL7VJNwnp7
cLTbWgHmpfPz5y9yIjQen1mSD8PWEJSHroD_00joVZZ4S2_id405qjQe6jh0vkyq87p7_gQggH
cQg4vDek29EwIv-MZ0QgdH6y0Knd402AON8TsS5aMcvwY |
http://cp.mcafee.com/d/2DRPoAd6QmnD3hO-Ntd4TsSCyOeu7nju76QQmhPPRTSrCQQmhPM
WWrPNJd5AsYYzssMrQxmIOvwENoyoH4Z3BPpyN4Nm9W7bCS4Q6i6FLZvASnDCneLsKztAQsYMC
YCqenKmKDp55mZSel3PWApmU6CSjr1K_9TLuZXTLsTsSkWfE4ZDOVJKDNEnzYSUMbIzCnXBZ8l
HcDVNyN4Nm9W7bwHrjApZkmXlJIj_w09Is--yr1vF6y02J9W739EwfGhd43Iz_Eq88lCq82fPD
a14Qgmf_ouwq86Rzh0nbCy01ipoArKrZzCB4t6l
skype: tabish121 | twitter: @tabish121
blog:
http://cp.mcafee.com/d/FZsS73hJ5BVMQsLInjhdTdFEIzDxRQTxNJd5AsYZtZCVJd5AsYe
KCYYrjhp7ff8T7c6Z8lHcDUacm8CaNfgVsSoIhclyuxOVJxd1AxGr_nVdBVVBPHTbETpd7fc9L
9CzBXBHFShhlLtzBgY-F6lK1FJwSMrLOtXTLuZXTdTdw0VOl8iEjUwvkzhqI-nMSTjUQbN-rso
5ShPbZO-AaRCjYUNoyoH4Z3BMlJFOc-GbtGSS9_M04SevvhdwLQzh01mAZ3xAQg7R8Cy1Sh_Qd
44aPd417VPB0yq8b7_Ifgd43qNEwbBPh00FcIidTdKLPH3zP8
Without seeing the log messages from that article all we can say is, you
still have some messages in there.
--
Tim Bish
Sr Software Engineer | RedHat Inc.
tim.b...@redhat.com | www.fusesource.com | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/
