Hi Rafael,
I’m just an ActiveMQ classic user.
I will report the behavior of ActiveMQ I experienced.
As far as I know the ActiveMQ classic journal is composed of append only
files.
Each file should be by default 32MB in size (this can be changed, I think
is controlled by journalLogFileSize)
A journal file gets released only when ALL the messages within are consumed.
Furthermore, messages form different queues are appended to and mixed
within the same file until the maximum size gets reached.
As consequence if you have unprocessed messages (DLQ or offline consumers)
the used storage will increase, usually in the same order of magnitude as
the total traffic served by the broker: a small percentage of unconsumed
messages will prevent journal file cleanup.
For example an unprocessed 1KByte message will retain 32MB on disk.
As far as I know there is no building functionality to perform journal
compaction.
In our case the developed internally a small application to offload
unprocessed messages to an external database/ requeue DLQ messages to the
same DLQ in order to free storage.
In my experience persistent Topics are quite hard to monitor, if the list
of subscribers is not dynamic (known when application starts) you could
think to use VirtualTopics so that you will get back to a normal queue uses
case for each consumer.
Regards,
Marc
Il giorno gio 29 set 2022 alle ore 23:06 Rafael Pimenta <
rafa.spime...@gmail.com> ha scritto:
> Hi Dears,
> I'm using ActiveMQ 5.15.9 and after about one month our customers started
> to complain about the Kahadb folder increase until reaching its limit and
> crash the system. Then they delete the folder to recover the system, but we
> are losing data in this process.
> We activate the kahadb log and seems that some old messages are getting
> stuck for some reason and do not allow the ActiveMQ to delete the db-x.log
> files:
>
> *2022-09-29 17:39:51,555 | TRACE | ackMessageFileMap:* {244=[244, 243],
> 245=[123], 246=[244, 246], 247=[246, 247], 248=[123], 249=[249, 247],
> 250=[163, 211, 249, 250, 234, 237], 251=[250, 243, 251, 247], 252=[140,
> 108, 97], 253=[249, 253, 251], 254=[253, 254], 255=[140, 141], 256=[256,
> 250, 251, 253, 254], 257=[256, 257], 258=[141], 259=[257, 259], 260=[259,
> 260, 254], 261=[140], 262=[260, 262], 263=[262, 263], 264=[264, 262, 254,
> 263], 265=[264, 265, 253], 266=[265, 249, 266, 250, 253], 267=[159],
> 268=[266, 268, 253, 79], 269=[259, 268, 269, 262, 254, 263], 270=[159],
> 271=[269, 271], 272=[272, 265, 266, 271], 273=[163], 274=[272, 274, 271],
> 275=[274, 275], 276=[163], 277=[265, 274, 266, 275, 277], 278=[277, 278],
> 279=[163], 280=[280, 278, 266], 281=[280, 281], 282=[281, 282], 283=[159],
> 284=[284, 282], 285=[284, 285], 286=[174], 287=[285, 287], 288=[288, 287],
> 289=[174], 290=[288, 290], 291=[290, 291], 292=[163], 293=[293, 291],
> 294=[293, 294], 295=[294, 295], 296=[296, 295], 297=[296, 297], 298=[174,
> 159], 299=[297, 299], 300=[300, 299], 301=[180], 302=[300, 302], 303=[302,
> 303], 304=[180], 305=[305, 303], 306=[305, 306], 307=[182], 308=[308, 306,
> 287], 309=[308, 309], 310=[182], 311=[309, 311], 312=[312, 311], 313=[312,
> 313], 314=[313, 314], 315=[284, 314, 315], 316=[174], 317=[317, 315],
> 318=[317, 318], 319=[198], 320=[320, 318], 321=[320, 321], 322=[199],
> 323=[321, 323], 324=[324, 323], 325=[199], 326=[272, 274, 324, 326, 278,
> 294, 296], 327=[326, 327], 328=[328, 327], 329=[328, 329], 330=[329, 330],
> 331=[208], 332=[332, 330], 333=[332, 333], 334=[208], 335=[250, 251, 333,
> 253, 335], 336=[336, 335], 337=[336, 337, 211], 338=[20], 339=[337, 339],
> 342=[337, 342, 339], 343=[342, 343], 344=[20, 21], 345=[345, 343],
> 346=[345, 346], 347=[337, 345, 346, 347], 348=[348, 345, 347], 349=[348,
> 337, 349], 350=[349, 350], 351=[272, 337, 274, 294, 296, 350, 302, 351,
> 271], 352=[352, 337, 351], 353=[352, 353, 337, 285], 354=[21], 355=[353,
> 355], 356=[337, 274, 355, 356, 294, 299, 302], 357=[24], 358=[356, 358],
> 359=[358, 359], 360=[256, 272, 337, 274, 294, 359, 263, 360, 296, 265,
> 266], 361=[337, 274, 263, 359, 360, 361, 265, 253, 302], 362=[361, 362],
> 363=[360, 362, 363], 364=[364, 361, 363], 365=[24], 366=[364, 366],
> 367=[337, 345, 346, 366, 367, 351], 368=[21], 369=[369, 367], 370=[369,
> 370], 371=[28], 372=[337, 369, 370, 372, 345, 346, 351], 373=[372, 373],
> 374=[373, 374], 375=[374, 375], 376=[376, 345, 375, 351], 377=[376, 377],
> 378=[369, 374, 376, 377, 345, 378, 351], 379=[28], 380=[380, 378],
> 381=[380, 381], 382=[28], 383=[381, 383], 384=[384, 381, 383], 385=[384,
> 385, 366], 386=[352, 384, 385, 386, 350, 351, 367, 383], 387=[337, 385,
> 386, 387, 345, 378, 348, 366, 351], 388=[388, 387], 389=[352, 384, 388,
> 389, 351, 383], 390=[376, 369, 380, 389, 390, 374], 391=[390, 391],
> 392=[34], 393=[393, 387, 389, 390, 391], 394=[393, 394], 395=[34],
> 396=[385, 259, 387, 390, 296, 360, 265, 394, 266, 363, 396, 366, 337, 274,
> 243, 277, 278, 250, 284, 253], 397=[396, 397, 394], 398=[28, 34], 399=[380,
> 397, 374, 399], 400=[400, 384, 390, 362, 380, 396, 350, 399, 351],
> 401=[400, 401], 402=[37], 403=[401, 403], 404=[404, 403], 405=[37, 38],
> 406=[404, 406], 407=[406, 407], 408=[34, 38, 39], 409=[409, 407], 410=[409,
> 410], 411=[409, 410, 411], 412=[40, 39], 413=[413, 411], 414=[401, 409,
> 389, 413, 414], 415=[40], 416=[416, 414], 417=[416, 417], 418=[38],
> 419=[417, 419], 420=[420, 414, 419], 421=[400, 403, 420, 421], 422=[385,
> 419, 420, 421, 422], 423=[387, 419, 404, 420, 421, 422, 423, 394, 414],
> 424=[424, 419, 420, 423], 425=[424, 425], 426=[419, 404, 420, 422, 425,
> 426, 394, 414], 427=[426, 419, 427, 420], 428=[428, 427], 429=[400, 424,
> 403, 428, 429], 430=[48], 431=[419, 403, 404, 420, 424, 429, 414, 431],
> 432=[432, 431], 433=[48, 49], 434=[432, 434], 435=[424, 434, 435], 436=[49,
> 51], 437=[437, 435], 438=[428, 437, 438, 431], 439=[48, 49, 51, 52, 40],
> 440=[438, 440], 441=[440, 441], 442=[52, 53, 21], 443=[438, 441, 443],
> 444=[438, 440, 443, 444], 445=[438, 444, 445], 446=[445, 446], 447=[438,
> 446, 447], 448=[448, 438, 440, 445, 446, 447], 449=[448, 449], 450=[53,
> 54], 451=[449, 451], 452=[337, 451, 419, 452, 404, 420, 438, 394, 414],
> 453=[49, 54, 55, 40], 454=[452, 454], 455=[454, 455], 456=[55, 57],
> 457=[449, 438, 455, 457, 441], 458=[457, 458], 459=[458, 459], 460=[49, 53,
> 57, 58], 461=[437, 438, 459, 461], 462=[438, 458, 459, 461, 462], 463=[58,
> 60], 464=[464, 462], 465=[464, 465, 459], 466=[465, 466], 467=[466, 467],
> 468=[466, 467, 468], 469=[468, 469, 459], 470=[469, 470], 471=[470, 471],
> 472=[471, 472], 473=[472, 473], 474=[471, 472, 473, 474], 475=[474, 475],
> 476=[475, 476], 477=[476, 477], 478=[477, 478], 479=[34, 40, 60, 61],
> 480=[480, 478], 481=[480, 481], 482=[61], 483=[481, 483], 484=[483, 484],
> 485=[64, 40], 486=[484, 486], 487=[486, 487], 488=[487, 488], 489=[64, 66],
> 490=[488, 490], 491=[490, 491], 492=[66, 67], 493=[491, 493], 494=[493,
> 494], 495=[64, 67, 68], 496=[496, 464, 465, 494], 497=[496, 497, 438],
> 498=[497, 498], 499=[68, 70], 500=[465, 497, 498, 500, 438], 501=[500,
> 501], 502=[70, 71], 503=[420, 484, 501, 437, 438, 486, 503, 457, 490, 459,
> 446], 504=[503, 504], 505=[71, 73], 506=[504, 506], 507=[506, 507],
> 508=[507, 508], 509=[508, 509], 510=[509, 510], 511=[510, 511], 512=[512,
> 438, 511], 513=[73], 514=[512, 514], 515=[514, 515], 516=[76], 517=[515,
> 517], 518=[386, 387, 452, 517, 421, 518, 424, 459, 397, 429, 465, 500, 438,
> 503, 506], 519=[352, 337, 437, 518, 438, 519, 345, 381, 511], 520=[519,
> 520], 521=[519, 520, 521], 522=[521, 522], 523=[515, 521, 522, 523],
> 524=[76, 77], 525=[515, 521, 522, 523, 525], 526=[514, 515, 521, 525, 526],
> 527=[77, 79], 528=[528, 526], 529=[528, 529], 530=[79], 531=[529, 531],
> 532=[531, 532, 520], 533=[532, 533], 534=[533, 534], 535=[512, 517, 534,
> 535], 536=[81], 537=[535, 537], 538=[537, 538], 539=[81, 82, 71], 540=[538,
> 540], 541=[540, 541], 542=[82, 83], 543=[535, 541, 543], 544=[544, 543],
> 545=[544, 545], 546=[83], 547=[545, 547], 548=[544, 545, 547, 531, 548,
> 535, 521, 538, 543], 549=[79], 550=[548, 550], 551=[548, 550, 551],
> 552=[97], 553=[551, 553], 554=[337, 531, 438, 535, 553, 554], 555=[97],
> 556=[548, 554, 538, 556], 557=[545, 556, 557], 558=[557, 558], 559=[64, 68,
> 70], 560=[560, 544, 554, 558], 561=[560, 561], 562=[64, 82, 68, 70, 55,
> 105, 57], 563=[561, 563], 564=[563, 564, 537, 558], 565=[564, 565],
> 566=[565, 566, 556, 558], 567=[566, 567], 568=[567, 568], 569=[568, 569],
> 570=[566, 569, 570, 558], 571=[570, 571], 572=[211], 573=[571, 573],
> 574=[573, 574], 575=[211, 214], 576=[576, 574], 577=[576, 544, 577, 561,
> 554, 557], 578=[97, 105], 579=[577, 579, 565, 566], 580=[579, 580],
> 581=[214], 582=[580, 582], 583=[582, 583], 584=[566, 583, 584, 558],
> 585=[108], 586=[566, 584, 586], 587=[108], 588=[583, 586, 588], 589=[211],
> 590=[586, 588, 590], 591=[590, 591], 592=[211], 593=[593, 586, 591],
> 594=[593, 594, 586], 595=[594, 595], 596=[595, 596, 586], 597=[596, 597],
> 598=[111], 599=[596, 597, 599, 586], 600=[597, 599, 600], 601=[597, 599,
> 600, 601], 602=[601, 602, 586, 588], 603=[211], 604=[583, 599, 602, 586,
> 604], 605=[604, 605], 606=[586, 605, 606], 607=[113, 97, 108, 111],
> 608=[608, 596, 599, 586, 606], 609=[608, 609], 610=[113], 611=[609, 611,
> 586, 588], 612=[611, 612, 583, 599, 586], 613=[608, 612, 613, 599, 600],
> 614=[609, 596, 613, 614, 599, 586], 615=[614, 615], 616=[608, 615, 599,
> 616, 600, 586], 617=[615, 616, 617], 618=[611, 613, 617, 618], 619=[618,
> 619], 620=[234], 621=[611, 613, 619, 621], 622=[608, 609, 611, 621, 622],
> 623=[234], 624=[624, 608, 612, 613, 614, 615, 622], 625=[624, 625, 616,
> 621], 626=[234, 237], 627=[625, 627], 628=[624, 625, 627, 628, 615, 619,
> 622], 629=[237], 630=[628, 630], 631=[630, 631], 632=[97, 108, 79],
> 633=[624, 625, 630, 631, 633, 622], 634=[633, 634], 635=[599, 631, 634,
> 635, 621], 636=[97], 637=[635, 637], 638=[619, 637, 638, 622], 639=[243]} |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,557 | TRACE |* not removing data file: 244 as
> contained ack(s) refer to referenced file: [244, 243] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,559 | TRACE |* not removing data file: 245 as
> contained ack(s) refer to referenced file: [123] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,561 | TRACE |* not removing data file: 246 as
> contained ack(s) refer to referenced file: [244, 246] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,563 | TRACE |* not removing data file: 247 as
> contained ack(s) refer to referenced file: [246, 247] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,564 | WARN | *Stealing link for clientId
> TR_VP-018:30246 From Connection Transport Connection to: tcp://
> 172.24.96.125:42412 | org.apache.activemq.broker.region.RegionBroker |
> ActiveMQ Transport: tcp:///172.24.96.125:42414@1883
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
> contained ack(s) refer to referenced file: [123] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
> contained ack(s) refer to referenced file: [249, 247] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
> contained ack(s) refer to referenced file: [163, 211, 249, 250, 234, 237] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
> contained ack(s) refer to referenced file: [250, 243, 251, 247] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
> contained ack(s) refer to referenced file: [123] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
> contained ack(s) refer to referenced file: [249, 247] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
> contained ack(s) refer to referenced file: [163, 211, 249, 250, 234, 237] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
> contained ack(s) refer to referenced file: [250, 243, 251, 247] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 252 as
> contained ack(s) refer to referenced file: [140, 108, 97] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 253 as
> contained ack(s) refer to referenced file: [249, 253, 251] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 254 as
> contained ack(s) refer to referenced file: [253, 254] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
> contained ack(s) refer to referenced file: [140, 141] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker not removing data file: 252 as contained ack(s) refer to
> referenced file: [140, 108, 97] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 253 as
> contained ack(s) refer to referenced file: [249, 253, 251] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 254 as
> contained ack(s) refer to referenced file: [253, 254] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
> contained ack(s) refer to referenced file: [140, 141] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
>
> We also use durable topics as the devices that subscribe to these topics
> and send MQTT messages to others work in a very bad network environment,
> this is the way we found to avoid data loss while offline.
> I think the problem is related to durable topics with offline
> subscriptions, but how can I confirm this assumption, eg which topics? What
> signatures?
>
> I much appreciate any thoughts!
> Thanks in advance
>
