Hi,
I wondered if someone on the Qpid mailing list can throw some light on this
issue we are seeing.
We are using Qpid C++ brokers and clients of version 0.24.
We are using Topic Exchanges.
We create a sender on a particular session, then a message gets sent on that
sender followed by deleting the sender.
This is done a lot of times in a period of time.
What we are seeing is that the memory usage on Qpid is increasing and a lot of
memory allocations sitting on the std::deque and that number increases over time
There are 2 problems we are seeing:
1. That memory seems to be increasing on the std::deque list inside Qpid.
2. When we fail over the broker, there seems to be a lot of messages that
get replayed on the topics to the clients looking for received messages on the
same topic. These are message that have already been received before.
Below is a snipet of what we have seen.
Note that in the second half below, there are allocations which show the number
of bytes@address for time in -Minutes:seconds. These are blocks allocated that
are "alive" so to speak.
Can someone explain why we would be seeing this and perhaps how we can avoid
the "replays".
Thanks
Nitin Shah
58) 22656 total bytes from ::new in 354 separate allocations
/opt/atlas/kit/bin/StartEquipmentMgr (operator new(unsigned long)+0x65)
[0x0x9aa791]
lib/libqpidmessaging.so.2 (__gnu_cxx::new_allocator<void**>::allocate(unsigned
long, void const*)+0x42) [0x0x7f1119659518]
/lib/libqpidmessaging.so.2 (std::_Deque_base<void*, std::allocator<void*>
>::_M_allocate_map(unsigned long)+0x3d) [0x0x7f111965928d]
lib/libqpidmessaging.so.2 (std::_Deque_base<void*, std::allocator<void*>
>::_M_initialize_map(unsigned long)+0x7e) [0x0x7f1119659056]
lib/libqpidmessaging.so.2 (std::_Deque_base<void*, std::allocator<void*>
>::_Deque_base()+0x2b) [0x0x7f1119658bb1]
lib/libqpidmessaging.so.2 (std::deque<void*, std::allocator<void*>
>::deque()+0x1a) [0x0x7f1119658914]
lib/libqpidmessaging.so.2
(boost::ptr_container_detail::reversible_ptr_container<boost::ptr_container_detail::sequence_config<qpid::client::amqp0_10::OutgoingMessage,
std::deque<void*, std::allocator<void*> > >,
boost::heap_clone_allocator>::reversible_ptr_container()+0x1a)
[0x0x7f11196583f8]
/opt/atlas/kit/lib/libqpidmessaging.so.2
(boost::ptr_sequence_adapter<qpid::client::amqp0_10::OutgoingMessage,
std::deque<void*, std::allocator<void*> >,
boost::heap_clone_allocator>::ptr_sequence_adapter()+0x1a) [0x0x7f1119657880]
/opt/atlas/kit/lib/libqpidmessaging.so.2
(boost::ptr_deque<qpid::client::amqp0_10::OutgoingMessage,
boost::heap_clone_allocator, std::allocator<void*> >::ptr_deque()+0x1a)
[0x0x7f1119656220]
/opt/atlas/kit/lib/libqpidmessaging.so.2
(qpid::client::amqp0_10::SenderImpl::SenderImpl(qpid::client::amqp0_10::SessionImpl&,
std::string const&, qpid::messaging::Address const&)+0x13b) [0x0x7f1119654e21]
lib/libqpidmessaging.so.2
(qpid::client::amqp0_10::SessionImpl::createSenderImpl(qpid::messaging::Address
const&)+0x91) [0x0x7f1119646255]
lib/libqpidmessaging.so.2
(qpid::client::amqp0_10::SessionImpl::CreateSender::operator()()+0x2c)
[0x0x7f111964897a]
lib/libqpidmessaging.so.2 (bool
qpid::client::amqp0_10::SessionImpl::execute<qpid::client::amqp0_10::SessionImpl::CreateSender>(qpid::client::amqp0_10::SessionImpl::CreateSender&)+0x2c)
[0x0x7f111964d918]
/opt/atlas/kit/lib/libqpidmessaging.so.2 (qpid::messaging::Sender
qpid::client::amqp0_10::SessionImpl::get1<qpid::client::amqp0_10::SessionImpl::CreateSender,
qpid::messaging::Sender,
qpid::messaging::Address>(qpid::messaging::Address)+0x40) [0x0x7f1119649d98]
lib/libqpidmessaging.so.2
(qpid::client::amqp0_10::SessionImpl::createSender(qpid::messaging::Address
const&)+0x40) [0x0x7f1119646192]
lib/libqpidmessaging.so.2 (qpid::messaging::Session::createSender(std::string
const&)+0x51) [0x0x7f1119614525]
Blocks still allocated...
64@0x7f110c0158f0 -49m:9:10.190 64@0x7f110c0171f0 -49m:8:484.470
64@0x7f110c018190 -57m:5:169.301
64@0x7f110c01a3f0 -58m:8:56.395 64@0x7f110c01b330 -55m:4:604.704
64@0x7f110c01eec0 -54m:9:587.496 64@0x7f110c01f960 -56m:6:845.442
64@0x7f110c021070 -51m:8:198.576 64@0x7f110c0224a0 -53m:8:169.705
64@0x7f110c024740 -53m:8:487.554 64@0x7f110c029280 -58m:7:771.68
64@0x7f110c02d740 -53m:8:730.295 64@0x7f110c03ce40 -53m:9:200.21
64@0x7f110c03e3f0 -44m:7:785.601 64@0x7f110c040d40 -54m:8:738.430
64@0x7f110c041930 -45m:9:364.762 64@0x7f110c044470 -48m:8:177.276
64@0x7f110c048020 -53m:7:785.827 64@0x7f110c049a40 -43m:8:288.672
64@0x7f110c04aaa0 -49m:8:250.625 64@0x7f110c04c860 -11m:8:260.76
64@0x7f110c04fd30 -56m:7:124.826 64@0x7f110c050790 -36m:8:398.188
64@0x7f110c056fd0 -35m:9:326.379 64@0x7f110c057a40 -29m:8:852.362
64@0x7f110c059c90 -25m:6:948.696 64@0x7f110c05acd0 -51m:8:748.650
64@0x7f110c05d6c0 -52m:9:437.52 64@0x7f110c05e050 -50m:7:324.127
64@0x7f110c062470 -52m:8:728.162 64@0x7f110c063f60 -55m:5:598.884
64@0x7f110c063fb0 -51m:9:108.609 64@0x7f110c0685d0 -41m:7:30.685
64@0x7f110c068b30 -43m:8:564.190 64@0x7f110c069cf0 -50m:8:230.432
64@0x7f110c06e640 -47m:9:311.997 64@0x7f110c0708a0 -41m:8:448.186
64@0x7f110c074450 -38m:8:108.632 64@0x7f110c076880 -31m:8:678.348
64@0x7f110c076e60 -20m:7:538.796 64@0x7f110c077b20 -47m:9:671.732
64@0x7f110c079530 -56m:6:443.499 64@0x7f110c079b00 -46m:8:663.976
64@0x7f110c07be00 -54m:8:516.943 64@0x7f110c07ce00 -51m:9:390.781
64@0x7f110c07d9c0 -36m:7:628.346 64@0x7f110c07e250 -48m:9:24.194
64@0x7f110c0804d0 -49m:7:495.720 64@0x7f110c0810f0 -40m:8:97.964
64@0x7f110c084f80 -45m:8:852.273 64@0x7f110c085b20 -46m:9:90.499
64@0x7f110c087900 -55m:4:941.179 64@0x7f110c08a360 -44m:8:218.906
64@0x7f110c08b0d0 -56m:6:70.711 64@0x7f110c0922c0 -54m:9:300.442
64@0x7f110c093b40 -43m:9:404.55 64@0x7f110c094290 -50m:7:630.993
64@0x7f110c097000 -36m:7:229.792 64@0x7f110c09c8c0 -39m:7:575.783
64@0x7f110c09dea0 -34m:7:502.700 64@0x7f110c09e120 -42m:8:478.86
64@0x7f110c09f430 -23m:8:637.169 64@0x7f110c0a0980 -52m:9:730.971
64@0x7f110c0a12f0 -35m:7:131.423 64@0x7f110c0a1740 -38m:6:519.818
64@0x7f110c0a3030 -34m:8:78.349 64@0x7f110c0a3250 -29m:8:430.329
64@0x7f110c0a3d70 -45m:8:334.112 64@0x7f110c0a62f0 -54m:9:21.327
64@0x7f110c0a73c0 -35m:8:11.580 64@0x7f110c0a9090 -48m:9:276.653
64@0x7f110c0adb80 -43m:8:821.318 64@0x7f110c0b18a0 -22m:8:282.649
64@0x7f110c0b3020 -46m:8:329.831 64@0x7f110c0b3460 -37m:8:647.929
64@0x7f110c0b3850 -40m:9:16.811 64@0x7f110c0b6980 -57m:6:905.678
64@0x7f110c0b71c0 -32m:6:322.970 64@0x7f110c0b75a0 -47m:7:712.432
64@0x7f110c0b75f0 -55m:6:246.596 64@0x7f110c0b8e70 -40m:8:441.526
64@0x7f110c0b93d0 -31m:8:367.139 64@0x7f110c0b95f0 -20m:8:338.732
64@0x7f110c0bfa10 -21m:8:101.943 64@0x7f110c0c0020 -40m:8:713.599
64@0x7f110c0c1940 -50m:8:527.709 64@0x7f110c0c2980 -38m:7:435.784
64@0x7f110c0c3a30 -41m:8:780.711 64@0x7f110c0c4c10 -51m:8:456.469
64@0x7f110c0c5900 -23m:9:12.53 64@0x7f110c0c60c0 -42m:7:259.771
64@0x7f110c0c7e80 -36m:9:437.578 64@0x7f110c0c81f0 -42m:9:506.447
64@0x7f110c0ccf70 -41m:7:436.847 64@0x7f110c0cea10 -39m:8:682.504
64@0x7f110c0d24d0 -43m:9:112.895 64@0x7f110c0d6ea0 -29m:6:779.131
64@0x7f110c0d7170 -17m:7:547.842 64@0x7f110c0d8e00 -31m:7:899.271
64@0x7f110c0d9b20 -42m:7:701.900 64@0x7f110c0da080 -20m:7:152.483
64@0x7f110c0daf30 -30m:8:192.754 64@0x7f110c0db9b0 -45m:7:836.585
64@0x7f110c0dca50 -27m:6:325.95 64@0x7f110c0dfe40 -40m:9:295.998
64@0x7f110c0e2a80 -22m:7:873.76 64@0x7f110c0e3ce0 -58m:7:379.583
64@0x7f110c0e53a0 -54m:8:292.950 64@0x7f110c0e65e0 -28m:8:604.121
64@0x7f110c0e6c60 -35m:7:580.31 64@0x7f110c0e6d50 -44m:8:490.833
64@0x7f110c0e8180 -55m:6:953.684 64@0x7f110c0e8fc0 -39m:8:206.488
64@0x7f110c0eac90 -27m:8:172.131 64@0x7f110c0ec970 -30m:7:585.675
64@0x7f110c0f0860 -56m:5:792.627 64@0x7f110c0f0db0 -51m:9:676.148
64@0x7f110c0f3670 -41m:7:842.441 64@0x7f110c0f36c0 -44m:8:782.190
64@0x7f110c0f62e0 -45m:9:716.218 64@0x7f110c0f67e0 -42m:8:125.616
64@0x7f110c0f7ac0 -55m:4:292.662 64@0x7f110c0f7b10 -58m:6:981.242
64@0x7f110c0f9d50 -44m:9:94.242 64@0x7f110c0fa250 -42m:9:40.277
64@0x7f110c0fbd10 -48m:8:482.905 64@0x7f110c0fd810 -35m:8:589.658
64@0x7f110c0fd860 -30m:7:884.8 64@0x7f110c100320 -27m:6:814.788
64@0x7f110c101230 -47m:7:990.970 64@0x7f110c101ee0 -53m:8:978.999
64@0x7f110c1035d0 -47m:8:870.152 64@0x7f110c107830 -48m:7:783.764
64@0x7f110c10c130 -31m:9:12.108 64@0x7f110c1131b0 -40m:7:604.167
64@0x7f110c1163b0 -24m:3:566.37 64@0x7f110c117410 -37m:7:907.553
64@0x7f110c119480 -37m:7:206.584 64@0x7f110c11c790 -49m:7:944.104
64@0x7f110c11e770 -33m:8:80.110 64@0x7f110c11ecc0 -33m:7:253.649
64@0x7f110c1229d0 -38m:6:879.152 64@0x7f110c123430 -34m:6:832.369
64@0x7f110c125740 -52m:9:94.358 64@0x7f110c126820 -33m:7:528.432
64@0x7f110c1289a0 -31m:7:305.272 64@0x7f110c12a990 -57m:6:636.738
64@0x7f110c12d360 -47m:8:437.4 64@0x7f110c12de10 -34m:8:472.0
64@0x7f110c135320 -52m:8:205.240 64@0x7f110c137f10 -57m:5:953.314
64@0x7f110c139a40 -36m:8:2.152 64@0x7f110c13b3c0 -44m:9:475.915
64@0x7f110c13c160 -50m:8:926.600 64@0x7f110c13f890 -46m:7:447.867
64@0x7f110c1411e0 -41m:8:173.92 64@0x7f110c141870 -37m:6:857.673
64@0x7f110c142f20 -34m:7:787.950 64@0x7f110c143d20 -38m:7:151.938
64@0x7f110c145f80 -46m:7:708.926 64@0x7f110c146a20 -34m:7:198.284
64@0x7f110c149890 -36m:8:827.560 64@0x7f110c149c00 -48m:8:740.714
64@0x7f110c14a5b0 -50m:7:947.686 64@0x7f110c14d590 -43m:7:907.519
64@0x7f110c14dda0 -56m:5:518.989 64@0x7f110c14e470 -52m:8:456.336
64@0x7f110c151750 -46m:8:14.799 64@0x7f110c153ed0 -25m:5:700.637
64@0x7f110c154190 -30m:7:107.929 64@0x7f110c157ec0 -39m:9:220.462
64@0x7f110c158280 -57m:5:545.205 64@0x7f110c1583f0 -58m:6:694.217
64@0x7f110c15bae0 -29m:8:38.200 64@0x7f110c15e060 -39m:7:213.637
64@0x7f110c160690 -58m:6:416.913 64@0x7f110c1606e0 -57m:6:295.120
64@0x7f110c161450 -33m:8:403.975 64@0x7f110c161b30 -49m:8:722.131
64@0x7f110c162830 -35m:9:43.229 64@0x7f110c1675b0 -37m:7:513.589
64@0x7f110c168250 -32m:7:817.881 64@0x7f110c168560 -28m:7:356.152
64@0x7f110c1688c0 -14m:7:789.696 64@0x7f110c168d60 -30m:6:410.168
64@0x7f110c1697c0 -39m:7:923.673 64@0x7f110c169940 -37m:8:276.530
64@0x7f110c16a440 -38m:7:726.43 64@0x7f110c16b290 -45m:7:432.1
64@0x7f110c16b9e0 -18m:8:415.292 64@0x7f110c170d50 -29m:7:642.198
64@0x7f110c174150 -26m:6:204.62 64@0x7f110c174910 -31m:6:879.592
64@0x7f110c175ab0 -33m:7:795.839 64@0x7f110c177f40 -22m:8:619.390
64@0x7f110c179300 -27m:7:451.238 64@0x7f110c1793b0 -22m:8:969.947
64@0x7f110c179810 -32m:6:702.861 64@0x7f110c17a930 -33m:8:781.207
64@0x7f110c17b330 -27m:7:773.185 64@0x7f110c17bab0 -27m:7:166.688
64@0x7f110c17c950 -32m:6:11.895 64@0x7f110c17db90 -32m:7:432.30
64@0x7f110c182500 -23m:6:769.283 64@0x7f110c185530 -32m:7:74.736
64@0x7f110c1869c0 -25m:7:454.638 64@0x7f110c186ce0 -25m:6:112.202
64@0x7f110c1872f0 -17m:8:367.233 64@0x7f110c188400 -28m:5:928.577
64@0x7f110c18aa50 -21m:7:289.801 64@0x7f110c18b5b0 -19m:7:821.673
64@0x7f110c18bad0 -28m:7:911.150 64@0x7f110c18d110 -18m:7:101.532
64@0x7f110c18d280 -26m:7:235.638 64@0x7f110c18dea0 -16m:7:704.302
64@0x7f110c18f560 -29m:7:183.971 64@0x7f110c1900b0 -12m:9:121.606
64@0x7f110c190610 -26m:5:904.64 64@0x7f110c190660 -30m:5:756.151
64@0x7f110c1920c0 -10m:6:407.23 64@0x7f110c195d00 -23m:7:483.410
64@0x7f110c196360 -18m:7:958.179 64@0x7f110c197000 -22m:9:380.622
64@0x7f110c198a70 -21m:8:944.16 64@0x7f110c199320 -20m:8:800.52
64@0x7f110c19a010 -16m:8:450.683 64@0x7f110c19a370 -19m:7:161.421
64@0x7f110c19bda0 -19m:8:156.537 64@0x7f110c19c460 -26m:7:618.548
64@0x7f110c19c5d0 -16m:8:921.115 64@0x7f110c19cf60 -26m:6:860.593
64@0x7f110c19dbd0 -18m:6:593.855 64@0x7f110c19ff00 -20m:7:929.255
64@0x7f110c1a1440 -28m:6:858.12 64@0x7f110c1a15b0 -28m:6:380.539
64@0x7f110c1a2ec0 -11m:8:635.945 64@0x7f110c1a4470 -26m:6:511.469
64@0x7f110c1a6c70 -25m:6:524.397 64@0x7f110c1a73d0 -23m:8:303.89
64@0x7f110c1a8800 -17m:8:755.698 64@0x7f110c1a9770 -15m:8:982.823
64@0x7f110c1a9c10 -21m:7:715.810 64@0x7f110c1aa440 -19m:7:491.365
64@0x7f110c1aaf50 -17m:7:184.407 64@0x7f110c1add00 -20m:9:400.185
64@0x7f110c1af070 -23m:7:991.974 64@0x7f110c1b1410 -21m:6:985.864
64@0x7f110c1b1ce0 -21m:8:503.153 64@0x7f110c1b50a0 -24m:5:410.91
64@0x7f110c1b6280 -13m:7:671.4 64@0x7f110c1b7530 -19m:8:670.516
64@0x7f110c1b9e40 -25m:5:340.562 64@0x7f110c1ba6c0 -22m:7:163.423
64@0x7f110c1bc6d0 -18m:9:91.406 64@0x7f110c1c0040 -11m:7:429.371
64@0x7f110c1c11c0 -24m:3:197.906 64@0x7f110c1c26f0 -24m:4:801.762
64@0x7f110c1c2b60 -24m:3:953.97 64@0x7f110c1c3470 -19m:6:363.210
64@0x7f110c1c5280 -24m:4:358.660 64@0x7f110c1c7d00 -16m:8:50.198
64@0x7f110c1cbcf0 -12m:8:642.69 64@0x7f110c1cc590 -16m:7:63.866
64@0x7f110c1cd070 -15m:7:147.872 64@0x7f110c1cde20 -11m:7:832.429
64@0x7f110c1d1530 -15m:6:649.368 64@0x7f110c1d16a0 -18m:7:552.699
64@0x7f110c1d40d0 -14m:6:958.498 64@0x7f110c1d5690 -8m:8:205.99
64@0x7f110c1da7f0 -14m:8:607.385 64@0x7f110c1dd630 -7m:8:106.222
64@0x7f110c1e3f30 -13m:8:836.392 64@0x7f110c1e7fa0 -12m:8:176.663
64@0x7f110c1e91c0 -17m:7:976.98 64@0x7f110c1ecc00 -9m:7:130.831
64@0x7f110c1ed410 -5m:9:126.441 64@0x7f110c1ed860 -8m:6:247.883
64@0x7f110c1ef350 -13m:8:81.832 64@0x7f110c1f1a40 -15m:7:829.76
64@0x7f110c1f1e30 -15m:8:584.551 64@0x7f110c1f2160 -17m:9:262.217
64@0x7f110c1f2270 -16m:7:380.343 64@0x7f110c1f4d70 -12m:6:982.634
64@0x7f110c1f8d80 -14m:7:378.656 64@0x7f110c1f9260 -9m:8:837.651
64@0x7f110c1fb460 -15m:8:259.363 64@0x7f110c1fc9c0 -12m:7:505.957
64@0x7f110c1fcda0 -10m:7:717.779 64@0x7f110c200240 -10m:8:74.819
64@0x7f110c201360 -13m:6:795.480 64@0x7f110c202470 -14m:6:636.199
64@0x7f110c2098b0 -1m:8:172.424 64@0x7f110c209cd0 -13m:9:358.66
64@0x7f110c20b9e0 -14m:8:194.433 64@0x7f110c20be80 -10m:7:73.556
64@0x7f110c20cd70 -13m:7:243.663 64@0x7f110c20d950 -7m:9:334.879
64@0x7f110c20dac0 -9m:7:737.102 64@0x7f110c20efd0 -12m:7:834.413
64@0x7f110c212f30 -10m:9:18.166 64@0x7f110c2130a0 -10m:8:519.923
64@0x7f110c216ab0 -9m:8:78.609 64@0x7f110c21a5c0 -4m:9:305.818
64@0x7f110c21abb0 -6m:6:473.212 64@0x7f110c21c610 -11m:6:819.765
64@0x7f110c21ca60 -11m:6:295.279 64@0x7f110c220990 -9m:6:457.510
64@0x7f110c221480 -8m:6:735.429 64@0x7f110c223500 -8m:8:656.934
64@0x7f110c223f00 -9m:8:462.445 64@0x7f110c227aa0 -3m:7:5.438
64@0x7f110c22a390 -8m:9:136.236 64@0x7f110c22da60 -5m:6:973.514
64@0x7f110c22ed70 -2m:8:32.716 64@0x7f110c22f190 -5m:7:768.399
64@0x7f110c22fa30 -4m:7:392.737 64@0x7f110c231b80 -7m:7:27.773
64@0x7f110c234300 -7m:6:455.315 64@0x7f110c2347e0 -3m:9:391.195
64@0x7f110c2353f0 -6m:8:414.318 64@0x7f110c235770 -8m:7:545.445
64@0x7f110c236510 -7m:7:529.708 64@0x7f110c2372b0 -6m:7:222.261
64@0x7f110c237ca0 -5m:7:421.195 64@0x7f110c238640 -5m:8:632.251
64@0x7f110c23b330 -6m:8:944.321 64@0x7f110c23bfd0 -7m:8:839.280
64@0x7f110c23e9f0 -1m:7:728.729 64@0x7f110c2406b0 -5m:8:149.788
64@0x7f110c244e50 -2m:7:254.872 64@0x7f110c245a90 -0m:7:247.586
64@0x7f110c248ec0 -6m:7:577.950 64@0x7f110c249fd0 -6m:7:965.186
64@0x7f110c24be30 -4m:8:184.840 64@0x7f110c24fbf0 -0m:6:611.912
64@0x7f110c250970 -3m:8:281.339 64@0x7f110c252380 -2m:8:412.816
64@0x7f110c253b10 -4m:7:814.554 64@0x7f110c254a20 -1m:8:809.146
64@0x7f110c256a60 -0m:6:79.197 64@0x7f110c25a840 -3m:8:700.725
64@0x7f110c25aee0 -2m:7:680.366 64@0x7f110c25c970 -4m:8:576.890
64@0x7f110c25cb40 -1m:9:393.517 64@0x7f110c25d260 -4m:6:721.703
64@0x7f110c2619c0 -1m:7:278.515 64@0x7f110c262040 -3m:7:413.76
64@0x7f110c263510 -3m:7:851.882 64@0x7f110c263b10 -1m:6:848.748
64@0x7f110c264490 -2m:6:541.230 64@0x7f110c265780 -0m:8:807.787
64@0x7f110c269f00 -2m:6:33.133 64@0x7f110c27ca90 -0m:8:437.69
64@0x7f110c27eb20 -0m:8:60.559
