Re: [bitcoin-dev] Memory leaks?

[Rusty Russell via bitcoin-dev]

Btc Drak via bitcoin-dev  writes:
> I think this thread has gotten to the stage where it should be moved
> to an issue on Github and not continue to CC the bitcoin-dev list (or
> any other list tbh).

Agreed.  I couldn't see an issue, so I've opened one.  Let's track this
there, please.

https://github.com/bitcoin/bitcoin/issues/6876

Cheers,
Rusty.
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Btc Drak via bitcoin-dev]

I think this thread has gotten to the stage where it should be moved
to an issue on Github and not continue to CC the bitcoin-dev list (or
any other list tbh).

On Thu, Oct 22, 2015 at 5:27 PM, Jonathan Toomim via bitcoin-dev
 wrote:
> You may want to add a cron job to restart bitcoind every day or two as a
> damage control mechanism until we figure this out.
>
> On Oct 22, 2015, at 9:06 AM, Multipool Admin  wrote:
>
> This is a real issue.  The bitcoind process is getting killed every few days
> when it reaches around 55gb of usage on my server.
>
> On Oct 21, 2015 12:29 AM, "Tom Zander via bitcoin-dev"
>  wrote:
>>
>> On Tuesday 20 Oct 2015 20:01:16 Jonathan Toomim wrote:
>> >  claimed that he had this memory usage issue on Linux, but not on Mac OS
>> > X,
>> > under a GBT workload in both situations. If this is true, that would
>> > suggest this might be a fragmentation issue due to poor memory
>> > allocation.
>>
>> Please make sure you measure your memory usage correctly on Linux, it is
>> notoriously easy to get misleading info from tools like top.
>>
>> I use this one on Linux.
>>
>> $cat ~/bin/showmemusage
>> #!/bin/sh
>> if test -z "$1"; then
>> echo "need a pid"
>> exit
>> fi
>>
>> mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
>> sed 's#^#+#' ) | bc`
>> echo "$mem KB"
>>
>> ___
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoin-xt+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
> ___
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Jonathan Toomim via bitcoin-dev]

The method I was using was essentially

grep VmRSS /proc/$pid/status

Comparing these two methods, I get

Your method (PSS):
2408313
My method (RSS):
VmRSS:   2410396 kB

On Oct 21, 2015, at 12:29 AM, Tom Zander  wrote:

> On Tuesday 20 Oct 2015 20:01:16 Jonathan Toomim wrote:
> Please make sure you measure your memory usage correctly on Linux, it is
> notoriously easy to get misleading info from tools like top.
> 
> I use this one on Linux.
> 
> $cat ~/bin/showmemusage
> #!/bin/sh
> if test -z "$1"; then
>echo "need a pid"
>exit
> fi
> 
> mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
> sed 's#^#+#' ) | bc`
> echo "$mem KB"




signature.asc
Description: Message signed with OpenPGP using GPGMail
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Tom Zander via bitcoin-dev]

On Tuesday 20 Oct 2015 20:01:16 Jonathan Toomim wrote:
>  claimed that he had this memory usage issue on Linux, but not on Mac OS X,
> under a GBT workload in both situations. If this is true, that would
> suggest this might be a fragmentation issue due to poor memory allocation.

Please make sure you measure your memory usage correctly on Linux, it is 
notoriously easy to get misleading info from tools like top.

I use this one on Linux. 

$cat ~/bin/showmemusage 
#!/bin/sh
if test -z "$1"; then
echo "need a pid"
exit
fi

mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
sed 's#^#+#' ) | bc`
echo "$mem KB"


signature.asc
Description: This is a digitally signed message part.
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Ross Bennetts via bitcoin-dev]

On Wed, Oct 21, 2015 at 2:01 PM, Jonathan Toomim  wrote:

> 3. One user on the bitcointalk p2pool thread (
> https://bitcointalk.org/index.php?topic=18313.msg12733791#msg12733791)
> claimed that he had this memory usage issue on Linux, but not on Mac OS X,
> under a GBT workload in both situations.
>
>
Thanks for the thorough investigation.
FYI, I believe I've experienced this issue on both Linux & Mac OSX.
I've moved my P2Pool/bitcoind node (rossbennetts.com:9332) from Ubuntu to
OSX since my P2Pool instability began and they both seem to become flaky
after a few hours of uptime (since ~v0.10.something). I've tried both core
and xt.
Two other bitcoin nodes, one connected to electrum-server and one
standalone, seem fine.

//long term lurker, first time poster...
   Ross...

-- 
Ross M. W. Bennetts
https://rossbennetts.com/

E-mail: ross.benne...@gmail.com
Twitter: https://twitter.com/rmwb/
Flickr: https://flickr.com/photos/rmwb/
Instagram: https://instagram.com/rmwb/
Facebook: https://facebook.com/ross.bennetts
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Jonathan Toomim via bitcoin-dev]

I did that Sunday twice. I'll report the results soon. Short version is that it 
looks like valgrind is just finding 200 kB to 600 kB of pblocktemplate, which 
is declared as a static pointer. Not exactly the multi-GB leak I'm looking for, 
but possibly related.

I've also got two bitcoind processes running on the same machine that I started 
at the same time, running on different ports, all with the same settings, but 
one of which is serving getblocktemplate every 5-6 seconds and the other is 
not, while logging RSS on both every 6 seconds. RSS for the non-serving node is 
now 734 MB, and for the serving node 1997 MB. Graphs coming soon.


On Oct 20, 2015, at 3:12 AM, Mike Hearn  wrote:

> OK, then running under Valgrind whilst sending gbt RPCs would be the next 
> step.
> 
> On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin  wrote:
> My nodes are continuously running getblocktemplate and getinfo, and I also 
> suspected the issue is in either gbt or the rpc server.
> 
> The instance only takes a few hours to get up to that memory usage.
> 
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" 
>  wrote:
> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan  
> wrote:
>> This is *most likely* the mempool, but is just not reported correctly.
> 
> I did some testing with PR #6410's better mempool reporting. The improved 
> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is 
> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":). The 
> excess memory usage that I'm seeing is on the order of 100x higher than the 
> mempool "bytes": value. As such, I think it's unlikely that this is the 
> mempool, or at least not normal/correct mempool behavior.
> 
> Another user (ad...@multipool.us) reported 35 GB of RSS usage. I'm guessing 
> his bitcoind has been running longer than any of mine. His server definitely 
> has more RAM. I don't know which email list he is subscribed to (probably 
> XT), so I'm sharing it with both lists to make sure you're all aware of how 
> big an issue this can be.
> 
>> In the meantime you can mitigate the mempool growth by setting `-mintxfee`, 
>> see
>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
> 
> I have mintxfee and minrelaytxfee set to about 0.3, which is high enough 
> to exclude essentially all of the of the 14700-14800 byte flood transactions. 
> My nodes' mempools only contain about one or two blocks' worth of 
> transactions. So I don't think this is correct either.
> 
> 
> 
> Some additional notes on this issue:
> 
> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a 
> Core bitcoind process (commit d78a880) overnight with no mining connected to 
> it, and (IIRC -- my memory is fuzzy) when I woke up it was using around 400 
> MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran 
> ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around 800 
> MB. I then ran getblocktemplate every 5 seconds for about 30 minutes, and RSS 
> climbed to 1180 MB. An hour after that with more getblocktemplates, and now 
> RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is 
> still showing "usage" around 25MB or less.
> 
> I'll do some more testing with this and see if I can make it repeatable, and 
> record the results more carefully. Expect a follow-up from me in a day or two.
> 
> 2. valgrind did not show anything super promising. It did report this:
> 
> ==6880== LEAK SUMMARY:
> ==6880==definitely lost: 0 bytes in 0 blocks
> ==6880==indirectly lost: 0 bytes in 0 blocks
> ==6880==  possibly lost: 288 bytes in 1 blocks
> ==6880==still reachable: 10,552 bytes in 39 blocks
> ==6880== suppressed: 0 bytes in 0 blocks
> (Bitcoin Core commit d78a880)
> 
> and this:
> ==6778== LEAK SUMMARY:
> ==6778==definitely lost: 0 bytes in 0 blocks
> ==6778==indirectly lost: 0 bytes in 0 blocks
> ==6778==  possibly lost: 320 bytes in 1 blocks
> ==6778==still reachable: 10,080 bytes in 32 blocks
> ==6778== suppressed: 0 bytes in 0 blocks
> (Bitcoin XT commit fe446d)
> 
> I haven't found anything in there yet that I think would produce the multi-GB 
> memory usage after running for a few days, but I could be missing it. Email 
> me if you want the full log.
> 
> I did not try running getblocktemplate while valgrind was running. I'll have 
> to try that. I also have not let valgrind run for more than an hour.
> 
> 
> 
> P.S.: Sorry for all the cross-post confusion and consequent flamewar fallout. 
> While it's probably too late for this thread, I'll make sure to post in a 
> manner that keeps the threads clearly separate in the future (e.g. different 
> subject lines).
> 
> ___
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org

Re: [bitcoin-dev] Memory leaks?

[Jonathan Toomim via bitcoin-dev]

More notes:

1. I ran a side-by-side comparison with two bitcoind processes (Core, same 
recent git commit as before) on the same computer with the same settings 
running on different ports. With both processes, I logged RSS (via 
/proc/$pid/status) every 6 seconds. With one of those processes, I also ran 
bitcoin-cli getblocktemplate > /dev/null every 6 seconds. I let that run for 
about 30 hours. A graph and links to the CSVs of raw data are below. Results 
seem pretty clear: the getblocktemplate RPC is implicated in this issue.


http://toom.im/files/memlog8518.csv
http://toom.im/files/memlog-nogbt-8503.csv
http://toom.im/files/bitcoind_memory_usage_gbt.png


2. I ran valgrind twice, for about 6 hours each, on bitcoind while hitting it 
with getblocktemplate every 6 hours. Full valgrind output can be found at these 
two URLs:

http://toom.im/files/valgrind-gbt-1.log
http://toom.im/files/valgrind-gbt-2.log

The summary:

==4064== LEAK SUMMARY:
==4064==definitely lost: 0 bytes in 0 blocks
==4064==indirectly lost: 0 bytes in 0 blocks
==4064==  possibly lost: 288 bytes in 1 blocks
==4064==still reachable: 527,594 bytes in 4,367 blocks
==4064== suppressed: 0 bytes in 0 blocks
The main components of that still reachable section seem to just be one output 
of CreateNewBlock that's cached in case another getblocktemplate request is 
received before any new transactions come in:

==4064== 98,304 bytes in 1 blocks are still reachable in loss record 39 of 40
==4064==at 0x4C29180: operator new(unsigned long) (vg_replace_malloc.c:324)
==4064==by 0x28EAA1: 
__gnu_cxx::new_allocator::allocate(unsigned long, void const*) 
(new_allocator.h:104)
==4064==by 0x27EE44: __gnu_cxx::__alloc_traits::allocate(std::allocator&, unsigned long) (alloc_traits.h:182)
==4064==by 0x26DFB0: std::_Vector_base::_M_allocate(unsigned long) (stl_vector.h:170)
==4064==by 0x2D5BDE: std::vector::_M_insert_aux(__gnu_cxx::__normal_iterator >, CTransaction 
const&) (vector.tcc:353)
==4064==by 0x2D3FF8: std::vector::push_back(CTransaction const&) (stl_vector.h:925)
==4064==by 0x2D113E: CreateNewBlock(CScript const&) (miner.cpp:298)
==4064==by 0x442D78: getblocktemplate(UniValue const&, bool) 
(rpcmining.cpp:513)
==4064==by 0x390CEB: CRPCTable::execute(std::string const&, UniValue 
const&) const (rpcserver.cpp:526)
==4064==by 0x41C5AB: HTTPReq_JSONRPC(HTTPRequest*, std::string const&) 
(httprpc.cpp:125)
==4064==by 0x3559BD: boost::detail::function::void_function_invoker2::invoke(boost::detail::function::function_buffer&, HTTPRequest*, 
std::string const&) (function_template.hpp:112)
==4064==by 0x422520: boost::function2::operator()(HTTPRequest*, std::string const&) const 
(function_template.hpp:767)

There are a few other similar loss records (mostly referring to pblock or 
pblocktemplate in CreateNewBlock(...), but I see nothing that can explain the 
multi-GB memory consumption.

3. One user on the bitcointalk p2pool thread 
(https://bitcointalk.org/index.php?topic=18313.msg12733791#msg12733791) claimed 
that he had this memory usage issue on Linux, but not on Mac OS X, under a GBT 
workload in both situations. If this is true, that would suggest this might be 
a fragmentation issue due to poor memory allocation. The other likely 
hypothesis is bloated caches. Looking into those two possibilities will be my 
next steps.



On Oct 20, 2015, at 5:39 AM, Jonathan Toomim  wrote:

> I did that Sunday twice. I'll report the results soon. Short version is that 
> it looks like valgrind is just finding 200 kB to 600 kB of pblocktemplate, 
> which is declared as a static pointer. Not exactly the multi-GB leak I'm 
> looking for, but possibly related.
> 
> I've also got two bitcoind processes running on the same machine that I 
> started at the same time, running on different ports, all with the same 
> settings, but one of which is serving getblocktemplate every 5-6 seconds and 
> the other is not, while logging RSS on both every 6 seconds. RSS for the 
> non-serving node is now 734 MB, and for the serving node 1997 MB. Graphs 
> coming soon.
> 
> 
> On Oct 20, 2015, at 3:12 AM, Mike Hearn  wrote:
> 
>> OK, then running under Valgrind whilst sending gbt RPCs would be the next 
>> step.
>> 
>> On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin  wrote:
>> My nodes are continuously running getblocktemplate and getinfo, and I also 
>> suspected the issue is in either gbt or the rpc server.
>> 
>> The instance only takes a few hours to get up to that memory usage.
>> 
>> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" 
>>  wrote:
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der 

Re: [bitcoin-dev] Memory leaks?

[Multipool Admin via bitcoin-dev]

My nodes are continuously running getblocktemplate and getinfo, and I also
suspected the issue is in either gbt or the rpc server.

The instance only takes a few hours to get up to that memory usage.
On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan 
> wrote:
>
> This is *most likely* the mempool, but is just not reported correctly.
>
>
> I did some testing with PR #6410's better mempool reporting. The improved
> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
> The excess memory usage that I'm seeing is on the order of 100x higher than
> the mempool "bytes": value. As such, I think it's unlikely that this is the
> mempool, or at least not normal/correct mempool behavior.
>
> Another user (ad...@multipool.us) reported 35 GB of RSS usage. I'm
> guessing his bitcoind has been running longer than any of mine. His server
> definitely has more RAM. I don't know which email list he is subscribed to
> (probably XT), so I'm sharing it with both lists to make sure you're all
> aware of how big an issue this can be.
>
> In the meantime you can mitigate the mempool growth by setting
> `-mintxfee`, see
>
> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>
>
> I have mintxfee and minrelaytxfee set to about 0.3, which is high
> enough to exclude essentially all of the of the 14700-14800 byte flood
> transactions. My nodes' mempools only contain about one or two blocks'
> worth of transactions. So I don't think this is correct either.
>
>
>
> Some additional notes on this issue:
>
> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
> Core bitcoind process (commit d78a880) overnight with no mining connected
> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
> getmempoolinfo is still showing "usage" around 25MB or less.
>
> I'll do some more testing with this and see if I can make it repeatable,
> and record the results more carefully. Expect a follow-up from me in a day
> or two.
>
> 2. valgrind did not show anything super promising. It did report this:
>
> ==6880== LEAK SUMMARY:
> ==6880==definitely lost: 0 bytes in 0 blocks
> ==6880==indirectly lost: 0 bytes in 0 blocks
> ==6880==  possibly lost: 288 bytes in 1 blocks
> ==6880==still reachable: 10,552 bytes in 39 blocks
> ==6880== suppressed: 0 bytes in 0 blocks
> (Bitcoin Core commit d78a880)
>
> and this:
> ==6778== LEAK SUMMARY:
> ==6778==definitely lost: 0 bytes in 0 blocks
> ==6778==indirectly lost: 0 bytes in 0 blocks
> ==6778==  possibly lost: 320 bytes in 1 blocks
> ==6778==still reachable: 10,080 bytes in 32 blocks
> ==6778== suppressed: 0 bytes in 0 blocks
> (Bitcoin XT commit fe446d)
>
> I haven't found anything in there yet that I think would produce the
> multi-GB memory usage after running for a few days, but I could be missing
> it. Email me if you want the full log.
>
> I did not try running getblocktemplate while valgrind was running. I'll
> have to try that. I also have not let valgrind run for more than an hour.
>
>
>
> P.S.: Sorry for all the cross-post confusion and consequent flamewar
> fallout. While it's probably too late for this thread, I'll make sure to
> post in a manner that keeps the threads clearly separate in the future
> (e.g. different subject lines).
>
> ___
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Multipool Admin via bitcoin-dev]

I should also mention that this is definitely not an attack coming from
connected nodes.  My node experiencing the issue is only connected to 3
other nodes, all of which I control (via connect=).

--Adam

On Mon, Oct 19, 2015 at 12:17 PM, Multipool Admin 
wrote:

> My nodes are continuously running getblocktemplate and getinfo, and I also
> suspected the issue is in either gbt or the rpc server.
>
> The instance only takes a few hours to get up to that memory usage.
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan 
>> wrote:
>>
>> This is *most likely* the mempool, but is just not reported correctly.
>>
>>
>> I did some testing with PR #6410's better mempool reporting. The improved
>> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
>> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
>> The excess memory usage that I'm seeing is on the order of 100x higher than
>> the mempool "bytes": value. As such, I think it's unlikely that this is the
>> mempool, or at least not normal/correct mempool behavior.
>>
>> Another user (ad...@multipool.us) reported 35 GB of RSS usage. I'm
>> guessing his bitcoind has been running longer than any of mine. His server
>> definitely has more RAM. I don't know which email list he is subscribed to
>> (probably XT), so I'm sharing it with both lists to make sure you're all
>> aware of how big an issue this can be.
>>
>> In the meantime you can mitigate the mempool growth by setting
>> `-mintxfee`, see
>>
>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>>
>>
>> I have mintxfee and minrelaytxfee set to about 0.3, which is high
>> enough to exclude essentially all of the of the 14700-14800 byte flood
>> transactions. My nodes' mempools only contain about one or two blocks'
>> worth of transactions. So I don't think this is correct either.
>>
>>
>>
>> Some additional notes on this issue:
>>
>> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
>> Core bitcoind process (commit d78a880) overnight with no mining connected
>> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
>> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
>> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
>> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
>> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
>> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
>> getmempoolinfo is still showing "usage" around 25MB or less.
>>
>> I'll do some more testing with this and see if I can make it repeatable,
>> and record the results more carefully. Expect a follow-up from me in a day
>> or two.
>>
>> 2. valgrind did not show anything super promising. It did report this:
>>
>> ==6880== LEAK SUMMARY:
>> ==6880==definitely lost: 0 bytes in 0 blocks
>> ==6880==indirectly lost: 0 bytes in 0 blocks
>> ==6880==  possibly lost: 288 bytes in 1 blocks
>> ==6880==still reachable: 10,552 bytes in 39 blocks
>> ==6880== suppressed: 0 bytes in 0 blocks
>> (Bitcoin Core commit d78a880)
>>
>> and this:
>> ==6778== LEAK SUMMARY:
>> ==6778==definitely lost: 0 bytes in 0 blocks
>> ==6778==indirectly lost: 0 bytes in 0 blocks
>> ==6778==  possibly lost: 320 bytes in 1 blocks
>> ==6778==still reachable: 10,080 bytes in 32 blocks
>> ==6778== suppressed: 0 bytes in 0 blocks
>> (Bitcoin XT commit fe446d)
>>
>> I haven't found anything in there yet that I think would produce the
>> multi-GB memory usage after running for a few days, but I could be missing
>> it. Email me if you want the full log.
>>
>> I did not try running getblocktemplate while valgrind was running. I'll
>> have to try that. I also have not let valgrind run for more than an hour.
>>
>>
>>
>> P.S.: Sorry for all the cross-post confusion and consequent flamewar
>> fallout. While it's probably too late for this thread, I'll make sure to
>> post in a manner that keeps the threads clearly separate in the future
>> (e.g. different subject lines).
>>
>> ___
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Adam McKenna via bitcoin-dev]

I can do better than that:

  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND 


10625   20   0 41.310g 0.035t  51172 S   1.0 57.6  47:11.17 
bitcoind 

Any idea what's going on here?

$ cat /proc/10625/status|grep -i VM
VmPeak: 43647420 kB
VmSize: 43319932 kB
VmLck:60 kB
VmPin: 0 kB
VmHWM:  38286252 kB
VmRSS:  38014300 kB
VmData: 43103812 kB
VmStk:   236 kB
VmExe:  8844 kB
VmLib:194656 kB
VmPTE: 79932 kB
VmSwap:0 kB


On Tuesday, October 13, 2015 at 2:56:15 PM UTC-7, Jonathan Toomim (Toomim 
Bros) wrote:
>
> I just noticed that several of my running bitcoind processes were using 
> around 3+ GB of RAM, even though the mempool itself seemed to be under 
> control.
>
> @prime:~/bin$ ./bitcoin-cli getmempoolinfo
> {
> "size" : 1896,
> "bytes" : 37341328
> }
>
> [total memory usage not shown -- I restarted bitcoind as soon as I 
> noticed, and didn't copy it down from top]
>
> 37 MB mempool, >3 GB RAM usage. Normally, when there aren't a lot of 
> unconfirmed txns floating around the network, memory usage is around 600 
> MB, so this is quite unusual.
>
> After restarting the process and letting it run for a few minutes, I get:
>
>   PID USER  PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
> [###] [] 20   0 1402M  317M 49836 S  1.0  8.2  0:41.71 ./bitcoind 
> -daemon
>
> @prime:~/bin$ ./bitcoin-cli getmempoolinfo
> {
> "size" : 1072,
> "bytes" : 67
> }
>
> 0.67 MB mempool, 317 MB RAM usage. Much more reasonable.
>
>
> Here's another node I'm running that has been online longer, before 
> restarting:
>
>   PID USER  PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
> [###] [] 20   0 4961M 3540M 11080 S  2.8 45.3  8h20:11 
> bin/bitcoind -daemon
>
> @feather:~$ bin/bitcoin-cli getmempoolinfo
> {
> "size" : 3045,
> "bytes" : 39656126
> }
>
> 39 MB mempool, 3540 MB total memory usage. After restarting bitcoind, I 
> see:
>
> []@feather:~$ bin/bitcoin-cli stop
> Bitcoin server stopping
> []@feather:~$ bin/bitcoind -daemon
> Bitcoin server starting
> []@feather:~$ sleep 10; bin/bitcoin-cli getmempoolinfo
> {
> "size" : 39,
> "bytes" : 47037
> }
>
>
>   PID USER  PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
> [###] [] 20   0 1640M  247M 67960 S  0.0  3.2  0:05.17 
> bin/bitcoind -daemon
>
>
>
>
> Does anybody have any guesses where we might be leaking memory, or what is 
> using the additional 2.4 GB? I've been using minrelaytxfee=0.3 or 
> similar on my nodes. Maybe there's a leak in the minrelaytxfee code path? 
> Has anyone else seen something similar?
>
> This issue appears to happen both with Bitcoin Core 0.10.1 and with 
> Bitcoin XT 0.11B.
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[odinn via bitcoin-dev]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

For the record, Mr. Hearn, you do not own this list.  I submit to you
that you have very little to say on this matter at this stage and your
idle threats to "ban people" based on their preferences, suggestions,
or characterizations of your chosen software project are at best very
silly.

I am not and never was on bitcoin...@googlegroups.com and thus have
clipped that from the reply-to.

Mike Hearn via bitcoin-dev:
> Leaks are not the only explanation possible. Caches and
> fragmentation can also give this sort of effect. Unfortunately the
> tools to debug this aren't great. You could try a build with
> tcmalloc and use it to investigate heap stats.
> 
> Odinn, trolling like a 3 year old will get you swiftly banned. Last
> warning. On 14 Oct 2015 9:58 am, "Tom Zander" 
> wrote:
> 
>> On Tuesday 13 Oct 2015 14:56:08 Jonathan Toomim  via bitcoin-dev
>> wrote:
>>> Does anybody have any guesses where we might be leaking memory,
>>> or what
>> is
>>> using the additional 2.4 GB? I've been using
>>> minrelaytxfee=0.3 or similar on my nodes. Maybe there's a
>>> leak in the minrelaytxfee code path? Has anyone else seen
>>> something similar?
>> 
>> I suggest running it in valgrind with --leak-check=full for 10
>> minutes.
>> 
>> valgrind --leak-check=full src/bitcoind 2>&1 | tee out
>> 
>> This at least will show you any memory leaks at exit. Naturally,
>> the leaks you observe may just be design issues where cache can 
>> grow to much and when the cache is cleaned on shutdown you won't
>> see it in the valgrind output.
>> 
>> -- You received this message because you are subscribed to the
>> Google Groups "bitcoin-xt" group. To unsubscribe from this group
>> and stop receiving emails from it, send an email to
>> bitcoin-xt+unsubscr...@googlegroups.com. For more options, visit
>> https://groups.google.com/d/optout.
>> 
> 
> 
> 
> ___ bitcoin-dev mailing
> list bitcoin-dev@lists.linuxfoundation.org 
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 

- -- 
http://abis.io ~
"a protocol concept to enable decentralization
and expansion of a giving economy, and a new social good"
https://keybase.io/odinn
-BEGIN PGP SIGNATURE-

iQEcBAEBCgAGBQJWHnKFAAoJEGxwq/inSG8CoN8H/i4m748xSMkXLgsOO2nqghXr
K3EI98BucU5XF4M/0qW0EjN1PHdkl+EjtUt0POT3mG3hl66PaoA04nMgDrND7V+w
sEXICchAVNx5+AleT65U60iibASZZIlZaXmOtdtCgz7GulmMfsfnNV2IRRvsSO1A
Nl0PuEqPW1/rsJDA58tDb8y2ltMEo5Zi2AYDMvD/AfSuNBqdHM/2IrWSPUwDB7NN
TLq5WXyW5mv7qywIu3/8jk0za6RN4gc1DmpIJHjm4bO+4FoF0oytcaOg5X8uOC1B
pOxhvEM2fTjziXaBJVha/6lrGxfi8/mdLBE68hjB3Q6/KDF9VrugdG0JK0iuDW8=
=YqPh
-END PGP SIGNATURE-
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Mike Hearn via bitcoin-dev]

Leaks are not the only explanation possible. Caches and fragmentation can
also give this sort of effect. Unfortunately the tools to debug this aren't
great. You could try a build with tcmalloc and use it to investigate heap
stats.

Odinn, trolling like a 3 year old will get you swiftly banned. Last warning.
On 14 Oct 2015 9:58 am, "Tom Zander"  wrote:

> On Tuesday 13 Oct 2015 14:56:08 Jonathan Toomim  via bitcoin-dev wrote:
> > Does anybody have any guesses where we might be leaking memory, or what
> is
> > using the additional 2.4 GB? I've been using minrelaytxfee=0.3 or
> > similar on my nodes. Maybe there's a leak in the minrelaytxfee code path?
> > Has anyone else seen something similar?
>
> I suggest running it in valgrind with --leak-check=full for 10 minutes.
>
>   valgrind --leak-check=full src/bitcoind 2>&1 | tee out
>
> This at least will show you any memory leaks at exit.
> Naturally, the leaks you observe may just be design issues where cache can
> grow to much and when the cache is cleaned on shutdown you won't see it in
> the
> valgrind output.
>
> --
> You received this message because you are subscribed to the Google Groups
> "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoin-xt+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Pavel Janík via bitcoin-dev]

> I just noticed that several of my running bitcoind processes were using 
> around 3+ GB of RAM, even though the mempool itself seemed to be under 
> control.
> 
> @prime:~/bin$ ./bitcoin-cli getmempoolinfo
> {
> "size" : 1896,
> "bytes" : 37341328
> }
> 
> [total memory usage not shown -- I restarted bitcoind as soon as I noticed, 
> and didn't copy it down from top]

The newer versions contain much more accurate reporting of memorypool usage 
that reflect the real memory usage:

{
  "size": 58536,
  "bytes": 770044257,
  "usage": 1652554832
}

See PR #6410 (Implement accurate memory accounting for mempool).
--  
Pavel Janík




___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Wladimir J. van der Laan via bitcoin-dev]

On Tue, Oct 13, 2015 at 02:56:08PM -0700, Jonathan Toomim (Toomim Bros) via 
bitcoin-dev wrote:
> I just noticed that several of my running bitcoind processes were using 
> around 3+ GB of RAM, even though the mempool itself seemed to be under 
> control.
> 
> @prime:~/bin$ ./bitcoin-cli getmempoolinfo
> {
> "size" : 1896,
> "bytes" : 37341328
> }
> 
> [total memory usage not shown -- I restarted bitcoind as soon as I noticed, 
> and didn't copy it down from top]

This is *most likely* the mempool, but is just not reported correctly.

In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding

This will be the default in 0.11.1 and 0.10.3. 0.12.0 will have better mempool 
management, as well as more precise reporting (as paveljanik already mentions.)

Wladimir
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Dave Scotese via bitcoin-dev]

It was about 360MB (30 minutes ago?), but is now about 460MB.  I'm sure it
won't keep going up that fast.
{
"size" : 3413,
"bytes" : 41892350
}


On Tue, Oct 13, 2015 at 5:08 PM, Jonathan Toomim (Toomim Bros) 
wrote:

> 16 million divided by 1085 transactions is almost 15Kb per transaction =
> unlikely, right?
>
>
> The recent spam was about 15 kB per transaction, so that part sounds right.
>
> The anomalous thing that I saw was that the total bitcoind process usage
> was about 50-100x higher than I would have expected if the mempool was the
> main determinant of memory usage scaling. Can you tell me how much memory
> Task Manager is reporting your bitcoin process as using both today and
> tomorrow?
>
> On Oct 13, 2015, at 4:52 PM, Dave Scotese via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>
> {
> "size" : 1085,
> "bytes" : 16151768
> }
> It has been running about a day.  I'll report tomorrow too.  This is a
> Windows 8.1 box.
> 16 million divided by 1085 transactions is almost 15Kb per transaction =
> unlikely, right?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoin-xt+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
I like to provide some work at no charge to prove my value. Do you need a
techie?
I own Litmocracy  and Meme Racing
 (in alpha).
I'm the webmaster for The Voluntaryist  which
now accepts Bitcoin.
I also code for The Dollar Vigilante .
"He ought to find it more profitable to play by the rules" - Satoshi
Nakamoto
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Jonathan Toomim (Toomim Bros) via bitcoin-dev]

On Oct 13, 2015, at 3:49 PM, odinn  wrote:

> Signed PGP part
> It would also help to know what operating system(s) you are
> using for both the oldie and the freshie.

Linux feather 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) 
x86_64 GNU/Linux
Linux server 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux
Linux prime 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux

This excessive memory consumption was seen on 3 machines, all of which run 
Debian. All three machines run p2pool as well as bitcoind. Two run XT, one runs 
Core.

> 
> You should compare this to having set up a node on a completely clean
> computer.


I can't afford to do that. All of the servers I have are being used for 
something. Also, I'm not sure what it is you're trying to test for with that 
suggestion. The numbers I'm reporting are for bitcoind's resident set, not for 
the whole server's memory usage. I don't see how other processes running on the 
same machine are relevant unless you are suggesting that RPC calls (e.g. 
getblocktemplate) might be somehow responsible.

> 
> Also, dump your XT, is poo.


Not relevant. I addressed this message to both the Core and XT lists because 
the issue appears to affect both forks. Let's keep blocksize and governance 
debates to their own threads, please.

Repeating request: Has anyone else seen something similar? Can you report your 
mempool size and total bitcoind resident set size for your running full nodes?


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Dave Scotese via bitcoin-dev]

{
"size" : 1085,
"bytes" : 16151768
}

It has been running about a day.  I'll report tomorrow too.  This is a
Windows 8.1 box.

16 million divided by 1085 transactions is almost 15Kb per transaction =
unlikely, right?

On Tue, Oct 13, 2015 at 4:14 PM, Jonathan Toomim (Toomim Bros) 
wrote:

>
> On Oct 13, 2015, at 3:49 PM, odinn 
> wrote:
>
> Signed PGP part
> It would also help to know what operating system(s) you are
> using for both the oldie and the freshie.
>
>
> Linux feather 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3
> (2015-08-04) x86_64 GNU/Linux
> Linux server 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux
> Linux prime 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux
>
> This excessive memory consumption was seen on 3 machines, all of which run
> Debian. All three machines run p2pool as well as bitcoind. Two run XT, one
> runs Core.
>
> You should compare this to having set up a node on a completely clean
> computer.
>
>
> I can't afford to do that. All of the servers I have are being used for
> something. Also, I'm not sure what it is you're trying to test for with
> that suggestion. The numbers I'm reporting are for bitcoind's resident set,
> not for the whole server's memory usage. I don't see how other processes
> running on the same machine are relevant unless you are suggesting that RPC
> calls (e.g. getblocktemplate) might be somehow responsible.
>
> Also, dump your XT, is poo.
>
>
> Not relevant. I addressed this message to both the Core and XT lists
> because the issue appears to affect both forks. Let's keep blocksize and
> governance debates to their own threads, please.
>
> Repeating request: Has anyone else seen something similar? Can you report
> your mempool size and total bitcoind resident set size for your running
> full nodes?
>
> --
> You received this message because you are subscribed to the Google Groups
> "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoin-xt+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
I like to provide some work at no charge to prove my value. Do you need a
techie?
I own Litmocracy  and Meme Racing
 (in alpha).
I'm the webmaster for The Voluntaryist  which
now accepts Bitcoin.
I also code for The Dollar Vigilante .
"He ought to find it more profitable to play by the rules" - Satoshi
Nakamoto
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Jonathan Toomim (Toomim Bros) via bitcoin-dev]

> 16 million divided by 1085 transactions is almost 15Kb per transaction = 
> unlikely, right?


The recent spam was about 15 kB per transaction, so that part sounds right.

The anomalous thing that I saw was that the total bitcoind process usage was 
about 50-100x higher than I would have expected if the mempool was the main 
determinant of memory usage scaling. Can you tell me how much memory Task 
Manager is reporting your bitcoin process as using both today and tomorrow?

On Oct 13, 2015, at 4:52 PM, Dave Scotese via bitcoin-dev 
 wrote:

> 
> 
> {
> "size" : 1085,
> "bytes" : 16151768
> }
> It has been running about a day.  I'll report tomorrow too.  This is a 
> Windows 8.1 box.
> 16 million divided by 1085 transactions is almost 15Kb per transaction = 
> unlikely, right?



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[odinn via bitcoin-dev]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

You should compare this to having set up a node on a completely clean
computer.  It would also help to know what operating system(s) you are
using for both the oldie and the freshie.

Also, dump your XT, is poo.  Then try again, look at Core nodes on
your oldie and freshie.  Watch them for a bit.

Cheers,

O

Jonathan Toomim (Toomim Bros) via bitcoin-dev:
> I just noticed that several of my running bitcoind processes were
> using around 3+ GB of RAM, even though the mempool itself seemed to
> be under control.
> 
> @prime:~/bin$ ./bitcoin-cli getmempoolinfo { "size" : 1896, 
> "bytes" : 37341328 }
> 
> [total memory usage not shown -- I restarted bitcoind as soon as I
> noticed, and didn't copy it down from top]
> 
> 37 MB mempool, >3 GB RAM usage. Normally, when there aren't a lot
> of unconfirmed txns floating around the network, memory usage is
> around 600 MB, so this is quite unusual.
> 
> After restarting the process and letting it run for a few minutes,
> I get:
> 
> PID USER  PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+
> Command [###] [] 20   0 1402M  317M 49836 S  1.0  8.2
> 0:41.71 ./bitcoind -daemon
> 
> @prime:~/bin$ ./bitcoin-cli getmempoolinfo { "size" : 1072, 
> "bytes" : 67 }
> 
> 0.67 MB mempool, 317 MB RAM usage. Much more reasonable.
> 
> 
> Here's another node I'm running that has been online longer, before
> restarting:
> 
> PID USER  PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+
> Command [###] [] 20   0 4961M 3540M 11080 S  2.8 45.3
> 8h20:11 bin/bitcoind -daemon
> 
> @feather:~$ bin/bitcoin-cli getmempoolinfo { "size" : 3045, 
> "bytes" : 39656126 }
> 
> 39 MB mempool, 3540 MB total memory usage. After restarting
> bitcoind, I see:
> 
> []@feather:~$ bin/bitcoin-cli stop Bitcoin server stopping 
> []@feather:~$ bin/bitcoind -daemon Bitcoin server starting 
> []@feather:~$ sleep 10; bin/bitcoin-cli getmempoolinfo { "size"
> : 39, "bytes" : 47037 }
> 
> 
> PID USER  PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+
> Command [###] [] 20   0 1640M  247M 67960 S  0.0  3.2
> 0:05.17 bin/bitcoind -daemon
> 
> 
> 
> 
> Does anybody have any guesses where we might be leaking memory, or
> what is using the additional 2.4 GB? I've been using
> minrelaytxfee=0.3 or similar on my nodes. Maybe there's a leak
> in the minrelaytxfee code path? Has anyone else seen something
> similar?
> 
> This issue appears to happen both with Bitcoin Core 0.10.1 and with
> Bitcoin XT 0.11B.
> 
> 
> 
> ___ bitcoin-dev mailing
> list bitcoin-dev@lists.linuxfoundation.org 
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 

- -- 
http://abis.io ~
"a protocol concept to enable decentralization
and expansion of a giving economy, and a new social good"
https://keybase.io/odinn
-BEGIN PGP SIGNATURE-

iQEcBAEBCgAGBQJWHYqAAAoJEGxwq/inSG8Cp8YIAJa9xWA+rNY9ZOTWjOEZGc7Q
IpXqpIyZprhEog6By/rQ7Te+xdUuaIZrMPESEoQIdMlDjRu7V2CWKGr1LbbDf2v9
A6nMhE19tazDSMAcqHlcuWRfpex3C5QD93Oo7h0QvioNLc8cseNsYzqvOW40vIFL
SPJOfor6IFLEi6/0t7OBhVyaXZdhI7XD1IDxeD67IOafCDwHgixFZWS4aCpz4axj
i4B8DNsVgDdeYI2STBiqnL9Sopdnc1q2CwC1ENszR+sCXwIB9vdPOtIjhtWGk1gi
f+/I8IUP/jn2xIjAGixjEePCIFLFV3kI+B6akH8+TsXB4G9C3zHEHTsfCrmLblg=
=VMV3
-END PGP SIGNATURE-
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev