On Sat, Sep 24, 2016 at 10:42:24AM +0300, Evgeniy Ivanov wrote: > Hi Mathieu, > > On Sat, Sep 24, 2016 at 12:59 AM, Mathieu Desnoyers > <mathieu.desnoy...@efficios.com> wrote: > > ----- On Sep 22, 2016, at 3:14 PM, Evgeniy Ivanov lolkaanti...@gmail.com > > wrote: > > > >> Hi all, > >> > >> I'm investigating high memory usage of my program: RSS varies between > >> executions in range 20-50 GB, though it should be determenistic. I've > >> found that all the memory is allocated in this stack: > >> > >> Allocated 17673781248 bytes in 556 allocations > >> cds_lfht_alloc_bucket_table3 from liburcu-cds.so.2.0.0 > >> _do_cds_lfht_resize from liburcu-cds.so.2.0.0 > >> do_resize_cb from liburcu-cds.so.2.0.0 > >> call_rcu_thread from liburcu-qsbr.so.2.0.0 > >> start_thread from libpthread-2.12.so > >> clone from libc-2.12.so > >> > >> According pstack it should be quiescent state. Call thread waits on > >> syscall: > >> syscall > >> call_rcu_thread > >> start_thread > >> clone > >> > >> We use urcu-0.8.7, only rculfhash (QSBR). Is it some kind of leak in > >> RCU or any chance I misuse it? What would you recommend to > >> troubleshoot the situation? > > > > urcu-qsbr is the fastest flavor of urcu, but it is rather tricky to use > > well. > > Make sure that: > > > > - Each registered thread periodically reach a quiescent state, by: > > - Invoking rcu_quiescent_state periodically, and > > - Making sure to surround any blocking for relatively large amount of > > time by rcu_thread_offline()/rcu_thread_online(). > > > > In urcu-qsbr, the "default" state of threads is to be within a RCU > > read-side. > > Therefore, if you omit any of the two advice above, you end up in a > > situation > > where grace periods never complete, and therefore no call_rcu() callbacks > > can > > be processed. This effectively acts like a big memory leak. > > It was the original assumption, but in memory stacks I don't see such > allocations for my data. Instead huge allocations happen right in > call_rcu_thread. Memory footprint for my app is about 20 GB, erasing > RCU data is a rare operation, so almost 20 GB in rcu thread looks > suspecios. I'll try to not erase any RCU protected data and reproduce > the issue (complicated thing is that under memory tracer it happens > not so often).
Interesting. Trying to figure out why your call_rcu_thread() would ever allocate memory. Ah! Do your RCU callbacks allocate memory? Thanx, Paul > > Hoping this helps, > > > > Thanks, > > > > Mathieu > > > > > > -- > > Mathieu Desnoyers > > EfficiOS Inc. > > http://www.efficios.com > > > > -- > Cheers, > Evgeniy > _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev