----- On Sep 24, 2016, at 3:42 AM, Evgeniy Ivanov [email protected] wrote:
> Hi Mathieu, > > On Sat, Sep 24, 2016 at 12:59 AM, Mathieu Desnoyers > <[email protected]> wrote: >> ----- On Sep 22, 2016, at 3:14 PM, Evgeniy Ivanov [email protected] >> wrote: >> >>> Hi all, >>> >>> I'm investigating high memory usage of my program: RSS varies between >>> executions in range 20-50 GB, though it should be determenistic. I've >>> found that all the memory is allocated in this stack: >>> >>> Allocated 17673781248 bytes in 556 allocations >>> cds_lfht_alloc_bucket_table3 from liburcu-cds.so.2.0.0 >>> _do_cds_lfht_resize from liburcu-cds.so.2.0.0 >>> do_resize_cb from liburcu-cds.so.2.0.0 >>> call_rcu_thread from liburcu-qsbr.so.2.0.0 >>> start_thread from libpthread-2.12.so >>> clone from libc-2.12.so >>> >>> According pstack it should be quiescent state. Call thread waits on >>> syscall: >>> syscall >>> call_rcu_thread >>> start_thread >>> clone >>> >>> We use urcu-0.8.7, only rculfhash (QSBR). Is it some kind of leak in >>> RCU or any chance I misuse it? What would you recommend to >>> troubleshoot the situation? >> >> urcu-qsbr is the fastest flavor of urcu, but it is rather tricky to use well. >> Make sure that: >> >> - Each registered thread periodically reach a quiescent state, by: >> - Invoking rcu_quiescent_state periodically, and >> - Making sure to surround any blocking for relatively large amount of >> time by rcu_thread_offline()/rcu_thread_online(). >> >> In urcu-qsbr, the "default" state of threads is to be within a RCU read-side. >> Therefore, if you omit any of the two advice above, you end up in a situation >> where grace periods never complete, and therefore no call_rcu() callbacks can >> be processed. This effectively acts like a big memory leak. > > It was the original assumption, but in memory stacks I don't see such > allocations for my data. Instead huge allocations happen right in > call_rcu_thread. Memory footprint for my app is about 20 GB, erasing > RCU data is a rare operation, so almost 20 GB in rcu thread looks > suspecios. I'll try to not erase any RCU protected data and reproduce > the issue (complicated thing is that under memory tracer it happens > not so often). Can you provide extra info on the parameters used to create the urcu hash table ? It looks like a lot of hash table buckets are allocated. Moreover, what hash function do you use ? It may be caused by a combination of having lots of collisions due to poor hashing, which then try to increase the number of hash table buckets, and not specifying appropriate limits for the number of buckets when creating the hash table. Thanks, Mathieu > > >> Hoping this helps, >> >> Thanks, >> >> Mathieu >> >> >> -- >> Mathieu Desnoyers >> EfficiOS Inc. >> http://www.efficios.com > > > > -- > Cheers, > Evgeniy > _______________________________________________ > lttng-dev mailing list > [email protected] > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
