That explains a lot. Our SHM space is hard coded to 64MB, which limits me big-time.
It still doesn't explain why I was able to add procname to the context but adding also vpid and vtid failed when the number of cores was 24 rather than 12. I mean - the formula you give (and we use per-UID buffers) doesn't seem to be related to what's in context and what's not... Now, on the 12 core (6+HT) system, I would need - nb_traced_users(=1) * subbuf_size(=256KB) * nr_subbuf(=16) * nb_cpus(=12) = 1*256KB*16*16 = 48MB But the same arguments worked just fine on the 24-core system, including adding procname to context. Adding vtid + vpid worked on the 12-core, but failed on the 24-core system. Any ideas why? Amit Margalit IBM XIV - Storage Reinvented XIV-NAS Development Team Tel. 03-689-7774 Fax. 03-689-7230 From: Mathieu Desnoyers <[email protected]> To: Amit Margalit/Israel/IBM@IBMIL Cc: [email protected] Date: 07/10/2013 04:59 PM Subject: Re: [lttng-dev] How much shared memory is used by LTTng (UST) ? * Amit Margalit ([email protected]) wrote: > Hi, > > Chasing down an issue with adding some context (see > https://bugs.lttng.org/issues/579), I've encountered something that makes > me think the failures I am seeing are related to amount of shared memory > that LTTng uses... > > In short - when the subbuffers have a large enough size and number, adding > things to context on a multi-core system can cause libringbuffer/shm.c in > lttng-ust to complain about "No space left on device", leading to > everything looking OK, except that the trace directory stays empty. > > I am working on a 6-core (12 with HT) system that has 24GB of RAM. This is > one configuration, but we also have dual-CPU systems (i.e. 12 cores / 24 > with HT) with 96GB. > > I am using 16 subbuffers of 256KB. In this setup, I have no problem adding > procname, vpid and vtid to the context. > > When I tried the same on the 12-core dual-CPU system, it failed as > explained above. I tried reducing the subbuffer sizes to 64KB, and the > problem disappeared. > > So - how can I tell how much shared memory LTTng wants, and - if you know > - how can I configure my system to allow more than the default? First thing, try using per-uid buffers rather than the 2.2 default (per-pid buffers). If you do that, then you will need: nb_traced_users * subbuf_size * nr_subbuf * nb_cpus as shm space. If you stay with per-pid buffers, then you need nb_processes * subbuf_size * nr_subbuf * nb_cpus as shm space. On my system here, I notice that each shm creates files within /run/shm (this location may differ on your system). It is a tmpfs filesystem. You might want to increase its size. For instance, here, it is 1.6GB. Thanks, Mathieu > > Thanks, > > Amit Margalit > IBM XIV - Storage Reinvented > XIV-NAS Development Team > Tel. 03-689-7774 > Fax. 03-689-7230 > _______________________________________________ > lttng-dev mailing list > [email protected] > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
_______________________________________________ lttng-dev mailing list [email protected] http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
