Re: [lttng-dev] [PATCH lttng-ust] Improve tracelog handling, reduce exported functions

[Mathieu Desnoyers via lttng-dev]

- On May 20, 2021, at 1:43 PM, Norbert Lange nolang...@gmail.com wrote:

> Am Do., 20. Mai 2021 um 19:18 Uhr schrieb Mathieu Desnoyers
> :
>>
>>
>>
>> - On May 20, 2021, at 12:51 PM, Norbert Lange nolang...@gmail.com wrote:
>>
>> > Am Do., 20. Mai 2021 um 18:25 Uhr schrieb Mathieu Desnoyers
>> > :
>> >>
>> >> - On May 20, 2021, at 11:54 AM, Norbert Lange nolang...@gmail.com 
>> >> wrote:
>> >> [...]
>> >>
>> >> >> What prevents you from linking against lttng-ust.so again ?
>> >> >
>> >> > I did not poke around enough with Lttng to be confident it wont have
>> >> > side effects,
>> >> > I really don't want it active in production. It doesn't seem there is
>> >> > much public knowledge with Xenomai either.
>> >> > lttng-ust.so will spawn threads, lttng-ust-tracepoint.so is mostly 
>> >> > passive,
>> >>
>> >> There is indeed a split between instrumentation and runtime threads done
>> >> with lttng-ust-tracepoint.so vs lttng-ust.so.
>> >>
>> >> I understand that this split is missing for tracelog and tracef, and
>> >> would be a good thing to have.
>> >>
>> >> I would be interested to move the tracelog and tracef implementation
>> >> from liblttng-ust.so to liblttng-ust-tracepoint.so, even this late
>> >> in the -rc cycle, because all users of tracelog/tracef need to link
>> >> against liblttng-ust-tracepoint.so anyway. So moving these symbols
>> >> should not affect anyone.
>> >>
>> >> Can you give it a try and let me know if it works for you ?
>> >
>> > Will take some time, whats the timeframe you need for feedback?
>>
>> Here is the tentative commit:
>>
>> https://review.lttng.org/c/lttng-ust/+/5927 Move tracef/tracelog symbols to
>> liblttng-ust-tracepoint.so
> 
> Well... this is certainly an improvement. I am not completely happy
> though: "... users now link against
> liblttng-ust-tracepoint.so explicitly"

I'm abandoning this change for now.

It's too late in the rc cycle for doing an ABI breaking change. Also, this
can eventually be done gradually by introducing a new .so with new symbols,
and mapping the tracelog/tracef APIs to those new symbols in a future release.
So I don't think it justifies breaking ABI at this stage.

> 
> My homecooked solution currently works like this:
> 
> *) define the probes from  with
> TRACEPOINT_PROBE_DYNAMIC_LINKAGE,
>link them in the application, together with other dynamic probes
> *) build a separate library with *other* tracepoints, lets call it
> libtracepoint.so
> *) don't link the application to any lttng library.
> 
> Which means:
> 
> 1) the application works without lttng libraries. tracepoints are no-ops
> 2) if available then liblttng-ust-tracepoint.so is loaded (constructor
> function from your headers). tracepoints are no-ops
> 3) if the application dlopen's libtracepoint.so and in turn
> liblttng-ust.so then tracepoints work.
> 
> I'd lose option 1 compared to reimplementing tracelog using homecooked
> lttng-ust-tracelog tracepoints.
> 
> So, are there any issues using  that way,
> it seems to work fine,
> are there mutliple competing instances now?
> (I am not re-using any bit from tracelog.h, I am just after using the
> tracepoint definition).
> 
> I mean I could dlsym all the functions, but tracelog has 1 per
> loglevel and really ugly long names ;)

I recommend that you re-use the parts of tracelog which are useful to
you, but that you implement your own events within your provider .so
with your own event names, so there is no clash with upstream lttng-ust.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

[MONTET Julien via lttng-dev]

Hello Mathieu, Norbert and Jan,

Thank you for all of your explainations and the overview of the system.
No I didn't change the ipipe patch for the vDSO, I may try this.
If I have correctly understood, this patch prevents Cobalt from entering in a 
deadlock when the kernel is using the vDSO and the program interrupts the 
kernel at the same time. On which kernel does it word (aroubd 4.19) ?
I currently try to avoid kernel 5.4 because I remember I faced some boot issues 
(but it is on another topic).

Here the issues i faced (drawn on TraceCompass). Are these the deadlocks we are 
talking about ?
https://postimg.cc/BP4G3bF0 (on 11:02:56:380)
https://postimg.cc/q6wHvrcC

Regards,



De : Norbert Lange 
Envoyé : jeudi 20 mai 2021 17:39
À : Mathieu Desnoyers 
Cc : MONTET Julien ; lttng-dev 
; Jan Kiszka ; Xenomai 

Objet : Re: [lttng-dev] LTTng - Xenomai : different results between 
timestamp-lttng and rt_time_read()

Am Do., 20. Mai 2021 um 17:09 Uhr schrieb Mathieu Desnoyers
:
>
> - On May 20, 2021, at 9:56 AM, Mathieu Desnoyers 
> mathieu.desnoy...@efficios.com wrote:
>
> > - On May 20, 2021, at 9:54 AM, lttng-dev lttng-dev@lists.lttng.org 
> > wrote:
> >
> >> - On May 20, 2021, at 5:11 AM, lttng-dev lttng-dev@lists.lttng.org 
> >> wrote:
> >>
> >>> Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
> >>> :
> 
>  Hi Norbert,
> 
>  Thank you for your answer !
> 
>  Yes, I am using a Xenomai cobalt - xenomai is 3.1
>  cat /proc/xenomai/version => 3.1
> 
>  After the installation, I tested "test tools" in /proc/xenomai/ and it 
>  worked
>  nice.
> >>>
> >>> Just asked to make sure, thought the scripts usual add some -xeno tag
> >>> to the kernel version.
> >>>
>  What do you mean by "it might deadlock really good" ?
> >>>
> >>> clock_gettime will either use a syscall (kills realtime always) or is
> >>> optimized via VDSO (which very likely is your case).
> >>>
> >>> What happens is that the kernel will take a spinlock, then write new
> >>> values, then releases the spinlock.
> >>> your program will aswell spin (but just to see if the spinlock is
> >>> free), read the values and interpolates them.
> >>>
> >>> But if your program interrupts the kernel while the kernel holds the
> >>> lock (all on the same cpu core), then it will spin forever and the
> >>> kernel will never execute.
> >>
> >> Just one clarification: the specific locking strategy used by the
> >> Linux kernel monotonic clock vDSO is a "seqlock", where the kernel
> >> sets a bit which keeps concurrent readers looping until they observe
> >
> > When I say "sets a bit", I actually mean "increment a sequence counter",
> > and readers observe either odd or even state, thus knowing whether
> > they need to retry, and whether the value read before/after reading
> > the data structure changed.
>
> Looking again at the Linux kernel's kernel/time/vsyscall.c implementation
> of vdso_update_{begin,end}, I notice that interrupts are disabled across
> the entire update. So I understand that the Interrupt pipeline (I-pipe)
> interrupt gets delivered even when the kernel disables interrupts. Did
> you consider modifying the I-pipe kernel patch to change the vdso update so
> it updates the vdso from within an I-pipe virq handler ?

Yes, I did use an non-upstreamed patch for a while to get things in order:
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.xenomai.org%2Fpipermail%2Fxenomai%2F2018-December%2F040134.htmldata=04%7C01%7Cjulien.montet%40reseau.eseo.fr%7Cef0b71ac314f4ab2321f08d91ba57c9d%7C4d7ad1591265437ab9f62946247d5bf9%7C0%7C0%7C637571219835495365%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=dOiRFzeKFQA%2B25R6aqrjL2ZJMkV5c782DSBGiHHoYZc%3Dreserved=0

I would prefer just a NMI safe source that might jump back a bit, no matter how.

> AFAIU this would allow Xenomai userspace to use the Linux kernel vDSO
> clock sources.

The Xenomai folks are trying to get their next-gen abstraction "dovetail" closer
coupled to the kernel, AFAIR their will be VDSO support and
unification of the clock sources.

Still need to get stuff running today =)

Norbert
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev