Re: [Xenomai-core] Testing LTTng: first insights
Philippe Gerum wrote: > ... > Ok, I'm not going to argue about cleanliness or brokenness about this... > issue, because it's first and foremost a matter of taste. The main > point, I think, is this one: is the clock device something which may be > changed on-the-fly during real-time operations, in such a way that we > would have to atomically switch the device and get its frequency? > Moreover, would this all-in-one approach guarantee anything when we use > the frequency value to rescale timeout values, much later, long after > the atomic section has been exited? > > Because the answer is no, then I would not go for yet another interface > breaking backward compatibility (even if I agree that we could live with > a few patches being obsoleted), just for the purpose of getting the > frequency in some atomic fashion, a guarantee which would not leave > beyond the return point from ipipe_request_tickdev(), and above all, a > guarantee we have no actual use of. This is where "cleanliness" may bite > harder than perceived "brokenness". Let's wait for my patches, and you may understand what I mean (e.g. that the timer frequency is delivered by the selected clock_event_device - atomically or not). Jan ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Testing LTTng: first insights
Jan Kiszka wrote: >> sysinfo.tmfreq was meant to be such value. > > Well, then it is totally mis-initialised on x86 so far, same on powerpc. > In fact, there it is the _clock_ frequency, not the timer frequency. So > what is meant by this? One comment actually call it "Timebase frequency" > (powerpc). This naming looks rather inconsistent. > That's the issue I just pointed out. Normally, tmfreq _should have been_ used as the timer frequency (or timebase frequency for powerpc), and initialized that way. Because no Xenomai port actually cared, all ports ended up doing the same thing as x86, i.e. setting this field to the CPU freq because internally, x86 over APIC uses TSC values to program the local timer. The same way, ia64 passes values from the on-chip clock (ITC) to the ITM, and so on. Agreed 100%, it's lame, but since nobody actually never used tmfreq the right way, it always ended up being set the wrong way. I suggest to use it the right way from now on. >> But to play cleanly, we would have to critical_enter first, >>> look up the currently used clock_event_device, maybe even validate that >>> it was hijacked, and then return its frequency. The problem with this >>> API is, that it is by nature unsynchronised with ipipe_request_tickdev, >>> thus would not always be able to return a valid frequency. >>> >> You get no special guarantee from getting the frequency out of the >> request_tickdev call, because if the point is to prevent anyone from >> changing/removing such device while you are using such value at Xenomai >> level, then we can't, by design. >> >> So we may as well read per_cpu(ipipe_tick_cpu_device) from >> ipipe_get_sysinfo() to access the current tick device installed, without >> any downside. After all, the timer is something which has to be >> considered as a reasonably stable property of the system. Btw, we don't >> currently handle any change of frequency of the underlying hw timer, so >> changing the device would not actually work, I guess. >> >> The next question may be, should we handle such situation? I tend to >> think that we should not, because we just cannot accept any flaky >> situation due to a misbehaving time source which would make the kernel >> downgrade the current clock device, even temporarily, anyway. We have to >> be confident in the current time source when operating. >> >>> Actually, we would only exclude a few patches when going the >>> ipipe_request_tickdev way: those few that were clockevent-aware up to >>> today. For all others (namely 2.6.20 on i386 and up to 2.6.23 on >>> x86_64), we would simply fall back to our current inaccurate approach. I >>> think this is more acceptable than an ipipe_get_sysinfo extension. >>> >> The archdep section from the sysinfo struct has been meant to be >> extended, really, so I think it's actually cleaner to use it - if >> possible - for this purpose, instead of adding new ad hoc interfaces for >> dealing with a particular kernel feature. > > The timer frequency is coupled with the process of requesting a specific > tick device (I'm only talking about GENERIC_CLOCKEVENTS now, because > this is what matters mid-term). Only after deciding what event source to > use, one can obtain its frequency, thus passing it along the request > process appears more than logical and clean to me. Providing this > information "out of band" is broken IMHO, specifically as we need this > value within a very well-define time window (after switching the > clock_event, before using it). > Ok, I'm not going to argue about cleanliness or brokenness about this issue, because it's first and foremost a matter of taste. The main point, I think, is this one: is the clock device something which may be changed on-the-fly during real-time operations, in such a way that we would have to atomically switch the device and get its frequency? Moreover, would this all-in-one approach guarantee anything when we use the frequency value to rescale timeout values, much later, long after the atomic section has been exited? Because the answer is no, then I would not go for yet another interface breaking backward compatibility (even if I agree that we could live with a few patches being obsoleted), just for the purpose of getting the frequency in some atomic fashion, a guarantee which would not leave beyond the return point from ipipe_request_tickdev(), and above all, a guarantee we have no actual use of. This is where "cleanliness" may bite harder than perceived "brokenness". > However, that may not prevent us from doing something different for > legacy kernels and not-yet CLOCKEVENT'ized archs, but only temporarily. > > Jan > -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Testing LTTng: first insights
Philippe Gerum wrote: > Jan Kiszka wrote: >> Philippe Gerum wrote: >>> Jan Kiszka wrote: Jan Kiszka wrote: > ... > Xenomai is loaded at this time but not yet used. Linux runs in tickless > highres mode and obviously had programmed the host timer to fire here. > But instead of one timer IRQ (233) + its handling, we see an additional > early shot (about 3 µs too early here - the longer the timer is > programmed in advance, the larger the error gets) before the xntimer > finally expires. But at the same time, /proc/xenomai/latency reports > 1000 (1 µs). So there must be more discrepancy between TSC and APIC > timebases, no? Well, nothing critical, but at least suboptimal, maybe > pointing at some hidden minor bug. Once time permits, I will check the > APIC frequency and the delay calculation on my box and compare it to > what Linux uses. Looks like Xenomai is using an inaccurate APIC frequency (probably since ages): 10.383 MHz on one of my boxes vs. 10.39591 MHz according to Linux' calibration ( (1,000,000,000 ns * clock_event_device.mult) >> clock_event_device.shift ), which is based on the PM-timer. As the real frequency is higher, the APIC fires earlier than we want to. Consider, e.g., the 4 ms host tick period => 5 µs too early! This correlates with my LTTng traces. >>> Oops. Once again, this proves that having a permanent trace facility in >>> place is key to uncover bugs. >>> I will try to fix this issue by extending the ipipe_request_tickdev interface so that it returns also the frequency of the requested tick device - as long as Xenomai 2.4 is not released, such an API breakage should not cause any hassle IMHO. >>> You may want to have a look at ipipe_get_sysinfo() first, and track the >>> use of the tmfreq field in Xenomai. This may be what you want to fix, >> Nope tmfreq is not involved. The problem is: >> >> rthal_timerfreq_arg = apic_read(APIC_TMICT) * HZ; >> > > Actually, ports should use rthal_get_timerfreq() to get this value which > in turn calls into the I-pipe to determine it accurately, instead of > approximating it by themselves (this is not to say that the I-pipe > always does this, though). Only ARM has it right currently. > >>> IIUC. This would keep the older patches usable with the next Xenomai >>> version, which is very desirable. >> We could extend the information ipipe_get_sysinfo returns by the timer >> frequency. > > sysinfo.tmfreq was meant to be such value. Well, then it is totally mis-initialised on x86 so far, same on powerpc. In fact, there it is the _clock_ frequency, not the timer frequency. So what is meant by this? One comment actually call it "Timebase frequency" (powerpc). This naming looks rather inconsistent. > > But to play cleanly, we would have to critical_enter first, >> look up the currently used clock_event_device, maybe even validate that >> it was hijacked, and then return its frequency. The problem with this >> API is, that it is by nature unsynchronised with ipipe_request_tickdev, >> thus would not always be able to return a valid frequency. >> > > You get no special guarantee from getting the frequency out of the > request_tickdev call, because if the point is to prevent anyone from > changing/removing such device while you are using such value at Xenomai > level, then we can't, by design. > > So we may as well read per_cpu(ipipe_tick_cpu_device) from > ipipe_get_sysinfo() to access the current tick device installed, without > any downside. After all, the timer is something which has to be > considered as a reasonably stable property of the system. Btw, we don't > currently handle any change of frequency of the underlying hw timer, so > changing the device would not actually work, I guess. > > The next question may be, should we handle such situation? I tend to > think that we should not, because we just cannot accept any flaky > situation due to a misbehaving time source which would make the kernel > downgrade the current clock device, even temporarily, anyway. We have to > be confident in the current time source when operating. > >> Actually, we would only exclude a few patches when going the >> ipipe_request_tickdev way: those few that were clockevent-aware up to >> today. For all others (namely 2.6.20 on i386 and up to 2.6.23 on >> x86_64), we would simply fall back to our current inaccurate approach. I >> think this is more acceptable than an ipipe_get_sysinfo extension. >> > > The archdep section from the sysinfo struct has been meant to be > extended, really, so I think it's actually cleaner to use it - if > possible - for this purpose, instead of adding new ad hoc interfaces for > dealing with a particular kernel feature. The timer frequency is coupled with the process of requesting a specific tick device (I'm only talking about GENERIC_CLOCKEVENTS now, because this is what matters mid-term). Only after deciding wh
Re: [Xenomai-core] Testing LTTng: first insights
Jan Kiszka wrote: > Philippe Gerum wrote: >> Jan Kiszka wrote: >>> Jan Kiszka wrote: ... Xenomai is loaded at this time but not yet used. Linux runs in tickless highres mode and obviously had programmed the host timer to fire here. But instead of one timer IRQ (233) + its handling, we see an additional early shot (about 3 µs too early here - the longer the timer is programmed in advance, the larger the error gets) before the xntimer finally expires. But at the same time, /proc/xenomai/latency reports 1000 (1 µs). So there must be more discrepancy between TSC and APIC timebases, no? Well, nothing critical, but at least suboptimal, maybe pointing at some hidden minor bug. Once time permits, I will check the APIC frequency and the delay calculation on my box and compare it to what Linux uses. >>> Looks like Xenomai is using an inaccurate APIC frequency (probably since >>> ages): 10.383 MHz on one of my boxes vs. 10.39591 MHz according to >>> Linux' calibration ( (1,000,000,000 ns * clock_event_device.mult) >> >>> clock_event_device.shift ), which is based on the PM-timer. As the real >>> frequency is higher, the APIC fires earlier than we want to. Consider, >>> e.g., the 4 ms host tick period => 5 µs too early! This correlates with >>> my LTTng traces. >>> >> Oops. Once again, this proves that having a permanent trace facility in >> place is key to uncover bugs. >> >>> I will try to fix this issue by extending the ipipe_request_tickdev >>> interface so that it returns also the frequency of the requested tick >>> device - as long as Xenomai 2.4 is not released, such an API breakage >>> should not cause any hassle IMHO. >>> >> You may want to have a look at ipipe_get_sysinfo() first, and track the >> use of the tmfreq field in Xenomai. This may be what you want to fix, > > Nope tmfreq is not involved. The problem is: > > rthal_timerfreq_arg = apic_read(APIC_TMICT) * HZ; > Actually, ports should use rthal_get_timerfreq() to get this value which in turn calls into the I-pipe to determine it accurately, instead of approximating it by themselves (this is not to say that the I-pipe always does this, though). Only ARM has it right currently. >> IIUC. This would keep the older patches usable with the next Xenomai >> version, which is very desirable. > > We could extend the information ipipe_get_sysinfo returns by the timer > frequency. sysinfo.tmfreq was meant to be such value. But to play cleanly, we would have to critical_enter first, > look up the currently used clock_event_device, maybe even validate that > it was hijacked, and then return its frequency. The problem with this > API is, that it is by nature unsynchronised with ipipe_request_tickdev, > thus would not always be able to return a valid frequency. > You get no special guarantee from getting the frequency out of the request_tickdev call, because if the point is to prevent anyone from changing/removing such device while you are using such value at Xenomai level, then we can't, by design. So we may as well read per_cpu(ipipe_tick_cpu_device) from ipipe_get_sysinfo() to access the current tick device installed, without any downside. After all, the timer is something which has to be considered as a reasonably stable property of the system. Btw, we don't currently handle any change of frequency of the underlying hw timer, so changing the device would not actually work, I guess. The next question may be, should we handle such situation? I tend to think that we should not, because we just cannot accept any flaky situation due to a misbehaving time source which would make the kernel downgrade the current clock device, even temporarily, anyway. We have to be confident in the current time source when operating. > Actually, we would only exclude a few patches when going the > ipipe_request_tickdev way: those few that were clockevent-aware up to > today. For all others (namely 2.6.20 on i386 and up to 2.6.23 on > x86_64), we would simply fall back to our current inaccurate approach. I > think this is more acceptable than an ipipe_get_sysinfo extension. > The archdep section from the sysinfo struct has been meant to be extended, really, so I think it's actually cleaner to use it - if possible - for this purpose, instead of adding new ad hoc interfaces for dealing with a particular kernel feature. > Hmm, maybe we could install a temporary API for x86_64 so that users > don't have to wait for 2.6.24 to get an accurate APIC. This would be > removed again with the first unified x86-ipipe patch. > > Jan > -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Testing LTTng: first insights
Philippe Gerum wrote: > Jan Kiszka wrote: >> Jan Kiszka wrote: >>> ... >>> Xenomai is loaded at this time but not yet used. Linux runs in tickless >>> highres mode and obviously had programmed the host timer to fire here. >>> But instead of one timer IRQ (233) + its handling, we see an additional >>> early shot (about 3 µs too early here - the longer the timer is >>> programmed in advance, the larger the error gets) before the xntimer >>> finally expires. But at the same time, /proc/xenomai/latency reports >>> 1000 (1 µs). So there must be more discrepancy between TSC and APIC >>> timebases, no? Well, nothing critical, but at least suboptimal, maybe >>> pointing at some hidden minor bug. Once time permits, I will check the >>> APIC frequency and the delay calculation on my box and compare it to >>> what Linux uses. >> Looks like Xenomai is using an inaccurate APIC frequency (probably since >> ages): 10.383 MHz on one of my boxes vs. 10.39591 MHz according to >> Linux' calibration ( (1,000,000,000 ns * clock_event_device.mult) >> >> clock_event_device.shift ), which is based on the PM-timer. As the real >> frequency is higher, the APIC fires earlier than we want to. Consider, >> e.g., the 4 ms host tick period => 5 µs too early! This correlates with >> my LTTng traces. >> > > Oops. Once again, this proves that having a permanent trace facility in > place is key to uncover bugs. > >> I will try to fix this issue by extending the ipipe_request_tickdev >> interface so that it returns also the frequency of the requested tick >> device - as long as Xenomai 2.4 is not released, such an API breakage >> should not cause any hassle IMHO. >> > > You may want to have a look at ipipe_get_sysinfo() first, and track the > use of the tmfreq field in Xenomai. This may be what you want to fix, Nope tmfreq is not involved. The problem is: rthal_timerfreq_arg = apic_read(APIC_TMICT) * HZ; > IIUC. This would keep the older patches usable with the next Xenomai > version, which is very desirable. We could extend the information ipipe_get_sysinfo returns by the timer frequency. But to play cleanly, we would have to critical_enter first, look up the currently used clock_event_device, maybe even validate that it was hijacked, and then return its frequency. The problem with this API is, that it is by nature unsynchronised with ipipe_request_tickdev, thus would not always be able to return a valid frequency. Actually, we would only exclude a few patches when going the ipipe_request_tickdev way: those few that were clockevent-aware up to today. For all others (namely 2.6.20 on i386 and up to 2.6.23 on x86_64), we would simply fall back to our current inaccurate approach. I think this is more acceptable than an ipipe_get_sysinfo extension. Hmm, maybe we could install a temporary API for x86_64 so that users don't have to wait for 2.6.24 to get an accurate APIC. This would be removed again with the first unified x86-ipipe patch. Jan signature.asc Description: OpenPGP digital signature ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Testing LTTng: first insights
Jan Kiszka wrote: > Jan Kiszka wrote: >> ... >> Xenomai is loaded at this time but not yet used. Linux runs in tickless >> highres mode and obviously had programmed the host timer to fire here. >> But instead of one timer IRQ (233) + its handling, we see an additional >> early shot (about 3 µs too early here - the longer the timer is >> programmed in advance, the larger the error gets) before the xntimer >> finally expires. But at the same time, /proc/xenomai/latency reports >> 1000 (1 µs). So there must be more discrepancy between TSC and APIC >> timebases, no? Well, nothing critical, but at least suboptimal, maybe >> pointing at some hidden minor bug. Once time permits, I will check the >> APIC frequency and the delay calculation on my box and compare it to >> what Linux uses. > > Looks like Xenomai is using an inaccurate APIC frequency (probably since > ages): 10.383 MHz on one of my boxes vs. 10.39591 MHz according to > Linux' calibration ( (1,000,000,000 ns * clock_event_device.mult) >> > clock_event_device.shift ), which is based on the PM-timer. As the real > frequency is higher, the APIC fires earlier than we want to. Consider, > e.g., the 4 ms host tick period => 5 µs too early! This correlates with > my LTTng traces. > Oops. Once again, this proves that having a permanent trace facility in place is key to uncover bugs. > I will try to fix this issue by extending the ipipe_request_tickdev > interface so that it returns also the frequency of the requested tick > device - as long as Xenomai 2.4 is not released, such an API breakage > should not cause any hassle IMHO. > You may want to have a look at ipipe_get_sysinfo() first, and track the use of the tmfreq field in Xenomai. This may be what you want to fix, IIUC. This would keep the older patches usable with the next Xenomai version, which is very desirable. > (At this chance, I will also try to kill clock_event_device.delta. I > think I have a nicer approach...) > Ack. > Jan > -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Testing LTTng: first insights
Jan Kiszka wrote: > ... > Xenomai is loaded at this time but not yet used. Linux runs in tickless > highres mode and obviously had programmed the host timer to fire here. > But instead of one timer IRQ (233) + its handling, we see an additional > early shot (about 3 µs too early here - the longer the timer is > programmed in advance, the larger the error gets) before the xntimer > finally expires. But at the same time, /proc/xenomai/latency reports > 1000 (1 µs). So there must be more discrepancy between TSC and APIC > timebases, no? Well, nothing critical, but at least suboptimal, maybe > pointing at some hidden minor bug. Once time permits, I will check the > APIC frequency and the delay calculation on my box and compare it to > what Linux uses. Looks like Xenomai is using an inaccurate APIC frequency (probably since ages): 10.383 MHz on one of my boxes vs. 10.39591 MHz according to Linux' calibration ( (1,000,000,000 ns * clock_event_device.mult) >> clock_event_device.shift ), which is based on the PM-timer. As the real frequency is higher, the APIC fires earlier than we want to. Consider, e.g., the 4 ms host tick period => 5 µs too early! This correlates with my LTTng traces. I will try to fix this issue by extending the ipipe_request_tickdev interface so that it returns also the frequency of the requested tick device - as long as Xenomai 2.4 is not released, such an API breakage should not cause any hassle IMHO. (At this chance, I will also try to kill clock_event_device.delta. I think I have a nicer approach...) Jan -- Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
