Re: [Xenomai-core] Synchronising TSC and periodic timer
Philippe Gerum wrote: > Jan Kiszka wrote: >> ... >> On the other hand, the advantage of TSC-based synchronised inter-tick >> timestamps is that you can do things like >> >> sleep_until(rt_timer_ns2ticks(rtdm_clock_read() + 100)) >> >> without risking an error beyond +/- 1 tick (+jitter). With current >> jiffies vs. TSC in periodic mode, this is not easily possible. You have >> to sync in the application, creating another error source when the delay >> between acquiring the TSC and sync'ing the TSC on jiffies is too long. >> > > The proper way to solve this is rather to emulate the periodic mode over > the oneshot machinery, so that we stop having this +/- 1 tick error > margin. The periodic mode as it is now is purely a x86 legacy; even on > some ppc boards where the auto-reload feature is available from the > decrementer, Xeno doesn't use it. > > The more I think of the x86 situation, the more I find it quite silly. I > mean, picking the periodic mode means that 1) all delays can be > expressed as multiples of a given constant interval, 2) the constant > interval must be large enough so that you don't put your board on its > knees, by processing useless ticks most of the time. What one saves here > - using periodic mode - is a couple of outb's per tick on the ISA bus, > since the PIT handles this automatically without software intervention > once set up properly. We already know that the programming overhead > (i.e. introduced by those outb's) is perfectly bearable even for high > frequency sampling like 10Khz loops in aperiodic mode. So why on earth > do we care about saving two outb's and get a lousy timing accuracy in > the same move, for constant interval delays which are necessarily going > to be much larger than those already supported by the aperiodic mode? Er... > > This is a shift in the underlying logic of the periodic mode we are > discussing here actually. It used to be a mode where timing accuracy was > only approximate, mostly to deal with timeouts, in the watchdog sense. > Now, it is becoming a way to rely on a constant interval unit, while > still keeping a high timing accuracy. I'm ok with this, since we don't > rely on true PIT (except for x86, which is fixable) when running in > periodic mode, so I see no problem in raising the level of timing > accuracy of such mode. Existing stuff would not break because of such > change, but improve instead for people who care for exact durations in > periodic mode. Yep, getting rid of as much periodic mode limitations as reasonable in a transparent way sounds very good to me. Jan signature.asc Description: OpenPGP digital signature ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Rodrigo Rosenfeld Rosas wrote: I would choose the first case, since I think all rt-programmers are smarter (or should be) then the average programmer. Uh? no, we aren't smarter. Being smarter would require us to stop spending countless hours working on ten lines of code, just for the purpose of saving a couple of microseconds on the so-called "fast path"... :o> -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Jan Kiszka wrote: Philippe Gerum wrote: Given the description above, just that some skin might return either nucleus ticks or corrected timestamps to the applications, which would in turn do some arithmetics for converting values they got from the skin between both scales internally, and mistakenly use the result while assuming that both scales are always in sync. In this situation, and during a fraction of the time (i.e. the jitter), both scales might not be in sync, and the result would be unusable. This said, this kind of issue could be solved by big fat warnings in documentation, explicitely saying that conversions between both scales might be meaningless. So the worst-case is when a user derives some relative times from two different time sources, one purely tick-base, the other improved by inter-tick TSC (when available on that arch)? Let's say the user takes timestamp t1 = rtdm_clock_read() (via some driver) and a bit later t2 = rt_timer_tick2ns(rt_timer_read()). t1 was set to the last tick Tn + the number of TSC ticks since then: t1 = Tn * tick_period + TSC_offset With, e.g., Tn=1001, tick_period = 1000 us, and TSC_offset = 589 us: t1 = 1001 * 1000 us + 589 us = 1001569 us As the next tick may have not stroke yet when taking t2, that value converted to us can be smaller: t2 = Tn * tick_period = 1001000 Now the difference between t2 and t1 becomes negative (-589 us), although the user may expect it t2-t1 >= 0. Is this non-monotony your concern? No, because both are sourced from xnpod_get_time(). My concern is that the offset correction is going to cause situations where actual count of ticks kept by the nucleus might be different from corrected_time / period, due to the jitter issue, albeit the corrected timestamp is expected to depend on the count of ticks. So you end up having two time scales which are expected to be in sync, albeit there might not under certain conditions. On the other hand, the advantage of TSC-based synchronised inter-tick timestamps is that you can do things like sleep_until(rt_timer_ns2ticks(rtdm_clock_read() + 100)) without risking an error beyond +/- 1 tick (+jitter). With current jiffies vs. TSC in periodic mode, this is not easily possible. You have to sync in the application, creating another error source when the delay between acquiring the TSC and sync'ing the TSC on jiffies is too long. The proper way to solve this is rather to emulate the periodic mode over the oneshot machinery, so that we stop having this +/- 1 tick error margin. The periodic mode as it is now is purely a x86 legacy; even on some ppc boards where the auto-reload feature is available from the decrementer, Xeno doesn't use it. The more I think of the x86 situation, the more I find it quite silly. I mean, picking the periodic mode means that 1) all delays can be expressed as multiples of a given constant interval, 2) the constant interval must be large enough so that you don't put your board on its knees, by processing useless ticks most of the time. What one saves here - using periodic mode - is a couple of outb's per tick on the ISA bus, since the PIT handles this automatically without software intervention once set up properly. We already know that the programming overhead (i.e. introduced by those outb's) is perfectly bearable even for high frequency sampling like 10Khz loops in aperiodic mode. So why on earth do we care about saving two outb's and get a lousy timing accuracy in the same move, for constant interval delays which are necessarily going to be much larger than those already supported by the aperiodic mode? Er... This is a shift in the underlying logic of the periodic mode we are discussing here actually. It used to be a mode where timing accuracy was only approximate, mostly to deal with timeouts, in the watchdog sense. Now, it is becoming a way to rely on a constant interval unit, while still keeping a high timing accuracy. I'm ok with this, since we don't rely on true PIT (except for x86, which is fixable) when running in periodic mode, so I see no problem in raising the level of timing accuracy of such mode. Existing stuff would not break because of such change, but improve instead for people who care for exact durations in periodic mode. And what would prevent us from improving the accuracy of other timestamping API functions beyond RTDM as well, e.g. on converting from ticks to nanos in rt_timer_ticks2ns()? I don't understand why rt_timer_ticks2ns() should be impacted by such extension. This service must keep a constant behaviour, regardless of any outstanding timing issue. I mean, 3 ticks from a 1Khz clock rate must always return 3,000,000 nanos, unless you stop passing count of ticks but fractional/compound values instead. Forget about this, it was (pre-lunch) nonsense. The bottom-line is that we should not blur the line between periodic and aperiodic timing modes, just for getting precise timestamps in the former case
Re: [Xenomai-core] Synchronising TSC and periodic timer
Jan Kiszka escreveu: We discussed a lot about how to prevent the user shooting him/herself in the knee with inter-tick timestamps, but I still think that rtdm_clock_read_tsc() would even be worse in this regard. What do you think abou this documentation: "This function is meant to be used in periodic mode for getting a high timestamp, independently from the system timer's tick. Its return values should not be used mixed with rtdm_clock_read() values because they are not syncronised. Driver authors[developers?] are advised to state this on driver documentation for the final users where returning these values to them for avoiding confusion. Note: This function is available for uniprocessor systems only in the meantime." I think it explains and will not make confusion in driver developers... If sometime someone give a good solution to the syncronisation problem between multiple processors, this can be changed... > ... Rodrigo. ___ Yahoo! doce lar. Faça do Yahoo! sua homepage. http://br.yahoo.com/homepageset.html ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Rodrigo Rosenfeld Rosas wrote: > Em Segunda 20 Março 2006 13:51, Philippe Gerum escreveu: > >> ... >> I think that you should try convincing Jan that rtdm_clock_tsc() might >> be a good idea to provide, instead of tweaking rtdm_clock_read() in a >> way which changes its underlying logic. ;o) > > Yes, that is exactly what I want! :) > I don't see any reason for changing rtdm_timer_read() neither. I think that > the most common usage of high-precision timestamps is for relative time > cases. It doesn't need to be in sync with Xenomai's timer... It is best to > keep things simple. > > What do you think Jan? We discussed a lot about how to prevent the user shooting him/herself in the knee with inter-tick timestamps, but I still think that rtdm_clock_read_tsc() would even be worse in this regard. xnarch_get_cpu_tsc() and derived skin services are not supposed to deliver consistent results across multiple CPUs, are they? While the user could avoid such scenarios by locking tasks on a specific CPU, drivers cannot - at least so far. So, to safely introduce such a low-level service for RTDM, I think we need A) CPU affinity for RTDM-registered IRQs B) CPU affinity for RTDM kernel tasks C) Some well written docs, explaining how to safely use TSCs at driver level and how to provide them to the user (the latter aspect makes me worry most) While A) and B) might be useful for other (though rare) scenarios as well, C) will still require a very good understanding and interface design from the driver writer, while I don't see comparable error dimensions with the improved rtdm_clock_read(). Comparing apples (rtdm_clock_read()) to oranges (rt_timer_read()), there will be some error around a tick period. But comparing apple[0] (rtdm_clock_read_tsc() on CPU#0) to apple[1] (rtdm_clock_read_tsc() on CPU#1), the error could become *much* larger and the design and documentation effort to avoid this will be significant. Ok, as a simple resolution for this problem, I could imagine introducing a TSC timestamping service to RTDM that always fall back to that level of accuracy which is guaranteed to be consistent - either because we run in aperiodic mode, or on uniprocessor, or thanks to some magic synchronisation between all CPU clocks. This would have to be decided at build time, in the first version likely by checking for (multiprocessor || aperiodic) to switch to xnpod_get_time(). Jan signature.asc Description: OpenPGP digital signature ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Em Segunda 20 Março 2006 13:51, Philippe Gerum escreveu: >... >I think that you should try convincing Jan that rtdm_clock_tsc() might >be a good idea to provide, instead of tweaking rtdm_clock_read() in a >way which changes its underlying logic. ;o) Yes, that is exactly what I want! :) I don't see any reason for changing rtdm_timer_read() neither. I think that the most common usage of high-precision timestamps is for relative time cases. It doesn't need to be in sync with Xenomai's timer... It is best to keep things simple. What do you think Jan? P.S: Sorry for the last message, Philippe. I didn't see that one at the time. ___ Yahoo! Acesso Grátis - Internet rápida e grátis. Instale o discador agora! http://br.acesso.yahoo.com ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Em Segunda 20 Março 2006 12:23, Philippe Gerum escreveu: >... >It's not a matter of dealing with users always doing The Right Thing, >but preferably preventing people from doing the wrong one. But we then have two problems and there are tradeoffs here. In one hand we want to avoid users from making mistakes. In the other hand we want to provide a way for solving the issue that generated this thread. A solution for the last one would be to have a function that would return a high-precision timestamp, not necessarily in sync with xenomai's timer, since it would be used for relative time calculations, but should be in sync between multiple CPUs. But this solution would rise the possibility for a user to do a wrong thing. Of course, the functions should be well documented and states the lack of sync if it is the case. So, we have to choose between turning it possible to have such design (as the example I have last message) or avoid people from doing the wrong thing. I would choose the first case, since I think all rt-programmers are smarter (or should be) then the average programmer. They must have attention when dealing with rt-programming. So, reading a documentation and understanding it should not be a hard task for them... In the other hand, if the second approach was chosen, a user wanting to use a rt-video interface will be forced to use the aperiodic timer for having reliable timestamps... Rodrigo. ___ Yahoo! Acesso Grátis - Internet rápida e grátis. Instale o discador agora! http://br.acesso.yahoo.com ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Rodrigo Rosenfeld Rosas wrote: Philippe Gerum wrote: ... Given the description above, just that some skin might return either nucleus ticks or corrected timestamps to the applications, which would in turn do some arithmetics for converting values they got from the skin between both scales internally, and mistakenly use the result while assuming that both scales are always in sync. In this situation, and during a fraction of the time (i.e. the jitter), both scales might not be in sync, and the result would be unusable. But I still can't find a real situation where the user would need these values to be in sync... ... But maybe we are still discussing different issues actually, so it would be useful that the core issue that triggered the discussion about periodic mode precision be exposed again. The core issue is that: I think the driver development should be kind of independent from the user programs. That said, if the driver needs a precise timestamp, it should be able to get it, even if the user is fine with a, say, 100ms periodic tick. If the user have a loop with a deadline of 100ms, and if it takes two consecutive images for estimating speed in the same loop, the user will need to have a higher precision timestamps for the images. So, the driver will need a high precision timer reading for making possible to provide those timestamps... This is exactely what rt_timer_tsc() or clock_gettime(CLOCK_MONOTONIC) [or whatever ends up calling xnarch_get_cpu_tsc() from the underlying architecture] are there for. The thing is that rt_timer_read() is expected to return values compatible with the timer mode, always. But rt_timer_tsc() is there to return the most precise timestamp available from the underlying architecture, regardless of the current timing mode. If no TSC exists on x86, then it is going to be emulated (using the PIT's channel #2), but in any case, you will get a high precision timestamp, up to the best precision the architecture can provide, that is. The key issue is to acknowledge the fact that periodic ticks and precise timestamps are two _unrelated_ units. What I'm reluctant to is to try finding some artificial binding between both units, because there is none (being stable, that is). In your example above, you would be able to estimate the elapsed time using something like rt_timer_tsc(), and convert this to ticks using rt_timer_ns2ticks(rt_timer_tsc2ns(timestamp)). The main problem would be the rounding here, and work around the lack of precision the periodic mode currently exhibits due to the constant delay between timer shots. I think that you should try convincing Jan that rtdm_clock_tsc() might be a good idea to provide, instead of tweaking rtdm_clock_read() in a way which changes its underlying logic. ;o) I hope that was what you were asking for... Regards, Rodrigo. ___ Yahoo! Acesso Grátis - Internet rápida e grátis. Instale o discador agora! http://br.acesso.yahoo.com ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Philippe Gerum wrote: >... >Given the description above, just that some skin might return either >nucleus ticks or corrected timestamps to the applications, which would >in turn do some arithmetics for converting values they got from the skin >between both scales internally, and mistakenly use the result while >assuming that both scales are always in sync. In this situation, and >during a fraction of the time (i.e. the jitter), both scales might not >be in sync, and the result would be unusable. But I still can't find a real situation where the user would need these values to be in sync... >... >But maybe we are still discussing different issues actually, so it would >be useful that the core issue that triggered the discussion about >periodic mode precision be exposed again. The core issue is that: I think the driver development should be kind of independent from the user programs. That said, if the driver needs a precise timestamp, it should be able to get it, even if the user is fine with a, say, 100ms periodic tick. If the user have a loop with a deadline of 100ms, and if it takes two consecutive images for estimating speed in the same loop, the user will need to have a higher precision timestamps for the images. So, the driver will need a high precision timer reading for making possible to provide those timestamps... I hope that was what you were asking for... Regards, Rodrigo. ___ Yahoo! Acesso Grátis - Internet rápida e grátis. Instale o discador agora! http://br.acesso.yahoo.com ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Rodrigo Rosenfeld Rosas wrote: Philippe Gerum wrote: ... Given the description above, just that some skin might return either nucleus ticks or corrected timestamps to the applications, which would in turn do some arithmetics for converting values they got from the skin between both scales internally, and mistakenly use the result while assuming that both scales are always in sync. In this situation, and during a fraction of the time (i.e. the jitter), both scales might not be in sync, and the result would be unusable. But I still can't find a real situation where the user would need these values to be in sync... It's not a matter of dealing with users always doing The Right Thing, but preferably preventing people from doing the wrong one. -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Philippe Gerum wrote: > Jan Kiszka wrote: >> Philippe Gerum wrote: >> >>> ... >>> The issue that worries me - provided that we bound the adjustment offset >>> to the duration of one tick after some jittery - is that any attempt to >>> get intra-tick precision would lead to a possible discrepancy regarding >>> the elapsed time according to those two different scales, between the >>> actual count of jiffies tracked by the timer ISR on the timekeeper CPU, >>> and the corrected time value returned by rtdm_read_clock. And this >>> discrepancy would last for the total duration of the jitter. E.g., for a >>> 100 us period, xnpod_get_time() could return 2 albeit rtdm_read_clock >>> returns 300, instead of 200. Spuriously mixing both units in >>> applications would lead to some funky chaos. >>> >> >> >> Trying to pick up this thread again, I just tried to understand your >> concerns, but failed so far to imagine a concrete scenario. Could you >> sketch such a "funky chaotic" situation from the application point of >> view? > > Given the description above, just that some skin might return either > nucleus ticks or corrected timestamps to the applications, which would > in turn do some arithmetics for converting values they got from the skin > between both scales internally, and mistakenly use the result while > assuming that both scales are always in sync. In this situation, and > during a fraction of the time (i.e. the jitter), both scales might not > be in sync, and the result would be unusable. This said, this kind of > issue could be solved by big fat warnings in documentation, explicitely > saying that conversions between both scales might be meaningless. So the worst-case is when a user derives some relative times from two different time sources, one purely tick-base, the other improved by inter-tick TSC (when available on that arch)? Let's say the user takes timestamp t1 = rtdm_clock_read() (via some driver) and a bit later t2 = rt_timer_tick2ns(rt_timer_read()). t1 was set to the last tick Tn + the number of TSC ticks since then: t1 = Tn * tick_period + TSC_offset With, e.g., Tn=1001, tick_period = 1000 us, and TSC_offset = 589 us: t1 = 1001 * 1000 us + 589 us = 1001569 us As the next tick may have not stroke yet when taking t2, that value converted to us can be smaller: t2 = Tn * tick_period = 1001000 Now the difference between t2 and t1 becomes negative (-589 us), although the user may expect it t2-t1 >= 0. Is this non-monotony your concern? On the other hand, the advantage of TSC-based synchronised inter-tick timestamps is that you can do things like sleep_until(rt_timer_ns2ticks(rtdm_clock_read() + 100)) without risking an error beyond +/- 1 tick (+jitter). With current jiffies vs. TSC in periodic mode, this is not easily possible. You have to sync in the application, creating another error source when the delay between acquiring the TSC and sync'ing the TSC on jiffies is too long. > > And what would prevent us from improving the accuracy of other >> timestamping API functions beyond RTDM as well, e.g. on converting from >> ticks to nanos in rt_timer_ticks2ns()? >> > > I don't understand why rt_timer_ticks2ns() should be impacted by such > extension. This service must keep a constant behaviour, regardless of > any outstanding timing issue. I mean, 3 ticks from a 1Khz clock rate > must always return 3,000,000 nanos, unless you stop passing count of > ticks but fractional/compound values instead. Forget about this, it was (pre-lunch) nonsense. > > The bottom-line is that we should not blur the line between periodic and > aperiodic timing modes, just for getting precise timestamps in the > former case. Additionally, and x86-wise, when no TSC is available on the > target system, rt_timer_tsc() already returns a timestamp obtained from > the 8254's channel #2 we use as a free running counter, which is the > most precise source we have at hand to do so. > > Periodic mode bears its own limitation, which is basically a loss of > accuracy we trade against a lower overhead (even if that does not mean > much except perhaps on x86). What we could do is reducing the jittery > involved in periodic ticks, by always emulating periodic mode over > aperiodic shots instead of using e.g. the 8254 in PIT mode (and remove > the need for the double scale on x86, tsc + 8254 channel #1), but not > change the basic meaning of periodic timing. Hmm, interesting, and it also reminds of a long pending (slightly OT) question I have: why not creating the infrastructure (a dedicated periodic timer) for providing round-robin scheduling even in aperiodic mode? > > But maybe we are still discussing different issues actually, so it would > be useful that the core issue that triggered the discussion about > periodic mode precision be exposed again. Yep, Rodrigo...? Jan signature.asc Description: OpenPGP digital signature ___ Xenomai-core mailing list
Re: [Xenomai-core] Synchronising TSC and periodic timer
Jan Kiszka wrote: Philippe Gerum wrote: ... The issue that worries me - provided that we bound the adjustment offset to the duration of one tick after some jittery - is that any attempt to get intra-tick precision would lead to a possible discrepancy regarding the elapsed time according to those two different scales, between the actual count of jiffies tracked by the timer ISR on the timekeeper CPU, and the corrected time value returned by rtdm_read_clock. And this discrepancy would last for the total duration of the jitter. E.g., for a 100 us period, xnpod_get_time() could return 2 albeit rtdm_read_clock returns 300, instead of 200. Spuriously mixing both units in applications would lead to some funky chaos. Trying to pick up this thread again, I just tried to understand your concerns, but failed so far to imagine a concrete scenario. Could you sketch such a "funky chaotic" situation from the application point of view? Given the description above, just that some skin might return either nucleus ticks or corrected timestamps to the applications, which would in turn do some arithmetics for converting values they got from the skin between both scales internally, and mistakenly use the result while assuming that both scales are always in sync. In this situation, and during a fraction of the time (i.e. the jitter), both scales might not be in sync, and the result would be unusable. This said, this kind of issue could be solved by big fat warnings in documentation, explicitely saying that conversions between both scales might be meaningless. And what would prevent us from improving the accuracy of other timestamping API functions beyond RTDM as well, e.g. on converting from ticks to nanos in rt_timer_ticks2ns()? I don't understand why rt_timer_ticks2ns() should be impacted by such extension. This service must keep a constant behaviour, regardless of any outstanding timing issue. I mean, 3 ticks from a 1Khz clock rate must always return 3,000,000 nanos, unless you stop passing count of ticks but fractional/compound values instead. The bottom-line is that we should not blur the line between periodic and aperiodic timing modes, just for getting precise timestamps in the former case. Additionally, and x86-wise, when no TSC is available on the target system, rt_timer_tsc() already returns a timestamp obtained from the 8254's channel #2 we use as a free running counter, which is the most precise source we have at hand to do so. Periodic mode bears its own limitation, which is basically a loss of accuracy we trade against a lower overhead (even if that does not mean much except perhaps on x86). What we could do is reducing the jittery involved in periodic ticks, by always emulating periodic mode over aperiodic shots instead of using e.g. the 8254 in PIT mode (and remove the need for the double scale on x86, tsc + 8254 channel #1), but not change the basic meaning of periodic timing. But maybe we are still discussing different issues actually, so it would be useful that the core issue that triggered the discussion about periodic mode precision be exposed again. -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Philippe Gerum wrote: > ... > The issue that worries me - provided that we bound the adjustment offset > to the duration of one tick after some jittery - is that any attempt to > get intra-tick precision would lead to a possible discrepancy regarding > the elapsed time according to those two different scales, between the > actual count of jiffies tracked by the timer ISR on the timekeeper CPU, > and the corrected time value returned by rtdm_read_clock. And this > discrepancy would last for the total duration of the jitter. E.g., for a > 100 us period, xnpod_get_time() could return 2 albeit rtdm_read_clock > returns 300, instead of 200. Spuriously mixing both units in > applications would lead to some funky chaos. > Trying to pick up this thread again, I just tried to understand your concerns, but failed so far to imagine a concrete scenario. Could you sketch such a "funky chaotic" situation from the application point of view? And what would prevent us from improving the accuracy of other timestamping API functions beyond RTDM as well, e.g. on converting from ticks to nanos in rt_timer_ticks2ns()? Jan signature.asc Description: OpenPGP digital signature ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Gilles Chanteperdrix wrote: > Jan Kiszka wrote: > > Gilles Chanteperdrix wrote: > > > Jan Kiszka wrote: > > > > Likely too simple: The periodic IRQ seems to pop up on every CPU so > that > > > > the TSC could be recorded, but will this happen synchronously? At > least > > > > we will see (IRQ) jitters, and those jitters could already create in > the > > > > single-CPU case a non-monotonic clock... > > > > > > I do not know how this issue is solved in Linux, but there seem to be a > > > simple solution: before adding the tsc offset to the last tick time, > > > this tsc offset should be compared with the tick duration in tsc > > > counts; if it is greater, then replace with the tick duration in tsc. > > > > > > > Hmm, I would rather express it in absolute tsc values, i.e. always save > > the tuple (absolute_tsc, jiffies): > > > > [timer IRQ] > > new_tsc = read_rsc() - old_tsc; > > if (new_tsc < old_tsc + period_in_tsc_ticks) > >new_tsc = old_tsc + period_in_tsc; > > old_tsc = new_tsc; > > > > Disclaimer: I haven't thought about potential accuracy side effects of > > this implementation, e.g. what would happen over the long term if the > > condition is always fulfilled and executed... > > Here is what I meant: > > [timer IRQ] > irq_tsc = rdtsc(); > irq_jitter_ns = read_8254() > > [xnpod_gettime_offset] > offset_ns = tsc2ns(rdtsc() - irq_tsc()) + irq_jitter_ns > if (offset_ns > period_ns) > offset_ns = period_ns; /* Avoid non monotonic clock. */ > Ah, I see. Hmm, wouldn't this create some "resolution hole" between the time offset_ns exceeds a precise period and the time the next IRQ actually strikes? The returned timestamps would then just stick to last (irq_tsc + irq_jitter_ns) until the next update occurs. Jan ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Jan Kiszka wrote: > Gilles Chanteperdrix wrote: > > Jan Kiszka wrote: > > > Likely too simple: The periodic IRQ seems to pop up on every CPU so that > > > the TSC could be recorded, but will this happen synchronously? At least > > > we will see (IRQ) jitters, and those jitters could already create in the > > > single-CPU case a non-monotonic clock... > > > > I do not know how this issue is solved in Linux, but there seem to be a > > simple solution: before adding the tsc offset to the last tick time, > > this tsc offset should be compared with the tick duration in tsc > > counts; if it is greater, then replace with the tick duration in tsc. > > > > Hmm, I would rather express it in absolute tsc values, i.e. always save > the tuple (absolute_tsc, jiffies): > > [timer IRQ] > new_tsc = read_rsc() - old_tsc; > if (new_tsc < old_tsc + period_in_tsc_ticks) > new_tsc = old_tsc + period_in_tsc; > old_tsc = new_tsc; > > Disclaimer: I haven't thought about potential accuracy side effects of > this implementation, e.g. what would happen over the long term if the > condition is always fulfilled and executed... Here is what I meant: [timer IRQ] irq_tsc = rdtsc(); irq_jitter_ns = read_8254() [xnpod_gettime_offset] offset_ns = tsc2ns(rdtsc() - irq_tsc()) + irq_jitter_ns if (offset_ns > period_ns) offset_ns = period_ns; /* Avoid non monotonic clock. */ -- Gilles Chanteperdrix. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Gilles Chanteperdrix wrote: Philippe Gerum wrote: > At worst, you would see an old timestamp from a previous shot while the timer IRQ > announcing the most accurate one is still outstanding but untaken, but I think > that you would still have something behaving in a monotonic way though. > > > Does anyone ever studied if and how Linux synchronises across CPUs? > > There was some activity around the problematic AMD64 multicores, but I > > haven't looked at the details and if it's actually solved now. > > > > Only once during boot AFAICT, see arch/i386/kernel/smpboot.c. This said, TSC > synchronization would not work on NUMA boxen. I think Jan is talking about using TSC to get intra-ticks precise clock, by adding tsc offsets to the time derived from the clock irqs count. This would allow, for example, to run the "latency" test with the timer set in periodic mode. The issue with non-monotonic values happens if two clock interrupts are distant from a bit more than one tick, because of the jitter. Reading the time just before the second irq then yield a greater value than the one read just after the second irq. The issue that worries me - provided that we bound the adjustment offset to the duration of one tick after some jittery - is that any attempt to get intra-tick precision would lead to a possible discrepancy regarding the elapsed time according to those two different scales, between the actual count of jiffies tracked by the timer ISR on the timekeeper CPU, and the corrected time value returned by rtdm_read_clock. And this discrepancy would last for the total duration of the jitter. E.g., for a 100 us period, xnpod_get_time() could return 2 albeit rtdm_read_clock returns 300, instead of 200. Spuriously mixing both units in applications would lead to some funky chaos. -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Fillod Stephane wrote: > Jan Kiszka wrote: >> Gilles Chanteperdrix wrote: >>> Jan Kiszka wrote: >>> > Likely too simple: The periodic IRQ seems to pop up on every CPU > so that >>> > the TSC could be recorded, but will this happen synchronously? At > least >>> > we will see (IRQ) jitters, and those jitters could already create > in the >>> > single-CPU case a non-monotonic clock... >>> >>> I do not know how this issue is solved in Linux, but there seem to be > a >>> simple solution: before adding the tsc offset to the last tick time, >>> this tsc offset should be compared with the tick duration in tsc >>> counts; if it is greater, then replace with the tick duration in tsc. >>> >> Hmm, I would rather express it in absolute tsc values, i.e. always save >> the tuple (absolute_tsc, jiffies): >> >> [timer IRQ] >> new_tsc = read_rsc() - old_tsc; >> if (new_tsc < old_tsc + period_in_tsc_ticks) >> new_tsc = old_tsc + period_in_tsc; >> old_tsc = new_tsc; >> >> Disclaimer: I haven't thought about potential accuracy side effects of >> this implementation, e.g. what would happen over the long term if the >> condition is always fulfilled and executed... > > Aha! the wonderful world of PLL's :-) > Yes! I was waiting for someone to throw this in ;). But do we really need them, or are there so cheap implementations that the overhead doesn't matter? Jan signature.asc Description: OpenPGP digital signature ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Gilles Chanteperdrix wrote: > Jan Kiszka wrote: > > Likely too simple: The periodic IRQ seems to pop up on every CPU so that > > the TSC could be recorded, but will this happen synchronously? At least > > we will see (IRQ) jitters, and those jitters could already create in the > > single-CPU case a non-monotonic clock... > > I do not know how this issue is solved in Linux, but there seem to be a > simple solution: before adding the tsc offset to the last tick time, > this tsc offset should be compared with the tick duration in tsc > counts; if it is greater, then replace with the tick duration in tsc. > Hmm, I would rather express it in absolute tsc values, i.e. always save the tuple (absolute_tsc, jiffies): [timer IRQ] new_tsc = read_rsc() - old_tsc; if (new_tsc < old_tsc + period_in_tsc_ticks) new_tsc = old_tsc + period_in_tsc; old_tsc = new_tsc; Disclaimer: I haven't thought about potential accuracy side effects of this implementation, e.g. what would happen over the long term if the condition is always fulfilled and executed... Jan signature.asc Description: OpenPGP digital signature ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Jan Kiszka wrote: > Likely too simple: The periodic IRQ seems to pop up on every CPU so that > the TSC could be recorded, but will this happen synchronously? At least > we will see (IRQ) jitters, and those jitters could already create in the > single-CPU case a non-monotonic clock... I do not know how this issue is solved in Linux, but there seem to be a simple solution: before adding the tsc offset to the last tick time, this tsc offset should be compared with the tick duration in tsc counts; if it is greater, then replace with the tick duration in tsc. -- Gilles Chanteperdrix. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Philippe Gerum wrote: > At worst, you would see an old timestamp from a previous shot while the > timer IRQ > announcing the most accurate one is still outstanding but untaken, but I > think > that you would still have something behaving in a monotonic way though. > > > Does anyone ever studied if and how Linux synchronises across CPUs? > > There was some activity around the problematic AMD64 multicores, but I > > haven't looked at the details and if it's actually solved now. > > > > Only once during boot AFAICT, see arch/i386/kernel/smpboot.c. This said, TSC > synchronization would not work on NUMA boxen. I think Jan is talking about using TSC to get intra-ticks precise clock, by adding tsc offsets to the time derived from the clock irqs count. This would allow, for example, to run the "latency" test with the timer set in periodic mode. The issue with non-monotonic values happens if two clock interrupts are distant from a bit more than one tick, because of the jitter. Reading the time just before the second irq then yield a greater value than the one read just after the second irq. -- Gilles Chanteperdrix. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Jan Kiszka wrote: Jan Kiszka wrote: Hi, for those who haven't followed the endless "RTDM and Timer functions" thread: we are currently discussing a way to provide high-resolution timestamps in periodic mode for RTDM users. It was suggested to use the TSC for this, but I noted that this source will not be in sync with the periodic system timer and may even be out of sync across multiple CPUs. A straight-forward approach to overcome this might be to record the current TSC value together with the current jiffies in xntimer_do_tick_periodic(). This tuple (per CPU) could then be provided to skin implementers in order to let them offer a high-resolution timestamp source even in periodic mode. Hmm, sounds too simple actually, so I'm waiting now for someone pointing out the pitfalls. Likely too simple: The periodic IRQ seems to pop up on every CPU so that Talking about x86, indeed it does, so that each per-CPU scheduler can have its own time feed. The basic principle is as follows on Xenomai SMP/x86: - a spare APIC vector is used to relay the timer ticks to the Xenomai domain, grabbing the APIC timer to trigger those ticks (see RTHAL_APIC_TIMER_VECTOR). - Xenomai timer ticks are programmed to occur on all CPUs, feeding the per-CPU schedulers with timing data, so that each CPU can in turn update the global scheduling state according to possible wakeups/suspensions. - when a thread needs to be awaken or suspended on a remote CPU after a local tick on the local one, an IPI is sent to invoke the rescheduling procedure on the remote CPU (see xnpod_schedule)). This also applies to Xeno's ia64 port. the TSC could be recorded, but will this happen synchronously? Nope. You could not predict the state of hardware and/or virtual interrupt masking on a particular CPU when the IRQ get raised anyway. At least we will see (IRQ) jitters, and those jitters could already create in the single-CPU case a non-monotonic clock... At worst, you would see an old timestamp from a previous shot while the timer IRQ announcing the most accurate one is still outstanding but untaken, but I think that you would still have something behaving in a monotonic way though. Does anyone ever studied if and how Linux synchronises across CPUs? There was some activity around the problematic AMD64 multicores, but I haven't looked at the details and if it's actually solved now. Only once during boot AFAICT, see arch/i386/kernel/smpboot.c. This said, TSC synchronization would not work on NUMA boxen. Jan ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Anders Blomdell wrote:
> Jan Kiszka wrote:
>> Jan Kiszka wrote:
>>
>>> Hi,
>>>
>>> for those who haven't followed the endless "RTDM and Timer functions"
>>> thread: we are currently discussing a way to provide high-resolution
>>> timestamps in periodic mode for RTDM users. It was suggested to use the
>>> TSC for this, but I noted that this source will not be in sync with the
>>> periodic system timer and may even be out of sync across multiple CPUs.
>>>
>>> A straight-forward approach to overcome this might be to record the
>>> current TSC value together with the current jiffies in
>>> xntimer_do_tick_periodic(). This tuple (per CPU) could then be provided
>>> to skin implementers in order to let them offer a high-resolution
>>> timestamp source even in periodic mode. Hmm, sounds too simple actually,
>>> so I'm waiting now for someone pointing out the pitfalls.
>>
>>
>> Likely too simple: The periodic IRQ seems to pop up on every CPU so that
>> the TSC could be recorded, but will this happen synchronously? At least
>> we will see (IRQ) jitters, and those jitters could already create in the
>> single-CPU case a non-monotonic clock...
> Returning a struct with { jiffies, cpu#, tsc, clockscaling, ... } and
> routines to compare ( lt | equal | gt | unordered ) and calculate
> differences { diff, accuracy }. And then people (of course) will send
> them over the network and compare items emanating from different systems!
>
Well, I think it's more obvious that timestamps taken on box A will not
necessarily match timestamps of box B. Moreover, tasks on box A will not
that often get migrated automatically to box B, while this can happen
inside SMP boxes if you do not explicitly bind to a certain CPU.
For the local usage I have this scheme in mind:
local_irqs_off();
read_cpu_tsc();
get_last_cpu_tsc_offset();
local_irqs_on();
calc_corrected_time();
So, nothing has to be exported to the skin user.
Or did I misunderstand what you wanted to express?
Jan
signature.asc
Description: OpenPGP digital signature
___
Xenomai-core mailing list
Xenomai-core@gna.org
https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Synchronising TSC and periodic timer
Jan Kiszka wrote: > Hi, > > for those who haven't followed the endless "RTDM and Timer functions" > thread: we are currently discussing a way to provide high-resolution > timestamps in periodic mode for RTDM users. It was suggested to use the > TSC for this, but I noted that this source will not be in sync with the > periodic system timer and may even be out of sync across multiple CPUs. > > A straight-forward approach to overcome this might be to record the > current TSC value together with the current jiffies in > xntimer_do_tick_periodic(). This tuple (per CPU) could then be provided > to skin implementers in order to let them offer a high-resolution > timestamp source even in periodic mode. Hmm, sounds too simple actually, > so I'm waiting now for someone pointing out the pitfalls. Likely too simple: The periodic IRQ seems to pop up on every CPU so that the TSC could be recorded, but will this happen synchronously? At least we will see (IRQ) jitters, and those jitters could already create in the single-CPU case a non-monotonic clock... Does anyone ever studied if and how Linux synchronises across CPUs? There was some activity around the problematic AMD64 multicores, but I haven't looked at the details and if it's actually solved now. Jan signature.asc Description: OpenPGP digital signature ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
