Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
On Thu, May 12, 2011 at 02:03:34AM +0200, Gilles Chanteperdrix wrote: the issue on ixp looks like the last one to be fixed on arm. If you have time, could you try the following program? It makes a very basic test, but not having a big-endian ixp at end, I am wondering about very basic assumptions...: ~ armeb-unknown-linux-gnueabi-gcc -Wall arm.c arm.c: In function 'f': arm.c:7: warning: control reaches end of non-void function arm.c: In function 'g': arm.c:13: warning: control reaches end of non-void function and then # ./a.out f: g: If you do not have time, do you agree for a revert of the KUSER_TSC code to the one we had before, and which is working for you? I sorry about not getting back about this. I have had crunch trying to meet a deadline, but now that is over. Tomorrow I'm on the road, but next week looks good for IXP testing. And thanks, thanks, for all your good work, Richard ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
On 04/14/2011 06:42 PM, Richard Cochran wrote:
On Sun, Apr 10, 2011 at 01:22:08PM +0200, Gilles Chanteperdrix wrote:
To help debugging this, please run the kernel which crashes with I-pipe
enabled, without Xenomai, and the attached test, in order to see if the
tsc behaves correctly.
Getting back to this, I did try the test program with ipipe 2.6.35 but
without xenomai. It seems to run fine. But I only ran it for a few
minutes. The exact version was ipipe-2.6.35.9-arm-1.18-01 plus endian
fix, and I attached the config.
Hi Richard,
the issue on ixp looks like the last one to be fixed on arm. If you have
time, could you try the following program? It makes a very basic test,
but not having a big-endian ixp at end, I am wondering about very basic
assumptions...:
#include stdio.h
__attribute__((naked)) unsigned long long f(unsigned long long *x)
{
asm(ldrd r0, [r0]\n\t
mov pc, lr);
}
__attribute__((naked)) unsigned long long g(unsigned long long *x)
{
asm(ldm r0, {r0, r1}\n\t
mov pc, lr);
}
int main(void)
{
unsigned long long x = 0xULL;
printf(f: %Lx\n, f(x));
printf(g: %Lx\n, g(x));
return 0;
}
If you do not have time, do you agree for a revert of the KUSER_TSC code
to the one we had before, and which is working for you?
Regards.
--
Gilles.
___
Xenomai-core mailing list
Xenomai-core@gna.org
https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
Gilles Chanteperdrix wrote: Richard Cochran wrote: On Sun, Apr 24, 2011 at 10:15:00PM +0200, Gilles Chanteperdrix wrote: some temporary results on the benchmark here: http://www.xenomai.org/~gch/latency-at91sam9263.png The worst case latency seems not to vary much over time, it looks like it is decreasing a bit, but the differences may well be in the measurement noise. Unlocked context switch actually improves the latency. I will update the png as I get new results. Gilles, That is a very interesting graph. It might be nice to have a performance benchmarks page on the wiki, including that figure, plus the description of the test setup. I can contribute IXP and some PowerPC results. Yes, we should set this up. The xeno-test currently in the head branch is made to help doing this. Regarding the measured performance, the graphs are shaped like the letter, H (or like a Bactrian camel, with two humps). I appears to me that the peaks of the 2.4.10 lines (red, green) are standing clearly to the left of the 2.5.6 lines (blue, purple), by about 10 microseconds. I would say that 2.4.10 outperforms 2.5.6, on average. I added the average latencies (except for 2.4.10 with the I-pipe 1.14-04 patch). The peak corresponds roughly to the average lat max column in latency results. Both the I-pipe version and Xenomai version seem to contribute to the increased average latency. The unlocked context switch also improves the worst-case at the expense of the average. But there is hope, 2.6.33-1.18-03 introduces pic muting and re-enables irq in the middle of gpio demuxing, which seems to improve the average latency. Hi Richard, I jumped to conclusions too quickly. I am not sure the results I obtain really are meaningful. I am probably not doing it enough rigorously. Regards. -- Gilles. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
On Sun, Apr 24, 2011 at 10:15:00PM +0200, Gilles Chanteperdrix wrote: some temporary results on the benchmark here: http://www.xenomai.org/~gch/latency-at91sam9263.png The worst case latency seems not to vary much over time, it looks like it is decreasing a bit, but the differences may well be in the measurement noise. Unlocked context switch actually improves the latency. I will update the png as I get new results. Gilles, That is a very interesting graph. It might be nice to have a performance benchmarks page on the wiki, including that figure, plus the description of the test setup. I can contribute IXP and some PowerPC results. Regarding the measured performance, the graphs are shaped like the letter, H (or like a Bactrian camel, with two humps). I appears to me that the peaks of the 2.4.10 lines (red, green) are standing clearly to the left of the 2.5.6 lines (blue, purple), by about 10 microseconds. I would say that 2.4.10 outperforms 2.5.6, on average. The absolute worst case appears to be about the same, or perhaps slightly improved in 2.5.6. So the differences are not huge, but still I think 2.4.10 is looking better. Thanks, Richard ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
Gilles Chanteperdrix wrote: Gilles Chanteperdrix wrote: Richard Cochran wrote: On Sun, Apr 10, 2011 at 01:22:08PM +0200, Gilles Chanteperdrix wrote: To help debugging this, please run the kernel which crashes with I-pipe enabled, without Xenomai, and the attached test, in order to see if the tsc behaves correctly. Getting back to this, I did try the test program with ipipe 2.6.35 but without xenomai. It seems to run fine. But I only ran it for a few minutes. The exact version was ipipe-2.6.35.9-arm-1.18-01 plus endian fix, and I attached the config. I enabled the *third* #if-elif-else case thinking it to be the correct one to test. (see below) (...) Yes, that was the correct thing to do. I assumed that when you enabled Xenomai, the system was running, and that it frozen when you started a Xenomai application. Which is not your case at all. So, assuming I got that right, then the new tsc code passes the test. Next I'll try to get some better information about the freeze. Any other ideas? We have only one commit to inspect, that should not be so hard. Next thing to try is to see what changed in the other parts of this commit. We should look at the interrupt handler, ipipe_mach_set_dec, ipipe_mach_release_timer to see what change had such an effect. I will try to reorder the functions to have a more readable diff. You can also check if we pass the rthal_calibrate_timer function. Hi Richard, I am currently running some benchmarks on an AT91SAM9263 board, to see whether we have some performance regression. Hi Richard, some temporary results on the benchmark here: http://www.xenomai.org/~gch/latency-at91sam9263.png The worst case latency seems not to vary much over time, it looks like it is decreasing a bit, but the differences may well be in the measurement noise. Unlocked context switch actually improves the latency. I will update the png as I get new results. Regards. -- Gilles. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
On Fri, Apr 22, 2011 at 07:55:35AM +0200, Gilles Chanteperdrix wrote: I also had a look at the culprit patch, reducing it to the bare minimum (no useless whitespace changes and no function moves), and it boils down to only two differences: 1- the fact that we use the generic clocksource created by ipipe_tsc_register, which probably has a shift of 25 or 26 bits instead of the 20 bits shift of the original clocksource, and returns a 64 bits value instead of 32 bits. 2- the fact that the tsc update is done with hardware irqs off in the original patch. Thanks for looking into this for me. I tried the patch, but the result is the same as before: - Kernel freezes immediately if xenomai is not a module. - If xenomai is a module, the nucleus loads, but the first skin module freezes the machine. I will take a closer look next week to see if I can find out at least where the freeze happens. Thanks, Richard ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
Richard Cochran wrote: On Fri, Apr 22, 2011 at 07:55:35AM +0200, Gilles Chanteperdrix wrote: I also had a look at the culprit patch, reducing it to the bare minimum (no useless whitespace changes and no function moves), and it boils down to only two differences: 1- the fact that we use the generic clocksource created by ipipe_tsc_register, which probably has a shift of 25 or 26 bits instead of the 20 bits shift of the original clocksource, and returns a 64 bits value instead of 32 bits. 2- the fact that the tsc update is done with hardware irqs off in the original patch. Thanks for looking into this for me. I tried the patch, but the result is the same as before: - Kernel freezes immediately if xenomai is not a module. - If xenomai is a module, the nucleus loads, but the first skin module freezes the machine. I will take a closer look next week to see if I can find out at least where the freeze happens. Ok. It means that rthal_calibrate_timer works, the system freezes when Xenomai steals the timer. However, since the ipipe_mach_set_dec, ipipe_mach_release_timer code did not change, it does not really make sense. But in order to trace the issue, you should probably trace the calls to ipipe_mach_set_dec/ipipe_mach_acktimer to see which one is the last before the freeze. On my side, I have run benches on at91sam9263, but am not done yet, though yet I have not found a lot of differences in latencies between 2.4.10 and 2.5.6. Which version of the I-pipe patch were you running with 2.4.10? -- Gilles. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
Gilles Chanteperdrix wrote: Gilles Chanteperdrix wrote: Richard Cochran wrote: On Sun, Apr 10, 2011 at 01:22:08PM +0200, Gilles Chanteperdrix wrote: To help debugging this, please run the kernel which crashes with I-pipe enabled, without Xenomai, and the attached test, in order to see if the tsc behaves correctly. Getting back to this, I did try the test program with ipipe 2.6.35 but without xenomai. It seems to run fine. But I only ran it for a few minutes. The exact version was ipipe-2.6.35.9-arm-1.18-01 plus endian fix, and I attached the config. I enabled the *third* #if-elif-else case thinking it to be the correct one to test. (see below) (...) Yes, that was the correct thing to do. I assumed that when you enabled Xenomai, the system was running, and that it frozen when you started a Xenomai application. Which is not your case at all. So, assuming I got that right, then the new tsc code passes the test. Next I'll try to get some better information about the freeze. Any other ideas? We have only one commit to inspect, that should not be so hard. Next thing to try is to see what changed in the other parts of this commit. We should look at the interrupt handler, ipipe_mach_set_dec, ipipe_mach_release_timer to see what change had such an effect. I will try to reorder the functions to have a more readable diff. You can also check if we pass the rthal_calibrate_timer function. Hi Richard, I am currently running some benchmarks on an AT91SAM9263 board, to see whether we have some performance regression. The FCSE support in the I-pipe patch for Linux 2.6.30 needs this additional patch: http://git.xenomai.org/?p=ipipe-gch.git;a=commit;h=50e97478bd64cc28937179989d17ebc1930829b4 Otherwise you get segmentation faults when exceeding the 95 processes limit. -- Gilles. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
Gilles Chanteperdrix wrote:
Richard Cochran wrote:
On Sun, Apr 10, 2011 at 01:22:08PM +0200, Gilles Chanteperdrix wrote:
To help debugging this, please run the kernel which crashes with I-pipe
enabled, without Xenomai, and the attached test, in order to see if the
tsc behaves correctly.
Getting back to this, I did try the test program with ipipe 2.6.35 but
without xenomai. It seems to run fine. But I only ran it for a few
minutes. The exact version was ipipe-2.6.35.9-arm-1.18-01 plus endian
fix, and I attached the config.
I enabled the *third* #if-elif-else case thinking it to be the correct
one to test. (see below)
(...)
Yes, that was the correct thing to do. I assumed that when you enabled
Xenomai, the system was running, and that it frozen when you started a
Xenomai application. Which is not your case at all.
So, assuming I got that right, then the new tsc code passes the test.
Next I'll try to get some better information about the freeze. Any
other ideas?
We have only one commit to inspect, that should not be so hard. Next
thing to try is to see what changed in the other parts of this commit.
We should look at the interrupt handler, ipipe_mach_set_dec,
ipipe_mach_release_timer to see what change had such an effect. I will
try to reorder the functions to have a more readable diff. You can also
check if we pass the rthal_calibrate_timer function.
Hi Richard,
I am currently running some benchmarks on an AT91SAM9263 board, to see
whether we have some performance regression.
I also had a look at the culprit patch, reducing it to the bare minimum
(no useless whitespace changes and no function moves), and it boils down
to only two differences:
1- the fact that we use the generic clocksource created by
ipipe_tsc_register, which probably has a shift of 25 or 26 bits instead
of the 20 bits shift of the original clocksource, and returns a 64 bits
value instead of 32 bits.
2- the fact that the tsc update is done with hardware irqs off in the
original patch.
I do not think 1 causes any problem, since the tsc test works (though,
you should run the test at least the time necessary for the counter to
wrap, a few times, the wrap time should be around one minute at 60 MHz).
For 2, here is a patch you could try:
diff --git a/arch/arm/kernel/ipipe_tsc.c b/arch/arm/kernel/ipipe_tsc.c
index a9de4f9..9fdda34 100644
--- a/arch/arm/kernel/ipipe_tsc.c
+++ b/arch/arm/kernel/ipipe_tsc.c
@@ -99,6 +99,9 @@ void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo
*info)
void __ipipe_tsc_update(void)
{
+ unsigned long flags;
+
+ local_irq_save_hw(flags);
if (tsc_info.type == IPIPE_TSC_TYPE_DECREMENTER) {
unsigned cnt = *(unsigned *)tsc_info.counter_vaddr;
int offset = ipipe_tsc_value-last_cnt - cnt;
@@ -106,8 +109,10 @@ void __ipipe_tsc_update(void)
offset += 0x1;
ipipe_tsc_value-last_tsc += offset + 1;
ipipe_tsc_value-last_cnt = cnt - 1;
+ local_irq_restore_hw(flags);
return;
}
ipipe_tsc_value-last_tsc = __ipipe_tsc_get() - 1;
+ local_irq_restore_hw(flags);
}
EXPORT_SYMBOL(__ipipe_tsc_get);
--
Gilles.
___
Xenomai-core mailing list
Xenomai-core@gna.org
https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
Richard Cochran wrote: On Tue, Apr 12, 2011 at 09:17:43AM +0200, Gilles Chanteperdrix wrote: Not only that. The aim of the test is to trigger the worst case path. I suspect you can not trigger it with a 10 minutes tests. As you probably remember, I was once running Xenomai on IXP465, and the latency with Xenomai 2.4 and FCSE was around 150us. But that was FCSE best effort, no? Guaranteed, but with all the load in the noltp_hell script (that is nework load, I/O load, hackbench, etc...). -- Gilles. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
On Sun, Apr 10, 2011 at 01:22:08PM +0200, Gilles Chanteperdrix wrote:
To help debugging this, please run the kernel which crashes with I-pipe
enabled, without Xenomai, and the attached test, in order to see if the
tsc behaves correctly.
Getting back to this, I did try the test program with ipipe 2.6.35 but
without xenomai. It seems to run fine. But I only ran it for a few
minutes. The exact version was ipipe-2.6.35.9-arm-1.18-01 plus endian
fix, and I attached the config.
I enabled the *third* #if-elif-else case thinking it to be the correct
one to test. (see below)
#include stdio.h
#if 0
#include native/timer.h
#define rdtsc() rt_timer_tsc()
#define init_tsc() do { \
} while(0)
#elif 1
Changed to 0...
#else
/* The newest version */
... to use this one.
#endif
So, assuming I got that right, then the new tsc code passes the test.
Next I'll try to get some better information about the freeze. Any
other ideas?
Thanks,
Richard
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.35.9
# Wed Apr 13 20:48:58 2011
#
CONFIG_ARM=y
CONFIG_SYS_SUPPORTS_APM_EMULATION=y
CONFIG_GENERIC_GPIO=y
CONFIG_GENERIC_TIME=y
# CONFIG_ARCH_USES_GETTIMEOFFSET is not set
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_HAVE_PROC_CPU=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_HARDIRQS_SW_RESEND=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ZONE_DMA=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y
CONFIG_VECTORS_BASE=0x
CONFIG_DEFCONFIG_LIST=/lib/modules/$UNAME_RELEASE/.config
CONFIG_CONSTRUCTORS=y
#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=
CONFIG_LOCALVERSION=
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_LZO is not set
# CONFIG_SWAP is not set
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
# CONFIG_POSIX_MQUEUE is not set
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_TASKSTATS is not set
# CONFIG_AUDIT is not set
#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_TREE_PREEMPT_RCU is not set
# CONFIG_TINY_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=32
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=14
# CONFIG_CGROUPS is not set
# CONFIG_SYSFS_DEPRECATED_V2 is not set
# CONFIG_RELAY is not set
# CONFIG_NAMESPACES is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=
# CONFIG_RD_GZIP is not set
# CONFIG_RD_BZIP2 is not set
# CONFIG_RD_LZMA is not set
# CONFIG_RD_LZO is not set
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_EMBEDDED=y
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
# CONFIG_KALLSYMS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
# CONFIG_SHMEM is not set
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y
CONFIG_PERF_USE_VMALLOC=y
#
# Kernel Performance Events And Counters
#
# CONFIG_PERF_EVENTS is not set
# CONFIG_PERF_COUNTERS is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
# CONFIG_SLUB_DEBUG is not set
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
CONFIG_HAVE_OPROFILE=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
#
# GCOV-based kernel profiling
#
# CONFIG_SLOW_WORK is not set
CONFIG_HAVE_GENERIC_DMA_COHERENT=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_BLOCK=y
# CONFIG_LBDAF is not set
# CONFIG_BLK_DEV_BSG is not set
# CONFIG_BLK_DEV_INTEGRITY is not set
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_DEADLINE is not set
# CONFIG_IOSCHED_CFQ is not set
# CONFIG_DEFAULT_DEADLINE is not set
# CONFIG_DEFAULT_CFQ is not set
CONFIG_DEFAULT_NOOP=y
CONFIG_DEFAULT_IOSCHED=noop
# CONFIG_INLINE_SPIN_TRYLOCK is not set
# CONFIG_INLINE_SPIN_TRYLOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK is not set
# CONFIG_INLINE_SPIN_LOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK_IRQ is not set
# CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set
CONFIG_INLINE_SPIN_UNLOCK=y
# CONFIG_INLINE_SPIN_UNLOCK_BH is not set
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
# CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_READ_TRYLOCK is not set
# CONFIG_INLINE_READ_LOCK is not set
# CONFIG_INLINE_READ_LOCK_BH is not set
# CONFIG_INLINE_READ_LOCK_IRQ is not set
# CONFIG_INLINE_READ_LOCK_IRQSAVE is not set
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
On Tue, Apr 12, 2011 at 09:17:43AM +0200, Gilles Chanteperdrix wrote: Not only that. The aim of the test is to trigger the worst case path. I suspect you can not trigger it with a 10 minutes tests. As you probably remember, I was once running Xenomai on IXP465, and the latency with Xenomai 2.4 and FCSE was around 150us. But that was FCSE best effort, no? Richard ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
Richard Cochran wrote: On Mon, Apr 11, 2011 at 11:26:33PM +0200, Gilles Chanteperdrix wrote: Wait a minute. You are comparing results obtained after 2 or 3, or 10 minutes of runtime? I am not sure such results are meaningful. I do my benchmarks with the noltp_hell test: http://git.xenomai.org/?p=mkrootfs.git;a=blob_plain;f=tests/noltp_hell;hb=HEAD Of course, the longer the test, the more confidence you have. Not only that. The aim of the test is to trigger the worst case path. I suspect you can not trigger it with a 10 minutes tests. As you probably remember, I was once running Xenomai on IXP465, and the latency with Xenomai 2.4 and FCSE was around 150us. So, I suspect the test you show us are not really meaningful. Improving the worst case latency sometimes results in a higher average latency. Each release of the I-pipe patch is tested with the boards at my disposal, and the latency on the oldest board I have, a csb637, has been the same for a long time. But I can test again a 2.4.10 to confirm this. In any case, if you want to investigate the difference, the best tool is the I-pipe tracer. As I said, I am not sure it worked with all versions, but I can help get it working. But, when tracking down kernel freezes and toggling two component versions and a half dozen CONFIG options, sometimes you do short tests just to sample the combination. Exactly, and IMO, we should not get sidetracked in premature statements and conclusions about performance of the system when we are tracking a stability issue. Let us solve the stability issue first, then we will look into any potential performance issue, and we will do it with a proper load of the system, and with a four hours test. What compiler are you using by the way? -- Gilles. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
On Tue, Apr 12, 2011 at 09:17:43AM +0200, Gilles Chanteperdrix wrote: What compiler are you using by the way? I compiled this one myself using crosstool-ng. At the time I had first tried gcc 4.3, but you advised me that it would not work for xenomai. Target: armeb-unknown-linux-gnueabi Configured with: /home/cochran/abul-gnueabi/targets/src/gcc-4.2.4/configure --build=i486-build_pc-linux-gnu --host=i486-build_pc-linux-gnu --target=armeb-unknown-linux-gnueabi --prefix=/home/cochran/x-tools/armeb-unknown-linux-gnueabi --with-sysroot=/home/cochran/x-tools/armeb-unknown-linux-gnueabi/armeb-unknown-linux-gnueabi//sys-root --enable-languages=c,c++ --disable-multilib --with-arch=armv5te --with-cpu=xscale --with-tune=xscale --with-float=soft --with-gmp=/home/cochran/x-tools/armeb-unknown-linux-gnueabi --with-mpfr=/home/cochran/x-tools/armeb-unknown-linux-gnueabi --disable-sjlj-exceptions --enable-__cxa_atexit --with-local-prefix=/home/cochran/x-tools/armeb-unknown-linux-gnueabi/armeb-unknown-linux-gnueabi//sys-root --disable-nls --enable-threads=posix --enable-symvers=gnu --enable-c99 --enable-long-long --enable-target-optspace Thread model: posix gcc version 4.2.4 Thanks, Richard ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
Richard Cochran wrote: On Tue, Apr 12, 2011 at 09:17:43AM +0200, Gilles Chanteperdrix wrote: What compiler are you using by the way? I compiled this one myself using crosstool-ng. At the time I had first tried gcc 4.3, but you advised me that it would not work for xenomai. All codesourcery version work with 2.5.6, including the latest one which is based on gcc 4.5. Ah, but they do not support big endian. I was just asking to know if you were not using something like gcc 3.x. gcc 4.2 is OK, though it has some bug with stack unwinding which we do not care about. -- Gilles. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
Richard Cochran wrote: On Sun, Apr 10, 2011 at 01:22:08PM +0200, Gilles Chanteperdrix wrote: Also, about the performances, Xenomai 2.4 did not have the Xenomai preemptible context switches. Maybe with FCSE, it results in reduced latencies to disable this option in Xenomai 2.5. So, are you saying that XENO_HW_UNLOCKED_SWITCH=n might improve latency? The help for this option says... This option reduces interrupt latency when costly cache and TLB flushes are required to switch context, and may improve concurrency on some SMP/multi-core systems as well. You definitely want to enable that option on embedded ARM platforms. so I am confused. Is is a trade-off. With this option enabled a context switch may be interrupted, this improves interrupt latency, but at the expense of scheduling latency. It makes a lot of sense without FCSE, when each context switch implies a 200us cache flush. A bit less with FCSE in guaranteed mode. -- Gilles. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
On Sun, Apr 10, 2011 at 11:34:14AM +0200, Gilles Chanteperdrix wrote: Richard Cochran wrote: I will try xenomai 2.5 with ipipe 2.6.35 next... I meant to say, Xenomai 2.4 with ipipe 2.6.35, but this does not work because the kernel definitions have changed: include/asm-generic/xenomai/hal.h: In function 'rthal_get_cpufreq': include/asm-generic/xenomai/hal.h:168: error: 'struct ipipe_sysinfo' has no member named 'cpufreq' include/asm-generic/xenomai/hal.h: In function 'rthal_get_timerfreq': include/asm-generic/xenomai/hal.h:175: error: 'struct ipipe_sysinfo' has no member named 'archdep' But I think its not worth the effort to try that combination. Ok, please try Xenomai 2.5.6 with I-pipe for 2.6.30, in order to know if the difference comes from the I-pipe of from Xenomai. So here is what I measured. Despite all the performance enhancements added in Xenomai 2.5, it seems that 2.4 still performs better on my platform. The difference isn't huge, but it is visible. |+--++---+-+-+-+--| | Kernel | | Xeno | IPIPE | | | | | | Vers | FCSE | Vers | DEBUG | lat min | lat avg | lat max | duration | |+--++---+-+-+-+--| | v30| yes | v2.4.10| no| 8.430 | 17.310 | 63.510 | 00:02:35 | | v30| yes | v2.5.6 [1] | no| 7.260 | 32.745 | 80.775 | 00:03:09 | | v30| yes | v2.5.6 [2] | no| 7.140 | 34.950 | 78.375 | 00:03:21 | |+--++---+-+-+-+--| | v31| yes | v2.5.6 | no| 7.215 | 34.020 | 85.350 | 00:02:58 | | v33| yes | v2.5.6 | yes | 17.250 | 55.815 | 127.516 | 00:02:00 | | v35| yes | v2.5.6 | yes | 7.140 | 34.200 | 84.180 | 00:02:17 | | v35-revert | yes | v2.5.6 | no| 7.350 | 29.430 | 85.185 | 00:05:50 | | v35| no | v2.5.6 | no| 6.960 | 109.500 | 214.156 | 00:09:41 | |+--++---+-+-+-+--| [1] CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH=y [2] CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH=n Gilles, I will try your tsc test program later this week... Thanks, Richard ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
On Sat, Apr 09, 2011 at 09:50:16PM +0200, Gilles Chanteperdrix wrote: Just to have an idea where the issue come from, could you try reverting all the changes which were made on the tsc and timer? i.e. revert to the original ipipe_mach_get_tsc and ipipe_mach_set_dec? The exact commit to revert is this one: http://git.xenomai.org/?p=ipipe-gch.git;a=commitdiff;h=cbd591ed5797f105ff45e04cca3bc939388624ea;hp=a2761e92ecd573bc3a3b068d4ebb7cc06313213c Okay, with cbd591ed57 reverted, it works fine even without IPIPE_DEBUG. The performance is still about 20 usec worse than xenomai 2.4 with ipipe 2.6.30. |+--+-+---+-+-+-+--| | Kernel | | Xeno| IPIPE | | | | | | Vers | FCSE | Vers| DEBUG | lat min | lat avg | lat max | duration | |+--+-+---+-+-+-+--| | v30| yes | v2.4.10 | no| 8.430 | 17.310 | 63.510 | 00:02:35 | | v31| yes | v2.5.6 | no| 7.215 | 34.020 | 85.350 | 00:02:58 | | v33| yes | v2.5.6 | yes | 17.250 | 55.815 | 127.516 | 00:02:00 | | v35| yes | v2.5.6 | yes | 7.140 | 34.200 | 84.180 | 00:02:17 | | v35-revert | yes | v2.5.6 | no| 7.350 | 29.430 | 85.185 | 00:05:50 | | v35| no | v2.5.6 | no| 6.960 | 109.500 | 214.156 | 00:09:41 | |+--+-+---+-+-+-+--| I will try xenomai 2.5 with ipipe 2.6.35 next... Thanks, Richard ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
Richard Cochran wrote: On Sat, Apr 09, 2011 at 09:50:16PM +0200, Gilles Chanteperdrix wrote: Just to have an idea where the issue come from, could you try reverting all the changes which were made on the tsc and timer? i.e. revert to the original ipipe_mach_get_tsc and ipipe_mach_set_dec? The exact commit to revert is this one: http://git.xenomai.org/?p=ipipe-gch.git;a=commitdiff;h=cbd591ed5797f105ff45e04cca3bc939388624ea;hp=a2761e92ecd573bc3a3b068d4ebb7cc06313213c Okay, with cbd591ed57 reverted, it works fine even without IPIPE_DEBUG. The performance is still about 20 usec worse than xenomai 2.4 with ipipe 2.6.30. |+--+-+---+-+-+-+--| | Kernel | | Xeno| IPIPE | | | | | | Vers | FCSE | Vers| DEBUG | lat min | lat avg | lat max | duration | |+--+-+---+-+-+-+--| | v30| yes | v2.4.10 | no| 8.430 | 17.310 | 63.510 | 00:02:35 | | v31| yes | v2.5.6 | no| 7.215 | 34.020 | 85.350 | 00:02:58 | | v33| yes | v2.5.6 | yes | 17.250 | 55.815 | 127.516 | 00:02:00 | | v35| yes | v2.5.6 | yes | 7.140 | 34.200 | 84.180 | 00:02:17 | | v35-revert | yes | v2.5.6 | no| 7.350 | 29.430 | 85.185 | 00:05:50 | | v35| no | v2.5.6 | no| 6.960 | 109.500 | 214.156 | 00:09:41 | |+--+-+---+-+-+-+--| I will try xenomai 2.5 with ipipe 2.6.35 next... Ok, please try Xenomai 2.5.6 with I-pipe for 2.6.30, in order to know if the difference comes from the I-pipe of from Xenomai. You can also play with the various I-pipe versions, going back in the releases. Note that with FCSEv4, you probably want preemptible cache flushes with FCSE guaranteed. And of course, please do not enable FCSE_DEBUG for performances testing. You can also try and enable the I-pipe tracer. But I am not sure it works on all versions. -- Gilles. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
On Sun, Apr 10, 2011 at 01:22:08PM +0200, Gilles Chanteperdrix wrote: Also, about the performances, Xenomai 2.4 did not have the Xenomai preemptible context switches. Maybe with FCSE, it results in reduced latencies to disable this option in Xenomai 2.5. So, are you saying that XENO_HW_UNLOCKED_SWITCH=n might improve latency? The help for this option says... This option reduces interrupt latency when costly cache and TLB flushes are required to switch context, and may improve concurrency on some SMP/multi-core systems as well. You definitely want to enable that option on embedded ARM platforms. so I am confused. Thanks, Richard ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
[Xenomai-core] arm ixp: more trouble with recent xenomai
After fixing the ipipe big endian issue, I have run into some more problems on my arm ixp425. Ipipe kernel 2.6.31 with xenomai 2.5.6 boots and runs fine. Ipipe kernel .33 and .35 both boot, but freeze when xenomai is enabled. I tried building xenomai as modules, and I found that loading xeno_nucleus is okay, but the system freezes immediately on loading xeno_native or xeno_posix. I tried disabling various CONFIG options, and I found by accident that enabling IPIPE_DEBUG allows the system to run just fine. Any hints on where to go from here? BTW, I also started to compare latency results from various combinations of ipipe/xenomai. The results a summarized in the table below. For the tests, I cleared /proc/xenomai/latency to zero and ran two instances of busy.sh, each one in its own xterm, via ssh. Toward the end of each run, I stopped the scripts at let the system idle for about ten seconds, in order to get a reading of the minimum latency, too. Thanks in advance, Richard * latency |+-+---+-+-+-+--| | Kernel | Xeno| IPIPE | | | | | | Vers | Vers| DEBUG | lat min | lat avg | lat max | duration | |+-+---+-+-+-+--| | v30| v2.4.10 | no| 8.430 | 17.310 | 63.510 | 00:02:35 | | v31| v2.5.6 | no| 7.215 | 34.020 | 85.350 | 00:02:58 | | v33| v2.5.6 | yes | 17.250 | 55.815 | 127.516 | 00:02:00 | | v35| v2.5.6 | yes | 7.140 | 34.200 | 84.180 | 00:02:17 | |+-+---+-+-+-+--| * busy.sh #!/bin/sh while [ 1 ] ; do find / cat /dev/mem /dev/null hackbench -g 1 -T 4 -s 1000 -l 1000 calibrator 400 1M /tmp/out done ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
On Sat, Apr 09, 2011 at 08:41:22PM +0200, Richard Cochran wrote: I tried disabling various CONFIG options, and I found by accident that enabling IPIPE_DEBUG allows the system to run just fine. Update: It is not enough for me to enable IPIPE_DEBUG. The kernels that boot have all of the XENO_OPT_DEBUG options enabled. Disabling XENO_OPT_DEBUG results in a kernel that freezes. I will try to find out which of these options makes a difference. Thanks, Richard ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
Richard Cochran wrote: On Sat, Apr 09, 2011 at 08:41:22PM +0200, Richard Cochran wrote: I tried disabling various CONFIG options, and I found by accident that enabling IPIPE_DEBUG allows the system to run just fine. Update: It is not enough for me to enable IPIPE_DEBUG. The kernels that boot have all of the XENO_OPT_DEBUG options enabled. Disabling XENO_OPT_DEBUG results in a kernel that freezes. I will try to find out which of these options makes a difference. Just to have an idea where the issue come from, could you try reverting all the changes which were made on the tsc and timer? i.e. revert to the original ipipe_mach_get_tsc and ipipe_mach_set_dec? -- Gilles. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
Richard Cochran wrote: cat /dev/mem /dev/null Are you really doing this? This is not a good idea. Please do your tests without this, this alone can cause the hardware to freeze. In fact, I would recommend using the new xeno-test available in head. -- Gilles. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
Gilles Chanteperdrix wrote: Richard Cochran wrote: On Sat, Apr 09, 2011 at 08:41:22PM +0200, Richard Cochran wrote: I tried disabling various CONFIG options, and I found by accident that enabling IPIPE_DEBUG allows the system to run just fine. Update: It is not enough for me to enable IPIPE_DEBUG. The kernels that boot have all of the XENO_OPT_DEBUG options enabled. Disabling XENO_OPT_DEBUG results in a kernel that freezes. I will try to find out which of these options makes a difference. Just to have an idea where the issue come from, could you try reverting all the changes which were made on the tsc and timer? i.e. revert to the original ipipe_mach_get_tsc and ipipe_mach_set_dec? The exact commit to revert is this one: http://git.xenomai.org/?p=ipipe-gch.git;a=commitdiff;h=cbd591ed5797f105ff45e04cca3bc939388624ea;hp=a2761e92ecd573bc3a3b068d4ebb7cc06313213c Also please try disabling the FCSE code. This one changed too... -- Gilles. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] arm ixp: more trouble with recent xenomai
On Sat, Apr 09, 2011 at 09:50:16PM +0200, Gilles Chanteperdrix wrote: Gilles Chanteperdrix wrote: Richard Cochran wrote: Update: It is not enough for me to enable IPIPE_DEBUG. The kernels that boot have all of the XENO_OPT_DEBUG options enabled. Disabling XENO_OPT_DEBUG results in a kernel that freezes. I will try to find out which of these options makes a difference. Okay, I confused myself on this one: I don't need XENO_OPT_DEBUG to boot, just IPIPE_DEBUG. Just to have an idea where the issue come from, could you try reverting all the changes which were made on the tsc and timer? i.e. revert to the original ipipe_mach_get_tsc and ipipe_mach_set_dec? The exact commit to revert is this one: http://git.xenomai.org/?p=ipipe-gch.git;a=commitdiff;h=cbd591ed5797f105ff45e04cca3bc939388624ea;hp=a2761e92ecd573bc3a3b068d4ebb7cc06313213c Will do. Also please try disabling the FCSE code. This one changed too... With IPIPE_DEBUG=n and FCSE=n it also boots. But I really need FCSE guaranteed! Thanks, Richard ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
