Hello all,
Sorry it took me so long to test the patch that was released so fast. I was tied up
with a non-linux project.
On a first glance this patch seemed to remedy the problem - at least the idle rtlinux/
idle linux system does NOT get stuck on "TLB IPI wait". Yet the slightest load
(network, ftp download of 500K file) brings the system to its knee with the "stuck on
TLB IPI wait on CPU#0" messages displayed at a slow rate (NO rtl module inserted).
I have tried to enable the MTRR (Memory Type Range Register) support in "Processor
Type and parameters" configuration. It is recommended for fixing the problem with
buggy SMP BIOSes which only set MTRRs for the boot CPU and not the secondary CPUs.
Helas, the system dies even at boot time with "hda: lost interrupt" message.
The boot messages have the following entries (same as reported by Surya):
Apr 3 11:40:06 dualpro kernel: RTL started
Apr 3 11:40:06 dualpro kernel: Unable to handle kernel NULL pointer dereference at
virtual address 00000000
........
Apr 3 11:40:06 dualpro kernel: stuck on TLB IPI wait (CPU#0)
Apr 3 11:40:06 dualpro kernel: stuck on TLB IPI wait (CPU#0)
I've tried rtai-1.2 with linux2.2.14 kernel. I do not see any "TLB IPI wait" messages
(as Paolo explained, RTAI's way of doing it is now to leave tlb invalidate IPIs as
hard interrupts). But when I load rtai modules with ldmod script, I get a very similar
message "Unable to handle kernel NULL pointer dereference at virtual address
00000000", and system becomes very unstable (for sure dies when trying rtai example
stress).
I have used a copyto script for patching, make calibrate, and used spm apic scheduler
(make instapic).
All this tells me that something is wrong with my hardware. The motherboard Intel
Buckeye B440FX DP is "fully MPS 1.4 compliant with appropriate Pentium Pro extensions"
(though linux boot messages show version 1.1 compatibility ???). Processors are not
overclocked (200 MHz). The motherboard is not very modern and probably not very many
rtlinux/rtai users will have it - it has a separate CPU "Britanny" board, PCI-to-PCI
bridge and requires a special power supply (not ATX nor AT style).
I am sure both rtlinux and rtai were successfully tested on SMP systems. It would be
nice to know some examples of SMP system configuration that work great with rtlinux /
rtai (Pentium Pro and Pentium II/ III).
Thanks,
Sergey.
----- Original Message -----
From: <[EMAIL PROTECTED]>
To: Surya <[EMAIL PROTECTED]>
Cc: <[EMAIL PROTECTED]>; RT-Linux <[EMAIL PROTECTED]>
Sent: Friday, March 31, 2000 8:58 AM
Subject: Re: [rtl] SMP error
> Anyone who would like to test a new patch for this problem can find
> a patch against 2.2.14 called test_tlb_fix_patch.2.1.14 in the v2 ftp
> directory
>
> On Fri, Mar 31, 2000 at 01:30:48AM -0800, Surya wrote:
> > Hi all,
> > Sorry my chipset is PR440FX (not 440LX as mentioned previously). Same
> > results on upgrading to 2.2.14+RTLV2.2. I have observed on my machine a common
> > error message of "kernel NULL pointer dereference....." just after the "RTL
> > started" message in the kernel log messages file (/var/log/messages). Kernel
> > is getting stuck even while it is booting. I have attached a snapshot of one
> > of the booting sequence messages from the /var/log/messages file. You can see
> > the "stuck on TLB IPI wait (CPU#0)" message in the file a couple of times
> > while booting. Hope this helps in getting at a solution.
> >
> > I hope there are people successfully using a SMP machine. I would like to
> > know their configuration and the kernel versions they are using. I am unable
> > to run RTLinux on my machine at all. Hoping to hear for your solutions......
> >
> > Surya.
> >
> > [EMAIL PROTECTED] wrote:
> >
> > > Try updating to 2.2. But you can cause this warning with long RT
> > > interrupt handlers or tasks. The problem is that Linux on CPU_A is
> > > sending a message to Linux on CPU_B asking it to synchronize on clearing
> > > the memory management TLB cache. Linux on CPU_A then waits for CPU_B to
> > > ack. If CPU_B is busy doing some RT activity, it will not ack in time and
> > > CPU_A will complain. If the warning is causing real problems, tell me and
> > > I'll fix it: there is a easy fix, but we have not yet put it in.
>
> > Feb 23 18:34:12 localhost syslogd 1.3-3: restart.
> > Feb 23 18:34:12 localhost syslog: syslogd startup succeeded
> > Feb 23 18:34:12 localhost kernel: klogd 1.3-3, log source = /proc/kmsg started.
> > Feb 23 18:34:12 localhost kernel: Inspecting /boot/System.map
> > Feb 23 18:34:12 localhost syslog: klogd startup succeeded
> > Feb 23 18:34:12 localhost kernel: Loaded 7533 symbols from /boot/System.map.
> > Feb 23 18:34:12 localhost kernel: Symbols match kernel version 2.2.13.
> > Feb 23 18:34:12 localhost kernel: No module symbols loaded.
> > Feb 23 18:34:12 localhost kernel: Linux version 2.2.13-rtl2.0 (root@surya) (gcc
>version egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)) #14 SMP Wed Feb 23 18:17:34
>PST 2000
> > Feb 23 18:34:12 localhost kernel: Intel MultiProcessor Specification v1.4
> > Feb 23 18:34:12 localhost kernel: Virtual Wire compatibility mode.
> > Feb 23 18:34:12 localhost kernel: OEM ID: INTEL Product ID: PR440FX APIC
>at: 0xFEC08000
> > Feb 23 18:34:12 localhost kernel: Processor #0 Pentium(tm) Pro APIC version 17
> > Feb 23 18:34:12 localhost kernel: Processor #12 Pentium(tm) Pro APIC version 17
> > Feb 23 18:34:12 localhost kernel: I/O APIC #13 Version 17 at 0xFEC00000.
> > Feb 23 18:34:12 localhost kernel: Processors: 2
> > Feb 23 18:34:12 localhost kernel: mapped APIC to ffffe000 (fec08000)
> > Feb 23 18:34:12 localhost kernel: mapped IOAPIC to ffffd000 (fec00000)
> > Feb 23 18:34:12 localhost kernel: Detected 198666647 Hz processor.
> > Feb 23 18:34:12 localhost kernel: Console: colour VGA+ 80x25
> > Feb 23 18:34:12 localhost kernel: Calibrating delay loop... 198.25 BogoMIPS
> > Feb 23 18:34:12 localhost kernel: Memory: 127396k/131072k available (1172k kernel
>code, 416k reserved, 2040k data, 48k init)
> > Feb 23 18:34:12 localhost kernel: Checking 386/387 coupling... OK, FPU using
>exception 16 error reporting.
> > Feb 23 18:34:12 localhost kernel: Checking 'hlt' instruction... OK.
> > Feb 23 18:34:12 localhost kernel: POSIX conformance testing by UNIFIX
> > Feb 23 18:34:12 localhost kernel: mtrr: v1.35a (19990819) Richard Gooch
>([EMAIL PROTECTED])
> > Feb 23 18:34:12 localhost kernel: per-CPU timeslice cutoff: 50.16 usecs.
> > Feb 23 18:34:12 localhost kernel: CPU0: Intel Pentium Pro stepping 09
> > Feb 23 18:34:12 localhost kernel: calibrating APIC timer ...
> > Feb 23 18:34:12 localhost kernel: ..... CPU clock speed is 198.6717 MHz.
> > Feb 23 18:34:12 localhost kernel: ..... system bus clock speed is 66.2236 MHz.
> > Feb 23 18:34:12 localhost kernel: Booting processor 12 eip 2000
> > Feb 23 18:34:12 localhost kernel: Calibrating delay loop... 198.25 BogoMIPS
> > Feb 23 18:34:13 localhost kernel: OK.
> > Feb 23 18:34:13 localhost kernel: CPU12: Intel Pentium Pro stepping 09
> > Feb 23 18:34:13 localhost kernel: Total of 2 processors activated (396.49
>BogoMIPS).
> > Feb 23 18:34:13 localhost kernel: enabling symmetric IO mode... ...done.
> > Feb 23 18:34:13 localhost kernel: ENABLING IO-APIC IRQs
> > Feb 23 18:34:13 localhost kernel: init IO_APIC IRQs
> > Feb 23 18:34:13 localhost kernel: IO-APIC pin 0, 2, 10, 11, 13, 16, 19, 20, 21,
>22, 23 not connected.
> > Feb 23 18:34:13 localhost kernel: ...trying to set up timer as ExtINT... .. (found
>pin 0) ... works.
> > Feb 23 18:34:13 localhost kernel: number of MP IRQ sources: 15.
> > Feb 23 18:34:13 localhost kernel: number of IO-APIC registers: 24.
> > Feb 23 18:34:13 localhost kernel: testing the IO APIC.......................
> > Feb 23 18:34:13 localhost kernel: .... register #00: 0D000000
> > Feb 23 18:34:13 localhost kernel: ....... : physical APIC id: 0D
> > Feb 23 18:34:13 localhost kernel: .... register #01: 00170011
> > Feb 23 18:34:13 localhost kernel: ....... : max redirection entries: 0017
> > Feb 23 18:34:13 localhost kernel: ....... : IO APIC version: 0011
> > Feb 23 18:34:13 localhost kernel: .... register #02: 00000000
> > Feb 23 18:34:13 localhost kernel: ....... : arbitration: 00
> > Feb 23 18:34:13 localhost kernel: .... IRQ redirection table:
> > Feb 23 18:34:13 localhost kernel: NR Log Phy Mask Trig IRR Pol Stat Dest Deli
>Vect:
> > Feb 23 18:34:13 localhost kernel: 00 000 00 0 0 0 0 0 0 7 61
> > Feb 23 18:34:13 localhost kernel: 01 000 00 0 0 0 0 0 1 1 69
> > Feb 23 18:34:13 localhost kernel: 02 000 00 1 0 0 0 0 0 0 00
> > Feb 23 18:34:13 localhost kernel: 03 000 00 0 0 0 0 0 1 1 71
> > Feb 23 18:34:13 localhost kernel: 04 000 00 0 0 0 0 0 1 1 79
> > Feb 23 18:34:13 localhost kernel: 05 000 00 0 0 0 0 0 1 1 81
> > Feb 23 18:34:13 localhost kernel: 06 000 00 0 0 0 0 0 1 1 89
> > Feb 23 18:34:13 localhost kernel: 07 000 00 0 0 0 0 0 1 1 91
> > Feb 23 18:34:13 localhost kernel: 08 000 00 0 0 0 0 0 1 1 99
> > Feb 23 18:34:13 localhost kernel: 09 000 00 0 0 0 0 0 1 1 A1
> > Feb 23 18:34:13 localhost kernel: 0a 000 00 1 0 0 0 0 0 0 00
> > Feb 23 18:34:13 localhost kernel: 0b 000 00 1 0 0 0 0 0 0 00
> > Feb 23 18:34:13 localhost kernel: 0c 000 00 0 0 0 0 0 1 1 A9
> > Feb 23 18:34:13 localhost kernel: 0d 000 00 1 0 0 0 0 0 0 00
> > Feb 23 18:34:13 localhost kernel: 0e 000 00 0 0 0 0 0 1 1 B1
> > Feb 23 18:34:13 localhost kernel: 0f 000 00 0 0 0 0 0 1 1 B9
> > Feb 23 18:34:13 localhost kernel: 10 000 00 1 0 0 0 0 0 0 00
> > Feb 23 18:34:13 localhost kernel: 11 0FF 0F 1 1 0 1 0 1 1 C1
> > Feb 23 18:34:13 localhost kernel: 12 0FF 0F 1 1 0 1 0 1 1 C9
> > Feb 23 18:34:13 localhost kernel: 13 000 00 1 0 0 0 0 0 0 00
> > Feb 23 18:34:13 localhost kernel: 14 000 00 1 0 0 0 0 0 0 00
> > Feb 23 18:34:13 localhost kernel: 15 000 00 1 0 0 0 0 0 0 00
> > Feb 23 18:34:13 localhost kernel: 16 000 00 1 0 0 0 0 0 0 00
> > Feb 23 18:34:13 localhost kernel: 17 000 00 1 0 0 0 0 0 0 00
> > Feb 23 18:34:13 localhost kernel: .................................... done.
> > Feb 23 18:34:13 localhost kernel: RTL started
> > Feb 23 18:34:13 localhost kernel: Unable to handle kernel NULL pointer dereference
>at virtual address 00000000
> > Feb 23 18:34:13 localhost kernel: current->tss.cr3 = 00101000, %cr3 = 00101000
> > Feb 23 18:34:13 localhost kernel: *pde = 00000000
> > Feb 23 18:34:13 localhost kernel: Oops: 0002
> > Feb 23 18:34:13 localhost kernel: CPU: 12
> > Feb 23 18:34:13 localhost kernel: EIP: 0010:[<00000000>]
> > Feb 23 18:34:13 localhost kernel: EFLAGS: 00010087
> > Feb 23 18:34:13 localhost kernel: eax: 00000000 ebx: 0000000c ecx: c7ffa000
>edx: 00000018
> > Feb 23 18:34:13 localhost kernel: esi: 00000041 edi: c7ffbf7c ebp: c7ffbf74
>esp: c7ffbf5c
> > Feb 23 18:34:13 localhost kernel: ds: 0018 es: 0018 ss: 0018
> > Feb 23 18:34:13 localhost kernel: Process swapper (pid: 0, process nr: 1,
>stackpage=c7ffb000)
> > Feb 23 18:34:13 localhost kernel: Stack: c7ffa000 c7ffa000 c027fca0 00000041
>c7ffbf90 00000000 00000000 PCI: PCI BIOS revision 2.10 entry at 0xfd9a1
> > Feb 23 18:34:13 localhost kernel: c010ab8c PCI: Using configuration type 1
> > Feb 23 18:34:13 localhost kernel:
> > Feb 23 18:34:13 localhost kernel: PCI: Probing PCI hardware
> > Feb 23 18:34:13 localhost kernel: c7ffa000 c7ffa000 00000002 c7ffa000 c027fca0
>00000000 00000600 00000018
> > Feb 23 18:34:13 localhost kernel: 00000018 00000041 c01079a5 00000010
>00000246 00000000 00000000 00000000
> > Feb 23 18:34:13 localhost kernel: Call Trace: [common_smp_interrupt+24/48]
>[cpu_idle+61/80] [do_IRQ+69/72] [rtl_intercept+116/424] [common_interrupt+24/48]
> > Feb 23 18:34:13 localhost kernel: Code: <1>Unable to handle kernel NULL pointer
>dereference at virtual address 00000000
> > Feb 23 18:34:13 localhost kernel: current->tss.cr3 = 00101000, %cr3 = 00101000
> > Feb 23 18:34:13 localhost kernel: *pde = 00000000
> > Feb 23 18:34:13 localhost kernel: PCI->APIC IRQ transform: (B0,I6,P0) -> 18
> > Feb 23 18:34:13 localhost kernel: PCI->APIC IRQ transform: (B0,I9,P0) -> 17
> > Feb 23 18:34:13 localhost kernel: PCI->APIC IRQ transform: (B0,I17,P0) -> 18
> > Feb 23 18:34:13 localhost kernel: PCI: 00:00 [8086/1237]: Passive release enable
>(00)
> > Feb 23 18:34:13 localhost kernel: Linux NET4.0 for Linux 2.2
> > Feb 23 18:34:13 localhost kernel: Based upon Swansea University Computer Society
>NET3.039
> > Feb 23 18:34:13 localhost kernel: NET4: Unix domain sockets 1.0 for Linux NET4.0.
> > Feb 23 18:34:13 localhost kernel: NET4: Linux TCP/IP 1.0 for NET4.0
> > Feb 23 18:34:13 localhost kernel: IP Protocols: ICMP, UDP, TCP
> > Feb 23 18:34:13 localhost kernel: Starting kswapd v 1.5
> > Feb 23 18:34:13 localhost kernel: Detected PS/2 Mouse Port.
> > Feb 23 18:34:13 localhost kernel: Unable to handle kernel NULL pointer dereference
>at virtual address 00000000
> > Feb 23 18:34:13 localhost kernel: current->tss.cr3 = 00101000, %cr3 = 00101000
> > Feb 23 18:34:13 localhost kernel: *pde = 00000000
> > Feb 23 18:34:13 localhost kernel: Serial driver version 4.27 with no serial
>options enabled
> > Feb 23 18:34:13 localhost kernel: ttyS00 at 0x03f8 (irq = 4) is a 16550A
> > Feb 23 18:34:13 localhost kernel: ttyS01 at 0x02f8 (irq = 3) is a 16550A
> > Feb 23 18:34:13 localhost kernel: pty: 256 Unix98 ptys configured
> > Feb 23 18:34:13 localhost kernel: Real Time Clock Driver v1.09
> > Feb 23 18:34:13 localhost kernel: PIIX3: IDE controller on PCI bus 00 dev 39
> > Feb 23 18:34:13 localhost kernel: PIIX3: not 100% native mode: will probe irqs
>later
> > Feb 23 18:34:13 localhost kernel: ide0: BM-DMA at 0xffa0-0xffa7, BIOS
>settings: hda:pio, hdb:pio
> > Feb 23 18:34:13 localhost kernel: ide1: BM-DMA at 0xffa8-0xffaf, BIOS
>settings: hdc:pio, hdd:pio
> > Feb 23 18:34:13 localhost kernel: hdc: FX120T, ATAPI CDROM drive
> > Feb 23 18:34:13 localhost kernel: ide1 at 0x170-0x177,0x376 on irq 15
> > Feb 23 18:34:13 localhost kernel: hdc: ATAPI 12X CD-ROM drive, 256kB Cache
> > Feb 23 18:34:13 localhost kernel: Uniform CDROM driver Revision: 2.56
> > Feb 23 18:34:13 localhost kernel: Floppy drive(s): fd0 is 1.44M
> > Feb 23 18:34:13 localhost kernel: FDC 0 is a National Semiconductor PC87306
> > Feb 23 18:34:13 localhost kernel: stuck on TLB IPI wait (CPU#0)
> > Feb 23 18:34:13 localhost kernel: (scsi0) <Adaptec AIC-7880 Ultra SCSI host
>adapter> found at PCI 9/0
> > Feb 23 18:34:13 localhost kernel: (scsi0) Wide Channel, SCSI ID=7, 16/255 SCBs
> > Feb 23 18:34:13 localhost kernel: (scsi0) Downloading sequencer code... 413
>instructions downloaded
> > Feb 23 18:34:13 localhost kernel: scsi0 : Adaptec AHA274x/284x/294x
>(EISA/VLB/PCI-Fast SCSI) 5.1.20/3.2.4
> > Feb 23 18:34:13 localhost kernel: <Adaptec AIC-7880 Ultra SCSI host
>adapter>
> > Feb 23 18:34:13 localhost kernel: scsi : 1 host.
> > Feb 23 18:34:13 localhost kernel: (scsi0:0:1:0) Synchronous at 20.0 Mbyte/sec,
>offset 8.
> > Feb 23 18:34:13 localhost kernel: Vendor: SEAGATE Model: ST19101W
>Rev: 0011
> > Feb 23 18:34:13 localhost kernel: Type: Direct-Access
>ANSI SCSI revision: 02
> > Feb 23 18:34:13 localhost kernel: Detected scsi disk sda at scsi0, channel 0, id
>1, lun 0
> > Feb 23 18:34:13 localhost kernel: scsi : detected 1 SCSI disk total.
> > Feb 23 18:34:13 localhost kernel: SCSI device sda: hdwr sector= 512 bytes.
>Sectors= 17783240 [8683 MB] [8.7 GB]
> > Feb 23 18:34:14 localhost kernel: eth0: Intel EtherExpress Pro 10/100 at 0xff40,
>00:A0:C9:14:0E:CA, IRQ 18.
> > Feb 23 18:34:14 localhost kernel: Board assembly 645520-034, Physical connectors
>present: RJ45
> > Feb 23 18:34:14 localhost kernel: Primary interface chip DP83840 PHY #1.
> > Feb 23 18:34:14 localhost kernel: DP83840 specific setup, setting register 23 to
>8462.
> > Feb 23 18:34:14 localhost kernel: General self-test: passed.
> > Feb 23 18:34:14 localhost kernel: Serial sub-system self-test: passed.
> > Feb 23 18:34:14 localhost kernel: Internal registers self-test: passed.
> > Feb 23 18:34:14 localhost kernel: ROM checksum self-test: passed (0x49caa8d6).
> > Feb 23 18:34:14 localhost kernel: Receiver lock-up workaround activated.
> > Feb 23 18:34:14 localhost kernel: Partition check:
> > Feb 23 18:34:14 localhost kernel: sda: sda1 sda2 < sda5 sda6 sda7 >
> > Feb 23 18:34:14 localhost kernel: VFS: Mounted root (ext2 filesystem) readonly.
> > Feb 23 18:34:14 localhost kernel: Freeing unused kernel memory: 48k freed
> > Feb 23 18:34:14 localhost kernel: stuck on TLB IPI wait (CPU#0)
> > Feb 23 18:34:14 localhost kernel: stuck on TLB IPI wait (CPU#0)
> > Feb 23 18:34:14 localhost kernel: Adding Swap: 313228k swap-space (priority -1)
> > Feb 23 18:34:12 localhost atd: atd startup succeeded
> > Feb 23 18:34:14 localhost crond: crond startup succeeded
> > Feb 23 18:34:15 localhost rc: Starting pcmcia succeeded
> > Feb 23 18:34:15 localhost inet: inetd startup succeeded
> > Feb 23 18:34:15 localhost lpd: lpd startup succeeded
> > Feb 23 18:34:15 localhost keytable: Loading keymap: Loading system font:
> > Feb 23 18:34:15 localhost rc: Starting keytable succeeded
> > Feb 23 18:34:16 localhost sendmail: sendmail startup succeeded
> > Feb 23 18:34:16 localhost gpm: gpm startup succeeded
> > Feb 23 18:34:17 localhost xfs: xfs startup succeeded
> > Feb 23 18:34:17 localhost linuxconf: Linuxconf final setup
> > Feb 23 18:34:19 localhost rc: Starting linuxconf succeeded
> > Feb 23 18:34:30 localhost PAM_pwdb[498]: (login) session opened for user root by
>LOGIN(uid=0)
> > Feb 23 18:40:00 localhost kernel: stuck on TLB IPI wait (CPU#0)
> > Feb 23 18:40:00 localhost last message repeated 20 times
> > Feb 23 18:46:04 localhost PAM_pwdb[498]: (login) session closed for user root
> > Feb 23 18:46:06 localhost rc: Stopping keytable succeeded
> > Feb 23 18:46:07 localhost Font Server[463]: terminating
> > Feb 23 18:46:07 localhost xfs: xfs shutdown succeeded
> > Feb 23 18:46:07 localhost gpm: Shutting down gpm mouse services:
> > Feb 23 18:46:07 localhost gpm: gpm
> > Feb 23 18:46:07 localhost gpm:
> > Feb 23 18:46:07 localhost rc: Stopping gpm succeeded
> > Feb 23 18:46:08 localhost sendmail: sendmail shutdown succeeded
> > Feb 23 18:46:08 localhost inet: inetd shutdown succeeded
> > Feb 23 18:46:08 localhost atd: atd shutdown succeeded
> > Feb 23 18:46:09 localhost crond: crond shutdown succeeded
> > Feb 23 18:46:09 localhost lpd: lpd shutdown succeeded
> > Feb 23 18:46:10 localhost dd: 1+0 records in
> > Feb 23 18:46:10 localhost dd: 1+0 records out
> > Feb 23 18:46:10 localhost random: Saving random seed succeeded
> > Feb 23 18:46:10 localhost portmap: portmap shutdown succeeded
> > Feb 23 18:46:11 localhost network: Shutting down interface eth0 succeeded
> > Feb 23 18:46:11 localhost network: Disabling IPv4 automatic defragmentation
>succeeded
> > Feb 23 18:46:11 localhost kernel: Kernel logging (proc) stopped.
> > Feb 23 18:46:11 localhost kernel: Kernel log daemon terminating.
> > Feb 23 18:46:12 localhost syslog: klogd shutdown succeeded
> > Feb 23 18:46:13 localhost exiting on signal 15
>
> --
> ---------------------------------------------------------
> Victor Yodaiken
> FSMLabs: www.fsmlabs.com www.rtlinux.com
> FSMLabs is a servicemark and a service of
> VJY Associates L.L.C, New Mexico.
>
> -- [rtl] ---
> To unsubscribe:
> echo "unsubscribe rtl" | mail [EMAIL PROTECTED] OR
> echo "unsubscribe rtl <Your_email>" | mail [EMAIL PROTECTED]
> ---
> For more information on Real-Time Linux see:
> http://www.rtlinux.org/rtlinux/
>
-- [rtl] ---
To unsubscribe:
echo "unsubscribe rtl" | mail [EMAIL PROTECTED] OR
echo "unsubscribe rtl <Your_email>" | mail [EMAIL PROTECTED]
---
For more information on Real-Time Linux see:
http://www.rtlinux.org/rtlinux/
