Problems with kernel 3.6.x (vm ?) (was : Is kernel 3.6.1 or filestreams option toxic ?)

2012-10-23 Thread Yann Dupont

Le 22/10/2012 16:14, Yann Dupont a écrit :

Hello. This mail is a follow up of a message on XFS mailing list. I had 
hang with 3.6.1, and then , damage on XFS filesystem.


3.6.1 is not alone. Tried 3.6.2, and had another hang with quite a 
different trace this time , so not really sure the 2 problems are related .
Anyway the problem is maybe not XFS, but is just a consequence of what 
seems more like kernel problems.


cc: to linux-kernel


Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.991908] 
INFO: task ceph-osd:4409 blocked for more than 120 seconds.
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.991954] 
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.991999] 
ceph-osdD 88084c049030 0 4409  1 0x
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992003]  
88084c048d60 0086 880a1421de78 880a17caa820
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992054]  
880a1421dfd8 880a1421dfd8 880a1421dfd8 88084c048d60
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992105]  
03373001 88084c048d60 88051775cb20 
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992156] 
Call Trace:
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992184]  
[] ? rwsem_down_failed_common+0xbd/0x150
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992215]  
[] ? call_rwsem_down_write_failed+0x13/0x20
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992248]  
[] ? cap_mmap_addr+0x50/0x50
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992275]  
[] ? down_write+0x1c/0x1d
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992303]  
[] ? vm_mmap_pgoff+0x64/0xb0
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992331]  
[] ? sys_mmap_pgoff+0x5c/0x190
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992360]  
[] ? do_sys_open+0x161/0x1e0
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992387]  
[] ? system_call_fastpath+0x1a/0x1f
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992423] 
INFO: task ceph-osd:25297 blocked for more than 120 seconds.
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992451] 
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992495] 
ceph-osdD 8801bce7b1a0 0 25297  1 0x
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992497]  
8801bce7aed0 0086 88025d903fd8 880a17cab580
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992548]  
88025d903fd8 88025d903fd8 88025d903fd8 8801bce7aed0
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992599]  
8801bce7aed0 8801bce7aed0 88051775cb20 
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992650] 
Call Trace:
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992673]  
[] ? rwsem_down_failed_common+0xbd/0x150
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992702]  
[] ? call_rwsem_down_read_failed+0x14/0x30
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992732]  
[] ? down_read+0xe/0x10
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992759]  
[] ? do_page_fault+0x16c/0x460
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992787]  
[] ? release_sock+0xd2/0x150
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992815]  
[] ? inet_stream_connect+0x4b/0x70
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992844]  
[] ? sys_connect+0xa5/0xe0
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992871]  
[] ? fd_install+0x33/0x70
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992898]  
[] ? page_fault+0x25/0x30
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992925] 
INFO: task ceph-osd:32469 blocked for more than 120 seconds.
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992953] 
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992996] 
ceph-osdD 880556237b30 0 32469  1 0x
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992999]  
880556237860 0086 88059fe5dfd8 880a17c742e0
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.993050]  
88059fe5dfd8 88059fe5dfd8 88059fe5dfd8 880556237860
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.993101]  
880556237860 880556237860 88051775cb20 
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.993153] 
Call 

Problems with kernel 3.6.x (vm ?) (was : Is kernel 3.6.1 or filestreams option toxic ?)

2012-10-23 Thread Yann Dupont

Le 22/10/2012 16:14, Yann Dupont a écrit :

Hello. This mail is a follow up of a message on XFS mailing list. I had 
hang with 3.6.1, and then , damage on XFS filesystem.


3.6.1 is not alone. Tried 3.6.2, and had another hang with quite a 
different trace this time , so not really sure the 2 problems are related .
Anyway the problem is maybe not XFS, but is just a consequence of what 
seems more like kernel problems.


cc: to linux-kernel


Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.991908] 
INFO: task ceph-osd:4409 blocked for more than 120 seconds.
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.991954] 
echo 0  /proc/sys/kernel/hung_task_timeout_secs disables this message.
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.991999] 
ceph-osdD 88084c049030 0 4409  1 0x
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992003]  
88084c048d60 0086 880a1421de78 880a17caa820
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992054]  
880a1421dfd8 880a1421dfd8 880a1421dfd8 88084c048d60
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992105]  
03373001 88084c048d60 88051775cb20 
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992156] 
Call Trace:
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992184]  
[813c52fd] ? rwsem_down_failed_common+0xbd/0x150
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992215]  
[812094a3] ? call_rwsem_down_write_failed+0x13/0x20
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992248]  
[811b83e0] ? cap_mmap_addr+0x50/0x50
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992275]  
[813c3cbc] ? down_write+0x1c/0x1d
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992303]  
[810fcf74] ? vm_mmap_pgoff+0x64/0xb0
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992331]  
[8110d4cc] ? sys_mmap_pgoff+0x5c/0x190
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992360]  
[811357f1] ? do_sys_open+0x161/0x1e0
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992387]  
[813c5ffd] ? system_call_fastpath+0x1a/0x1f
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992423] 
INFO: task ceph-osd:25297 blocked for more than 120 seconds.
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992451] 
echo 0  /proc/sys/kernel/hung_task_timeout_secs disables this message.
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992495] 
ceph-osdD 8801bce7b1a0 0 25297  1 0x
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992497]  
8801bce7aed0 0086 88025d903fd8 880a17cab580
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992548]  
88025d903fd8 88025d903fd8 88025d903fd8 8801bce7aed0
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992599]  
8801bce7aed0 8801bce7aed0 88051775cb20 
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992650] 
Call Trace:
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992673]  
[813c52fd] ? rwsem_down_failed_common+0xbd/0x150
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992702]  
[81209474] ? call_rwsem_down_read_failed+0x14/0x30
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992732]  
[813c3c9e] ? down_read+0xe/0x10
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992759]  
[8103129c] ? do_page_fault+0x16c/0x460
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992787]  
[81305862] ? release_sock+0xd2/0x150
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992815]  
[8137aceb] ? inet_stream_connect+0x4b/0x70
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992844]  
[81302b55] ? sys_connect+0xa5/0xe0
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992871]  
[811343e3] ? fd_install+0x33/0x70
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992898]  
[813c5a75] ? page_fault+0x25/0x30
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992925] 
INFO: task ceph-osd:32469 blocked for more than 120 seconds.
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992953] 
echo 0  /proc/sys/kernel/hung_task_timeout_secs disables this message.
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992996] 
ceph-osdD 880556237b30 0 32469  1 0x
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.992999]  
880556237860 0086 88059fe5dfd8 880a17c742e0
Oct 22 20:54:29 braeval.u14.univ-nantes.prive kernel: [629576.993050]  
88059fe5dfd8 88059fe5dfd8 88059fe5dfd8

Re: E1000 - page allocation failure - saga continues :(

2005-04-20 Thread Yann Dupont
Lukas Hejtmanek a écrit :

>On Tue, Apr 19, 2005 at 09:23:46AM +0200, Yann Dupont wrote:
>  
>
>>Do you have turned NAPI on ??? I tried without it off on e1000 and ...
>>surprise !
>>Don't have any messages since 12H now (usually I got those in less than 1H)
>>
>>
>
>I have NAPI on. I tried to turn it off but my test failed, I can see allocation
>failure again.
>
>  
>
Well. forgives me :)
I have re turned NAPI On and my box is still happy 19H later...

So it's obviously not napi.

The problem is beetween the 2 incarnations of kernel (2.6.11.7 with
kswapd meesages on thoses who works well), I've changed some more options
Not exactly the best way to track bugs :(

Anyway i'll try to catch THE option that make the kernel not so happy
under heavy stress. Stay tuned,

-- 
Yann Dupont, Cri de l'université de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [EMAIL PROTECTED]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: E1000 - page allocation failure - saga continues :(

2005-04-20 Thread Yann Dupont
Lukas Hejtmanek a crit :

On Tue, Apr 19, 2005 at 09:23:46AM +0200, Yann Dupont wrote:
  

Do you have turned NAPI on ??? I tried without it off on e1000 and ...
surprise !
Don't have any messages since 12H now (usually I got those in less than 1H)



I have NAPI on. I tried to turn it off but my test failed, I can see allocation
failure again.

  

Well. forgives me :)
I have re turned NAPI On and my box is still happy 19H later...

So it's obviously not napi.

The problem is beetween the 2 incarnations of kernel (2.6.11.7 with
kswapd meesages on thoses who works well), I've changed some more options
Not exactly the best way to track bugs :(

Anyway i'll try to catch THE option that make the kernel not so happy
under heavy stress. Stay tuned,

-- 
Yann Dupont, Cri de l'universit de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [EMAIL PROTECTED]

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: E1000 - page allocation failure - saga continues :(

2005-04-19 Thread Yann Dupont
Nick Piggin a Ãcrit :

>
>>Do you have turned NAPI on ??? I tried without it off on e1000 and ...
>>surprise !
>>Don't have any messages since 12H now (usually I got those in less than 1H)
>>
>>
>>
>
>Possibly kswapd might be unable to get enough CPU to free memory.
>
>  
>
Ok, so what you're saying is that turning NAPI off is just slowing down
things enough to not be hit by
this problem , right ?


-- 
Yann Dupont, Cri de l'università de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [EMAIL PROTECTED]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: E1000 - page allocation failure - saga continues :(

2005-04-19 Thread Yann Dupont
Lukas Hejtmanek a écrit :

>On Mon, Apr 18, 2005 at 02:10:31PM +0200, Yann Dupont wrote:
>  
>
>>I have those problems too. The (temporary ?) fix is to raise the
>>min_free_kb to an higher value.
>>echo 65535 > /proc/sys/vm/min_free_kbytes
>>
>>Maybe such an high value is totally silly, but at least I don't have
>>those messages.
>>
>>
>
>I know that kernel 2.6.6-bk4 works. So were there some memory manager changes
>since 2.6.6? If so it looks like there are some bugs. 
>On the other hand, ethernet driver should not allocate much memory but rather
>drop packets.
>
>Btw, are you using some TCP tweaks? E.g. I have default TCP window size 1MB.
>
>  
>
Do you have turned NAPI on ??? I tried without it off on e1000 and ...
surprise !
Don't have any messages since 12H now (usually I got those in less than 1H)

-- 
Yann Dupont, Cri de l'université de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [EMAIL PROTECTED]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: E1000 - page allocation failure - saga continues :(

2005-04-19 Thread Yann Dupont
Lukas Hejtmanek a crit :

On Mon, Apr 18, 2005 at 02:10:31PM +0200, Yann Dupont wrote:
  

I have those problems too. The (temporary ?) fix is to raise the
min_free_kb to an higher value.
echo 65535  /proc/sys/vm/min_free_kbytes

Maybe such an high value is totally silly, but at least I don't have
those messages.



I know that kernel 2.6.6-bk4 works. So were there some memory manager changes
since 2.6.6? If so it looks like there are some bugs. 
On the other hand, ethernet driver should not allocate much memory but rather
drop packets.

Btw, are you using some TCP tweaks? E.g. I have default TCP window size 1MB.

  

Do you have turned NAPI on ??? I tried without it off on e1000 and ...
surprise !
Don't have any messages since 12H now (usually I got those in less than 1H)

-- 
Yann Dupont, Cri de l'universit de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [EMAIL PROTECTED]

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: E1000 - page allocation failure - saga continues :(

2005-04-19 Thread Yann Dupont
Nick Piggin a crit :


Do you have turned NAPI on ??? I tried without it off on e1000 and ...
surprise !
Don't have any messages since 12H now (usually I got those in less than 1H)




Possibly kswapd might be unable to get enough CPU to free memory.

  

Ok, so what you're saying is that turning NAPI off is just slowing down
things enough to not be hit by
this problem , right ?


-- 
Yann Dupont, Cri de l'universit de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [EMAIL PROTECTED]

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: E1000 - page allocation failure - saga continues :(

2005-04-18 Thread Yann Dupont
Lukas Hejtmanek a écrit :

>On Mon, Apr 18, 2005 at 02:24:47PM +0200, Yann Dupont wrote:
>  
>
>>>I know that kernel 2.6.6-bk4 works. So were there some memory manager changes
>>>since 2.6.6? If so it looks like there are some bugs. 
>>>On the other hand, ethernet driver should not allocate much memory but rather
>>>drop packets.
>>>
>>>Btw, are you using some TCP tweaks? E.g. I have default TCP window size 1MB.
>>>
>>>  
>>>
>>No tweaking at all. No jumbo frames.
>>
>>
>
>There were assumptions that it is XFS related. Are you using XFS on that box?
>
>I'm able to deterministically produce this error:
>on XFS partition store a file from network using multiple threads. If file size
>is bigger then total memory, then it fails after major part of memory is used
>for a file cache.
>
>  
>
Ah yes, this is the case.
XFS all over ...

The server is quite heavily stressed, we have a bunch of servers
rsyncing on a big SAN volume - formatted with XFS, that's right.
(and, if that matters, XFS in on top of a EVMS volume (on top of a LVM2
region)...)

-- 
Yann Dupont, Cri de l'université de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [EMAIL PROTECTED]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: E1000 - page allocation failure - saga continues :(

2005-04-18 Thread Yann Dupont
Lukas Hejtmanek a écrit :

>On Mon, Apr 18, 2005 at 02:10:31PM +0200, Yann Dupont wrote:
>  
>
>>I have those problems too. The (temporary ?) fix is to raise the
>>min_free_kb to an higher value.
>>echo 65535 > /proc/sys/vm/min_free_kbytes
>>
>>Maybe such an high value is totally silly, but at least I don't have
>>those messages.
>>
>>
>
>I know that kernel 2.6.6-bk4 works. So were there some memory manager changes
>since 2.6.6? If so it looks like there are some bugs. 
>On the other hand, ethernet driver should not allocate much memory but rather
>drop packets.
>
>Btw, are you using some TCP tweaks? E.g. I have default TCP window size 1MB.
>
>  
>
No tweaking at all. No jumbo frames.


-- 
Yann Dupont, Cri de l'université de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [EMAIL PROTECTED]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: E1000 - page allocation failure - saga continues :(

2005-04-18 Thread Yann Dupont
Lukas Hejtmanek a écrit :

>Hello,
>
>today I tried 2.6.11.7 kernel with hoping that allocation failures disappear.
>Unfortunately they did not.
>
>Default min_free_kb is 3200kB.
>
>Here is stack trace:
>
>swapper: page allocation failure. order:0, mode:0x20
> [] __alloc_pages+0x2b3/0x420
> [] kmem_getpages+0x31/0xa0
> [] cache_grow+0xae/0x160
> [] ip_rcv_finish+0x0/0x280
> [] cache_alloc_refill+0x17b/0x230
> [] __kmalloc+0x88/0xa0
> [] alloc_skb+0x47/0xf0
> [] e1000_alloc_rx_buffers+0x57/0x100
> [] e1000_clean_rx_irq+0x1bf/0x4c0
>  
>
I have those problems too. The (temporary ?) fix is to raise the
min_free_kb to an higher value.
echo 65535 > /proc/sys/vm/min_free_kbytes

Maybe such an high value is totally silly, but at least I don't have
those messages.

Sincerely yours,

-- 
Yann Dupont, Cri de l'université de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [EMAIL PROTECTED]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: E1000 - page allocation failure - saga continues :(

2005-04-18 Thread Yann Dupont
Lukas Hejtmanek a crit :

Hello,

today I tried 2.6.11.7 kernel with hoping that allocation failures disappear.
Unfortunately they did not.

Default min_free_kb is 3200kB.

Here is stack trace:

swapper: page allocation failure. order:0, mode:0x20
 [c0139783] __alloc_pages+0x2b3/0x420
 [c013c4f1] kmem_getpages+0x31/0xa0
 [c013d22e] cache_grow+0xae/0x160
 [c0342b50] ip_rcv_finish+0x0/0x280
 [c013d45b] cache_alloc_refill+0x17b/0x230
 [c013d7d8] __kmalloc+0x88/0xa0
 [c0327ce7] alloc_skb+0x47/0xf0
 [c02bff97] e1000_alloc_rx_buffers+0x57/0x100
 [c02bfc3f] e1000_clean_rx_irq+0x1bf/0x4c0
  

I have those problems too. The (temporary ?) fix is to raise the
min_free_kb to an higher value.
echo 65535  /proc/sys/vm/min_free_kbytes

Maybe such an high value is totally silly, but at least I don't have
those messages.

Sincerely yours,

-- 
Yann Dupont, Cri de l'universit de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [EMAIL PROTECTED]

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: E1000 - page allocation failure - saga continues :(

2005-04-18 Thread Yann Dupont
Lukas Hejtmanek a crit :

On Mon, Apr 18, 2005 at 02:10:31PM +0200, Yann Dupont wrote:
  

I have those problems too. The (temporary ?) fix is to raise the
min_free_kb to an higher value.
echo 65535  /proc/sys/vm/min_free_kbytes

Maybe such an high value is totally silly, but at least I don't have
those messages.



I know that kernel 2.6.6-bk4 works. So were there some memory manager changes
since 2.6.6? If so it looks like there are some bugs. 
On the other hand, ethernet driver should not allocate much memory but rather
drop packets.

Btw, are you using some TCP tweaks? E.g. I have default TCP window size 1MB.

  

No tweaking at all. No jumbo frames.


-- 
Yann Dupont, Cri de l'universit de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [EMAIL PROTECTED]

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: E1000 - page allocation failure - saga continues :(

2005-04-18 Thread Yann Dupont
Lukas Hejtmanek a crit :

On Mon, Apr 18, 2005 at 02:24:47PM +0200, Yann Dupont wrote:
  

I know that kernel 2.6.6-bk4 works. So were there some memory manager changes
since 2.6.6? If so it looks like there are some bugs. 
On the other hand, ethernet driver should not allocate much memory but rather
drop packets.

Btw, are you using some TCP tweaks? E.g. I have default TCP window size 1MB.

  

No tweaking at all. No jumbo frames.



There were assumptions that it is XFS related. Are you using XFS on that box?

I'm able to deterministically produce this error:
on XFS partition store a file from network using multiple threads. If file size
is bigger then total memory, then it fails after major part of memory is used
for a file cache.

  

Ah yes, this is the case.
XFS all over ...

The server is quite heavily stressed, we have a bunch of servers
rsyncing on a big SAN volume - formatted with XFS, that's right.
(and, if that matters, XFS in on top of a EVMS volume (on top of a LVM2
region)...)

-- 
Yann Dupont, Cri de l'universit de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [EMAIL PROTECTED]

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: PATCH 2.4.4.ac9: Tulip net driver fixes

2001-05-15 Thread Yann Dupont

Le 14 May 2001 14:36:05 -0400, Jeff Garzik a écrit :
> Mads Martin Jørgensen wrote:

> Attached is a patch against 2.4.4-ac9 which includes the changes found
> in tulip-devel 1.1.6...   (tulip-devel is sort of a misnomer; right now
> it's really just a staging and testing point for fixes which go straight
> into the tulip-stable series)
> 
> I just checked against ac9 and it applies cleanly here.


Still the same issue here : On a quad port card, if only 1 port is up,
all is fine. This can be eth0, eth1, eth2 or eth3, it doesn't matter.

As soon as more than 1 port is up, the machine freeze. no oops, 
not event CTRL-Scrollback, nothing.

As the use I made of the 4 port eth is bridging, you can imagine
the freeze appears very shortly after boot ;-) 

Anyway. using the de4x5 driver for now.

Yann.

-- 
\|/  \|/ Fac. des sciences de Nantes-Linux-Python-IPv6-ATM-BONOM
"@'/ ,. \@"  Tel :(+33) [0]251125865(AM)[0]251125857(PM)[0]251125868(Fax)
/_| \__/ |_\ [EMAIL PROTECTED]
   \__U_/http://www.unantes.univ-nantes.fr/~dupont

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



Re: PATCH 2.4.4.ac9: Tulip net driver fixes

2001-05-15 Thread Yann Dupont

Le 14 May 2001 14:36:05 -0400, Jeff Garzik a écrit :
> Mads Martin Jørgensen wrote:

> Attached is a patch against 2.4.4-ac9 which includes the changes found
> in tulip-devel 1.1.6...   (tulip-devel is sort of a misnomer; right now
> it's really just a staging and testing point for fixes which go straight
> into the tulip-stable series)
> 
> I just checked against ac9 and it applies cleanly here.


Still the same issue here : On a quad port card, if only 1 port is up,
all is fine. This can be eth0, eth1, eth2 or eth3, it doesn't matter.

As soon as more than 1 port is up, the machine freeze. no oops, 
not event CTRL-Scrollback, nothing.

As the use I made of the 4 port eth is bridging, you can imagine
the freeze appears very shortly after boot ;-) 

Anyway. using the de4x5 driver for now.

Yann.

-- 
\|/  \|/ Fac. des sciences de Nantes-Linux-Python-IPv6-ATM-BONOM
"@'/ ,. \@"  Tel :(+33) [0]251125865(AM)[0]251125857(PM)[0]251125868(Fax)
/_| \__/ |_\ [EMAIL PROTECTED]
   \__U_/http://www.unantes.univ-nantes.fr/~dupont

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



Re: PATCH 2.4.4.ac9: Tulip net driver fixes

2001-05-15 Thread Yann Dupont

Le 14 May 2001 14:36:05 -0400, Jeff Garzik a écrit :
 Mads Martin Jørgensen wrote:

 Attached is a patch against 2.4.4-ac9 which includes the changes found
 in tulip-devel 1.1.6...   (tulip-devel is sort of a misnomer; right now
 it's really just a staging and testing point for fixes which go straight
 into the tulip-stable series)
 
 I just checked against ac9 and it applies cleanly here.


Still the same issue here : On a quad port card, if only 1 port is up,
all is fine. This can be eth0, eth1, eth2 or eth3, it doesn't matter.

As soon as more than 1 port is up, the machine freeze. no oops, 
not event CTRL-Scrollback, nothing.

As the use I made of the 4 port eth is bridging, you can imagine
the freeze appears very shortly after boot ;-) 

Anyway. using the de4x5 driver for now.

Yann.

-- 
\|/  \|/ Fac. des sciences de Nantes-Linux-Python-IPv6-ATM-BONOM
@'/ ,. \@  Tel :(+33) [0]251125865(AM)[0]251125857(PM)[0]251125868(Fax)
/_| \__/ |_\ [EMAIL PROTECTED]
   \__U_/http://www.unantes.univ-nantes.fr/~dupont

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



Re: PATCH 2.4.4.ac9: Tulip net driver fixes

2001-05-15 Thread Yann Dupont

Le 14 May 2001 14:36:05 -0400, Jeff Garzik a écrit :
 Mads Martin Jørgensen wrote:

 Attached is a patch against 2.4.4-ac9 which includes the changes found
 in tulip-devel 1.1.6...   (tulip-devel is sort of a misnomer; right now
 it's really just a staging and testing point for fixes which go straight
 into the tulip-stable series)
 
 I just checked against ac9 and it applies cleanly here.


Still the same issue here : On a quad port card, if only 1 port is up,
all is fine. This can be eth0, eth1, eth2 or eth3, it doesn't matter.

As soon as more than 1 port is up, the machine freeze. no oops, 
not event CTRL-Scrollback, nothing.

As the use I made of the 4 port eth is bridging, you can imagine
the freeze appears very shortly after boot ;-) 

Anyway. using the de4x5 driver for now.

Yann.

-- 
\|/  \|/ Fac. des sciences de Nantes-Linux-Python-IPv6-ATM-BONOM
@'/ ,. \@  Tel :(+33) [0]251125865(AM)[0]251125857(PM)[0]251125868(Fax)
/_| \__/ |_\ [EMAIL PROTECTED]
   \__U_/http://www.unantes.univ-nantes.fr/~dupont

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



Re: Deadlock/crash with Quad tulip card in 2.4

2001-05-11 Thread Yann Dupont

Le 10 May 2001 12:08:08 -0400, Jeff Garzik a écrit :
> Yann Dupont wrote:

> I'm working on fixing this right now; until then, the "de4x5" driver
> should work for you.
> 

Yes that's true, it's working like a charm now. I didn't know the de4x5
driver was working on 4 port Ethernet 21140...

Thanks for the quick answer.

-- 
\|/  \|/ Fac. des sciences de Nantes-Linux-Python-IPv6-ATM-BONOM
"@'/ ,. \@"  Tel :(+33) [0]251125865(AM)[0]251125857(PM)[0]251125868(Fax)
/_| \__/ |_\ [EMAIL PROTECTED]
   \__U_/http://www.unantes.univ-nantes.fr/~dupont

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



Re: Deadlock/crash with Quad tulip card in 2.4

2001-05-11 Thread Yann Dupont

Le 10 May 2001 12:08:08 -0400, Jeff Garzik a écrit :
 Yann Dupont wrote:

 I'm working on fixing this right now; until then, the de4x5 driver
 should work for you.
 

Yes that's true, it's working like a charm now. I didn't know the de4x5
driver was working on 4 port Ethernet 21140...

Thanks for the quick answer.

-- 
\|/  \|/ Fac. des sciences de Nantes-Linux-Python-IPv6-ATM-BONOM
@'/ ,. \@  Tel :(+33) [0]251125865(AM)[0]251125857(PM)[0]251125868(Fax)
/_| \__/ |_\ [EMAIL PROTECTED]
   \__U_/http://www.unantes.univ-nantes.fr/~dupont

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



Deadlock/crash with Quad tulip card in 2.4

2001-05-10 Thread Yann Dupont

Hello. I'm having problem with Quad eth100 tulip (digital 21140) cards.

The machine was acting as a bridge with kernel 2.2 very reliably.

Now, when I boot 2.4.4-ac6, the machine hangs - no oops, nothing. Just
hang.

even CTRL-Scroll lock does nothing.

I supected a bridge problem (I know 2.2  & 2.4 bridge are different) But
it's not the case as I reproduce the bug on another machine (with
another motherboard. kernel 2.4.4-ac6 where the bridge is not
configured.)

single boot is OK,

ifconfig eth0 up is ok (MII negotiation is OK)
ifconfig eth1 up crash the machine -

on another boot i tried this :

ifconfig eth0 up
ifconfig eth0 down

ifconfig eth1 up is OK
ifconfig eth1 down ... etc etc

This works as long as only one interface is up. 

If only 1 interface is up, the machine works reliably.

just putting 2 interfaces up hang the machine.

Can this be an  IRQ sharing / bridge  problem  ??? (all interface shares
the same IRQ and are after a bridge)

I don't know if this is ac-series specific. I'll try tomorrow.

Any Idea how I can test further ? Without oops it's not easy...

Yann Dupont.

-- 
\|/  \|/ Fac. des sciences de Nantes-Linux-Python-IPv6-ATM-BONOM
"@'/ ,. \@"  Tel :(+33) [0]251125865(AM)[0]251125857(PM)[0]251125868(Fax)
/_| \__/ |_\ [EMAIL PROTECTED]
   \__U_/http://www.unantes.univ-nantes.fr/~dupont

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



Deadlock/crash with Quad tulip card in 2.4

2001-05-10 Thread Yann Dupont

Hello. I'm having problem with Quad eth100 tulip (digital 21140) cards.

The machine was acting as a bridge with kernel 2.2 very reliably.

Now, when I boot 2.4.4-ac6, the machine hangs - no oops, nothing. Just
hang.

even CTRL-Scroll lock does nothing.

I supected a bridge problem (I know 2.2   2.4 bridge are different) But
it's not the case as I reproduce the bug on another machine (with
another motherboard. kernel 2.4.4-ac6 where the bridge is not
configured.)

single boot is OK,

ifconfig eth0 up is ok (MII negotiation is OK)
ifconfig eth1 up crash the machine -

on another boot i tried this :

ifconfig eth0 up
ifconfig eth0 down

ifconfig eth1 up is OK
ifconfig eth1 down ... etc etc

This works as long as only one interface is up. 

If only 1 interface is up, the machine works reliably.

just putting 2 interfaces up hang the machine.

Can this be an  IRQ sharing / bridge  problem  ??? (all interface shares
the same IRQ and are after a bridge)

I don't know if this is ac-series specific. I'll try tomorrow.

Any Idea how I can test further ? Without oops it's not easy...

Yann Dupont.

-- 
\|/  \|/ Fac. des sciences de Nantes-Linux-Python-IPv6-ATM-BONOM
@'/ ,. \@  Tel :(+33) [0]251125865(AM)[0]251125857(PM)[0]251125868(Fax)
/_| \__/ |_\ [EMAIL PROTECTED]
   \__U_/http://www.unantes.univ-nantes.fr/~dupont

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



Re: Oracle 8I & Kernel 2.4.3 : Sane ?

2001-04-03 Thread Yann Dupont

Le 02 Apr 2001 23:04:50 +0100, Alan Cox a écrit :
> > I noticed that 2.4.3 contains some fixs for shared memory -
> > So the final question IS :
> > 
> > Is oracle 8.1.5 + Kernel 2.4.3 a sane combination ?
> 
> Probably not yet but getting closer.

Ok, thanks for the quick answer. That's good news.

> 
> > In general is oracle + Kernel 2.4 working ? 
> 
> Ditto.
> 
> The shm and rawio fixes are very recent
> 

Ok. That needs some test ...

Yann Dupont.
-- 
\|/  \|/ Fac. des sciences de Nantes-Linux-Python-IPv6-ATM-BONOM
"@'/ ,. \@"  Tel :(+33) [0]251125865(AM)[0]251125857(PM)[0]251125868(Fax)
/_| \__/ |_\ [EMAIL PROTECTED]
   \__U_/http://www.unantes.univ-nantes.fr/~dupont

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



Re: Oracle 8I Kernel 2.4.3 : Sane ?

2001-04-03 Thread Yann Dupont

Le 02 Apr 2001 23:04:50 +0100, Alan Cox a crit :
  I noticed that 2.4.3 contains some fixs for shared memory -
  So the final question IS :
  
  Is oracle 8.1.5 + Kernel 2.4.3 a sane combination ?
 
 Probably not yet but getting closer.

Ok, thanks for the quick answer. That's good news.

 
  In general is oracle + Kernel 2.4 working ? 
 
 Ditto.
 
 The shm and rawio fixes are very recent
 

Ok. That needs some test ...

Yann Dupont.
-- 
\|/  \|/ Fac. des sciences de Nantes-Linux-Python-IPv6-ATM-BONOM
"@'/ ,. \@"  Tel :(+33) [0]251125865(AM)[0]251125857(PM)[0]251125868(Fax)
/_| \__/ |_\ [EMAIL PROTECTED]
   \__U_/http://www.unantes.univ-nantes.fr/~dupont

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



Oracle 8I & Kernel 2.4.3 : Sane ?

2001-04-02 Thread Yann Dupont

Hello. Some times ago, I tried Kernel 2.4.2 with oracle 8i 
(old version, 8.1.5)

Oracle was working fine at start, but after some hours we encountered
problems (I'm not database guru, I'm not able to say what was really not
working, it seems like a "rollback segment" was unable to be modified /
stuck)

I noticed that 2.4.3 contains some fixs for shared memory -
So the final question IS :

Is oracle 8.1.5 + Kernel 2.4.3 a sane combination ?
In general is oracle + Kernel 2.4 working ? 

(BTW, if that matter this is on a 4-xeon Proc + 3GB ram)

Yann Dupont.
-- 
\|/  \|/ Fac. des sciences de Nantes-Linux-Python-IPv6-ATM-BONOM
"@'/ ,. \@"  Tel :(+33) [0]251125865(AM)[0]251125857(PM)[0]251125868(Fax)
/_| \__/ |_\ [EMAIL PROTECTED]
   \__U_/http://www.unantes.univ-nantes.fr/~dupont

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



Oracle 8I Kernel 2.4.3 : Sane ?

2001-04-02 Thread Yann Dupont

Hello. Some times ago, I tried Kernel 2.4.2 with oracle 8i 
(old version, 8.1.5)

Oracle was working fine at start, but after some hours we encountered
problems (I'm not database guru, I'm not able to say what was really not
working, it seems like a "rollback segment" was unable to be modified /
stuck)

I noticed that 2.4.3 contains some fixs for shared memory -
So the final question IS :

Is oracle 8.1.5 + Kernel 2.4.3 a sane combination ?
In general is oracle + Kernel 2.4 working ? 

(BTW, if that matter this is on a 4-xeon Proc + 3GB ram)

Yann Dupont.
-- 
\|/  \|/ Fac. des sciences de Nantes-Linux-Python-IPv6-ATM-BONOM
"@'/ ,. \@"  Tel :(+33) [0]251125865(AM)[0]251125857(PM)[0]251125868(Fax)
/_| \__/ |_\ [EMAIL PROTECTED]
   \__U_/http://www.unantes.univ-nantes.fr/~dupont

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



__alloc_pages: 3-order allocation failed. for 2.4.1-pre9

2001-01-23 Thread Yann Dupont

I had thoses errors messages on 2.4.1-pre9 :

__alloc_pages: 2-order allocation failed.
__alloc_pages: 3-order allocation failed.
__alloc_pages: 3-order allocation failed.
(ad nauseum)

I was doing backups on scsi streamers using tar ;

The files are on a md0 array, file system is reiserfs;

the machine has 128MB RAM, Swap is 512MB and was pratically unused ;

the tar then failed miserably -But the system was OK after that; 
Please note that this was after an hour or so of backup... 
And that I was doing 2 backups in the same time ...

The machine is an old pentium pro 200,
the scsi cards are Buslogic BT958 ( I have 3 of them ; 3x9 Gb on 1, 3x9b
on 2, 2 scsi tape on 3)

I remember sawing that those errors were due to improperly written
drivers . Is the buslogic driver or tape driver are to blame here ?? Or
maybe this is a vm balancing issue ?

Side note : Free gives this :

admin@dumbo:~$ free
 total   used   free sharedbuffers
cached
Mem:126592 122856   3736  0 100968
8960
-/+ buffers/cache:  12928 113664
Swap:   522104   2352 519752

As we can expect, buffers are large, maybe a little too much ?

Yann Dupont.

-- 
\|/  \|/ Fac. des sciences de Nantes-Linux-Python-IPv6-ATM-BONOM
"@'/ ,. \@"  Tel :(+33) [0]251125865(AM)[0]251125857(PM)[0]251125868(Fax)
/_| \__/ |_\ [EMAIL PROTECTED]
   \__U_/http://www.unantes.univ-nantes.fr/~dupont

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/



__alloc_pages: 3-order allocation failed. for 2.4.1-pre9

2001-01-23 Thread Yann Dupont

I had thoses errors messages on 2.4.1-pre9 :

__alloc_pages: 2-order allocation failed.
__alloc_pages: 3-order allocation failed.
__alloc_pages: 3-order allocation failed.
(ad nauseum)

I was doing backups on scsi streamers using tar ;

The files are on a md0 array, file system is reiserfs;

the machine has 128MB RAM, Swap is 512MB and was pratically unused ;

the tar then failed miserably -But the system was OK after that; 
Please note that this was after an hour or so of backup... 
And that I was doing 2 backups in the same time ...

The machine is an old pentium pro 200,
the scsi cards are Buslogic BT958 ( I have 3 of them ; 3x9 Gb on 1, 3x9b
on 2, 2 scsi tape on 3)

I remember sawing that those errors were due to improperly written
drivers . Is the buslogic driver or tape driver are to blame here ?? Or
maybe this is a vm balancing issue ?

Side note : Free gives this :

admin@dumbo:~$ free
 total   used   free sharedbuffers
cached
Mem:126592 122856   3736  0 100968
8960
-/+ buffers/cache:  12928 113664
Swap:   522104   2352 519752

As we can expect, buffers are large, maybe a little too much ?

Yann Dupont.

-- 
\|/  \|/ Fac. des sciences de Nantes-Linux-Python-IPv6-ATM-BONOM
"@'/ ,. \@"  Tel :(+33) [0]251125865(AM)[0]251125857(PM)[0]251125868(Fax)
/_| \__/ |_\ [EMAIL PROTECTED]
   \__U_/http://www.unantes.univ-nantes.fr/~dupont

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/