On Thu, Jun 6, 2013 at 12:30 AM, Peter P Waskiewicz Jr
<[email protected]> wrote:
> What about the pre-emption behavior of the kernel? Namely Processor type
> and Features -> Preemption Model. Are you using no preemption, or forced
> preemption?
Ok. I've done testing. Yes, we were building with PREEMPT_FULL.
I've done some further testing and can re-create the problem on
vanilla, non-preempt kernels. See below.
# uname -a
Linux (none) 3.0.80-rt108 #2 Thu Jun 6 16:09:35 UTC 2013 ppc GNU/Linux
And I still get the slab corruption leading up to the kernel panic:
Slab corruption: size-2048 start=ee2b2070, len=2048
Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
Last user: [<c0208514>](skb_release_data+0xb4/0xc8)
020: 6b 6b ff ff ff ff ff ff 00 0d ed 47 d9 87 81 00
030: 00 f2 08 06 00 01 08 00 06 04 00 01 00 0d ed 47
040: d9 87 0a f1 0a ea 00 00 00 00 00 00 0a f1 0a ea
050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
060: 00 00 09 81 d2 0f 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
Next obj: start=ee2b2888, len=2048
Redzone: 0xd84156c5635688c0/0xd84156c5635688c0.
Last user: [<c0209b8c>](__netdev_alloc_skb+0x28/0x60)
000: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
010: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
Slab corruption: size-2048 start=ed401480, len=2048
Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
Last user: [<c0208514>](skb_release_data+0xb4/0xc8)
020: 6b 6b ff ff ff ff ff ff e0 db 55 e4 ce f9 08 00
030: 45 00 01 3e 3e 1a 00 00 80 11 ca c0 0a ca 0d 42
040: 0a ca 0d ff 00 8a 00 8a 01 2a a5 96 11 0e af 81
050: 0a ca 0d 42 00 8a 01 14 00 00 20 45 42 45 4f 45
060: 45 46 43 45 4c 45 50 45 44 45 49 45 4f 45 43 43
070: 41 43 41 43 41 43 41 43 41 41 41 00 20 46 44 45
Prev obj: start=ed400c68, len=2048
Redzone: 0xd84156c5635688c0/0xd84156c5635688c0.
Last user: [<c0209b8c>](__netdev_alloc_skb+0x28/0x60)
000: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
010: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
Unable to handle kernel paging request for data at address 0x20454c45
Faulting instruction address: 0xc0062498
Oops: Kernel access of bad area, sig: 11 [#1]
SEL35xx Platform
Modules linked in:
NIP: c0062498 LR: c02084d8 CTR: c000cbbc
REGS: ee85bc60 TRAP: 0300 Not tainted (3.0.80-rt108)
MSR: 00009032 <EE,ME,IR,DR> CR: 24008248 XER: 00000000
DAR: 20454c45, DSISR: 20000000
TASK = ef3e5830[4616] 'ifconfig' THREAD: ee85a000
GPR00: 00000000 ee85bd10 ef3e5830 20454c45 2d746baa 000005f2 00000002 00000000
GPR08: c03b14e4 ed7471a8 ee85bcd0 00005c26 00000000 10087a48 bfe0e41c 10064ae4
GPR16: 10064bc0 bfe0e40c 00000000 bfe0e3f4 00000228 00000000 00008914 c019a488
GPR24: c019a9cc ed70f4b0 0000005c ed70f340 ef063120 00000000 00000001 ee62bd30
NIP [c0062498] put_page+0x0/0x34
LR [c02084d8] skb_release_data+0x78/0xc8
Call Trace:
[ee85bd20] [c020810c] __kfree_skb+0x18/0xbc
[ee85bd30] [c0195734] e1000_clean_rx_ring+0x10c/0x1a4
[ee85bd60] [c01957f4] e1000_clean_all_rx_rings+0x28/0x54
[ee85bd70] [c0198d40] e1000_close+0x30/0xb4
[ee85bd90] [c0212408] __dev_close_many+0xa0/0xe0
[ee85bda0] [c02141a0] __dev_close+0x2c/0x4c
[ee85bdc0] [c0210a58] __dev_change_flags+0xb8/0x140
[ee85bde0] [c0212324] dev_change_flags+0x1c/0x60
[ee85be00] [c0267594] devinet_ioctl+0x2a4/0x700
[ee85be60] [c026839c] inet_ioctl+0xc8/0xfc
[ee85be70] [c02006d4] sock_ioctl+0x260/0x2a0
[ee85be90] [c009145c] vfs_ioctl+0x2c/0x58
[ee85bea0] [c0091bc8] do_vfs_ioctl+0x610/0x698
[ee85bf10] [c0091ca8] sys_ioctl+0x58/0x88
[ee85bf40] [c000e674] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xff35a3c
LR = 0xff359a0
Instruction dump:
419e0018 3c80c006 38630180 38842abc 38a00000 4bfffe65 80010014 bbc10008
38210010 7c0803a6 4e800020 4bffff54
<80030000> 7c691b78 700bc000 41a20008
Kernel panic - not syncing: Fatal exception
Call Trace:
[ee85bb90] [c0007b80] show_stack+0x58/0x154 (unreliable)
[ee85bbd0] [c001c3a8] panic+0xa8/0x1cc
[ee85bc20] [c000b1f0] die+0x178/0x19c
[ee85bc40] [c0011a44] bad_page_fault+0xe8/0xfc
[ee85bc50] [c000eb14] handle_page_fault+0x7c/0x80
--- Exception: 300 at put_page+0x0/0x34
LR = skb_release_data+0x78/0xc8
[ee85bd10] [00000000] (null) (unreliable)
[ee85bd20] [c020810c] __kfree_skb+0x18/0xbc
[ee85bd30] [c0195734] e1000_clean_rx_ring+0x10c/0x1a4
[ee85bd60] [c01957f4] e1000_clean_all_rx_rings+0x28/0x54
[ee85bd70] [c0198d40] e1000_close+0x30/0xb4
[ee85bd90] [c0212408] __dev_close_many+0xa0/0xe0
[ee85bda0] [c02141a0] __dev_close+0x2c/0x4c
[ee85bdc0] [c0210a58] __dev_change_flags+0xb8/0x140
[ee85bde0] [c0212324] dev_change_flags+0x1c/0x60
[ee85be00] [c0267594] devinet_ioctl+0x2a4/0x700
[ee85be60] [c026839c] inet_ioctl+0xc8/0xfc
[ee85be70] [c02006d4] sock_ioctl+0x260/0x2a0
[ee85be90] [c009145c] vfs_ioctl+0x2c/0x58
[ee85bea0] [c0091bc8] do_vfs_ioctl+0x610/0x698
[ee85bf10] [c0091ca8] sys_ioctl+0x58/0x88
[ee85bf40] [c000e674] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xff35a3c
LR = 0xff359a0
And with a vanilla, no-preempt kernel:
# uname -a
Linux (none) 3.0.80 #5 Thu Jun 6 16:26:15 UTC 2013 ppc GNU/Linux
slab error in verify_redzone_free(): cache `size-2048': memory outside
object was overwritten
Call Trace:
[eebc7c70] [c0007b80] show_stack+0x58/0x154 (unreliable)
[eebc7cb0] [c007bb0c] __slab_error+0x2c/0x3c
[eebc7cc0] [c007c0d0] cache_free_debugcheck+0x184/0x274
[eebc7cf0] [c007c36c] kfree+0x90/0x10c
[eebc7d10] [c02079e4] skb_release_data+0xb4/0xc8
[eebc7d20] [c02075dc] __kfree_skb+0x18/0xbc
[eebc7d30] [c0194d50] e1000_clean_rx_ring+0x10c/0x1a4
[eebc7d60] [c0194e10] e1000_clean_all_rx_rings+0x28/0x54
[eebc7d70] [c019835c] e1000_close+0x30/0xb4
[eebc7d90] [c02118d8] __dev_close_many+0xa0/0xe0
[eebc7da0] [c0213670] __dev_close+0x2c/0x4c
[eebc7dc0] [c020ff28] __dev_change_flags+0xb8/0x140
[eebc7de0] [c02117f4] dev_change_flags+0x1c/0x60
[eebc7e00] [c02669b4] devinet_ioctl+0x2a4/0x700
[eebc7e60] [c02677bc] inet_ioctl+0xc8/0xfc
[eebc7e70] [c01ffba4] sock_ioctl+0x260/0x2a0
[eebc7e90] [c0090a80] vfs_ioctl+0x2c/0x58
[eebc7ea0] [c00911ec] do_vfs_ioctl+0x610/0x698
[eebc7f10] [c00912cc] sys_ioctl+0x58/0x88
[eebc7f40] [c000e674] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xff35a3c
LR = 0xff359a0
ee26d718: redzone 1:0x8011c7500aca0dd4, redzone 2:0xd84156c5635688c0.
slab error in verify_redzone_free(): cache `size-2048': memory outside
object was overwritten
Call Trace:
[eebc7c70] [c0007b80] show_stack+0x58/0x154 (unreliable)
[eebc7cb0] [c007bb0c] __slab_error+0x2c/0x3c
[eebc7cc0] [c007c0d0] cache_free_debugcheck+0x184/0x274
[eebc7cf0] [c007c36c] kfree+0x90/0x10c
[eebc7d10] [c02079e4] skb_release_data+0xb4/0xc8
[eebc7d20] [c02075dc] __kfree_skb+0x18/0xbc
[eebc7d30] [c0194d50] e1000_clean_rx_ring+0x10c/0x1a4
[eebc7d60] [c0194e10] e1000_clean_all_rx_rings+0x28/0x54
[eebc7d70] [c019835c] e1000_close+0x30/0xb4
[eebc7d90] [c02118d8] __dev_close_many+0xa0/0xe0
[eebc7da0] [c0213670] __dev_close+0x2c/0x4c
[eebc7dc0] [c020ff28] __dev_change_flags+0xb8/0x140
[eebc7de0] [c02117f4] dev_change_flags+0x1c/0x60
[eebc7e00] [c02669b4] devinet_ioctl+0x2a4/0x700
[eebc7e60] [c02677bc] inet_ioctl+0xc8/0xfc
[eebc7e70] [c01ffba4] sock_ioctl+0x260/0x2a0
[eebc7e90] [c0090a80] vfs_ioctl+0x2c/0x58
[eebc7ea0] [c00911ec] do_vfs_ioctl+0x610/0x698
[eebc7f10] [c00912cc] sys_ioctl+0x58/0x88
[eebc7f40] [c000e674] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xff35a3c
LR = 0xff359a0
ee26cf00: redzone 1:0x8011bcea0aca0d81, redzone 2:0xd4bed92bb5840800.
eth2: no IPv6 routers present
slab error in verify_redzone_free(): cache `size-2048': memory outside
object was overwritten
Call Trace:
[eebc7c70] [c0007b80] show_stack+0x58/0x154 (unreliable)
[eebc7cb0] [c007bb0c] __slab_error+0x2c/0x3c
[eebc7cc0] [c007c0d0] cache_free_debugcheck+0x184/0x274
[eebc7cf0] [c007c36c] kfree+0x90/0x10c
[eebc7d10] [c02079e4] skb_release_data+0xb4/0xc8
[eebc7d20] [c02075dc] __kfree_skb+0x18/0xbc
[eebc7d30] [c0194d50] e1000_clean_rx_ring+0x10c/0x1a4
[eebc7d60] [c0194e10] e1000_clean_all_rx_rings+0x28/0x54
[eebc7d70] [c019835c] e1000_close+0x30/0xb4
[eebc7d90] [c02118d8] __dev_close_many+0xa0/0xe0
[eebc7da0] [c0213670] __dev_close+0x2c/0x4c
[eebc7dc0] [c020ff28] __dev_change_flags+0xb8/0x140
[eebc7de0] [c02117f4] dev_change_flags+0x1c/0x60
[eebc7e00] [c02669b4] devinet_ioctl+0x2a4/0x700
[eebc7e60] [c02677bc] inet_ioctl+0xc8/0xfc
[eebc7e70] [c01ffba4] sock_ioctl+0x260/0x2a0
[eebc7e90] [c0090a80] vfs_ioctl+0x2c/0x58
[eebc7ea0] [c00911ec] do_vfs_ioctl+0x610/0x698
[eebc7f10] [c00912cc] sys_ioctl+0x58/0x88
[eebc7f40] [c000e674] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xff35a3c
LR = 0xff359a0
ee26c6e8: redzone 1:0x8011c0320aca0d71, redzone 2:0x1aa0dd11c10800.
slab error in verify_redzone_free(): cache `size-2048': memory outside
object was overwritten
Call Trace:
[eebc7c70] [c0007b80] show_stack+0x58/0x154 (unreliable)
[eebc7cb0] [c007bb0c] __slab_error+0x2c/0x3c
[eebc7cc0] [c007c0d0] cache_free_debugcheck+0x184/0x274
[eebc7cf0] [c007c36c] kfree+0x90/0x10c
[eebc7d10] [c02079e4] skb_release_data+0xb4/0xc8
[eebc7d20] [c02075dc] __kfree_skb+0x18/0xbc
[eebc7d30] [c0194d50] e1000_clean_rx_ring+0x10c/0x1a4
[eebc7d60] [c0194e10] e1000_clean_all_rx_rings+0x28/0x54
[eebc7d70] [c019835c] e1000_close+0x30/0xb4
[eebc7d90] [c02118d8] __dev_close_many+0xa0/0xe0
[eebc7da0] [c0213670] __dev_close+0x2c/0x4c
[eebc7dc0] [c020ff28] __dev_change_flags+0xb8/0x140
[eebc7de0] [c02117f4] dev_change_flags+0x1c/0x60
[eebc7e00] [c02669b4] devinet_ioctl+0x2a4/0x700
[eebc7e60] [c02677bc] inet_ioctl+0xc8/0xfc
[eebc7e70] [c01ffba4] sock_ioctl+0x260/0x2a0
[eebc7e90] [c0090a80] vfs_ioctl+0x2c/0x58
[eebc7ea0] [c00911ec] do_vfs_ioctl+0x610/0x698
[eebc7f10] [c00912cc] sys_ioctl+0x58/0x88
[eebc7f40] [c000e674] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xff35a3c
LR = 0xff359a0
ee26f738: redzone 1:0x1372a318240aca, redzone 2:0xd84156c5635688c0.
slab error in verify_redzone_free(): cache `size-2048': memory outside
object was overwritten
Call Trace:
[eebc7c70] [c0007b80] show_stack+0x58/0x154 (unreliable)
[eebc7cb0] [c007bb0c] __slab_error+0x2c/0x3c
[eebc7cc0] [c007c0d0] cache_free_debugcheck+0x184/0x274
[eebc7cf0] [c007c36c] kfree+0x90/0x10c
[eebc7d10] [c02079e4] skb_release_data+0xb4/0xc8
[eebc7d20] [c02075dc] __kfree_skb+0x18/0xbc
[eebc7d30] [c0194d50] e1000_clean_rx_ring+0x10c/0x1a4
[eebc7d60] [c0194e10] e1000_clean_all_rx_rings+0x28/0x54
[eebc7d70] [c019835c] e1000_close+0x30/0xb4
[eebc7d90] [c02118d8] __dev_close_many+0xa0/0xe0
[eebc7da0] [c0213670] __dev_close+0x2c/0x4c
[eebc7dc0] [c020ff28] __dev_change_flags+0xb8/0x140
[eebc7de0] [c02117f4] dev_change_flags+0x1c/0x60
[eebc7e00] [c02669b4] devinet_ioctl+0x2a4/0x700
[eebc7e60] [c02677bc] inet_ioctl+0xc8/0xfc
[eebc7e70] [c01ffba4] sock_ioctl+0x260/0x2a0
[eebc7e90] [c0090a80] vfs_ioctl+0x2c/0x58
[eebc7ea0] [c00911ec] do_vfs_ioctl+0x610/0x698
[eebc7f10] [c00912cc] sys_ioctl+0x58/0x88
[eebc7f40] [c000e674] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xff35a3c
LR = 0xff359a0
ee26ef20: redzone 1:0x8011a40b0aca0d18, redzone 2:0x1372a318240806.
slab error in verify_redzone_free(): cache `size-2048': memory outside
object was overwritten
Call Trace:
[eebc7c70] [c0007b80] show_stack+0x58/0x154 (unreliable)
[eebc7cb0] [c007bb0c] __slab_error+0x2c/0x3c
[eebc7cc0] [c007c0d0] cache_free_debugcheck+0x184/0x274
[eebc7cf0] [c007c36c] kfree+0x90/0x10c
[eebc7d10] [c02079e4] skb_release_data+0xb4/0xc8
[eebc7d20] [c02075dc] __kfree_skb+0x18/0xbc
[eebc7d30] [c0194d50] e1000_clean_rx_ring+0x10c/0x1a4
[eebc7d60] [c0194e10] e1000_clean_all_rx_rings+0x28/0x54
[eebc7d70] [c019835c] e1000_close+0x30/0xb4
[eebc7d90] [c02118d8] __dev_close_many+0xa0/0xe0
[eebc7da0] [c0213670] __dev_close+0x2c/0x4c
[eebc7dc0] [c020ff28] __dev_change_flags+0xb8/0x140
[eebc7de0] [c02117f4] dev_change_flags+0x1c/0x60
[eebc7e00] [c02669b4] devinet_ioctl+0x2a4/0x700
[eebc7e60] [c02677bc] inet_ioctl+0xc8/0xfc
[eebc7e70] [c01ffba4] sock_ioctl+0x260/0x2a0
[eebc7e90] [c0090a80] vfs_ioctl+0x2c/0x58
[eebc7ea0] [c00911ec] do_vfs_ioctl+0x610/0x698
[eebc7f10] [c00912cc] sys_ioctl+0x58/0x88
[eebc7f40] [c000e674] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xff35a3c
LR = 0xff359a0
ee26e708: redzone 1:0x8011d7910aca0d95, redzone 2:0xd4bed98ac1d30800.
slab: Internal list corruption detected in cache 'size-2048'(3), slabp
ee26c6c0(65535). Hexdump:
000: 00 10 01 00 00 20 02 00 00 00 06 e8 ee 26 c6 e8
010: 00 00 ff ff ff ff ff ff 18 03 73 d2 ce ee 08 00
020: 45 00 00 44 49 73 00 00
------------[ cut here ]------------
kernel BUG at mm/slab.c:2998!
Oops: Exception in kernel mode, sig: 5 [#1]
SEL35xx Platform
Modules linked in:
NIP: c007b6d0 LR: c007b6d0 CTR: c01535a0
REGS: eebc7bc0 TRAP: 0700 Not tainted (3.0.80)
MSR: 00021032 <ME,CE,IR,DR> CR: 24008282 XER: 20000000
TASK = eea92030[4693] 'ifconfig' THREAD: eebc6000
GPR00: c007b6d0 eebc7c70 eea92030 00000001 00005a9e ffffffff c01509e0 00004000
GPR08: c03a0000 c03c9540 00005a9e 00004000 44008282 10087a48 bfeadbcc 10064ae4
GPR16: 10064bc0 bfeadbbc 00000000 bfeadba4 c03d0000 0000000c ef000a80 00000000
GPR24: 00100100 00200200 00000001 c03234bc c03234c4 ef000a80 ee26c6c0 00000028
NIP [c007b6d0] check_slabp+0xd8/0xf4
LR [c007b6d0] check_slabp+0xd8/0xf4
Call Trace:
[eebc7c70] [c007b6d0] check_slabp+0xd8/0xf4 (unreliable)
[eebc7c90] [c007c714] free_block+0xa4/0x1c8
[eebc7cd0] [c007c26c] cache_flusharray+0xac/0x11c
[eebc7cf0] [c007c3b8] kfree+0xdc/0x10c
[eebc7d10] [c02079e4] skb_release_data+0xb4/0xc8
[eebc7d20] [c02075dc] __kfree_skb+0x18/0xbc
[eebc7d30] [c0194d50] e1000_clean_rx_ring+0x10c/0x1a4
[eebc7d60] [c0194e10] e1000_clean_all_rx_rings+0x28/0x54
[eebc7d70] [c019835c] e1000_close+0x30/0xb4
[eebc7d90] [c02118d8] __dev_close_many+0xa0/0xe0
[eebc7da0] [c0213670] __dev_close+0x2c/0x4c
[eebc7dc0] [c020ff28] __dev_change_flags+0xb8/0x140
[eebc7de0] [c02117f4] dev_change_flags+0x1c/0x60
[eebc7e00] [c02669b4] devinet_ioctl+0x2a4/0x700
[eebc7e60] [c02677bc] inet_ioctl+0xc8/0xfc
[eebc7e70] [c01ffba4] sock_ioctl+0x260/0x2a0
[eebc7e90] [c0090a80] vfs_ioctl+0x2c/0x58
[eebc7ea0] [c00911ec] do_vfs_ioctl+0x610/0x698
[eebc7f10] [c00912cc] sys_ioctl+0x58/0x88
[eebc7f40] [c000e674] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xff35a3c
LR = 0xff359a0
Instruction dump:
7f83e378 3bff0001 4bfa1d75 813d001c 73e0000f 5529103a 3929001c 7f9f4840
419cffcc 3c60c034 38638c20 4bfa1d51 <0fe00000> 48000000 80010024 bb61000c
Kernel panic - not syncing: Fatal exception
Call Trace:
[eebc7a10] [c0007b80] show_stack+0x58/0x154 (unreliable)
[eebc7a50] [c001c290] panic+0xa8/0x1cc
[eebc7aa0] [c000b1f0] die+0x178/0x19c
[eebc7ac0] [c000b518] _exception+0x58/0x110
[eebc7bb0] [c000ecc0] ret_from_except_full+0x0/0x4c
--- Exception: 700 at check_slabp+0xd8/0xf4
LR = check_slabp+0xd8/0xf4
[eebc7c90] [c007c714] free_block+0xa4/0x1c8
[eebc7cd0] [c007c26c] cache_flusharray+0xac/0x11c
[eebc7cf0] [c007c3b8] kfree+0xdc/0x10c
[eebc7d10] [c02079e4] skb_release_data+0xb4/0xc8
[eebc7d20] [c02075dc] __kfree_skb+0x18/0xbc
[eebc7d30] [c0194d50] e1000_clean_rx_ring+0x10c/0x1a4
[eebc7d60] [c0194e10] e1000_clean_all_rx_rings+0x28/0x54
[eebc7d70] [c019835c] e1000_close+0x30/0xb4
[eebc7d90] [c02118d8] __dev_close_many+0xa0/0xe0
[eebc7da0] [c0213670] __dev_close+0x2c/0x4c
[eebc7dc0] [c020ff28] __dev_change_flags+0xb8/0x140
[eebc7de0] [c02117f4] dev_change_flags+0x1c/0x60
[eebc7e00] [c02669b4] devinet_ioctl+0x2a4/0x700
[eebc7e60] [c02677bc] inet_ioctl+0xc8/0xfc
[eebc7e70] [c01ffba4] sock_ioctl+0x260/0x2a0
[eebc7e90] [c0090a80] vfs_ioctl+0x2c/0x58
[eebc7ea0] [c00911ec] do_vfs_ioctl+0x610/0x698
[eebc7f10] [c00912cc] sys_ioctl+0x58/0x88
[eebc7f40] [c000e674] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xff35a3c
LR = 0xff359a0
------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
