Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Christoph Lameter <[EMAIL PROTECTED]> wrote: > I finally figured out the second issue. Took some time to get that figure > out. Sorry. But now all the bug reports make sense. [...] Impressive Christoph. Indeed, this fixes my problem on latest -git (its hg equivalent :-)). Well done. (Tested with CONFIG_SLAB_DEBUG without slub_debug; all was well. Then with slub_debug & slabinfo -v also; again all was well. I had many LOCKDEP and its relatives also compiled in.) Now hoping it'll be in 2.6.22-rc3 :-)). > Signed-off-by: Christoph Lameter <[EMAIL PROTECTED]> > > --- > mm/slub.c |7 +++ > 1 file changed, 7 insertions(+) > > Index: slub/mm/slub.c > === > --- slub.orig/mm/slub.c 2007-05-25 18:28:42.0 -0700 > +++ slub/mm/slub.c2007-05-25 18:29:46.0 -0700 > @@ -2473,6 +2473,7 @@ void __init kmem_cache_init(void) >*/ > create_kmalloc_cache(_caches[0], "kmem_cache_node", > sizeof(struct kmem_cache_node), GFP_KERNEL); > + kmalloc_caches[0].refcount = -1; > #endif > > /* Able to allocate the per node structures */ > @@ -2520,6 +2521,12 @@ static int slab_unmergeable(struct kmem_ > if (s->ctor) > return 1; > > + /* > + * We may have set a slab to be unmergeable during bootstrap. > + */ > + if (s->refcount < 0) > + return 1; > + > return 0; > } ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
I finally figured out the second issue. Took some time to get that figure out. Sorry. But now all the bug reports make sense. Here is the fix SLUB: Fix NUMA / SYSFS bootstrap issue The kmem_cache_node cache is very special because it is needed for NUMA bootstrap. Under certain conditions (like for example if lockdep is enabled and significantly increases the size of spinlock_t) the structure may become exactly the size as one of the larger caches in the kmalloc array. That early during bootstrap we cannot perform merging properly. The unique id for the kmem_cache_node cache will match one of the kmalloc array. Sysfs will complain about a duplicate directory entry. All of this occurs while the console is not yet fully operational. Thus boot may appear to be silently failing. The kmem_cache_node cache is very special. During early boostrap the main allocation function is not operational yet and so we have to run our own small special alloc function during early boot. It is also special in that it is never freed. We really do not want any merging on that cache. Set the refcount -1 and forbid merging of slabs that have a negative refcount. Signed-off-by: Christoph Lameter <[EMAIL PROTECTED]> --- mm/slub.c |7 +++ 1 file changed, 7 insertions(+) Index: slub/mm/slub.c === --- slub.orig/mm/slub.c 2007-05-25 18:28:42.0 -0700 +++ slub/mm/slub.c 2007-05-25 18:29:46.0 -0700 @@ -2473,6 +2473,7 @@ void __init kmem_cache_init(void) */ create_kmalloc_cache(_caches[0], "kmem_cache_node", sizeof(struct kmem_cache_node), GFP_KERNEL); + kmalloc_caches[0].refcount = -1; #endif /* Able to allocate the per node structures */ @@ -2520,6 +2521,12 @@ static int slab_unmergeable(struct kmem_ if (s->ctor) return 1; + /* +* We may have set a slab to be unmergeable during bootstrap. +*/ + if (s->refcount < 0) + return 1; + return 0; } - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Fri, May 25 2007, Christoph Lameter wrote: > On Fri, 25 May 2007, Jens Axboe wrote: > > > The .config I sent is the one that the last boot of the machine was > > based on. The kernel is 2.6.22-rc2 with the patch moving the #ifdef from > > you, and it works for me with or without CONFIG_SLUB_DEBUG set. > > I was not talking about your config but the .config that started this > thread. OK, that also wasn't clear. I'll repeat, if you want me to test anything further, please let that be known explicitly. I have reported what works and doesn't, as far as I'm concerned the patch works for me without incident. -- Jens Axboe - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Fri, 25 May 2007, Jens Axboe wrote: > The .config I sent is the one that the last boot of the machine was > based on. The kernel is 2.6.22-rc2 with the patch moving the #ifdef from > you, and it works for me with or without CONFIG_SLUB_DEBUG set. I was not talking about your config but the .config that started this thread. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Thu, May 24 2007, Christoph Lameter wrote: > On Thu, 24 May 2007, Jens Axboe wrote: > > > On Wed, May 23 2007, Christoph Lameter wrote: > > > On Wed, 23 May 2007, Jens Axboe wrote: > > > > > > > That works for me with the patch, .config attached. > > > > > > H... That means the .config sent initially here was bogus. > > > > ? > > > > Considering we're trying to help you fix bugs in your code, you could be > > considerably more helpful. > > I am trying figure out what is wrong and somehow that is not helpful? The > data that we have right now about the bug does not give us a clear You are doing what you are supposed to do, fixing your broken code. We are the ones that are helping you do that, by reporting the problem and testing solutions. I think you have your attributions backwards. > picture. One explanation that would have made it consistent would be that > the .config posted was not reflecting the actual kernel in use. The .config I sent is the one that the last boot of the machine was based on. The kernel is 2.6.22-rc2 with the patch moving the #ifdef from you, and it works for me with or without CONFIG_SLUB_DEBUG set. -- Jens Axboe - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Thu, May 24 2007, Christoph Lameter wrote: On Thu, 24 May 2007, Jens Axboe wrote: On Wed, May 23 2007, Christoph Lameter wrote: On Wed, 23 May 2007, Jens Axboe wrote: That works for me with the patch, .config attached. H... That means the .config sent initially here was bogus. ? Considering we're trying to help you fix bugs in your code, you could be considerably more helpful. I am trying figure out what is wrong and somehow that is not helpful? The data that we have right now about the bug does not give us a clear You are doing what you are supposed to do, fixing your broken code. We are the ones that are helping you do that, by reporting the problem and testing solutions. I think you have your attributions backwards. picture. One explanation that would have made it consistent would be that the .config posted was not reflecting the actual kernel in use. The .config I sent is the one that the last boot of the machine was based on. The kernel is 2.6.22-rc2 with the patch moving the #ifdef from you, and it works for me with or without CONFIG_SLUB_DEBUG set. -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Fri, 25 May 2007, Jens Axboe wrote: The .config I sent is the one that the last boot of the machine was based on. The kernel is 2.6.22-rc2 with the patch moving the #ifdef from you, and it works for me with or without CONFIG_SLUB_DEBUG set. I was not talking about your config but the .config that started this thread. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Fri, May 25 2007, Christoph Lameter wrote: On Fri, 25 May 2007, Jens Axboe wrote: The .config I sent is the one that the last boot of the machine was based on. The kernel is 2.6.22-rc2 with the patch moving the #ifdef from you, and it works for me with or without CONFIG_SLUB_DEBUG set. I was not talking about your config but the .config that started this thread. OK, that also wasn't clear. I'll repeat, if you want me to test anything further, please let that be known explicitly. I have reported what works and doesn't, as far as I'm concerned the patch works for me without incident. -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
I finally figured out the second issue. Took some time to get that figure out. Sorry. But now all the bug reports make sense. Here is the fix SLUB: Fix NUMA / SYSFS bootstrap issue The kmem_cache_node cache is very special because it is needed for NUMA bootstrap. Under certain conditions (like for example if lockdep is enabled and significantly increases the size of spinlock_t) the structure may become exactly the size as one of the larger caches in the kmalloc array. That early during bootstrap we cannot perform merging properly. The unique id for the kmem_cache_node cache will match one of the kmalloc array. Sysfs will complain about a duplicate directory entry. All of this occurs while the console is not yet fully operational. Thus boot may appear to be silently failing. The kmem_cache_node cache is very special. During early boostrap the main allocation function is not operational yet and so we have to run our own small special alloc function during early boot. It is also special in that it is never freed. We really do not want any merging on that cache. Set the refcount -1 and forbid merging of slabs that have a negative refcount. Signed-off-by: Christoph Lameter [EMAIL PROTECTED] --- mm/slub.c |7 +++ 1 file changed, 7 insertions(+) Index: slub/mm/slub.c === --- slub.orig/mm/slub.c 2007-05-25 18:28:42.0 -0700 +++ slub/mm/slub.c 2007-05-25 18:29:46.0 -0700 @@ -2473,6 +2473,7 @@ void __init kmem_cache_init(void) */ create_kmalloc_cache(kmalloc_caches[0], kmem_cache_node, sizeof(struct kmem_cache_node), GFP_KERNEL); + kmalloc_caches[0].refcount = -1; #endif /* Able to allocate the per node structures */ @@ -2520,6 +2521,12 @@ static int slab_unmergeable(struct kmem_ if (s-ctor) return 1; + /* +* We may have set a slab to be unmergeable during bootstrap. +*/ + if (s-refcount 0) + return 1; + return 0; } - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Christoph Lameter [EMAIL PROTECTED] wrote: I finally figured out the second issue. Took some time to get that figure out. Sorry. But now all the bug reports make sense. [...] Impressive Christoph. Indeed, this fixes my problem on latest -git (its hg equivalent :-)). Well done. (Tested with CONFIG_SLAB_DEBUG without slub_debug; all was well. Then with slub_debug slabinfo -v also; again all was well. I had many LOCKDEP and its relatives also compiled in.) Now hoping it'll be in 2.6.22-rc3 :-)). Signed-off-by: Christoph Lameter [EMAIL PROTECTED] --- mm/slub.c |7 +++ 1 file changed, 7 insertions(+) Index: slub/mm/slub.c === --- slub.orig/mm/slub.c 2007-05-25 18:28:42.0 -0700 +++ slub/mm/slub.c2007-05-25 18:29:46.0 -0700 @@ -2473,6 +2473,7 @@ void __init kmem_cache_init(void) */ create_kmalloc_cache(kmalloc_caches[0], kmem_cache_node, sizeof(struct kmem_cache_node), GFP_KERNEL); + kmalloc_caches[0].refcount = -1; #endif /* Able to allocate the per node structures */ @@ -2520,6 +2521,12 @@ static int slab_unmergeable(struct kmem_ if (s-ctor) return 1; + /* + * We may have set a slab to be unmergeable during bootstrap. + */ + if (s-refcount 0) + return 1; + return 0; } ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Christoph Lameter <[EMAIL PROTECTED]> wrote: > slabinfo -v [...] > Any corrupted objects will be reported to the syslog. (Only only could have made that sentence better.) Sorry I didn't know slabinfo -v produces nothing on stdout, stderr, rather all findings reported to syslog. Right? You don't have to answer this question, if my assumptions are right. If so, when ran it produced nothing in the syslog. Sorry mate, I'm unable to spend a lot of time on slub debugging. I'll nonetheless keep slub as my preferred memory allocator for the rest of the 2.6.22-rc process. If anything interesting turns up that I could contribute during this phase, you'll be the first know I promise. Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Fri, 25 May 2007, Srihari Vijayaraghavan wrote: > I know slabinfo_without_v might not be good enough for you. For completeness, > I attach it here anyway. Right. "slabinfo" output is useless. You need to specify -v for it to have any effect. slabinfo -v can detect object corruptions without the kernel having to touch the object. It will scan all objects and it will only work if run as root. Any corrupted objects will be reported to the syslog. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Christoph Lameter <[EMAIL PROTECTED]> wrote: [...] > You need to be root to do this. Sorry. Of course, I tried it as root also. (Kernel hg commit a4c9979b8d42 is 2.6.22-rc2 + all commits available as of a few minutes ago) [EMAIL PROTECTED] ~]# whoami root [EMAIL PROTECTED] ~]# uname -a Linux fedora7 2.6.22-rc2-a4c9979b8d42 #1 SMP Fri May 25 06:32:20 EST 2007 x86_64 x86_64 x86_64 GNU/Linux [EMAIL PROTECTED] ~]# egrep -i 'slub' /proc/cmdline /boot/config-2.6.22-rc2-a4c9979b8d42 /proc/cmdline:ro root=LABEL=/1234 slub_debug /boot/config-2.6.22-rc2-a4c9979b8d42:CONFIG_SLUB_DEBUG=y /boot/config-2.6.22-rc2-a4c9979b8d42:CONFIG_SLUB=y [EMAIL PROTECTED] ~]# md5sum /home/hari/linux-2.6/Documentation/vm/slabinfo.c 07a832596203afba9739c27daa059174 /home/hari/linux-2.6/Documentation/vm/slabinfo.c [EMAIL PROTECTED] ~]# gcc --version gcc (GCC) 4.1.2 20070502 (Red Hat 4.1.2-12) ... [EMAIL PROTECTED] ~]# gcc -o /usr/local/bin/slabinfo /home/hari/linux-2.6/Documentation/vm/slabinfo.c ; echo $? 0 [EMAIL PROTECTED] ~]# /usr/local/bin/slabinfo -v > slabinfo.txt; echo $? 0 [EMAIL PROTECTED] ~]# ll slabinfo.txt -rw-r--r-- 1 root root 0 2007-05-25 07:11 slabinfo.txt [EMAIL PROTECTED] ~]# /usr/local/bin/slabinfo > slabinfo_without_v.txt; echo $? 0 [EMAIL PROTECTED] ~]# ll slabinfo_without_v.txt -rw-r--r-- 1 root root 7932 2007-05-25 07:14 slabinfo_without_v.txt I know slabinfo_without_v might not be good enough for you. For completeness, I attach it here anyway. (I couldn't attach the 0 byte slabinfo.txt, otherwise I'd've given you that also. It's unbearable to lose an enormous 0 byte of information.) Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html Name Objects ObjsizeSpace Slabs/Part/Cpu O/S O %Fr %Ef Flg Acpi-Namespace 635 3269.6K 17/1/0 39 0 5 29 PZFU Acpi-Operand 1975 64 274.4K 67/4/0 30 0 5 46 PZFU Acpi-Parse 0 40 8.1K 2/2/0 36 0 100 0 PZFU Acpi-ParseExt0 64 8.1K 2/2/0 30 0 100 0 PZFU Acpi-State 0 80 8.1K 2/2/0 26 0 100 0 PZFU anon_vma 1714 64 237.5K 58/2/2 30 0 3 46 PZFU arp_cache1 300 4.0K 1/1/0 10 0 100 7 APZFU avc_node 498 7286.0K21/15/1 28 0 71 41 PZFU avtab_node 110636 2410.7M 2635/1/0 42 0 0 24 PZFU bdev_cache 36134465.5K 8/1/05 1 12 73 APaZFU bio 2 104 8.1K 2/0/2 21 0 0 2 APZFU biovec-1 2 16 8.1K 2/0/2 46 0 0 0 APZFU biovec-128 2204816.3K 2/2/03 1 100 25 APZFU biovec-162 256 8.1K 2/2/0 10 0 100 6 APZFU biovec-256 2409616.3K 2/0/01 1 0 50 APZFU biovec-4 2 64 8.1K 2/2/0 21 0 100 1 APZFU biovec-642102416.3K 2/2/07 1 100 12 APZFU blkdev_ioc 50 6412.2K 3/3/0 30 0 100 26 PZFU blkdev_queue21175249.1K 6/1/04 1 16 74 PZFU blkdev_requests 16 28016.3K 4/2/2 11 0 50 27 PZFU buffer_head 5420 104 970.7K237/3/2 23 0 1 58 PaZFU cfq_io_context 51 15216.3K 4/3/0 18 0 75 47 PZFU cfq_queue 58 14420.4K 5/4/0 18 0 80 40 PZFU dentry 21791 240 6.8M 1677/0/2 13 0 0 76 PaZFU dnotify_cache1 40 4.0K 1/1/0 36 0 100 0 PZFU ext3_inode_cache 8478137613.9M 1697/1/25 1 0 83 PaZFU ext3_xattr 823 88 139.2K 34/1/1 25 0 2 52 PaZFU fasync_cache 2 24 4.0K 1/1/0 42 0 100 1 PZFU file_lock_cache 7 216 8.1K 2/2/0 14 0 100 18 PZFU files_cache 67 76877.8K 19/3/24 0 15 66 APZFU filp 2531 328 1.1M283/0/29 0 0 71 APZFU fs_cache66 11216.3K 4/2/2 21 0 50 45 APZFU hugetlbfs_inode_cache11008 8.1K 1/1/07 1 100 12 PZFU idr_layer_cache194 528 135.1K 33/2/06 0 6 75 PZFU inode_cache 56171008 6.5M804/0/27 1 0 85 PaZFU inotify_event_cache
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Thu, 24 May 2007, Srihari Vijayaraghavan wrote: > slabinfo -v produces this error message: > Cannot write to Acpi-Namespace/validate You need to be root to do this. Sorry. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Thu, 24 May 2007, Jens Axboe wrote: > On Wed, May 23 2007, Christoph Lameter wrote: > > On Wed, 23 May 2007, Jens Axboe wrote: > > > > > That works for me with the patch, .config attached. > > > > H... That means the .config sent initially here was bogus. > > ? > > Considering we're trying to help you fix bugs in your code, you could be > considerably more helpful. I am trying figure out what is wrong and somehow that is not helpful? The data that we have right now about the bug does not give us a clear picture. One explanation that would have made it consistent would be that the .config posted was not reflecting the actual kernel in use. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Christoph Lameter <[EMAIL PROTECTED]> wrote: > On Thu, 24 May 2007, Srihari Vijayaraghavan wrote: [...] > Could you boot with slub_debug and then run Done. > slabinfo -v > > to validate all slabs? If there is anything wrong with an object then it > should show in the syslog. slabinfo -v produces this error message: Cannot write to Acpi-Namespace/validate slabinfo with no argument worked fine; the result is attached. Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html Name Objects ObjsizeSpace Slabs/Part/Cpu O/S O %Fr %Ef Flg Acpi-Namespace 635 3269.6K 17/1/0 39 0 5 29 PZFU Acpi-Operand 1975 64 274.4K 67/4/0 30 0 5 46 PZFU Acpi-Parse 0 40 8.1K 2/2/0 36 0 100 0 PZFU Acpi-ParseExt0 64 8.1K 2/2/0 30 0 100 0 PZFU Acpi-State 0 80 8.1K 2/2/0 26 0 100 0 PZFU anon_vma 1636 64 233.4K 57/4/2 30 0 7 44 PZFU arp_cache2 300 4.0K 1/1/0 10 0 100 14 APZFU avc_node 503 7281.9K20/10/2 28 0 50 44 PZFU avtab_node 110636 2410.7M 2635/1/0 42 0 0 24 PZFU bdev_cache 36134465.5K 8/2/05 1 25 73 APaZFU bio 2 104 8.1K 2/0/2 21 0 0 2 APZFU biovec-1 2 16 8.1K 2/0/2 46 0 0 0 APZFU biovec-128 2204816.3K 2/2/03 1 100 25 APZFU biovec-162 256 8.1K 2/1/1 10 0 50 6 APZFU biovec-256 2409616.3K 2/0/01 1 0 50 APZFU biovec-4 2 64 8.1K 2/0/2 21 0 0 1 APZFU biovec-642102416.3K 2/1/17 1 50 12 APZFU blkdev_ioc 42 64 8.1K 2/1/1 30 0 50 32 PZFU blkdev_queue21175249.1K 6/1/04 1 16 74 PZFU blkdev_requests 16 28016.3K 4/1/2 11 0 25 27 PZFU buffer_head 5315 104 954.3K233/4/2 23 0 1 57 PaZFU cfq_io_context 45 15216.3K 4/3/1 18 0 75 41 PZFU cfq_queue 52 14416.3K 4/3/1 18 0 75 45 PZFU dentry 21166 240 6.6M 1629/2/2 13 0 0 76 PaZFU dnotify_cache1 40 4.0K 1/1/0 36 0 100 0 PZFU ext3_inode_cache 8357137613.6M 1672/2/25 1 0 83 PaZFU ext3_xattr 842 88 143.3K 35/0/2 25 0 0 51 PaZFU fasync_cache 2 24 4.0K 1/1/0 42 0 100 1 PZFU file_lock_cache 4 216 8.1K 2/2/0 14 0 100 10 PZFU files_cache 60 76869.6K 17/3/24 0 17 66 APZFU filp 2190 328 1.0M245/0/29 0 0 71 APZFU fs_cache59 11216.3K 4/2/2 21 0 50 40 APZFU hugetlbfs_inode_cache11008 8.1K 1/1/07 1 100 12 PZFU idr_layer_cache194 528 135.1K 33/1/06 0 3 75 PZFU inet_peer_cache 1 64 4.0K 1/1/0 21 0 100 1 APZFU inode_cache 54031008 6.3M773/2/27 1 0 86 PaZFU inotify_event_cache 0 40 8.1K 2/2/0 36 0 100 0 PZFU inotify_watch_cache 81 7216.3K 4/3/0 28 0 75 35 PZFU ip_dst_cache43 32020.4K 5/0/29 0 0 67 APZFU ip_fib_alias10 48 8.1K 2/2/0 32 0 100 5 APZFU ip_fib_hash 10 40 8.1K 2/2/0 32 0 100 4 APZFU jbd_4k 0409616.3K 2/0/21 1 0 0 PZFU journal_handle 0 24 8.1K 2/0/2 42 0 0 0 PZFU journal_head28 9612.2K 3/1/2 24 0 33 21 PZFU key_jar 10 216 8.1K 2/2/0 12 0 100 26 APZFU kmalloc-1024 2971024 360.4K 44/2/27 1 4 84 PZFU kmalloc-128100 12824.5K 6/3/0 20 0 50 52 PZFU kmalloc-131072 1 131072 262.1K 1/0/01 6 0 50 PZFU kmalloc-16 53453 16 4.7M 1164/6/2 46 0 0 17 PZFU kmalloc-16384 12
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Wed, May 23 2007, Christoph Lameter wrote: > On Wed, 23 May 2007, Jens Axboe wrote: > > > That works for me with the patch, .config attached. > > H... That means the .config sent initially here was bogus. ? Considering we're trying to help you fix bugs in your code, you could be considerably more helpful. -- Jens Axboe - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Wed, May 23 2007, Christoph Lameter wrote: On Wed, 23 May 2007, Jens Axboe wrote: That works for me with the patch, .config attached. H... That means the .config sent initially here was bogus. ? Considering we're trying to help you fix bugs in your code, you could be considerably more helpful. -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Christoph Lameter [EMAIL PROTECTED] wrote: On Thu, 24 May 2007, Srihari Vijayaraghavan wrote: [...] Could you boot with slub_debug and then run Done. slabinfo -v to validate all slabs? If there is anything wrong with an object then it should show in the syslog. slabinfo -v produces this error message: Cannot write to Acpi-Namespace/validate slabinfo with no argument worked fine; the result is attached. Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html Name Objects ObjsizeSpace Slabs/Part/Cpu O/S O %Fr %Ef Flg Acpi-Namespace 635 3269.6K 17/1/0 39 0 5 29 PZFU Acpi-Operand 1975 64 274.4K 67/4/0 30 0 5 46 PZFU Acpi-Parse 0 40 8.1K 2/2/0 36 0 100 0 PZFU Acpi-ParseExt0 64 8.1K 2/2/0 30 0 100 0 PZFU Acpi-State 0 80 8.1K 2/2/0 26 0 100 0 PZFU anon_vma 1636 64 233.4K 57/4/2 30 0 7 44 PZFU arp_cache2 300 4.0K 1/1/0 10 0 100 14 APZFU avc_node 503 7281.9K20/10/2 28 0 50 44 PZFU avtab_node 110636 2410.7M 2635/1/0 42 0 0 24 PZFU bdev_cache 36134465.5K 8/2/05 1 25 73 APaZFU bio 2 104 8.1K 2/0/2 21 0 0 2 APZFU biovec-1 2 16 8.1K 2/0/2 46 0 0 0 APZFU biovec-128 2204816.3K 2/2/03 1 100 25 APZFU biovec-162 256 8.1K 2/1/1 10 0 50 6 APZFU biovec-256 2409616.3K 2/0/01 1 0 50 APZFU biovec-4 2 64 8.1K 2/0/2 21 0 0 1 APZFU biovec-642102416.3K 2/1/17 1 50 12 APZFU blkdev_ioc 42 64 8.1K 2/1/1 30 0 50 32 PZFU blkdev_queue21175249.1K 6/1/04 1 16 74 PZFU blkdev_requests 16 28016.3K 4/1/2 11 0 25 27 PZFU buffer_head 5315 104 954.3K233/4/2 23 0 1 57 PaZFU cfq_io_context 45 15216.3K 4/3/1 18 0 75 41 PZFU cfq_queue 52 14416.3K 4/3/1 18 0 75 45 PZFU dentry 21166 240 6.6M 1629/2/2 13 0 0 76 PaZFU dnotify_cache1 40 4.0K 1/1/0 36 0 100 0 PZFU ext3_inode_cache 8357137613.6M 1672/2/25 1 0 83 PaZFU ext3_xattr 842 88 143.3K 35/0/2 25 0 0 51 PaZFU fasync_cache 2 24 4.0K 1/1/0 42 0 100 1 PZFU file_lock_cache 4 216 8.1K 2/2/0 14 0 100 10 PZFU files_cache 60 76869.6K 17/3/24 0 17 66 APZFU filp 2190 328 1.0M245/0/29 0 0 71 APZFU fs_cache59 11216.3K 4/2/2 21 0 50 40 APZFU hugetlbfs_inode_cache11008 8.1K 1/1/07 1 100 12 PZFU idr_layer_cache194 528 135.1K 33/1/06 0 3 75 PZFU inet_peer_cache 1 64 4.0K 1/1/0 21 0 100 1 APZFU inode_cache 54031008 6.3M773/2/27 1 0 86 PaZFU inotify_event_cache 0 40 8.1K 2/2/0 36 0 100 0 PZFU inotify_watch_cache 81 7216.3K 4/3/0 28 0 75 35 PZFU ip_dst_cache43 32020.4K 5/0/29 0 0 67 APZFU ip_fib_alias10 48 8.1K 2/2/0 32 0 100 5 APZFU ip_fib_hash 10 40 8.1K 2/2/0 32 0 100 4 APZFU jbd_4k 0409616.3K 2/0/21 1 0 0 PZFU journal_handle 0 24 8.1K 2/0/2 42 0 0 0 PZFU journal_head28 9612.2K 3/1/2 24 0 33 21 PZFU key_jar 10 216 8.1K 2/2/0 12 0 100 26 APZFU kmalloc-1024 2971024 360.4K 44/2/27 1 4 84 PZFU kmalloc-128100 12824.5K 6/3/0 20 0 50 52 PZFU kmalloc-131072 1 131072 262.1K 1/0/01 6 0 50 PZFU kmalloc-16 53453 16 4.7M 1164/6/2 46 0 0 17 PZFU kmalloc-16384 12 16384
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Thu, 24 May 2007, Jens Axboe wrote: On Wed, May 23 2007, Christoph Lameter wrote: On Wed, 23 May 2007, Jens Axboe wrote: That works for me with the patch, .config attached. H... That means the .config sent initially here was bogus. ? Considering we're trying to help you fix bugs in your code, you could be considerably more helpful. I am trying figure out what is wrong and somehow that is not helpful? The data that we have right now about the bug does not give us a clear picture. One explanation that would have made it consistent would be that the .config posted was not reflecting the actual kernel in use. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Thu, 24 May 2007, Srihari Vijayaraghavan wrote: slabinfo -v produces this error message: Cannot write to Acpi-Namespace/validate You need to be root to do this. Sorry. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Christoph Lameter [EMAIL PROTECTED] wrote: [...] You need to be root to do this. Sorry. Of course, I tried it as root also. (Kernel hg commit a4c9979b8d42 is 2.6.22-rc2 + all commits available as of a few minutes ago) [EMAIL PROTECTED] ~]# whoami root [EMAIL PROTECTED] ~]# uname -a Linux fedora7 2.6.22-rc2-a4c9979b8d42 #1 SMP Fri May 25 06:32:20 EST 2007 x86_64 x86_64 x86_64 GNU/Linux [EMAIL PROTECTED] ~]# egrep -i 'slub' /proc/cmdline /boot/config-2.6.22-rc2-a4c9979b8d42 /proc/cmdline:ro root=LABEL=/1234 slub_debug /boot/config-2.6.22-rc2-a4c9979b8d42:CONFIG_SLUB_DEBUG=y /boot/config-2.6.22-rc2-a4c9979b8d42:CONFIG_SLUB=y [EMAIL PROTECTED] ~]# md5sum /home/hari/linux-2.6/Documentation/vm/slabinfo.c 07a832596203afba9739c27daa059174 /home/hari/linux-2.6/Documentation/vm/slabinfo.c [EMAIL PROTECTED] ~]# gcc --version gcc (GCC) 4.1.2 20070502 (Red Hat 4.1.2-12) ... [EMAIL PROTECTED] ~]# gcc -o /usr/local/bin/slabinfo /home/hari/linux-2.6/Documentation/vm/slabinfo.c ; echo $? 0 [EMAIL PROTECTED] ~]# /usr/local/bin/slabinfo -v slabinfo.txt; echo $? 0 [EMAIL PROTECTED] ~]# ll slabinfo.txt -rw-r--r-- 1 root root 0 2007-05-25 07:11 slabinfo.txt [EMAIL PROTECTED] ~]# /usr/local/bin/slabinfo slabinfo_without_v.txt; echo $? 0 [EMAIL PROTECTED] ~]# ll slabinfo_without_v.txt -rw-r--r-- 1 root root 7932 2007-05-25 07:14 slabinfo_without_v.txt I know slabinfo_without_v might not be good enough for you. For completeness, I attach it here anyway. (I couldn't attach the 0 byte slabinfo.txt, otherwise I'd've given you that also. It's unbearable to lose an enormous 0 byte of information.) Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html Name Objects ObjsizeSpace Slabs/Part/Cpu O/S O %Fr %Ef Flg Acpi-Namespace 635 3269.6K 17/1/0 39 0 5 29 PZFU Acpi-Operand 1975 64 274.4K 67/4/0 30 0 5 46 PZFU Acpi-Parse 0 40 8.1K 2/2/0 36 0 100 0 PZFU Acpi-ParseExt0 64 8.1K 2/2/0 30 0 100 0 PZFU Acpi-State 0 80 8.1K 2/2/0 26 0 100 0 PZFU anon_vma 1714 64 237.5K 58/2/2 30 0 3 46 PZFU arp_cache1 300 4.0K 1/1/0 10 0 100 7 APZFU avc_node 498 7286.0K21/15/1 28 0 71 41 PZFU avtab_node 110636 2410.7M 2635/1/0 42 0 0 24 PZFU bdev_cache 36134465.5K 8/1/05 1 12 73 APaZFU bio 2 104 8.1K 2/0/2 21 0 0 2 APZFU biovec-1 2 16 8.1K 2/0/2 46 0 0 0 APZFU biovec-128 2204816.3K 2/2/03 1 100 25 APZFU biovec-162 256 8.1K 2/2/0 10 0 100 6 APZFU biovec-256 2409616.3K 2/0/01 1 0 50 APZFU biovec-4 2 64 8.1K 2/2/0 21 0 100 1 APZFU biovec-642102416.3K 2/2/07 1 100 12 APZFU blkdev_ioc 50 6412.2K 3/3/0 30 0 100 26 PZFU blkdev_queue21175249.1K 6/1/04 1 16 74 PZFU blkdev_requests 16 28016.3K 4/2/2 11 0 50 27 PZFU buffer_head 5420 104 970.7K237/3/2 23 0 1 58 PaZFU cfq_io_context 51 15216.3K 4/3/0 18 0 75 47 PZFU cfq_queue 58 14420.4K 5/4/0 18 0 80 40 PZFU dentry 21791 240 6.8M 1677/0/2 13 0 0 76 PaZFU dnotify_cache1 40 4.0K 1/1/0 36 0 100 0 PZFU ext3_inode_cache 8478137613.9M 1697/1/25 1 0 83 PaZFU ext3_xattr 823 88 139.2K 34/1/1 25 0 2 52 PaZFU fasync_cache 2 24 4.0K 1/1/0 42 0 100 1 PZFU file_lock_cache 7 216 8.1K 2/2/0 14 0 100 18 PZFU files_cache 67 76877.8K 19/3/24 0 15 66 APZFU filp 2531 328 1.1M283/0/29 0 0 71 APZFU fs_cache66 11216.3K 4/2/2 21 0 50 45 APZFU hugetlbfs_inode_cache11008 8.1K 1/1/07 1 100 12 PZFU idr_layer_cache194 528 135.1K 33/2/06 0 6 75 PZFU inode_cache 56171008 6.5M804/0/27 1 0 85 PaZFU inotify_event_cache 0
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Fri, 25 May 2007, Srihari Vijayaraghavan wrote: I know slabinfo_without_v might not be good enough for you. For completeness, I attach it here anyway. Right. slabinfo output is useless. You need to specify -v for it to have any effect. slabinfo -v can detect object corruptions without the kernel having to touch the object. It will scan all objects and it will only work if run as root. Any corrupted objects will be reported to the syslog. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Christoph Lameter [EMAIL PROTECTED] wrote: slabinfo -v [...] Any corrupted objects will be reported to the syslog. (Only only could have made that sentence better.) Sorry I didn't know slabinfo -v produces nothing on stdout, stderr, rather all findings reported to syslog. Right? You don't have to answer this question, if my assumptions are right. If so, when ran it produced nothing in the syslog. Sorry mate, I'm unable to spend a lot of time on slub debugging. I'll nonetheless keep slub as my preferred memory allocator for the rest of the 2.6.22-rc process. If anything interesting turns up that I could contribute during this phase, you'll be the first know I promise. Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Thu, 24 May 2007, Srihari Vijayaraghavan wrote: > I'm stunned. Honestly, I have no possible explanations for this behaviour. Do > you? I need more time to work out (until otherwise you might know a reason). Hmmm... Bad. We have conflicting reports and no clear way to trigger the bug. This may actually only trigger depending on how certain objects is place in memory. Could you boot with slub_debug and then run slabinfo -v to validate all slabs? If there is anything wrong with an object then it should show in the syslog. slabinfo.c can be found at Documentation/vm/slabinfo.c > Oh, while at it, I created this patch (to ask you more info on what > slub_nomerge is for): See Documentation/vm/slub.txt which covers most of those. slub_nomerge is not mentioned. It removes an optimization and that is only useful for debugging strange cornercases. I hope to remove it someday. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Christoph Lameter <[EMAIL PROTECTED]> wrote: > On Wed, 23 May 2007, Srihari Vijayaraghavan wrote: > > > I'm personally very happy that slub works stably without slub debug > options, > > because that's what I'd run in a production env. Thanks to your patch, > slub is > > quite stable without the slub debug for me :-)). But it'd to nice to have > a > > working slub debug for test env., as you'd undoubtedly be aware of, of > course > > :-). Just my humble opinion. > > Just switching off lockdep should make you happy until we figure out what > is wrong? I'm very happy :-). No problems. Has something important been merged in the last day or so to mainstream (http://www.kernel.org/hg/linux-2.6 starting hg commit 57110:0a8a879062a0)? On 57199:239bf3811cb2 with CONFIG_LOCKDEP, CONFIG_PROVE_LOCKING & cousins with CONFIG_SLUB_DEBUG without slub_nomerge even, I'm unable to reproduce the panic now. My current .config is attached as config-2.6.22-rc2-hg-slub.txt I'm stunned. Honestly, I have no possible explanations for this behaviour. Do you? I need more time to work out (until otherwise you might know a reason). (Perhaps, when I introduced a few debug options in the kernel hacking section, I inadvertently introduced a timing issue that now masks the panic away. Just my hypothesis.) A few things did happen in the last 24 hours time): 1. I pulled 90 changesets from the linux-2.6 hg repo. Easy to rollback. 2. I did a yum update on my Fedora 7 test system (I don't believe it pulled anything important in ref to this problem). Not so easy to rollback. 3. Adding a few extra debug options in the "kernel hacking" section. Easy to rollback also. I need more time (something I really don't have, but it's my karma for kernel community) to eliminate the things. I'll now stop updating everything so not to introduce any new variables.) Oh, while at it, I created this patch (to ask you more info on what slub_nomerge is for): I'm also attaching my current working .config Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html --- linux-2.6-239bf3811cb2-slub/Documentation/kernel-parameters.txt.orig 2007-05-24 07:41:33.0 +1000 +++ linux-2.6-239bf3811cb2-slub/Documentation/kernel-parameters.txt 2007-05-24 07:43:34.0 +1000 @@ -1593,6 +1593,9 @@ slram= [HW,MTD] + slub_debug ... + slub_nomerge... + smart2= [HW] Format: [,[,...,]] # # Automatically generated make config: don't edit # Linux kernel version: 2.6.22-rc2 # Thu May 24 07:46:01 2007 # CONFIG_X86_64=y CONFIG_64BIT=y CONFIG_X86=y CONFIG_GENERIC_TIME=y CONFIG_GENERIC_TIME_VSYSCALL=y CONFIG_ZONE_DMA32=y CONFIG_LOCKDEP_SUPPORT=y CONFIG_STACKTRACE_SUPPORT=y CONFIG_SEMAPHORE_SLEEPERS=y CONFIG_MMU=y CONFIG_ZONE_DMA=y CONFIG_RWSEM_GENERIC_SPINLOCK=y CONFIG_GENERIC_HWEIGHT=y CONFIG_GENERIC_CALIBRATE_DELAY=y CONFIG_X86_CMPXCHG=y CONFIG_EARLY_PRINTK=y CONFIG_GENERIC_ISA_DMA=y CONFIG_GENERIC_IOMAP=y CONFIG_ARCH_MAY_HAVE_PC_FDC=y CONFIG_ARCH_POPULATES_NODE_MAP=y CONFIG_DMI=y CONFIG_AUDIT_ARCH=y CONFIG_GENERIC_BUG=y # CONFIG_ARCH_HAS_ILOG2_U32 is not set # CONFIG_ARCH_HAS_ILOG2_U64 is not set CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config" # # Code maturity level options # CONFIG_EXPERIMENTAL=y CONFIG_LOCK_KERNEL=y CONFIG_INIT_ENV_ARG_LIMIT=32 # # General setup # CONFIG_LOCALVERSION="-hg-slub" # CONFIG_LOCALVERSION_AUTO is not set CONFIG_SWAP=y CONFIG_SYSVIPC=y CONFIG_IPC_NS=y CONFIG_SYSVIPC_SYSCTL=y CONFIG_POSIX_MQUEUE=y CONFIG_BSD_PROCESS_ACCT=y # CONFIG_BSD_PROCESS_ACCT_V3 is not set # CONFIG_TASKSTATS is not set CONFIG_UTS_NS=y CONFIG_AUDIT=y CONFIG_AUDITSYSCALL=y # CONFIG_IKCONFIG is not set CONFIG_LOG_BUF_SHIFT=17 # CONFIG_CPUSETS is not set CONFIG_SYSFS_DEPRECATED=y CONFIG_RELAY=y CONFIG_BLK_DEV_INITRD=y CONFIG_INITRAMFS_SOURCE="" CONFIG_CC_OPTIMIZE_FOR_SIZE=y CONFIG_SYSCTL=y CONFIG_EMBEDDED=y CONFIG_UID16=y CONFIG_SYSCTL_SYSCALL=y CONFIG_KALLSYMS=y CONFIG_KALLSYMS_ALL=y CONFIG_KALLSYMS_EXTRA_PASS=y CONFIG_HOTPLUG=y CONFIG_PRINTK=y CONFIG_BUG=y CONFIG_ELF_CORE=y CONFIG_BASE_FULL=y CONFIG_FUTEX=y CONFIG_ANON_INODES=y CONFIG_EPOLL=y CONFIG_SIGNALFD=y CONFIG_TIMERFD=y CONFIG_EVENTFD=y CONFIG_SHMEM=y CONFIG_VM_EVENT_COUNTERS=y CONFIG_SLUB_DEBUG=y # CONFIG_SLAB is not set CONFIG_SLUB=y # CONFIG_SLOB is not set CONFIG_RT_MUTEXES=y # CONFIG_TINY_SHMEM is not set CONFIG_BASE_SMALL=0 # # Loadable module support # CONFIG_MODULES=y CONFIG_MODULE_UNLOAD=y # CONFIG_MODULE_FORCE_UNLOAD is not set CONFIG_MODVERSIONS=y CONFIG_MODULE_SRCVERSION_ALL=y CONFIG_KMOD=y CONFIG_STOP_MACHINE=y # # Block layer # CONFIG_BLOCK=y CONFIG_BLK_DEV_IO_TRACE=y # # IO Schedulers # CONFIG_IOSCHED_NOOP=y # CONFIG_IOSCHED_AS is not
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Wed, 23 May 2007, Srihari Vijayaraghavan wrote: > I'm personally very happy that slub works stably without slub debug options, > because that's what I'd run in a production env. Thanks to your patch, slub is > quite stable without the slub debug for me :-)). But it'd to nice to have a > working slub debug for test env., as you'd undoubtedly be aware of, of course > :-). Just my humble opinion. Just switching off lockdep should make you happy until we figure out what is wrong? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Wed, 23 May 2007, Srihari Vijayaraghavan wrote: > > and then try to boot without slub_debug. > > I guess you mean with CONFIG_SLUB_CONFIG=y? If so, I built another kernel with > CONFIG_SLUB_DEBUG=y (plus all of the above) & tested it. It panics by default, > but with slub_nomerge it works just fine (tested under moderate load). > > (the panic message produced by CONFIG_SLUB_DEBUG=y was the exact same call > trace as my very first email in this email thread with slightly different > address on a couple of functions, but rest remains the same) Ahh... At least we are getting to the original problem. > I'm personally very happy that slub works stably without slub debug options, > because that's what I'd run in a production env. Thanks to your patch, slub is > quite stable without the slub debug for me :-)). But it'd to nice to have a > working slub debug for test env., as you'd undoubtedly be aware of, of course > :-). Just my humble opinion. > > > If that fails then boot with slub_nomerge So lockdep has issues with slab merging? If locks are tracking within slabs then I imagine that lockdep gets confused if we put them together. > Yup, I had to use slub_nomerge; without that CONFIG_SLUB_DEBUG=y kernel > panics. (I haven't tested the UP case though. I did try nosmp & maxcpus=1, but > they had no effect on the panic. Do you want me to test UP case for > CONFIG_SLUB_DEBUG=y without slub_nomerge?) Yes. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Wed, 23 May 2007, Jens Axboe wrote: > That works for me with the patch, .config attached. H... That means the .config sent initially here was bogus. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Christoph Lameter <[EMAIL PROTECTED]> wrote: > On Wed, 23 May 2007, Srihari Vijayaraghavan wrote: [...] > Yup. compile with > > CONFIG_NUMA > > CONFIG_LOCKDEP > > CONFIG_DEBUG_LOCK_ALLOCS (All the tests in this email was conducted on top of your patch) Yup done that. The resulting kernel (without CONFIG_SLUB_DEBUG=y) is quite stable; passed a few iterations of kernel compile tests successfully. > and then try to boot without slub_debug. I guess you mean with CONFIG_SLUB_CONFIG=y? If so, I built another kernel with CONFIG_SLUB_DEBUG=y (plus all of the above) & tested it. It panics by default, but with slub_nomerge it works just fine (tested under moderate load). (the panic message produced by CONFIG_SLUB_DEBUG=y was the exact same call trace as my very first email in this email thread with slightly different address on a couple of functions, but rest remains the same) I'm personally very happy that slub works stably without slub debug options, because that's what I'd run in a production env. Thanks to your patch, slub is quite stable without the slub debug for me :-)). But it'd to nice to have a working slub debug for test env., as you'd undoubtedly be aware of, of course :-). Just my humble opinion. > If that fails then boot with slub_nomerge Yup, I had to use slub_nomerge; without that CONFIG_SLUB_DEBUG=y kernel panics. (I haven't tested the UP case though. I did try nosmp & maxcpus=1, but they had no effect on the panic. Do you want me to test UP case for CONFIG_SLUB_DEBUG=y without slub_nomerge?) Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Tue, May 22 2007, Christoph Lameter wrote: > On Wed, 23 May 2007, Srihari Vijayaraghavan wrote: > > > (If you want me to test it with other slub or kernel debug options please > > let > > me know. It just takes a lot of time to eliminate the variables, if there > > are > > problems.) > > Yup. compile with > > CONFIG_NUMA > > CONFIG_LOCKDEP > > CONFIG_DEBUG_LOCK_ALLOCS > > and then try to boot without slub_debug. That works for me with the patch, .config attached. -- Jens Axboe .config Description: application/config
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Tue, May 22 2007, Christoph Lameter wrote: On Wed, 23 May 2007, Srihari Vijayaraghavan wrote: (If you want me to test it with other slub or kernel debug options please let me know. It just takes a lot of time to eliminate the variables, if there are problems.) Yup. compile with CONFIG_NUMA CONFIG_LOCKDEP CONFIG_DEBUG_LOCK_ALLOCS and then try to boot without slub_debug. That works for me with the patch, .config attached. -- Jens Axboe .config Description: application/config
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Christoph Lameter [EMAIL PROTECTED] wrote: On Wed, 23 May 2007, Srihari Vijayaraghavan wrote: [...] Yup. compile with CONFIG_NUMA CONFIG_LOCKDEP CONFIG_DEBUG_LOCK_ALLOCS (All the tests in this email was conducted on top of your patch) Yup done that. The resulting kernel (without CONFIG_SLUB_DEBUG=y) is quite stable; passed a few iterations of kernel compile tests successfully. and then try to boot without slub_debug. I guess you mean with CONFIG_SLUB_CONFIG=y? If so, I built another kernel with CONFIG_SLUB_DEBUG=y (plus all of the above) tested it. It panics by default, but with slub_nomerge it works just fine (tested under moderate load). (the panic message produced by CONFIG_SLUB_DEBUG=y was the exact same call trace as my very first email in this email thread with slightly different address on a couple of functions, but rest remains the same) I'm personally very happy that slub works stably without slub debug options, because that's what I'd run in a production env. Thanks to your patch, slub is quite stable without the slub debug for me :-)). But it'd to nice to have a working slub debug for test env., as you'd undoubtedly be aware of, of course :-). Just my humble opinion. If that fails then boot with slub_nomerge Yup, I had to use slub_nomerge; without that CONFIG_SLUB_DEBUG=y kernel panics. (I haven't tested the UP case though. I did try nosmp maxcpus=1, but they had no effect on the panic. Do you want me to test UP case for CONFIG_SLUB_DEBUG=y without slub_nomerge?) Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Wed, 23 May 2007, Jens Axboe wrote: That works for me with the patch, .config attached. H... That means the .config sent initially here was bogus. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Wed, 23 May 2007, Srihari Vijayaraghavan wrote: and then try to boot without slub_debug. I guess you mean with CONFIG_SLUB_CONFIG=y? If so, I built another kernel with CONFIG_SLUB_DEBUG=y (plus all of the above) tested it. It panics by default, but with slub_nomerge it works just fine (tested under moderate load). (the panic message produced by CONFIG_SLUB_DEBUG=y was the exact same call trace as my very first email in this email thread with slightly different address on a couple of functions, but rest remains the same) Ahh... At least we are getting to the original problem. I'm personally very happy that slub works stably without slub debug options, because that's what I'd run in a production env. Thanks to your patch, slub is quite stable without the slub debug for me :-)). But it'd to nice to have a working slub debug for test env., as you'd undoubtedly be aware of, of course :-). Just my humble opinion. If that fails then boot with slub_nomerge So lockdep has issues with slab merging? If locks are tracking within slabs then I imagine that lockdep gets confused if we put them together. Yup, I had to use slub_nomerge; without that CONFIG_SLUB_DEBUG=y kernel panics. (I haven't tested the UP case though. I did try nosmp maxcpus=1, but they had no effect on the panic. Do you want me to test UP case for CONFIG_SLUB_DEBUG=y without slub_nomerge?) Yes. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Wed, 23 May 2007, Srihari Vijayaraghavan wrote: I'm personally very happy that slub works stably without slub debug options, because that's what I'd run in a production env. Thanks to your patch, slub is quite stable without the slub debug for me :-)). But it'd to nice to have a working slub debug for test env., as you'd undoubtedly be aware of, of course :-). Just my humble opinion. Just switching off lockdep should make you happy until we figure out what is wrong? - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Christoph Lameter [EMAIL PROTECTED] wrote: On Wed, 23 May 2007, Srihari Vijayaraghavan wrote: I'm personally very happy that slub works stably without slub debug options, because that's what I'd run in a production env. Thanks to your patch, slub is quite stable without the slub debug for me :-)). But it'd to nice to have a working slub debug for test env., as you'd undoubtedly be aware of, of course :-). Just my humble opinion. Just switching off lockdep should make you happy until we figure out what is wrong? I'm very happy :-). No problems. Has something important been merged in the last day or so to mainstream (http://www.kernel.org/hg/linux-2.6 starting hg commit 57110:0a8a879062a0)? On 57199:239bf3811cb2 with CONFIG_LOCKDEP, CONFIG_PROVE_LOCKING cousins with CONFIG_SLUB_DEBUG without slub_nomerge even, I'm unable to reproduce the panic now. My current .config is attached as config-2.6.22-rc2-hg-slub.txt I'm stunned. Honestly, I have no possible explanations for this behaviour. Do you? I need more time to work out (until otherwise you might know a reason). (Perhaps, when I introduced a few debug options in the kernel hacking section, I inadvertently introduced a timing issue that now masks the panic away. Just my hypothesis.) A few things did happen in the last 24 hours time): 1. I pulled 90 changesets from the linux-2.6 hg repo. Easy to rollback. 2. I did a yum update on my Fedora 7 test system (I don't believe it pulled anything important in ref to this problem). Not so easy to rollback. 3. Adding a few extra debug options in the kernel hacking section. Easy to rollback also. I need more time (something I really don't have, but it's my karma for kernel community) to eliminate the things. I'll now stop updating everything so not to introduce any new variables.) Oh, while at it, I created this patch (to ask you more info on what slub_nomerge is for): I'm also attaching my current working .config Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html --- linux-2.6-239bf3811cb2-slub/Documentation/kernel-parameters.txt.orig 2007-05-24 07:41:33.0 +1000 +++ linux-2.6-239bf3811cb2-slub/Documentation/kernel-parameters.txt 2007-05-24 07:43:34.0 +1000 @@ -1593,6 +1593,9 @@ slram= [HW,MTD] + slub_debug ... + slub_nomerge... + smart2= [HW] Format: io1[,io2[,...,io8]] # # Automatically generated make config: don't edit # Linux kernel version: 2.6.22-rc2 # Thu May 24 07:46:01 2007 # CONFIG_X86_64=y CONFIG_64BIT=y CONFIG_X86=y CONFIG_GENERIC_TIME=y CONFIG_GENERIC_TIME_VSYSCALL=y CONFIG_ZONE_DMA32=y CONFIG_LOCKDEP_SUPPORT=y CONFIG_STACKTRACE_SUPPORT=y CONFIG_SEMAPHORE_SLEEPERS=y CONFIG_MMU=y CONFIG_ZONE_DMA=y CONFIG_RWSEM_GENERIC_SPINLOCK=y CONFIG_GENERIC_HWEIGHT=y CONFIG_GENERIC_CALIBRATE_DELAY=y CONFIG_X86_CMPXCHG=y CONFIG_EARLY_PRINTK=y CONFIG_GENERIC_ISA_DMA=y CONFIG_GENERIC_IOMAP=y CONFIG_ARCH_MAY_HAVE_PC_FDC=y CONFIG_ARCH_POPULATES_NODE_MAP=y CONFIG_DMI=y CONFIG_AUDIT_ARCH=y CONFIG_GENERIC_BUG=y # CONFIG_ARCH_HAS_ILOG2_U32 is not set # CONFIG_ARCH_HAS_ILOG2_U64 is not set CONFIG_DEFCONFIG_LIST=/lib/modules/$UNAME_RELEASE/.config # # Code maturity level options # CONFIG_EXPERIMENTAL=y CONFIG_LOCK_KERNEL=y CONFIG_INIT_ENV_ARG_LIMIT=32 # # General setup # CONFIG_LOCALVERSION=-hg-slub # CONFIG_LOCALVERSION_AUTO is not set CONFIG_SWAP=y CONFIG_SYSVIPC=y CONFIG_IPC_NS=y CONFIG_SYSVIPC_SYSCTL=y CONFIG_POSIX_MQUEUE=y CONFIG_BSD_PROCESS_ACCT=y # CONFIG_BSD_PROCESS_ACCT_V3 is not set # CONFIG_TASKSTATS is not set CONFIG_UTS_NS=y CONFIG_AUDIT=y CONFIG_AUDITSYSCALL=y # CONFIG_IKCONFIG is not set CONFIG_LOG_BUF_SHIFT=17 # CONFIG_CPUSETS is not set CONFIG_SYSFS_DEPRECATED=y CONFIG_RELAY=y CONFIG_BLK_DEV_INITRD=y CONFIG_INITRAMFS_SOURCE= CONFIG_CC_OPTIMIZE_FOR_SIZE=y CONFIG_SYSCTL=y CONFIG_EMBEDDED=y CONFIG_UID16=y CONFIG_SYSCTL_SYSCALL=y CONFIG_KALLSYMS=y CONFIG_KALLSYMS_ALL=y CONFIG_KALLSYMS_EXTRA_PASS=y CONFIG_HOTPLUG=y CONFIG_PRINTK=y CONFIG_BUG=y CONFIG_ELF_CORE=y CONFIG_BASE_FULL=y CONFIG_FUTEX=y CONFIG_ANON_INODES=y CONFIG_EPOLL=y CONFIG_SIGNALFD=y CONFIG_TIMERFD=y CONFIG_EVENTFD=y CONFIG_SHMEM=y CONFIG_VM_EVENT_COUNTERS=y CONFIG_SLUB_DEBUG=y # CONFIG_SLAB is not set CONFIG_SLUB=y # CONFIG_SLOB is not set CONFIG_RT_MUTEXES=y # CONFIG_TINY_SHMEM is not set CONFIG_BASE_SMALL=0 # # Loadable module support # CONFIG_MODULES=y CONFIG_MODULE_UNLOAD=y # CONFIG_MODULE_FORCE_UNLOAD is not set CONFIG_MODVERSIONS=y CONFIG_MODULE_SRCVERSION_ALL=y CONFIG_KMOD=y CONFIG_STOP_MACHINE=y # # Block layer # CONFIG_BLOCK=y CONFIG_BLK_DEV_IO_TRACE=y # # IO Schedulers # CONFIG_IOSCHED_NOOP=y # CONFIG_IOSCHED_AS is not set #
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Thu, 24 May 2007, Srihari Vijayaraghavan wrote: I'm stunned. Honestly, I have no possible explanations for this behaviour. Do you? I need more time to work out (until otherwise you might know a reason). Hmmm... Bad. We have conflicting reports and no clear way to trigger the bug. This may actually only trigger depending on how certain objects is place in memory. Could you boot with slub_debug and then run slabinfo -v to validate all slabs? If there is anything wrong with an object then it should show in the syslog. slabinfo.c can be found at Documentation/vm/slabinfo.c Oh, while at it, I created this patch (to ask you more info on what slub_nomerge is for): See Documentation/vm/slub.txt which covers most of those. slub_nomerge is not mentioned. It removes an optimization and that is only useful for debugging strange cornercases. I hope to remove it someday. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Wed, 23 May 2007, Srihari Vijayaraghavan wrote: > (If you want me to test it with other slub or kernel debug options please let > me know. It just takes a lot of time to eliminate the variables, if there are > problems.) Yup. compile with CONFIG_NUMA CONFIG_LOCKDEP CONFIG_DEBUG_LOCK_ALLOCS and then try to boot without slub_debug. If that fails then boot with slub_nomerge - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Hugh Dickins <[EMAIL PROTECTED]> wrote: > On Tue, 22 May 2007, Srihari Vijayaraghavan wrote: [...] > > Compiled slub with SMP & CONFIG_PROVE_LOCKING. No luck. It still hangs > solid [...] > You've made no mention of trying the patch I sent yesterday, or better, > the patch Christoph replied with to replace it. Please clarify whether > you're getting the above after applying one of those patches - thanks. Sorry about that :-(. Was a busy day & skipped a few emails :-(. Indeed with Christoph's patch (over the hg commit 57109:745e2df8ec60), without CONFIG_SLUB_DEBUG, but with CONFIG_PROVE_LOCKING only under the kernel debugging section, on CONFIG_SMP the slub kerenel is quite stable. (It's running some stress tests now, but things are looking good.) (If you want me to test it with other slub or kernel debug options please let me know. It just takes a lot of time to eliminate the variables, if there are problems.) Thanks folks. ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Tue, May 22 2007, Christoph Lameter wrote: > On Tue, 22 May 2007, Jens Axboe wrote: > > > > You've made no mention of trying the patch I sent yesterday, or better, > > > the patch Christoph replied with to replace it. Please clarify whether > > > you're getting the above after applying one of those patches - thanks. > > > > Christophs patch works for me! > > So you are able to run slub with the patch and > > 1. CONFIG_NUMA set (CONFIG_SMP works fine here too) > > 2. CONFIG_LOCKDEP > > 3. CONFIG_DEBUG_LOCK_ALLOCS > > ? > > That fails here. I didn't say that. Previously a kernel without SLUB_DEBUG set would not work, with the patch it does. That is all I tested so far. -- Jens Axboe - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Tue, 22 May 2007, Jens Axboe wrote: > > You've made no mention of trying the patch I sent yesterday, or better, > > the patch Christoph replied with to replace it. Please clarify whether > > you're getting the above after applying one of those patches - thanks. > > Christophs patch works for me! So you are able to run slub with the patch and 1. CONFIG_NUMA set (CONFIG_SMP works fine here too) 2. CONFIG_LOCKDEP 3. CONFIG_DEBUG_LOCK_ALLOCS ? That fails here. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Tue, May 22 2007, Hugh Dickins wrote: > On Tue, 22 May 2007, Srihari Vijayaraghavan wrote: > > --- Ingo Molnar <[EMAIL PROTECTED]> wrote: > > > * Srihari Vijayaraghavan <[EMAIL PROTECTED]> wrote: > > > > Yup, with CONFIG_SMP=n, I'm unable to reproduce the problem. It's > > > > quite stable actually (having completed a dozen kernel compile > > > > sessions so far). > > > > [...] > > > > > could you enable CONFIG_PROVE_LOCKING - does it spit out any warning > > > into the syslog? > > > > Compiled slub with SMP & CONFIG_PROVE_LOCKING. No luck. It still hangs solid > > after the second spinlock lockup call trace. > > > > Here's the relevant sections of the kernel logs: > > > > ... > > Freeing unused kernel memory: 228k freed > > BUG: spinlock bad magic on CPU#1, init/1 > > lock: 81011f5f1100, .magic: 8101, .owner: /-1, .owner_cpu: -1 > > > > Call Trace: > > [] _raw_spin_lock+0x22/0xf6 > > [] vma_adjust+0x21c/0x446 > > [] vma_adjust+0x21c/0x446 > > [] vma_merge+0x10c/0x195 > > [] do_mmap_pgoff+0x3f5/0x794 > > [] _spin_unlock_irq+0x24/0x27 > > [] sys_mmap+0xe5/0x110 > > [] system_call+0x7e/0x83 > > ... > > PM: Adding info for No Bus:vcsa1 > > BUG: spinlock lockup on CPU#1, hostname/369, 81011f5f1fc0 > > > > Call Trace: > > [] _raw_spin_lock+0xcf/0xf6 > > [] anon_vma_unlink+0x1c/0x68 > > [] anon_vma_unlink+0x1c/0x68 > > [] free_pgtables+0x69/0xc4 > > [] exit_mmap+0x91/0xeb > > [] mmput+0x2c/0x9f > > [] do_exit+0x22e/0x82e > > [] sys_exit_group+0x0/0xe > > [] system_call+0x7e/0x83 > > > > > > Surprisingly, with CONFIG_SMP=n, CONFIG_PROVE_LOCKING produces this with > > slub > > (then hangs solid): > > > > Freeing unused kernel memory: 188k freed > > BUG: spinlock lockup on CPU#0, init/1, 81011e9d3160 > > > > Call Trace: > > [] _raw_spin_lock+0xca/0xe8 > > [] vma_adjust+0x218/0x442 > > [] vma_adjust+0x218/0x442 > > [] vma_merge+0x10c/0x195 > > [] do_mmap_pgoff+0x3f5/0x790 > > [] _spin_unlock_irq+0x24/0x27 > > [] sys_mmap+0xe5/0x110 > > [] system_call+0x7e/0x83 > > > > To recap: > > 1. No problems with slub on CONFIG_SMP=n & CONFIG_PROVE_LOCKING=n > > 2. Problem with slub on CONFIG_SMP=n & CONFIG_PROVE_LOCKING=y (perhaps a. > > some > > locking issue when slub is activated or b. something is wrong with 'prove > > locking' mechanism when slub is activated or c. something else I don't see) > > 3. Problem with slub on CONFIG_SMP=y (even without CONFIG_PROVE_LOCKING=y) > > You've made no mention of trying the patch I sent yesterday, or better, > the patch Christoph replied with to replace it. Please clarify whether > you're getting the above after applying one of those patches - thanks. Christophs patch works for me! -- Jens Axboe - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Tue, 22 May 2007, Srihari Vijayaraghavan wrote: > --- Ingo Molnar <[EMAIL PROTECTED]> wrote: > > * Srihari Vijayaraghavan <[EMAIL PROTECTED]> wrote: > > > Yup, with CONFIG_SMP=n, I'm unable to reproduce the problem. It's > > > quite stable actually (having completed a dozen kernel compile > > > sessions so far). > > [...] > > > could you enable CONFIG_PROVE_LOCKING - does it spit out any warning > > into the syslog? > > Compiled slub with SMP & CONFIG_PROVE_LOCKING. No luck. It still hangs solid > after the second spinlock lockup call trace. > > Here's the relevant sections of the kernel logs: > > ... > Freeing unused kernel memory: 228k freed > BUG: spinlock bad magic on CPU#1, init/1 > lock: 81011f5f1100, .magic: 8101, .owner: /-1, .owner_cpu: -1 > > Call Trace: > [] _raw_spin_lock+0x22/0xf6 > [] vma_adjust+0x21c/0x446 > [] vma_adjust+0x21c/0x446 > [] vma_merge+0x10c/0x195 > [] do_mmap_pgoff+0x3f5/0x794 > [] _spin_unlock_irq+0x24/0x27 > [] sys_mmap+0xe5/0x110 > [] system_call+0x7e/0x83 > ... > PM: Adding info for No Bus:vcsa1 > BUG: spinlock lockup on CPU#1, hostname/369, 81011f5f1fc0 > > Call Trace: > [] _raw_spin_lock+0xcf/0xf6 > [] anon_vma_unlink+0x1c/0x68 > [] anon_vma_unlink+0x1c/0x68 > [] free_pgtables+0x69/0xc4 > [] exit_mmap+0x91/0xeb > [] mmput+0x2c/0x9f > [] do_exit+0x22e/0x82e > [] sys_exit_group+0x0/0xe > [] system_call+0x7e/0x83 > > > Surprisingly, with CONFIG_SMP=n, CONFIG_PROVE_LOCKING produces this with slub > (then hangs solid): > > Freeing unused kernel memory: 188k freed > BUG: spinlock lockup on CPU#0, init/1, 81011e9d3160 > > Call Trace: > [] _raw_spin_lock+0xca/0xe8 > [] vma_adjust+0x218/0x442 > [] vma_adjust+0x218/0x442 > [] vma_merge+0x10c/0x195 > [] do_mmap_pgoff+0x3f5/0x790 > [] _spin_unlock_irq+0x24/0x27 > [] sys_mmap+0xe5/0x110 > [] system_call+0x7e/0x83 > > To recap: > 1. No problems with slub on CONFIG_SMP=n & CONFIG_PROVE_LOCKING=n > 2. Problem with slub on CONFIG_SMP=n & CONFIG_PROVE_LOCKING=y (perhaps a. some > locking issue when slub is activated or b. something is wrong with 'prove > locking' mechanism when slub is activated or c. something else I don't see) > 3. Problem with slub on CONFIG_SMP=y (even without CONFIG_PROVE_LOCKING=y) You've made no mention of trying the patch I sent yesterday, or better, the patch Christoph replied with to replace it. Please clarify whether you're getting the above after applying one of those patches - thanks. Hugh - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Ingo Molnar <[EMAIL PROTECTED]> wrote: [...] > yes - PROVE_LOCKING reactivates spinlocks even on UP. At least this > suggests that you'd have gotten the hang even with maxcpus=1 - i.e. the > spinlock corruption is not caused by some genuine SMP race. You're right on the mark there: even with maxcpus=1 or nosmp (or with both), with PROVE_LOCKING it hung with slub. I didn't mention that so not to confuse the matter, but you rightly figured that out too. Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
* Srihari Vijayaraghavan <[EMAIL PROTECTED]> wrote: > Compiled slub with SMP & CONFIG_PROVE_LOCKING. No luck. It still hangs > solid after the second spinlock lockup call trace. hm. This suggests that the spinlock got corrupted - otherwise lockdep would have complained about the lockup before the spinlock-debug code had its chance. > Surprisingly, with CONFIG_SMP=n, CONFIG_PROVE_LOCKING produces this > with slub (then hangs solid): yes - PROVE_LOCKING reactivates spinlocks even on UP. At least this suggests that you'd have gotten the hang even with maxcpus=1 - i.e. the spinlock corruption is not caused by some genuine SMP race. furthermore, PROVE_LOCKING also turns on DEBUG_LOCK_ALLOC, so we now know that it's most likely not use-after-free type of corruption. Ingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Ingo Molnar <[EMAIL PROTECTED]> wrote: > * Srihari Vijayaraghavan <[EMAIL PROTECTED]> wrote: > > Yup, with CONFIG_SMP=n, I'm unable to reproduce the problem. It's > > quite stable actually (having completed a dozen kernel compile > > sessions so far). [...] > could you enable CONFIG_PROVE_LOCKING - does it spit out any warning > into the syslog? Compiled slub with SMP & CONFIG_PROVE_LOCKING. No luck. It still hangs solid after the second spinlock lockup call trace. Here's the relevant sections of the kernel logs: ... Freeing unused kernel memory: 228k freed BUG: spinlock bad magic on CPU#1, init/1 lock: 81011f5f1100, .magic: 8101, .owner: /-1, .owner_cpu: -1 Call Trace: [] _raw_spin_lock+0x22/0xf6 [] vma_adjust+0x21c/0x446 [] vma_adjust+0x21c/0x446 [] vma_merge+0x10c/0x195 [] do_mmap_pgoff+0x3f5/0x794 [] _spin_unlock_irq+0x24/0x27 [] sys_mmap+0xe5/0x110 [] system_call+0x7e/0x83 ... PM: Adding info for No Bus:vcsa1 BUG: spinlock lockup on CPU#1, hostname/369, 81011f5f1fc0 Call Trace: [] _raw_spin_lock+0xcf/0xf6 [] anon_vma_unlink+0x1c/0x68 [] anon_vma_unlink+0x1c/0x68 [] free_pgtables+0x69/0xc4 [] exit_mmap+0x91/0xeb [] mmput+0x2c/0x9f [] do_exit+0x22e/0x82e [] sys_exit_group+0x0/0xe [] system_call+0x7e/0x83 Surprisingly, with CONFIG_SMP=n, CONFIG_PROVE_LOCKING produces this with slub (then hangs solid): Freeing unused kernel memory: 188k freed BUG: spinlock lockup on CPU#0, init/1, 81011e9d3160 Call Trace: [] _raw_spin_lock+0xca/0xe8 [] vma_adjust+0x218/0x442 [] vma_adjust+0x218/0x442 [] vma_merge+0x10c/0x195 [] do_mmap_pgoff+0x3f5/0x790 [] _spin_unlock_irq+0x24/0x27 [] sys_mmap+0xe5/0x110 [] system_call+0x7e/0x83 To recap: 1. No problems with slub on CONFIG_SMP=n & CONFIG_PROVE_LOCKING=n 2. Problem with slub on CONFIG_SMP=n & CONFIG_PROVE_LOCKING=y (perhaps a. some locking issue when slub is activated or b. something is wrong with 'prove locking' mechanism when slub is activated or c. something else I don't see) 3. Problem with slub on CONFIG_SMP=y (even without CONFIG_PROVE_LOCKING=y) Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
* Srihari Vijayaraghavan <[EMAIL PROTECTED]> wrote: > Just another data point: with the flick of CONFIG_SMP, I'm in control > of the hangs/crashes ;-). > > Yup, with CONFIG_SMP=n, I'm unable to reproduce the problem. It's > quite stable actually (having completed a dozen kernel compile > sessions so far). > > I suspected this after seeing spinlock issues on both cores of my CPU > in my earlier report. Just a stab in the dark with some unexpected > luck :-). could you enable CONFIG_PROVE_LOCKING - does it spit out any warning into the syslog? Ingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
Just another data point: with the flick of CONFIG_SMP, I'm in control of the hangs/crashes ;-). Yup, with CONFIG_SMP=n, I'm unable to reproduce the problem. It's quite stable actually (having completed a dozen kernel compile sessions so far). I suspected this after seeing spinlock issues on both cores of my CPU in my earlier report. Just a stab in the dark with some unexpected luck :-). (Tested on http://www.kernel.org/hg/linux-2.6 changeset at 57106:3f42ad5de7cd. That's the latest hg commit there.) Thanks PS: For the record everything except sysrq support is disabled in the kernel hacking section. Also the CONFIG_SLAB_DEBUG is off. Kernel preemption is also off, after proving it had no influence. Just to keep the test case simple. ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, May 21 2007, Christoph Lameter wrote: > Ok booting SLUB without "slub_debug" and having > > CONFIG_DEBUG_LOCK_ALLOC > CONFIG_LOCKDEP > > will trigger the problem. > > So I guess the issue is that lockdep does a slab alloc while we get the > slab lock during slab alloc? If I have CONFIG_SLUB_DEBUG set, then box boots fine. If I don't have it set, then it hard hangs sometime after boot: [...] ReiserFS: sda2: Using r5 hash to sort names VFS: Mounted root (reiserfs filesystem) readonly. Freeing unused kernel memory: 180k freed which is usually followed by messages bringing up the network link, etc. Let me know which tests you want run, or which patches to try. -- Jens Axboe - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Tue, 22 May 2007, Jens Axboe wrote: You've made no mention of trying the patch I sent yesterday, or better, the patch Christoph replied with to replace it. Please clarify whether you're getting the above after applying one of those patches - thanks. Christophs patch works for me! So you are able to run slub with the patch and 1. CONFIG_NUMA set (CONFIG_SMP works fine here too) 2. CONFIG_LOCKDEP 3. CONFIG_DEBUG_LOCK_ALLOCS ? That fails here. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Tue, May 22 2007, Christoph Lameter wrote: On Tue, 22 May 2007, Jens Axboe wrote: You've made no mention of trying the patch I sent yesterday, or better, the patch Christoph replied with to replace it. Please clarify whether you're getting the above after applying one of those patches - thanks. Christophs patch works for me! So you are able to run slub with the patch and 1. CONFIG_NUMA set (CONFIG_SMP works fine here too) 2. CONFIG_LOCKDEP 3. CONFIG_DEBUG_LOCK_ALLOCS ? That fails here. I didn't say that. Previously a kernel without SLUB_DEBUG set would not work, with the patch it does. That is all I tested so far. -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Hugh Dickins [EMAIL PROTECTED] wrote: On Tue, 22 May 2007, Srihari Vijayaraghavan wrote: [...] Compiled slub with SMP CONFIG_PROVE_LOCKING. No luck. It still hangs solid [...] You've made no mention of trying the patch I sent yesterday, or better, the patch Christoph replied with to replace it. Please clarify whether you're getting the above after applying one of those patches - thanks. Sorry about that :-(. Was a busy day skipped a few emails :-(. Indeed with Christoph's patch (over the hg commit 57109:745e2df8ec60), without CONFIG_SLUB_DEBUG, but with CONFIG_PROVE_LOCKING only under the kernel debugging section, on CONFIG_SMP the slub kerenel is quite stable. (It's running some stress tests now, but things are looking good.) (If you want me to test it with other slub or kernel debug options please let me know. It just takes a lot of time to eliminate the variables, if there are problems.) Thanks folks. ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Wed, 23 May 2007, Srihari Vijayaraghavan wrote: (If you want me to test it with other slub or kernel debug options please let me know. It just takes a lot of time to eliminate the variables, if there are problems.) Yup. compile with CONFIG_NUMA CONFIG_LOCKDEP CONFIG_DEBUG_LOCK_ALLOCS and then try to boot without slub_debug. If that fails then boot with slub_nomerge - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, May 21 2007, Christoph Lameter wrote: Ok booting SLUB without slub_debug and having CONFIG_DEBUG_LOCK_ALLOC CONFIG_LOCKDEP will trigger the problem. So I guess the issue is that lockdep does a slab alloc while we get the slab lock during slab alloc? If I have CONFIG_SLUB_DEBUG set, then box boots fine. If I don't have it set, then it hard hangs sometime after boot: [...] ReiserFS: sda2: Using r5 hash to sort names VFS: Mounted root (reiserfs filesystem) readonly. Freeing unused kernel memory: 180k freed which is usually followed by messages bringing up the network link, etc. Let me know which tests you want run, or which patches to try. -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
Just another data point: with the flick of CONFIG_SMP, I'm in control of the hangs/crashes ;-). Yup, with CONFIG_SMP=n, I'm unable to reproduce the problem. It's quite stable actually (having completed a dozen kernel compile sessions so far). I suspected this after seeing spinlock issues on both cores of my CPU in my earlier report. Just a stab in the dark with some unexpected luck :-). (Tested on http://www.kernel.org/hg/linux-2.6 changeset at 57106:3f42ad5de7cd. That's the latest hg commit there.) Thanks PS: For the record everything except sysrq support is disabled in the kernel hacking section. Also the CONFIG_SLAB_DEBUG is off. Kernel preemption is also off, after proving it had no influence. Just to keep the test case simple. ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
* Srihari Vijayaraghavan [EMAIL PROTECTED] wrote: Just another data point: with the flick of CONFIG_SMP, I'm in control of the hangs/crashes ;-). Yup, with CONFIG_SMP=n, I'm unable to reproduce the problem. It's quite stable actually (having completed a dozen kernel compile sessions so far). I suspected this after seeing spinlock issues on both cores of my CPU in my earlier report. Just a stab in the dark with some unexpected luck :-). could you enable CONFIG_PROVE_LOCKING - does it spit out any warning into the syslog? Ingo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Ingo Molnar [EMAIL PROTECTED] wrote: * Srihari Vijayaraghavan [EMAIL PROTECTED] wrote: Yup, with CONFIG_SMP=n, I'm unable to reproduce the problem. It's quite stable actually (having completed a dozen kernel compile sessions so far). [...] could you enable CONFIG_PROVE_LOCKING - does it spit out any warning into the syslog? Compiled slub with SMP CONFIG_PROVE_LOCKING. No luck. It still hangs solid after the second spinlock lockup call trace. Here's the relevant sections of the kernel logs: ... Freeing unused kernel memory: 228k freed BUG: spinlock bad magic on CPU#1, init/1 lock: 81011f5f1100, .magic: 8101, .owner: none/-1, .owner_cpu: -1 Call Trace: [802f326a] _raw_spin_lock+0x22/0xf6 [8026b2d5] vma_adjust+0x21c/0x446 [8026b2d5] vma_adjust+0x21c/0x446 [8026b9d4] vma_merge+0x10c/0x195 [8026c757] do_mmap_pgoff+0x3f5/0x794 [803fff0c] _spin_unlock_irq+0x24/0x27 [8020f414] sys_mmap+0xe5/0x110 [80209dde] system_call+0x7e/0x83 ... PM: Adding info for No Bus:vcsa1 BUG: spinlock lockup on CPU#1, hostname/369, 81011f5f1fc0 Call Trace: [802f3317] _raw_spin_lock+0xcf/0xf6 [8026ec9c] anon_vma_unlink+0x1c/0x68 [8026ec9c] anon_vma_unlink+0x1c/0x68 [80269aa0] free_pgtables+0x69/0xc4 [8026ad0e] exit_mmap+0x91/0xeb [80228cea] mmput+0x2c/0x9f [8022df72] do_exit+0x22e/0x82e [8022e5f4] sys_exit_group+0x0/0xe [80209dde] system_call+0x7e/0x83 Surprisingly, with CONFIG_SMP=n, CONFIG_PROVE_LOCKING produces this with slub (then hangs solid): Freeing unused kernel memory: 188k freed BUG: spinlock lockup on CPU#0, init/1, 81011e9d3160 Call Trace: [802eca20] _raw_spin_lock+0xca/0xe8 [80265d6d] vma_adjust+0x218/0x442 [80265d6d] vma_adjust+0x218/0x442 [8026646b] vma_merge+0x10c/0x195 [802671d5] do_mmap_pgoff+0x3f5/0x790 [803f6e84] _spin_unlock_irq+0x24/0x27 [8020ead0] sys_mmap+0xe5/0x110 [80209cce] system_call+0x7e/0x83 To recap: 1. No problems with slub on CONFIG_SMP=n CONFIG_PROVE_LOCKING=n 2. Problem with slub on CONFIG_SMP=n CONFIG_PROVE_LOCKING=y (perhaps a. some locking issue when slub is activated or b. something is wrong with 'prove locking' mechanism when slub is activated or c. something else I don't see) 3. Problem with slub on CONFIG_SMP=y (even without CONFIG_PROVE_LOCKING=y) Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
* Srihari Vijayaraghavan [EMAIL PROTECTED] wrote: Compiled slub with SMP CONFIG_PROVE_LOCKING. No luck. It still hangs solid after the second spinlock lockup call trace. hm. This suggests that the spinlock got corrupted - otherwise lockdep would have complained about the lockup before the spinlock-debug code had its chance. Surprisingly, with CONFIG_SMP=n, CONFIG_PROVE_LOCKING produces this with slub (then hangs solid): yes - PROVE_LOCKING reactivates spinlocks even on UP. At least this suggests that you'd have gotten the hang even with maxcpus=1 - i.e. the spinlock corruption is not caused by some genuine SMP race. furthermore, PROVE_LOCKING also turns on DEBUG_LOCK_ALLOC, so we now know that it's most likely not use-after-free type of corruption. Ingo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Ingo Molnar [EMAIL PROTECTED] wrote: [...] yes - PROVE_LOCKING reactivates spinlocks even on UP. At least this suggests that you'd have gotten the hang even with maxcpus=1 - i.e. the spinlock corruption is not caused by some genuine SMP race. You're right on the mark there: even with maxcpus=1 or nosmp (or with both), with PROVE_LOCKING it hung with slub. I didn't mention that so not to confuse the matter, but you rightly figured that out too. Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Tue, 22 May 2007, Srihari Vijayaraghavan wrote: --- Ingo Molnar [EMAIL PROTECTED] wrote: * Srihari Vijayaraghavan [EMAIL PROTECTED] wrote: Yup, with CONFIG_SMP=n, I'm unable to reproduce the problem. It's quite stable actually (having completed a dozen kernel compile sessions so far). [...] could you enable CONFIG_PROVE_LOCKING - does it spit out any warning into the syslog? Compiled slub with SMP CONFIG_PROVE_LOCKING. No luck. It still hangs solid after the second spinlock lockup call trace. Here's the relevant sections of the kernel logs: ... Freeing unused kernel memory: 228k freed BUG: spinlock bad magic on CPU#1, init/1 lock: 81011f5f1100, .magic: 8101, .owner: none/-1, .owner_cpu: -1 Call Trace: [802f326a] _raw_spin_lock+0x22/0xf6 [8026b2d5] vma_adjust+0x21c/0x446 [8026b2d5] vma_adjust+0x21c/0x446 [8026b9d4] vma_merge+0x10c/0x195 [8026c757] do_mmap_pgoff+0x3f5/0x794 [803fff0c] _spin_unlock_irq+0x24/0x27 [8020f414] sys_mmap+0xe5/0x110 [80209dde] system_call+0x7e/0x83 ... PM: Adding info for No Bus:vcsa1 BUG: spinlock lockup on CPU#1, hostname/369, 81011f5f1fc0 Call Trace: [802f3317] _raw_spin_lock+0xcf/0xf6 [8026ec9c] anon_vma_unlink+0x1c/0x68 [8026ec9c] anon_vma_unlink+0x1c/0x68 [80269aa0] free_pgtables+0x69/0xc4 [8026ad0e] exit_mmap+0x91/0xeb [80228cea] mmput+0x2c/0x9f [8022df72] do_exit+0x22e/0x82e [8022e5f4] sys_exit_group+0x0/0xe [80209dde] system_call+0x7e/0x83 Surprisingly, with CONFIG_SMP=n, CONFIG_PROVE_LOCKING produces this with slub (then hangs solid): Freeing unused kernel memory: 188k freed BUG: spinlock lockup on CPU#0, init/1, 81011e9d3160 Call Trace: [802eca20] _raw_spin_lock+0xca/0xe8 [80265d6d] vma_adjust+0x218/0x442 [80265d6d] vma_adjust+0x218/0x442 [8026646b] vma_merge+0x10c/0x195 [802671d5] do_mmap_pgoff+0x3f5/0x790 [803f6e84] _spin_unlock_irq+0x24/0x27 [8020ead0] sys_mmap+0xe5/0x110 [80209cce] system_call+0x7e/0x83 To recap: 1. No problems with slub on CONFIG_SMP=n CONFIG_PROVE_LOCKING=n 2. Problem with slub on CONFIG_SMP=n CONFIG_PROVE_LOCKING=y (perhaps a. some locking issue when slub is activated or b. something is wrong with 'prove locking' mechanism when slub is activated or c. something else I don't see) 3. Problem with slub on CONFIG_SMP=y (even without CONFIG_PROVE_LOCKING=y) You've made no mention of trying the patch I sent yesterday, or better, the patch Christoph replied with to replace it. Please clarify whether you're getting the above after applying one of those patches - thanks. Hugh - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Tue, May 22 2007, Hugh Dickins wrote: On Tue, 22 May 2007, Srihari Vijayaraghavan wrote: --- Ingo Molnar [EMAIL PROTECTED] wrote: * Srihari Vijayaraghavan [EMAIL PROTECTED] wrote: Yup, with CONFIG_SMP=n, I'm unable to reproduce the problem. It's quite stable actually (having completed a dozen kernel compile sessions so far). [...] could you enable CONFIG_PROVE_LOCKING - does it spit out any warning into the syslog? Compiled slub with SMP CONFIG_PROVE_LOCKING. No luck. It still hangs solid after the second spinlock lockup call trace. Here's the relevant sections of the kernel logs: ... Freeing unused kernel memory: 228k freed BUG: spinlock bad magic on CPU#1, init/1 lock: 81011f5f1100, .magic: 8101, .owner: none/-1, .owner_cpu: -1 Call Trace: [802f326a] _raw_spin_lock+0x22/0xf6 [8026b2d5] vma_adjust+0x21c/0x446 [8026b2d5] vma_adjust+0x21c/0x446 [8026b9d4] vma_merge+0x10c/0x195 [8026c757] do_mmap_pgoff+0x3f5/0x794 [803fff0c] _spin_unlock_irq+0x24/0x27 [8020f414] sys_mmap+0xe5/0x110 [80209dde] system_call+0x7e/0x83 ... PM: Adding info for No Bus:vcsa1 BUG: spinlock lockup on CPU#1, hostname/369, 81011f5f1fc0 Call Trace: [802f3317] _raw_spin_lock+0xcf/0xf6 [8026ec9c] anon_vma_unlink+0x1c/0x68 [8026ec9c] anon_vma_unlink+0x1c/0x68 [80269aa0] free_pgtables+0x69/0xc4 [8026ad0e] exit_mmap+0x91/0xeb [80228cea] mmput+0x2c/0x9f [8022df72] do_exit+0x22e/0x82e [8022e5f4] sys_exit_group+0x0/0xe [80209dde] system_call+0x7e/0x83 Surprisingly, with CONFIG_SMP=n, CONFIG_PROVE_LOCKING produces this with slub (then hangs solid): Freeing unused kernel memory: 188k freed BUG: spinlock lockup on CPU#0, init/1, 81011e9d3160 Call Trace: [802eca20] _raw_spin_lock+0xca/0xe8 [80265d6d] vma_adjust+0x218/0x442 [80265d6d] vma_adjust+0x218/0x442 [8026646b] vma_merge+0x10c/0x195 [802671d5] do_mmap_pgoff+0x3f5/0x790 [803f6e84] _spin_unlock_irq+0x24/0x27 [8020ead0] sys_mmap+0xe5/0x110 [80209cce] system_call+0x7e/0x83 To recap: 1. No problems with slub on CONFIG_SMP=n CONFIG_PROVE_LOCKING=n 2. Problem with slub on CONFIG_SMP=n CONFIG_PROVE_LOCKING=y (perhaps a. some locking issue when slub is activated or b. something is wrong with 'prove locking' mechanism when slub is activated or c. something else I don't see) 3. Problem with slub on CONFIG_SMP=y (even without CONFIG_PROVE_LOCKING=y) You've made no mention of trying the patch I sent yesterday, or better, the patch Christoph replied with to replace it. Please clarify whether you're getting the above after applying one of those patches - thanks. Christophs patch works for me! -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Christoph Lameter wrote: > So I guess the issue is that lockdep does a slab alloc while we get the > slab lock during slab alloc? Lockdep is not available on IA64 where I would be able to figure it out using a simulator. x86_64 early printk support seems to be broken? No output with earlyprintk=serial. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
Ok booting SLUB without "slub_debug" and having CONFIG_DEBUG_LOCK_ALLOC CONFIG_LOCKDEP will trigger the problem. So I guess the issue is that lockdep does a slab alloc while we get the slab lock during slab alloc? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Hugh Dickins wrote: > > the problem. And the earlier case that you reported was a version of SLUB > > that did not have the capability to switch off SLUB_DEBUG. > > Are you sure? If CONFIG_EMBEDDED=y then -rc1 and -rc2 give you the > choice to turn it off. Right. That went in indeed shortly before the rc1 release. So if both had disabled SLUB_DEBUG then this could be the reason for the failure. However, CONFIG_SLUB_DEBUG was set in the .config that was included when this bug was reported. So I think we still have not found the real problem yet. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Christoph Lameter wrote: > On Mon, 21 May 2007, Jens Axboe wrote: > > > I can test whatever you want tomorrow morning, it was 100% repeatable > > here. So which one, your patch or Hughs? > > This was an effect of me suggesting to switch off CONFIG_SLUB_DEBUG. If > you did not run without CONFIG_SLUB_DEBUG then you were not affected by > the problem. And the earlier case that you reported was a version of SLUB > that did not have the capability to switch off SLUB_DEBUG. Are you sure? If CONFIG_EMBEDDED=y then -rc1 and -rc2 give you the choice to turn it off. > Is there any way we can boot with "slub_debug" on the kernel command line > and get early printk output? Looks like an early failure that may have to > do with other kernel debug options being on (lockdep or so?) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, May 21 2007, Christoph Lameter wrote: > On Mon, 21 May 2007, Jens Axboe wrote: > > > I can test whatever you want tomorrow morning, it was 100% repeatable > > here. So which one, your patch or Hughs? > > This was an effect of me suggesting to switch off CONFIG_SLUB_DEBUG. If > you did not run without CONFIG_SLUB_DEBUG then you were not affected by > the problem. And the earlier case that you reported was a version of SLUB > that did not have the capability to switch off SLUB_DEBUG. > > Is there any way we can boot with "slub_debug" on the kernel command line > and get early printk output? Looks like an early failure that may have to > do with other kernel debug options being on (lockdep or so?) Sure, I will try that as well. For reference, the last kernel tested on that box was HEAD 129a84de2347002f09721cda3155ccfd19fade40. -- Jens Axboe - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, May 21 2007, Hugh Dickins wrote: > On Mon, 21 May 2007, Jens Axboe wrote: > > > > I can test whatever you want tomorrow morning, it was 100% repeatable > > here. So which one, your patch or Hughs? > > Great, thanks: Christoph's please - I'm sure he'll agree! Alright, will do! -- Jens Axboe - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Jens Axboe wrote: > I can test whatever you want tomorrow morning, it was 100% repeatable > here. So which one, your patch or Hughs? This was an effect of me suggesting to switch off CONFIG_SLUB_DEBUG. If you did not run without CONFIG_SLUB_DEBUG then you were not affected by the problem. And the earlier case that you reported was a version of SLUB that did not have the capability to switch off SLUB_DEBUG. Is there any way we can boot with "slub_debug" on the kernel command line and get early printk output? Looks like an early failure that may have to do with other kernel debug options being on (lockdep or so?) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Christoph Lameter wrote: > On Mon, 21 May 2007, Hugh Dickins wrote: > > On Mon, 21 May 2007, Christoph Lameter wrote: > > > On Mon, 21 May 2007, Hugh Dickins wrote: > > > > SLUB Debug: Fix object size calculation > > > > > > The object size calculation is wrong if !CONFIG_SLUB_DEBUG because > > > the #ifdef CONFIG_SLUB_DEBUG is now switching off the size adjustments > > > for DESTROY_BY_RCU and ctor. > > > > > > Signed-off-by: Christoph Lameter <[EMAIL PROTECTED]> Acked-by: Hugh Dickins <[EMAIL PROTECTED]> - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Jens Axboe wrote: > > I can test whatever you want tomorrow morning, it was 100% repeatable > here. So which one, your patch or Hughs? Great, thanks: Christoph's please - I'm sure he'll agree! Hugh - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, May 21 2007, Christoph Lameter wrote: > On Mon, 21 May 2007, Hugh Dickins wrote: > > > On Mon, 21 May 2007, Christoph Lameter wrote: > > > On Mon, 21 May 2007, Hugh Dickins wrote: > > > > > > > Yes, sounded the same to me too: I couldn't reproduce it or see anything > > > > wrong in the code back then. But Srihari's info about CONFIG_DEBUG_SLUB > > > > off has helped a lot: I was then able to reproduce it on my x86_64, and > > > > after a lot of staring at the code, the problem became obvious... > > > > > > Right. The #ifdef CONFIG_SLUB_DEBUG is at the wrong location. The best fix > > > is to moving the #ifdef otherwise the size is still wrong for the > > > ctor case. > > > > ? My patch did handle the ctor case. > > True. I was thinking about just checking the problem case that we had > here. > > > > SLUB Debug: Fix object size calculation > > > > > > The object size calculation is wrong if !CONFIG_SLUB_DEBUG because > > > the #ifdef CONFIG_SLUB_DEBUG is now switching off the size adjustments > > > for DESTROY_BY_RCU and ctor. > > > > > > Signed-off-by: Christoph Lameter <[EMAIL PROTECTED]> > > > > Yes, I think that should do it too. The reason behind my repeating > > the block was to handle the case where SLAB_POISON is passed to > > kmem_cache_create, but CONFIG_SLUB_DEBUG is off. But apparently > > that case would hit the BUG_ON(flags & ~CREATE_MASK), therefore > > your patch is simpler and better. Quite a maze. > > Would you ack my patch? I do not want to repeat the block. I can test whatever you want tomorrow morning, it was 100% repeatable here. So which one, your patch or Hughs? -- Jens Axboe - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Hugh Dickins wrote: > On Mon, 21 May 2007, Christoph Lameter wrote: > > On Mon, 21 May 2007, Hugh Dickins wrote: > > > > > Yes, sounded the same to me too: I couldn't reproduce it or see anything > > > wrong in the code back then. But Srihari's info about CONFIG_DEBUG_SLUB > > > off has helped a lot: I was then able to reproduce it on my x86_64, and > > > after a lot of staring at the code, the problem became obvious... > > > > Right. The #ifdef CONFIG_SLUB_DEBUG is at the wrong location. The best fix > > is to moving the #ifdef otherwise the size is still wrong for the > > ctor case. > > ? My patch did handle the ctor case. True. I was thinking about just checking the problem case that we had here. > > SLUB Debug: Fix object size calculation > > > > The object size calculation is wrong if !CONFIG_SLUB_DEBUG because > > the #ifdef CONFIG_SLUB_DEBUG is now switching off the size adjustments > > for DESTROY_BY_RCU and ctor. > > > > Signed-off-by: Christoph Lameter <[EMAIL PROTECTED]> > > Yes, I think that should do it too. The reason behind my repeating > the block was to handle the case where SLAB_POISON is passed to > kmem_cache_create, but CONFIG_SLUB_DEBUG is off. But apparently > that case would hit the BUG_ON(flags & ~CREATE_MASK), therefore > your patch is simpler and better. Quite a maze. Would you ack my patch? I do not want to repeat the block. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Christoph Lameter wrote:
> On Mon, 21 May 2007, Hugh Dickins wrote:
>
> > Yes, sounded the same to me too: I couldn't reproduce it or see anything
> > wrong in the code back then. But Srihari's info about CONFIG_DEBUG_SLUB
> > off has helped a lot: I was then able to reproduce it on my x86_64, and
> > after a lot of staring at the code, the problem became obvious...
>
> Right. The #ifdef CONFIG_SLUB_DEBUG is at the wrong location. The best fix
> is to moving the #ifdef otherwise the size is still wrong for the
> ctor case.
? My patch did handle the ctor case.
>
> SLUB Debug: Fix object size calculation
>
> The object size calculation is wrong if !CONFIG_SLUB_DEBUG because
> the #ifdef CONFIG_SLUB_DEBUG is now switching off the size adjustments
> for DESTROY_BY_RCU and ctor.
>
> Signed-off-by: Christoph Lameter <[EMAIL PROTECTED]>
Yes, I think that should do it too. The reason behind my repeating
the block was to handle the case where SLAB_POISON is passed to
kmem_cache_create, but CONFIG_SLUB_DEBUG is off. But apparently
that case would hit the BUG_ON(flags & ~CREATE_MASK), therefore
your patch is simpler and better. Quite a maze.
Hugh
>
> ---
> mm/slub.c |2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> Index: slub/mm/slub.c
> ===
> --- slub.orig/mm/slub.c 2007-05-21 11:21:36.0 -0700
> +++ slub/mm/slub.c2007-05-21 11:21:49.0 -0700
> @@ -1943,7 +1943,6 @@ static int calculate_sizes(struct kmem_c
>*/
> s->inuse = size;
>
> -#ifdef CONFIG_SLUB_DEBUG
> if (((flags & (SLAB_DESTROY_BY_RCU | SLAB_POISON)) ||
> s->ctor)) {
> /*
> @@ -1958,6 +1957,7 @@ static int calculate_sizes(struct kmem_c
> size += sizeof(void *);
> }
>
> +#ifdef CONFIG_SLUB_DEBUG
> if (flags & SLAB_STORE_USER)
> /*
>* Need to store information about allocs and frees after
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Hugh Dickins wrote:
> Yes, sounded the same to me too: I couldn't reproduce it or see anything
> wrong in the code back then. But Srihari's info about CONFIG_DEBUG_SLUB
> off has helped a lot: I was then able to reproduce it on my x86_64, and
> after a lot of staring at the code, the problem became obvious...
Right. The #ifdef CONFIG_SLUB_DEBUG is at the wrong location. The best fix
is to moving the #ifdef otherwise the size is still wrong for the
ctor case.
SLUB Debug: Fix object size calculation
The object size calculation is wrong if !CONFIG_SLUB_DEBUG because
the #ifdef CONFIG_SLUB_DEBUG is now switching off the size adjustments
for DESTROY_BY_RCU and ctor.
Signed-off-by: Christoph Lameter <[EMAIL PROTECTED]>
---
mm/slub.c |2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Index: slub/mm/slub.c
===
--- slub.orig/mm/slub.c 2007-05-21 11:21:36.0 -0700
+++ slub/mm/slub.c 2007-05-21 11:21:49.0 -0700
@@ -1943,7 +1943,6 @@ static int calculate_sizes(struct kmem_c
*/
s->inuse = size;
-#ifdef CONFIG_SLUB_DEBUG
if (((flags & (SLAB_DESTROY_BY_RCU | SLAB_POISON)) ||
s->ctor)) {
/*
@@ -1958,6 +1957,7 @@ static int calculate_sizes(struct kmem_c
size += sizeof(void *);
}
+#ifdef CONFIG_SLUB_DEBUG
if (flags & SLAB_STORE_USER)
/*
* Need to store information about allocs and frees after
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Christoph Lameter wrote:
> On Sun, 20 May 2007, Srihari Vijayaraghavan wrote:
>
> > With no CONFIG_SLUB_DEBUG, things have slightly improved. No more panic.
> > Good.
> > Serial console is working. Good. But there is another problem:
>
> > Freeing unused kernel memory: 308k freed
> > BUG: spinlock bad magic on CPU#1, init/1
> > lock: 81011ec0a100, .magic: 8101, .owner: /-1, .owner_cpu: -1
> >
> > Call Trace:
> > [] _raw_spin_lock+0x22/0xf6
> > [] vma_adjust+0x219/0x454
> > [] vma_adjust+0x219/0x454
> > [] vma_merge+0x147/0x1f4
> > [] do_mmap_pgoff+0x414/0x7c7
> > [] _spin_unlock_irq+0x24/0x27
> > [] sys_mmap+0xe5/0x110
> > [] system_call+0x7e/0x83
>
> H. We have seen this before
>
> http://marc.info/?l=linux-kernel=117891943401284=2
Yes, sounded the same to me too: I couldn't reproduce it or see anything
wrong in the code back then. But Srihari's info about CONFIG_DEBUG_SLUB
off has helped a lot: I was then able to reproduce it on my x86_64, and
after a lot of staring at the code, the problem became obvious...
[PATCH] slub: fix size adjustment when SLUB_DEBUG off
SLUB was corrupting the anon_vma's spinlock when not configured for
SLUB_DEBUG: the size adjustment necessary for SLAB_DESTROY_BY_RCU or
constructor was inside the CONFIG_SLUB_DEBUG appropriate for poisoning.
Signed-off-by: Hugh Dickins <[EMAIL PROTECTED]>
---
mm/slub.c | 18 ++
1 file changed, 14 insertions(+), 4 deletions(-)
--- 2.6.22-rc2/mm/slub.c2007-05-21 13:13:20.0 +0100
+++ linux/mm/slub.c 2007-05-21 18:05:57.0 +0100
@@ -1917,16 +1917,26 @@ static int calculate_sizes(struct kmem_c
*/
s->inuse = size;
+ if ((flags & SLAB_DESTROY_BY_RCU) || s->ctor) {
+ /*
+* Relocate free pointer after the object if it is not
+* permitted to overwrite the first word of the object on
+* kmem_cache_free.
+*
+* This is the case if we do RCU or have a constructor.
+*/
+ s->offset = size;
+ size += sizeof(void *);
+ }
+
#ifdef CONFIG_SLUB_DEBUG
- if (((flags & (SLAB_DESTROY_BY_RCU | SLAB_POISON)) ||
- s->ctor)) {
+ if (s->flags & __OBJECT_POISON) {
/*
* Relocate free pointer after the object if it is not
* permitted to overwrite the first word of the object on
* kmem_cache_free.
*
-* This is the case if we do RCU, have a constructor or
-* destructor or are poisoning the objects.
+* This is the case if we are poisoning the objects.
*/
s->offset = size;
size += sizeof(void *);
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Sun, 20 May 2007, Srihari Vijayaraghavan wrote: > With no CONFIG_SLUB_DEBUG, things have slightly improved. No more panic. Good. > Serial console is working. Good. But there is another problem: Well this indicates that something destroys the sysfs pointer structure that SLUB is using. Could you reenable CONFIG_SLUB_DEBUG and boot with the option "slub_debug" on the kernel command line? Also enable the list debug options in kernel hacking. This should show up on the console since the failure is in slub_sysfs_init which is long after the console has been enabled. I wonder why it did not show so far? > Freeing unused kernel memory: 308k freed > BUG: spinlock bad magic on CPU#1, init/1 > lock: 81011ec0a100, .magic: 8101, .owner: /-1, .owner_cpu: -1 > > Call Trace: > [] _raw_spin_lock+0x22/0xf6 > [] vma_adjust+0x219/0x454 > [] vma_adjust+0x219/0x454 > [] vma_merge+0x147/0x1f4 > [] do_mmap_pgoff+0x414/0x7c7 > [] _spin_unlock_irq+0x24/0x27 > [] sys_mmap+0xe5/0x110 > [] system_call+0x7e/0x83 H. We have seen this before http://marc.info/?l=linux-kernel=117891943401284=2 I suspect this is due to an interaction with other debug flags you have set. What I see in your .config is CONFIG_DEBUG_RT_MUTEXES=y Try to switch this off if switching off the lockdep flags does not hel. CONFIG_DEBUG_PI_LIST=y Will go off with the above # CONFIG_RT_MUTEX_TESTER is not set Leave off CONFIG_DEBUG_SPINLOCK=y CONFIG_DEBUG_MUTEXES=y Try toggling if the disabling lockdep flags does not work. CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y CONFIG_LOCKDEP=y I suspect the above are the main candidates for trouble. Could you switch those off? # CONFIG_DEBUG_LOCKDEP is not set CONFIG_TRACE_IRQFLAGS=y CONFIG_DEBUG_SPINLOCK_SLEEP=y # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set CONFIG_STACKTRACE=y # CONFIG_DEBUG_KOBJECT is not set Debug kobject may be helpful. CONFIG_DEBUG_BUGVERBOSE=y - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Sun, 20 May 2007, Srihari Vijayaraghavan wrote: With no CONFIG_SLUB_DEBUG, things have slightly improved. No more panic. Good. Serial console is working. Good. But there is another problem: Well this indicates that something destroys the sysfs pointer structure that SLUB is using. Could you reenable CONFIG_SLUB_DEBUG and boot with the option slub_debug on the kernel command line? Also enable the list debug options in kernel hacking. This should show up on the console since the failure is in slub_sysfs_init which is long after the console has been enabled. I wonder why it did not show so far? Freeing unused kernel memory: 308k freed BUG: spinlock bad magic on CPU#1, init/1 lock: 81011ec0a100, .magic: 8101, .owner: none/-1, .owner_cpu: -1 Call Trace: [802fc516] _raw_spin_lock+0x22/0xf6 [8026f684] vma_adjust+0x219/0x454 [8026f684] vma_adjust+0x219/0x454 [8026fe0d] vma_merge+0x147/0x1f4 [80270b55] do_mmap_pgoff+0x414/0x7c7 [8040aa34] _spin_unlock_irq+0x24/0x27 [8020f528] sys_mmap+0xe5/0x110 [80209dde] system_call+0x7e/0x83 H. We have seen this before http://marc.info/?l=linux-kernelm=117891943401284w=2 I suspect this is due to an interaction with other debug flags you have set. What I see in your .config is CONFIG_DEBUG_RT_MUTEXES=y Try to switch this off if switching off the lockdep flags does not hel. CONFIG_DEBUG_PI_LIST=y Will go off with the above # CONFIG_RT_MUTEX_TESTER is not set Leave off CONFIG_DEBUG_SPINLOCK=y CONFIG_DEBUG_MUTEXES=y Try toggling if the disabling lockdep flags does not work. CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y CONFIG_LOCKDEP=y I suspect the above are the main candidates for trouble. Could you switch those off? # CONFIG_DEBUG_LOCKDEP is not set CONFIG_TRACE_IRQFLAGS=y CONFIG_DEBUG_SPINLOCK_SLEEP=y # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set CONFIG_STACKTRACE=y # CONFIG_DEBUG_KOBJECT is not set Debug kobject may be helpful. CONFIG_DEBUG_BUGVERBOSE=y - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Christoph Lameter wrote:
On Sun, 20 May 2007, Srihari Vijayaraghavan wrote:
With no CONFIG_SLUB_DEBUG, things have slightly improved. No more panic.
Good.
Serial console is working. Good. But there is another problem:
Freeing unused kernel memory: 308k freed
BUG: spinlock bad magic on CPU#1, init/1
lock: 81011ec0a100, .magic: 8101, .owner: none/-1, .owner_cpu: -1
Call Trace:
[802fc516] _raw_spin_lock+0x22/0xf6
[8026f684] vma_adjust+0x219/0x454
[8026f684] vma_adjust+0x219/0x454
[8026fe0d] vma_merge+0x147/0x1f4
[80270b55] do_mmap_pgoff+0x414/0x7c7
[8040aa34] _spin_unlock_irq+0x24/0x27
[8020f528] sys_mmap+0xe5/0x110
[80209dde] system_call+0x7e/0x83
H. We have seen this before
http://marc.info/?l=linux-kernelm=117891943401284w=2
Yes, sounded the same to me too: I couldn't reproduce it or see anything
wrong in the code back then. But Srihari's info about CONFIG_DEBUG_SLUB
off has helped a lot: I was then able to reproduce it on my x86_64, and
after a lot of staring at the code, the problem became obvious...
[PATCH] slub: fix size adjustment when SLUB_DEBUG off
SLUB was corrupting the anon_vma's spinlock when not configured for
SLUB_DEBUG: the size adjustment necessary for SLAB_DESTROY_BY_RCU or
constructor was inside the CONFIG_SLUB_DEBUG appropriate for poisoning.
Signed-off-by: Hugh Dickins [EMAIL PROTECTED]
---
mm/slub.c | 18 ++
1 file changed, 14 insertions(+), 4 deletions(-)
--- 2.6.22-rc2/mm/slub.c2007-05-21 13:13:20.0 +0100
+++ linux/mm/slub.c 2007-05-21 18:05:57.0 +0100
@@ -1917,16 +1917,26 @@ static int calculate_sizes(struct kmem_c
*/
s-inuse = size;
+ if ((flags SLAB_DESTROY_BY_RCU) || s-ctor) {
+ /*
+* Relocate free pointer after the object if it is not
+* permitted to overwrite the first word of the object on
+* kmem_cache_free.
+*
+* This is the case if we do RCU or have a constructor.
+*/
+ s-offset = size;
+ size += sizeof(void *);
+ }
+
#ifdef CONFIG_SLUB_DEBUG
- if (((flags (SLAB_DESTROY_BY_RCU | SLAB_POISON)) ||
- s-ctor)) {
+ if (s-flags __OBJECT_POISON) {
/*
* Relocate free pointer after the object if it is not
* permitted to overwrite the first word of the object on
* kmem_cache_free.
*
-* This is the case if we do RCU, have a constructor or
-* destructor or are poisoning the objects.
+* This is the case if we are poisoning the objects.
*/
s-offset = size;
size += sizeof(void *);
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Hugh Dickins wrote:
Yes, sounded the same to me too: I couldn't reproduce it or see anything
wrong in the code back then. But Srihari's info about CONFIG_DEBUG_SLUB
off has helped a lot: I was then able to reproduce it on my x86_64, and
after a lot of staring at the code, the problem became obvious...
Right. The #ifdef CONFIG_SLUB_DEBUG is at the wrong location. The best fix
is to moving the #ifdef otherwise the size is still wrong for the
ctor case.
SLUB Debug: Fix object size calculation
The object size calculation is wrong if !CONFIG_SLUB_DEBUG because
the #ifdef CONFIG_SLUB_DEBUG is now switching off the size adjustments
for DESTROY_BY_RCU and ctor.
Signed-off-by: Christoph Lameter [EMAIL PROTECTED]
---
mm/slub.c |2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Index: slub/mm/slub.c
===
--- slub.orig/mm/slub.c 2007-05-21 11:21:36.0 -0700
+++ slub/mm/slub.c 2007-05-21 11:21:49.0 -0700
@@ -1943,7 +1943,6 @@ static int calculate_sizes(struct kmem_c
*/
s-inuse = size;
-#ifdef CONFIG_SLUB_DEBUG
if (((flags (SLAB_DESTROY_BY_RCU | SLAB_POISON)) ||
s-ctor)) {
/*
@@ -1958,6 +1957,7 @@ static int calculate_sizes(struct kmem_c
size += sizeof(void *);
}
+#ifdef CONFIG_SLUB_DEBUG
if (flags SLAB_STORE_USER)
/*
* Need to store information about allocs and frees after
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Christoph Lameter wrote:
On Mon, 21 May 2007, Hugh Dickins wrote:
Yes, sounded the same to me too: I couldn't reproduce it or see anything
wrong in the code back then. But Srihari's info about CONFIG_DEBUG_SLUB
off has helped a lot: I was then able to reproduce it on my x86_64, and
after a lot of staring at the code, the problem became obvious...
Right. The #ifdef CONFIG_SLUB_DEBUG is at the wrong location. The best fix
is to moving the #ifdef otherwise the size is still wrong for the
ctor case.
? My patch did handle the ctor case.
SLUB Debug: Fix object size calculation
The object size calculation is wrong if !CONFIG_SLUB_DEBUG because
the #ifdef CONFIG_SLUB_DEBUG is now switching off the size adjustments
for DESTROY_BY_RCU and ctor.
Signed-off-by: Christoph Lameter [EMAIL PROTECTED]
Yes, I think that should do it too. The reason behind my repeating
the block was to handle the case where SLAB_POISON is passed to
kmem_cache_create, but CONFIG_SLUB_DEBUG is off. But apparently
that case would hit the BUG_ON(flags ~CREATE_MASK), therefore
your patch is simpler and better. Quite a maze.
Hugh
---
mm/slub.c |2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Index: slub/mm/slub.c
===
--- slub.orig/mm/slub.c 2007-05-21 11:21:36.0 -0700
+++ slub/mm/slub.c2007-05-21 11:21:49.0 -0700
@@ -1943,7 +1943,6 @@ static int calculate_sizes(struct kmem_c
*/
s-inuse = size;
-#ifdef CONFIG_SLUB_DEBUG
if (((flags (SLAB_DESTROY_BY_RCU | SLAB_POISON)) ||
s-ctor)) {
/*
@@ -1958,6 +1957,7 @@ static int calculate_sizes(struct kmem_c
size += sizeof(void *);
}
+#ifdef CONFIG_SLUB_DEBUG
if (flags SLAB_STORE_USER)
/*
* Need to store information about allocs and frees after
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Hugh Dickins wrote: On Mon, 21 May 2007, Christoph Lameter wrote: On Mon, 21 May 2007, Hugh Dickins wrote: Yes, sounded the same to me too: I couldn't reproduce it or see anything wrong in the code back then. But Srihari's info about CONFIG_DEBUG_SLUB off has helped a lot: I was then able to reproduce it on my x86_64, and after a lot of staring at the code, the problem became obvious... Right. The #ifdef CONFIG_SLUB_DEBUG is at the wrong location. The best fix is to moving the #ifdef otherwise the size is still wrong for the ctor case. ? My patch did handle the ctor case. True. I was thinking about just checking the problem case that we had here. SLUB Debug: Fix object size calculation The object size calculation is wrong if !CONFIG_SLUB_DEBUG because the #ifdef CONFIG_SLUB_DEBUG is now switching off the size adjustments for DESTROY_BY_RCU and ctor. Signed-off-by: Christoph Lameter [EMAIL PROTECTED] Yes, I think that should do it too. The reason behind my repeating the block was to handle the case where SLAB_POISON is passed to kmem_cache_create, but CONFIG_SLUB_DEBUG is off. But apparently that case would hit the BUG_ON(flags ~CREATE_MASK), therefore your patch is simpler and better. Quite a maze. Would you ack my patch? I do not want to repeat the block. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, May 21 2007, Christoph Lameter wrote: On Mon, 21 May 2007, Hugh Dickins wrote: On Mon, 21 May 2007, Christoph Lameter wrote: On Mon, 21 May 2007, Hugh Dickins wrote: Yes, sounded the same to me too: I couldn't reproduce it or see anything wrong in the code back then. But Srihari's info about CONFIG_DEBUG_SLUB off has helped a lot: I was then able to reproduce it on my x86_64, and after a lot of staring at the code, the problem became obvious... Right. The #ifdef CONFIG_SLUB_DEBUG is at the wrong location. The best fix is to moving the #ifdef otherwise the size is still wrong for the ctor case. ? My patch did handle the ctor case. True. I was thinking about just checking the problem case that we had here. SLUB Debug: Fix object size calculation The object size calculation is wrong if !CONFIG_SLUB_DEBUG because the #ifdef CONFIG_SLUB_DEBUG is now switching off the size adjustments for DESTROY_BY_RCU and ctor. Signed-off-by: Christoph Lameter [EMAIL PROTECTED] Yes, I think that should do it too. The reason behind my repeating the block was to handle the case where SLAB_POISON is passed to kmem_cache_create, but CONFIG_SLUB_DEBUG is off. But apparently that case would hit the BUG_ON(flags ~CREATE_MASK), therefore your patch is simpler and better. Quite a maze. Would you ack my patch? I do not want to repeat the block. I can test whatever you want tomorrow morning, it was 100% repeatable here. So which one, your patch or Hughs? -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Christoph Lameter wrote: On Mon, 21 May 2007, Hugh Dickins wrote: On Mon, 21 May 2007, Christoph Lameter wrote: On Mon, 21 May 2007, Hugh Dickins wrote: SLUB Debug: Fix object size calculation The object size calculation is wrong if !CONFIG_SLUB_DEBUG because the #ifdef CONFIG_SLUB_DEBUG is now switching off the size adjustments for DESTROY_BY_RCU and ctor. Signed-off-by: Christoph Lameter [EMAIL PROTECTED] Acked-by: Hugh Dickins [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Jens Axboe wrote: I can test whatever you want tomorrow morning, it was 100% repeatable here. So which one, your patch or Hughs? Great, thanks: Christoph's please - I'm sure he'll agree! Hugh - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, May 21 2007, Hugh Dickins wrote: On Mon, 21 May 2007, Jens Axboe wrote: I can test whatever you want tomorrow morning, it was 100% repeatable here. So which one, your patch or Hughs? Great, thanks: Christoph's please - I'm sure he'll agree! Alright, will do! -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Jens Axboe wrote: I can test whatever you want tomorrow morning, it was 100% repeatable here. So which one, your patch or Hughs? This was an effect of me suggesting to switch off CONFIG_SLUB_DEBUG. If you did not run without CONFIG_SLUB_DEBUG then you were not affected by the problem. And the earlier case that you reported was a version of SLUB that did not have the capability to switch off SLUB_DEBUG. Is there any way we can boot with slub_debug on the kernel command line and get early printk output? Looks like an early failure that may have to do with other kernel debug options being on (lockdep or so?) - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, May 21 2007, Christoph Lameter wrote: On Mon, 21 May 2007, Jens Axboe wrote: I can test whatever you want tomorrow morning, it was 100% repeatable here. So which one, your patch or Hughs? This was an effect of me suggesting to switch off CONFIG_SLUB_DEBUG. If you did not run without CONFIG_SLUB_DEBUG then you were not affected by the problem. And the earlier case that you reported was a version of SLUB that did not have the capability to switch off SLUB_DEBUG. Is there any way we can boot with slub_debug on the kernel command line and get early printk output? Looks like an early failure that may have to do with other kernel debug options being on (lockdep or so?) Sure, I will try that as well. For reference, the last kernel tested on that box was HEAD 129a84de2347002f09721cda3155ccfd19fade40. -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Christoph Lameter wrote: On Mon, 21 May 2007, Jens Axboe wrote: I can test whatever you want tomorrow morning, it was 100% repeatable here. So which one, your patch or Hughs? This was an effect of me suggesting to switch off CONFIG_SLUB_DEBUG. If you did not run without CONFIG_SLUB_DEBUG then you were not affected by the problem. And the earlier case that you reported was a version of SLUB that did not have the capability to switch off SLUB_DEBUG. Are you sure? If CONFIG_EMBEDDED=y then -rc1 and -rc2 give you the choice to turn it off. Is there any way we can boot with slub_debug on the kernel command line and get early printk output? Looks like an early failure that may have to do with other kernel debug options being on (lockdep or so?) - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Hugh Dickins wrote: the problem. And the earlier case that you reported was a version of SLUB that did not have the capability to switch off SLUB_DEBUG. Are you sure? If CONFIG_EMBEDDED=y then -rc1 and -rc2 give you the choice to turn it off. Right. That went in indeed shortly before the rc1 release. So if both had disabled SLUB_DEBUG then this could be the reason for the failure. However, CONFIG_SLUB_DEBUG was set in the .config that was included when this bug was reported. So I think we still have not found the real problem yet. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
Ok booting SLUB without slub_debug and having CONFIG_DEBUG_LOCK_ALLOC CONFIG_LOCKDEP will trigger the problem. So I guess the issue is that lockdep does a slab alloc while we get the slab lock during slab alloc? - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Mon, 21 May 2007, Christoph Lameter wrote: So I guess the issue is that lockdep does a slab alloc while we get the slab lock during slab alloc? Lockdep is not available on IA64 where I would be able to figure it out using a simulator. x86_64 early printk support seems to be broken? No output with earlyprintk=serial. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Satyam Sharma <[EMAIL PROTECTED]> wrote:
> On 5/20/07, Srihari Vijayaraghavan <[EMAIL PROTECTED]>
> wrote:
> > --- Christoph Lameter <[EMAIL PROTECTED]> wrote:
> > > On Sun, 20 May 2007, Srihari Vijayaraghavan wrote:
> > >
> > > > Code: 0f ob eb fe 48 8b 1b 48 8b 0x 0f 18 08 48 81 fb 60 cb 51 80
> > > > RIP [...] slab_sysfs_init+0x49/0x98
> > > > RSP [...]
> > > > kernel panic - not syncing: Attempted to kill init!
> > >
> > > sysfs? If you build without CONFIG_SLUB_DEBUG then SLUB will not use
> > > sysfs. You can switch that off in the embedded section.
> >
> > Fair enough. Would you the section name it appears in 'menuconfig'?
> >
> > I looked under 'Kernel hacking' section (where the slab debugging appears
> if
> > slab is turned on as 'Debug slab memory allocation') & in the 'General
> setup'
> > section also. I unable to figure out how to disable slub debugging :-(.
>
> SLUB_DEBUG is both "bool if EMBEDDED" and "depends on SLUB", which
> means just getting it to be visible in "menuconfig" is quite a challenge :-)
>
> You first need to go into the choice where you pick the allocators ("Choose
> allocator --->" which is the last option in General Setup), and then pick
> SLUB as your allocator.
>
> Then, from the General Setup again, turn on the "Configure standard
> kernel features (for small systems) --->" just above the previous option
> and then go into that menu. Now the option to disable SLUB_DEBUG
> (which is y by default) becomes visible, so just deselect it to disable it.
Thanks for that. I've just figured that out a few minutes ago :-). Was a
tricky one indeed. Then I captured and posted the current kernel bug report
while having slub just then.
Thanks
___
How would you spend $50,000 to create a more sustainable environment in
Australia? Go to Yahoo!7 Answers and share your idea.
http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Srihari Vijayaraghavan <[EMAIL PROTECTED]> wrote: > --- Christoph Lameter <[EMAIL PROTECTED]> wrote: > > On Sun, 20 May 2007, Srihari Vijayaraghavan wrote: > > > > > Code: 0f ob eb fe 48 8b 1b 48 8b 0x 0f 18 08 48 81 fb 60 cb 51 80 > > > RIP [...] slab_sysfs_init+0x49/0x98 > > > RSP [...] > > > kernel panic - not syncing: Attempted to kill init! > > > > sysfs? If you build without CONFIG_SLUB_DEBUG then SLUB will not use > > sysfs. You can switch that off in the embedded section. > > Fair enough. Would you the section name it appears in 'menuconfig'? [Sorry for replying to my own email. I've made some progress since then, so here goes ...] Sorry for the noise. I've discovered it under "General setup - Configure standard kernel features (for small systems)". With no CONFIG_SLUB_DEBUG, things have slightly improved. No more panic. Good. Serial console is working. Good. But there is another problem: ... Freeing unused kernel memory: 308k freed BUG: spinlock bad magic on CPU#1, init/1 lock: 81011ec0a100, .magic: 8101, .owner: /-1, .owner_cpu: -1 Call Trace: [] _raw_spin_lock+0x22/0xf6 [] vma_adjust+0x219/0x454 [] vma_adjust+0x219/0x454 [] vma_merge+0x147/0x1f4 [] do_mmap_pgoff+0x414/0x7c7 [] _spin_unlock_irq+0x24/0x27 [] sys_mmap+0xe5/0x110 [] system_call+0x7e/0x83 ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver ... PM: Adding info for No Bus:vcsa1 BUG: spinlock lockup on CPU#0, sh/385, 81011ca8ffc0 Call Trace: [] _raw_spin_lock+0xcf/0xf6 [] anon_vma_unlink+0x1c/0x68 [] anon_vma_unlink+0x1c/0x68 [] free_pgtables+0x69/0xc3 [] exit_mmap+0x91/0xeb [] mmput+0x45/0xb8 [] flush_old_exec+0x65f/0x941 [] vfs_read+0x13f/0x153 [] load_elf_binary+0x0/0x197f [] load_elf_binary+0x460/0x197f [] __alloc_pages+0x72/0x2d4 [] load_elf_binary+0x0/0x197f [] search_binary_handler+0xc4/0x25f [] do_execve+0x188/0x231 [] sys_execve+0x36/0x8a [] stub_execve+0x67/0xb0 It just simply hangs there. With slab all is well, of course. (If you want me to activate all kernel debugging options, do advise, I'm happy to do that. Nothing changed in ref to my .config from the past email save the CONFIG_SLUB_DEBUG is gone now.) The dmesg is attached for your reference. Thanks ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html Linux version 2.6.22-rc2-ahci-slub ([EMAIL PROTECTED]) (gcc version 4.1.2 20070502 (Red Hat 4.1.2-12)) #6 SMP Sun May 20 23:05:08 EST 2007 Command line: ro root=LABEL=/1234 console=ttyS0,115200 console=tty0 BIOS-provided physical RAM map: BIOS-e820: - 0009f000 (usable) BIOS-e820: 0009f000 - 000a (reserved) BIOS-e820: 000f - 0010 (reserved) BIOS-e820: 0010 - dfee (usable) BIOS-e820: dfee - dfee3000 (ACPI NVS) BIOS-e820: dfee3000 - dfef (ACPI data) BIOS-e820: dfef - dff0 (reserved) BIOS-e820: e000 - f000 (reserved) BIOS-e820: fec0 - 0001 (reserved) BIOS-e820: 0001 - 00012000 (usable) end_pfn_map = 1179648 DMI 2.4 present. ACPI: RSDP 000F8110, 0024 (r2 ATI ) ACPI: XSDT DFEE30C0, 0044 (r1 ATIASUSACPI 42302E31 AWRD0) ACPI: FACP DFEE83C0, 00F4 (r3 ATIASUSACPI 42302E31 AWRD0) ACPI: DSDT DFEE3240, 511E (r1 ATIASUSACPI 1000 MSFT 300) ACPI: FACS DFEE, 0040 ACPI: SSDT DFEE85C0, 02CC (r1 PTLTD POWERNOW1 LTP1) ACPI: MCFG DFEE8980, 003C (r1 ATIASUSACPI 42302E31 AWRD0) ACPI: APIC DFEE8500, 0068 (r1 ATIASUSACPI 42302E31 AWRD0) Scanning NUMA topology in Northbridge 24 No NUMA configuration found Faking a node at -00012000 Bootmem setup node 0 -00012000 Zone PFN ranges: DMA 0 -> 4096 DMA324096 -> 1048576 Normal1048576 -> 1179648 early_node_map[3] active PFN ranges 0:0 -> 159 0: 256 -> 917216 0: 1048576 -> 1179648 ATI board detected. Disabling timer routing over 8254. ACPI: PM-Timer IO Port: 0x4008 ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Processor #0 (Bootup-CPU) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) Processor #1 ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) ACPI: IOAPIC (id[0x02] address[0xfec0] gsi_base[0]) IOAPIC[0]: apic_id 2, address 0xfec0, GSI 0-23 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) Setting APIC routing to flat Using ACPI (MADT) for SMP configuration information Allocating PCI resources starting at f100 (gap:
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On 5/20/07, Srihari Vijayaraghavan <[EMAIL PROTECTED]> wrote:
--- Christoph Lameter <[EMAIL PROTECTED]> wrote:
> On Sun, 20 May 2007, Srihari Vijayaraghavan wrote:
>
> > Code: 0f ob eb fe 48 8b 1b 48 8b 0x 0f 18 08 48 81 fb 60 cb 51 80
> > RIP [...] slab_sysfs_init+0x49/0x98
> > RSP [...]
> > kernel panic - not syncing: Attempted to kill init!
>
> sysfs? If you build without CONFIG_SLUB_DEBUG then SLUB will not use
> sysfs. You can switch that off in the embedded section.
Fair enough. Would you the section name it appears in 'menuconfig'?
I looked under 'Kernel hacking' section (where the slab debugging appears if
slab is turned on as 'Debug slab memory allocation') & in the 'General setup'
section also. I unable to figure out how to disable slub debugging :-(.
SLUB_DEBUG is both "bool if EMBEDDED" and "depends on SLUB", which
means just getting it to be visible in "menuconfig" is quite a challenge :-)
You first need to go into the choice where you pick the allocators ("Choose
allocator --->" which is the last option in General Setup), and then pick
SLUB as your allocator.
Then, from the General Setup again, turn on the "Configure standard
kernel features (for small systems) --->" just above the previous option
and then go into that menu. Now the option to disable SLUB_DEBUG
(which is y by default) becomes visible, so just deselect it to disable it.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Christoph Lameter <[EMAIL PROTECTED]> wrote: > On Sun, 20 May 2007, Srihari Vijayaraghavan wrote: > > > Code: 0f ob eb fe 48 8b 1b 48 8b 0x 0f 18 08 48 81 fb 60 cb 51 80 > > RIP [...] slab_sysfs_init+0x49/0x98 > > RSP [...] > > kernel panic - not syncing: Attempted to kill init! > > sysfs? If you build without CONFIG_SLUB_DEBUG then SLUB will not use > sysfs. You can switch that off in the embedded section. Fair enough. Would you the section name it appears in 'menuconfig'? I looked under 'Kernel hacking' section (where the slab debugging appears if slab is turned on as 'Debug slab memory allocation') & in the 'General setup' section also. I unable to figure out how to disable slub debugging :-(. Pls give me a pointer to it. Thanks (Won't it be right to disable it by default when a user selects slub such that a user needs to manually turn on the slub debugging in menuconfig?) ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Sun, 20 May 2007, Srihari Vijayaraghavan wrote: > Code: 0f ob eb fe 48 8b 1b 48 8b 0x 0f 18 08 48 81 fb 60 cb 51 80 > RIP [...] slab_sysfs_init+0x49/0x98 > RSP [...] > kernel panic - not syncing: Attempted to kill init! sysfs? If you build without CONFIG_SLUB_DEBUG then SLUB will not use sysfs. You can switch that off in the embedded section. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On Sun, 20 May 2007, Srihari Vijayaraghavan wrote: Code: 0f ob eb fe 48 8b 1b 48 8b 0x 0f 18 08 48 81 fb 60 cb 51 80 RIP [...] slab_sysfs_init+0x49/0x98 RSP [...] kernel panic - not syncing: Attempted to kill init! sysfs? If you build without CONFIG_SLUB_DEBUG then SLUB will not use sysfs. You can switch that off in the embedded section. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
--- Christoph Lameter [EMAIL PROTECTED] wrote: On Sun, 20 May 2007, Srihari Vijayaraghavan wrote: Code: 0f ob eb fe 48 8b 1b 48 8b 0x 0f 18 08 48 81 fb 60 cb 51 80 RIP [...] slab_sysfs_init+0x49/0x98 RSP [...] kernel panic - not syncing: Attempted to kill init! sysfs? If you build without CONFIG_SLUB_DEBUG then SLUB will not use sysfs. You can switch that off in the embedded section. Fair enough. Would you the section name it appears in 'menuconfig'? I looked under 'Kernel hacking' section (where the slab debugging appears if slab is turned on as 'Debug slab memory allocation') in the 'General setup' section also. I unable to figure out how to disable slub debugging :-(. Pls give me a pointer to it. Thanks (Won't it be right to disable it by default when a user selects slub such that a user needs to manually turn on the slub debugging in menuconfig?) ___ How would you spend $50,000 to create a more sustainable environment in Australia? Go to Yahoo!7 Answers and share your idea. http://advision.webevents.yahoo.com/aunz/lifestyle/answers/y7ans-babp_reg.html - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PROBLEM] 2.6.22-rc2 panics on x86-64 with slub
On 5/20/07, Srihari Vijayaraghavan [EMAIL PROTECTED] wrote: --- Christoph Lameter [EMAIL PROTECTED] wrote: On Sun, 20 May 2007, Srihari Vijayaraghavan wrote: Code: 0f ob eb fe 48 8b 1b 48 8b 0x 0f 18 08 48 81 fb 60 cb 51 80 RIP [...] slab_sysfs_init+0x49/0x98 RSP [...] kernel panic - not syncing: Attempted to kill init! sysfs? If you build without CONFIG_SLUB_DEBUG then SLUB will not use sysfs. You can switch that off in the embedded section. Fair enough. Would you the section name it appears in 'menuconfig'? I looked under 'Kernel hacking' section (where the slab debugging appears if slab is turned on as 'Debug slab memory allocation') in the 'General setup' section also. I unable to figure out how to disable slub debugging :-(. SLUB_DEBUG is both bool if EMBEDDED and depends on SLUB, which means just getting it to be visible in menuconfig is quite a challenge :-) You first need to go into the choice where you pick the allocators (Choose allocator --- which is the last option in General Setup), and then pick SLUB as your allocator. Then, from the General Setup again, turn on the Configure standard kernel features (for small systems) --- just above the previous option and then go into that menu. Now the option to disable SLUB_DEBUG (which is y by default) becomes visible, so just deselect it to disable it. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
