Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
How is this going? regards, 2013/10/11 Fengguang Wu : >> I think something like the below may address the issue - I've only build >> tested this so far. >> >> We have another case where DRM does the wrong thing here too - a similar >> thing goes on with connectors as well. I've not yet looked into this, >> but I'll take a look later today. >> >> Fengguang - could you give this some runs through your marvellous test >> system and see whether it fixes the problem you're seeing please? > > Russell, I applied the patch (as commit 17e57131d4c2) on top of > v3.12-rc4 and here are the test results for the kernels with > > CONFIG_DRM=y > CONFIG_DEBUG_KOBJECT_RELEASE=y > > As you may see, it's pure improvements. :) > > Tested-by: Fengguang Wu > > > i386-randconfig-c3-1009 > > +---+---+--+ > | | v3.12-rc4 | > 17e57131d4c2 | > +---+---+--+ > | has_kernel_error_warning | 63| > | > | BUG:kernel_early_hang_without_any_printk_output | 7 | > | > | WARNING:CPU:PID:at_fs/sysfs/dir.c:sysfs_add_one() | 56| > | > | WARNING:CPU:PID:at_lib/kobject.c:kobject_add_internal() | 56| > | > | WARNING:CPU:PID:at_drivers/base/driver.c:driver_unregister() | 56| > | > | WARNING:CPU:PID:at_lib/debugobjects.c:debug_print_object()| 56| > | > | INFO:trying_to_register_non-static_key| 56| > | > | BUG:unable_to_handle_kernel_NULL_pointer_dereference_at(null) | 56| > | > | Oops:PREEMPT_SMP | 56| > | > | Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 56| > | > | BUG:kernel_boot_oops | 56| > | > | good_boots| 0 | > 9| > +---+---+--+ > > x86_64-randconfig-j5-1009 > > ++---+--+ > || v3.12-rc4 | 17e57131d4c2 | > ++---+--+ > | good_boots | 30| 9| > ++---+--+ > > x86_64-randconfig-j3-1009 > > +---+---+--+ > | | v3.12-rc4 | > 17e57131d4c2 | > +---+---+--+ > | has_kernel_error_warning | 60| > | > | WARNING:CPU:PID:at_fs/sysfs/dir.c:sysfs_add_one() | 60| > | > | WARNING:CPU:PID:at_lib/kobject.c:kobject_add_internal() | 60| > | > | WARNING:CPU:PID:at_drivers/base/driver.c:driver_unregister() | 60| > | > | WARNING:CPU:PID:at_lib/debugobjects.c:debug_print_object()| 60| > | > | WARNING:CPU:PID:at_arch/x86/kernel/irq_64.c:handle_irq() | 3 | > | > | BUG:unable_to_handle_kernel_NULL_pointer_dereference_at | 59| > | > | Oops:DEBUG_PAGEALLOC | 28| > | > | WARNING:CPU:PID:at_lib/list_debug.c:__list_del_entry()| 28| > | > | BUG:scheduling_while_atomic | 30| > | > | INFO:lockdep_is_turned_off| 30| > | > | BUG:unable_to_handle_kernel_paging_request_at | 55| > | > | kernel_BUG_at_kernel/smpboot.c| 13| > | > | invalid_opcode:DEBUG_PAGEALLOC| 13| > | > | general_protection_fault:DEBUG_PAGEALLOC | 2 | > | > | Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 4 | > | > | BUG:kernel_boot_oops | 60| > | > | WARNING:CPU:PID:at_kernel/workqueue.c:work_fixup_activate() | 24| > | > | WARNING:CPU:PID:at_kernel/workqueue.c:__queue_work() | 10| > | > | BUG:unable_to_handle_kernel_NULL_pointer_dereference_at(null) | 2 | > | > | BUG:unable_to_handle_kernel | 1 | > | > |
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
How is this going? regards, 2013/10/11 Fengguang Wu fengguang...@intel.com: I think something like the below may address the issue - I've only build tested this so far. We have another case where DRM does the wrong thing here too - a similar thing goes on with connectors as well. I've not yet looked into this, but I'll take a look later today. Fengguang - could you give this some runs through your marvellous test system and see whether it fixes the problem you're seeing please? Russell, I applied the patch (as commit 17e57131d4c2) on top of v3.12-rc4 and here are the test results for the kernels with CONFIG_DRM=y CONFIG_DEBUG_KOBJECT_RELEASE=y As you may see, it's pure improvements. :) Tested-by: Fengguang Wu fengguang...@intel.com i386-randconfig-c3-1009 +---+---+--+ | | v3.12-rc4 | 17e57131d4c2 | +---+---+--+ | has_kernel_error_warning | 63| | | BUG:kernel_early_hang_without_any_printk_output | 7 | | | WARNING:CPU:PID:at_fs/sysfs/dir.c:sysfs_add_one() | 56| | | WARNING:CPU:PID:at_lib/kobject.c:kobject_add_internal() | 56| | | WARNING:CPU:PID:at_drivers/base/driver.c:driver_unregister() | 56| | | WARNING:CPU:PID:at_lib/debugobjects.c:debug_print_object()| 56| | | INFO:trying_to_register_non-static_key| 56| | | BUG:unable_to_handle_kernel_NULL_pointer_dereference_at(null) | 56| | | Oops:PREEMPT_SMP | 56| | | Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 56| | | BUG:kernel_boot_oops | 56| | | good_boots| 0 | 9| +---+---+--+ x86_64-randconfig-j5-1009 ++---+--+ || v3.12-rc4 | 17e57131d4c2 | ++---+--+ | good_boots | 30| 9| ++---+--+ x86_64-randconfig-j3-1009 +---+---+--+ | | v3.12-rc4 | 17e57131d4c2 | +---+---+--+ | has_kernel_error_warning | 60| | | WARNING:CPU:PID:at_fs/sysfs/dir.c:sysfs_add_one() | 60| | | WARNING:CPU:PID:at_lib/kobject.c:kobject_add_internal() | 60| | | WARNING:CPU:PID:at_drivers/base/driver.c:driver_unregister() | 60| | | WARNING:CPU:PID:at_lib/debugobjects.c:debug_print_object()| 60| | | WARNING:CPU:PID:at_arch/x86/kernel/irq_64.c:handle_irq() | 3 | | | BUG:unable_to_handle_kernel_NULL_pointer_dereference_at | 59| | | Oops:DEBUG_PAGEALLOC | 28| | | WARNING:CPU:PID:at_lib/list_debug.c:__list_del_entry()| 28| | | BUG:scheduling_while_atomic | 30| | | INFO:lockdep_is_turned_off| 30| | | BUG:unable_to_handle_kernel_paging_request_at | 55| | | kernel_BUG_at_kernel/smpboot.c| 13| | | invalid_opcode:DEBUG_PAGEALLOC| 13| | | general_protection_fault:DEBUG_PAGEALLOC | 2 | | | Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 4 | | | BUG:kernel_boot_oops | 60| | | WARNING:CPU:PID:at_kernel/workqueue.c:work_fixup_activate() | 24| | | WARNING:CPU:PID:at_kernel/workqueue.c:__queue_work() | 10| | | BUG:unable_to_handle_kernel_NULL_pointer_dereference_at(null) | 2 | | | BUG:unable_to_handle_kernel | 1 | | | BUG:unable_to_handle_kernel_NULL_point| 1 |
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
> I think something like the below may address the issue - I've only build > tested this so far. > > We have another case where DRM does the wrong thing here too - a similar > thing goes on with connectors as well. I've not yet looked into this, > but I'll take a look later today. > > Fengguang - could you give this some runs through your marvellous test > system and see whether it fixes the problem you're seeing please? Russell, I applied the patch (as commit 17e57131d4c2) on top of v3.12-rc4 and here are the test results for the kernels with CONFIG_DRM=y CONFIG_DEBUG_KOBJECT_RELEASE=y As you may see, it's pure improvements. :) Tested-by: Fengguang Wu i386-randconfig-c3-1009 +---+---+--+ | | v3.12-rc4 | 17e57131d4c2 | +---+---+--+ | has_kernel_error_warning | 63| | | BUG:kernel_early_hang_without_any_printk_output | 7 | | | WARNING:CPU:PID:at_fs/sysfs/dir.c:sysfs_add_one() | 56| | | WARNING:CPU:PID:at_lib/kobject.c:kobject_add_internal() | 56| | | WARNING:CPU:PID:at_drivers/base/driver.c:driver_unregister() | 56| | | WARNING:CPU:PID:at_lib/debugobjects.c:debug_print_object()| 56| | | INFO:trying_to_register_non-static_key| 56| | | BUG:unable_to_handle_kernel_NULL_pointer_dereference_at(null) | 56| | | Oops:PREEMPT_SMP | 56| | | Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 56| | | BUG:kernel_boot_oops | 56| | | good_boots| 0 | 9 | +---+---+--+ x86_64-randconfig-j5-1009 ++---+--+ || v3.12-rc4 | 17e57131d4c2 | ++---+--+ | good_boots | 30| 9| ++---+--+ x86_64-randconfig-j3-1009 +---+---+--+ | | v3.12-rc4 | 17e57131d4c2 | +---+---+--+ | has_kernel_error_warning | 60| | | WARNING:CPU:PID:at_fs/sysfs/dir.c:sysfs_add_one() | 60| | | WARNING:CPU:PID:at_lib/kobject.c:kobject_add_internal() | 60| | | WARNING:CPU:PID:at_drivers/base/driver.c:driver_unregister() | 60| | | WARNING:CPU:PID:at_lib/debugobjects.c:debug_print_object()| 60| | | WARNING:CPU:PID:at_arch/x86/kernel/irq_64.c:handle_irq() | 3 | | | BUG:unable_to_handle_kernel_NULL_pointer_dereference_at | 59| | | Oops:DEBUG_PAGEALLOC | 28| | | WARNING:CPU:PID:at_lib/list_debug.c:__list_del_entry()| 28| | | BUG:scheduling_while_atomic | 30| | | INFO:lockdep_is_turned_off| 30| | | BUG:unable_to_handle_kernel_paging_request_at | 55| | | kernel_BUG_at_kernel/smpboot.c| 13| | | invalid_opcode:DEBUG_PAGEALLOC| 13| | | general_protection_fault:DEBUG_PAGEALLOC | 2 | | | Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 4 | | | BUG:kernel_boot_oops | 60| | | WARNING:CPU:PID:at_kernel/workqueue.c:work_fixup_activate() | 24| | | WARNING:CPU:PID:at_kernel/workqueue.c:__queue_work() | 10| | | BUG:unable_to_handle_kernel_NULL_pointer_dereference_at(null) | 2 | | | BUG:unable_to_handle_kernel | 1 | | | BUG:unable_to_handle_kernel_NULL_point| 1 | | | BUG:unable_to_handle_kernel_NULL_pointer_dereference | 4 | | | BUG:unable_to_handle_kernel_NULL_pointer_derefe | 1 |
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
> Damn gmail screwed up my reply-all,
>
> anyhoo I get the feeling I'd rather do this like fbdev does and stop using
> an embedded kdev for this if I can. The lifetime of the minor and the sysfs
> objects aren't necessarily that tied together esp with hot unplug of
> USB devices.
>
> e.g. when a USB device goes away I don't want the sysfs files to
> remain hanging around,
> or the userspace device nodes, however I can't remove all the internal
> driver state until userspace finally closes all open device
> files,
>
> I think this is a bit of a bad misdesign of the whole device api that
> sysfs exposure and device exposure is tied
> to objects you are encourgaged to embed.
>
> I think maybe I'll just copy fbdev and do a separate device_create
> with a nonembedded dev.
>
Okay it compiles but not boot tested yet.,attached as well just in
case gmail eats it.
>From be06235b2d2a26c0e52c099e70922b71f1245b9e Mon Sep 17 00:00:00 2001
From: Dave Airlie
Date: Fri, 11 Oct 2013 14:07:25 +1000
Subject: [PATCH] drm/sysfs: sort out minor and connector device object
lifetimes.
So drm was abusing device lifetimes, by having embedded device structures
in the minor and connector it meant that the lifetime of the internal drm
objects (drm_minor and drm_connector) were tied to the lifetime of the device
files in sysfs, so if something kept those files opened the current code
would kfree the objects and things would go downhill from there.
Now in reality there is no need for these lifetimes to be so intertwined,
especailly with hotplugging of devices where we wish to remove the sysfs
and userspace facing pieces before we can unwind the internal objects due
to open userspace files or mmaps, so split the objects out so the struct
device is no longer embedded and do what fbdev does and just allocate
and remove the sysfs inodes separately.
Signed-off-by: Dave Airlie
---
drivers/gpu/drm/drm_sysfs.c | 94 ++-
drivers/gpu/drm/i915/i915_irq.c | 8 ++--
drivers/gpu/drm/i915/i915_sysfs.c | 28 ++--
include/drm/drmP.h| 2 +-
include/drm/drm_crtc.h| 2 +-
5 files changed, 54 insertions(+), 80 deletions(-)
diff --git a/drivers/gpu/drm/drm_sysfs.c b/drivers/gpu/drm/drm_sysfs.c
index 2290b3b..dae42c7 100644
--- a/drivers/gpu/drm/drm_sysfs.c
+++ b/drivers/gpu/drm/drm_sysfs.c
@@ -22,8 +22,8 @@
#include
#include
-#define to_drm_minor(d) container_of(d, struct drm_minor, kdev)
-#define to_drm_connector(d) container_of(d, struct drm_connector, kdev)
+#define to_drm_minor(d) dev_get_drvdata(d)
+#define to_drm_connector(d) dev_get_drvdata(d)
static struct device_type drm_sysfs_device_minor = {
.name = "drm_minor"
@@ -162,20 +162,6 @@ void drm_sysfs_destroy(void)
drm_class = NULL;
}
-/**
- * drm_sysfs_device_release - do nothing
- * @dev: Linux device
- *
- * Normally, this would free the DRM device associated with @dev, along
- * with cleaning up any other stuff. But we do that in the DRM core, so
- * this function can just return and hope that the core does its job.
- */
-static void drm_sysfs_device_release(struct device *dev)
-{
-memset(dev, 0, sizeof(struct device));
-return;
-}
-
/*
* Connector properties
*/
@@ -394,29 +380,26 @@ int drm_sysfs_connector_add(struct drm_connector
*connector)
int i;
int ret;
-/* We shouldn't get called more than once for the same connector */
-BUG_ON(device_is_registered(>kdev));
-
-connector->kdev.parent = >primary->kdev;
-connector->kdev.class = drm_class;
-connector->kdev.release = drm_sysfs_device_release;
+if (connector->kdev)
+return 0;
+/* We shouldn't get called more than once for the same connector */
+connector->kdev = device_create(drm_class, dev->primary->kdev,
+0, connector, "card%d-%s",
+dev->primary->index, drm_get_connector_name(connector));
DRM_DEBUG("adding \"%s\" to sysfs\n",
drm_get_connector_name(connector));
-dev_set_name(>kdev, "card%d-%s",
- dev->primary->index, drm_get_connector_name(connector));
-ret = device_register(>kdev);
-
-if (ret) {
-DRM_ERROR("failed to register connector device: %d\n", ret);
+if (IS_ERR(connector->kdev)) {
+DRM_ERROR("failed to register connector device: %ld\n",
PTR_ERR(connector->kdev));
+ret = PTR_ERR(connector->kdev);
goto out;
}
/* Standard attributes */
for (attr_cnt = 0; attr_cnt < ARRAY_SIZE(connector_attrs); attr_cnt++) {
-ret = device_create_file(>kdev, _attrs[attr_cnt]);
+ret = device_create_file(connector->kdev, _attrs[attr_cnt]);
if (ret)
goto err_out_files;
}
@@ -433,7 +416,7 @@ int drm_sysfs_connector_add(struct drm_connector *connector)
case DRM_MODE_CONNECTOR_Component:
case DRM_MODE_CONNECTOR_TV:
for (opt_cnt = 0; opt_cnt <
ARRAY_SIZE(connector_attrs_opt1);
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Thu, Oct 10, 2013 at 8:53 PM, Russell King - ARM Linux
wrote:
> On Thu, Oct 10, 2013 at 10:19:20AM +0100, Russell King - ARM Linux wrote:
>> On Thu, Oct 10, 2013 at 03:23:45AM +0100, Dave Airlie wrote:
>> >
>> > > I think David Arlie also needs a quiet talking to about how to use the
>> > > device model:
>> > >
>> > > int drm_sysfs_device_add(struct drm_minor *minor)
>> > > {
>> > > minor->kdev.release = drm_sysfs_device_release;
>> > > ...
>> > > err = device_register(>kdev);
>> > > }
>> > >
>> > > static void drm_sysfs_device_release(struct device *dev)
>> > > {
>> > > memset(dev, 0, sizeof(struct device));
>> > > return;
>> > > }
>> > >
>> > > Since when has that been acceptable in a release function?
>> >
>> > Well the commit that added it had a reason that seems to cover some other
>> > device model abuses, so maybe someone who actually understands the device
>> > model (all 2 people) can review usage.
>>
>> Please - there's more than two people who understand this stuff.
>>
>> You appear to manage to understand the refcounting principle for things
>> like the DRM framebuffers, DRM buffer objects etc, and the driver model
>> (and kobjects in general) are no different.
>>
>> I've just been reading the code around here little more, and hit the usb
>> implementation. I don't see how USB drivers get cleaned up when they're
>> disconnected from the USB bus. I see drm_unplug_dev() which gets called
>> on a USB disconnection (from udl/udl_drv.c) which effectively makes the
>> device unavailable, but I don't see anything which frees anything. I see
>> nothing calling drm_put_dev() here. How does the drm_device in this case
>> get freed?
>
> Don't worry about the above - I've worked out how USB works in that regard.
> However, I can't say that I like the feel of the code in drm_unplug_dev()
> and drm_put_dev() - if these two can run simultaneously in two threads of
> execution, there's the chance that things will go awry. I don't think
> that can happen due to the way the unplugged-ness is dealt with, but it
> doesn't "feel" safe.
>
> I think something like the below may address the issue - I've only build
> tested this so far.
>
> We have another case where DRM does the wrong thing here too - a similar
> thing goes on with connectors as well. I've not yet looked into this,
> but I'll take a look later today.
>
Damn gmail screwed up my reply-all,
anyhoo I get the feeling I'd rather do this like fbdev does and stop using
an embedded kdev for this if I can. The lifetime of the minor and the sysfs
objects aren't necessarily that tied together esp with hot unplug of
USB devices.
e.g. when a USB device goes away I don't want the sysfs files to
remain hanging around,
or the userspace device nodes, however I can't remove all the internal
driver state until userspace finally closes all open device
files,
I think this is a bit of a bad misdesign of the whole device api that
sysfs exposure and device exposure is tied
to objects you are encourgaged to embed.
I think maybe I'll just copy fbdev and do a separate device_create
with a nonembedded dev.
Dave.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Thu, Oct 10, 2013 at 10:19:20AM +0100, Russell King - ARM Linux wrote:
> On Thu, Oct 10, 2013 at 03:23:45AM +0100, Dave Airlie wrote:
> >
> > > I think David Arlie also needs a quiet talking to about how to use the
> > > device model:
> > >
> > > int drm_sysfs_device_add(struct drm_minor *minor)
> > > {
> > > minor->kdev.release = drm_sysfs_device_release;
> > > ...
> > > err = device_register(>kdev);
> > > }
> > >
> > > static void drm_sysfs_device_release(struct device *dev)
> > > {
> > > memset(dev, 0, sizeof(struct device));
> > > return;
> > > }
> > >
> > > Since when has that been acceptable in a release function?
> >
> > Well the commit that added it had a reason that seems to cover some other
> > device model abuses, so maybe someone who actually understands the device
> > model (all 2 people) can review usage.
>
> Please - there's more than two people who understand this stuff.
>
> You appear to manage to understand the refcounting principle for things
> like the DRM framebuffers, DRM buffer objects etc, and the driver model
> (and kobjects in general) are no different.
>
> I've just been reading the code around here little more, and hit the usb
> implementation. I don't see how USB drivers get cleaned up when they're
> disconnected from the USB bus. I see drm_unplug_dev() which gets called
> on a USB disconnection (from udl/udl_drv.c) which effectively makes the
> device unavailable, but I don't see anything which frees anything. I see
> nothing calling drm_put_dev() here. How does the drm_device in this case
> get freed?
Don't worry about the above - I've worked out how USB works in that regard.
However, I can't say that I like the feel of the code in drm_unplug_dev()
and drm_put_dev() - if these two can run simultaneously in two threads of
execution, there's the chance that things will go awry. I don't think
that can happen due to the way the unplugged-ness is dealt with, but it
doesn't "feel" safe.
I think something like the below may address the issue - I've only build
tested this so far.
We have another case where DRM does the wrong thing here too - a similar
thing goes on with connectors as well. I've not yet looked into this,
but I'll take a look later today.
Fengguang - could you give this some runs through your marvellous test
system and see whether it fixes the problem you're seeing please?
David - maybe you can find some time to look at the Armada driver I
re-posted last weekend?
drivers/gpu/drm/drm_stub.c |8 +---
drivers/gpu/drm/drm_sysfs.c | 42 +-
include/drm/drmP.h |1 +
3 files changed, 39 insertions(+), 12 deletions(-)
diff --git a/drivers/gpu/drm/drm_stub.c b/drivers/gpu/drm/drm_stub.c
index 39d8645..1a837e1 100644
--- a/drivers/gpu/drm/drm_stub.c
+++ b/drivers/gpu/drm/drm_stub.c
@@ -396,6 +396,7 @@ EXPORT_SYMBOL(drm_get_minor);
int drm_put_minor(struct drm_minor **minor_p)
{
struct drm_minor *minor = *minor_p;
+ int minor_id = minor->index;
DRM_DEBUG("release secondary minor %d\n", minor->index);
@@ -403,11 +404,12 @@ int drm_put_minor(struct drm_minor **minor_p)
drm_debugfs_cleanup(minor);
#endif
- drm_sysfs_device_remove(minor);
+ if (!drm_device_is_unplugged(minor->dev))
+ drm_sysfs_device_remove(minor);
- idr_remove(_minors_idr, minor->index);
+ idr_remove(_minors_idr, minor_id);
- kfree(minor);
+ drm_sysfs_device_put(minor);
*minor_p = NULL;
return 0;
}
diff --git a/drivers/gpu/drm/drm_sysfs.c b/drivers/gpu/drm/drm_sysfs.c
index 2290b3b..e0733a0 100644
--- a/drivers/gpu/drm/drm_sysfs.c
+++ b/drivers/gpu/drm/drm_sysfs.c
@@ -170,7 +170,7 @@ void drm_sysfs_destroy(void)
* with cleaning up any other stuff. But we do that in the DRM core, so
* this function can just return and hope that the core does its job.
*/
-static void drm_sysfs_device_release(struct device *dev)
+static void drm_sysfs_connector_release(struct device *dev)
{
memset(dev, 0, sizeof(struct device));
return;
@@ -399,7 +399,7 @@ int drm_sysfs_connector_add(struct drm_connector *connector)
connector->kdev.parent = >primary->kdev;
connector->kdev.class = drm_class;
- connector->kdev.release = drm_sysfs_device_release;
+ connector->kdev.release = drm_sysfs_connector_release;
DRM_DEBUG("adding \"%s\" to sysfs\n",
drm_get_connector_name(connector));
@@ -512,6 +512,17 @@ void drm_sysfs_hotplug_event(struct drm_device *dev)
}
EXPORT_SYMBOL(drm_sysfs_hotplug_event);
+/*
+ * We can only free the drm_minor once its embedded struct device has
+ * been released.
+ */
+static void drm_sysfs_device_release(struct device *dev)
+{
+ struct drm_minor *minor = container_of(dev, struct drm_minor, kdev);
+
+ kfree(minor);
+}
+
/**
* drm_sysfs_device_add - adds a class device to sysfs for a character
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Thu, Oct 10, 2013 at 03:23:45AM +0100, Dave Airlie wrote:
>
> > I think David Arlie also needs a quiet talking to about how to use the
> > device model:
> >
> > int drm_sysfs_device_add(struct drm_minor *minor)
> > {
> > minor->kdev.release = drm_sysfs_device_release;
> > ...
> > err = device_register(>kdev);
> > }
> >
> > static void drm_sysfs_device_release(struct device *dev)
> > {
> > memset(dev, 0, sizeof(struct device));
> > return;
> > }
> >
> > Since when has that been acceptable in a release function?
>
> Well the commit that added it had a reason that seems to cover some other
> device model abuses, so maybe someone who actually understands the device
> model (all 2 people) can review usage.
Please - there's more than two people who understand this stuff.
You appear to manage to understand the refcounting principle for things
like the DRM framebuffers, DRM buffer objects etc, and the driver model
(and kobjects in general) are no different.
I've just been reading the code around here little more, and hit the usb
implementation. I don't see how USB drivers get cleaned up when they're
disconnected from the USB bus. I see drm_unplug_dev() which gets called
on a USB disconnection (from udl/udl_drv.c) which effectively makes the
device unavailable, but I don't see anything which frees anything. I see
nothing calling drm_put_dev() here. How does the drm_device in this case
get freed?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Thu, Oct 10, 2013 at 03:23:45AM +0100, Dave Airlie wrote:
I think David Arlie also needs a quiet talking to about how to use the
device model:
int drm_sysfs_device_add(struct drm_minor *minor)
{
minor-kdev.release = drm_sysfs_device_release;
...
err = device_register(minor-kdev);
}
static void drm_sysfs_device_release(struct device *dev)
{
memset(dev, 0, sizeof(struct device));
return;
}
Since when has that been acceptable in a release function?
Well the commit that added it had a reason that seems to cover some other
device model abuses, so maybe someone who actually understands the device
model (all 2 people) can review usage.
Please - there's more than two people who understand this stuff.
You appear to manage to understand the refcounting principle for things
like the DRM framebuffers, DRM buffer objects etc, and the driver model
(and kobjects in general) are no different.
I've just been reading the code around here little more, and hit the usb
implementation. I don't see how USB drivers get cleaned up when they're
disconnected from the USB bus. I see drm_unplug_dev() which gets called
on a USB disconnection (from udl/udl_drv.c) which effectively makes the
device unavailable, but I don't see anything which frees anything. I see
nothing calling drm_put_dev() here. How does the drm_device in this case
get freed?
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Thu, Oct 10, 2013 at 10:19:20AM +0100, Russell King - ARM Linux wrote:
On Thu, Oct 10, 2013 at 03:23:45AM +0100, Dave Airlie wrote:
I think David Arlie also needs a quiet talking to about how to use the
device model:
int drm_sysfs_device_add(struct drm_minor *minor)
{
minor-kdev.release = drm_sysfs_device_release;
...
err = device_register(minor-kdev);
}
static void drm_sysfs_device_release(struct device *dev)
{
memset(dev, 0, sizeof(struct device));
return;
}
Since when has that been acceptable in a release function?
Well the commit that added it had a reason that seems to cover some other
device model abuses, so maybe someone who actually understands the device
model (all 2 people) can review usage.
Please - there's more than two people who understand this stuff.
You appear to manage to understand the refcounting principle for things
like the DRM framebuffers, DRM buffer objects etc, and the driver model
(and kobjects in general) are no different.
I've just been reading the code around here little more, and hit the usb
implementation. I don't see how USB drivers get cleaned up when they're
disconnected from the USB bus. I see drm_unplug_dev() which gets called
on a USB disconnection (from udl/udl_drv.c) which effectively makes the
device unavailable, but I don't see anything which frees anything. I see
nothing calling drm_put_dev() here. How does the drm_device in this case
get freed?
Don't worry about the above - I've worked out how USB works in that regard.
However, I can't say that I like the feel of the code in drm_unplug_dev()
and drm_put_dev() - if these two can run simultaneously in two threads of
execution, there's the chance that things will go awry. I don't think
that can happen due to the way the unplugged-ness is dealt with, but it
doesn't feel safe.
I think something like the below may address the issue - I've only build
tested this so far.
We have another case where DRM does the wrong thing here too - a similar
thing goes on with connectors as well. I've not yet looked into this,
but I'll take a look later today.
Fengguang - could you give this some runs through your marvellous test
system and see whether it fixes the problem you're seeing please?
David - maybe you can find some time to look at the Armada driver I
re-posted last weekend?
drivers/gpu/drm/drm_stub.c |8 +---
drivers/gpu/drm/drm_sysfs.c | 42 +-
include/drm/drmP.h |1 +
3 files changed, 39 insertions(+), 12 deletions(-)
diff --git a/drivers/gpu/drm/drm_stub.c b/drivers/gpu/drm/drm_stub.c
index 39d8645..1a837e1 100644
--- a/drivers/gpu/drm/drm_stub.c
+++ b/drivers/gpu/drm/drm_stub.c
@@ -396,6 +396,7 @@ EXPORT_SYMBOL(drm_get_minor);
int drm_put_minor(struct drm_minor **minor_p)
{
struct drm_minor *minor = *minor_p;
+ int minor_id = minor-index;
DRM_DEBUG(release secondary minor %d\n, minor-index);
@@ -403,11 +404,12 @@ int drm_put_minor(struct drm_minor **minor_p)
drm_debugfs_cleanup(minor);
#endif
- drm_sysfs_device_remove(minor);
+ if (!drm_device_is_unplugged(minor-dev))
+ drm_sysfs_device_remove(minor);
- idr_remove(drm_minors_idr, minor-index);
+ idr_remove(drm_minors_idr, minor_id);
- kfree(minor);
+ drm_sysfs_device_put(minor);
*minor_p = NULL;
return 0;
}
diff --git a/drivers/gpu/drm/drm_sysfs.c b/drivers/gpu/drm/drm_sysfs.c
index 2290b3b..e0733a0 100644
--- a/drivers/gpu/drm/drm_sysfs.c
+++ b/drivers/gpu/drm/drm_sysfs.c
@@ -170,7 +170,7 @@ void drm_sysfs_destroy(void)
* with cleaning up any other stuff. But we do that in the DRM core, so
* this function can just return and hope that the core does its job.
*/
-static void drm_sysfs_device_release(struct device *dev)
+static void drm_sysfs_connector_release(struct device *dev)
{
memset(dev, 0, sizeof(struct device));
return;
@@ -399,7 +399,7 @@ int drm_sysfs_connector_add(struct drm_connector *connector)
connector-kdev.parent = dev-primary-kdev;
connector-kdev.class = drm_class;
- connector-kdev.release = drm_sysfs_device_release;
+ connector-kdev.release = drm_sysfs_connector_release;
DRM_DEBUG(adding \%s\ to sysfs\n,
drm_get_connector_name(connector));
@@ -512,6 +512,17 @@ void drm_sysfs_hotplug_event(struct drm_device *dev)
}
EXPORT_SYMBOL(drm_sysfs_hotplug_event);
+/*
+ * We can only free the drm_minor once its embedded struct device has
+ * been released.
+ */
+static void drm_sysfs_device_release(struct device *dev)
+{
+ struct drm_minor *minor = container_of(dev, struct drm_minor, kdev);
+
+ kfree(minor);
+}
+
/**
* drm_sysfs_device_add - adds a class device to sysfs for a character driver
* @dev: DRM device to be added
@@ -554,17 +565,30 @@ int
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Thu, Oct 10, 2013 at 8:53 PM, Russell King - ARM Linux
li...@arm.linux.org.uk wrote:
On Thu, Oct 10, 2013 at 10:19:20AM +0100, Russell King - ARM Linux wrote:
On Thu, Oct 10, 2013 at 03:23:45AM +0100, Dave Airlie wrote:
I think David Arlie also needs a quiet talking to about how to use the
device model:
int drm_sysfs_device_add(struct drm_minor *minor)
{
minor-kdev.release = drm_sysfs_device_release;
...
err = device_register(minor-kdev);
}
static void drm_sysfs_device_release(struct device *dev)
{
memset(dev, 0, sizeof(struct device));
return;
}
Since when has that been acceptable in a release function?
Well the commit that added it had a reason that seems to cover some other
device model abuses, so maybe someone who actually understands the device
model (all 2 people) can review usage.
Please - there's more than two people who understand this stuff.
You appear to manage to understand the refcounting principle for things
like the DRM framebuffers, DRM buffer objects etc, and the driver model
(and kobjects in general) are no different.
I've just been reading the code around here little more, and hit the usb
implementation. I don't see how USB drivers get cleaned up when they're
disconnected from the USB bus. I see drm_unplug_dev() which gets called
on a USB disconnection (from udl/udl_drv.c) which effectively makes the
device unavailable, but I don't see anything which frees anything. I see
nothing calling drm_put_dev() here. How does the drm_device in this case
get freed?
Don't worry about the above - I've worked out how USB works in that regard.
However, I can't say that I like the feel of the code in drm_unplug_dev()
and drm_put_dev() - if these two can run simultaneously in two threads of
execution, there's the chance that things will go awry. I don't think
that can happen due to the way the unplugged-ness is dealt with, but it
doesn't feel safe.
I think something like the below may address the issue - I've only build
tested this so far.
We have another case where DRM does the wrong thing here too - a similar
thing goes on with connectors as well. I've not yet looked into this,
but I'll take a look later today.
Damn gmail screwed up my reply-all,
anyhoo I get the feeling I'd rather do this like fbdev does and stop using
an embedded kdev for this if I can. The lifetime of the minor and the sysfs
objects aren't necessarily that tied together esp with hot unplug of
USB devices.
e.g. when a USB device goes away I don't want the sysfs files to
remain hanging around,
or the userspace device nodes, however I can't remove all the internal
driver state until userspace finally closes all open device
files,
I think this is a bit of a bad misdesign of the whole device api that
sysfs exposure and device exposure is tied
to objects you are encourgaged to embed.
I think maybe I'll just copy fbdev and do a separate device_create
with a nonembedded dev.
Dave.
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
Damn gmail screwed up my reply-all,
anyhoo I get the feeling I'd rather do this like fbdev does and stop using
an embedded kdev for this if I can. The lifetime of the minor and the sysfs
objects aren't necessarily that tied together esp with hot unplug of
USB devices.
e.g. when a USB device goes away I don't want the sysfs files to
remain hanging around,
or the userspace device nodes, however I can't remove all the internal
driver state until userspace finally closes all open device
files,
I think this is a bit of a bad misdesign of the whole device api that
sysfs exposure and device exposure is tied
to objects you are encourgaged to embed.
I think maybe I'll just copy fbdev and do a separate device_create
with a nonembedded dev.
Okay it compiles but not boot tested yet.,attached as well just in
case gmail eats it.
From be06235b2d2a26c0e52c099e70922b71f1245b9e Mon Sep 17 00:00:00 2001
From: Dave Airlie airl...@redhat.com
Date: Fri, 11 Oct 2013 14:07:25 +1000
Subject: [PATCH] drm/sysfs: sort out minor and connector device object
lifetimes.
So drm was abusing device lifetimes, by having embedded device structures
in the minor and connector it meant that the lifetime of the internal drm
objects (drm_minor and drm_connector) were tied to the lifetime of the device
files in sysfs, so if something kept those files opened the current code
would kfree the objects and things would go downhill from there.
Now in reality there is no need for these lifetimes to be so intertwined,
especailly with hotplugging of devices where we wish to remove the sysfs
and userspace facing pieces before we can unwind the internal objects due
to open userspace files or mmaps, so split the objects out so the struct
device is no longer embedded and do what fbdev does and just allocate
and remove the sysfs inodes separately.
Signed-off-by: Dave Airlie airl...@redhat.com
---
drivers/gpu/drm/drm_sysfs.c | 94 ++-
drivers/gpu/drm/i915/i915_irq.c | 8 ++--
drivers/gpu/drm/i915/i915_sysfs.c | 28 ++--
include/drm/drmP.h| 2 +-
include/drm/drm_crtc.h| 2 +-
5 files changed, 54 insertions(+), 80 deletions(-)
diff --git a/drivers/gpu/drm/drm_sysfs.c b/drivers/gpu/drm/drm_sysfs.c
index 2290b3b..dae42c7 100644
--- a/drivers/gpu/drm/drm_sysfs.c
+++ b/drivers/gpu/drm/drm_sysfs.c
@@ -22,8 +22,8 @@
#include drm/drm_core.h
#include drm/drmP.h
-#define to_drm_minor(d) container_of(d, struct drm_minor, kdev)
-#define to_drm_connector(d) container_of(d, struct drm_connector, kdev)
+#define to_drm_minor(d) dev_get_drvdata(d)
+#define to_drm_connector(d) dev_get_drvdata(d)
static struct device_type drm_sysfs_device_minor = {
.name = drm_minor
@@ -162,20 +162,6 @@ void drm_sysfs_destroy(void)
drm_class = NULL;
}
-/**
- * drm_sysfs_device_release - do nothing
- * @dev: Linux device
- *
- * Normally, this would free the DRM device associated with @dev, along
- * with cleaning up any other stuff. But we do that in the DRM core, so
- * this function can just return and hope that the core does its job.
- */
-static void drm_sysfs_device_release(struct device *dev)
-{
-memset(dev, 0, sizeof(struct device));
-return;
-}
-
/*
* Connector properties
*/
@@ -394,29 +380,26 @@ int drm_sysfs_connector_add(struct drm_connector
*connector)
int i;
int ret;
-/* We shouldn't get called more than once for the same connector */
-BUG_ON(device_is_registered(connector-kdev));
-
-connector-kdev.parent = dev-primary-kdev;
-connector-kdev.class = drm_class;
-connector-kdev.release = drm_sysfs_device_release;
+if (connector-kdev)
+return 0;
+/* We shouldn't get called more than once for the same connector */
+connector-kdev = device_create(drm_class, dev-primary-kdev,
+0, connector, card%d-%s,
+dev-primary-index, drm_get_connector_name(connector));
DRM_DEBUG(adding \%s\ to sysfs\n,
drm_get_connector_name(connector));
-dev_set_name(connector-kdev, card%d-%s,
- dev-primary-index, drm_get_connector_name(connector));
-ret = device_register(connector-kdev);
-
-if (ret) {
-DRM_ERROR(failed to register connector device: %d\n, ret);
+if (IS_ERR(connector-kdev)) {
+DRM_ERROR(failed to register connector device: %ld\n,
PTR_ERR(connector-kdev));
+ret = PTR_ERR(connector-kdev);
goto out;
}
/* Standard attributes */
for (attr_cnt = 0; attr_cnt ARRAY_SIZE(connector_attrs); attr_cnt++) {
-ret = device_create_file(connector-kdev, connector_attrs[attr_cnt]);
+ret = device_create_file(connector-kdev, connector_attrs[attr_cnt]);
if (ret)
goto err_out_files;
}
@@ -433,7 +416,7 @@ int drm_sysfs_connector_add(struct drm_connector *connector)
case DRM_MODE_CONNECTOR_Component:
case DRM_MODE_CONNECTOR_TV:
for
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
I think something like the below may address the issue - I've only build tested this so far. We have another case where DRM does the wrong thing here too - a similar thing goes on with connectors as well. I've not yet looked into this, but I'll take a look later today. Fengguang - could you give this some runs through your marvellous test system and see whether it fixes the problem you're seeing please? Russell, I applied the patch (as commit 17e57131d4c2) on top of v3.12-rc4 and here are the test results for the kernels with CONFIG_DRM=y CONFIG_DEBUG_KOBJECT_RELEASE=y As you may see, it's pure improvements. :) Tested-by: Fengguang Wu fengguang...@intel.com i386-randconfig-c3-1009 +---+---+--+ | | v3.12-rc4 | 17e57131d4c2 | +---+---+--+ | has_kernel_error_warning | 63| | | BUG:kernel_early_hang_without_any_printk_output | 7 | | | WARNING:CPU:PID:at_fs/sysfs/dir.c:sysfs_add_one() | 56| | | WARNING:CPU:PID:at_lib/kobject.c:kobject_add_internal() | 56| | | WARNING:CPU:PID:at_drivers/base/driver.c:driver_unregister() | 56| | | WARNING:CPU:PID:at_lib/debugobjects.c:debug_print_object()| 56| | | INFO:trying_to_register_non-static_key| 56| | | BUG:unable_to_handle_kernel_NULL_pointer_dereference_at(null) | 56| | | Oops:PREEMPT_SMP | 56| | | Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 56| | | BUG:kernel_boot_oops | 56| | | good_boots| 0 | 9 | +---+---+--+ x86_64-randconfig-j5-1009 ++---+--+ || v3.12-rc4 | 17e57131d4c2 | ++---+--+ | good_boots | 30| 9| ++---+--+ x86_64-randconfig-j3-1009 +---+---+--+ | | v3.12-rc4 | 17e57131d4c2 | +---+---+--+ | has_kernel_error_warning | 60| | | WARNING:CPU:PID:at_fs/sysfs/dir.c:sysfs_add_one() | 60| | | WARNING:CPU:PID:at_lib/kobject.c:kobject_add_internal() | 60| | | WARNING:CPU:PID:at_drivers/base/driver.c:driver_unregister() | 60| | | WARNING:CPU:PID:at_lib/debugobjects.c:debug_print_object()| 60| | | WARNING:CPU:PID:at_arch/x86/kernel/irq_64.c:handle_irq() | 3 | | | BUG:unable_to_handle_kernel_NULL_pointer_dereference_at | 59| | | Oops:DEBUG_PAGEALLOC | 28| | | WARNING:CPU:PID:at_lib/list_debug.c:__list_del_entry()| 28| | | BUG:scheduling_while_atomic | 30| | | INFO:lockdep_is_turned_off| 30| | | BUG:unable_to_handle_kernel_paging_request_at | 55| | | kernel_BUG_at_kernel/smpboot.c| 13| | | invalid_opcode:DEBUG_PAGEALLOC| 13| | | general_protection_fault:DEBUG_PAGEALLOC | 2 | | | Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 4 | | | BUG:kernel_boot_oops | 60| | | WARNING:CPU:PID:at_kernel/workqueue.c:work_fixup_activate() | 24| | | WARNING:CPU:PID:at_kernel/workqueue.c:__queue_work() | 10| | | BUG:unable_to_handle_kernel_NULL_pointer_dereference_at(null) | 2 | | | BUG:unable_to_handle_kernel | 1 | | | BUG:unable_to_handle_kernel_NULL_point| 1 | | | BUG:unable_to_handle_kernel_NULL_pointer_dereference | 4 | | | BUG:unable_to_handle_kernel_NULL_pointer_derefe
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Wed, Oct 9, 2013 at 7:23 PM, Dave Airlie wrote: > > Well the commit that added it had a reason that seems to cover some other > device model abuses, so maybe someone who actually understands the device > model (all 2 people) can review usage Actually, I think it's the same bug. You *cannot* just reuse the storage. Doing a "memset()" doesn't improve anything. The basic issue is that if you reuse it, you're buggy. End of story. Why? It's refcounted, and it's out of your hands. Reusing it is wrong - because it might still be used. The fact that you "released" it is immaterial. Others can have refcounts (and through /sys etc, historically really do have them). Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
> I think David Arlie also needs a quiet talking to about how to use the
> device model:
>
> int drm_sysfs_device_add(struct drm_minor *minor)
> {
> minor->kdev.release = drm_sysfs_device_release;
> ...
> err = device_register(>kdev);
> }
>
> static void drm_sysfs_device_release(struct device *dev)
> {
> memset(dev, 0, sizeof(struct device));
> return;
> }
>
> Since when has that been acceptable in a release function?
Well the commit that added it had a reason that seems to cover some other
device model abuses, so maybe someone who actually understands the device
model (all 2 people) can review usage.
Dave.
commit 77d26dc9b9805f322f5a1f6e559b18ad66205bd9
Author: Ma Ling
Date: Thu Apr 16 17:51:25 2009 +0800
drm: clean dirty memory after device release
In current code we register/unregister connector object by
drm_sysfs_connector_add/remove function.
However under some cases, we need to dynamically register or
unregister device
multiple times, so we have to go through register -> unregister
->register
routine.
Because after device_unregister function our memory is dirty, we need
to do
clean operation in order to re-register the device, otherwise the
system
will crash. The patch intends to clean device after device release.
Signed-off-by: Ma Ling
Signed-off-by: Dave Airlie
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 8, 2013 at 8:45 PM, Linus Torvalds wrote: > On Tue, Oct 8, 2013 at 3:48 PM, Greg Kroah-Hartman > wrote: >> >> It's going to make things really noisy at boot time, but then it should >> settle down and not be bad at all. Let's try it and see if it helps or >> not. > > Yeah. And quite frankly, normally that whole DEBUG_KOBJECT_RELEASE > thing is hopefully only enabled in debug kernels (like maybe the > Fedora rawhide one, or at developers), so being a bit more verbose is > likely ok. We haven't enabled it yet. Dave hit issues with it early in the merge window and we left it off. That being said, it's something we'll look at again soon and having it be a bit more verbose wouldn't be horrible. josh -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 8, 2013 at 8:45 PM, Linus Torvalds torva...@linux-foundation.org wrote: On Tue, Oct 8, 2013 at 3:48 PM, Greg Kroah-Hartman gre...@linuxfoundation.org wrote: It's going to make things really noisy at boot time, but then it should settle down and not be bad at all. Let's try it and see if it helps or not. Yeah. And quite frankly, normally that whole DEBUG_KOBJECT_RELEASE thing is hopefully only enabled in debug kernels (like maybe the Fedora rawhide one, or at developers), so being a bit more verbose is likely ok. We haven't enabled it yet. Dave hit issues with it early in the merge window and we left it off. That being said, it's something we'll look at again soon and having it be a bit more verbose wouldn't be horrible. josh -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
I think David Arlie also needs a quiet talking to about how to use the
device model:
int drm_sysfs_device_add(struct drm_minor *minor)
{
minor-kdev.release = drm_sysfs_device_release;
...
err = device_register(minor-kdev);
}
static void drm_sysfs_device_release(struct device *dev)
{
memset(dev, 0, sizeof(struct device));
return;
}
Since when has that been acceptable in a release function?
Well the commit that added it had a reason that seems to cover some other
device model abuses, so maybe someone who actually understands the device
model (all 2 people) can review usage.
Dave.
commit 77d26dc9b9805f322f5a1f6e559b18ad66205bd9
Author: Ma Ling ling...@intel.com
Date: Thu Apr 16 17:51:25 2009 +0800
drm: clean dirty memory after device release
In current code we register/unregister connector object by
drm_sysfs_connector_add/remove function.
However under some cases, we need to dynamically register or
unregister device
multiple times, so we have to go through register - unregister
-register
routine.
Because after device_unregister function our memory is dirty, we need
to do
clean operation in order to re-register the device, otherwise the
system
will crash. The patch intends to clean device after device release.
Signed-off-by: Ma Ling ling...@intel.com
Signed-off-by: Dave Airlie airl...@redhat.com
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Wed, Oct 9, 2013 at 7:23 PM, Dave Airlie airl...@linux.ie wrote: Well the commit that added it had a reason that seems to cover some other device model abuses, so maybe someone who actually understands the device model (all 2 people) can review usage Actually, I think it's the same bug. You *cannot* just reuse the storage. Doing a memset() doesn't improve anything. The basic issue is that if you reuse it, you're buggy. End of story. Why? It's refcounted, and it's out of your hands. Reusing it is wrong - because it might still be used. The fact that you released it is immaterial. Others can have refcounts (and through /sys etc, historically really do have them). Linus -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 08, 2013 at 05:45:37PM -0700, Linus Torvalds wrote: > normally that whole DEBUG_KOBJECT_RELEASE > thing is hopefully only enabled in debug kernels (like maybe the > Fedora rawhide one Nope. After spending a couple of days fruitlessly trying to get my machine to boot with it enabled, I think we decided it's not worth the pain. Debugging "hangs instantly and silently on boot" bugs are hard enough when you're in front of the machine, doing so via bugzilla isn't something I'd wish on my worst enemy. Dave -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 8, 2013 at 3:48 PM, Greg Kroah-Hartman wrote: > > It's going to make things really noisy at boot time, but then it should > settle down and not be bad at all. Let's try it and see if it helps or > not. Yeah. And quite frankly, normally that whole DEBUG_KOBJECT_RELEASE thing is hopefully only enabled in debug kernels (like maybe the Fedora rawhide one, or at developers), so being a bit more verbose is likely ok. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 08, 2013 at 03:48:40PM -0700, Greg KH wrote:
> On Tue, Oct 08, 2013 at 11:14:17PM +0100, Russell King - ARM Linux wrote:
> > On Tue, Oct 08, 2013 at 08:17:42PM +0800, Fengguang Wu wrote:
> > > I find the above debug messages very helpful in locating the buggy
> > > driver. How about enabling it whenever CONFIG_DEBUG_KOBJECT_RELEASE is
> > > enabled? Something like
> > >
> > > #ifdef CONFIG_DEBUG_KOBJECT_RELEASE
> > > - pr_debug("kobject: '%s' (%p): %s, parent %p (delayed)\n",
> > > + printk(KERN_INFO "kobject: '%s' (%p): %s, parent %p (delayed)\n",
> > > kobject_name(kobj), kobj, __func__, kobj->parent);
> > >
> > > pr_debug() won't be displayed by default, and it depends on
> > > CONFIG_DYNAMIC_DEBUG.
> >
> > Please consider using pr_info() instead of printk(KERN_INFO - it's
> > slightly less typing.
Good suggestion!
> > I can see that being a useful change while we have these problematical
> > instances to track down, but I do wonder whether it'll make the thing
> > too noisy. Does anyone have other opinions on this point? Linus?
> > Greg?
>
> It's going to make things really noisy at boot time, but then it should
> settle down and not be bad at all.
FYI, in the randconfig kernel involved in this thread, it will emit
about 20 lines of dmesg.
> Let's try it and see if it helps or not.
OK, I'll submit a patch. These messages would allow me to do a
statistic analyze of similar bugs: since I'm testing 100+ new
randconfigs every day, some of them will include the buggy drivers and
some not, we can collect all these lines
[2.929669] kobject: 'drm' (880006db2848): kobject_release, parent
88189648 (delayed)
...
[3.750200] kobject: 'mc13783-adc' (880007707000): kobject_release,
parent 88189248 (delayed)
, count how many times each one shows up in the GOOD kernel boots and
how many show up in the BAD boots. Then we may be able to learn which
drivers are likely buggy and should be manually checked.
Thanks,
Fengguang
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 08, 2013 at 11:14:17PM +0100, Russell King - ARM Linux wrote:
> On Tue, Oct 08, 2013 at 08:17:42PM +0800, Fengguang Wu wrote:
> > I find the above debug messages very helpful in locating the buggy
> > driver. How about enabling it whenever CONFIG_DEBUG_KOBJECT_RELEASE is
> > enabled? Something like
> >
> > #ifdef CONFIG_DEBUG_KOBJECT_RELEASE
> > - pr_debug("kobject: '%s' (%p): %s, parent %p (delayed)\n",
> > + printk(KERN_INFO "kobject: '%s' (%p): %s, parent %p (delayed)\n",
> > kobject_name(kobj), kobj, __func__, kobj->parent);
> >
> > pr_debug() won't be displayed by default, and it depends on
> > CONFIG_DYNAMIC_DEBUG.
>
> Please consider using pr_info() instead of printk(KERN_INFO - it's
> slightly less typing.
>
> I can see that being a useful change while we have these problematical
> instances to track down, but I do wonder whether it'll make the thing
> too noisy. Does anyone have other opinions on this point? Linus?
> Greg?
It's going to make things really noisy at boot time, but then it should
settle down and not be bad at all. Let's try it and see if it helps or
not.
thanks,
greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 08, 2013 at 08:17:42PM +0800, Fengguang Wu wrote:
> I find the above debug messages very helpful in locating the buggy
> driver. How about enabling it whenever CONFIG_DEBUG_KOBJECT_RELEASE is
> enabled? Something like
>
> #ifdef CONFIG_DEBUG_KOBJECT_RELEASE
> - pr_debug("kobject: '%s' (%p): %s, parent %p (delayed)\n",
> + printk(KERN_INFO "kobject: '%s' (%p): %s, parent %p (delayed)\n",
> kobject_name(kobj), kobj, __func__, kobj->parent);
>
> pr_debug() won't be displayed by default, and it depends on
> CONFIG_DYNAMIC_DEBUG.
Please consider using pr_info() instead of printk(KERN_INFO - it's
slightly less typing.
I can see that being a useful change while we have these problematical
instances to track down, but I do wonder whether it'll make the thing
too noisy. Does anyone have other opinions on this point? Linus?
Greg?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
Hi Russell,
> I'm now trying to disable all drivers shows up in the kobject_release
> messages:
>
> [2.756392] kobject: 'ipmi_si' (880007764a00): kobject_release, parent
> 881b7648 (delayed)
> [2.758091] kobject: 'ipmi_si' (880007764800): kobject_release, parent
> 88221248 (delayed)
> [2.759858] kobject: 'ipmi_si' (880007764c00): kobject_release, parent
> 88189248 (delayed)
> [2.929669] kobject: 'drm' (880006db2848): kobject_release, parent
> 88189648 (delayed)
> [2.932143] kobject: 'drm' (880006daf000): kobject_release, parent
> (null) (delayed)
> [2.941844] kobject: 'controlD64' (880006db2020): kobject_release,
> parent (null) (delayed)
> [2.958432] kobject: 'parport_pc.956' (880006db2020): kobject_release,
> parent (null) (delayed)
> [2.965698] kobject: 'parport_pc.888' (880006dc5820): kobject_release,
> parent (null) (delayed)
> [2.972583] kobject: 'parport_pc.632' (880006dc5020): kobject_release,
> parent (null) (delayed)
> [3.031704] kobject: 'physmap-flash' (880006ddc800): kobject_release,
> parent 88189248 (delayed)
> [3.055119] kobject: 'docg3' (880006de3c00): kobject_release, parent
> 88189248 (delayed)
> [3.496256] kobject: 'gpio-vbus' (880006817400): kobject_release,
> parent 88189248 (delayed)
> [3.619023] kobject: '(null)' (88000777baf0): kobject_release, parent
> (null) (delayed)
> [3.657587] kobject: 'proc-osm' (88000684be00): kobject_release,
> parent 880006849848 (delayed)
> [3.662546] kobject: 'mc13xxx-rtc' (880006851e00): kobject_release,
> parent 88189248 (delayed)
> [3.669144] kobject: 'rtc-msm6242' (880006851c00): kobject_release,
> parent 88189248 (delayed)
> [3.677494] kobject: 'pcap-rtc' (880006851a00): kobject_release,
> parent 88189248 (delayed)
> [3.680280] kobject: 'rtc-rp5c01' (880006855a00): kobject_release,
> parent 88189248 (delayed)
> [3.750200] kobject: 'mc13783-adc' (880007707000): kobject_release,
> parent 88189248 (delayed)
I find the above debug messages very helpful in locating the buggy
driver. How about enabling it whenever CONFIG_DEBUG_KOBJECT_RELEASE is
enabled? Something like
#ifdef CONFIG_DEBUG_KOBJECT_RELEASE
- pr_debug("kobject: '%s' (%p): %s, parent %p (delayed)\n",
+ printk(KERN_INFO "kobject: '%s' (%p): %s, parent %p (delayed)\n",
kobject_name(kobj), kobj, __func__, kobj->parent);
pr_debug() won't be displayed by default, and it depends on
CONFIG_DYNAMIC_DEBUG.
> with some manual bisects, I find a good config (attached) that can
> reliably boot the kernel up.
>
> Based on that config, I tried adding parport_pc and see that it still
> boots fine.
>
> Adding drm, however will bring back the oops. Will try a kernel based
> on the original kconfig with drm disabled only.
Thanks,
Fengguang
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
* Ingo Molnar wrote:
> * Fengguang Wu wrote:
>
> > After enabling CONFIG_DEBUG_OBJECTS_TIMERS=y, it will issue a WARNING
> > followed by a "BUG: ..."
>
> Cool!
>
> > [2.802167] parport_pc 00:04: reported by Plug and Play ACPI
> > [2.803818] parport0: PC-style at 0x378, irq 7 [PCSPP(,...)]
> > [2.806035] kobject: 'parport_pc.956' (880006dc3820):
> > kobject_release, parent (null) (delayed)
> > [2.808626] [ cut here ]
> >
> > [2.809776] WARNING: CPU: 1 PID: 1 at
> > /c/wfg/linux/lib/debugobjects.c:260 debug_print_object+0x7c/0x8d()
> > [2.812433] ODEBUG: init active (active state 0) object type: timer_list
> > hint: (null)
> > ..
> > [3.796079] BUG: unable to handle kernel NULL pointer dereference at
> > (null)
>
> Mind posting the full backtrace? It should show the actual callsite that
> zaps the timer. I don't think we got that dump so far, I've only seen
> timer list corruption backtraces. (but I haven't read the whole thread.)
I see you already posted the dmesg in another part of this thread, so the
relevant call-trace for parport_pc.c is:
> [2.812579] Call Trace:
> [2.812579] [] dump_stack+0x4f/0x84
> [2.812579] [] warn_slowpath_common+0x72/0x8c
> [2.812579] [] ? debug_print_object+0x7c/0x8d
> [2.812579] [] warn_slowpath_fmt+0x41/0x43
> [2.812579] [] debug_print_object+0x7c/0x8d
> [2.812579] [] __debug_object_init+0x27c/0x2c6
> [2.812579] [] ? __debug_object_init+0x2b7/0x2c6
> [2.812579] [] debug_object_init+0x14/0x16
> [2.812579] [] init_timer_key+0x23/0x65
> [2.812579] [] kobject_release+0x90/0xba
> [2.812579] [] kobject_put+0x4d/0x51
> [2.812579] [] put_device+0x12/0x14
> [2.812579] [] platform_device_put+0x12/0x14
> [2.812579] [] platform_device_unregister+0x16/0x1a
> [2.812579] [] parport_pc_probe_port+0x7c4/0x7d9
> [2.812579] [] parport_pc_init+0x2b0/0x317
> [2.812579] [] ? parport_setup+0x147/0x147
> [2.812579] [] do_one_initcall+0x8e/0x132
> [2.812579] [] ? param_array_set+0x7f/0xf2
> [2.812579] [] ? parse_args+0x1ad/0x26c
> [2.812579] [] kernel_init_freeable+0x132/0x1bc
> [2.812579] [] ? do_early_param+0x86/0x86
> [2.812579] [] ? rest_init+0xc3/0xc3
> [2.812579] [] kernel_init+0x9/0xcc
> [2.812579] [] ret_from_fork+0x7c/0xb0
> [2.812579] [] ? rest_init+0xc3/0xc3
> [2.812579] ---[ end trace ce7dd707bef7dc3e ]---
It appears a delayed callback is already active for that platform device?
Looks weird, because in theory platform devices are allocated anew in
parport_pc_probe_port():
pdev = platform_device_register_simple("parport_pc",
base, NULL, 0);
if (IS_ERR(pdev))
return NULL;
dev = >dev;
Which is allocated via zalloc(), so no reuse.
and then released if the hardware port is not present:
out1:
if (pdev)
platform_device_unregister(pdev);
Thanks,
Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
* Fengguang Wu wrote: > After enabling CONFIG_DEBUG_OBJECTS_TIMERS=y, it will issue a WARNING > followed by a "BUG: ..." Cool! > [2.802167] parport_pc 00:04: reported by Plug and Play ACPI > [2.803818] parport0: PC-style at 0x378, irq 7 [PCSPP(,...)] > [2.806035] kobject: 'parport_pc.956' (880006dc3820): kobject_release, > parent (null) (delayed) > [2.808626] [ cut here ] > > [2.809776] WARNING: CPU: 1 PID: 1 at /c/wfg/linux/lib/debugobjects.c:260 > debug_print_object+0x7c/0x8d() > [2.812433] ODEBUG: init active (active state 0) object type: timer_list > hint: (null) > .. > [3.796079] BUG: unable to handle kernel NULL pointer dereference at > (null) Mind posting the full backtrace? It should show the actual callsite that zaps the timer. I don't think we got that dump so far, I've only seen timer list corruption backtraces. (but I haven't read the whole thread.) Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 08, 2013 at 09:58:16AM +0200, Ingo Molnar wrote: > > * Linus Torvalds wrote: > > > On Mon, Oct 7, 2013 at 1:35 AM, Fengguang Wu wrote: > > > On Mon, Oct 07, 2013 at 01:12:17AM -0700, Linus Torvalds wrote: > > > > > > My pleasure! Here are 100 randomly selected call traces. Also attached > > > several full dmesgs and the kconfig. > > > > Ok, they may be randomly selected, but they are all the same. Which is > > good, I guess, we're only talking about one bug. > > > > Anyway, they all have RIP:run_timer_softirq+0x12c/0x1b8, and the code is > > > >0: 8b 65 c8 mov-0x38(%rbp),%esp > >3: 4d 39 ec cmp%r13,%r12 > >6: 0f 84 2f ff ff ff je 0xff3b > >c: 41 8b 4c 24 18 mov0x18(%r12),%ecx > > 11: 4d 8b 74 24 20 mov0x20(%r12),%r14 > > 16: 4d 8b 7c 24 28 mov0x28(%r12),%r15 > > 1b: 4c 89 63 38 mov%r12,0x38(%rbx) > > 1f: 49 8b 44 24 08 mov0x8(%r12),%rax > > 24: 49 8b 14 24 mov(%r12),%rdx > > 28: 83 e1 02 and$0x2,%ecx > > 2b:* 48 89 42 08 mov%rax,0x8(%rdx) <-- trapping instruction > > 2f: 48 89 10 mov%rdx,(%rax) > > 32: 48 b8 00 02 20 00 00 movabs $0xdead00200200,%rax > > > > where that constant is LIST_POISON2 and the "and $2" seems to be > > TIMER_IRQSAFE. So the trapping instruction *looks* like it's doing > > __list_del() on the timer, and timer->next is NULL. > > > > So somebody added a timer, and then deallocated/cleared the structure > > before it triggered. The problem is, I can't see a way to figure out > > _who_ did that. > > I think CONFIG_DEBUG_OBJECTS_TIMERS=y should be able to detect that? It did help expose more information, and earlier. w/o debugobjects, we hit the "BUG: ..." directly: [2.964097] BUG: unable to handle kernel NULL pointer dereference at 0008 [2.96] IP: [] run_timer_softirq+0x126/0x1da [2.968060] PGD 0 [2.968060] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC [2.968060] CPU: 0 PID: 95 Comm: kworker/0:2 Not tainted 3.11.0-rc2-00010-gc817a67-dirty #5 [2.968060] Workqueue: events flush_to_ldisc [2.968060] task: 8800068544c0 ti: 880006856000 task.ti: 880006856000 [2.968060] RIP: 0010:[] [] run_timer_softirq+0x126/0x1da After enabling CONFIG_DEBUG_OBJECTS_TIMERS=y, it will issue a WARNING followed by a "BUG: ..." [2.802167] parport_pc 00:04: reported by Plug and Play ACPI [2.803818] parport0: PC-style at 0x378, irq 7 [PCSPP(,...)] [2.806035] kobject: 'parport_pc.956' (880006dc3820): kobject_release, parent (null) (delayed) [2.808626] [ cut here ] [2.809776] WARNING: CPU: 1 PID: 1 at /c/wfg/linux/lib/debugobjects.c:260 debug_print_object+0x7c/0x8d() [2.812433] ODEBUG: init active (active state 0) object type: timer_list hint: (null) .. [3.796079] BUG: unable to handle kernel NULL pointer dereference at (null) > Debugobjects hooks into deallocation paths and complains immediately if a > live timer is zapped that way. > > If the corrupion does not involve deallocation then it might be more > difficult to detect but not impossible either: for example if an object is > not freed but reused incorrectly then a repeat use of any timer function > will cause the debugobjects (and/or the timer code) to complain. > > So I'd suggest trying debugobjects, it should catch a fair number of > non-exotic object corruption patterns. Good to know that, thanks for the info! Regards, Fengguang -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 08, 2013 at 10:14:52AM +0800, Fengguang Wu wrote:
> [2.785188] kobject: 'drm' (880006dae048): kobject_release, parent
> 88189648 (delayed)
> [2.787362] kobject: 'drm' (880006dafe00): kobject_release, parent
> (null) (delayed)
> [2.789674] [drm] radeon kernel modesetting enabled.
> [2.791798] [drm:drm_proc_init] *ERROR* Cannot create /proc/dri/0
> [2.793280] [drm:drm_get_minor] *ERROR* DRM: Failed to initialize
> /proc/dri.
> [2.795591] kobject: 'controlD64' (880006dc3820): kobject_release,
> parent (null) (delayed)
> [2.797988] cirrus: probe of :00:02.0 failed with error -1
> [2.799675] usbcore: registered new interface driver udl
> [2.802167] parport_pc 00:04: reported by Plug and Play ACPI
> [2.803818] parport0: PC-style at 0x378, irq 7 [PCSPP(,...)]
> [2.806035] kobject: 'parport_pc.956' (880006dc3820): kobject_release,
> parent (null) (delayed)
Look very carefully at the above addresses of the controlD64 object and
the parport_pc.956 object. They're both the same - and the first hasn't
been run yet. It's a double addition because the culpret is not parport_pc,
but is controlD64 instead.
And here we have it:
int drm_put_minor(struct drm_minor **minor_p)
{
...
drm_sysfs_device_remove(minor);
...
kfree(minor);
}
void drm_sysfs_device_remove(struct drm_minor *minor)
{
if (minor->kdev.parent)
device_unregister(>kdev);
minor->kdev.parent = NULL;
}
I think David Arlie also needs a quiet talking to about how to use the
device model:
int drm_sysfs_device_add(struct drm_minor *minor)
{
minor->kdev.release = drm_sysfs_device_release;
...
err = device_register(>kdev);
}
static void drm_sysfs_device_release(struct device *dev)
{
memset(dev, 0, sizeof(struct device));
return;
}
Since when has that been acceptable in a release function?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
* Linus Torvalds wrote: > On Mon, Oct 7, 2013 at 1:35 AM, Fengguang Wu wrote: > > On Mon, Oct 07, 2013 at 01:12:17AM -0700, Linus Torvalds wrote: > > > > My pleasure! Here are 100 randomly selected call traces. Also attached > > several full dmesgs and the kconfig. > > Ok, they may be randomly selected, but they are all the same. Which is > good, I guess, we're only talking about one bug. > > Anyway, they all have RIP:run_timer_softirq+0x12c/0x1b8, and the code is > >0: 8b 65 c8 mov-0x38(%rbp),%esp >3: 4d 39 ec cmp%r13,%r12 >6: 0f 84 2f ff ff ff je 0xff3b >c: 41 8b 4c 24 18 mov0x18(%r12),%ecx > 11: 4d 8b 74 24 20 mov0x20(%r12),%r14 > 16: 4d 8b 7c 24 28 mov0x28(%r12),%r15 > 1b: 4c 89 63 38 mov%r12,0x38(%rbx) > 1f: 49 8b 44 24 08 mov0x8(%r12),%rax > 24: 49 8b 14 24 mov(%r12),%rdx > 28: 83 e1 02 and$0x2,%ecx > 2b:* 48 89 42 08 mov%rax,0x8(%rdx) <-- trapping instruction > 2f: 48 89 10 mov%rdx,(%rax) > 32: 48 b8 00 02 20 00 00 movabs $0xdead00200200,%rax > > where that constant is LIST_POISON2 and the "and $2" seems to be > TIMER_IRQSAFE. So the trapping instruction *looks* like it's doing > __list_del() on the timer, and timer->next is NULL. > > So somebody added a timer, and then deallocated/cleared the structure > before it triggered. The problem is, I can't see a way to figure out > _who_ did that. I think CONFIG_DEBUG_OBJECTS_TIMERS=y should be able to detect that? Debugobjects hooks into deallocation paths and complains immediately if a live timer is zapped that way. If the corrupion does not involve deallocation then it might be more difficult to detect but not impossible either: for example if an object is not freed but reused incorrectly then a repeat use of any timer function will cause the debugobjects (and/or the timer code) to complain. So I'd suggest trying debugobjects, it should catch a fair number of non-exotic object corruption patterns. Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
* Linus Torvalds torva...@linux-foundation.org wrote: On Mon, Oct 7, 2013 at 1:35 AM, Fengguang Wu fengguang...@intel.com wrote: On Mon, Oct 07, 2013 at 01:12:17AM -0700, Linus Torvalds wrote: My pleasure! Here are 100 randomly selected call traces. Also attached several full dmesgs and the kconfig. Ok, they may be randomly selected, but they are all the same. Which is good, I guess, we're only talking about one bug. Anyway, they all have RIP:run_timer_softirq+0x12c/0x1b8, and the code is 0: 8b 65 c8 mov-0x38(%rbp),%esp 3: 4d 39 ec cmp%r13,%r12 6: 0f 84 2f ff ff ff je 0xff3b c: 41 8b 4c 24 18 mov0x18(%r12),%ecx 11: 4d 8b 74 24 20 mov0x20(%r12),%r14 16: 4d 8b 7c 24 28 mov0x28(%r12),%r15 1b: 4c 89 63 38 mov%r12,0x38(%rbx) 1f: 49 8b 44 24 08 mov0x8(%r12),%rax 24: 49 8b 14 24 mov(%r12),%rdx 28: 83 e1 02 and$0x2,%ecx 2b:* 48 89 42 08 mov%rax,0x8(%rdx) -- trapping instruction 2f: 48 89 10 mov%rdx,(%rax) 32: 48 b8 00 02 20 00 00 movabs $0xdead00200200,%rax where that constant is LIST_POISON2 and the and $2 seems to be TIMER_IRQSAFE. So the trapping instruction *looks* like it's doing __list_del() on the timer, and timer-next is NULL. So somebody added a timer, and then deallocated/cleared the structure before it triggered. The problem is, I can't see a way to figure out _who_ did that. I think CONFIG_DEBUG_OBJECTS_TIMERS=y should be able to detect that? Debugobjects hooks into deallocation paths and complains immediately if a live timer is zapped that way. If the corrupion does not involve deallocation then it might be more difficult to detect but not impossible either: for example if an object is not freed but reused incorrectly then a repeat use of any timer function will cause the debugobjects (and/or the timer code) to complain. So I'd suggest trying debugobjects, it should catch a fair number of non-exotic object corruption patterns. Thanks, Ingo -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 08, 2013 at 10:14:52AM +0800, Fengguang Wu wrote:
[2.785188] kobject: 'drm' (880006dae048): kobject_release, parent
88189648 (delayed)
[2.787362] kobject: 'drm' (880006dafe00): kobject_release, parent
(null) (delayed)
[2.789674] [drm] radeon kernel modesetting enabled.
[2.791798] [drm:drm_proc_init] *ERROR* Cannot create /proc/dri/0
[2.793280] [drm:drm_get_minor] *ERROR* DRM: Failed to initialize
/proc/dri.
[2.795591] kobject: 'controlD64' (880006dc3820): kobject_release,
parent (null) (delayed)
[2.797988] cirrus: probe of :00:02.0 failed with error -1
[2.799675] usbcore: registered new interface driver udl
[2.802167] parport_pc 00:04: reported by Plug and Play ACPI
[2.803818] parport0: PC-style at 0x378, irq 7 [PCSPP(,...)]
[2.806035] kobject: 'parport_pc.956' (880006dc3820): kobject_release,
parent (null) (delayed)
Look very carefully at the above addresses of the controlD64 object and
the parport_pc.956 object. They're both the same - and the first hasn't
been run yet. It's a double addition because the culpret is not parport_pc,
but is controlD64 instead.
And here we have it:
int drm_put_minor(struct drm_minor **minor_p)
{
...
drm_sysfs_device_remove(minor);
...
kfree(minor);
}
void drm_sysfs_device_remove(struct drm_minor *minor)
{
if (minor-kdev.parent)
device_unregister(minor-kdev);
minor-kdev.parent = NULL;
}
I think David Arlie also needs a quiet talking to about how to use the
device model:
int drm_sysfs_device_add(struct drm_minor *minor)
{
minor-kdev.release = drm_sysfs_device_release;
...
err = device_register(minor-kdev);
}
static void drm_sysfs_device_release(struct device *dev)
{
memset(dev, 0, sizeof(struct device));
return;
}
Since when has that been acceptable in a release function?
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 08, 2013 at 09:58:16AM +0200, Ingo Molnar wrote: * Linus Torvalds torva...@linux-foundation.org wrote: On Mon, Oct 7, 2013 at 1:35 AM, Fengguang Wu fengguang...@intel.com wrote: On Mon, Oct 07, 2013 at 01:12:17AM -0700, Linus Torvalds wrote: My pleasure! Here are 100 randomly selected call traces. Also attached several full dmesgs and the kconfig. Ok, they may be randomly selected, but they are all the same. Which is good, I guess, we're only talking about one bug. Anyway, they all have RIP:run_timer_softirq+0x12c/0x1b8, and the code is 0: 8b 65 c8 mov-0x38(%rbp),%esp 3: 4d 39 ec cmp%r13,%r12 6: 0f 84 2f ff ff ff je 0xff3b c: 41 8b 4c 24 18 mov0x18(%r12),%ecx 11: 4d 8b 74 24 20 mov0x20(%r12),%r14 16: 4d 8b 7c 24 28 mov0x28(%r12),%r15 1b: 4c 89 63 38 mov%r12,0x38(%rbx) 1f: 49 8b 44 24 08 mov0x8(%r12),%rax 24: 49 8b 14 24 mov(%r12),%rdx 28: 83 e1 02 and$0x2,%ecx 2b:* 48 89 42 08 mov%rax,0x8(%rdx) -- trapping instruction 2f: 48 89 10 mov%rdx,(%rax) 32: 48 b8 00 02 20 00 00 movabs $0xdead00200200,%rax where that constant is LIST_POISON2 and the and $2 seems to be TIMER_IRQSAFE. So the trapping instruction *looks* like it's doing __list_del() on the timer, and timer-next is NULL. So somebody added a timer, and then deallocated/cleared the structure before it triggered. The problem is, I can't see a way to figure out _who_ did that. I think CONFIG_DEBUG_OBJECTS_TIMERS=y should be able to detect that? It did help expose more information, and earlier. w/o debugobjects, we hit the BUG: ... directly: [2.964097] BUG: unable to handle kernel NULL pointer dereference at 0008 [2.96] IP: [81098f60] run_timer_softirq+0x126/0x1da [2.968060] PGD 0 [2.968060] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC [2.968060] CPU: 0 PID: 95 Comm: kworker/0:2 Not tainted 3.11.0-rc2-00010-gc817a67-dirty #5 [2.968060] Workqueue: events flush_to_ldisc [2.968060] task: 8800068544c0 ti: 880006856000 task.ti: 880006856000 [2.968060] RIP: 0010:[81098f60] [81098f60] run_timer_softirq+0x126/0x1da After enabling CONFIG_DEBUG_OBJECTS_TIMERS=y, it will issue a WARNING followed by a BUG: ... [2.802167] parport_pc 00:04: reported by Plug and Play ACPI [2.803818] parport0: PC-style at 0x378, irq 7 [PCSPP(,...)] [2.806035] kobject: 'parport_pc.956' (880006dc3820): kobject_release, parent (null) (delayed) [2.808626] [ cut here ] [2.809776] WARNING: CPU: 1 PID: 1 at /c/wfg/linux/lib/debugobjects.c:260 debug_print_object+0x7c/0x8d() [2.812433] ODEBUG: init active (active state 0) object type: timer_list hint: (null) .. [3.796079] BUG: unable to handle kernel NULL pointer dereference at (null) Debugobjects hooks into deallocation paths and complains immediately if a live timer is zapped that way. If the corrupion does not involve deallocation then it might be more difficult to detect but not impossible either: for example if an object is not freed but reused incorrectly then a repeat use of any timer function will cause the debugobjects (and/or the timer code) to complain. So I'd suggest trying debugobjects, it should catch a fair number of non-exotic object corruption patterns. Good to know that, thanks for the info! Regards, Fengguang -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
* Fengguang Wu fengguang...@intel.com wrote: After enabling CONFIG_DEBUG_OBJECTS_TIMERS=y, it will issue a WARNING followed by a BUG: ... Cool! [2.802167] parport_pc 00:04: reported by Plug and Play ACPI [2.803818] parport0: PC-style at 0x378, irq 7 [PCSPP(,...)] [2.806035] kobject: 'parport_pc.956' (880006dc3820): kobject_release, parent (null) (delayed) [2.808626] [ cut here ] [2.809776] WARNING: CPU: 1 PID: 1 at /c/wfg/linux/lib/debugobjects.c:260 debug_print_object+0x7c/0x8d() [2.812433] ODEBUG: init active (active state 0) object type: timer_list hint: (null) .. [3.796079] BUG: unable to handle kernel NULL pointer dereference at (null) Mind posting the full backtrace? It should show the actual callsite that zaps the timer. I don't think we got that dump so far, I've only seen timer list corruption backtraces. (but I haven't read the whole thread.) Thanks, Ingo -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
* Ingo Molnar mi...@kernel.org wrote: * Fengguang Wu fengguang...@intel.com wrote: After enabling CONFIG_DEBUG_OBJECTS_TIMERS=y, it will issue a WARNING followed by a BUG: ... Cool! [2.802167] parport_pc 00:04: reported by Plug and Play ACPI [2.803818] parport0: PC-style at 0x378, irq 7 [PCSPP(,...)] [2.806035] kobject: 'parport_pc.956' (880006dc3820): kobject_release, parent (null) (delayed) [2.808626] [ cut here ] [2.809776] WARNING: CPU: 1 PID: 1 at /c/wfg/linux/lib/debugobjects.c:260 debug_print_object+0x7c/0x8d() [2.812433] ODEBUG: init active (active state 0) object type: timer_list hint: (null) .. [3.796079] BUG: unable to handle kernel NULL pointer dereference at (null) Mind posting the full backtrace? It should show the actual callsite that zaps the timer. I don't think we got that dump so far, I've only seen timer list corruption backtraces. (but I haven't read the whole thread.) I see you already posted the dmesg in another part of this thread, so the relevant call-trace for parport_pc.c is: [2.812579] Call Trace: [2.812579] [8184ea6e] dump_stack+0x4f/0x84 [2.812579] [8108f38d] warn_slowpath_common+0x72/0x8c [2.812579] [81223681] ? debug_print_object+0x7c/0x8d [2.812579] [8108f40b] warn_slowpath_fmt+0x41/0x43 [2.812579] [81223681] debug_print_object+0x7c/0x8d [2.812579] [812239e0] __debug_object_init+0x27c/0x2c6 [2.812579] [81223a1b] ? __debug_object_init+0x2b7/0x2c6 [2.812579] [81223a3e] debug_object_init+0x14/0x16 [2.812579] [810992e8] init_timer_key+0x23/0x65 [2.812579] [8121352f] kobject_release+0x90/0xba [2.812579] [812137ac] kobject_put+0x4d/0x51 [2.812579] [81482cfb] put_device+0x12/0x14 [2.812579] [814873a2] platform_device_put+0x12/0x14 [2.812579] [81487727] platform_device_unregister+0x16/0x1a [2.812579] [814810f9] parport_pc_probe_port+0x7c4/0x7d9 [2.812579] [81e36c75] parport_pc_init+0x2b0/0x317 [2.812579] [81e369c5] ? parport_setup+0x147/0x147 [2.812579] [81e11e0c] do_one_initcall+0x8e/0x132 [2.812579] [810a7000] ? param_array_set+0x7f/0xf2 [2.812579] [810a7275] ? parse_args+0x1ad/0x26c [2.812579] [81e11fe2] kernel_init_freeable+0x132/0x1bc [2.812579] [81e116e3] ? do_early_param+0x86/0x86 [2.812579] [818488bf] ? rest_init+0xc3/0xc3 [2.812579] [818488c8] kernel_init+0x9/0xcc [2.812579] [81859e3c] ret_from_fork+0x7c/0xb0 [2.812579] [818488bf] ? rest_init+0xc3/0xc3 [2.812579] ---[ end trace ce7dd707bef7dc3e ]--- It appears a delayed callback is already active for that platform device? Looks weird, because in theory platform devices are allocated anew in parport_pc_probe_port(): pdev = platform_device_register_simple(parport_pc, base, NULL, 0); if (IS_ERR(pdev)) return NULL; dev = pdev-dev; Which is allocated via zalloc(), so no reuse. and then released if the hardware port is not present: out1: if (pdev) platform_device_unregister(pdev); Thanks, Ingo -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
Hi Russell, I'm now trying to disable all drivers shows up in the kobject_release messages: [2.756392] kobject: 'ipmi_si' (880007764a00): kobject_release, parent 881b7648 (delayed) [2.758091] kobject: 'ipmi_si' (880007764800): kobject_release, parent 88221248 (delayed) [2.759858] kobject: 'ipmi_si' (880007764c00): kobject_release, parent 88189248 (delayed) [2.929669] kobject: 'drm' (880006db2848): kobject_release, parent 88189648 (delayed) [2.932143] kobject: 'drm' (880006daf000): kobject_release, parent (null) (delayed) [2.941844] kobject: 'controlD64' (880006db2020): kobject_release, parent (null) (delayed) [2.958432] kobject: 'parport_pc.956' (880006db2020): kobject_release, parent (null) (delayed) [2.965698] kobject: 'parport_pc.888' (880006dc5820): kobject_release, parent (null) (delayed) [2.972583] kobject: 'parport_pc.632' (880006dc5020): kobject_release, parent (null) (delayed) [3.031704] kobject: 'physmap-flash' (880006ddc800): kobject_release, parent 88189248 (delayed) [3.055119] kobject: 'docg3' (880006de3c00): kobject_release, parent 88189248 (delayed) [3.496256] kobject: 'gpio-vbus' (880006817400): kobject_release, parent 88189248 (delayed) [3.619023] kobject: '(null)' (88000777baf0): kobject_release, parent (null) (delayed) [3.657587] kobject: 'proc-osm' (88000684be00): kobject_release, parent 880006849848 (delayed) [3.662546] kobject: 'mc13xxx-rtc' (880006851e00): kobject_release, parent 88189248 (delayed) [3.669144] kobject: 'rtc-msm6242' (880006851c00): kobject_release, parent 88189248 (delayed) [3.677494] kobject: 'pcap-rtc' (880006851a00): kobject_release, parent 88189248 (delayed) [3.680280] kobject: 'rtc-rp5c01' (880006855a00): kobject_release, parent 88189248 (delayed) [3.750200] kobject: 'mc13783-adc' (880007707000): kobject_release, parent 88189248 (delayed) I find the above debug messages very helpful in locating the buggy driver. How about enabling it whenever CONFIG_DEBUG_KOBJECT_RELEASE is enabled? Something like #ifdef CONFIG_DEBUG_KOBJECT_RELEASE - pr_debug(kobject: '%s' (%p): %s, parent %p (delayed)\n, + printk(KERN_INFO kobject: '%s' (%p): %s, parent %p (delayed)\n, kobject_name(kobj), kobj, __func__, kobj-parent); pr_debug() won't be displayed by default, and it depends on CONFIG_DYNAMIC_DEBUG. with some manual bisects, I find a good config (attached) that can reliably boot the kernel up. Based on that config, I tried adding parport_pc and see that it still boots fine. Adding drm, however will bring back the oops. Will try a kernel based on the original kconfig with drm disabled only. Thanks, Fengguang -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 08, 2013 at 08:17:42PM +0800, Fengguang Wu wrote: I find the above debug messages very helpful in locating the buggy driver. How about enabling it whenever CONFIG_DEBUG_KOBJECT_RELEASE is enabled? Something like #ifdef CONFIG_DEBUG_KOBJECT_RELEASE - pr_debug(kobject: '%s' (%p): %s, parent %p (delayed)\n, + printk(KERN_INFO kobject: '%s' (%p): %s, parent %p (delayed)\n, kobject_name(kobj), kobj, __func__, kobj-parent); pr_debug() won't be displayed by default, and it depends on CONFIG_DYNAMIC_DEBUG. Please consider using pr_info() instead of printk(KERN_INFO - it's slightly less typing. I can see that being a useful change while we have these problematical instances to track down, but I do wonder whether it'll make the thing too noisy. Does anyone have other opinions on this point? Linus? Greg? -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 08, 2013 at 11:14:17PM +0100, Russell King - ARM Linux wrote: On Tue, Oct 08, 2013 at 08:17:42PM +0800, Fengguang Wu wrote: I find the above debug messages very helpful in locating the buggy driver. How about enabling it whenever CONFIG_DEBUG_KOBJECT_RELEASE is enabled? Something like #ifdef CONFIG_DEBUG_KOBJECT_RELEASE - pr_debug(kobject: '%s' (%p): %s, parent %p (delayed)\n, + printk(KERN_INFO kobject: '%s' (%p): %s, parent %p (delayed)\n, kobject_name(kobj), kobj, __func__, kobj-parent); pr_debug() won't be displayed by default, and it depends on CONFIG_DYNAMIC_DEBUG. Please consider using pr_info() instead of printk(KERN_INFO - it's slightly less typing. I can see that being a useful change while we have these problematical instances to track down, but I do wonder whether it'll make the thing too noisy. Does anyone have other opinions on this point? Linus? Greg? It's going to make things really noisy at boot time, but then it should settle down and not be bad at all. Let's try it and see if it helps or not. thanks, greg k-h -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 8, 2013 at 3:48 PM, Greg Kroah-Hartman gre...@linuxfoundation.org wrote: It's going to make things really noisy at boot time, but then it should settle down and not be bad at all. Let's try it and see if it helps or not. Yeah. And quite frankly, normally that whole DEBUG_KOBJECT_RELEASE thing is hopefully only enabled in debug kernels (like maybe the Fedora rawhide one, or at developers), so being a bit more verbose is likely ok. Linus -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 08, 2013 at 03:48:40PM -0700, Greg KH wrote: On Tue, Oct 08, 2013 at 11:14:17PM +0100, Russell King - ARM Linux wrote: On Tue, Oct 08, 2013 at 08:17:42PM +0800, Fengguang Wu wrote: I find the above debug messages very helpful in locating the buggy driver. How about enabling it whenever CONFIG_DEBUG_KOBJECT_RELEASE is enabled? Something like #ifdef CONFIG_DEBUG_KOBJECT_RELEASE - pr_debug(kobject: '%s' (%p): %s, parent %p (delayed)\n, + printk(KERN_INFO kobject: '%s' (%p): %s, parent %p (delayed)\n, kobject_name(kobj), kobj, __func__, kobj-parent); pr_debug() won't be displayed by default, and it depends on CONFIG_DYNAMIC_DEBUG. Please consider using pr_info() instead of printk(KERN_INFO - it's slightly less typing. Good suggestion! I can see that being a useful change while we have these problematical instances to track down, but I do wonder whether it'll make the thing too noisy. Does anyone have other opinions on this point? Linus? Greg? It's going to make things really noisy at boot time, but then it should settle down and not be bad at all. FYI, in the randconfig kernel involved in this thread, it will emit about 20 lines of dmesg. Let's try it and see if it helps or not. OK, I'll submit a patch. These messages would allow me to do a statistic analyze of similar bugs: since I'm testing 100+ new randconfigs every day, some of them will include the buggy drivers and some not, we can collect all these lines [2.929669] kobject: 'drm' (880006db2848): kobject_release, parent 88189648 (delayed) ... [3.750200] kobject: 'mc13783-adc' (880007707000): kobject_release, parent 88189248 (delayed) , count how many times each one shows up in the GOOD kernel boots and how many show up in the BAD boots. Then we may be able to learn which drivers are likely buggy and should be manually checked. Thanks, Fengguang -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Tue, Oct 08, 2013 at 05:45:37PM -0700, Linus Torvalds wrote: normally that whole DEBUG_KOBJECT_RELEASE thing is hopefully only enabled in debug kernels (like maybe the Fedora rawhide one Nope. After spending a couple of days fruitlessly trying to get my machine to boot with it enabled, I think we decided it's not worth the pain. Debugging hangs instantly and silently on boot bugs are hard enough when you're in front of the machine, doing so via bugzilla isn't something I'd wish on my worst enemy. Dave -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 07, 2013 at 08:29:26PM -0700, Linus Torvalds wrote: > On Mon, Oct 7, 2013 at 8:11 PM, Fengguang Wu wrote: > > > > Yeah, I see no timer usage in parport_pc driver, so it's still questionable. > > The timer itself comes simply from the delayed_work that is used to > delay the freeing of the kobject. > > So that is not the surprising part. OK. > The surprising part is that I don't see parport_pc doing anything > odd/bad with its kobject embedded in the 'struct dev'. It seems to > just do a platform_device_register_simple() followed by a > platform_device_unregister(). > > At least that's true for the normal parport_pc_probe_port() case that > just passes in a NULL dev... But I only glanced at the driver, so I > might have missed something. > > > with some manual bisects, I find a good config (attached) that can > > reliably boot the kernel up. > > > > Based on that config, I tried adding parport_pc and see that it still > > boots fine. > > > > Adding drm, however will bring back the oops. Will try a kernel based > > on the original kconfig with drm disabled only. FYI I just confirmed that the original bad kconfig can be made bootable by simply disabling CONFIG_DRM. > Ok. The list corruption (which also pointed at parport_pc) might well > be corrupted by removing the entries before or after the parport_pc, > and moving the corruption to parport_pc that way (through the > "prev->next = next" thing in list handling). So maybe it was something > else all along. You could enable CONFIG_DEBUG_LIST to see if that > triggers some dump earlier.. OK, I'll try enabling CONFIG_DEBUG_LIST. Thanks, Fengguang -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 7, 2013 at 8:11 PM, Fengguang Wu wrote: > > Yeah, I see no timer usage in parport_pc driver, so it's still questionable. The timer itself comes simply from the delayed_work that is used to delay the freeing of the kobject. So that is not the surprising part. The surprising part is that I don't see parport_pc doing anything odd/bad with its kobject embedded in the 'struct dev'. It seems to just do a platform_device_register_simple() followed by a platform_device_unregister(). At least that's true for the normal parport_pc_probe_port() case that just passes in a NULL dev... But I only glanced at the driver, so I might have missed something. > with some manual bisects, I find a good config (attached) that can > reliably boot the kernel up. > > Based on that config, I tried adding parport_pc and see that it still > boots fine. > > Adding drm, however will bring back the oops. Will try a kernel based > on the original kconfig with drm disabled only. Ok. The list corruption (which also pointed at parport_pc) might well be corrupted by removing the entries before or after the parport_pc, and moving the corruption to parport_pc that way (through the "prev->next = next" thing in list handling). So maybe it was something else all along. You could enable CONFIG_DEBUG_LIST to see if that triggers some dump earlier.. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
I'll catch up with your emails if it kills me.. On Mon, Oct 7, 2013 at 7:36 PM, Fengguang Wu wrote: > On Tue, Oct 08, 2013 at 10:14:52AM +0800, Fengguang Wu wrote: > > Disabled PARPORT_PC: > > # CONFIG_PARPORT is not set > > and retest shows another call trace. Here are two run logs: Ugh. Ok. So something else is wrong too. > [4.074903] registered taskstats version 1 > [4.077382] rtc-test rtc-test.0: setting system clock to 2013-09-09 > 06:44:08 UTC (1378709048) > [4.081688] debug: unmapping init [mem > 0x81df2000-0x81ec3fff] > /bin/sh: /proc/self/fd/9: No such file or directory > [4.148224] BUG: unable to handle kernel NULL pointer dereference at > (null) > [4.150690] IP: [< (null)>] (null) > [4.152050] PGD 6a79067 PUD 69d3067 PMD 0 > [4.152125] Oops: 0010 [#1] PREEMPT SMP DEBUG_PAGEALLOC > [4.152125] CPU: 0 PID: 130 Comm: sh Not tainted > 3.11.0-rc2-00010-gc817a67-dirty #7 > [4.152125] task: 880006a5e980 ti: 880006a3e000 task.ti: > 880006a3e000 > [4.152125] RIP: 0010:[<>] [< (null)>] > (null) > [4.152125] RSP: 0018:88000dc03e40 EFLAGS: 00010286 > [4.152125] RAX: 0001 RBX: 0102 RCX: > > [4.152125] RDX: 0f13 RSI: RDI: > > [4.152125] RBP: 88000dc03e98 R08: 160f R09: > 88000dc0a000 > [4.152125] R10: 88000dc0a000 R11: 880006a5e980 R12: > 880006a3ffd8 > [4.152125] R13: R14: R15: > 81f347d8 > [4.152125] FS: 7fe94d7b4700() GS:88000dc0() > knlGS: > [4.152125] CS: 0010 DS: ES: CR0: 80050033 > [4.152125] CR2: CR3: 06a49000 CR4: > 06b0 > [4.152125] Stack: > [4.152125] 810989e6 8109897b 810c73b8 > 88000dc03e58 > [4.152125] 821cbe60 > 81f33780 > [4.152125] 88000dc03ec8 81f34fd8 81f34bd8 > 88000dc03f08 > [4.152125] Call Trace: > [4.152125] > [4.152125] [] ? call_timer_fn+0x6b/0xde > [4.152125] [] ? detach_timer+0x46/0x46 > [4.152125] [] ? trace_hardirqs_on_caller+0x13a/0x18b > [4.152125] [] run_timer_softirq+0x187/0x1cf > [4.152125] [] __do_softirq+0xc6/0x18f > [4.152125] [] ? unlazy_walk+0x125/0x172 > [4.152125] [] irq_exit+0x58/0x9e > [4.152125] [] smp_apic_timer_interrupt+0x31/0x3e > [4.152125] [] apic_timer_interrupt+0x72/0x80 > [4.152125] > [4.152125] [] ? arch_local_irq_restore+0x6/0xd > [4.152125] [] lock_release+0x16e/0x17a > [4.152125] [] _raw_spin_unlock+0x1b/0x4f > [4.152125] [] unlazy_walk+0x125/0x172 > [4.152125] [] do_last+0x6cc/0x9b5 > [4.152125] [] ? inode_permission+0x3d/0x3f > [4.152125] [] path_openat+0x22a/0x48d > [4.152125] [] do_filp_open+0x35/0x85 > [4.152125] [] ? __alloc_fd+0xd8/0xea > [4.152125] [] do_sys_open+0x6b/0xfd > [4.152125] [] ? trace_hardirqs_on_caller+0x154/0x18b > [4.152125] [] SyS_open+0x19/0x1b > [4.152125] [] system_call_fastpath+0x16/0x1b > [4.152125] Code: Bad RIP value. And this one doesn't leave any hint of where it came from. Damn. The register contents aren't being very helpful either. The second one is the same thing.. And equally unhelpful as far as I can tell. I think we need more powerful debugging aids to make these useful. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 7, 2013 at 7:14 PM, Fengguang Wu wrote: > > I got a call trace containing parport_pc_probe_port() (is it the > culprit?) after recompiling kernel with > > CONFIG_DEBUG_OBJECTS_TIMERS=y > > and booting with "ignore_loglevel". Here is the log for two kernel boots. Ok, so it's definitely parport_pc. I don't see what it would do wrong, though. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 7, 2013 at 7:09 PM, Fengguang Wu wrote: > > Thanks for the hints! I run a kernel with pr_alert() for several times > and here is the screen log. Note that this kernel is compiled with gcc > 4.6.3 and the decoded code looks different than gcc 4.8.1 Ok, I think we have something. > NULL pointer dereference at 0008 > [3.901119] IP: [] run_timer_softirq+0x126/0x1da > [3.901119] PGD 0 > [3.901119] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC > [3.901119] CPU: 1 PID: 22 Comm: kworker/1:1 Not tainted > 3.11.0-rc2-00010-gc817a67-dirty #5 > [3.901119] Workqueue: events power_supply_changed_work > [3.901119] task: 882dc5c0 ti: 882de000 task.ti: > 882de000 > [3.901119] RIP: 0010:[] [] > run_timer_softirq+0x126/0x1da > [3.901119] RSP: :88000dd03ea8 EFLAGS: 00010002 > [3.901119] RAX: 880006dc28a0 RBX: 8815c000 RCX: > 88000dd03ec8 > [3.901119] RDX: 880006dc2860 RSI: 810a1b7b RDI: > > [3.901119] RBP: 88000dd03f08 R08: 0002 R09: > dead00200200 > [3.901119] R10: 8815c018 R11: 882dc5c0 R12: > 88000dd03ec8 > [3.901119] R13: 8815d858 R14: 8815d458 R15: > 8815d058 > [3.901119] FS: () GS:88000dd0() > knlGS: > [3.901119] CS: 0010 DS: ES: CR0: 8005003b > [3.901119] CR2: 0008 CR3: 01c47000 CR4: > 06a0 > [3.901119] Stack: > [3.901119] 88000dd0e1b0 88000dd0e170 810a1b7b > 00c0 > [3.901119] 880006dc28a0 88000dd0e300 88000dd03ef8 > 882dffd8 > [3.901119] 81c43088 1100 0141 > 0001 > [3.901119] Call Trace: > [3.901119] > [3.901119] [] ? __queue_work+0x1ee/0x1ee > [3.901119] [] __do_softirq+0xc6/0x18f > [3.901119] [] irq_exit+0x58/0x9e > [3.901119] [] smp_apic_timer_interrupt+0x31/0x3e > [3.901119] [] apic_timer_interrupt+0x72/0x80 > [3.901119] > [3.901119] [] ? retint_restore_args+0x13/0x13 > [3.901119] [] ? arch_local_irq_enable+0xb/0xd > [3.901119] [] ? trace_hardirqs_on+0xd/0xf > [3.901119] [] preempt_schedule_irq+0x36/0x5a > [3.901119] [] retint_kernel+0x26/0x30 > [3.901119] [] ? arch_local_irq_restore+0x6/0xd > [3.901119] [] vprintk_emit+0x3f9/0x422 > [3.901119] [] ? snprintf+0x34/0x36 > [3.901119] [] dev_vprintk_emit+0x1a3/0x1c9 > [3.901119] [] ? string.isra.3+0x3d/0xa2 > [3.901119] [] dev_printk_emit+0x34/0x36 > [3.901119] [] __dev_printk+0x71/0x78 > [3.901119] [] dev_printk+0x40/0x42 > [3.901119] [] ? __kmalloc+0x93/0xb8 > [3.901119] [] power_supply_uevent+0x183/0x1ec > [3.901119] [] dev_uevent+0x179/0x1a2 > [3.901119] [] ? kzalloc+0xf/0x11 > [3.901119] [] kobject_uevent_env+0x195/0x414 > [3.901119] [] kobject_uevent+0xb/0xd > [3.901119] [] power_supply_changed_work+0x60/0x65 > [3.901119] [] process_one_work+0x1de/0x2fd > [3.901119] [] ? process_one_work+0x174/0x2fd > [3.901119] [] worker_thread+0x159/0x1ee > [3.901119] [] ? rescuer_thread+0x27f/0x27f > [3.901119] [] kthread+0xac/0xb4 > [3.901119] [] ? __kthread_parkme+0x68/0x68 > [3.901119] [] ret_from_fork+0x7c/0xb0 > [3.901119] [] ? __kthread_parkme+0x68/0x68 > [3.901119] Code: 08 e9 99 00 00 00 4c 8b 40 18 48 8b 70 20 49 b9 00 02 20 > 00 00 00 ad de 48 8b 50 28 48 89 43 38 48 8b 38 48 8b 48 08 41 83 e0 02 <48> > 89 4f 08 48 89 39 f6 40 18 01 48 c7 00 00 00 00 00 4c 89 48 > [3.901119] RIP [] run_timer_softirq+0x126/0x1da So this is the same bug, now the code decodes to 5: 4c 8b 40 18 mov0x18(%rax),%r8 9: 48 8b 70 20 mov0x20(%rax),%rsi d: 49 b9 00 02 20 00 00 movabs $0xdead00200200,%r9 14: 00 ad de 17: 48 8b 50 28 mov0x28(%rax),%rdx 1b: 48 89 43 38 mov%rax,0x38(%rbx) 1f: 48 8b 38 mov(%rax),%rdi 22: 48 8b 48 08 mov0x8(%rax),%rcx 26: 41 83 e0 02 and$0x2,%r8d 2a:* 48 89 4f 08 mov%rcx,0x8(%rdi) <-- trapping instruction 2e: 48 89 39 mov%rdi,(%rcx) 31: f6 40 18 01 testb $0x1,0x18(%rax) 35: 48 c7 00 00 00 00 00 movq $0x0,(%rax) so the NULL pointer in %rdi (which _should_ be a list pointer) got loaded from %rax. So the prime suspect is an allocation near RAX: 880006dc28a0. and looking backwards in your dump, we have [2.807742] kobject: 'parport_pc.888' (880006dc2020): kobject_release, parent (null) (delayed) [2.810494] kobject: 'parport_pc.632' (880006dca820): kobject_release, parent (null) (delayed) so it looks like that "parparport_pc.632" thing. Another one, same code: > [3.992074] BUG: unable to handle kernel NULL
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 7, 2013 at 3:14 PM, Linus Torvalds wrote: > > I *think* r14 contains the function we're going to jump to in the > oops, and that could be interesting to know, but it's not decoded, so > you'd have to match it up against a symbol map... Actually, Fenguguan, never mind. Instead, change the "pr_debug()" in kobject_release() to "pr_alert()", so that it gets printed out. Or just boot with the "ignore_loglevel" thing so that debug messages are actually visible. At that point, we should be able to match the oops workqueue list address with the address of the delayed kernel object that gets printed out. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 07, 2013 at 11:29:25PM +0100, Russell King - ARM Linux wrote: > However, due to the problems with x86, that's fallen on its head and I > have no solution to get better debugging out which works across all > architectures. I'm stumpted by this. The final attempt at trying to sort this may be to try allocating a struct at release time to contain a pointer to the kobject and the delayed work queue, and maybe a stack trace. The thing I worry about is whether we can allocate memory at that point. GFP_ATOMIC maybe? Should be a small enough structure that it shouldn't eat too much into the reserved pools. I won't be coding it up tonight though! -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 07, 2013 at 03:14:48PM -0700, Linus Torvalds wrote: > On Mon, Oct 7, 2013 at 1:35 AM, Fengguang Wu wrote: > > On Mon, Oct 07, 2013 at 01:12:17AM -0700, Linus Torvalds wrote: > > > > My pleasure! Here are 100 randomly selected call traces. Also attached > > several full dmesgs and the kconfig. > > Ok, they may be randomly selected, but they are all the same. Which is > good, I guess, we're only talking about one bug. > > Anyway, they all have RIP:run_timer_softirq+0x12c/0x1b8, and the code is > >0: 8b 65 c8 mov-0x38(%rbp),%esp >3: 4d 39 ec cmp%r13,%r12 >6: 0f 84 2f ff ff ff je 0xff3b >c: 41 8b 4c 24 18 mov0x18(%r12),%ecx > 11: 4d 8b 74 24 20 mov0x20(%r12),%r14 > 16: 4d 8b 7c 24 28 mov0x28(%r12),%r15 > 1b: 4c 89 63 38 mov%r12,0x38(%rbx) > 1f: 49 8b 44 24 08 mov0x8(%r12),%rax > 24: 49 8b 14 24 mov(%r12),%rdx > 28: 83 e1 02 and$0x2,%ecx > 2b:* 48 89 42 08 mov%rax,0x8(%rdx) <-- trapping instruction > 2f: 48 89 10 mov%rdx,(%rax) > 32: 48 b8 00 02 20 00 00 movabs $0xdead00200200,%rax > > where that constant is LIST_POISON2 and the "and $2" seems to be > TIMER_IRQSAFE. So the trapping instruction *looks* like it's doing > __list_del() on the timer, and timer->next is NULL. > > So somebody added a timer, and then deallocated/cleared the structure > before it triggered. The problem is, I can't see a way to figure out > _who_ did that. > > I *think* r14 contains the function we're going to jump to in the > oops, and that could be interesting to know, but it's not decoded, so > you'd have to match it up against a symbol map... As with all of these, it will be a kobject, prompted by my delayed kobject release - we embed a delayed work structure in the kobject so that we can call the cleanup and detect if it was freed. However, early on, after Greg merged it, the problems with x86 were reported, and I tried all sorts of ways to avoid this. I tried allocating it separately, but that doesn't work because x86 registers kobjects really early. I tried a few other things as well. The idea of allocating the delayed work separately is that it doesn't get freed along with the kobject, and then we can start tracking more reportable state and/or tie it up with other kobject debug. However, due to the problems with x86, that's fallen on its head and I have no solution to get better debugging out which works across all architectures. I'm stumpted by this. However, one thing that this patch _is_ doing is it is uncovering the fact that the kernel is full of kobject refcount problems, and it seems many people just get this stuff wrong. That in itself is quite a problem. What I would say is that we should have had this delayed release either as standard in the kobject system from the start, or as a debug thing to stop these problems as soon as they were initially introduced. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 7, 2013 at 1:35 AM, Fengguang Wu wrote: > On Mon, Oct 07, 2013 at 01:12:17AM -0700, Linus Torvalds wrote: > > My pleasure! Here are 100 randomly selected call traces. Also attached > several full dmesgs and the kconfig. Ok, they may be randomly selected, but they are all the same. Which is good, I guess, we're only talking about one bug. Anyway, they all have RIP:run_timer_softirq+0x12c/0x1b8, and the code is 0: 8b 65 c8 mov-0x38(%rbp),%esp 3: 4d 39 ec cmp%r13,%r12 6: 0f 84 2f ff ff ff je 0xff3b c: 41 8b 4c 24 18 mov0x18(%r12),%ecx 11: 4d 8b 74 24 20 mov0x20(%r12),%r14 16: 4d 8b 7c 24 28 mov0x28(%r12),%r15 1b: 4c 89 63 38 mov%r12,0x38(%rbx) 1f: 49 8b 44 24 08 mov0x8(%r12),%rax 24: 49 8b 14 24 mov(%r12),%rdx 28: 83 e1 02 and$0x2,%ecx 2b:* 48 89 42 08 mov%rax,0x8(%rdx) <-- trapping instruction 2f: 48 89 10 mov%rdx,(%rax) 32: 48 b8 00 02 20 00 00 movabs $0xdead00200200,%rax where that constant is LIST_POISON2 and the "and $2" seems to be TIMER_IRQSAFE. So the trapping instruction *looks* like it's doing __list_del() on the timer, and timer->next is NULL. So somebody added a timer, and then deallocated/cleared the structure before it triggered. The problem is, I can't see a way to figure out _who_ did that. I *think* r14 contains the function we're going to jump to in the oops, and that could be interesting to know, but it's not decoded, so you'd have to match it up against a symbol map... Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Sun, Oct 6, 2013 at 10:10 PM, Fengguang Wu wrote:
>
> I retried bisect with "Oops:" and the first bad commit is
>
> commit c817a67ecba7c3 ("kobject: delayed kobject release: help find buggy
> drivers")
Ok, that makes way more sense.
> That commit has already helped expose some bugs, however I suspect there are
> still many hidden ones. In this particular bisect, the commit produces 85 good
> dmesgs and 2055 bad dmesgs, exposing all sorts of error messages
>
>1929 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
>1921 BUG: unable to handle kernel NULL pointer dereference at
> 0008
>1897 Kernel panic - not syncing: Fatal exception in interrupt
..
Ok, can you post some of the more promising oopses so that we can try
to figure out what kobject it is that causes problems?
Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Sun, Oct 6, 2013 at 10:10 PM, Fengguang Wu fengguang...@intel.com wrote: I retried bisect with Oops: and the first bad commit is commit c817a67ecba7c3 (kobject: delayed kobject release: help find buggy drivers) Ok, that makes way more sense. That commit has already helped expose some bugs, however I suspect there are still many hidden ones. In this particular bisect, the commit produces 85 good dmesgs and 2055 bad dmesgs, exposing all sorts of error messages 1929 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC 1921 BUG: unable to handle kernel NULL pointer dereference at 0008 1897 Kernel panic - not syncing: Fatal exception in interrupt .. Ok, can you post some of the more promising oopses so that we can try to figure out what kobject it is that causes problems? Linus -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 7, 2013 at 1:35 AM, Fengguang Wu fengguang...@intel.com wrote: On Mon, Oct 07, 2013 at 01:12:17AM -0700, Linus Torvalds wrote: My pleasure! Here are 100 randomly selected call traces. Also attached several full dmesgs and the kconfig. Ok, they may be randomly selected, but they are all the same. Which is good, I guess, we're only talking about one bug. Anyway, they all have RIP:run_timer_softirq+0x12c/0x1b8, and the code is 0: 8b 65 c8 mov-0x38(%rbp),%esp 3: 4d 39 ec cmp%r13,%r12 6: 0f 84 2f ff ff ff je 0xff3b c: 41 8b 4c 24 18 mov0x18(%r12),%ecx 11: 4d 8b 74 24 20 mov0x20(%r12),%r14 16: 4d 8b 7c 24 28 mov0x28(%r12),%r15 1b: 4c 89 63 38 mov%r12,0x38(%rbx) 1f: 49 8b 44 24 08 mov0x8(%r12),%rax 24: 49 8b 14 24 mov(%r12),%rdx 28: 83 e1 02 and$0x2,%ecx 2b:* 48 89 42 08 mov%rax,0x8(%rdx) -- trapping instruction 2f: 48 89 10 mov%rdx,(%rax) 32: 48 b8 00 02 20 00 00 movabs $0xdead00200200,%rax where that constant is LIST_POISON2 and the and $2 seems to be TIMER_IRQSAFE. So the trapping instruction *looks* like it's doing __list_del() on the timer, and timer-next is NULL. So somebody added a timer, and then deallocated/cleared the structure before it triggered. The problem is, I can't see a way to figure out _who_ did that. I *think* r14 contains the function we're going to jump to in the oops, and that could be interesting to know, but it's not decoded, so you'd have to match it up against a symbol map... Linus -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 07, 2013 at 03:14:48PM -0700, Linus Torvalds wrote: On Mon, Oct 7, 2013 at 1:35 AM, Fengguang Wu fengguang...@intel.com wrote: On Mon, Oct 07, 2013 at 01:12:17AM -0700, Linus Torvalds wrote: My pleasure! Here are 100 randomly selected call traces. Also attached several full dmesgs and the kconfig. Ok, they may be randomly selected, but they are all the same. Which is good, I guess, we're only talking about one bug. Anyway, they all have RIP:run_timer_softirq+0x12c/0x1b8, and the code is 0: 8b 65 c8 mov-0x38(%rbp),%esp 3: 4d 39 ec cmp%r13,%r12 6: 0f 84 2f ff ff ff je 0xff3b c: 41 8b 4c 24 18 mov0x18(%r12),%ecx 11: 4d 8b 74 24 20 mov0x20(%r12),%r14 16: 4d 8b 7c 24 28 mov0x28(%r12),%r15 1b: 4c 89 63 38 mov%r12,0x38(%rbx) 1f: 49 8b 44 24 08 mov0x8(%r12),%rax 24: 49 8b 14 24 mov(%r12),%rdx 28: 83 e1 02 and$0x2,%ecx 2b:* 48 89 42 08 mov%rax,0x8(%rdx) -- trapping instruction 2f: 48 89 10 mov%rdx,(%rax) 32: 48 b8 00 02 20 00 00 movabs $0xdead00200200,%rax where that constant is LIST_POISON2 and the and $2 seems to be TIMER_IRQSAFE. So the trapping instruction *looks* like it's doing __list_del() on the timer, and timer-next is NULL. So somebody added a timer, and then deallocated/cleared the structure before it triggered. The problem is, I can't see a way to figure out _who_ did that. I *think* r14 contains the function we're going to jump to in the oops, and that could be interesting to know, but it's not decoded, so you'd have to match it up against a symbol map... As with all of these, it will be a kobject, prompted by my delayed kobject release - we embed a delayed work structure in the kobject so that we can call the cleanup and detect if it was freed. However, early on, after Greg merged it, the problems with x86 were reported, and I tried all sorts of ways to avoid this. I tried allocating it separately, but that doesn't work because x86 registers kobjects really early. I tried a few other things as well. The idea of allocating the delayed work separately is that it doesn't get freed along with the kobject, and then we can start tracking more reportable state and/or tie it up with other kobject debug. However, due to the problems with x86, that's fallen on its head and I have no solution to get better debugging out which works across all architectures. I'm stumpted by this. However, one thing that this patch _is_ doing is it is uncovering the fact that the kernel is full of kobject refcount problems, and it seems many people just get this stuff wrong. That in itself is quite a problem. What I would say is that we should have had this delayed release either as standard in the kobject system from the start, or as a debug thing to stop these problems as soon as they were initially introduced. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 07, 2013 at 11:29:25PM +0100, Russell King - ARM Linux wrote: However, due to the problems with x86, that's fallen on its head and I have no solution to get better debugging out which works across all architectures. I'm stumpted by this. The final attempt at trying to sort this may be to try allocating a struct at release time to contain a pointer to the kobject and the delayed work queue, and maybe a stack trace. The thing I worry about is whether we can allocate memory at that point. GFP_ATOMIC maybe? Should be a small enough structure that it shouldn't eat too much into the reserved pools. I won't be coding it up tonight though! -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 7, 2013 at 3:14 PM, Linus Torvalds torva...@linux-foundation.org wrote: I *think* r14 contains the function we're going to jump to in the oops, and that could be interesting to know, but it's not decoded, so you'd have to match it up against a symbol map... Actually, Fenguguan, never mind. Instead, change the pr_debug() in kobject_release() to pr_alert(), so that it gets printed out. Or just boot with the ignore_loglevel thing so that debug messages are actually visible. At that point, we should be able to match the oops workqueue list address with the address of the delayed kernel object that gets printed out. Linus -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 7, 2013 at 7:09 PM, Fengguang Wu fengguang...@intel.com wrote: Thanks for the hints! I run a kernel with pr_alert() for several times and here is the screen log. Note that this kernel is compiled with gcc 4.6.3 and the decoded code looks different than gcc 4.8.1 Ok, I think we have something. NULL pointer dereference at 0008 [3.901119] IP: [81098f60] run_timer_softirq+0x126/0x1da [3.901119] PGD 0 [3.901119] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC [3.901119] CPU: 1 PID: 22 Comm: kworker/1:1 Not tainted 3.11.0-rc2-00010-gc817a67-dirty #5 [3.901119] Workqueue: events power_supply_changed_work [3.901119] task: 882dc5c0 ti: 882de000 task.ti: 882de000 [3.901119] RIP: 0010:[81098f60] [81098f60] run_timer_softirq+0x126/0x1da [3.901119] RSP: :88000dd03ea8 EFLAGS: 00010002 [3.901119] RAX: 880006dc28a0 RBX: 8815c000 RCX: 88000dd03ec8 [3.901119] RDX: 880006dc2860 RSI: 810a1b7b RDI: [3.901119] RBP: 88000dd03f08 R08: 0002 R09: dead00200200 [3.901119] R10: 8815c018 R11: 882dc5c0 R12: 88000dd03ec8 [3.901119] R13: 8815d858 R14: 8815d458 R15: 8815d058 [3.901119] FS: () GS:88000dd0() knlGS: [3.901119] CS: 0010 DS: ES: CR0: 8005003b [3.901119] CR2: 0008 CR3: 01c47000 CR4: 06a0 [3.901119] Stack: [3.901119] 88000dd0e1b0 88000dd0e170 810a1b7b 00c0 [3.901119] 880006dc28a0 88000dd0e300 88000dd03ef8 882dffd8 [3.901119] 81c43088 1100 0141 0001 [3.901119] Call Trace: [3.901119] IRQ [3.901119] [810a1b7b] ? __queue_work+0x1ee/0x1ee [3.901119] [8109555f] __do_softirq+0xc6/0x18f [3.901119] [81095745] irq_exit+0x58/0x9e [3.901119] [81044e69] smp_apic_timer_interrupt+0x31/0x3e [3.901119] [8185a772] apic_timer_interrupt+0x72/0x80 [3.901119] EOI [3.901119] [81859277] ? retint_restore_args+0x13/0x13 [3.901119] [810af994] ? arch_local_irq_enable+0xb/0xd [3.901119] [810c707f] ? trace_hardirqs_on+0xd/0xf [3.901119] [81857904] preempt_schedule_irq+0x36/0x5a [3.901119] [818593a6] retint_kernel+0x26/0x30 [3.901119] [8108f47c] ? arch_local_irq_restore+0x6/0xd [3.901119] [81090db8] vprintk_emit+0x3f9/0x422 [3.901119] [8121a640] ? snprintf+0x34/0x36 [3.901119] [81483ca2] dev_vprintk_emit+0x1a3/0x1c9 [3.901119] [81219385] ? string.isra.3+0x3d/0xa2 [3.901119] [81483cfc] dev_printk_emit+0x34/0x36 [3.901119] [81483d6f] __dev_printk+0x71/0x78 [3.901119] [81483db6] dev_printk+0x40/0x42 [3.901119] [8111b1b4] ? __kmalloc+0x93/0xb8 [3.901119] [815dd5a4] power_supply_uevent+0x183/0x1ec [3.901119] [81483451] dev_uevent+0x179/0x1a2 [3.901119] [812130dc] ? kzalloc+0xf/0x11 [3.901119] [812142ce] kobject_uevent_env+0x195/0x414 [3.901119] [81214558] kobject_uevent+0xb/0xd [3.901119] [815dcc14] power_supply_changed_work+0x60/0x65 [3.901119] [810a2733] process_one_work+0x1de/0x2fd [3.901119] [810a26c9] ? process_one_work+0x174/0x2fd [3.901119] [810a2c54] worker_thread+0x159/0x1ee [3.901119] [810a2afb] ? rescuer_thread+0x27f/0x27f [3.901119] [810a859a] kthread+0xac/0xb4 [3.901119] [810a84ee] ? __kthread_parkme+0x68/0x68 [3.901119] [81859a3c] ret_from_fork+0x7c/0xb0 [3.901119] [810a84ee] ? __kthread_parkme+0x68/0x68 [3.901119] Code: 08 e9 99 00 00 00 4c 8b 40 18 48 8b 70 20 49 b9 00 02 20 00 00 00 ad de 48 8b 50 28 48 89 43 38 48 8b 38 48 8b 48 08 41 83 e0 02 48 89 4f 08 48 89 39 f6 40 18 01 48 c7 00 00 00 00 00 4c 89 48 [3.901119] RIP [81098f60] run_timer_softirq+0x126/0x1da So this is the same bug, now the code decodes to 5: 4c 8b 40 18 mov0x18(%rax),%r8 9: 48 8b 70 20 mov0x20(%rax),%rsi d: 49 b9 00 02 20 00 00 movabs $0xdead00200200,%r9 14: 00 ad de 17: 48 8b 50 28 mov0x28(%rax),%rdx 1b: 48 89 43 38 mov%rax,0x38(%rbx) 1f: 48 8b 38 mov(%rax),%rdi 22: 48 8b 48 08 mov0x8(%rax),%rcx 26: 41 83 e0 02 and$0x2,%r8d 2a:* 48 89 4f 08 mov%rcx,0x8(%rdi) -- trapping instruction 2e: 48 89 39 mov%rdi,(%rcx) 31: f6 40 18 01 testb $0x1,0x18(%rax) 35: 48 c7 00 00 00 00 00 movq $0x0,(%rax) so the NULL
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 7, 2013 at 7:14 PM, Fengguang Wu fengguang...@intel.com wrote: I got a call trace containing parport_pc_probe_port() (is it the culprit?) after recompiling kernel with CONFIG_DEBUG_OBJECTS_TIMERS=y and booting with ignore_loglevel. Here is the log for two kernel boots. Ok, so it's definitely parport_pc. I don't see what it would do wrong, though. Linus -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
I'll catch up with your emails if it kills me.. On Mon, Oct 7, 2013 at 7:36 PM, Fengguang Wu fengguang...@intel.com wrote: On Tue, Oct 08, 2013 at 10:14:52AM +0800, Fengguang Wu wrote: Disabled PARPORT_PC: # CONFIG_PARPORT is not set and retest shows another call trace. Here are two run logs: Ugh. Ok. So something else is wrong too. [4.074903] registered taskstats version 1 [4.077382] rtc-test rtc-test.0: setting system clock to 2013-09-09 06:44:08 UTC (1378709048) [4.081688] debug: unmapping init [mem 0x81df2000-0x81ec3fff] /bin/sh: /proc/self/fd/9: No such file or directory [4.148224] BUG: unable to handle kernel NULL pointer dereference at (null) [4.150690] IP: [ (null)] (null) [4.152050] PGD 6a79067 PUD 69d3067 PMD 0 [4.152125] Oops: 0010 [#1] PREEMPT SMP DEBUG_PAGEALLOC [4.152125] CPU: 0 PID: 130 Comm: sh Not tainted 3.11.0-rc2-00010-gc817a67-dirty #7 [4.152125] task: 880006a5e980 ti: 880006a3e000 task.ti: 880006a3e000 [4.152125] RIP: 0010:[] [ (null)] (null) [4.152125] RSP: 0018:88000dc03e40 EFLAGS: 00010286 [4.152125] RAX: 0001 RBX: 0102 RCX: [4.152125] RDX: 0f13 RSI: RDI: [4.152125] RBP: 88000dc03e98 R08: 160f R09: 88000dc0a000 [4.152125] R10: 88000dc0a000 R11: 880006a5e980 R12: 880006a3ffd8 [4.152125] R13: R14: R15: 81f347d8 [4.152125] FS: 7fe94d7b4700() GS:88000dc0() knlGS: [4.152125] CS: 0010 DS: ES: CR0: 80050033 [4.152125] CR2: CR3: 06a49000 CR4: 06b0 [4.152125] Stack: [4.152125] 810989e6 8109897b 810c73b8 88000dc03e58 [4.152125] 821cbe60 81f33780 [4.152125] 88000dc03ec8 81f34fd8 81f34bd8 88000dc03f08 [4.152125] Call Trace: [4.152125] IRQ [4.152125] [810989e6] ? call_timer_fn+0x6b/0xde [4.152125] [8109897b] ? detach_timer+0x46/0x46 [4.152125] [810c73b8] ? trace_hardirqs_on_caller+0x13a/0x18b [4.152125] [81098fcb] run_timer_softirq+0x187/0x1cf [4.152125] [8109555f] __do_softirq+0xc6/0x18f [4.152125] [811264a7] ? unlazy_walk+0x125/0x172 [4.152125] [81095745] irq_exit+0x58/0x9e [4.152125] [81044e69] smp_apic_timer_interrupt+0x31/0x3e [4.152125] [818547b2] apic_timer_interrupt+0x72/0x80 [4.152125] EOI [4.152125] [810c563d] ? arch_local_irq_restore+0x6/0xd [4.152125] [810c9235] lock_release+0x16e/0x17a [4.152125] [81852754] _raw_spin_unlock+0x1b/0x4f [4.152125] [811264a7] unlazy_walk+0x125/0x172 [4.152125] [81128578] do_last+0x6cc/0x9b5 [4.152125] [81126108] ? inode_permission+0x3d/0x3f [4.152125] [81128a8b] path_openat+0x22a/0x48d [4.152125] [81128d23] do_filp_open+0x35/0x85 [4.152125] [811337e9] ? __alloc_fd+0xd8/0xea [4.152125] [8111d932] do_sys_open+0x6b/0xfd [4.152125] [810c73d2] ? trace_hardirqs_on_caller+0x154/0x18b [4.152125] [8111d9dd] SyS_open+0x19/0x1b [4.152125] [81853b29] system_call_fastpath+0x16/0x1b [4.152125] Code: Bad RIP value. And this one doesn't leave any hint of where it came from. Damn. The register contents aren't being very helpful either. The second one is the same thing.. And equally unhelpful as far as I can tell. I think we need more powerful debugging aids to make these useful. Linus -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 7, 2013 at 8:11 PM, Fengguang Wu fengguang...@intel.com wrote: Yeah, I see no timer usage in parport_pc driver, so it's still questionable. The timer itself comes simply from the delayed_work that is used to delay the freeing of the kobject. So that is not the surprising part. The surprising part is that I don't see parport_pc doing anything odd/bad with its kobject embedded in the 'struct dev'. It seems to just do a platform_device_register_simple() followed by a platform_device_unregister(). At least that's true for the normal parport_pc_probe_port() case that just passes in a NULL dev... But I only glanced at the driver, so I might have missed something. with some manual bisects, I find a good config (attached) that can reliably boot the kernel up. Based on that config, I tried adding parport_pc and see that it still boots fine. Adding drm, however will bring back the oops. Will try a kernel based on the original kconfig with drm disabled only. Ok. The list corruption (which also pointed at parport_pc) might well be corrupted by removing the entries before or after the parport_pc, and moving the corruption to parport_pc that way (through the prev-next = next thing in list handling). So maybe it was something else all along. You could enable CONFIG_DEBUG_LIST to see if that triggers some dump earlier.. Linus -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 07, 2013 at 08:29:26PM -0700, Linus Torvalds wrote: On Mon, Oct 7, 2013 at 8:11 PM, Fengguang Wu fengguang...@intel.com wrote: Yeah, I see no timer usage in parport_pc driver, so it's still questionable. The timer itself comes simply from the delayed_work that is used to delay the freeing of the kobject. So that is not the surprising part. OK. The surprising part is that I don't see parport_pc doing anything odd/bad with its kobject embedded in the 'struct dev'. It seems to just do a platform_device_register_simple() followed by a platform_device_unregister(). At least that's true for the normal parport_pc_probe_port() case that just passes in a NULL dev... But I only glanced at the driver, so I might have missed something. with some manual bisects, I find a good config (attached) that can reliably boot the kernel up. Based on that config, I tried adding parport_pc and see that it still boots fine. Adding drm, however will bring back the oops. Will try a kernel based on the original kconfig with drm disabled only. FYI I just confirmed that the original bad kconfig can be made bootable by simply disabling CONFIG_DRM. Ok. The list corruption (which also pointed at parport_pc) might well be corrupted by removing the entries before or after the parport_pc, and moving the corruption to parport_pc that way (through the prev-next = next thing in list handling). So maybe it was something else all along. You could enable CONFIG_DEBUG_LIST to see if that triggers some dump earlier.. OK, I'll try enabling CONFIG_DEBUG_LIST. Thanks, Fengguang -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 07, 2013 at 10:11:18AM +0800, Fengguang Wu wrote:
> On Sun, Oct 06, 2013 at 10:26:24AM -0700, Linus Torvalds wrote:
> > On Sun, Oct 6, 2013 at 1:23 AM, Fengguang Wu wrote:
> > >
> > > I got the below dmesg and the first bad commit is commit cf39c8e5352b:
> > > Merge tag 'stable/for-linus-3.12-rc0-tag' of
> > > git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
> >
> > Ugh. How reliable is the double fault? Because bisecting it to the
> > merge that didn't even have any conflicts in it as far as I can
> > remember means that there's something really subtle going on wrt some
> > semantic conflict or other. Or, alternatively, it means that the
> > bisect failed because the double fault isn't 100% reliable..
>
> Oops, it's not a reliable bisect...
>
> The "first" bad commit cf39c8e5352b4fb9efedfe7e9acb566a85ed847c runs
> and produces 25 good dmesgs and 3530 bad dmesgs, however only 1 of the
> bad boots has "double fault:" in its dmesg.
>
> Looking into all the 3530 bad dmesgs, I find all kinds of bug messages:
>
> $ grep_crash_head -h dmesg-* | sed 's/^[^a-zA-Z]*//' | sort | uniq -c | sort
> -nr
>
>3086 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
>3047 BUG: unable to handle kernel NULL pointer dereference at
> 0008
>3046 Kernel panic - not syncing: Fatal exception in interrupt
I retried bisect with "Oops:" and the first bad commit is
commit c817a67ecba7c3c2aaa104796d78f160af60920d
Author: Russell King
Date: Thu Jun 27 15:06:14 2013 +0100
kobject: delayed kobject release: help find buggy drivers
Implement debugging for kobject release functions. kobjects are
reference counted, so the drop of the last reference to them is not
predictable. However, the common case is for the last reference to be
the kobject's removal from a subsystem, which results in the release
function being immediately called.
This can hide subtle bugs, which can occur when another thread holds a
reference to the kobject at the same time that a kobject is removed.
This results in the release method being delayed.
In order to make these kinds of problems more visible, the following
patch implements a delayed release; this has the effect that the
release function will be out of order with respect to the removal of
the kobject in the same manner that it would be if a reference was
being held.
This provides us with an easy way to allow driver writers to debug
their drivers and fix otherwise hidden problems.
Signed-off-by: Russell King
Signed-off-by: Greg Kroah-Hartman
That commit has already helped expose some bugs, however I suspect there are
still many hidden ones. In this particular bisect, the commit produces 85 good
dmesgs and 2055 bad dmesgs, exposing all sorts of error messages
1929 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
1921 BUG: unable to handle kernel NULL pointer dereference at
0008
1897 Kernel panic - not syncing: Fatal exception in interrupt
56 WARNING: CPU: 0 PID: 1 at /c/wfg/mm/kernel/workqueue.c:590
set_work_data+0x33/0x50()
28 kernel BUG at /c/wfg/mm/mm/slab.c:3011!
27 invalid opcode: [#1] PREEMPT SMP DEBUG_PAGEALLOC
19 Kernel panic - not syncing: Attempted to kill init! exitcode=0x000b
16 INFO: lockdep is turned off.
13 general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC
13 BUG: unable to handle kernel
8 BUG: sleeping function called from invalid context at
/c/wfg/mm/kernel/rwsem.c:20
6 WARNING: CPU: 0 PID: 0 at /c/wfg/mm/lib/debugobjects.c:260
debug_print_object+0x7c/0x8b()
6 WARNING: CPU: 0 PID: 0 at /c/wfg/mm/kernel/workqueue.c:457
work_fixup_activate+0x6a/0x6f()
6 WARNING: CPU: 0 PID: 0 at /c/wfg/mm/kernel/workqueue.c:1378
__queue_work+0x1a1/0x1ee()
5 WARNING: CPU: 1 PID: 0 at /c/wfg/mm/lib/debugobjects.c:260
debug_print_object+0x7c/0x8b()
5 WARNING: CPU: 1 PID: 0 at /c/wfg/mm/kernel/workqueue.c:457
work_fixup_activate+0x6a/0x6f()
5 WARNING: CPU: 1 PID: 0 at /c/wfg/mm/kernel/workqueue.c:1378
__queue_work+0x1a1/0x1ee()
5 Oops: 0002 [#1]
4 WARNING: CPU: 1 PID: 0 at /c/wfg/mm/kernel/workqueue.c:590
set_work_data+0x33/0x50()
4 Oops: [#2] PREEMPT SMP DEBUG_PAGEALLOC
4 BUG: unable to handle kernel NULL pointer dereference
3 Oops: [#1] PREEMPT SMP DEBUG_PAGEALLOC
3 BUG: kernel boot crashed
2 WARNING: CPU: 1 PID: 1 at /c/wfg/mm/kernel/workqueue.c:590
set_work_data+0x33/0x50()
2 BUG: unable to handle kernel paging request at ffa8
2 BUG: unable to handle kernel NULL pointer dereference at
00a0
2 BUG: unable to handle kernel NULL pointer dereference at
0017
2 BUG: scheduling while atomic: rc.local/135/0x1002
In comparison, its parent commit 7c42721fe0 ("char: tile-srom: fix
build error") boots fine 10001 times w/o a single error. It also goes
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Sun, Oct 06, 2013 at 10:26:24AM -0700, Linus Torvalds wrote: > On Sun, Oct 6, 2013 at 1:23 AM, Fengguang Wu wrote: > > > > I got the below dmesg and the first bad commit is commit cf39c8e5352b: > > Merge tag 'stable/for-linus-3.12-rc0-tag' of > > git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip > > Ugh. How reliable is the double fault? Because bisecting it to the > merge that didn't even have any conflicts in it as far as I can > remember means that there's something really subtle going on wrt some > semantic conflict or other. Or, alternatively, it means that the > bisect failed because the double fault isn't 100% reliable.. Oops, it's not a reliable bisect... The "first" bad commit cf39c8e5352b4fb9efedfe7e9acb566a85ed847c runs and produces 25 good dmesgs and 3530 bad dmesgs, however only 1 of the bad boots has "double fault:" in its dmesg. Looking into all the 3530 bad dmesgs, I find all kinds of bug messages: $ grep_crash_head -h dmesg-* | sed 's/^[^a-zA-Z]*//' | sort | uniq -c | sort -nr 3086 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC 3047 BUG: unable to handle kernel NULL pointer dereference at 0008 3046 Kernel panic - not syncing: Fatal exception in interrupt 2969 BUG: kernel boot oops 374 BUG: kernel test oops 255 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50() 167 kernel BUG at /c/wfg/linux-drm/mm/slab.c:3011! 167 invalid opcode: [#1] PREEMPT SMP DEBUG_PAGEALLOC 148 Kernel panic - not syncing: Attempted to kill init! exitcode=0x000b 48 INFO: lockdep is turned off. 43 BUG: unable to handle kernel 33 BUG: kernel boot crashed 30 BUG: sleeping function called from invalid context at /c/wfg/linux-drm/kernel/rwsem.c:20 27 general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC 17 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b() 17 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f() 17 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee() 13 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50() 13 BUG: unable to handle kernel NULL pointer dereference at (null) 12 Oops: 0010 [#1] PREEMPT SMP DEBUG_PAGEALLOC 11 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b() 11 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f() 11 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee() 11 Oops: [#1] PREEMPT SMP DEBUG_PAGEALLOC 9 INFO: trying to register non-static key. 9 BUG: scheduling while atomic: init/136/0x1002 8 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50() 8 BUG: unable to handle kernel NULL pointer dereference 6 Oops: [#2] PREEMPT SMP DEBUG_PAGEALLOC 5 BUG: unable to handle kernel paging request at ffa8 5 BUG: Bad page map in process init pte: pmd:06d9e067 5 BUG: Bad page map in process init pte: pmd:06d9e067 4 Oops: 0002 [#1] 4 Kernel panic - not syncing: Attempted to kill the idle task! 4 BUG: unable to handle kernel paging request at 88000cd94000 3 invalid opcode: [#2] PREEMPT SMP DEBUG_PAGEALLOC 3 WARNING: CPU: 1 PID: 95 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b() 3 WARNING: CPU: 1 PID: 95 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f() 3 WARNING: CPU: 1 PID: 95 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee() 3 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b() 3 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f() 3 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee() 3 WARNING: CPU: 0 PID: 116 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50() 3 BUG: kernel boot hang 3 BUG: Bad page map in process init pte:81f0fa00 pmd:06d9e067 3 BUG: Bad page map in process init pte:81b52e93 pmd:06d9e067 3 BUG: Bad page map in process init pte:dead4ead pmd:06d9e067 2 kernel BUG at /c/wfg/linux-drm/include/linux/mm.h:286! 2 general protection fault: [#2] PREEMPT SMP DEBUG_PAGEALLOC 2 WARNING: CPU: 1 PID: 130 at /c/wfg/linux-drm/drivers/tty/tty_mutex.c:23 tty_lock_nested+0x34/0x83() 2 WARNING: CPU: 1 PID: 121 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50() 2 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
- torva...@linux-foundation.org wrote: > On Sun, Oct 6, 2013 at 1:23 AM, Fengguang Wu > wrote: > > > > I got the below dmesg and the first bad commit is commit > cf39c8e5352b: > > Merge tag 'stable/for-linus-3.12-rc0-tag' of > git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip > > Ugh. How reliable is the double fault? Because bisecting it to the > merge that didn't even have any conflicts in it as far as I can > remember means that there's something really subtle going on wrt some > semantic conflict or other. Or, alternatively, it means that the > bisect failed because the double fault isn't 100% reliable.. > > Anyway, the stack is crap when the original fault happens at > "boot_tvec_bases+0x1fe", and that causes the double fault debug code > to take *another* fault, which means that it doesn't even show the > right code sequence. Too bad. So ignore the latter part of the oops, > but the top part looks valid: > > > [4.136137] double fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC > > [4.137521] CPU: 0 PID: 132 Comm: bootlogd Not tainted > 3.12.0-rc2-00153-g14951f2 #129 > > [4.139156] task: 88000c9a6580 ti: 88000c9ba000 task.ti: > 88000c9ba000 > > [4.140042] RIP: 0010:[] [] > boot_tvec_bases+0x1fe/0x2080 > > [4.140042] RSP: 0018:88000cd8 EFLAGS: 00010212 > > [4.140042] RAX: 004f RBX: 0100 RCX: > > > [4.140042] RDX: 0f1e RSI: 81f746a8 RDI: > 81f31c48 > > [4.140042] RBP: 88000f003ee0 R08: R09: > > > [4.140042] R10: 0001 R11: 88000f00a000 R12: > 88000c9bbfd8 > > [4.140042] R13: 81f31c48 R14: 81f31c48 R15: > 81f31c48 > > [4.140042] FS: 7fb1f9662700() GS:88000f00() > knlGS: > > [4.140042] CS: 0010 DS: ES: CR0: 80050033 > > [4.140042] CR2: 88000cc8 CR3: 0c9cd000 CR4: > 06b0 > > [4.140042] Stack: > > > but it has jumped into a data section and is executing random data as > code, and there is no sign of where it jumped *from*, since the > random > code clearly corrupted the stack - resulting in the double fault in > the first place. > > So the oops is almost entirely useless as a debug aid in this > situation. I'm almost hoping that your bisect was wrong, and you > could > try to see if you could do that again.. For what it's worth, the commit in question touches almost exclusively Xen files, the only exception being lib/swiotlb.c (with what appear to be fairly trivial changes). And CONFIG_XEN in the config file for this report is not set. -boris -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Sun, Oct 6, 2013 at 1:23 AM, Fengguang Wu wrote: > > I got the below dmesg and the first bad commit is commit cf39c8e5352b: > Merge tag 'stable/for-linus-3.12-rc0-tag' of > git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip Ugh. How reliable is the double fault? Because bisecting it to the merge that didn't even have any conflicts in it as far as I can remember means that there's something really subtle going on wrt some semantic conflict or other. Or, alternatively, it means that the bisect failed because the double fault isn't 100% reliable.. Anyway, the stack is crap when the original fault happens at "boot_tvec_bases+0x1fe", and that causes the double fault debug code to take *another* fault, which means that it doesn't even show the right code sequence. Too bad. So ignore the latter part of the oops, but the top part looks valid: > [4.136137] double fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC > [4.137521] CPU: 0 PID: 132 Comm: bootlogd Not tainted > 3.12.0-rc2-00153-g14951f2 #129 > [4.139156] task: 88000c9a6580 ti: 88000c9ba000 task.ti: > 88000c9ba000 > [4.140042] RIP: 0010:[] [] > boot_tvec_bases+0x1fe/0x2080 > [4.140042] RSP: 0018:88000cd8 EFLAGS: 00010212 > [4.140042] RAX: 004f RBX: 0100 RCX: > > [4.140042] RDX: 0f1e RSI: 81f746a8 RDI: > 81f31c48 > [4.140042] RBP: 88000f003ee0 R08: R09: > > [4.140042] R10: 0001 R11: 88000f00a000 R12: > 88000c9bbfd8 > [4.140042] R13: 81f31c48 R14: 81f31c48 R15: > 81f31c48 > [4.140042] FS: 7fb1f9662700() GS:88000f00() > knlGS: > [4.140042] CS: 0010 DS: ES: CR0: 80050033 > [4.140042] CR2: 88000cc8 CR3: 0c9cd000 CR4: > 06b0 > [4.140042] Stack: but it has jumped into a data section and is executing random data as code, and there is no sign of where it jumped *from*, since the random code clearly corrupted the stack - resulting in the double fault in the first place. So the oops is almost entirely useless as a debug aid in this situation. I'm almost hoping that your bisect was wrong, and you could try to see if you could do that again.. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Sun, Oct 6, 2013 at 1:23 AM, Fengguang Wu fengguang...@intel.com wrote: I got the below dmesg and the first bad commit is commit cf39c8e5352b: Merge tag 'stable/for-linus-3.12-rc0-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip Ugh. How reliable is the double fault? Because bisecting it to the merge that didn't even have any conflicts in it as far as I can remember means that there's something really subtle going on wrt some semantic conflict or other. Or, alternatively, it means that the bisect failed because the double fault isn't 100% reliable.. Anyway, the stack is crap when the original fault happens at boot_tvec_bases+0x1fe, and that causes the double fault debug code to take *another* fault, which means that it doesn't even show the right code sequence. Too bad. So ignore the latter part of the oops, but the top part looks valid: [4.136137] double fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC [4.137521] CPU: 0 PID: 132 Comm: bootlogd Not tainted 3.12.0-rc2-00153-g14951f2 #129 [4.139156] task: 88000c9a6580 ti: 88000c9ba000 task.ti: 88000c9ba000 [4.140042] RIP: 0010:[81f31c7e] [81f31c7e] boot_tvec_bases+0x1fe/0x2080 [4.140042] RSP: 0018:88000cd8 EFLAGS: 00010212 [4.140042] RAX: 004f RBX: 0100 RCX: [4.140042] RDX: 0f1e RSI: 81f746a8 RDI: 81f31c48 [4.140042] RBP: 88000f003ee0 R08: R09: [4.140042] R10: 0001 R11: 88000f00a000 R12: 88000c9bbfd8 [4.140042] R13: 81f31c48 R14: 81f31c48 R15: 81f31c48 [4.140042] FS: 7fb1f9662700() GS:88000f00() knlGS: [4.140042] CS: 0010 DS: ES: CR0: 80050033 [4.140042] CR2: 88000cc8 CR3: 0c9cd000 CR4: 06b0 [4.140042] Stack: boom, it crashes again here but it has jumped into a data section and is executing random data as code, and there is no sign of where it jumped *from*, since the random code clearly corrupted the stack - resulting in the double fault in the first place. So the oops is almost entirely useless as a debug aid in this situation. I'm almost hoping that your bisect was wrong, and you could try to see if you could do that again.. Linus -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
- torva...@linux-foundation.org wrote: On Sun, Oct 6, 2013 at 1:23 AM, Fengguang Wu fengguang...@intel.com wrote: I got the below dmesg and the first bad commit is commit cf39c8e5352b: Merge tag 'stable/for-linus-3.12-rc0-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip Ugh. How reliable is the double fault? Because bisecting it to the merge that didn't even have any conflicts in it as far as I can remember means that there's something really subtle going on wrt some semantic conflict or other. Or, alternatively, it means that the bisect failed because the double fault isn't 100% reliable.. Anyway, the stack is crap when the original fault happens at boot_tvec_bases+0x1fe, and that causes the double fault debug code to take *another* fault, which means that it doesn't even show the right code sequence. Too bad. So ignore the latter part of the oops, but the top part looks valid: [4.136137] double fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC [4.137521] CPU: 0 PID: 132 Comm: bootlogd Not tainted 3.12.0-rc2-00153-g14951f2 #129 [4.139156] task: 88000c9a6580 ti: 88000c9ba000 task.ti: 88000c9ba000 [4.140042] RIP: 0010:[81f31c7e] [81f31c7e] boot_tvec_bases+0x1fe/0x2080 [4.140042] RSP: 0018:88000cd8 EFLAGS: 00010212 [4.140042] RAX: 004f RBX: 0100 RCX: [4.140042] RDX: 0f1e RSI: 81f746a8 RDI: 81f31c48 [4.140042] RBP: 88000f003ee0 R08: R09: [4.140042] R10: 0001 R11: 88000f00a000 R12: 88000c9bbfd8 [4.140042] R13: 81f31c48 R14: 81f31c48 R15: 81f31c48 [4.140042] FS: 7fb1f9662700() GS:88000f00() knlGS: [4.140042] CS: 0010 DS: ES: CR0: 80050033 [4.140042] CR2: 88000cc8 CR3: 0c9cd000 CR4: 06b0 [4.140042] Stack: boom, it crashes again here but it has jumped into a data section and is executing random data as code, and there is no sign of where it jumped *from*, since the random code clearly corrupted the stack - resulting in the double fault in the first place. So the oops is almost entirely useless as a debug aid in this situation. I'm almost hoping that your bisect was wrong, and you could try to see if you could do that again.. For what it's worth, the commit in question touches almost exclusively Xen files, the only exception being lib/swiotlb.c (with what appear to be fairly trivial changes). And CONFIG_XEN in the config file for this report is not set. -boris -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Sun, Oct 06, 2013 at 10:26:24AM -0700, Linus Torvalds wrote: On Sun, Oct 6, 2013 at 1:23 AM, Fengguang Wu fengguang...@intel.com wrote: I got the below dmesg and the first bad commit is commit cf39c8e5352b: Merge tag 'stable/for-linus-3.12-rc0-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip Ugh. How reliable is the double fault? Because bisecting it to the merge that didn't even have any conflicts in it as far as I can remember means that there's something really subtle going on wrt some semantic conflict or other. Or, alternatively, it means that the bisect failed because the double fault isn't 100% reliable.. Oops, it's not a reliable bisect... The first bad commit cf39c8e5352b4fb9efedfe7e9acb566a85ed847c runs and produces 25 good dmesgs and 3530 bad dmesgs, however only 1 of the bad boots has double fault: in its dmesg. Looking into all the 3530 bad dmesgs, I find all kinds of bug messages: $ grep_crash_head -h dmesg-* | sed 's/^[^a-zA-Z]*//' | sort | uniq -c | sort -nr 3086 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC 3047 BUG: unable to handle kernel NULL pointer dereference at 0008 3046 Kernel panic - not syncing: Fatal exception in interrupt 2969 BUG: kernel boot oops 374 BUG: kernel test oops 255 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50() 167 kernel BUG at /c/wfg/linux-drm/mm/slab.c:3011! 167 invalid opcode: [#1] PREEMPT SMP DEBUG_PAGEALLOC 148 Kernel panic - not syncing: Attempted to kill init! exitcode=0x000b 48 INFO: lockdep is turned off. 43 BUG: unable to handle kernel 33 BUG: kernel boot crashed 30 BUG: sleeping function called from invalid context at /c/wfg/linux-drm/kernel/rwsem.c:20 27 general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC 17 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b() 17 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f() 17 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee() 13 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50() 13 BUG: unable to handle kernel NULL pointer dereference at (null) 12 Oops: 0010 [#1] PREEMPT SMP DEBUG_PAGEALLOC 11 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b() 11 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f() 11 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee() 11 Oops: [#1] PREEMPT SMP DEBUG_PAGEALLOC 9 INFO: trying to register non-static key. 9 BUG: scheduling while atomic: init/136/0x1002 8 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50() 8 BUG: unable to handle kernel NULL pointer dereference 6 Oops: [#2] PREEMPT SMP DEBUG_PAGEALLOC 5 BUG: unable to handle kernel paging request at ffa8 5 BUG: Bad page map in process init pte: pmd:06d9e067 5 BUG: Bad page map in process init pte: pmd:06d9e067 4 Oops: 0002 [#1] 4 Kernel panic - not syncing: Attempted to kill the idle task! 4 BUG: unable to handle kernel paging request at 88000cd94000 3 invalid opcode: [#2] PREEMPT SMP DEBUG_PAGEALLOC 3 WARNING: CPU: 1 PID: 95 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b() 3 WARNING: CPU: 1 PID: 95 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f() 3 WARNING: CPU: 1 PID: 95 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee() 3 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b() 3 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f() 3 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee() 3 WARNING: CPU: 0 PID: 116 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50() 3 BUG: kernel boot hang 3 BUG: Bad page map in process init pte:81f0fa00 pmd:06d9e067 3 BUG: Bad page map in process init pte:81b52e93 pmd:06d9e067 3 BUG: Bad page map in process init pte:dead4ead pmd:06d9e067 2 kernel BUG at /c/wfg/linux-drm/include/linux/mm.h:286! 2 general protection fault: [#2] PREEMPT SMP DEBUG_PAGEALLOC 2 WARNING: CPU: 1 PID: 130 at /c/wfg/linux-drm/drivers/tty/tty_mutex.c:23 tty_lock_nested+0x34/0x83() 2 WARNING: CPU: 1 PID: 121 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50() 2 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:591
Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
On Mon, Oct 07, 2013 at 10:11:18AM +0800, Fengguang Wu wrote: On Sun, Oct 06, 2013 at 10:26:24AM -0700, Linus Torvalds wrote: On Sun, Oct 6, 2013 at 1:23 AM, Fengguang Wu fengguang...@intel.com wrote: I got the below dmesg and the first bad commit is commit cf39c8e5352b: Merge tag 'stable/for-linus-3.12-rc0-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip Ugh. How reliable is the double fault? Because bisecting it to the merge that didn't even have any conflicts in it as far as I can remember means that there's something really subtle going on wrt some semantic conflict or other. Or, alternatively, it means that the bisect failed because the double fault isn't 100% reliable.. Oops, it's not a reliable bisect... The first bad commit cf39c8e5352b4fb9efedfe7e9acb566a85ed847c runs and produces 25 good dmesgs and 3530 bad dmesgs, however only 1 of the bad boots has double fault: in its dmesg. Looking into all the 3530 bad dmesgs, I find all kinds of bug messages: $ grep_crash_head -h dmesg-* | sed 's/^[^a-zA-Z]*//' | sort | uniq -c | sort -nr 3086 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC 3047 BUG: unable to handle kernel NULL pointer dereference at 0008 3046 Kernel panic - not syncing: Fatal exception in interrupt I retried bisect with Oops: and the first bad commit is commit c817a67ecba7c3c2aaa104796d78f160af60920d Author: Russell King rmk+ker...@arm.linux.org.uk Date: Thu Jun 27 15:06:14 2013 +0100 kobject: delayed kobject release: help find buggy drivers Implement debugging for kobject release functions. kobjects are reference counted, so the drop of the last reference to them is not predictable. However, the common case is for the last reference to be the kobject's removal from a subsystem, which results in the release function being immediately called. This can hide subtle bugs, which can occur when another thread holds a reference to the kobject at the same time that a kobject is removed. This results in the release method being delayed. In order to make these kinds of problems more visible, the following patch implements a delayed release; this has the effect that the release function will be out of order with respect to the removal of the kobject in the same manner that it would be if a reference was being held. This provides us with an easy way to allow driver writers to debug their drivers and fix otherwise hidden problems. Signed-off-by: Russell King rmk+ker...@arm.linux.org.uk Signed-off-by: Greg Kroah-Hartman gre...@linuxfoundation.org That commit has already helped expose some bugs, however I suspect there are still many hidden ones. In this particular bisect, the commit produces 85 good dmesgs and 2055 bad dmesgs, exposing all sorts of error messages 1929 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC 1921 BUG: unable to handle kernel NULL pointer dereference at 0008 1897 Kernel panic - not syncing: Fatal exception in interrupt 56 WARNING: CPU: 0 PID: 1 at /c/wfg/mm/kernel/workqueue.c:590 set_work_data+0x33/0x50() 28 kernel BUG at /c/wfg/mm/mm/slab.c:3011! 27 invalid opcode: [#1] PREEMPT SMP DEBUG_PAGEALLOC 19 Kernel panic - not syncing: Attempted to kill init! exitcode=0x000b 16 INFO: lockdep is turned off. 13 general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC 13 BUG: unable to handle kernel 8 BUG: sleeping function called from invalid context at /c/wfg/mm/kernel/rwsem.c:20 6 WARNING: CPU: 0 PID: 0 at /c/wfg/mm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b() 6 WARNING: CPU: 0 PID: 0 at /c/wfg/mm/kernel/workqueue.c:457 work_fixup_activate+0x6a/0x6f() 6 WARNING: CPU: 0 PID: 0 at /c/wfg/mm/kernel/workqueue.c:1378 __queue_work+0x1a1/0x1ee() 5 WARNING: CPU: 1 PID: 0 at /c/wfg/mm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b() 5 WARNING: CPU: 1 PID: 0 at /c/wfg/mm/kernel/workqueue.c:457 work_fixup_activate+0x6a/0x6f() 5 WARNING: CPU: 1 PID: 0 at /c/wfg/mm/kernel/workqueue.c:1378 __queue_work+0x1a1/0x1ee() 5 Oops: 0002 [#1] 4 WARNING: CPU: 1 PID: 0 at /c/wfg/mm/kernel/workqueue.c:590 set_work_data+0x33/0x50() 4 Oops: [#2] PREEMPT SMP DEBUG_PAGEALLOC 4 BUG: unable to handle kernel NULL pointer dereference 3 Oops: [#1] PREEMPT SMP DEBUG_PAGEALLOC 3 BUG: kernel boot crashed 2 WARNING: CPU: 1 PID: 1 at /c/wfg/mm/kernel/workqueue.c:590 set_work_data+0x33/0x50() 2 BUG: unable to handle kernel paging request at ffa8 2 BUG: unable to handle kernel NULL pointer dereference at 00a0 2 BUG: unable to handle kernel NULL pointer dereference at 0017 2 BUG: scheduling while atomic: rc.local/135/0x1002 In comparison, its parent commit 7c42721fe0 (char: tile-srom: fix build error) boots
