On Fri, 07 Jan 2011 14:18:22 -0600, Mike Christie wrote: michaelc> On 01/07/2011 01:37 AM, Max Matveev wrote:
>> I have the backtrace but it's almost useless - the code path is >> unbelievable: the only reliable bit is michaelc> Send it. There might be something useful. I've got 94 back traces in the dmesg, most of them (39 out of 94) are like this: BUG: scheduling while atomic: iscsi_q_31/0x10000001/7288 [<c061b52f>] schedule+0x43/0xa55 [<c05edc56>] tcp_transmit_skb+0x5c7/0x5f5 [<c05ef33b>] __tcp_push_pending_frames+0x474/0x752 [<c04211bc>] __cond_resched+0x16/0x34 [<c061bf67>] cond_resched+0x26/0x31 [<c05b8a8b>] lock_sock+0xe/0x96 [<c061d62f>] _spin_lock_bh+0x8/0x18 [<c05e525d>] tcp_sendmsg+0x12/0x9d2 [<c0471055>] cache_alloc_refill+0x60/0x482 [<c05fd04c>] inet_sendmsg+0x35/0x3f [<c05b623f>] sock_sendmsg+0xd3/0x11d [<c0436473>] autoremove_wake_function+0x0/0x2d [<c05bbec3>] __alloc_skb+0x49/0xfe [<f8acc815>] iscsi_tcp_segment_map+0x5b/0x6a [libiscsi_tcp] [<f8acca0e>] iscsi_tcp_segment_done+0x1c7/0x2a7 [libiscsi_tcp] [<c05b7f45>] kernel_sendmsg+0x27/0x35 [<f8ad33e8>] iscsi_sw_tcp_pdu_xmit+0x9e/0x1e8 [iscsi_tcp] [<f8acc246>] iscsi_tcp_task_xmit+0x25/0x204 [libiscsi_tcp] [<f8d9b622>] iscsi_xmit_task+0x24/0x4b [libiscsi2] [<f8d9c190>] iscsi_xmitworker+0x122/0x202 [libiscsi2] [<c04336e2>] run_workqueue+0x78/0xb5 [<f8d9c06e>] iscsi_xmitworker+0x0/0x202 [libiscsi2] [<c0433f96>] worker_thread+0xd9/0x10b [<c041fc47>] default_wake_function+0x0/0xc [<c0433ebd>] worker_thread+0x0/0x10b [<c04363af>] kthread+0xc0/0xed [<c04362ef>] kthread+0x0/0xed [<c0405c53>] kernel_thread_helper+0x7/0x10 The other two are repeated 13 and 11 times: BUG: scheduling while atomic: iscsi_q_31/0x10000001/7288 [<c061b52f>] schedule+0x43/0xa55 [<c045adc6>] get_page_from_freelist+0x96/0x370 [<c045b109>] __alloc_pages+0x69/0x2cf [<c04211bc>] __cond_resched+0x16/0x34 [<c061bf67>] cond_resched+0x26/0x31 [<c04eff85>] copy_from_user+0x31/0x5d [<c05e5865>] tcp_sendmsg+0x61a/0x9d2 [<c0471055>] cache_alloc_refill+0x60/0x482 [<c05fd04c>] inet_sendmsg+0x35/0x3f [<c05b623f>] sock_sendmsg+0xd3/0x11d [<c0436473>] autoremove_wake_function+0x0/0x2d [<c05bbec3>] __alloc_skb+0x49/0xfe [<f8acc815>] iscsi_tcp_segment_map+0x5b/0x6a [libiscsi_tcp] [<f8acca0e>] iscsi_tcp_segment_done+0x1c7/0x2a7 [libiscsi_tcp] [<c05b7f45>] kernel_sendmsg+0x27/0x35 [<f8ad33e8>] iscsi_sw_tcp_pdu_xmit+0x9e/0x1e8 [iscsi_tcp] [<f8acc246>] iscsi_tcp_task_xmit+0x25/0x204 [libiscsi_tcp] [<f8d9b622>] iscsi_xmit_task+0x24/0x4b [libiscsi2] [<f8d9c190>] iscsi_xmitworker+0x122/0x202 [libiscsi2] [<c04336e2>] run_workqueue+0x78/0xb5 [<f8d9c06e>] iscsi_xmitworker+0x0/0x202 [libiscsi2] [<c0433f96>] worker_thread+0xd9/0x10b [<c041fc47>] default_wake_function+0x0/0xc [<c0433ebd>] worker_thread+0x0/0x10b [<c04363af>] kthread+0xc0/0xed [<c04362ef>] kthread+0x0/0xed [<c0405c53>] kernel_thread_helper+0x7/0x10 BUG: scheduling while atomic: iscsi_q_31/0x10000001/7288 [<c061b52f>] schedule+0x43/0xa55 [<c04211bc>] __cond_resched+0x16/0x34 [<c061bf67>] cond_resched+0x26/0x31 [<c05b8a8b>] lock_sock+0xe/0x96 [<c05e525d>] tcp_sendmsg+0x12/0x9d2 [<c05e063d>] ip_output+0x22e/0x265 [<c05dfdf1>] ip_queue_xmit+0x3cd/0x40c [<c0471055>] cache_alloc_refill+0x60/0x482 [<c05fd04c>] inet_sendmsg+0x35/0x3f [<c05b623f>] sock_sendmsg+0xd3/0x11d [<c0436473>] autoremove_wake_function+0x0/0x2d [<c05bbec3>] __alloc_skb+0x49/0xfe [<f8acc815>] iscsi_tcp_segment_map+0x5b/0x6a [libiscsi_tcp] [<f8acca0e>] iscsi_tcp_segment_done+0x1c7/0x2a7 [libiscsi_tcp] [<c05b7f45>] kernel_sendmsg+0x27/0x35 [<f8ad33e8>] iscsi_sw_tcp_pdu_xmit+0x9e/0x1e8 [iscsi_tcp] [<f8acc246>] iscsi_tcp_task_xmit+0x25/0x204 [libiscsi_tcp] [<f8d9b622>] iscsi_xmit_task+0x24/0x4b [libiscsi2] [<f8d9c190>] iscsi_xmitworker+0x122/0x202 [libiscsi2] [<c04336e2>] run_workqueue+0x78/0xb5 [<f8d9c06e>] iscsi_xmitworker+0x0/0x202 [libiscsi2] [<c0433f96>] worker_thread+0xd9/0x10b [<c041fc47>] default_wake_function+0x0/0xc [<c0433ebd>] worker_thread+0x0/0x10b [<c04363af>] kthread+0xc0/0xed [<c04362ef>] kthread+0x0/0xed [<c0405c53>] kernel_thread_helper+0x7/0x10 I'm trying to get a kernel dump to hopefully get a better backtrace and check the state of the socket. >> He's running RHEL 5.5 i686 kernel - sorry, I cannot tell the version >> of open-iscsi bits in this kernel. michaelc> Just uname -a the kernel. I can map it to the version. I michaelc> think it should be close to upstream though. Linux taper 2.6.18-194.11.3.el5PAE #1 SMP Mon Aug 23 15:57:10 EDT 2010 i686 i686 i386 GNU/Linux i386 is important - in RHEL kmap_atomic in arch/i386/mm/highmem.c bumps preemp count which explains why this kind of messages is seen on i386 and not on x86_64. >> After some digging into the code I'm starting to wonder if kmap_atomic >> is the right thing to do when dealing with tcp: tcp_sendmsg() can >> legitemately sleep on full socket buffer or when trying to allocate an michaelc> Ack on the sleeping waiting for write space. But, for skb michaelc> allocation isn't the sock->sk_allocation used always or michaelc> what paths is it not used? My bad - I've assumed that iscsi_tcp is using GFP_KERNEL and will wait. max -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
