Re: [Qemu-devel] [Bug 1708442] [NEW] Crash(assert) during reading image from http url through qemu-nbd
03.08.2017, 17:01, "Eric Blake" <1708...@bugs.launchpad.net>:
> On 08/03/2017 07:12 AM, Andrey Smetanin wrote:
>> Public bug reported:
>>
>> Description:
>> During reading image from nbd device mounted by qemu-nbd server with url
>> backend I/O error happens
>> "blk_update_request: I/O error, dev nbd0, sector 42117" dmesg. After some
>> investigation I found that qemu-nbd server aborts in aio_co_enter() assert
>> in util/async.c:468.
>
> Based on the backtrace, this looks to be a bug in the block/curl.c
> driver, rather than the nbd/ or block/nbd.c code. If I'm right, it
> should be possible to reproduce the crash using qemu-io directly on the
> curl path, rather than adding the extra layer of an nbd client reading
> through qemu-nbd (then again, having the qemu-nbd layer may be what is
> allowing multiple parallel requests to hit the curl driver at once,
> while qemu-io is not quite as easy to provoke into performing
> complicated access patterns).
>
>> Steps to reproduce:
>>
>> 1) sudo go run qemu-nbd-bug-report/qemu-nbd-bug.go (see qemu-nbd-bug-
>> report.tar.gz)
>>
>> or try directly
>>
>> 1) qemu-nbd -c /dev/nbd0 -r -v --aio=native -f qcow2
>> json:{"file.driver":"http","file.url":"http://localhost:9666/image","file.readahead":3276800
>
> Presumably, you've got something serving the file at port 9666?
Yes, you are right. I'm using qemu-nbd-bug.go(see qemu-nbd-bug-report.tar.gz)
script which does it.
>
>> 2) try read whole nbd device while error in dmesg appears x
>>
>> Versions:
>>
>> 1) qemu built from sources(/configure --target-list=x86_64-softmmu
>> --disable-user --enable-curl --enable-linux-aio --enable-virtfs
>> --enable-debug --disable-pie
>> , top commit 5619c179057e24195ff19c8fe6d6a6cbcb16ed28):
>>
>> qemu-nbd -v
>> qemu-nbd 2.9.90 (v2.10.0-rc0-67-g5619c17)
>>
>> 2) libcurl(built from sources, top commit
>> 1767adf4399bb3be29121435e1bb1cc2bc05f7bf):
>>
>> curl -V
>> curl 7.55.0-DEV (Linux) libcurl/7.55.0-DEV OpenSSL/1.0.2g zlib/1.2.8
>>
>> Backtrace:
>> (gdb) bt
>> #0 0x7f7131426428 in __GI_raise (sig=sig@entry=6) at
>> ../sysdeps/unix/sysv/linux/raise.c:54
>> #1 0x7f713142802a in __GI_abort () at abort.c:89
>> #2 0x7f713141ebd7 in __assert_fail_base (fmt=,
>> assertion=assertion@entry=0x54c924 "self != co",
>> file=file@entry=0x54c871 "util/async.c", line=line@entry=468,
>> function=function@entry=0x54c980 <__PRETTY_FUNCTION__.24766>
>> "aio_co_enter") at assert.c:92
>> #3 0x7f713141ec82 in __GI___assert_fail (assertion=0x54c924 "self !=
>> co", file=0x54c871 "util/async.c", line=468,
>> function=0x54c980 <__PRETTY_FUNCTION__.24766> "aio_co_enter") at
>> assert.c:101
>> #4 0x004fe6a2 in aio_co_enter (ctx=0xf0ddb0, co=0xf14650) at
>> util/async.c:468
>> #5 0x004fe637 in aio_co_wake (co=0xf14650) at util/async.c:456
>> #6 0x00495c8a in curl_read_cb (ptr=0xf566d9, size=1, nmemb=16135,
>> opaque=0xf1cb90) at block/curl.c:275
>> #7 0x7f713242ac24 in Curl_client_chop_write () from
>> /usr/lib/x86_64-linux-gnu/libcurl.so
>> #8 0x7f713242ae03 in Curl_client_write () from
>> /usr/lib/x86_64-linux-gnu/libcurl.so
>> #9 0x7f713244e1cf in readwrite_data () from
>> /usr/lib/x86_64-linux-gnu/libcurl.so
>> #10 0x7f713244eb6f in Curl_readwrite () from
>> /usr/lib/x86_64-linux-gnu/libcurl.so
>> #11 0x7f713245c1bb in multi_runsingle () from
>> /usr/lib/x86_64-linux-gnu/libcurl.so
>> #12 0x7f713245d819 in multi_socket () from
>> /usr/lib/x86_64-linux-gnu/libcurl.so
>> #13 0x7f713245e067 in curl_multi_socket_action () from
>> /usr/lib/x86_64-linux-gnu/libcurl.so
>> #14 0x00497555 in curl_setup_preadv (bs=0xf16820,
>> acb=0x7f712d379860) at block/curl.c:918
>> #15 0x004975fb in curl_co_preadv (bs=0xf16820, offset=6556160,
>> bytes=512, qiov=0x7f712d379b40, flags=0) at block/curl.c:935
>
> The backtrace is definitely pointing at curl as being the problem.
>
> --
> Eric Blake, Principal Software Engineer
> Red Hat, Inc. +1-919-301-3266
> Virtualization: qemu.org | libvirt.org
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1708442
>
> Title:
> Crash(assert) during reading image from http url through qemu-nbd
>
> Status in QEMU:
> New
>
> Bug description:
> Description:
> During reading image from nbd device mounted by qemu-nbd server with url
> backend I/O error happens
> "blk_update_request: I/O error, dev nbd0, sector 42117" dmesg. After some
> investigation I found that qemu-nbd server aborts in aio_co_enter() assert in
> util/async.c:468.
>
> Steps to reproduce:
>
> 1) sudo go run qemu-nbd-bug-report/qemu-nbd-bug.go (see qemu-nbd-bug-
> report.tar.gz)
>
> or try directly
>
> 1) qemu-nbd -c /dev/nbd0 -r -v --aio=native -f qcow2
> json:{"file.driver":"http","file.url":"http://localhost:9666/image","file.readahead":3276800
> 2) try read whole nbd device w
Re: [Qemu-devel] [Bug 1708442] [NEW] Crash(assert) during reading image from http url through qemu-nbd
On 08/03/2017 07:12 AM, Andrey Smetanin wrote:
> Public bug reported:
>
> Description:
> During reading image from nbd device mounted by qemu-nbd server with url
> backend I/O error happens
> "blk_update_request: I/O error, dev nbd0, sector 42117" dmesg. After some
> investigation I found that qemu-nbd server aborts in aio_co_enter() assert in
> util/async.c:468.
>
Based on the backtrace, this looks to be a bug in the block/curl.c
driver, rather than the nbd/ or block/nbd.c code. If I'm right, it
should be possible to reproduce the crash using qemu-io directly on the
curl path, rather than adding the extra layer of an nbd client reading
through qemu-nbd (then again, having the qemu-nbd layer may be what is
allowing multiple parallel requests to hit the curl driver at once,
while qemu-io is not quite as easy to provoke into performing
complicated access patterns).
>
> Steps to reproduce:
>
> 1) sudo go run qemu-nbd-bug-report/qemu-nbd-bug.go (see qemu-nbd-bug-
> report.tar.gz)
>
> or try directly
>
> 1) qemu-nbd -c /dev/nbd0 -r -v --aio=native -f qcow2
> json:{"file.driver":"http","file.url":"http://localhost:9666/image","file.readahead":3276800
Presumably, you've got something serving the file at port 9666?
> 2) try read whole nbd device while error in dmesg appears x
>
> Versions:
>
> 1) qemu built from sources(/configure --target-list=x86_64-softmmu
> --disable-user --enable-curl --enable-linux-aio --enable-virtfs
> --enable-debug --disable-pie
> , top commit 5619c179057e24195ff19c8fe6d6a6cbcb16ed28):
>
> qemu-nbd -v
> qemu-nbd 2.9.90 (v2.10.0-rc0-67-g5619c17)
>
> 2) libcurl(built from sources, top commit
> 1767adf4399bb3be29121435e1bb1cc2bc05f7bf):
>
> curl -V
> curl 7.55.0-DEV (Linux) libcurl/7.55.0-DEV OpenSSL/1.0.2g zlib/1.2.8
>
>
> Backtrace:
> (gdb) bt
> #0 0x7f7131426428 in __GI_raise (sig=sig@entry=6) at
> ../sysdeps/unix/sysv/linux/raise.c:54
> #1 0x7f713142802a in __GI_abort () at abort.c:89
> #2 0x7f713141ebd7 in __assert_fail_base (fmt=,
> assertion=assertion@entry=0x54c924 "self != co",
> file=file@entry=0x54c871 "util/async.c", line=line@entry=468,
> function=function@entry=0x54c980 <__PRETTY_FUNCTION__.24766>
> "aio_co_enter") at assert.c:92
> #3 0x7f713141ec82 in __GI___assert_fail (assertion=0x54c924 "self !=
> co", file=0x54c871 "util/async.c", line=468,
> function=0x54c980 <__PRETTY_FUNCTION__.24766> "aio_co_enter") at
> assert.c:101
> #4 0x004fe6a2 in aio_co_enter (ctx=0xf0ddb0, co=0xf14650) at
> util/async.c:468
> #5 0x004fe637 in aio_co_wake (co=0xf14650) at util/async.c:456
> #6 0x00495c8a in curl_read_cb (ptr=0xf566d9, size=1, nmemb=16135,
> opaque=0xf1cb90) at block/curl.c:275
> #7 0x7f713242ac24 in Curl_client_chop_write () from
> /usr/lib/x86_64-linux-gnu/libcurl.so
> #8 0x7f713242ae03 in Curl_client_write () from
> /usr/lib/x86_64-linux-gnu/libcurl.so
> #9 0x7f713244e1cf in readwrite_data () from
> /usr/lib/x86_64-linux-gnu/libcurl.so
> #10 0x7f713244eb6f in Curl_readwrite () from
> /usr/lib/x86_64-linux-gnu/libcurl.so
> #11 0x7f713245c1bb in multi_runsingle () from
> /usr/lib/x86_64-linux-gnu/libcurl.so
> #12 0x7f713245d819 in multi_socket () from
> /usr/lib/x86_64-linux-gnu/libcurl.so
> #13 0x7f713245e067 in curl_multi_socket_action () from
> /usr/lib/x86_64-linux-gnu/libcurl.so
> #14 0x00497555 in curl_setup_preadv (bs=0xf16820, acb=0x7f712d379860)
> at block/curl.c:918
> #15 0x004975fb in curl_co_preadv (bs=0xf16820, offset=6556160,
> bytes=512, qiov=0x7f712d379b40, flags=0) at block/curl.c:935
The backtrace is definitely pointing at curl as being the problem.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
signature.asc
Description: OpenPGP digital signature
[Qemu-devel] [Bug 1708442] [NEW] Crash(assert) during reading image from http url through qemu-nbd
Public bug reported:
Description:
During reading image from nbd device mounted by qemu-nbd server with url
backend I/O error happens
"blk_update_request: I/O error, dev nbd0, sector 42117" dmesg. After some
investigation I found that qemu-nbd server aborts in aio_co_enter() assert in
util/async.c:468.
Steps to reproduce:
1) sudo go run qemu-nbd-bug-report/qemu-nbd-bug.go (see qemu-nbd-bug-
report.tar.gz)
or try directly
1) qemu-nbd -c /dev/nbd0 -r -v --aio=native -f qcow2
json:{"file.driver":"http","file.url":"http://localhost:9666/image","file.readahead":3276800
2) try read whole nbd device while error in dmesg appears x
Versions:
1) qemu built from sources(/configure --target-list=x86_64-softmmu
--disable-user --enable-curl --enable-linux-aio --enable-virtfs --enable-debug
--disable-pie
, top commit 5619c179057e24195ff19c8fe6d6a6cbcb16ed28):
qemu-nbd -v
qemu-nbd 2.9.90 (v2.10.0-rc0-67-g5619c17)
2) libcurl(built from sources, top commit
1767adf4399bb3be29121435e1bb1cc2bc05f7bf):
curl -V
curl 7.55.0-DEV (Linux) libcurl/7.55.0-DEV OpenSSL/1.0.2g zlib/1.2.8
Backtrace:
(gdb) bt
#0 0x7f7131426428 in __GI_raise (sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54
#1 0x7f713142802a in __GI_abort () at abort.c:89
#2 0x7f713141ebd7 in __assert_fail_base (fmt=,
assertion=assertion@entry=0x54c924 "self != co",
file=file@entry=0x54c871 "util/async.c", line=line@entry=468,
function=function@entry=0x54c980 <__PRETTY_FUNCTION__.24766>
"aio_co_enter") at assert.c:92
#3 0x7f713141ec82 in __GI___assert_fail (assertion=0x54c924 "self != co",
file=0x54c871 "util/async.c", line=468,
function=0x54c980 <__PRETTY_FUNCTION__.24766> "aio_co_enter") at
assert.c:101
#4 0x004fe6a2 in aio_co_enter (ctx=0xf0ddb0, co=0xf14650) at
util/async.c:468
#5 0x004fe637 in aio_co_wake (co=0xf14650) at util/async.c:456
#6 0x00495c8a in curl_read_cb (ptr=0xf566d9, size=1, nmemb=16135,
opaque=0xf1cb90) at block/curl.c:275
#7 0x7f713242ac24 in Curl_client_chop_write () from
/usr/lib/x86_64-linux-gnu/libcurl.so
#8 0x7f713242ae03 in Curl_client_write () from
/usr/lib/x86_64-linux-gnu/libcurl.so
#9 0x7f713244e1cf in readwrite_data () from
/usr/lib/x86_64-linux-gnu/libcurl.so
#10 0x7f713244eb6f in Curl_readwrite () from
/usr/lib/x86_64-linux-gnu/libcurl.so
#11 0x7f713245c1bb in multi_runsingle () from
/usr/lib/x86_64-linux-gnu/libcurl.so
#12 0x7f713245d819 in multi_socket () from
/usr/lib/x86_64-linux-gnu/libcurl.so
#13 0x7f713245e067 in curl_multi_socket_action () from
/usr/lib/x86_64-linux-gnu/libcurl.so
#14 0x00497555 in curl_setup_preadv (bs=0xf16820, acb=0x7f712d379860)
at block/curl.c:918
#15 0x004975fb in curl_co_preadv (bs=0xf16820, offset=6556160,
bytes=512, qiov=0x7f712d379b40, flags=0) at block/curl.c:935
#16 0x0047730f in bdrv_driver_preadv (bs=0xf16820, offset=6556160,
bytes=512, qiov=0x7f712d379b40, flags=0) at block/io.c:836
#17 0x00477c1f in bdrv_aligned_preadv (child=0xf1be20,
req=0x7f712d379a60, offset=6556160, bytes=512, align=1,
qiov=0x7f712d379b40, flags=0) at block/io.c:1086
#18 0x00478109 in bdrv_co_preadv (child=0xf1be20, offset=6556160,
bytes=512, qiov=0x7f712d379b40, flags=0) at block/io.c:1180
#19 0x00437498 in qcow2_co_preadv (bs=0xf0fdc0, offset=21563904,
bytes=512, qiov=0x7f712d379e80, flags=0)
at block/qcow2.c:1812
#20 0x0047730f in bdrv_driver_preadv (bs=0xf0fdc0, offset=21563904,
bytes=512, qiov=0x7f712d379e80, flags=0)
at block/io.c:836
#21 0x00477c1f in bdrv_aligned_preadv (child=0xf1c0d0,
req=0x7f712d379d30, offset=21563904, bytes=512, align=1,
qiov=0x7f712d379e80, flags=0) at block/io.c:1086
#22 0x00478109 in bdrv_co_preadv (child=0xf1c0d0, offset=21563904,
bytes=512, qiov=0x7f712d379e80, flags=0)
at block/io.c:1180
#23 0x004645ad in blk_co_preadv (blk=0xf1be90, offset=21563904,
bytes=512, qiov=0x7f712d379e80, flags=0)
at block/block-backend.c:991
#24 0x004646fa in blk_read_entry (opaque=0x7f712d379ea0) at
block/block-backend.c:1038
#25 0x0046481c in blk_prw (blk=0xf1be90, offset=21563904,
---Type to continue, or q to quit---
buf=0xf7f000
"2,NV\241t!\ti\312\vp\364\017Kl*\354\021\a\177\021\260\b\027\212\347\027\004\322\nG\340b\\\306pG\332\313\060\341;\002\360\063L\240\027T
\211\341\305\022АE\230\356DǮ}\211\bx\016\a\b\313\350\316\064.\017\372\032-R\376z\261\263\350|cQ<\016S_L\340A\221\366~L#\001+\271\204\065~\327\023\027I\211\343\361\276zT$4\336\273ˏ\353ʪ\234\016_Z|TMk\"\370\002\363~\334\332.\a\375\265mӌ{/%\304֎\374sFI\356\302\375F\340\332\324\021\202\232>\026\261\233\303tv\023\304\006\243\037\062BϏ\b\324rs\360'"...,
bytes=512, co_entry=0x4646aa , flags=0) at
block/block-backend.c:1074
#26 0x00464f81 in blk_pread (blk=0xf1be90, offset=21563904,
buf=0xf7f000, count=512) at block/block-backend.c:1227
#27 0x004906cb in nbd_trip (opaqu
