This bug appeared to surface on aws Bionic linux-aws kernel-version":
"4.15.0-1035.37", AMD64
We also see this occurring on arm64 instances a1.medium so it does not appear
to be arch specific.
AMD64 Instances: c5.large , m4.large, m5a.large, r3.large , r5.large,
t2.small and t3.medium.
This is not 100% reproducible, if affects different instance types
during each SRU cycle and appears to be random. However the failure,
although not identical bit for bit, is a buffer I/O error
I restarted a select few of the failed tests some passed completely and
some failed again.
I attempted to reproduce on m5a.large.
1.) Deploy m5a.large using cloud prov script with Bionic, ensure kernel is
4.15.0-1035.37 (in proposed at the time of writing this comment)
2.) Clone autotest and autotest-client-tests:
git clone https://github.com/autotest/autotest.git
git clone git://kernel.ubuntu.com/ubuntu/autotest-client-tests
3.) sudo apt-get update && sudo apt-get install python-pkg-resources
4.)
untu@ip-172-31-10-8:~/autotest/client$ sudo ./autotest-local -d
~/autotest-client-tests/ ~/autotest-client-
tests/ubuntu_nbd_smoke_test/control
19:42:55 INFO | Writing results to /home/ubuntu/autotest/client/results/default
19:42:55 INFO | START ---- ---- timestamp=1553283775 localtime=Mar
22 19:42:55
19:42:55 INFO | START ubuntu_nbd_smoke_test.nbd-smoke-test
ubuntu_nbd_smoke_test.nbd-smoke-test timestamp=1553283775 timeout=600
localtime=Mar 22 19:42:55
19:42:58 ERROR| [stderr] bs=1024, sz=134217728 bytes
19:42:58 ERROR| [stderr] timeout=30
19:42:58 ERROR| [stderr] e2fsck 1.44.1 (24-Mar-2018)
19:43:01 ERROR| Exception escaping from test:
Traceback (most recent call last):
File "/home/ubuntu/autotest/client/shared/test.py", line 411, in _exec
_call_test_function(self.execute, *p_args, **p_dargs)
File "/home/ubuntu/autotest/client/shared/test.py", line 823, in
_call_test_function
return func(*args, **dargs)
File "/home/ubuntu/autotest/client/shared/test.py", line 291, in execute
postprocess_profiled_run, args, dargs)
File "/home/ubuntu/autotest/client/shared/test.py", line 212, in
_call_run_once
self.run_once(*args, **dargs)
File
"/home/ubuntu/autotest-client-tests/ubuntu_nbd_smoke_test/ubuntu_nbd_smoke_test.py",
line 51, in run_once
self.results = utils.system_output(cmd, retain_output=True)
File "/home/ubuntu/autotest/client/shared/utils.py", line 1267, in
system_output
verbose=verbose, args=args).stdout
File "/home/ubuntu/autotest/client/shared/utils.py", line 918, in run
"Command returned non-zero exit status")
CmdError: Command
</home/ubuntu/autotest-client-tests/ubuntu_nbd_smoke_test/ubuntu_nbd_smoke_test.sh>
failed, rc=1, Command returned non-zero exit status
* Command:
/home/ubuntu/autotest-client-
tests/ubuntu_nbd_smoke_test/ubuntu_nbd_smoke_test.sh
Exit status: 1
Duration: 6.14667606354
stdout:
creating backing nbd image /tmp/nbd_image.img
--------------------------------------------------------------------------------
Image path: /tmp/nbd_image.img
Mount point: /mnt/nbd-test-7736
Date: Fri Mar 22 19:42:56 UTC 2019
Host: ip-172-31-10-8
Kernel: 4.15.0-1034-aws #36-Ubuntu SMP Tue Mar 5 23:17:16 UTC 2019
Machine: ip-172-31-10-8 x86_64 x86_64
CPUs online: 2
CPUs total: 2
Page size: 4096
Pages avail: 1641331
Pages total: 1977692
Free space:
Filesystem Size Used Avail Use% Mounted on
udev 3.8G 0 3.8G 0% /dev
tmpfs 773M 800K 772M 1% /run
/dev/nvme0n1p1 7.7G 1.8G 6.0G 23% /
tmpfs 3.8G 0 3.8G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
/dev/loop0 92M 92M 0 100% /snap/core/6531
/dev/loop1 18M 18M 0 100% /snap/amazon-ssm-agent/1068
tmpfs 773M 0 773M 0% /run/user/1000
--------------------------------------------------------------------------------
NBD device /dev/nbd0 created
found nbd export
NBD exports found:
test
starting client with NBD device /dev/nbd0
Negotiation: ..size = 128MB
creating ext4 on /dev/nbd0
mkfs on /dev/nbd0 succeeded after 0 attempt(s)
checking ext4 on /dev/nbd0
fsck from util-linux 2.31.1
/dev/nbd0: clean, 11/32768 files, 9787/131072 blocks
mount:
/dev/nbd0 on /mnt/nbd-test-7736 type ext4 (rw,relatime,data=ordered)
mounted on /dev/nbd0
free:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/nbd0 122835 1550 112111 2% /mnt/nbd-test-7736
creating large file /mnt/nbd-test-7736/largefile
-rw-r--r-- 1 root root 100M Mar 22 19:42 /mnt/nbd-test-7736/largefile
free:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/nbd0 122835 103951 9710 92% /mnt/nbd-test-7736
removing file /mnt/nbd-test-7736/largefile
unmounting /mnt/nbd-test-7736
stopping client
disconnect, sock, done
Found kernel warning, IO error and/or call trace
echo
[ 703.195406] creating backing nbd image /tmp/nbd_image.img
[ 704.101923] NBD device /dev/nbd0 created
[ 704.650458] found nbd export
[ 705.698030] starting client with NBD device /dev/nbd0
[ 705.702106] creating ext4 on /dev/nbd0
[ 706.093950] mkfs on /dev/nbd0 succeeded after 0 attempt(s)
[ 706.128680] checking ext4 on /dev/nbd0
[ 706.141553] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts:
(null)
[ 706.151522] mounted on /dev/nbd0
[ 706.153205] creating large file /mnt/nbd-test-7736/largefile
[ 706.645699] removing file /mnt/nbd-test-7736/largefile
[ 706.650698] unmounting /mnt/nbd-test-7736
[ 707.695006] stopping client
[ 707.696875] block nbd0: NBD_DISCONNECT
[ 707.696933] block nbd0: shutting down sockets
[ 707.697399] nbd0: detected capacity change from 0 to 134217728
[ 707.697426] print_req_error: I/O error, dev nbd0, sector 0
[ 707.699986] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.703114] print_req_error: I/O error, dev nbd0, sector 0
[ 707.705603] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.708843] print_req_error: I/O error, dev nbd0, sector 0
[ 707.711471] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.714714] ldm_validate_partition_table(): Disk read failed.
[ 707.714727] print_req_error: I/O error, dev nbd0, sector 0
[ 707.717407] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.720647] print_req_error: I/O error, dev nbd0, sector 0
[ 707.723258] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.726546] print_req_error: I/O error, dev nbd0, sector 0
[ 707.729177] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.732395] print_req_error: I/O error, dev nbd0, sector 0
[ 707.735010] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.738272] Dev nbd0: unable to read RDB block 0
[ 707.740565] print_req_error: I/O error, dev nbd0, sector 0
[ 707.743173] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.746462] print_req_error: I/O error, dev nbd0, sector 24
[ 707.749144] Buffer I/O error on dev nbd0, logical block 3, async page read
[ 707.752389] print_req_error: I/O error, dev nbd0, sector 0
[ 707.755007] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.759594] nbd0: unable to read partition table
[ 708.163286] Found kernel warning, IO error and/or call trace
[ 708.163332] echo
killing server
================================================================================
Completed
Kernel issues:
Found kernel warning, IO error and/or call trace:
TEST:
[ 703.195406] creating backing nbd image /tmp/nbd_image.img
[ 704.101923] NBD device /dev/nbd0 created
[ 704.650458] found nbd export
[ 705.698030] starting client with NBD device /dev/nbd0
[ 705.702106] creating ext4 on /dev/nbd0
[ 706.093950] mkfs on /dev/nbd0 succeeded after 0 attempt(s)
[ 706.128680] checking ext4 on /dev/nbd0
[ 706.141553] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts:
(null)
[ 706.151522] mounted on /dev/nbd0
[ 706.153205] creating large file /mnt/nbd-test-7736/largefile
[ 706.645699] removing file /mnt/nbd-test-7736/largefile
[ 706.650698] unmounting /mnt/nbd-test-7736
[ 707.695006] stopping client
[ 707.696875] block nbd0: NBD_DISCONNECT
[ 707.696933] block nbd0: shutting down sockets
[ 707.697399] nbd0: detected capacity change from 0 to 134217728
[ 707.697426] print_req_error: I/O error, dev nbd0, sector 0
[ 707.699986] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.703114] print_req_error: I/O error, dev nbd0, sector 0
[ 707.705603] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.708843] print_req_error: I/O error, dev nbd0, sector 0
[ 707.711471] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.714714] ldm_validate_partition_table(): Disk read failed.
[ 707.714727] print_req_error: I/O error, dev nbd0, sector 0
[ 707.717407] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.720647] print_req_error: I/O error, dev nbd0, sector 0
[ 707.723258] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.726546] print_req_error: I/O error, dev nbd0, sector 0
[ 707.729177] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.732395] print_req_error: I/O error, dev nbd0, sector 0
[ 707.735010] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.738272] Dev nbd0: unable to read RDB block 0
[ 707.740565] print_req_error: I/O error, dev nbd0, sector 0
[ 707.743173] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.746462] print_req_error: I/O error, dev nbd0, sector 24
[ 707.749144] Buffer I/O error on dev nbd0, logical block 3, async page read
[ 707.752389] print_req_error: I/O error, dev nbd0, sector 0
[ 707.755007] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.759594] nbd0: unable to read partition table
[ 708.163286] Found kernel warning, IO error and/or call trace
[ 708.163332] echo
stderr:
bs=1024, sz=134217728 bytes
timeout=30
e2fsck 1.44.1 (24-Mar-2018)
19:43:01 ERROR| child process failed
19:43:03 INFO | ERROR ubuntu_nbd_smoke_test.nbd-smoke-test
ubuntu_nbd_smoke_test.nbd-smoke-test timestamp=1553283783 localtime=Mar
22 19:43:03 Command
</home/ubuntu/autotest-client-tests/ubuntu_nbd_smoke_test/ubuntu_nbd_smoke_test.sh>
failed, rc=1, Command returned non-zero exit status
* Command:
/home/ubuntu/autotest-client-
tests/ubuntu_nbd_smoke_test/ubuntu_nbd_smoke_test.sh
Exit status: 1
Duration: 6.14667606354
stdout:
creating backing nbd image /tmp/nbd_image.img
--------------------------------------------------------------------------------
Image path: /tmp/nbd_image.img
Mount point: /mnt/nbd-test-7736
Date: Fri Mar 22 19:42:56 UTC 2019
Host: ip-172-31-10-8
Kernel: 4.15.0-1034-aws #36-Ubuntu SMP Tue Mar 5 23:17:16 UTC 2019
Machine: ip-172-31-10-8 x86_64 x86_64
CPUs online: 2
CPUs total: 2
Page size: 4096
Pages avail: 1641331
Pages total: 1977692
Free space:
Filesystem Size Used Avail Use% Mounted on
udev 3.8G 0 3.8G 0% /dev
tmpfs 773M 800K 772M 1% /run
/dev/nvme0n1p1 7.7G 1.8G 6.0G 23% /
tmpfs 3.8G 0 3.8G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
/dev/loop0 92M 92M 0 100% /snap/core/6531
/dev/loop1 18M 18M 0 100% /snap/amazon-ssm-agent/1068
tmpfs 773M 0 773M 0% /run/user/1000
--------------------------------------------------------------------------------
NBD device /dev/nbd0 created
found nbd export
NBD exports found:
test
starting client with NBD device /dev/nbd0
Negotiation: ..size = 128MB
creating ext4 on /dev/nbd0
mkfs on /dev/nbd0 succeeded after 0 attempt(s)
checking ext4 on /dev/nbd0
fsck from util-linux 2.31.1
/dev/nbd0: clean, 11/32768 files, 9787/131072 blocks
mount:
/dev/nbd0 on /mnt/nbd-test-7736 type ext4 (rw,relatime,data=ordered)
mounted on /dev/nbd0
free:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/nbd0 122835 1550 112111 2% /mnt/nbd-test-7736
creating large file /mnt/nbd-test-7736/largefile
-rw-r--r-- 1 root root 100M Mar 22 19:42 /mnt/nbd-test-7736/largefile
free:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/nbd0 122835 103951 9710 92% /mnt/nbd-test-7736
removing file /mnt/nbd-test-7736/largefile
unmounting /mnt/nbd-test-7736
stopping client
disconnect, sock, done
Found kernel warning, IO error and/or call trace
echo
[ 703.195406] creating backing nbd image /tmp/nbd_image.img
[ 704.101923] NBD device /dev/nbd0 created
[ 704.650458] found nbd export
[ 705.698030] starting client with NBD device /dev/nbd0
[ 705.702106] creating ext4 on /dev/nbd0
[ 706.093950] mkfs on /dev/nbd0 succeeded after 0 attempt(s)
[ 706.128680] checking ext4 on /dev/nbd0
[ 706.141553] EXT4-fs (nbd0): mounted filesystem with ordered data mode.
Opts: (null)
[ 706.151522] mounted on /dev/nbd0
[ 706.153205] creating large file /mnt/nbd-test-7736/largefile
[ 706.645699] removing file /mnt/nbd-test-7736/largefile
[ 706.650698] unmounting /mnt/nbd-test-7736
[ 707.695006] stopping client
[ 707.696875] block nbd0: NBD_DISCONNECT
[ 707.696933] block nbd0: shutting down sockets
[ 707.697399] nbd0: detected capacity change from 0 to 134217728
[ 707.697426] print_req_error: I/O error, dev nbd0, sector 0
[ 707.699986] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.703114] print_req_error: I/O error, dev nbd0, sector 0
[ 707.705603] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.708843] print_req_error: I/O error, dev nbd0, sector 0
[ 707.711471] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.714714] ldm_validate_partition_table(): Disk read failed.
[ 707.714727] print_req_error: I/O error, dev nbd0, sector 0
[ 707.717407] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.720647] print_req_error: I/O error, dev nbd0, sector 0
[ 707.723258] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.726546] print_req_error: I/O error, dev nbd0, sector 0
[ 707.729177] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.732395] print_req_error: I/O error, dev nbd0, sector 0
[ 707.735010] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.738272] Dev nbd0: unable to read RDB block 0
[ 707.740565] print_req_error: I/O error, dev nbd0, sector 0
[ 707.743173] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.746462] print_req_error: I/O error, dev nbd0, sector 24
[ 707.749144] Buffer I/O error on dev nbd0, logical block 3, async page read
[ 707.752389] print_req_error: I/O error, dev nbd0, sector 0
[ 707.755007] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.759594] nbd0: unable to read partition table
[ 708.163286] Found kernel warning, IO error and/or call trace
[ 708.163332] echo
killing server
================================================================================
Completed
Kernel issues:
Found kernel warning, IO error and/or call trace:
TEST:
[ 703.195406] creating backing nbd image /tmp/nbd_image.img
[ 704.101923] NBD device /dev/nbd0 created
[ 704.650458] found nbd export
[ 705.698030] starting client with NBD device /dev/nbd0
[ 705.702106] creating ext4 on /dev/nbd0
[ 706.093950] mkfs on /dev/nbd0 succeeded after 0 attempt(s)
[ 706.128680] checking ext4 on /dev/nbd0
[ 706.141553] EXT4-fs (nbd0): mounted filesystem with ordered data mode.
Opts: (null)
[ 706.151522] mounted on /dev/nbd0
[ 706.153205] creating large file /mnt/nbd-test-7736/largefile
[ 706.645699] removing file /mnt/nbd-test-7736/largefile
[ 706.650698] unmounting /mnt/nbd-test-7736
[ 707.695006] stopping client
[ 707.696875] block nbd0: NBD_DISCONNECT
[ 707.696933] block nbd0: shutting down sockets
[ 707.697399] nbd0: detected capacity change from 0 to 134217728
[ 707.697426] print_req_error: I/O error, dev nbd0, sector 0
[ 707.699986] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.703114] print_req_error: I/O error, dev nbd0, sector 0
[ 707.705603] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.708843] print_req_error: I/O error, dev nbd0, sector 0
[ 707.711471] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.714714] ldm_validate_partition_table(): Disk read failed.
[ 707.714727] print_req_error: I/O error, dev nbd0, sector 0
[ 707.717407] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.720647] print_req_error: I/O error, dev nbd0, sector 0
[ 707.723258] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.726546] print_req_error: I/O error, dev nbd0, sector 0
[ 707.729177] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.732395] print_req_error: I/O error, dev nbd0, sector 0
[ 707.735010] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.738272] Dev nbd0: unable to read RDB block 0
[ 707.740565] print_req_error: I/O error, dev nbd0, sector 0
[ 707.743173] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.746462] print_req_error: I/O error, dev nbd0, sector 24
[ 707.749144] Buffer I/O error on dev nbd0, logical block 3, async page read
[ 707.752389] print_req_error: I/O error, dev nbd0, sector 0
[ 707.755007] Buffer I/O error on dev nbd0, logical block 0, async page read
[ 707.759594] nbd0: unable to read partition table
[ 708.163286] Found kernel warning, IO error and/or call trace
[ 708.163332] echo
stderr:
bs=1024, sz=134217728 bytes
timeout=30
e2fsck 1.44.1 (24-Mar-2018)
19:43:03 INFO | END ERROR ubuntu_nbd_smoke_test.nbd-smoke-test
ubuntu_nbd_smoke_test.nbd-smoke-test timestamp=1553283783 localtime=Mar
22 19:43:03
19:43:03 INFO | END GOOD ---- ---- timestamp=1553283783
localtime=Mar 22 19:43:03
19:43:03 INFO | Report successfully generated at
/home/ubuntu/autotest/client/results/default/job_report.html
** Also affects: linux-aws (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1814237
Title:
ubuntu_nbd_smoke_test failed with I/O error on X-4.15
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1814237/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
