Re: AIO requests may be disordered by Qemu-kvm iothread with disk cache=writethrough, Bug or Feature?

[Stefan Hajnoczi]

On Thu, Oct 08, 2015 at 07:59:56PM +0800, charlie.song wrote:
> We recently try to use Linux AIO from guest OS and find that the IOthread 
> mechanism of Qemu-KVM will reorder I/O requests from guest OS 
> even when the AIO write requests are issued from a single thread in order. 
> This does not happen on the host OS however.

I think you are describing a situation where a guest submits multiple
overlapping I/O requests at the same time.

virtio-blk does not guarantee a specific request ordering, so the
application needs to wait for request completion if ordering matters.

io_submit(2) also does not make guarantees about ordering.

Stefan
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

AIO requests may be disordered by Qemu-kvm iothread with disk cache=writethrough, Bug or Feature?

[charlie.song]

Dear KVM Developers: 
I am Xiang Song from UCloud company. We currently encounter a weird 
phenomenon about Qemu-KVM IOthread. 
We recently try to use Linux AIO from guest OS and find that the IOthread 
mechanism of Qemu-KVM will reorder I/O requests from guest OS 
even when the AIO write requests are issued from a single thread in order. This 
does not happen on the host OS however.
We are not sure whether this is a feature of Qemu-KVM IOthread mechanism or 
a Bug.
 
The testbd is as following: (the guest disk device cache is configured to 
writethrough.)
CPU: Intel(R) Xeon(R) CPU E5-2650
QEMU version: 1.5.3
Host/Guest Kernel:  Both Linux 4.1.8 & Linux 2.6.32, OS type CentOS 6.5
Simplified Guest OS qemu cmd:  
/usr/libexec/qemu-kvm -machine rhel6.3.0,accel=kvm,usb=off -cpu kvm64 -smp 
8,sockets=8,cores=1,threads=1 
-drive 
file=/var/lib/libvirt/images/song-disk.img,if=none,id=drive-virtio-disk0,format=qcow2,serial=UCLOUD_DISK_VDA,cache=writethrough
 
-device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:22:d5:52,bus=pci.0,addr=0x4

The test code triggerring this phenomenon work as following: it use linux aio 
API to issue concurrent async write requests to a file. During exection it will 
continuously write data into target test file. There are total 'X' jobs, and 
each job is assigned a job id JOB_ID which starts from 0. Each job will write 
16 * 512
Byte data into the target file at offset =  JOB_ID * 512. (the data is repeated 
uint64_t  JOB_ID). 
There is only one thread handling 'X' jobs one by one through Linux AIO 
(io_submit) cmd. When handling jobs, it will continuously 
issuing AIO requests without waiting for AIO Callbacks. When it finishes, the 
file should look like:
 [00][1...1][2...2][3...3]...[X-1...X-1]
Then we use a check program to test the resulting file, it can continuously 
read the first 8 byte (uint64_t) of each sector and print it out. In normal 
cases,
 it's output is like:
  0 1 2 3  X-1

Exec  output: (Set X=32)
In our guest OS, the output is abnormal: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 
16 17 18 18 18 18 18 18 24 25 26 27 28 29 30 31. 
It can be seen that job20~job24 are overwrited by job19.
In our host OS, the output is as expected, 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31.


I can provide the example code if needed.

Best regards, song

2015-10-08


charlie.song 
  
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: AIO requests may be disordered by Qemu-kvm iothread with disk cache=writethrough, Bug or Feature?

[Fam Zheng]

On Thu, 10/08 19:59, charlie.song wrote:
> Dear KVM Developers: 
> I am Xiang Song from UCloud company. We currently encounter a weird 
> phenomenon about Qemu-KVM IOthread. 
> We recently try to use Linux AIO from guest OS and find that the IOthread 
> mechanism of Qemu-KVM will reorder I/O requests from guest OS 
> even when the AIO write requests are issued from a single thread in order. 
> This does not happen on the host OS however.
> We are not sure whether this is a feature of Qemu-KVM IOthread mechanism 
> or a Bug.
>  
> The testbd is as following: (the guest disk device cache is configured to 
> writethrough.)
> CPU: Intel(R) Xeon(R) CPU E5-2650
> QEMU version: 1.5.3
> Host/Guest Kernel:  Both Linux 4.1.8 & Linux 2.6.32, OS type CentOS 6.5
> Simplified Guest OS qemu cmd:  
> /usr/libexec/qemu-kvm -machine rhel6.3.0,accel=kvm,usb=off -cpu kvm64 -smp 
> 8,sockets=8,cores=1,threads=1 
> -drive 
> file=/var/lib/libvirt/images/song-disk.img,if=none,id=drive-virtio-disk0,format=qcow2,serial=UCLOUD_DISK_VDA,cache=writethrough
>  
> -device 
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:22:d5:52,bus=pci.0,addr=0x4

You mentioned iothread above but it's not in your command line?

> 
> The test code triggerring this phenomenon work as following: it use linux aio 
> API to issue concurrent async write requests to a file. During exection it 
> will 
> continuously write data into target test file. There are total 'X' jobs, and 
> each job is assigned a job id JOB_ID which starts from 0. Each job will write 
> 16 * 512
> Byte data into the target file at offset =  JOB_ID * 512. (the data is 
> repeated uint64_t  JOB_ID). 
> There is only one thread handling 'X' jobs one by one through Linux AIO 
> (io_submit) cmd. When handling jobs, it will continuously 
> issuing AIO requests without waiting for AIO Callbacks. When it finishes, the 
> file should look like:
>  [00][1...1][2...2][3...3]...[X-1...X-1]
> Then we use a check program to test the resulting file, it can 
> continuously read the first 8 byte (uint64_t) of each sector and print it 
> out. In normal cases,
>  it's output is like:
>   0 1 2 3  X-1
> 
> Exec  output: (Set X=32)
> In our guest OS, the output is abnormal: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 
> 15 16 17 18 18 18 18 18 18 24 25 26 27 28 29 30 31. 
> It can be seen that job20~job24 are overwrited by job19.
> In our host OS, the output is as expected, 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 
> 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31.

I'm not 100% sure but I don't think the returning of io_submit guarantees any
ordering, usually you need to wait for the callback to ensure that.

Fam

> 
> 
> I can provide the example code if needed.
> 
> Best regards, song
> 
> 2015-10-08
> 
> 
> charlie.song 
>   
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Re: AIO requests may be disordered by Qemu-kvm iothread with disk cache=writethrough, Bug or Feature?

[Fam Zheng]

On Fri, 10/09 11:25, charlie.song wrote:
> At 2015-10-08 23:37:02, "Fam Zheng"  wrote:
> >On Thu, 10/08 19:59, charlie.song wrote:
> >> Dear KVM Developers: 
> >> I am Xiang Song from UCloud company. We currently encounter a weird 
> >> phenomenon about Qemu-KVM IOthread. 
> >> We recently try to use Linux AIO from guest OS and find that the 
> >> IOthread mechanism of Qemu-KVM will reorder I/O requests from guest OS 
> >> even when the AIO write requests are issued from a single thread in order. 
> >> This does not happen on the host OS however.
> >> We are not sure whether this is a feature of Qemu-KVM IOthread 
> >> mechanism or a Bug.
> >>  
> >> The testbd is as following: (the guest disk device cache is configured to 
> >> writethrough.)
> >> CPU: Intel(R) Xeon(R) CPU E5-2650
> >> QEMU version: 1.5.3
> >> Host/Guest Kernel:  Both Linux 4.1.8 & Linux 2.6.32, OS type CentOS 6.5
> >> Simplified Guest OS qemu cmd:  
> >> /usr/libexec/qemu-kvm -machine rhel6.3.0,accel=kvm,usb=off -cpu kvm64 -smp 
> >> 8,sockets=8,cores=1,threads=1 
> >> -drive 
> >> file=/var/lib/libvirt/images/song-disk.img,if=none,id=drive-virtio-disk0,format=qcow2,serial=UCLOUD_DISK_VDA,cache=writethrough
> >>  
> >> -device 
> >> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:22:d5:52,bus=pci.0,addr=0x4
> >
> >You mentioned iothread above but it's not in your command line?
> I means the thread pool mechanism used by qemu-kvm to accelerate I/O
> processing.This is used by paio_submit (block/raw-posix.c) by default and
> with pool->max_threads = 64 as I know. (qemu-kvm version 1.5.3)

The thread pool parallism may reorder non-overlapping requests, but it
shouldn't cause any reordering of overlapping requests like the case in your io
pattern. QEMU ensures that.

Do you see this with aio=native?

Fam

> 
> >
> >> 
> >> The test code triggerring this phenomenon work as following: it use linux 
> >> aio API to issue concurrent async write requests to a file. During 
> >> exection it will 
> >> continuously write data into target test file. There are total 'X' jobs, 
> >> and each job is assigned a job id JOB_ID which starts from 0. Each job 
> >> will write 16 * 512
> >> Byte data into the target file at offset =  JOB_ID * 512. (the data is 
> >> repeated uint64_t  JOB_ID). 
> >> There is only one thread handling 'X' jobs one by one through Linux 
> >> AIO (io_submit) cmd. When handling jobs, it will continuously 
> >> issuing AIO requests without waiting for AIO Callbacks. When it finishes, 
> >> the file should look like:
> >>  [00][1...1][2...2][3...3]...[X-1...X-1]
> >> Then we use a check program to test the resulting file, it can 
> >> continuously read the first 8 byte (uint64_t) of each sector and print it 
> >> out. In normal cases,
> >>  it's output is like:
> >>   0 1 2 3  X-1
> >> 
> >> Exec  output: (Set X=32)
> >> In our guest OS, the output is abnormal: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 
> >> 14 15 16 17 18 18 18 18 18 18 24 25 26 27 28 29 30 31. 
> >> It can be seen that job20~job24 are overwrited by job19.
> >> In our host OS, the output is as expected, 0 1 2 3 4 5 6 7 8 9 10 11 12 13 
> >> 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31.
> >
> >I'm not 100% sure but I don't think the returning of io_submit guarantees any
> >ordering, usually you need to wait for the callback to ensure that.
> Is there any proof or artical about the ordering of io_submit requests?
> >
> >Fam
> >
> >> 
> >> 
> >> I can provide the example code if needed.
> >> 
> >> Best regards, song
> >> 
> >> 2015-10-08
> >> 
> >> 
> >> charlie.song 
> >>   
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe kvm" in
> >> the body of a message to majord...@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Re: AIO requests may be disordered by Qemu-kvm iothread with disk cache=writethrough, Bug or Feature?

[charlie.song]

At 2015-10-08 23:37:02, "Fam Zheng"  wrote:
>On Thu, 10/08 19:59, charlie.song wrote:
>> Dear KVM Developers: 
>> I am Xiang Song from UCloud company. We currently encounter a weird 
>> phenomenon about Qemu-KVM IOthread. 
>> We recently try to use Linux AIO from guest OS and find that the 
>> IOthread mechanism of Qemu-KVM will reorder I/O requests from guest OS 
>> even when the AIO write requests are issued from a single thread in order. 
>> This does not happen on the host OS however.
>> We are not sure whether this is a feature of Qemu-KVM IOthread mechanism 
>> or a Bug.
>>  
>> The testbd is as following: (the guest disk device cache is configured to 
>> writethrough.)
>> CPU: Intel(R) Xeon(R) CPU E5-2650
>> QEMU version: 1.5.3
>> Host/Guest Kernel:  Both Linux 4.1.8 & Linux 2.6.32, OS type CentOS 6.5
>> Simplified Guest OS qemu cmd:  
>> /usr/libexec/qemu-kvm -machine rhel6.3.0,accel=kvm,usb=off -cpu kvm64 -smp 
>> 8,sockets=8,cores=1,threads=1 
>> -drive 
>> file=/var/lib/libvirt/images/song-disk.img,if=none,id=drive-virtio-disk0,format=qcow2,serial=UCLOUD_DISK_VDA,cache=writethrough
>>  
>> -device 
>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:22:d5:52,bus=pci.0,addr=0x4
>
>You mentioned iothread above but it's not in your command line?
I means the thread pool mechanism used by qemu-kvm to accelerate I/O 
processing.This is used by paio_submit (block/raw-posix.c) by default and with
pool->max_threads = 64 as I know. (qemu-kvm version 1.5.3)

>
>> 
>> The test code triggerring this phenomenon work as following: it use linux 
>> aio API to issue concurrent async write requests to a file. During exection 
>> it will 
>> continuously write data into target test file. There are total 'X' jobs, and 
>> each job is assigned a job id JOB_ID which starts from 0. Each job will 
>> write 16 * 512
>> Byte data into the target file at offset =  JOB_ID * 512. (the data is 
>> repeated uint64_t  JOB_ID). 
>> There is only one thread handling 'X' jobs one by one through Linux AIO 
>> (io_submit) cmd. When handling jobs, it will continuously 
>> issuing AIO requests without waiting for AIO Callbacks. When it finishes, 
>> the file should look like:
>>  [00][1...1][2...2][3...3]...[X-1...X-1]
>> Then we use a check program to test the resulting file, it can 
>> continuously read the first 8 byte (uint64_t) of each sector and print it 
>> out. In normal cases,
>>  it's output is like:
>>   0 1 2 3  X-1
>> 
>> Exec  output: (Set X=32)
>> In our guest OS, the output is abnormal: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 
>> 15 16 17 18 18 18 18 18 18 24 25 26 27 28 29 30 31. 
>> It can be seen that job20~job24 are overwrited by job19.
>> In our host OS, the output is as expected, 0 1 2 3 4 5 6 7 8 9 10 11 12 13 
>> 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31.
>
>I'm not 100% sure but I don't think the returning of io_submit guarantees any
>ordering, usually you need to wait for the callback to ensure that.
Is there any proof or artical about the ordering of io_submit requests?
>
>Fam
>
>> 
>> 
>> I can provide the example code if needed.
>> 
>> Best regards, song
>> 
>> 2015-10-08
>> 
>> 
>> charlie.song 
>>   
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Re: Re: AIO requests may be disordered by Qemu-kvm iothread with disk cache=writethrough, Bug or Feature?

[charlie.song]

At 2015-10-09 12:33:20, "charlie.song"  wrote:
>At 2015-10-09 12:16:03, "Fam Zheng"  wrote:
>>On Fri, 10/09 11:25, charlie.song wrote:
>>> At 2015-10-08 23:37:02, "Fam Zheng"  wrote:
>>> >On Thu, 10/08 19:59, charlie.song wrote:
>>> >> Dear KVM Developers: 
>>> >> I am Xiang Song from UCloud company. We currently encounter a weird 
>>> >> phenomenon about Qemu-KVM IOthread. 
>>> >> We recently try to use Linux AIO from guest OS and find that the 
>>> >> IOthread mechanism of Qemu-KVM will reorder I/O requests from guest OS 
>>> >> even when the AIO write requests are issued from a single thread in 
>>> >> order. This does not happen on the host OS however.
>>> >> We are not sure whether this is a feature of Qemu-KVM IOthread 
>>> >> mechanism or a Bug.
>>> >>  
>>> >> The testbd is as following: (the guest disk device cache is configured 
>>> >> to writethrough.)
>>> >> CPU: Intel(R) Xeon(R) CPU E5-2650
>>> >> QEMU version: 1.5.3
>>> >> Host/Guest Kernel:  Both Linux 4.1.8 & Linux 2.6.32, OS type CentOS 6.5
>>> >> Simplified Guest OS qemu cmd:  
>>> >> /usr/libexec/qemu-kvm -machine rhel6.3.0,accel=kvm,usb=off -cpu kvm64 
>>> >> -smp 8,sockets=8,cores=1,threads=1 
>>> >> -drive 
>>> >> file=/var/lib/libvirt/images/song-disk.img,if=none,id=drive-virtio-disk0,format=qcow2,serial=UCLOUD_DISK_VDA,cache=writethrough
>>> >>  
>>> >> -device 
>>> >> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:22:d5:52,bus=pci.0,addr=0x4
>>> >
>>> >You mentioned iothread above but it's not in your command line?
>>> I means the thread pool mechanism used by qemu-kvm to accelerate I/O
>>> processing.This is used by paio_submit (block/raw-posix.c) by default and
>>> with pool->max_threads = 64 as I know. (qemu-kvm version 1.5.3)
>>
>>The thread pool parallism may reorder non-overlapping requests, but it
>>shouldn't cause any reordering of overlapping requests like the case in your 
>>io
>>pattern. QEMU ensures that.
>>
>>Do you see this with aio=native?
>We see this with both aio=threads and aio=native.
>Are you sure "QEMU ensures the ordering of overlapping requests"? 
>I can provide the example code if needed.
If I configure the guest disk IO with "cache=directsync,aio=native", this 
phenomenon disappears.

>
>>
>>Fam
>>
>>> 
>>> >
>>> >> 
>>> >> The test code triggerring this phenomenon work as following: it use 
>>> >> linux aio API to issue concurrent async write requests to a file. During 
>>> >> exection it will 
>>> >> continuously write data into target test file. There are total 'X' jobs, 
>>> >> and each job is assigned a job id JOB_ID which starts from 0. Each job 
>>> >> will write 16 * 512
>>> >> Byte data into the target file at offset =  JOB_ID * 512. (the data is 
>>> >> repeated uint64_t  JOB_ID). 
>>> >> There is only one thread handling 'X' jobs one by one through Linux 
>>> >> AIO (io_submit) cmd. When handling jobs, it will continuously 
>>> >> issuing AIO requests without waiting for AIO Callbacks. When it 
>>> >> finishes, the file should look like:
>>> >>  [00][1...1][2...2][3...3]...[X-1...X-1]
>>> >> Then we use a check program to test the resulting file, it can 
>>> >> continuously read the first 8 byte (uint64_t) of each sector and print 
>>> >> it out. In normal cases,
>>> >>  it's output is like:
>>> >>   0 1 2 3  X-1
>>> >> 
>>> >> Exec  output: (Set X=32)
>>> >> In our guest OS, the output is abnormal: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 
>>> >> 14 15 16 17 18 18 18 18 18 18 24 25 26 27 28 29 30 31. 
>>> >> It can be seen that job20~job24 are overwrited by job19.
>>> >> In our host OS, the output is as expected, 0 1 2 3 4 5 6 7 8 9 10 11 12 
>>> >> 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31.
>>> >
>>> >I'm not 100% sure but I don't think the returning of io_submit guarantees 
>>> >any
>>> >ordering, usually you need to wait for the callback to ensure that.
>>> Is there any proof or artical about the ordering of io_submit requests?
>>> >
>>> >Fam
>>> >
>>> >> 
>>> >> 
>>> >> I can provide the example code if needed.
>>> >> 
>>> >> Best regards, song
>>> >> 
>>> >> 2015-10-08
>>> >> 
>>> >> 
>>> >> charlie.song 
>>> >>   
>>> >> --
>>> >> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> >> the body of a message to majord...@vger.kernel.org
>>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> 
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majord...@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Re: Re: AIO requests may be disordered by Qemu-kvm iothread with disk cache=writethrough, Bug or Feature?

[charlie.song]

At 2015-10-09 12:16:03, "Fam Zheng"  wrote:
>On Fri, 10/09 11:25, charlie.song wrote:
>> At 2015-10-08 23:37:02, "Fam Zheng"  wrote:
>> >On Thu, 10/08 19:59, charlie.song wrote:
>> >> Dear KVM Developers: 
>> >> I am Xiang Song from UCloud company. We currently encounter a weird 
>> >> phenomenon about Qemu-KVM IOthread. 
>> >> We recently try to use Linux AIO from guest OS and find that the 
>> >> IOthread mechanism of Qemu-KVM will reorder I/O requests from guest OS 
>> >> even when the AIO write requests are issued from a single thread in 
>> >> order. This does not happen on the host OS however.
>> >> We are not sure whether this is a feature of Qemu-KVM IOthread 
>> >> mechanism or a Bug.
>> >>  
>> >> The testbd is as following: (the guest disk device cache is configured to 
>> >> writethrough.)
>> >> CPU: Intel(R) Xeon(R) CPU E5-2650
>> >> QEMU version: 1.5.3
>> >> Host/Guest Kernel:  Both Linux 4.1.8 & Linux 2.6.32, OS type CentOS 6.5
>> >> Simplified Guest OS qemu cmd:  
>> >> /usr/libexec/qemu-kvm -machine rhel6.3.0,accel=kvm,usb=off -cpu kvm64 
>> >> -smp 8,sockets=8,cores=1,threads=1 
>> >> -drive 
>> >> file=/var/lib/libvirt/images/song-disk.img,if=none,id=drive-virtio-disk0,format=qcow2,serial=UCLOUD_DISK_VDA,cache=writethrough
>> >>  
>> >> -device 
>> >> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:22:d5:52,bus=pci.0,addr=0x4
>> >
>> >You mentioned iothread above but it's not in your command line?
>> I means the thread pool mechanism used by qemu-kvm to accelerate I/O
>> processing.This is used by paio_submit (block/raw-posix.c) by default and
>> with pool->max_threads = 64 as I know. (qemu-kvm version 1.5.3)
>
>The thread pool parallism may reorder non-overlapping requests, but it
>shouldn't cause any reordering of overlapping requests like the case in your io
>pattern. QEMU ensures that.
>
>Do you see this with aio=native?
We see this with both aio=threads and aio=native.
Are you sure "QEMU ensures the ordering of overlapping requests"? 
I can provide the example code if needed.

>
>Fam
>
>> 
>> >
>> >> 
>> >> The test code triggerring this phenomenon work as following: it use linux 
>> >> aio API to issue concurrent async write requests to a file. During 
>> >> exection it will 
>> >> continuously write data into target test file. There are total 'X' jobs, 
>> >> and each job is assigned a job id JOB_ID which starts from 0. Each job 
>> >> will write 16 * 512
>> >> Byte data into the target file at offset =  JOB_ID * 512. (the data is 
>> >> repeated uint64_t  JOB_ID). 
>> >> There is only one thread handling 'X' jobs one by one through Linux 
>> >> AIO (io_submit) cmd. When handling jobs, it will continuously 
>> >> issuing AIO requests without waiting for AIO Callbacks. When it finishes, 
>> >> the file should look like:
>> >>  [00][1...1][2...2][3...3]...[X-1...X-1]
>> >> Then we use a check program to test the resulting file, it can 
>> >> continuously read the first 8 byte (uint64_t) of each sector and print it 
>> >> out. In normal cases,
>> >>  it's output is like:
>> >>   0 1 2 3  X-1
>> >> 
>> >> Exec  output: (Set X=32)
>> >> In our guest OS, the output is abnormal: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 
>> >> 14 15 16 17 18 18 18 18 18 18 24 25 26 27 28 29 30 31. 
>> >> It can be seen that job20~job24 are overwrited by job19.
>> >> In our host OS, the output is as expected, 0 1 2 3 4 5 6 7 8 9 10 11 12 
>> >> 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31.
>> >
>> >I'm not 100% sure but I don't think the returning of io_submit guarantees 
>> >any
>> >ordering, usually you need to wait for the callback to ensure that.
>> Is there any proof or artical about the ordering of io_submit requests?
>> >
>> >Fam
>> >
>> >> 
>> >> 
>> >> I can provide the example code if needed.
>> >> 
>> >> Best regards, song
>> >> 
>> >> 2015-10-08
>> >> 
>> >> 
>> >> charlie.song 
>> >>   
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> >> the body of a message to majord...@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html