Hi Mikhail,
Yes, io_uring can help Nginx achieve same performance with lower resource cost.
This is the key improvement of io_uring I think. It can't break through the HW
limitation.
BR,
Ping
-----Original Message-----
From: nginx-devel <nginx-devel-boun...@nginx.org> On Behalf Of Mikhail
Isachenkov
Sent: Friday, February 26, 2021 4:41 PM
To: nginx-devel@nginx.org
Subject: Re: [PATCH] Add io_uring support in AIO(async io) module
Hi Zhao Ping,
I created i3en.6xlarge instance, 2 SSDs with 250k iops each (in raid0) and
reproduced your results. Please find dstat output below.
1 worker, io_uring:
usr sys idl wai stl| read writ| recv send| in out | int csw
4 4 92 0 0|1412M 0 |5742k 1484M| 0 0 | 244k 87k
1 worker, libaio:
usr sys idl wai stl| read writ| recv send| in out | int csw
1 1 95 3 0| 276M 0 |1386k 289M| 0 0 | 50k 8961
4 workers, io_uring:
usr sys idl wai stl| read writ| recv send| in out | int csw
6 6 18 70 0|1349M 0 |7240k 1445M| 0 0 | 296k 120k
4 workers, libaio:
usr sys idl wai stl| read writ| recv send| in out | int csw
3 2 82 13 0| 890M 0 |3570k 931M| 0 0 | 139k 31k
I ran test on i3.metal instance too (with 8 SSD in raid0), results are
different but io_uring still faster:
1 worker, io_uring:
usr sys idl wai stl| read writ| recv send| in out | int csw
1 2 97 0 0|1372M 0 |5845k 1442M| 0 0 | 168k 51k
1 worker, libaio:
usr sys idl wai stl| read writ| recv send| in out | int csw
1 1 98 0 0| 972M 20k|4083k 1023M| 0 0 | 126k 2014
4 workers, io_uring:
usr sys idl wai stl| read writ| recv send| in out | int csw
3 4 92 0 0|2211M 0 |9314k 2346M| 0 0 | 251k 198k
4 workers, libaio:
usr sys idl wai stl| read writ| recv send| in out | int csw
3 3 94 0 0|1867M 0 | 10M 1995M| 0 0 | 220k 38k
There is no difference between libaio and io_uring with a large number of
worker processes (>=12).
26.02.2021 04:22, Zhao, Ping пишет:
> Hi Mikhail,
>
> My Nvme SSD reports 562k IOPS with bs 4k, 128 depth using io_uring.
>
> ./fio -name=fiotest -filename=/dev/nvme2n1 -iodepth=128 -thread
> -rw=randread -ioengine=io_uring -sqthread_poll=1 -direct=1 -bs=4k
> -size=10G -numjobs=1 -runtime=600 -group_reporting
>
> Jobs: 1 (f=1): [r(1)][100.0%][r=2173MiB/s][r=556k IOPS][eta 00m:00s]
> fiotest: (groupid=0, jobs=1): err= 0: pid=23828: Fri Feb 26 03:55:40 2021
> read: IOPS=562k, BW=2196MiB/s (2303MB/s)(10.0GiB/4663msec)
>
> BR,
> Ping
>
> -----Original Message-----
> From: nginx-devel <nginx-devel-boun...@nginx.org> On Behalf Of Mikhail
> Isachenkov
> Sent: Thursday, February 25, 2021 7:01 PM
> To: nginx-devel@nginx.org
> Subject: Re: [PATCH] Add io_uring support in AIO(async io) module
>
> Hi Zhao Ping,
>
> Looks like general-purpose AWS EC2 instances does not optimized to high
> random I/O even with NVMe SSD; I'll try to test it again on bare-metal,
> storage-optimized instance.
>
> How many 4k random iops your storage can handle? (I'd like to run test on the
> instance with the same storage performance, according to
> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-optimized-instances.html#i2-instances-diskperf).
>
> Thanks in advance!
>
> 25.02.2021 09:59, Zhao, Ping пишет:
>> Hi Mikhail,
>>
>> I tried with CentOS 8.1.1911 + fedora-kernel-5.10.10-200.fc33, and with your
>> test steps & scripts:
>>
>> 1. Created 90k files on nvme SSD with each size 100KB.
>> 2. Created separate cgroup 'nginx': mkdir /sys/fs/cgroup/memory/nginx
>> 3. Limit Nginx to 2GB: echo 2G >
>> /sys/fs/cgroup/memory/nginx/memory.limit_in_bytes
>> 4. Clear cache: echo 3 > /proc/sys/vm/drop_caches 5. Start nginx:
>> cgexec -g memory:nginx 6. Tested with wrk on client: ./wrk -d 30 -t
>> 100 -c 1000 -s add_random.lua http://...
>>
>> Io_uring can achieved 1.9+GB/s
>> usr sys idl wai stl| read writ| recv send| in out | int csw
>> 1 2 97 0 0|1923M 0 |7369k 2136M| 0 0 | 263k 119k
>> 1 2 97 0 0|1911M 0 |7329k 2123M| 0 0 | 262k 119k
>> 1 2 97 0 0|1910M 3592k|7318k 2118M| 0 0 | 264k 117k
>> 1 2 97 0 0|1923M 200k|7353k 2138M| 0 0 | 265k 118k
>> 1 2 97 0 0|1929M 0 |7376k 2142M| 0 0 | 264k 118k
>> 1 2 97 0 0|1924M 32k|7352k 2140M| 0 0 | 265k 118k
>> 1 2 97 0 0|1913M 0 |7320k 2122M| 0 0 | 263k 117k
>> 1 2 97 0 0|1921M 3544k|7336k 2132M| 0 0 | 264k 118k
>> 1 2 97 0 0|1933M 0 |7353k 2146M| 0 0 | 264k 120k
>>
>> Libaio is 260+MB/s
>> usr sys idl wai stl| read writ| recv send| in out | int csw
>> 0 0 98 1 0| 267M 0 |1022k 293M| 0 0 | 41k 12k
>> 0 0 98 1 0| 279M 740k|1093k 314M| 0 0 | 42k 12k
>> 1 0 98 1 0| 268M 0 |1013k 294M| 0 0 | 41k 12k
>> 0 0 98 1 0| 261M 0 |1002k 294M| 0 0 | 41k 12k
>> 1 0 98 1 0| 285M 0 |1057k 310M| 0 0 | 43k 13k
>> 0 0 98 1 0| 276M 4096B|1038k 307M| 0 0 | 42k 13k
>> 0 0 98 1 0| 273M 780k|1019k 303M| 0 0 | 42k 12k
>> 1 0 98 1 0| 275M 0 |1016k 305M| 0 0 | 42k 12k
>> 0 0 98 1 0| 254M 0 | 976k 294M| 0 0 | 40k 12k
>> 0 0 98 1 0| 265M 0 | 975k 293M| 0 0 | 41k 12k
>> 0 0 98 1 0| 269M 0 | 976k 295M| 0 0 | 41k 12k
>>
>> Comparing with your dstat data that disk read keeps at 250M/s, is there
>> anything locked the disk io bw?
>>
>> Regards,
>> Ping
>>
>> -----Original Message-----
>> From: nginx-devel <nginx-devel-boun...@nginx.org> On Behalf Of
>> Mikhail Isachenkov
>> Sent: Tuesday, February 9, 2021 9:31 PM
>> To: nginx-devel@nginx.org
>> Subject: Re: [PATCH] Add io_uring support in AIO(async io) module
>>
>> Hi Zhao Ping,
>>
>> Unfortunately, I still couldn't reproduce these results. Maybe you could
>> point me where I'm wrong? Please find my steps below and configuration/lua
>> script for wrk attached.
>>
>> 1. Create 90k files on SSD on Amazon EC2 instance. I created 1k, 100k, 1M
>> files.
>> 2. Create separate cgroup 'nginx': mkdir /sys/fs/cgroup/memory/nginx
>> 3. Limit memory to 80 Mb, for example: echo
>> 80M>/sys/fs/cgroup/memory/nginx/memory.limit_in_bytes
>> 4. Disable limit for locked memory: ulimit -l unlimited 5. Start nginx:
>> cgexec -g memory:nginx /usr/local/sbin/nginx 6. Run wrk on client: ./wrk -d
>> 30 -t 100 -c 1000 -s add_random.lua http://...
>>
>> I tried different values for limit_in_bytes (from 80M to 2G) and different
>> file sizes -- 1k, 100k, 1M. In fact, maximum bandwidth is the same with
>> libaio and io_uring.
>>
>> For example, with 100kb files and 1 worker process:
>>
>> free -lh
>> total used free shared buff/cache
>> available
>> Mem: 15Gi 212Mi 14Gi 13Mi 318Mi
>> 14Gi
>>
>> dstat/libaio
>> 5 6 73 17 0| 251M 0 |1253k 265M| 0 0 | 33k 1721
>> 4 4 73 17 0| 250M 0 |1267k 264M| 0 0 | 33k 1739
>> 6 5 72 16 0| 250M 924k|1308k 270M| 0 0 | 34k 2017
>> 5 5 72 17 0| 250M 0 |1277k 258M| 0 0 | 34k 1945
>> 5 5 73 17 0| 250M 0 |1215k 263M| 0 0 | 33k 1720
>> 5 5 72 16 0| 250M 0 |1311k 267M| 0 0 | 34k 1721
>> 5 5 73 16 0| 250M 0 |1280k 264M| 0 0 | 34k 1718
>> 6 6 72 16 0| 250M 24k|1362k 268M| 0 0 | 35k 1825
>> 5 5 73 17 0| 250M 0 |1342k 262M| 0 0 | 34k 1726
>> dstat/io_uring
>> 5 6 60 29 0| 250M 0 |1079k 226M| 0 0 | 36k 10k
>> 5 6 64 25 0| 251M 0 | 906k 204M| 0 0 | 32k 8607
>> 4 6 62 27 0| 250M 0 |1034k 221M| 0 0 | 35k 10k
>> 5 6 63 26 0| 250M 20k| 909k 209M| 0 0 | 32k 8595
>> 4 6 62 27 0| 250M 0 |1003k 217M| 0 0 | 35k 10k
>> 4 5 61 28 0| 250M 0 |1019k 226M| 0 0 | 35k 9700
>> 4 5 62 27 0| 250M 0 | 948k 210M| 0 0 | 32k 8433
>> 4 6 61 28 0| 250M 0 |1094k 216M| 0 0 | 35k 9811
>> 5 6 62 26 0| 250M 0 |1083k 226M| 0 0 | 35k 9479
>>
>> As you can see, libaio even faster a bit.
>>
>> 09.02.2021 11:36, Zhao, Ping пишет:
>>> Hi Mikhail,
>>>
>>> The performance improvement of Io_uring vs. libaio locates at disk io
>>> interface. So it needs exclude other factors when test, such as memory
>>> cache storage which is much faster than disk io.
>>>
>>> If I didn't use memory limitation, libaio and io_uring network bandwidth is
>>> very close because both of them use memory as cache file location, so we
>>> can't see the disk io change from it. In following data, as example, it
>>> used 17G memory as cache, network speed is same of io_uring and libaio,
>>> both of them has very few disk io load, which means very low
>>> io_uring/libaio usage.
>>>
>>> memory
>>> free -lh
>>> total used free shared buff/cache
>>> available
>>> Mem: 376Gi 3.2Gi 356Gi 209Mi 17Gi
>>> 370Gi
>>>
>>> libaio:
>>> ----total-usage---- -dsk/total- -net/total- ---paging-- ---system--
>>> usr sys idl wai stl| read writ| recv send| in out | int csw
>>> 1 1 99 0 0|4097B 80k|4554k 104M| 0 0 | 77k 1344
>>> 1 1 98 0 0|8192B 104k|9955k 236M| 0 0 | 151k 1449
>>> 1 1 97 0 0| 56k 32k| 10M 241M| 0 0 | 148k 1652
>>> 2 1 97 0 0| 16k 16k|9552k 223M| 0 0 | 142k 1366
>>> 1 1 97 0 0| 16k 24k|9959k 234M| 0 0 | 146k 1570
>>> 1 1 97 0 0| 0 1064k| 10M 237M| 0 0 | 150k 1472
>>> 2 1 97 0 0| 16k 48k|9650k 227M| 0 0 | 143k 1555
>>> 2 1 97 0 0| 12k 16k|9185k 216M| 0 0 | 139k 1304
>>>
>>> Io_uring:
>>> ----total-usage---- -dsk/total- -net/total- ---paging-- ---system--
>>> usr sys idl wai stl| read writ| recv send| in out | int csw
>>> 2 1 97 0 0| 0 0 |9866k 232M| 0 0 | 148k 1286
>>> 2 1 97 0 0| 0 0 |9388k 220M| 0 0 | 144k 1345
>>> 2 1 97 0 0| 0 0 |9080k 213M| 0 0 | 137k 1388
>>> 2 1 97 0 0| 0 0 |9611k 226M| 0 0 | 144k 1615
>>> 1 1 97 0 0| 0 232k|9830k 231M| 0 0 | 147k 1524
>>>
>>> I used a Intel Xeon server Platinum 8280L CPU @ 2.70GHz, with 376G memory,
>>> 50G network. If I limit nginx memory to 2GB, the cache memory will be about
>>> 2.6G and won't increase during test. And disk io speed is close to network
>>> speed which means this can shows the disk io change of libaio vs. io_uring.
>>> This shows io_uring performance improvement. My previous data is based on
>>> this configuration.
>>>
>>> Memory:
>>> free -lh
>>> total used free shared buff/cache
>>> available
>>> Mem: 376Gi 3.2Gi 370Gi 141Mi 2.6Gi
>>> 370Gi
>>>
>>> Libaio:
>>> ----total-usage---- -dsk/total- -net/total- ---paging-- ---system--
>>> usr sys idl wai stl| read writ| recv send| in out | int csw
>>> 1 0 98 1 0| 60M 0 |2925k 68M| 0 0 | 50k 16k
>>> 1 0 98 1 0| 60M 8192B|2923k 68M| 0 0 | 50k 16k
>>> 1 0 98 1 0| 61M 0 |2923k 68M| 0 0 | 50k 16k
>>> 0 0 98 1 0| 60M 0 |2929k 68M| 0 0 | 50k 16k
>>> 1 0 98 1 0| 60M 264k|2984k 69M| 0 0 | 51k 16k
>>>
>>> Io_uring:
>>> ----total-usage---- -dsk/total- -net/total- ---paging-- ---system--
>>> usr sys idl wai stl| read writ| recv send| in out | int csw
>>> 1 2 93 4 0| 192M 8192B|7951k 187M| 0 0 | 146k 90k
>>> 1 2 93 4 0| 196M 0 |7953k 187M| 0 0 | 144k 89k
>>> 1 2 93 4 0| 191M 300k|7854k 185M| 0 0 | 145k 87k
>>> 1 2 94 3 0| 186M 8192B|7861k 185M| 0 0 | 143k 86k
>>> 1 2 94 3 0| 180M 16k|7995k 188M| 0 0 | 146k 86k
>>> 2 1 94 3 0| 163M 16k|7273k 171M| 0 0 | 133k 80k
>>> 1 1 94 3 0| 173M 1308k|7995k 188M| 0 0 | 144k 83k
>>>
>>> Considering that server memory won't be always enough for cache storage
>>> when traffic increased and then Nginx will use disk as cache storage. In
>>> this case, io_uring will shows big performance improvement than libaio on
>>> disk io interface. This is the value of this patch.
>>>
>>> BR,
>>> Ping
>>>
>>> -----Original Message-----
>>> From: nginx-devel <nginx-devel-boun...@nginx.org> On Behalf Of
>>> Mikhail Isachenkov
>>> Sent: Tuesday, February 9, 2021 1:17 AM
>>> To: nginx-devel@nginx.org
>>> Subject: Re: [PATCH] Add io_uring support in AIO(async io) module
>>>
>>> Hi Zhao Ping,
>>>
>>> First of all, thank you for pointing me to AWS patch -- on Fedora 33 with
>>> 5.10 kernel I don't see any errors now.
>>>
>>> I've tested patch on Amazon EC2 NVMe SSD (and found this drive pretty
>>> fast!). Server is i3en.xlarge, client is c5n.2xlarge instance type, with up
>>> to 25 Gigabit network.
>>>
>>> As in previous test, I've created a number of 100kb files, but try to reach
>>> them via proxy_cache as on your stand. After warming up disk cache, I've
>>> got the following results:
>>>
>>> a) with 4 worker processes, I've got 3Gb/sec in all tests regardless of
>>> sendfile/libaio/io_uring.
>>>
>>> b) with 1 worker process, sendfile is faster (up to 1.9 Gb/sec) than libaio
>>> (1.40 Gb/sec) and io_uring (up to 1.45 Gb/sec).
>>>
>>> I didn't use any memory limitations, but I ran 'echo 3 >
>>> /proc/sys/vm/drop_caches' before each pass. When I try to limit memory to
>>> 2G with cgroups, results are generally the same. Maybe 2G is not enough?
>>>
>>> Could you please run the test for ~60 seconds, and run 'dstat' on other
>>> console? I'd like to check disk and network bandwidth at the same
>>> timestamps and compare them to mine.
>>>
>>> Thanks in advance!
>>>
>>> 07.02.2021 05:16, Zhao, Ping пишет:
>>>> Hi Mikhail,
>>>>
>>>> I reproduced your problem with kernel 5.8.0-1010-aws. And I tried
>>>> kernel 5.8.0 which doesn't has this problem. I can confirm there's
>>>> a regression of aws patch(linux-aws_5.8.0-1010.10.diff)
>>>>
>>>> Updated 'sendfile on' & 'aio off' test result with 4KB data which is
>>>> almost same as libaio:
>>>>
>>>> Nginx worker_processes 1:
>>>> 4k 100k 1M
>>>> Io_uring 220MB/s 1GB/s 1.3GB/s
>>>> Libaio 70MB/s 250MB/s 600MB/s(with -c 200, 1.0GB/s)
>>>> sendfile 70MB/s 260MB/s 700MB/s
>>>>
>>>>
>>>> Nginx worker_processes 4:
>>>> 4k 100k 1M
>>>> Io_uring 800MB/s 2.5GB/s 2.6GB/s(my nvme disk io maximum bw)
>>>> libaio 250MB/s 900MB/s 2.0GB/s
>>>> sendfile 250MB/s 900MB/s 1.6GB/s
>>>>
>>>> BR,
>>>> Ping
>>>>
>>>> -----Original Message-----
>>>> From: Zhao, Ping
>>>> Sent: Friday, February 5, 2021 2:43 PM
>>>> To: nginx-devel@nginx.org
>>>> Subject: RE: [PATCH] Add io_uring support in AIO(async io) module
>>>>
>>>> Hi Mikhail,
>>>>
>>>> Added 'sendfile on' & 'aio off' test result with previous table:
>>>>
>>>> Following is the test result with 100KB and 1MB: (4KB to be test)
>>>>
>>>> Nginx worker_processes 1:
>>>> 4k 100k 1M
>>>> Io_uring 220MB/s 1GB/s 1.3GB/s
>>>> Libaio 70MB/s 250MB/s 600MB/s(with -c 200, 1.0GB/s)
>>>> sendfile tbt 260MB/s 700MB/s
>>>>
>>>>
>>>> Nginx worker_processes 4:
>>>> 4k 100k 1M
>>>> Io_uring 800MB/s 2.5GB/s 2.6GB/s(my nvme disk io maximum bw)
>>>> libaio 250MB/s 900MB/s 2.0GB/s
>>>> sendfile tbt 900MB/s 1.6GB/s
>>>>
>>>> Regards,
>>>> Ping
>>>>
>>>> -----Original Message-----
>>>> From: nginx-devel <nginx-devel-boun...@nginx.org> On Behalf Of
>>>> Mikhail Isachenkov
>>>> Sent: Thursday, February 4, 2021 4:55 PM
>>>> To: nginx-devel@nginx.org
>>>> Subject: Re: [PATCH] Add io_uring support in AIO(async io) module
>>>>
>>>> Hi Zhao Ping,
>>>>
>>>> My test is much simpler than yours. I created
>>>> /usr/local/html/(11111...99999) files on SSD (100 kb size) and wrote small
>>>> lua script for wrk that adds 5 random digits to request. There are no such
>>>> errors without patch with aio enabled.
>>>> These files does not change during test.
>>>>
>>>> I'll try to reproduce this on CentOS 8 -- which repository do you use to
>>>> install 5.x kernel?
>>>>
>>>> Also, could you please run the test with 'sendfile on' and 'aio off' to
>>>> get reference numbers for sendfile too?
>>>>
>>>> Thanks in advance!
>>>>
>>>> 04.02.2021 10:08, Zhao, Ping пишет:
>>>>> Another possible cause is that "/usr/local/html/64746" was
>>>>> changed/removed when other user tried to read it.
>>>>>
>>>>> -----Original Message-----
>>>>> From: Zhao, Ping
>>>>> Sent: Thursday, February 4, 2021 10:33 AM
>>>>> To: nginx-devel@nginx.org
>>>>> Subject: RE: [PATCH] Add io_uring support in AIO(async io) module
>>>>>
>>>>> Hi Mikhail,
>>>>>
>>>>> I didn't see this error in my log. Following is my OS/Kernel:
>>>>> CentOS: 8.1.1911
>>>>> Kernel: 5.7.19
>>>>> Liburing: liburing-1.0.7-3.el8.x86_64,
>>>>> liburing-devel-1.0.7-3.el8.x86_64 (from yum repo)
>>>>>
>>>>> Regarding the error: 11: Resource temporarily unavailable. It's probably
>>>>> that too many read "/usr/local/html/64746" at one time which is still
>>>>> locked by previous read. I tried to repro this error with single file but
>>>>> it seems nginx auto store the signal file in memory and I don't see
>>>>> error. How do you perform the test? I want to repro this if possible.
>>>>>
>>>>> My nginx reported this error before:
>>>>> 2021/01/04 05:04:29 [alert] 50769#50769: *11498 pread() read only 7101 of
>>>>> 15530 from "/mnt/cache1/17/68aae9d816ec02340ee617b7ee52a117", client:
>>>>> 11.11.11.3, server: _, request: "GET
>>>>> /_100kobject?version=cdn003191&thread=64 HTTP/1.1", host:
>>>>> "11.11.11.1:8080"
>>>>> Which is fixed by my 2nd patch(Jan 25) already.
>>>>>
>>>>> BR,
>>>>> Ping
>>>>>
>>>>> -----Original Message-----
>>>>> From: nginx-devel <nginx-devel-boun...@nginx.org> On Behalf Of
>>>>> Mikhail Isachenkov
>>>>> Sent: Wednesday, February 3, 2021 10:11 PM
>>>>> To: nginx-devel@nginx.org
>>>>> Subject: Re: [PATCH] Add io_uring support in AIO(async io) module
>>>>>
>>>>> Hi Ping Zhao,
>>>>>
>>>>> When I try to repeat this test, I've got a huge number of these errors:
>>>>>
>>>>> 2021/02/03 10:22:48 [crit] 30018#30018: *2 aio read
>>>>> "/usr/local/html/64746" failed (11: Resource temporarily
>>>>> unavailable) while sending response to client, client: 127.0.0.1, server:
>>>>> localhost,
>>>>> request: "GET /64746 HTTP/1.1", host: "localhost"
>>>>>
>>>>> I tested this patch on Ubuntu 20.10 (5.8.0-1010-aws kernel version) and
>>>>> Fedora 33 (5.10.11-200.fc33.x86_64) with the same result.
>>>>>
>>>>> Did you get any errors in error log with patch applied? Which OS/kernel
>>>>> did you use for testing? Did you perform any specific tuning before
>>>>> running?
>>>>>
>>>>> 25.01.2021 11:24, Zhao, Ping пишет:
>>>>>> Hello, add a small update to correct the length when part of request
>>>>>> already received in previous.
>>>>>> This case may happen when using io_uring and throughput increased.
>>>>>>
>>>>>> # HG changeset patch
>>>>>> # User Ping Zhao <ping.z...@intel.com> # Date 1611566408 18000
>>>>>> # Mon Jan 25 04:20:08 2021 -0500
>>>>>> # Node ID f2c91860b7ac4b374fff4353a830cd9427e1d027
>>>>>> # Parent 1372f9ee2e829b5de5d12c05713c307e325e0369
>>>>>> Correct length calculation when part of request received.
>>>>>>
>>>>>> diff -r 1372f9ee2e82 -r f2c91860b7ac src/core/ngx_output_chain.c
>>>>>> --- a/src/core/ngx_output_chain.c Wed Jan 13 11:10:05 2021 -0500
>>>>>> +++ b/src/core/ngx_output_chain.c Mon Jan 25 04:20:08 2021 -0500
>>>>>> @@ -531,6 +531,14 @@
>>>>>>
>>>>>> size = ngx_buf_size(src);
>>>>>> size = ngx_min(size, dst->end - dst->pos);
>>>>>> +#if (NGX_HAVE_FILE_IOURING)
>>>>>> + /*
>>>>>> + * check if already received part of the request in previous,
>>>>>> + * calculate the remain length
>>>>>> + */
>>>>>> + if(dst->last > dst->pos && size > (dst->last - dst->pos))
>>>>>> + size = size - (dst->last - dst->pos); #endif
>>>>>>
>>>>>> sendfile = ctx->sendfile && !ctx->directio;
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: nginx-devel <nginx-devel-boun...@nginx.org> On Behalf Of
>>>>>> Zhao, Ping
>>>>>> Sent: Thursday, January 21, 2021 9:44 AM
>>>>>> To: nginx-devel@nginx.org
>>>>>> Subject: RE: [PATCH] Add io_uring support in AIO(async io) module
>>>>>>
>>>>>> Hi Vladimir,
>>>>>>
>>>>>> No special/extra configuration needed, but need check if 'aio on' and
>>>>>> 'sendfile off' is correctly set. This is my Nginx config for reference:
>>>>>>
>>>>>> user nobody;
>>>>>> daemon off;
>>>>>> worker_processes 1;
>>>>>> error_log error.log ;
>>>>>> events {
>>>>>> worker_connections 65535;
>>>>>> use epoll;
>>>>>> }
>>>>>>
>>>>>> http {
>>>>>> include mime.types;
>>>>>> default_type application/octet-stream;
>>>>>> access_log on;
>>>>>> aio on;
>>>>>> sendfile off;
>>>>>> directio 2k;
>>>>>>
>>>>>> # Cache Configurations
>>>>>> proxy_cache_path /mnt/cache0 levels=2
>>>>>> keys_zone=nginx-cache0:400m max_size=1400g inactive=4d
>>>>>> use_temp_path=off; ......
>>>>>>
>>>>>>
>>>>>> To better measure the disk io performance data, I do the following steps:
>>>>>> 1. To exclude other impact, and focus on disk io part.(This patch only
>>>>>> impact disk aio read process) Use cgroup to limit Nginx memory usage.
>>>>>> Otherwise Nginx may also use memory as cache storage and this may cause
>>>>>> test result not so straight.(since most cache hit in memory, disk io bw
>>>>>> is low, like my previous mail found which didn't exclude the memory
>>>>>> cache impact)
>>>>>> echo 2G > memory.limit_in_bytes
>>>>>> use ' cgexec -g memory:nginx' to start Nginx.
>>>>>>
>>>>>> 2. use wrk -t 100 -c 1000, with random 25000 http requests.
>>>>>> My previous test used -t 200 connections, comparing with -t
>>>>>> 1000, libaio performance drop more when connections numbers increased
>>>>>> from 200 to 1000, but io_uring doesn't. It's another advantage of
>>>>>> io_uring.
>>>>>>
>>>>>> 3. First clean the cache disk and run the test for 30 minutes to let
>>>>>> Nginx store the cache files to nvme disk as much as possible.
>>>>>>
>>>>>> 4. Rerun the test, this time Nginx will use ngx_file_aio_read to
>>>>>> extract the cache files in nvme cache disk. Use iostat to track
>>>>>> the io data. The data should be align with NIC bw since all data
>>>>>> should be from cache disk.(need exclude memory as cache storage
>>>>>> impact)
>>>>>>
>>>>>> Following is the test result:
>>>>>>
>>>>>> Nginx worker_processes 1:
>>>>>> 4k 100k 1M
>>>>>> Io_uring 220MB/s 1GB/s 1.3GB/s
>>>>>> Libaio 70MB/s 250MB/s 600MB/s(with -c 200, 1.0GB/s)
>>>>>>
>>>>>>
>>>>>> Nginx worker_processes 4:
>>>>>> 4k 100k 1M
>>>>>> Io_uring 800MB/s 2.5GB/s 2.6GB/s(my nvme disk io maximum bw)
>>>>>> libaio 250MB/s 900MB/s 2.0GB/s
>>>>>>
>>>>>> So for small request, io_uring has huge improvement than libaio. In
>>>>>> previous mail, because I didn't exclude the memory cache storage impact,
>>>>>> most cache file is stored in memory, very few are from disk in case of
>>>>>> 4k/100k. The data is not correct.(for 1M, because the cache is too big
>>>>>> to store in memory, it wat in disk) Also I enabled directio option
>>>>>> "directio 2k" this time to avoid this.
>>>>>>
>>>>>> Regards,
>>>>>> Ping
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: nginx-devel <nginx-devel-boun...@nginx.org> On Behalf Of
>>>>>> Vladimir Homutov
>>>>>> Sent: Wednesday, January 20, 2021 12:43 AM
>>>>>> To: nginx-devel@nginx.org
>>>>>> Subject: Re: [PATCH] Add io_uring support in AIO(async io) module
>>>>>>
>>>>>> On Tue, Jan 19, 2021 at 03:32:30AM +0000, Zhao, Ping wrote:
>>>>>>> It depends on if disk io is the performance hot spot or not. If
>>>>>>> yes, io_uring shows improvement than libaio. With 4KB/100KB
>>>>>>> length
>>>>>>> 1 Nginx thread it's hard to see performance difference because
>>>>>>> iostat is only around ~10MB/100MB per second. Disk io is not the
>>>>>>> performance bottle neck, both libaio and io_uring have the same
>>>>>>> performance. If you increase request size or Nginx threads
>>>>>>> number, for example 1MB length or Nginx thread number 4. In this
>>>>>>> case, disk io became the performance bottle neck, you will see io_uring
>>>>>>> performance improvement.
>>>>>>
>>>>>> Can you please provide full test results with specific nginx
>>>>>> configuration?
>>>>>>
>>>>>> _______________________________________________
>>>>>> nginx-devel mailing list
>>>>>> nginx-devel@nginx.org
>>>>>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>>>>> _______________________________________________
>>>>>> nginx-devel mailing list
>>>>>> nginx-devel@nginx.org
>>>>>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>>>>> _______________________________________________
>>>>>> nginx-devel mailing list
>>>>>> nginx-devel@nginx.org
>>>>>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Mikhail Isachenkov
>>>>> NGINX Professional Services
>>>>> _______________________________________________
>>>>> nginx-devel mailing list
>>>>> nginx-devel@nginx.org
>>>>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>>>> _______________________________________________
>>>>> nginx-devel mailing list
>>>>> nginx-devel@nginx.org
>>>>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Mikhail Isachenkov
>>>> NGINX Professional Services
>>>> _______________________________________________
>>>> nginx-devel mailing list
>>>> nginx-devel@nginx.org
>>>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>>> _______________________________________________
>>>> nginx-devel mailing list
>>>> nginx-devel@nginx.org
>>>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>>>
>>>
>>> --
>>> Best regards,
>>> Mikhail Isachenkov
>>> NGINX Professional Services
>>> _______________________________________________
>>> nginx-devel mailing list
>>> nginx-devel@nginx.org
>>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>> _______________________________________________
>>> nginx-devel mailing list
>>> nginx-devel@nginx.org
>>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>>
>>
>> --
>> Best regards,
>> Mikhail Isachenkov
>> NGINX Professional Services
>> _______________________________________________
>> nginx-devel mailing list
>> nginx-devel@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>
>
> --
> Best regards,
> Mikhail Isachenkov
> NGINX Professional Services
> _______________________________________________
> nginx-devel mailing list
> nginx-devel@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
> _______________________________________________
> nginx-devel mailing list
> nginx-devel@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
--
Best regards,
Mikhail Isachenkov
NGINX Professional Services
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
