llowinge commented on issue #6433: URL: https://github.com/apache/camel-quarkus/issues/6433#issuecomment-3520919672
I've proceeded even with more testing and investigation. I was firstly checking JVM performance, i thought tunning JVM flags/memory would help, but memory is not the issue. I've noticed from JFR files that garbage collector is slower on Windows (bigger delays between evacuation), but then i've noticed also I/O tab and it showed slower on Windows. I've tried also run the JasyptTest on "D:", "E:" (dev drive) and Ubuntu then i've run the same matrix on whole testsuite. There is no difference between dev-drive "E" and "D" (if i've set everything correctly, but i've run it with -Dquarkus.log.level=debug + -X for maven) and i've noticed just one thing and that was temporary folder still used on "C" (you can read about it https://ichard26.github.io/blog/2025/03/faster-pip-ci-on-windows-d-drive/), but even if i've changed it to "D" it didn't add up speed. It all lead to low I/O in general. So i've digged and found FIO (https://github.com/axboe/fio) measuring tool and i've run again on whole matrix and the results are, that Windows-latest is much slower then Ubuntu. That's it. I've used such command: ``` fio --name=fio_write_test --thread --ioengine=psync --bs=4k --size=1G --rw=write --directory=fio_test_dir --output=fio_output.json ``` which defines a specific I/O performance benchmark simulating cache-intensive, small-block, sequential write operations. The primary purpose of this test is to measure how quickly the Operating System's kernel can handle rapid data transfer requests, which is highly relevant for tasks like Maven compilation and Jandex indexing. Here are FIO results: Ubuntu: ``` fio_write_test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.36 Starting 1 thread fio_write_test: Laying out IO file (1 file / 1024MiB) fio_write_test: (groupid=0, jobs=1): err= 0: pid=3184: Wed Nov 12 08:58:38 2025 write: IOPS=334k, BW=1306MiB/s (1370MB/s)(1024MiB/784msec); 0 zone resets clat (usec): min=2, max=105, avg= 2.75, stdev= 1.64 lat (usec): min=2, max=105, avg= 2.79, stdev= 1.64 clat percentiles (nsec): | 1.00th=[ 2448], 5.00th=[ 2480], 10.00th=[ 2480], 20.00th=[ 2480], | 30.00th=[ 2480], 40.00th=[ 2512], 50.00th=[ 2512], 60.00th=[ 2512], | 70.00th=[ 2544], 80.00th=[ 2576], 90.00th=[ 2768], 95.00th=[ 3632], | 99.00th=[ 5664], 99.50th=[ 6560], 99.90th=[17280], 99.95th=[51968], | 99.99th=[68096] bw ( MiB/s): min= 1307, max= 1307, per=100.00%, avg=1307.86, stdev= 0.00, samples=1 iops : min=334812, max=334812, avg=334812.00, stdev= 0.00, samples=1 lat (usec) : 4=95.75%, 10=3.92%, 20=0.26%, 50=0.02%, 100=0.05% lat (usec) : 250=0.01% cpu : usr=12.77%, sys=87.10%, ctx=5, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=1306MiB/s (1370MB/s), 1306MiB/s-1306MiB/s (1370MB/s-1370MB/s), io=1024MiB (1074MB), run=784-784msec Disk stats (read/write): sdb: ios=0/423, sectors=0/21768, merge=0/2298, ticks=0/2680, in_queue=2680, util=1.74% ``` Windows default: ``` fio_write_test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.41 Starting 1 thread fio_write_test: Laying out IO file (1 file / 1024MiB) fio_write_test: (groupid=0, jobs=1): err= 0: pid=6080: Wed Nov 12 08:58:20 2025 write: IOPS=160k, BW=624MiB/s (655MB/s)(1024MiB/1640msec); 0 zone resets clat (usec): min=3, max=538, avg= 5.89, stdev= 5.60 lat (usec): min=3, max=539, avg= 5.95, stdev= 5.61 clat percentiles (nsec): | 1.00th=[ 3664], 5.00th=[ 3696], 10.00th=[ 3696], 20.00th=[ 3728], | 30.00th=[ 3760], 40.00th=[ 3792], 50.00th=[ 3856], 60.00th=[ 3984], | 70.00th=[ 5536], 80.00th=[ 6624], 90.00th=[ 8160], 95.00th=[15680], | 99.00th=[30848], 99.50th=[35072], 99.90th=[50944], 99.95th=[59648], | 99.99th=[85504] bw ( KiB/s): min=583576, max=700400, per=100.00%, avg=641405.33, stdev=58420.72, samples=3 iops : min=145894, max=175100, avg=160352.00, stdev=14605.16, samples=3 lat (usec) : 4=60.25%, 10=31.48%, 20=5.38%, 50=2.79%, 100=0.10% lat (usec) : 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=0.00%, sys=61.01%, ctx=0, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=624MiB/s (655MB/s), 624MiB/s-624MiB/s (655MB/s-655MB/s), io=1024MiB (1074MB), run=1640-1640msec ``` Windows dev drive: ``` fio_write_test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.41 Starting 1 thread fio_write_test: Laying out IO file (1 file / 1024MiB) fio_write_test: (groupid=0, jobs=1): err= 0: pid=6644: Wed Nov 12 08:58:45 2025 write: IOPS=192k, BW=751MiB/s (787MB/s)(1024MiB/1364msec); 0 zone resets clat (usec): min=3, max=771, avg= 4.90, stdev= 6.55 lat (usec): min=3, max=771, avg= 4.95, stdev= 6.61 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 9], | 99.00th=[ 19], 99.50th=[ 24], 99.90th=[ 49], 99.95th=[ 68], | 99.99th=[ 375] bw ( KiB/s): min=745133, max=796239, per=100.00%, avg=770686.00, stdev=36137.40, samples=2 iops : min=186283, max=199061, avg=192672.00, stdev=9035.41, samples=2 lat (usec) : 4=65.96%, 10=30.07%, 20=3.16%, 50=0.72%, 100=0.06% lat (usec) : 250=0.01%, 500=0.02%, 750=0.01%, 1000=0.01% cpu : usr=0.00%, sys=73.37%, ctx=0, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=751MiB/s (787MB/s), 751MiB/s-751MiB/s (787MB/s-787MB/s), io=1024MiB (1074MB), run=1364-1364msec ``` Summary: ``` Metric Ubuntu (Linux) Windows (Standard Disk D:) Windows (Dev Drive) Bandwidth (BW) 1306 MiB/s 624 MiB/s 751 MiB/s IOPS 334k 160k 192k Avg. Latency 2.79 µs 5.95 µs 4.95 µs Run Time (ms) 784 ms 1640 ms 1364 ms ``` The Ubuntu runner provides 1.7x to 2.1x better I/O performance (1306 MiB/s vs. 751 MiB/s) and significantly lower latency (2.79 µs vs. 4.95 µs) for small, cache-intensive writes. This confirms the fundamental difference in OS kernel I/O overhead. Note that this result show that dev-drive is slightly faster, but when i've run it first time, the "D" disk was slightly faster, so they are more or less the same. But the magnificant difference from Ubuntu speed is still evident. I'm also attaching source code for the modified CI for measuring the I/O https://github.com/llowinge/camel-quarkus/blob/e80f91fad5515f67bc034820dc402315207c9c8a/.github/workflows/ci-build.yaml#L697. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
