tirkarthi commented on issue #60265: URL: https://github.com/apache/airflow/issues/60265#issuecomment-3739696024
Another observation though slightly out of topic is the change in latency distribution with the number of concurrent requests for the same endpoint. Below is caddy running on 2080 reverse proxying to api-server instances running on 8000 and 8001 with 4 workers each. A grid page can request with a single dagrun can load 9 ui/ and 11 api/ requests. On the initial load with many parallel requests latest_run endpoint which is slightly lightweight takes 108ms on the initial load but subsequent loads take 35-40ms. | concurrency | p50 (s) | p75 (s) | p90 (s) | -------- | ------- |-------|------| | 1 | 0.0300 | 0.0322 | 0.0338 | | 10 | 0.0580 | 0.1018 | 0.1606 | | 50 | 0.2044 | 0.3054 | 0.5021 ``` hey -c 1 -H "Cookie: _token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxIiwianRpIjoiNWRiNDhmZjBiNDc2NGZhNjlhNTI1ZGZiZjZkMDM3Y2IiLCJpc3MiOltdLCJhdWQiOiJhcGFjaGUtYWlyZmxvdyIsIm5iZiI6MTc2ODIzNDUyNiwiZXhwIjoxNzY4MjM4MTI2LCJpYXQiOjE3NjgyMzQ1MjZ9.P-5yzoGlFEp8LRACuJhCdk74uuQ3VJPBvnovMtIqZ8OI3lJdBRUMAFCzauvsBdbRXBXvub7XBapHfIojEH7v_A" http://localhost:2080/ui/dags/example_complex/latest_run Summary: Total: 6.3623 secs Slowest: 0.3777 secs Fastest: 0.0232 secs Average: 0.0318 secs Requests/sec: 31.4350 Response time histogram: 0.023 [1] | 0.059 [198] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.094 [0] | 0.130 [0] | 0.165 [0] | 0.200 [0] | 0.236 [0] | 0.271 [0] | 0.307 [0] | 0.342 [0] | 0.378 [1] | Latency distribution: 10% in 0.0265 secs 25% in 0.0283 secs 50% in 0.0300 secs 75% in 0.0322 secs 90% in 0.0338 secs 95% in 0.0348 secs 99% in 0.0364 secs ``` ``` time hey -c 10 -H "Cookie: _token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxIiwianRpIjoiNWRiNDhmZjBiNDc2NGZhNjlhNTI1ZGZiZjZkMDM3Y2IiLCJpc3MiOltdLCJhdWQiOiJhcGFjaGUtYWlyZmxvdyIsIm5iZiI6MTc2ODIzNDUyNiwiZXhwIjoxNzY4MjM4MTI2LCJpYXQiOjE3NjgyMzQ1MjZ9.P-5yzoGlFEp8LRACuJhCdk74uuQ3VJPBvnovMtIqZ8OI3lJdBRUMAFCzauvsBdbRXBXvub7XBapHfIojEH7v_A" http://localhost:2080/ui/dags/example_complex/latest_run Summary: Total: 2.3252 secs Slowest: 0.7348 secs Fastest: 0.0283 secs Average: 0.0937 secs Requests/sec: 86.0129 Response time histogram: 0.028 [1] | 0.099 [148] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.170 [35] |■■■■■■■■■ 0.240 [7] |■■ 0.311 [2] |■ 0.382 [0] | 0.452 [0] | 0.523 [0] | 0.594 [0] | 0.664 [3] |■ 0.735 [4] |■ Latency distribution: 10% in 0.0314 secs 25% in 0.0348 secs 50% in 0.0580 secs 75% in 0.1018 secs 90% in 0.1606 secs 95% in 0.2400 secs 99% in 0.7250 secs ``` ``` hey -c 50 -H "Cookie: _token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxIiwianRpIjoiNWRiNDhmZjBiNDc2NGZhNjlhNTI1ZGZiZjZkMDM3Y2IiLCJpc3MiOltdLCJhdWQiOiJhcGFjaGUtYWlyZmxvdyIsIm5iZiI6MTc2ODIzNDUyNiwiZXhwIjoxNzY4MjM4MTI2LCJpYXQiOjE3NjgyMzQ1MjZ9.P-5yzoGlFEp8LRACuJhCdk74uuQ3VJPBvnovMtIqZ8OI3lJdBRUMAFCzauvsBdbRXBXvub7XBapHfIojEH7v_A" http://localhost:2080/ui/dags/example_complex/latest_run Summary: Total: 1.5111 secs Slowest: 1.1720 secs Fastest: 0.0307 secs Average: 0.2528 secs Requests/sec: 132.3577 Response time histogram: 0.031 [1] |■ 0.145 [70] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.259 [67] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.373 [25] |■■■■■■■■■■■■■■ 0.487 [16] |■■■■■■■■■ 0.601 [9] |■■■■■ 0.715 [0] | 0.830 [0] | 0.944 [5] |■■■ 1.058 [0] | 1.172 [7] |■■■■ Latency distribution: 10% in 0.0403 secs 25% in 0.0819 secs 50% in 0.2044 secs 75% in 0.3054 secs 90% in 0.5021 secs 95% in 0.9171 secs 99% in 1.1450 secs ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
