harshit283 opened a new issue #14874: URL: https://github.com/apache/airflow/issues/14874
Airflow Setup Used Airflow Version: 1.10.12 PostgreSQL version: 10 Airflow Configuration: AIRFLOW__WEBSERVER__WORKER_REFRESH_BATCH_SIZE: 0 AIRFLOW__WEBSERVER__WORKERS:10 AIRFLOW__CORE__STORE_SERIALIZED_DAGS: True AIRFLOW__CORE__STORE_DAG_CODE: True AIRFLOW__WEBSERVER__WORKER_CLASS: gevent **Issue details:** So we are doing some performance tests on airflow webserver to benchmark how many requests per second can a single container (kubernetes) of airflow webserver can handle, we are testing the airflow [experimental APIs](https://airflow.apache.org/docs/apache-airflow/stable/rest-api-ref.html#post--api-experimental-dags--DAG_ID--dag_runs). We are able to handle upto 8 RPS and on investigation it appears that the bottleneck is at the authentication layer. We are using the password based authentication where `check_password_hash` [method ](https://github.com/apache/airflow/blob/1.10.12/airflow/contrib/auth/backends/password_auth.py#L84) appears to be extremely slow in execution, due to which the authentication flow is contributing to 80-85% of total time taken for the Airflow API's. After bypassing this method airflow webserver is easily able to serve 50 RPS **We are looking for couple of things:** - Is there any way to fix this issue or if this is expected should we opt for any other authentication protocol? - Is this issue fixed in later versions of airflow? - Are other authentication protocols more performant? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
