edwardwang888 commented on a change in pull request #18356: URL: https://github.com/apache/airflow/pull/18356#discussion_r712664619
########## File path: docs/apache-airflow/concepts/scheduler.rst ########## @@ -138,18 +141,173 @@ The following databases are fully supported and provide an "optimal" experience: Microsoft SQLServer has not been tested with HA. + +Fine-tuning your Scheduler performance +-------------------------------------- + +What impacts scheduler's performance +"""""""""""""""""""""""""""""""""""" + +The Scheduler is responsible for two operations: + +* continuously parsing DAG files and synchronizing with the DAG in the database +* continuously scheduling tasks for execution + +Those two tasks are executed in parallel by the scheduler and run independently of each other in +different processes. In order to fine-tune your scheduler, you need to include a number of factors: + +* The kind of deployment you have + * what kind of filesystem you have to share the DAGs (impacts performance of continuous reading DAGs) + * how fast the filesystem is (in many cases of distributed cloud filesystem you can pay extra to get + more throughput/faster filesystem + * how much memory you have for your processing + * how much CPU you have available + * how much networking throughput you have available + +* The logic and definition of your DAG structure: + * how many DAG files you have + * how many DAGs you have in your files + * how large the DAG files are (remember scheduler needs to read and parse the file every n seconds) + * how complex they are (i.e. how fast they can be parsed, how many tasks and dependencies they have) + * whether parsing your DAGs involves heavy processing (Hint! It should not. See :doc:`/best-practices`) + +* The scheduler configuration + * How many schedulers you have + * How many parsing processes you have in your scheduler + * How much time scheduler waits between re-parsing of the same DAG (it happens continuously) + * How many task instances scheduler processes in one loop + * How many new DAG runs should be created/scheduled per loop + * Whether to execute "mini-scheduler" after completed task to speed up scheduling dependent tasks + * How often the scheduler should perform cleanup and check for orphaned tasks/adopting them + * Whether scheduler uses row-level locking + +In order to perform fine-tuning, it's good to understand how Scheduler works under-the-hood. +You can take a look at the ``Airflow Summit 2021`` +`Deep Dive into the Airflow Scheduler talk <https://youtu.be/DYC4-xElccE>`_ to perform the fine-tuning. + +How to approach Scheduler's fine-tuning +""""""""""""""""""""""""""""""""""""""" + +Airflow gives you a lot of "knobs" to turn to fine tune the performance but it's a separate task, +depending on your particular deployment, your DAG structure, hardware availability and expectations, +to decide which knobs to turn to get best effect for you. Part of the job when managing the +deployment is to decide what you are going to optimize for. Some users are ok with +30 seconds delays of new DAG parsing, at the expense of lower CPU usage, whereas some other users +expect the DAGs to be parsed almost instantly when they appear in the DAGs folder at the +expense of higher CPU usage for example. + +Airflow gives you the flexibility to decide, but you should find out what aspect of performance is +most important for you and decide which knobs you want to turn in which direction. + +Generally for fine-tuning, your approach should be the same as for any performance improvement and +optimizations (we will not recommend any specific tools - just use the tools that you usually use +to observe and monitor your systems): + +* its extremely important to monitor your system with the right set of tools that you usually use to + monitor your system. This document does not go into details of particular metrics and tools that you + can use, it just describes what kind of resources you should monitor, but you should follow your best + practices for monitoring to grab the right data. +* decide which aspect of performance is most important for you (what you want to improve) +* observe your system to see where your bottlenecks are: CPU, memory, I/O are the usual limiting factors +* based on your expectations and observations - decide what is your next improvement and go back to + the observation of your performance, bottlenecks. Performance improvement is an iterative process. + +What resources might limit Scheduler's performance +"""""""""""""""""""""""""""""""""""""""""""""""""" + +There are several areas of resource usage that you should pay attention to: + +* FileSystem performance. Airflow Scheduler relies heavily on parsing (sometimes a lot) of Python + files, which are often located on a shared filesystem. Airflow Scheduler continuously reads and + re-parses those files. The same files have to be made available to workers, so often they are + stored in a distributed filesystem. You can use various filesystems for that purpose (NFS, CIFS, EFS, + GCS fuse, Azure File System are good examples). There are various parameters you can control for those + filesystems and fine-tune their performance, but this is beyond the scope of this document. You should + observe statistics and usage of your filesystem to determine if problems come from the filesystem + performance. For example there are anecdotal evidences that increasing IOPS (and paying more) for the + EFS performance, dramatically improves stability and speed of parsing Airflow DAGs when EFS is used. +* Another solution to FileSystem performance, if it becomes your bottleneck, is to turn to alternative + mechanisms of distributing your DAGs. Embedding DAGs in your image and GitSync distribution have both + the property that the files are available locally for Scheduler and it does not have to use a + distributed filesystem to read the files, the files are available locally for the Scheduler and it is + usually as fast as it can be, especially if your machines use fast SSD disks for local storage. Those + distribution mechanisms have other characteristics that might make them not the best choice for you, + but if your problems with performance come from distributed filesystem performance, they might be the + best approach to follow. +* Database connections and Database usage might become a problem as you want to increase performance and + process more things in parallel. Airflow is known from being "database-connection hungry" - the more DAGs + you have and the more you want to process in parallel, the more database connections will be opened. + This is generally not a problem for MySQL as its model of handling connections is thread-based, but this + might be a problem for Postgres, where connection handling is process-based. It is a general consensus + that if you have even medium size Postgres-based Airflow installation, the best solution is to use + `PGBouncer <https://www.pgbouncer.org/>`_ as a proxy to your database. The :doc:`helm-chart:index` + supports PGBouncer out-of-the-box. For MsSQL we have not yet worked out the best practices as support + for MsSQL is still experimental. +* CPU usage is most important for FileProcessors - those are the processes that parse and execute + Python DAG files. Since Schedulers triggers such parsing continuously, when you have a lot of DAGs, + the processing might take a lot of CPU. You can mitigate it by decreasing the + :ref:`config:scheduler__min_file_process_interval`, but this is one of the mentioned trade-offs, + result of this is that changes to such files will be picked up slower and you will see delays between + submitting the files and getting them available in Airflow UI and executed by Scheduler. Optimizing + the way how your DAGs are built, avoiding external data sources is your best approach to improve CPU + usage. If you have more CPUs available, you can increase number of processing threads + :ref:`config:scheduler__parsing_processes`, Also Airflow Scheduler scales almost linearly with + several instances, so you can also add more Schedulers if your Scheduler's performance is CPU-bound. +* Airflow might use quite significant amount of memory when you try to get more performance out of it. + Often more performance is achieved in Airflow by increasing number of processes handling the load, + and each process requires whole interpreter of Python loaded, a lot of classes imported, temporary + in-memory storage. This can lead to memory pressure. You need to observe if your system is not using + more memory than it has - which results with using swap disk, which dramatically decreases performance. Review comment: Since you are referring to the scenario of using more memory than you have, I think it's clearer to word this way? ```suggestion in-memory storage. This can lead to memory pressure. You need to observe if your system is using more memory than it has - which results with using swap disk, which dramatically decreases performance. ``` ########## File path: docs/apache-airflow/best-practices.rst ########## @@ -255,9 +297,34 @@ No additional code needs to be written by the user to run this test. .. code-block:: bash - python your-dag-file.py + python your-dag-file.py + +Running the above command without any error ensures your DAG does not contain any uninstalled dependency, +syntax errors, etc. Make sure that you load your DAG in an environment that corresponds to your +scheduler environment - with the same dependencies, environment variables, common code referred from the +DAG. + +This is also a great way to check if your DAG loads faster after an optimization, if you wan to attempt +to optimize DAG loading time. Simply run the DAG and measure the time it takes, but again you have to +make sure your DAG runs wit the same dependencies, environment variables, common code. +Make sure to run it several time in succession to account for caching effects. Compare the results +before and after the optimization in order to asses the impact of the optimization. Review comment: ```suggestion before and after the optimization in order to assess the impact of the optimization. ``` ########## File path: docs/apache-airflow/best-practices.rst ########## @@ -241,11 +243,51 @@ each parameter by following the links): * :ref:`config:scheduler__parsing_processes` * :ref:`config:scheduler__file_parsing_sort_mode` +.. _best_practices/reducing_dag_complexity: + +Reducing DAG complexity +^^^^^^^^^^^^^^^^^^^^^^^ + +While Airflow is good in handling a lot of DAGs with a lot of task and dependencies between them, when you +have many complex DAGs, their complexity might impact performance of scheduling. One of the ways to keep +your Airflow instance performant and well utilized, you should strive to simplify and optimize your DAGs +whenever possible - you have to remember that DAG parsing process and creation is just executing +Python code and it's up to you to make it as performant as possible. There are no magic recipes for making +your DAG "less complex" - since this is a Python code, it's the DAG writer who controls the complexity of +their code. + +There are no "metrics" for DAG complexity, especially, there are no metrics that can tell you +whether your DAG is "simple enough". However - as with any Python code you can definitely tell that +your code is "simpler" or "faster" when you optimize it, the same can be said about DAG code. If you +want to optimize your DAGs there are the following actions you can take: + +* Make your DAG load faster. This is a single improvement advice that might be implemented in various ways + but this is the one that has biggest impact on scheduler's performance. Whenever you have a chance to make + your DAG load faster - go for it, if your goal is to improve performance. See below + :ref:`best_practices/dag_loader_test` on how to asses your DAG loading time. + +* Make your DAG generate lest tasks. Every task adds additional processing overhead for scheduling and Review comment: Typo, but `fewer` is gramatically correct 🙂 ```suggestion * Make your DAG generate fewer tasks. Every task adds additional processing overhead for scheduling and ``` ########## File path: docs/apache-airflow/best-practices.rst ########## @@ -255,9 +297,34 @@ No additional code needs to be written by the user to run this test. .. code-block:: bash - python your-dag-file.py + python your-dag-file.py + +Running the above command without any error ensures your DAG does not contain any uninstalled dependency, +syntax errors, etc. Make sure that you load your DAG in an environment that corresponds to your +scheduler environment - with the same dependencies, environment variables, common code referred from the +DAG. + +This is also a great way to check if your DAG loads faster after an optimization, if you wan to attempt Review comment: ```suggestion This is also a great way to check if your DAG loads faster after an optimization, if you want to attempt ``` ########## File path: docs/apache-airflow/best-practices.rst ########## @@ -255,9 +297,34 @@ No additional code needs to be written by the user to run this test. .. code-block:: bash - python your-dag-file.py + python your-dag-file.py + +Running the above command without any error ensures your DAG does not contain any uninstalled dependency, +syntax errors, etc. Make sure that you load your DAG in an environment that corresponds to your +scheduler environment - with the same dependencies, environment variables, common code referred from the +DAG. + +This is also a great way to check if your DAG loads faster after an optimization, if you wan to attempt +to optimize DAG loading time. Simply run the DAG and measure the time it takes, but again you have to +make sure your DAG runs wit the same dependencies, environment variables, common code. Review comment: ```suggestion make sure your DAG runs with the same dependencies, environment variables, common code. ``` ########## File path: docs/apache-airflow/concepts/scheduler.rst ########## @@ -138,18 +141,173 @@ The following databases are fully supported and provide an "optimal" experience: Microsoft SQLServer has not been tested with HA. + +Fine-tuning your Scheduler performance +-------------------------------------- + +What impacts scheduler's performance +"""""""""""""""""""""""""""""""""""" + +The Scheduler is responsible for two operations: + +* continuously parsing DAG files and synchronizing with the DAG in the database +* continuously scheduling tasks for execution + +Those two tasks are executed in parallel by the scheduler and run independently of each other in +different processes. In order to fine-tune your scheduler, you need to include a number of factors: + +* The kind of deployment you have + * what kind of filesystem you have to share the DAGs (impacts performance of continuous reading DAGs) Review comment: ```suggestion * what kind of filesystem you have to share the DAGs (impacts performance of continuously reading DAGs) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
