mik-laj commented on a change in pull request #6247: [AIRFLOW-5588] Add
Celery's architecture diagram
URL: https://github.com/apache/airflow/pull/6247#discussion_r333026664
##########
File path: docs/executor/celery.rst
##########
@@ -72,3 +72,74 @@ Some caveats:
- Make sure to set a visibility timeout in [celery_broker_transport_options]
that exceeds the ETA of your longest running task
- Tasks can consume resources. Make sure your worker has enough resources to
run ``worker_concurrency`` tasks
- Queue names are limited to 256 characters, but each broker backend might
have its own restrictions
+
+Architecture
+------------
+
+.. graphviz::
+
+ digraph A{
+ rankdir="TB"
+ node[shape="rectangle", style="rounded"]
+
+
+ subgraph cluster {
+ label="Cluster";
+ {rank = same; dag; database}
+ {rank = same; workers; scheduler; web}
+
+ workers[label="Workers"]
+ scheduler[label="Scheduler"]
+ web[label="Web server"]
+ database[label="Database"]
+ dag[label="DAG files"]
+
+ subgraph cluster_queue {
+ label="Queue";
+ {rank = same; queue_broker; queue_result_backend}
+ queue_broker[label="Queue broker"]
+ queue_result_backend[label="Result backend"]
+ }
+
+ scheduler->workers[label="1"]
+ web->database[label="2"]
+ web->dag[label="3"]
+
+ workers->database[label="4"]
+ workers->dag[label="5"]
+ workers->queue_result_backend[label="6"]
+ workers->queue_broker[label="7"]
+
+ scheduler->database[label="8"]
+ scheduler->dag[label="9"]
+ scheduler->queue_result_backend[label="10"]
+ scheduler->queue_broker[label="11"]
+ }
+ }
+
+Airflow consist of several components:
+
+* **Workers** - Execute the assigned tasks
+* **Scheduler** - Responsible for adding the necessary tasks to the queue
+* **Web server** - Server HTTP provides access to DAG/task status information
Review comment:
Updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
