potiuk commented on code in PR #37609: URL: https://github.com/apache/airflow/pull/37609#discussion_r1499108113
########## docs/apache-airflow/index.rst: ########## @@ -19,56 +19,71 @@ What is Airflow™? ========================================= `Apache Airflow™ <https://github.com/apache/airflow>`_ is an open-source platform for developing, scheduling, -and monitoring batch-oriented workflows. Airflow's extensible Python framework enables you to build workflows +and monitoring distributable, batch-oriented workflows. Airflow's extensible Python framework enables you to build workflows connecting with virtually any technology. A web interface helps manage the state of your workflows. Airflow is -deployable in many ways, varying from a single process on your laptop to a distributed setup to support even -the biggest workflows. +deployable in many ways, varying from a single process on your laptop to a distributed setup across a network of many +machines to support even the largest of workflows. Workflows as code ========================================= -The main characteristic of Airflow workflows is that all workflows are defined in Python code. "Workflows as -code" serves several purposes: +The main characteristic of Airflow workflows is that all workflows are defined in Python code as a +`Directed Acyclic Graph <https://en.wikipedia.org/wiki/Directed_acyclic_graph>`__. + +"Workflows as code" serves several purposes: - **Dynamic**: Airflow pipelines are configured as Python code, allowing for dynamic pipeline generation. - **Extensible**: The Airflow™ framework contains operators to connect with numerous technologies. All Airflow components are extensible to easily adjust to your environment. -- **Flexible**: Workflow parameterization is built-in leveraging the `Jinja <https://jinja.palletsprojects.com>`_ templating engine. +- **Flexible**: Workflows can be easily parameterized by using the built-in `Jinja <https://jinja.palletsprojects.com>`_ templating engine. Take a look at the following snippet of code: .. code-block:: python - from datetime import datetime + # Create tasks using a decorated DAG and pass values between those tasks. Review Comment: I like the proposal in general, but I have one big ask: I think we should not make the mapped-task example as the first thing the user sees. This is really a bit more "Advanced" use case and not one that ss the main strength of airflow, also it introduces quite a few concepts at once and they might be difficult to grasp by a new user. Immediate questions: * shoudl I always do that even in a simple case ? * what is that expand thing? * what the hell those values are. I think those **maybe** should folllow the basic example as a more advance case (which I would find very useful to add). This should be clearly marked with something like "while simple workflows are simple, you can also do much more complex tasks where - for example - you can replicate tasks to run in parallel, allowing airflow fo run workflow with semantics similar to MapReduce" - followed by the new example. That would be more appealing to the users who hit the page and have no idea about Airflow. This should be similar in concept to the "architecture overview" - where we start with a basic, simplest deployment of Airflow and then follow that with more complex cases that focus on distribution and eventually security-focused deployments. See https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/overview.html#architecture-diagrams -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
