turbaszek commented on a change in pull request #10802: URL: https://github.com/apache/airflow/pull/10802#discussion_r485137457
########## File path: README.md ########## @@ -132,20 +132,13 @@ Other ways of retrieving source code are "convenience" methods. For example, tag > Note: Airflow Summit 2020's "Production Docker Image" talk where context, > architecture and customization/extension methods are > [explained](https://youtu.be/wDr3Y7q2XoI). -## Beyond the Horizon +## Project Focus -Airflow **is not** a data streaming solution. Tasks do not move data from -one to the other (though tasks can exchange metadata!). Airflow is not -in the [Spark Streaming](http://spark.apache.org/streaming/) -or [Storm](https://storm.apache.org/) space, it is more comparable to -[Oozie](http://oozie.apache.org/) or -[Azkaban](https://azkaban.github.io/). +Airflow works best with DAGs that are mostly static and slowly changing. When the DAG structure is similarly from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](http://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/). -Workflows are expected to be mostly static or slowly changing. You can think -of the structure of the tasks in your workflow as slightly more dynamic -than a database structure would be. Airflow workflows are expected to look -similar from a run to the next, this allows for clarity around -unit of work and continuity. +Airflow tasks are ideally idempotent, and do not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [Xcom feature](https://airflow.apache.org/docs/stable/concepts.html#xcoms)). Review comment: You are right, I just think it would be wise to emphasize that it not petabytes of data ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
