I'm not sure that I understand the blog post. In what way are make and Airflow alike... at all? Airflow manages dependencies in the sense that it can build directed acyclic graphs (DAGs) of tasks that depend on each other in an execution order and output sense, but that isn't the same as managing build or output dependencies, which I understand to be make's bread and butter. Airflow *is* a task scheduler. That isn't an "additional feature" over the core software. Everything else around it is about managing and running those tasks in potentially highly available environments, and is deeply configurable for use with remote executors and other advanced use cases.
It would be absurd to try to use it as a make replacement, unless I'm missing something huge about what make is capable of and used for. Am I missing something about make that makes Airflow an apt "alternative" to consider? I'm not surprised it feels overblown as an alternative or even in and of itself to the author of that blog post, if they are approaching it as an alternative to make... As a side note, I've enjoyed using https://github.com/casey/just for when I just need something for writing reusable, simple commands (without needing to write .PHONY everywhere), though it has some quirks you only run into after using it for a while. It's very good for the simple job of writing out reusable commands and composing an environment for executing those commands in. It does not know about dependencies in the same way as make, however, and if I need the advanced parts of make's dependency graph, I might try composing the two... or just stick with make. As it stands, I rarely need the dependency graph for commands and builds myself, and when I have needed it, the simplest version of it as just a bash file test has been sufficient... though I know my usecases have been simple compared to some things folks do with make. Still, I can't really see how make and Airflow are sensible to compare... - Sara (she/her <https://pronoun.is/she/>) ----- Original message ----- De: Rodney Brown via luv-main <[email protected]> Para: Luv Main <[email protected]> Assunto: Diomidis Spinellis blog: Unix make vs Apache Airflow: (simple-rolap, a tool suite for relational online analytical processing) Data: Quarta, 16 de outubro de 2024 18:51 These may interest https://www.spinellis.gr/blog/20241015/ Unix make vs Apache Airflow ... With *make* approaching its 50th birthday, before writing the column I looked for possible modern and better alternatives I might be ignoring. ... Another suggested tool that matched my requirements was Apache Airflow <https://airflow.apache.org/>, which I decided to investigate further. I was disappointed by what I found. ... D. Spinellis, "Modular Data Analytics," in IEEE Software, vol. 41, no. 5, pp. 20-23, Sept.-Oct. 2024, doi: 10.1109/MS.2024.3409988. keywords: {Analytical models;Query processing;Online services;Data analysis}, A sophisticated analysis of data is often based on relational analytical processing (ROLAP) methods. These involve using SQL queries on a relational database system to perform on the data operations such as slice and dice, drill down, and roll up. Such queries can be complex, involving tens of tables and many intermediate steps. ... Handling Dependencies To facilitate the development of maintainable, time-efficient, and testable ROLAP queries, I developed simple-rolap (github.com/dspinellis/simple-rolap/), a small open source software framework that automates the dependency analysis and orchestrates the execution of multiple modular queries. ... The frameworkâs design is based on convention over configuration centered on two databases. A main database contains the primary queried data, which are assumed to be infrequently modified. A secondary database is used for caching derived intermediate ROLAP results. _______________________________________________ luv-main mailing list -- [email protected] To unsubscribe send an email to [email protected]
_______________________________________________ luv-main mailing list -- [email protected] To unsubscribe send an email to [email protected]
