Also Apache Hop provides some sort of integration to Beam. Hop is divided to Workflows (you ask) and Pipelines (similar to Beam). As far as I understand (!) Hop's workflows are not persistent, ie it can't recover from a node failure like Airflow can.
On Thu, Dec 14, 2023 at 3:18 PM data_nerd_666 <[email protected]> wrote: > Hi all, > > I am new to apache beam, and am very excited to find beam in apache > community. I see lots of use cases of using apache beam for data flow > (process large amount of batch/streaming data). I am just wondering whether > I can use apache beam for control flow (ETL workflow). I don't mean the > spark/flink job in the ETL workflow, I mean the ETL workflow itself. > Because ETL workflow is also a DAG which is very similar as the abstraction > of apache beam, but unfortunately I didn't find such use cases on internet. > So I'd like to ask this question in beam community to confirm whether I can > use apache beam for control flow (ETL workflow). If yes, please let me know > some success stories of this. Thanks > > > > -- Sincerely yours Mikhail Khludnev
