Hi!
You probably want to take a look at a scheduler such as
Airflow(https://airflow.apache.org/) or
Luigi(https://luigi.readthedocs.io/en/stable/index.html). If you are on
Google they have a Cloud Composer(https://cloud.google.com/composer/)
which is Airflow underneath. On AWS Glue is probably an option.
/Magnus
On 2019-12-31 13:04, Gershi, Noam wrote:
Hi,
What is the best way to scheduler Apache beam pipelines (execute the
same pipeline once per day, for the data of the last day)?
Will it be a different solution per runner?
citi_logo_mailciti_logo_mail*Noam Gershi*
Software Developer
*T*:+972 (3) 7405718
Mail_signature_blue
