Hi there!

Would it make sense to add an operator that is both the EmrAddStep operator
and the step sensor?

In a past role we were using Airflow heavily for all things EMR, and I
found myself writing an Operator that combined the emr_add_step operator
and the sensor, it made the pipelines simpler (less operator instances per
DAG) and retries were easy

There is still value in keeping those 2 other classes around when we don't
care about the result of an EMR step or we are polling for the completion
of an EMR step we did not start from Airflow, but for most tasks wouldn't a
"merged operator" make sense?

Thanks!
Daniel

Reply via email to