yegeniy commented on issue #6281: Run batches of (self-terminating) EMR JobFlows [AIRFLOW-5763] URL: https://github.com/apache/airflow/pull/6281#issuecomment-548026133 > couldn't this be implemented as multiple EmrCreateJobFlowOperators and some sensors at the end Yes, one of the use-cases I described could be replaced with existing EMR operators and sensors. I probably shouldn’t have emphasized this use-case as it’s not the main/novel use case. The novel use-case is that this operator allows for dynamic behavior at DagRun time without overwriting the DAG definition itself (that is, we get to keep the history in the UI and have parallel runs each with different definitions). I agree that this isn’t a common use case for Airflow, but internally I’ve found it to be a handy way to quickly iterate, develop, and run AWS EMR-based workflows that require multiple clusters to execute in sequence. Please feel free to close out this PR if it doesn’t belong in `apache/airflow`. I can just leave the code up on my branch if someone finds it useful in the future. On Wed, Oct 30, 2019 at 8:09 AM Ash Berlin-Taylor <[email protected]> wrote: > I'm not sure this operator is really useful enough to belong in Airflow > core. If I'm reading the comments/docs correctly couldn't this be > implemented as multiple EmrCreateJobFlowOperators and some sensors at the > end? > > Right now I don't feel this needs a whole new operator to support it. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/airflow/pull/6281?email_source=notifications&email_token=AACHPQUVAKYUETQ45EHKTOTQRF2NZA5CNFSM4I6MYH4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECT5OUA#issuecomment-547870544>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AACHPQW7DPGYNPW2T4WAFM3QRF2NZANCNFSM4I6MYH4A> > . >
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
