o-nikolas commented on issue #27979: URL: https://github.com/apache/airflow/issues/27979#issuecomment-1341586646
> @o-nikolas > > Few questions regarding the approach Being an operator, this would involve coupling it with each of possibly hundreds of steps and thus hardly find any use case, when actually we just wanted to stop the existing running/pending step and do a new addstep of the same step name ( However, we can still implement this Operator ) > > I was thinking, we could emulate the Clear functionality on this Operator : > > 1. Call clear on this operator -> With an extra parameter in the `EmrAddStepsOperator` as `cancel_step_when_cleared` boolean which clears the existing step and adds a new step ( works as a retrigger ) whenever clear is called on this operator and acts as a normal `EmrAddStepsOperator` ( Just add a new step - no cancel ) whenever this parameter is false > > 2. Call clear on this operator -> By default, If there is an existing named step in the current cluster with status 'PENDING/RUNNING' , it will send a cancel-step before calling a addstep -> add a new step of the same name for execution > > > I would require your thoughts on this , if I am going too overboard with these approaches sweat_smile Very interesting idea! I think it sounds like a handy feature. Though I don't know EMR itself as a service, and the common workflows associated with it, well enough to say which of the two approaches is the right call. Perhaps @dacort and @syedahsn can weigh in on the topic -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
