swapz-z commented on issue #27979:
URL: https://github.com/apache/airflow/issues/27979#issuecomment-1341526159
@o-nikolas
Few questions regarding the approach
Being an operator, this would involve coupling it with each of possibly
hundreds of steps and thus hardly find any use case, when actually we just
wanted to stop the existing running/pending step and do a new addstep of the
same step name ( However, we can still implement this Operator )
I was thinking, we could emulate the Clear functionality on this Operator :
1. Call clear on this operator -> With an extra parameter in the
`EmrAddStepsOperator` as `cancel_step_when_cleared` boolean which clears the
existing step and adds a new step ( works as a retrigger ) whenever clear is
called on this operator and acts as a normal `EmrAddStepsOperator` ( Just add a
new step - no cancel ) whenever this parameter is false
2. Call clear on this operator -> If there is an existing named step in the
current cluster with status 'PENDING/RUNNING' , it will send a cancel-step for
these steps -> And add a new step of the same name for execution
I would require your thoughts on this , if I am going too overboard with
these approaches 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
