amoghrajesh commented on PR #67118: URL: https://github.com/apache/airflow/pull/67118#issuecomment-4532674200
> Another question I would have is: earlier when I was talking to @kaxil , seems to me the direction to "build a triggerer for this purpose and make this operator deferrable" is preferred? > > Do we need to build two modes? Would like to understand @kaxil 's thoughts on this. > > No strong opinion on this though. @XD-DENG The mixin should not be thought of a second mode, rather it makes the existing sync path crash-safe. `SparkSubmitOperator` has always had the option to run synchronously and this addition just means if the worker dies mid-poll, the next retry reconnects to the running driver instead of blindly resubmitting or not having an option to recover safely. Sync becomes survivable without any architecture change on the user's side. If a team is/can already using deferrable operators, nothing here changes for them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
