michalsosn commented on issue #10385:
URL: https://github.com/apache/airflow/issues/10385#issuecomment-1758577018
### operators in dataproc module
```
not killable, don't start a process:
class DataprocCreateClusterOperator(GoogleCloudBaseOperator)
class DataprocScaleClusterOperator(GoogleCloudBaseOperator)
class DataprocDeleteClusterOperator(GoogleCloudBaseOperator)
class DataprocUpdateClusterOperator(GoogleCloudBaseOperator)
class DataprocDeleteBatchOperator(GoogleCloudBaseOperator)
class DataprocGetBatchOperator(GoogleCloudBaseOperator)
class DataprocListBatchesOperator(GoogleCloudBaseOperator)
class DataprocCancelOperationOperator(GoogleCloudBaseOperator)
class DataprocCreateWorkflowTemplateOperator(GoogleCloudBaseOperator)
killable, have on_kill, deprecated (by DataprocSubmitJobOperator):
class DataprocJobBaseOperator(GoogleCloudBaseOperator) // they have
on_kill implemented
def on_kill(self) -> None:
"""Callback called when the operator is killed; cancel any
running job."""
if self.dataproc_job_id:
self.hook.cancel_job(project_id=self.project_id,
job_id=self.dataproc_job_id, region=self.region)
class DataprocSubmitPigJobOperator(DataprocJobBaseOperator)
class DataprocSubmitHiveJobOperator(DataprocJobBaseOperator)
class DataprocSubmitSparkSqlJobOperator(DataprocJobBaseOperator)
class DataprocSubmitSparkJobOperator(DataprocJobBaseOperator)
class DataprocSubmitHadoopJobOperator(DataprocJobBaseOperator)
class DataprocSubmitPySparkJobOperator(DataprocJobBaseOperator)
killable, have on_kill:
class DataprocSubmitJobOperator(GoogleCloudBaseOperator) // on_kill
added 18.08.2020 in https://github.com/apache/airflow/issues/10381
def on_kill(self):
if self.job_id and self.cancel_on_kill:
self.hook.cancel_job(job_id=self.job_id,
project_id=self.project_id, region=self.region)
class DataprocCreateBatchOperator(GoogleCloudBaseOperator)
def on_kill(self):
if self.operation:
self.operation.cancel()
killable, no on_kill:
class
DataprocInstantiateWorkflowTemplateOperator(GoogleCloudBaseOperator)
class
DataprocInstantiateInlineWorkflowTemplateOperator(GoogleCloudBaseOperator)
```
Since there was a discussion about implementing the `on_kill` method in all
operators, I've prepared a summary.
It seems that, among the operators in the dataproc module, only the
`Workflow` operators, which are the target of this issue, do not have it
implemented yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
