tszerszen commented on issue #10381:
URL: https://github.com/apache/airflow/issues/10381#issuecomment-688672324
In package `airflow.providers.google.cloud.operators.dataproc` each
DataprocSubmitJobOperator inherits from `DataprocJobBaseOperator`.
`DataprocJobBaseOperator` has the following implementation of `on_kill`
method (lines 984-992):
```
def on_kill(self):
"""
Callback called when the operator is killed.
Cancel any running job.
"""
if self.dataproc_job_id:
self.hook.cancel_job(
project_id=self.project_id, job_id=self.dataproc_job_id,
location=self.region
)
```
@turbaszek wrote:
> This operator should implement on_kill method using cancel_job method of
DataprocHook so in case of termination we cancel running job.
Considering present implementation of `on_kill` method of
`DataprocJobBaseOperator` isn't it already done?
The code call's method `cancel_job` of `DataprocHook` when `on_kill` method
is run and property `dataproc_job_id` is present.
If I understand this correctly, the only thing that method lacks is:
> This option probably should be configurable (for example cancel_on_kill)
because of request_id parameter in a job definition
Thus the following code might do the job:
```
def on_kill(self, cancel_on_kill=True, request_id=None):
"""
Callback called when the operator is killed.
Cancel any running job.
Parameters:
cancel_on_kill (bool): Cancel job if true, defaults to True
request_id (job_id): Job ID to cancel, defaults to object property
dataproc_job_id
"""
if not cancel_on_kill:
return
job_id = self.dataproc_job_id
if request_id:
job_id = request_id
if self.dataproc_job_id:
self.hook.cancel_job(
project_id=self.project_id, job_id=job_id, location=self.region
)
```
Did I understand this issue correctly? If not I guess calling hook's
`cancel_job` method is not enough and I will investigate it further.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
