XD-DENG commented on a change in pull request #4919: [AIRFLOW-4093] Throw
exception if job failed or cancelled or retry too many times
URL: https://github.com/apache/airflow/pull/4919#discussion_r267356523
##########
File path: airflow/contrib/operators/aws_athena_operator.py
##########
@@ -74,7 +76,16 @@ def execute(self, context):
self.result_configuration['OutputLocation'] = self.output_location
self.query_execution_id = self.hook.run_query(self.query,
self.query_execution_context,
self.result_configuration, self.client_request_token)
- self.hook.poll_query_status(self.query_execution_id)
+ query_status = self.hook.poll_query_status(self.query_execution_id,
self.max_tries)
+
+ if not query_status or query_status in AWSAthenaHook.FAILURE_STATES:
+ raise Exception(
+ 'Athena job failed. Final state is {}, query_execution_id is
{}.'
+ .format(query_status, self.query_execution_id))
+ elif query_status in AWSAthenaHook.INTERMEDIATE_STATES:
Review comment:
Hi @bryanyang0528 , if you check
https://github.com/apache/airflow/blob/4655c3f2bbd6dbb442a9c8482559748bd9db0bd7/airflow/contrib/hooks/aws_athena_hook.py#L123-L140
You will notice that `query_state is None` or `query_state in
self.INTERMEDIATE_STATES` would not result in `break`. Instead,
`poll_query_status()` will only end with `query_state is None` or `query_state
in self.INTERMEDIATE_STATES` when `max_tries` is reached. It may be a too
strong assumption to say "`query_status` is `None` means `failed`".
On the other hand, `else:` (including `FAILURE_STATES`) cause an explicit
`break`, which is for sure `failed`.
Hope this clarifies.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
