XD-DENG commented on a change in pull request #4919: [AIRFLOW-4093] Add
exception if job failed or cancelled or retry too many times
URL: https://github.com/apache/airflow/pull/4919#discussion_r265860080
##########
File path: airflow/contrib/hooks/aws_athena_hook.py
##########
@@ -127,14 +127,21 @@ def poll_query_status(self, query_execution_id,
max_tries=None):
elif query_state in self.INTERMEDIATE_STATES:
self.log.info('Trial {try_number}: Query is still in an
intermediate state - {state}'
.format(try_number=try_number,
state=query_state))
+ elif query_state in self.FAILURE_STATES:
+ self.log.error('Trial {try_number}: Query is failed - {state}'
+ .format(try_number=try_number,
state=query_state))
+ raise Exception('Athena job
{state}.'.format(state=query_state))
else:
self.log.info('Trial {try_number}: Query execution completed.
Final state is {state}'
.format(try_number=try_number,
state=query_state))
final_query_state = query_state
break
Review comment:
Hi @bryanyang0528 . I had another check. I don't think we need to add
exception here.
This function `poll_query_status()` helps "_Poll the status of submitted
athena query until query state reaches final state_". If you check the current
implementation: if the `query_state` is in `self.FAILURE_STATES`, it will go to
`else` block, in that block the final `query_state` ( one of
`self.FAILURE_STATES`) will be assigned to `final_query_state`, then the
while-loop will be stopped by `break`. Eventually, `final_query_state` will be
returned.
But your change will break this logic. If the query_state is one of
`self.FAILURE_STATES`, we're not expecting an Exception here. Instead, we
expect the query_state to be returned (no matter if it's one of
`self.FAILURE_STATES`)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
