ambika-garg commented on issue #44618:
URL: https://github.com/apache/airflow/issues/44618#issuecomment-2572273229
Hi @Ohashiro, thank you for bringing this up and creating the PR to address
it. After reviewing the conversation, I see the issue lies in `get refresh
history` function in hook class, as sometimes it fails to return the "dataset
refresh histories" leading the dataset refresh to be marked as fail, even if it
actually succeeds. Please correct me if I’ve misunderstood.
So, I suggest, we should add the retry mechanism to the get refresh history
function only as below, it will retry to fetch histories, if it still fails,
then we just throw the exception. That would also mean we don't need any extra
exception class as you created "PowerBIDatasetRefreshStatusExecption"
```
@tenacity.retry(
stop=tenacity.stop_after_attempt(3),
wait=tenacity.wait_random_exponential(),
reraise=True,
retry=tenacity.retry_if_exception(should_retry_creation),
)
async def get_refresh_history(
self,
dataset_id: str,
group_id: str,
) -> list[dict[str, str]]:
"""
Retrieve the refresh history of the specified dataset from the given
group ID.
:param dataset_id: The dataset ID.
:param group_id: The workspace ID.
:return: Dictionary containing all the refresh histories of the
dataset.
"""
try:
response = await self.run(
url="myorg/groups/{group_id}/datasets/{dataset_id}/refreshes",
path_parameters={
"group_id": group_id,
"dataset_id": dataset_id,
},
)
refresh_histories = response.get("value")
if not refresh_histories: # Retry if refresh_histories is None
or empty
raise PowerBIDatasetRefreshException(
"Refresh histories are empty; retrying..."
)
return [self.raw_to_refresh_details(refresh_history) for
refresh_history in refresh_histories]
except Exception as error:
raise PowerBIDatasetRefreshException(f"Failed to retrieve
refresh history due to error: {error}")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
