omar-massfih opened a new issue, #74597:
URL: https://github.com/apache/airflow/issues/74597
### Provider and affected code
`apache-airflow-providers-microsoft-azure==15.2.0`. The same polling path is
present on `main`.
`PowerBIHook.get_refresh_details_by_refresh_id()` searches refresh history.
`raw_to_refresh_details()` maps history status `Unknown` to `In Progress` and
retains only `requestId`, `status`, and `serviceExceptionJson`. It drops
`refreshType` and never requests execution details. `PowerBITrigger` therefore
continues polling until its configured timeout if that history entry continues
reporting `Unknown`, even when enhanced-refresh execution details would expose
a terminal state. Execution engine `messages`, `extendedStatus`, and
attempt-level service exceptions are unavailable to the task.
This is confirmed code behavior, reproduced with mocked responses. I have
not verified a live Power BI response pair with history `Unknown` and execution
`extendedStatus: Failed`; the fixture below is synthetic, not a captured
production response. Microsoft's documented in-progress enhanced-refresh
example legitimately has general `status: Unknown`, so `Unknown` itself must
not become a failure.
### Sanitized reproduction / regression fixture
After submitting an enhanced refresh, retain its returned refresh ID. A
representative enhanced request body is `{"type": "full", "commitMode":
"transactional"}`. No real connection or dataset is needed to reproduce the
polling gap:
```python
from unittest.mock import AsyncMock, patch
import pytest
from airflow.providers.microsoft.azure.hooks.powerbi import PowerBIHook
@pytest.mark.asyncio
async def test_enhanced_refresh_failure():
hook = PowerBIHook()
history = {
"value": [{
"requestId": "submitted-refresh",
"refreshType": "ViaEnhancedApi",
"status": "Unknown",
}]
}
execution = {
"status": "Unknown",
"extendedStatus": "Failed",
"messages": [{"type": "Error", "message": "Synthetic engine
failure"}],
"serviceExceptionJson": '{"errorCode":"SyntheticRefreshError"}',
"refreshAttempts": [{
"serviceExceptionJson": '{"errorDescription":"Synthetic attempt
failure"}'
}],
}
with patch.object(hook, "run", new_callable=AsyncMock) as run:
run.side_effect = [history, execution]
details = await hook.get_refresh_details_by_refresh_id(
dataset_id="example-dataset",
group_id="example-group",
refresh_id="submitted-refresh",
)
assert details["status"] == "Failed"
assert "Synthetic engine failure" in details["error"]
```
On the affected implementation, the assertion receives `In Progress` and
only the history request is made. The second mocked response is never consumed.
### Expected behavior
Keep standard-refresh history polling. For the matching `ViaEnhancedApi`
history entry, request `GET
/groups/{groupId}/datasets/{datasetId}/refreshes/{refreshId}` using the
submitted refresh ID and the existing provider authentication.
Use meaningful `extendedStatus`, falling back to general `status`:
`Completed` succeeds; `Failed`, `TimedOut`, `Cancelled`, and `Disabled` fail.
Active or unknown states keep polling within the existing timeout. Include
engine messages and top-level / attempt-level service exceptions in failure
output. Messages alone must not terminate a refresh that is still running or
retrying.
Microsoft's [execution-details
API](https://learn.microsoft.com/en-us/rest/api/power-bi/datasets/get-refresh-execution-details-in-group)
documents `extendedStatus`, engine messages, refresh attempts, and HTTP 202
for ongoing refreshes. Its [refresh-history
API](https://learn.microsoft.com/en-us/rest/api/power-bi/datasets/get-refresh-history-in-group)
documents `refreshType`.
Related reports: #44613 concerns a task succeeding for a failed refresh;
#44618 concerns a refresh absent from history. Both reported provider 11.1.0.
They provide context but are not being marked as duplicates of this enhanced
execution-details polling gap.
I am preparing a fix with hook and trigger regression coverage.
---
Drafted-by: Codex (GPT-6) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
