Ohashiro commented on issue #44618: URL: https://github.com/apache/airflow/issues/44618#issuecomment-2567785221
Hello! Thank you very much for taking the time to look at the code and making this suggestion. I tried to reduce the `check_interval` to 5, unfortunately it did not seem to fix the issue on our side. Also, I thought that the default timeout was 1 week because of this line: https://github.com/apache/airflow/blob/608de6b97ef04307559d79b2044899adfb9b2a93/providers/src/airflow/providers/microsoft/azure/triggers/powerbi.py#L57 But maybe I got this wrong. Am I mistaken? Regarding the bug, I think the issue may come from the very first request to get the refresh status, at this line: https://github.com/apache/airflow/blob/f7da5e48a2740a2bfb1c3dd57bb463ae29599a26/providers/src/airflow/providers/microsoft/azure/triggers/powerbi.py#L116 This request is made by the PowerBi hook (using the method `get_refresh_details_by_refresh_id`) and I think that, sometimes, the API is not yet "up-to-date" with the latest refresh IDs and can't find the status of our refresh ID. So, we fall in this case: https://github.com/apache/airflow/blob/f7da5e48a2740a2bfb1c3dd57bb463ae29599a26/providers/src/airflow/providers/microsoft/azure/hooks/powerbi.py#L159 and the hook raises the error "Unable to fetch the details of dataset refresh with Request Id" (which is also what I find in my Airflow logs). Do you think it makes sense? I did try to add a retry on this request (and it seemed to fix the bug), but I understand that you don't want to introduce this type of custom retry mechanism. Maybe another option could be to wait "check_interval" seconds BEFORE sending the first request. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
