alexpilon666 opened a new issue, #33835: URL: https://github.com/apache/airflow/issues/33835
### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened Hello! We've migrated away from Airflow v2.0.2 to v2.4.2 around 10 days ago and been noticing a lot of errors in our Worker nodes (we have two nodes in Dev, 3 in Prod), all the same thing for different DAGs and Variables. See logs: [airflow-dev-worker-1.log](https://github.com/apache/airflow/files/12457352/airflow-dev-worker-1.log). The variables actually do exists in Airflow, see attached screenshot (values ommited for privacy/security reasons) . We thought this might be a specific issue in v2.4.2 so we upgraded to v2.6.3 (latest available version when deploying using Bitnami Helm Charts), and we're still having the same issues. ### What you think should happen instead N/A (everything included in previous section) ### How to reproduce - Have Airflow v2.0.2 installed with running DAGs - Upgrade to Airflow v2.4.2 - Check worker logs and see above shown logs - Upgrade to Airflow v2.6.3 - Check worker logs and still see above shown logs. ### Operating System Debian 11 (running in containers ### Versions of Apache Airflow Providers apache-airflow-providers-databricks==1.0.1 ### Deployment Other 3rd-party Helm chart ### Deployment details Bitnami Helm charts. - v14.0.3 for Airflow 2.4.2 - v15.0.0 for Airflow 2.6.3 - Helm version: 3.11.0 - K8s version: 1.23.5 (We're aware this version is deprecated/EOL, we're retiring this cluster soon) ### Anything else Problem occurs all the time (as seen in the attached logs). Unsure if this is related, but we're also having the same issues as some users, as seen in this discussion: https://github.com/apache/airflow/discussions/26255. This also started happening at an alarming rate after the upgrade to new versions. Either our workers will return a `404 Client Error: NOT FOUND for url: <url>` for our tasks (not all, but a lot, and often), or they'll return `*** Could not read served logs: Request URL is missing an 'http://' or 'https://' protocol.`, which is somehow worse. If related, we might be solving two issues at once, but I doubt it is. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
