potiuk opened a new pull request #11079:
URL: https://github.com/apache/airflow/pull/11079
We introduced deletion of the old artifacts as this was a
suspected culprit of Kubernetes Job failures. It turned out
eventually that those Kubernetes Job failures were caused by
the #11017 change, but it's good to do housekeeping of the
artifacts anyway.
The delete workflow action introduced in a hurry had two problems:
* it runs for every fork if they sync master. This is a bit
too invasive
* it fails continuously after 10 - 30 minutes every time
as we have too many old artifacts to delete (GitHub has
90 days retention policy so we have likely tens of
thousands of artifacts to delete)
* it runs every hour and it causes occasional API rate limit
exhaution (because we have too many artifacts to loop trough)
This PR introduces filtering with the repo, changes frequency
of deletion to be 4 times a day and adds script that we are
running manualy to delete those excessive artifacts now. Eventually
when the number of artifacts goes down the regular job shoul delete
maybe few hundreds of artifacts appearing within the 6 hours window
and it should stop failing.
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
