Abacn opened a new issue, #23020: URL: https://github.com/apache/beam/issues/23020
### What happened? Multiple performance tests perma-red since Aug 31st, including JDBC, Kafka, IO Tests on HDFS. The cause is the number of existing k8s endpoints has run out of quota. These end points all have namespace beam-performancetests-[testname]-[test-number]. From the test number it can be seen there are all old tests. Usually k8s service gets deleted at the end of the test. But if jenkins worker fails in the middle, which happened on Aug 30, those service is never cleaned up. Another leak found is that rc validation script creates a k8s cluster named after "xlang-kafka" for kafka taxi test: https://github.com/apache/beam/blob/31561e2ff13147aa80f9f811e2a94ebe57b25374/release/src/main/scripts/run_rc_validation.sh#L501 , but never gets deleted. Consequently every time rc validation there is a new cluster created and at some point it will reach cluster quota. There should be a cron job automatically clean up these services and clusters. ### Issue Priority Priority: 1 ### Issue Component Component: test-failures -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
