pabloem opened a new issue, #33644: URL: https://github.com/apache/beam/issues/33644
### What needs to happen? Many tests and examples in the Beam codebase create GCP assets that are not cleaned up. Some examples: - BQ datasets - Pubsub topics, subscriptions - BigTable instances? - Others? The task is to build automation that can **safely** disable-then-clean-up leftover assets from the GCP environment. The tool would: - List existing inventory of various kinds of assets - Figure out which ones are meant to be cleaned up (i.e. short-lived test assets, vs. long-lived project assets like base sources, base buckets, etc). - Wait a week or so - Disable and (eventually?) delete these assets Why? - A lot of these assets have billing implications for the project - A lot of these assets make navigating the project more difficult (e.g. reviewing Bigquery datasets) Risks: - We need to roll this out carefully to make sure we don't delete important assets. We'd probably start with strict lists of inclusions (i.e. everything that's not included will be excluded). ### Issue Priority Priority: 2 (default / most normal work should be filed as P2) ### Issue Components - [ ] Component: Python SDK - [ ] Component: Java SDK - [ ] Component: Go SDK - [ ] Component: Typescript SDK - [ ] Component: IO connector - [ ] Component: Beam YAML - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [x] Component: Infrastructure - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Samza Runner - [ ] Component: Twister2 Runner - [ ] Component: Hazelcast Jet Runner - [ ] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
