Abacn commented on code in PR #39826: URL: https://github.com/apache/beam/pull/39826#discussion_r3837557327
########## sdks/python/apache_beam/testing/README.md: ########## @@ -0,0 +1,109 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Audit Log and Justification: Implementation of TestPubsubContext +#### Date: 2026-08-20 + +This document serves as a living log of resource leaks detected in our GCP environment due to failures or premature interruptions in CI/CD pipelines (such as Jenkins or GitHub Actions). The purpose of this log is to centralize evidence of orphaned components and provide technical and economic justification for implementing the TestPubsubContext lifecycle manager across all integration tests. The engineering team is encouraged to document any new leaks detected in future test suites in this file to maintain strict control over infrastructure consumption. + +## Detection Methodology (Audit Script) + +To identify optimization opportunities and prevent the accumulation of phantom resources in GCP, we developed an automated audit script (`auditar_backlog_wordcount.py`). This script connects to the `apache-beam-testing` project and actively filters for orphaned subscriptions based on the prefixes used by our test suites. +``` +python + +from google.cloud import pubsub_v1 +from datetime import datetime, timezone, timedelta + +def audit_wordcount_subscriptions(project_id): + subscriber = pubsub_v1.SubscriberClient() + project_path = f"projects/{project_id}" + + print(f"Searching for orphan subscriptions with 'resource_sub' prefixes in: {project_id}...\n") + print(f"{'Orphan Subscription Detected':<70} | {'Status'}") + print("-" * 90) + total_leaks = 0 + + # List subscriptions in the GCP project + for sub in subscriber.list_subscriptions(project=project_path): + sub_name = sub.name.split("/")[-1] + + # Filter by those created by wordcount_it_test ('name of the resource') + if sub_name.startswith("resource_sub") or "resource_subscription" in sub_name: + total_leaks += 1 + print(f"{sub_name:<70} | ACTIVE (ORPHAN)") + + print("-" * 90) + print(f"Diagnosis: Detected {total_leaks} active orphan 'resource_sub' subscriptions in GCP.") + +if __name__ == "__main__": + audit_wordcount_subscriptions("apache-beam-testing") + +``` + +## Evidence: Leaks in Pub/Sub Integration Tests (`psit_`) + +During the execution of the main integration test suite, the standard cleanup mechanism proved insufficient when tests failed or were abruptly aborted. + +### **Critical findings:** Review Comment: This dairy like log doesn't belong to Beam repo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
