HansMarcus01 opened a new pull request, #39685:
URL: https://github.com/apache/beam/pull/39685
## Summary of changes
This pull request introduces TestPubsubContext, a reusable, robust, and safe
lifecycle manager designed specifically for Python GCP integration tests that
interact with Google Cloud Pub/Sub.
Similar to the Java SDK's TestPubsub.java rule, this Python Context Manager
provides a centralized utility to track, monitor, and clean up temporary
Pub/Sub topics and subscriptions created during test executions, significantly
reducing the risk of orphaned GCP resources and billing leaks (FinOps leaks) in
shared testing environments like apache-beam-testing.
## Technical Architecture & Triple-Layer Security Model
The TestPubsubContext class does not force the creation of resources, which
preserves compatibility with existing tests that implement custom provisioning.
Instead, it acts as an active monitor that registers created assets via
register_topic() and register_subscription() to guarantee their teardown using
a redundant Triple-Layer Security model:
- Layer 1: Active Teardown (Context Manager exit):
Wrapped in a standard Python with statement, the block guarantees that the
registered temporary topics and subscriptions are physically destroyed
immediately upon exit, regardless of whether the test finishes successfully or
encounters an assertion failure/exception.
- Layer 2: Cascading Teardown (Java-inspired Cleanup):
Before deleting a registered topic, the manager queries the GCP Pub/Sub API
for any third-party or sub-process subscriptions that might have attached
dynamically to our test topic during runtime, deleting them in a cascading
sequence to ensure no residual ghost subscriptions remain.
-Layer 3: Passive Autodestrucion (GCP TTL Backstop):
For catastrophic test execution failures (e.g., CI/CD runner VM power cuts
or hard cancellations), where the python thread is destroyed instantly before
Layer 1 can run, the subscriptions are created with an explicit expiration
policy of 1 day (86400 seconds) (expiration_policy), ensuring GCP will
automatically garbage-collect the orphaned resources within 24 hours.
## Graceful Teardown Policy for Debugging on Failures
To preserve developer productivity and ensure seamless debugging, the
context manager implements a Graceful Teardown policy when a test fails
(exc_type is not None):
Successful Runs: Immediate deletion of all registered topics and
subscriptions to reclaim 100% of storage space.
Failed Runs: Immediate deletion is bypassed, and the GCP TTL (Layer 3) is
allowed to backstop the resources. This grants developers a 24-hour grace
window to inspect the message backlog and error logs in the GCP Console before
the resources are automatically destroyed.
### Real-world Validation Output (Dry Run local execution)
We successfully validated the execution of `TestPubsubContext` integrated
with `pubsub_io_perf_test.py` running locally in our workstation against the
`apache-beam-testing` project.
The console log output below demonstrates that the manager dynamically
intercepts, structures, and registers **both topics and both subscriptions**
with unique execution IDs. It validates that the audit logging mechanism works
flawlessly prior to the GCP API call:
```text
PYTHONPATH=../../.. python pubsub_io_perf_test.py \
--test-pipeline-options=" \
--runner=DirectRunner \
--project=apache-beam-testing \
--pubsub_namespace_prefix=perf-test-local- \
--wait_until_finish_duration=15000 \
--input_options='{\"num_records\": 10, \"key_size\": 1, \"value_size\":
10}'"
INFO:root:Missing --publish_to_big_query option. Metrics will not be
published to BigQuery.
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Missing InfluxDB
options. Metrics will not be published to InfluxDB
[TestPubsubContext][LOG][TestPubsubContext] Registering Topic for
monitoring:
projects/apache-beam-testing/topics/perf-test-local-e96e58a7-b8d1-4fb3-bca3-860e12298b1b
[TestPubsubContext][LOG][TestPubsubContext] Registering Topic for
monitoring:
projects/apache-beam-testing/topics/perf-test-local-e96e58a7-b8d1-4fb3-bca3-860e12298b1b_matcher
[TestPubsubContext][LOG][TestPubsubContext] Registering Subscription for
monitoring:
projects/apache-beam-testing/subscriptions/perf-test-local-e96e58a7-b8d1-4fb3-bca3-860e12298b1b_read
[TestPubsubContext][LOG][TestPubsubContext] Registering Subscription for
monitoring:
projects/apache-beam-testing/subscriptions/perf-test-local-e96e58a7-b8d1-4fb3-bca3-860e12298b1b_read_matcher
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
