Abacn commented on code in PR #39615:
URL: https://github.com/apache/beam/pull/39615#discussion_r3760501584
##########
sdks/python/apache_beam/io/gcp/bigquery_tools.py:
##########
@@ -125,6 +125,13 @@
"GEOGRAPHY": str,
}
+# Duplicated logic with io/gcp/bigquery_change_history.py
+# Default table expiration for auto-created temp datasets: 24 hours in ms.
+# Tables created in the dataset auto-expire after this duration if not
+# explicitly deleted, acting as a safety net for orphaned temp tables
+# (e.g. pipeline crash before cleanup runs).
+_DEFAULT_TABLE_EXPIRATION_MS = 24 * 60 * 60 * 1000
Review Comment:
Isn't there a genuine data loss concern if there is a long running job that
some temp tables are materialized, and expired before all of them finalized to
destination table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
