eladkal commented on code in PR #63738:
URL: https://github.com/apache/airflow/pull/63738#discussion_r2976342798
##########
providers/apache/beam/src/airflow/providers/apache/beam/triggers/beam.py:
##########
@@ -23,6 +23,37 @@
from airflow.providers.apache.beam.hooks.beam import BeamAsyncHook
from airflow.triggers.base import BaseTrigger, TriggerEvent
+from airflow.utils.log.secrets_masker import mask_secret
+
+# Keys in pipeline variables that may contain sensitive values and should
+# be masked in log output. This list covers the most common Beam pipeline
+# options that carry credentials.
+_SENSITIVE_VARIABLE_KEYS = frozenset(
+ {
+ "password",
+ "secret",
+ "token",
+ "api_key",
+ "apikey",
+ "api_token",
+ "access_token",
+ "refresh_token",
+ "credentials",
+ "private_key",
+ "service_account_key",
+ }
+)
+
+
+def _register_sensitive_variables(variables: dict) -> None:
+ """Register values of sensitive keys with Airflow's SecretsMasker.
+
+ This ensures that any log line that contains these values will have them
+ automatically replaced with ``***``.
+ """
+ for key, value in variables.items():
+ if value and any(sensitive in key.lower() for sensitive in
_SENSITIVE_VARIABLE_KEYS):
+ mask_secret(value)
Review Comment:
Before we make the fix beam specific we need to understand if this happens
only in beam - I have doubts that it does.
Can you offer confirmation for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
