dondaum commented on issue #17604:
URL: https://github.com/apache/airflow/issues/17604#issuecomment-899081566
Great! Glad that I could help finding this issue.
So I had a look on your PR and also on the original PR that merged the
'masking feature'. My example above should actually also work without having to
change anything in our customer operator - I mean beside your changes on the
secret masking in the exceptions.
Reading through the docs whenever you either use a connection or variable
and if the variable key names appear in the configuration varibles
`AIRFLOW__CORE__SENSITIVE_VAR_CONN_NAMES` it should be handled by default.
In our setup and in the example above I get the env variables from an
Airflow variable thus I am wondering why that does not work - beside of course
the expection issue that your took care of.
So I tested a bit and I found another strange behavior that I could not
explain. It seems that variable names in the configuration that have only
capital letters somehow are not handled correctly and therefore their values
are not masked.
First test (masking does not work):
```Python
# Context
# Airflow variable name: testvar
# Airflow variable payload:
# {
# "DBT_WAREHOUSE": "secret"
# }
# Configuration
AIRFLOW__CORE__SENSITIVE_VAR_CONN_NAMES: "DBT_WAREHOUSE"
# Custom operator
class KubernetesPodOperator_(KubernetesPodOperator):
def __init__(
self,
*args,
**kwargs,
):
super().__init__(*args, **kwargs)
def _get_dummy_secret_var(self) -> List[k8s.V1EnvVar]:
try:
sec = Variable.get(
"testvar", deserialize_json=True
)
except KeyError:
self.log.warning(
"You have to add the variable in Airflow"
)
sec = None
return sec
def execute(self, context):
dummy_var = self._get_dummy_secret_var()
self.log.info(dummy_var)
self.log.info(
f"Custom KubernetesPodOperator runs K8S Task"
)
return super().execute(context)
...
# Output
[2021-08-15 16:00:30,237] {{dbt_k8s_operator.py:67}} INFO -
{'DBT_WAREHOUSE': 'secret'}
[2021-08-15 16:00:30,237] {{dbt_k8s_operator.py:69}} INFO - Custom
KubernetesPodOperator runs K8S Task
```
Second test (masking is working):
```Python
# Context
# Airflow variable name: testvar
# Airflow variable payload:
# {
# "DBT_WAREHOUSE": "secret"
# }
# Configuration
AIRFLOW__CORE__SENSITIVE_VAR_CONN_NAMES: "dbt_warehouse"
# Custom operator
class KubernetesPodOperator_(KubernetesPodOperator):
def __init__(
self,
*args,
**kwargs,
):
super().__init__(*args, **kwargs)
def _get_dummy_secret_var(self) -> List[k8s.V1EnvVar]:
try:
sec = Variable.get(
"testvar", deserialize_json=True
)
except KeyError:
self.log.warning(
"You have to add the variable in Airflow"
)
sec = None
return sec
def execute(self, context):
dummy_var = self._get_dummy_secret_var()
self.log.info(dummy_var)
self.log.info(
f"Custom KubernetesPodOperator runs K8S Task"
)
return super().execute(context)
...
# Output
[2021-08-15 16:04:18,864] {{dbt_k8s_operator.py:67}} INFO -
{'DBT_WAREHOUSE': '***'}
[2021-08-15 16:04:18,864] {{dbt_k8s_operator.py:69}} INFO - Custom
KubernetesPodOperator runs K8S Task
```
Thus the masing only works if you change the variable name in the Airflow
config from 'DBT_WAREHOUSE' to 'dbt_warehouse'. As our variables key names all
written in capital letters our previous masking attemps did not work.
Any clue why that happens?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
