dondaum opened a new issue #17604:
URL: https://github.com/apache/airflow/issues/17604
Hi core Airflow dev teams,
I would like to say a big 'thank you' first for this great plattform and for
all of your efforts to constantly improve and mantain it. It is just a joy to
work with.. Our company has recently switched to the Airflow 2 and we love the
overall improvements, such as scheduler speed, UI but also all your efforts in
knowledge sharing and community building.
**Description**
The KubernetesPodOperator does not have an option to either disable showing
environment variables information or masking variables in the task logs if a
failure or an error happen on a created pod. We use the KubernetesPodOperator
and k8s.V1EnvVar to pass sensitive information to a pod, such as target
connection, user name or passwords. If an error or failure happens the task
logs show all pod details including the environment variables. An option, such
as adding a new param to the KubernetesPodOperator that allows to either remove
env variables completely or mask env variables would prevent stop sharing those
senstive informations in the logs.
**Use case / motivation**
When running the KubernetesPodOperator with environment variables it would
be great to have the option to remove environmental variables or to mask
environment variables information in the task logs if there are failures or
errors.
**Are you willing to submit a PR?**
Yes I would.
Currently if the final_state not equals State.SUCCESS, we are handling this
by checking if we have env variables set and if so we start to masking them
with **** before raising the exception that then also logs the complete
remote_pod object. I guess this is rather a quick fix that we have applied. So
it would be good to think about other use cases or pod objects that could
benefit as well.
The relevant section is
`
launcher = self.create_pod_launcher()
if len(pod_list.items) == 1:
try_numbers_match = self._try_numbers_match(context,
pod_list.items[0])
final_state, remote_pod, result = self.handle_pod_overlap(
labels, try_numbers_match, launcher, pod_list.items[0]
)
else:
self.log.info("creating pod with labels %s and launcher %s",
labels, launcher)
final_state, remote_pod, result =
self.create_new_pod_for_operator(labels, launcher)
if final_state != State.SUCCESS:
raise AirflowException(f'Pod {self.pod.metadata.name}
returned a failure: {remote_pod}')
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
