[
https://issues.apache.org/jira/browse/AIRFLOW-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17003995#comment-17003995
]
ASF GitHub Bot commented on AIRFLOW-6350:
-----------------------------------------
potiuk commented on pull request #6917: [AIRFLOW-6350] security - spark submit
operator logging+exceptions sh…
URL: https://github.com/apache/airflow/pull/6917
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> security - spark submit operator logging+exceptions should mask passwords
> --------------------------------------------------------------------------
>
> Key: AIRFLOW-6350
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6350
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks, operators
> Affects Versions: 1.10.3
> Reporter: t oo
> Assignee: t oo
> Priority: Major
>
> contrib/hooks/spark_submit_hook.py
> Mask passwords in spark submit cmd AND error stacktrace
>
> *add*
> def _mask_cmd(self, connection_cmd):
> # Mask any password related fields in application args with key value pair
> where key contains password (case insensitive), e.g. HivePassword='abc'
> connection_cmd_masked =
> re.sub(r"(\S*?(?:secret|password)\S*?\s*=\s*')[^']*(?=')", r'\1******', '
> '.join(connection_cmd), flags=re.I)
> return connection_cmd_masked
>
> *BEFORE*
> self.log.info("Spark-Submit cmd: %s", connection_cmd)
>
> *AFTER*
> self.log.info("Spark-Submit cmd: %s", self._mask_cmd(connection_cmd))
>
> *BEFORE*
> if returncode or (self._is_kubernetes and self._spark_exit_code != 0):
> raise AirflowException(
> "Cannot execute: {}. Error code is: {}.".format(
> spark_submit_cmd, returncode
> )
> )
>
> *AFTER*
> if returncode or (self._is_kubernetes and self._spark_exit_code != 0):
> raise AirflowException(
> "Cannot execute: {}. Error code is: {}.".format(
> self._mask_cmd(spark_submit_cmd), returncode
> )
> )
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
