Re: Recommended way of piping credentials / sensitive information on templated operators

[Shah Altaf]

Would like to hear a little more from others, or any recommendations around
this.  We've got other systems like SQL connections, SSH with user:pass,
API tokens.

Is there any recommended way to hide these from the output logs?



On Mon, Feb 26, 2018 at 10:41 PM Hbw 
wrote:

> Aws profiles on the workers - the creds are on the machines, but not
> exposed.  Boto/cli takes these profile names instead of access key/secret
> for just this kind of use case.
>
> Sent from a device with less than stellar autocorrect
>
> > On Feb 26, 2018, at 1:22 PM, jeeyoung kim  wrote:
> >
> > Hi everyone,
> >
> > I’m wondering how people work around accidentally writing credentials on
> > bash operator template page / logs.
> >
> > For example, I may have PostgreSQL operator to copy data into Redshift.
> >
> > COPY TABLE_NAME from 's3://.../something.manifest.json'
> > access_key_id '{{ params.AWS_ACCESS_KEY }}'
> > secret_access_key '{{ params.AWS_SECRET_KEY }}'
> >
> > Or a command that exports from mongo
> >
> > mongoexport \
> >  --assertExists \
> >  -h {{ connection.host }} \
> >  {% if connection.login %} -u {{ connection.login }} {% endif %}\
> >  {% if connection.get_password() %} -p {{ connection.get_password()
> > }} {% endif %}\
> >  -d {{ connection.schema }}
> >  ...
> >
> > However, when this operator is executed (or when the template is rendered
> > on the UI), the credentials are written to the log files / clearly
> visible
> > on the UI, which is problematic.
> >
> > There are many other cases where this can happen, and I’m wondering what
> is
> > a solution for it.
> >
> > What would be ideal is:
> >
> >   - Prevent credentials from accidentally being shown in “show rendered
> >   template” screen.
> >   - Prevent credentials from being written to the logs.
> >
> > Thanks.
> >
> > -Jeeyoung Kim
> > ​
>

Re: Recommended way of piping credentials / sensitive information on templated operators

[Hbw]

Aws profiles on the workers - the creds are on the machines, but not exposed.  
Boto/cli takes these profile names instead of access key/secret for just this 
kind of use case.

Sent from a device with less than stellar autocorrect

> On Feb 26, 2018, at 1:22 PM, jeeyoung kim  wrote:
> 
> Hi everyone,
> 
> I’m wondering how people work around accidentally writing credentials on
> bash operator template page / logs.
> 
> For example, I may have PostgreSQL operator to copy data into Redshift.
> 
> COPY TABLE_NAME from 's3://.../something.manifest.json'
> access_key_id '{{ params.AWS_ACCESS_KEY }}'
> secret_access_key '{{ params.AWS_SECRET_KEY }}'
> 
> Or a command that exports from mongo
> 
> mongoexport \
>  --assertExists \
>  -h {{ connection.host }} \
>  {% if connection.login %} -u {{ connection.login }} {% endif %}\
>  {% if connection.get_password() %} -p {{ connection.get_password()
> }} {% endif %}\
>  -d {{ connection.schema }}
>  ...
> 
> However, when this operator is executed (or when the template is rendered
> on the UI), the credentials are written to the log files / clearly visible
> on the UI, which is problematic.
> 
> There are many other cases where this can happen, and I’m wondering what is
> a solution for it.
> 
> What would be ideal is:
> 
>   - Prevent credentials from accidentally being shown in “show rendered
>   template” screen.
>   - Prevent credentials from being written to the logs.
> 
> Thanks.
> 
> -Jeeyoung Kim
> ​

Re: Recommended way of piping credentials / sensitive information on templated operators

[Shoumitra Srivastava]

Hey jeeyoung,

Specifically for Redshift, we worked around that problem by using iam_roles
instead of credentials.

-Shoumitra

On Mon, Feb 26, 2018 at 11:22 AM, jeeyoung kim  wrote:

> Hi everyone,
>
> I’m wondering how people work around accidentally writing credentials on
> bash operator template page / logs.
>
> For example, I may have PostgreSQL operator to copy data into Redshift.
>
> COPY TABLE_NAME from 's3://.../something.manifest.json'
> access_key_id '{{ params.AWS_ACCESS_KEY }}'
> secret_access_key '{{ params.AWS_SECRET_KEY }}'
>
> Or a command that exports from mongo
>
> mongoexport \
>   --assertExists \
>   -h {{ connection.host }} \
>   {% if connection.login %} -u {{ connection.login }} {% endif %}\
>   {% if connection.get_password() %} -p {{ connection.get_password()
> }} {% endif %}\
>   -d {{ connection.schema }}
>   ...
>
> However, when this operator is executed (or when the template is rendered
> on the UI), the credentials are written to the log files / clearly visible
> on the UI, which is problematic.
>
> There are many other cases where this can happen, and I’m wondering what is
> a solution for it.
>
> What would be ideal is:
>
>- Prevent credentials from accidentally being shown in “show rendered
>template” screen.
>- Prevent credentials from being written to the logs.
>
> Thanks.
>
> -Jeeyoung Kim
> ​
>