Re: Where to hide credentials when submitting a beam pipeline

[Parker Coleman]

I asked this same question a few weeks ago, we needed a way to have our
pipeline connect to an Azure instance (I think the ticket actually came
from that discussion). We decided to put the credentials in Google Cloud
Storage, It can be locked down with restrictive permissions and its easy to
access from Dataflow.  Its working fine for us until we can pass it in as a
hidden pipeline option.





On Fri, Jul 7, 2017 at 1:39 PM, Lukasz Cwik  wrote:

> I think your looking for this feature:
> https://issues.apache.org/jira/browse/BEAM-2492
>
> On Fri, Jul 7, 2017 at 11:40 AM, Randal Moore  wrote:
>
>> Maybe this is more of a question for DataFlow - but I'm submitting a
>> pipeline that needs to access a rest service running in a GKE kubernetes
>> instance.  I need to pass in creds.  I started with pipeline-options which
>> work but all options get exposed on the DataFlow web pages.
>>
>> Is there a way to pass in pipelien options that get hidden?  Or another
>> way to pass in creds (e.g., like the kubernetes creds map?
>>
>> rdm
>>
>
>

Re: Where to hide credentials when submitting a beam pipeline

[Lukasz Cwik]

I think your looking for this feature:
https://issues.apache.org/jira/browse/BEAM-2492

On Fri, Jul 7, 2017 at 11:40 AM, Randal Moore  wrote:

> Maybe this is more of a question for DataFlow - but I'm submitting a
> pipeline that needs to access a rest service running in a GKE kubernetes
> instance.  I need to pass in creds.  I started with pipeline-options which
> work but all options get exposed on the DataFlow web pages.
>
> Is there a way to pass in pipelien options that get hidden?  Or another
> way to pass in creds (e.g., like the kubernetes creds map?
>
> rdm
>

Where to hide credentials when submitting a beam pipeline

[Randal Moore]

Maybe this is more of a question for DataFlow - but I'm submitting a
pipeline that needs to access a rest service running in a GKE kubernetes
instance.  I need to pass in creds.  I started with pipeline-options which
work but all options get exposed on the DataFlow web pages.

Is there a way to pass in pipelien options that get hidden?  Or another way
to pass in creds (e.g., like the kubernetes creds map?

rdm