Thanks for the response Paul. Crypto package would be my last resort. But it would still not solve (2) where I am looking to create the connection automatically.
Awaiting further answers.. Thanks, Nadeem On Tue, Jul 19, 2016 at 9:03 AM, Paul Minton <[email protected]> wrote: > Nadeem, this doesn't directly answer either 1) or 2), but have you > considered using the option "is_exrra_encrypted"? This encrypts the extra > json as it would for the rest of the credentials on the connection object > (ie using a fernet key and the encryption package) > > On Mon, Jul 18, 2016 at 10:00 PM, Nadeem Ahmed Nazeer <[email protected] > > > wrote: > > > Hi, > > > > Appreciate if someone could please provide assistance on this. > > > > Thanks, > > Nadeem > > > > On Fri, Jul 15, 2016 at 4:15 PM, Nadeem Ahmed Nazeer < > [email protected]> > > wrote: > > > > > Hello, > > > > > > We are using the S3Hook in several of our airflow TI's to read and > write > > > data from S3. > > > > > > We are creating a s3 connection from the UI where we choose the below > > > options. > > > > > > Conn Id - s3 > > > Conn Type - S3 > > > Extra - {"aws_access_key_id":"key", "aws_secret_access_key": "key"} > > > > > > In pipeline code we use this connection as below, > > > > > > s3 = S3Hook(s3_conn_id='s3') > > > > > > We are looking into other options to define this connection as it is a > > > security issue to have the keys open like this. We tried defining the > > > connection id and connection type alone in the UI without the keys. In > > this > > > case, the tasks that read from S3 succeed but the ones that delete or > > > create files/objects fail with '403 Forbidden' error from S3. Did some > > > digging in the S3_Hook code and found that if the keys are not in the > > Extra > > > parameter then it would use the boto config but that doesn't seem to > work > > > in my case for reasons I am unable to find. > > > > > > All our other python scripts interact with S3 using the boto config on > > the > > > system without any problems. > > > > > > 1) > > > Need help on why the s3 hook isn't using the boto config. Am I missing > to > > > pass some other parameters to this connection? > > > > > > 2) > > > How to define the s3 connection as environmental variable? We are > > > installing airflow via Chef and would want to have an environmental > > > variable like AIRFLOW_CONN_S3 created for this connection so that we > > don't > > > have to manually do it in the UI every time we run the setup. > > > > > > Documentation says, it has the connection has to be in a URI format. On > > > S3, I could access different buckets with the same connection. But > since > > it > > > has to be in URI format, does that mean i create one connection per > > bucket > > > and use it? Did not find any examples of this anywhere hence asking. > > > > > > Thanks, > > > Nadeem > > > > > >
