Thank you Robin, Your code provides a simple solution for the problem, although I believe that forcing host to be defined may be unnecessary because most use cases won't require it. After looking at the host hack used with boto I really think S3Hook should start using boto3, the fact that you can't pass a region string such as *us-east-1* using host instead doesn't seem standard and is not reusable for any other purpose (other than s3).
I think https://issues.apache.org/jira/browse/AIRFLOW-115 is the key point. I'll look into making a proposition on the matter. Thank you all for your help. On Thu, Mar 23, 2017 at 5:16 PM Gael Magnan <[email protected]> wrote: > Hi, > > we have had the same problem and basically did the same as Robin, the only > diff we have is we use the host from the connection instead of extra > params. > > I can't agree more with the statement that we need to really rethink the > S3Hook, and maybe make some basic S3/AWS operators. > > Gael > > Le jeu. 23 mars 2017 à 18:05, Robin Bartholdson < > [email protected]> a écrit : > > > I submitted our patch in the mean time, do test it and see if it works > for > > you :-) > > > > https://github.com/apache/incubator-airflow/pull/2181 > > https://issues.apache.org/jira/browse/AIRFLOW-1034 > > > > -Robin > > > > On 23 Mar 2017, at 17:43, Robin Bartholdson < > > [email protected]> wrote: > > > > There is some discussion in this JIRA, > > https://issues.apache.org/jira/browse/AIRFLOW-115, about moving to boto3 > > and restructuring along the lines you discuss. > > > > I totally agree that it would indeed be very useful! > > > > -Robin > > > > > > On 23 Mar 2017, at 16:53, Jorge Alpedrinha Ramos < > > [email protected]> wrote: > > > > Hi Sumit, > > > > I was actually referring to what Robin (thank you) pointed out which is > > reason enough for having region in S3Hook. > > > > There are even other situations having the region would be beneficial, > > that's why the AwsHook has that definition, although I would agree that > > their goals are different they typically refer to the same credentials, > > just different use cases. > > > > This makes me feel that it becomes redundant to have both an S3 > connection > > and an AWS connection with the same credentials just be able to access > > region. Wouldn't it be beneficial to implement S3Hook (and S3 connection) > > as a special case of AwsHook (and Aws connection) and be able to use both > > interchangeably ? > > > > I'm also aware that one uses boto and the other uses boto3, and it seems > > like the project is naturally slowly migrating towards the later. Maybe > > there is even some discussion regarding this that I'm not aware. > > > > If you think this is a desirable move, I may start to design this move, > > making sure nothing gets broken in the process. > > > > WDYT? > > > > On Thu, Mar 23, 2017 at 2:46 PM Robin Bartholdson < > > [email protected]> wrote: > > > > I _think_ that’s a different issue (do correct me if I’m wrong) > > > > For a number of regions (see the list on > > http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html), > > you have to specify an extra host-parameter in the boto-S3Connection as > > they only support signature V4 and the host-param is required to have > this > > working ( > > > > > http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError > > ) > > > > -Robin > > > > On 23 Mar 2017, at 15:10, Sumit Maheshwari <[email protected]> > wrote: > > > > Are we talking about this > > https://github.com/apache/incubator-airflow/pull/2176 ? > > > > > > > > > > On Thu, Mar 23, 2017 at 4:32 PM, Robin Bartholdson < > > [email protected]> wrote: > > > > Hi, > > > > We actually have a patch for this that allows you to specify the host > > parameter in the S3 config in the connections. > > > > Been thinking about submitting this for a while but have not had the > time, > > maybe now’s the time to do it :-) > > > > If you want to open a JIRA I can submit the patch we are using in > > production! > > > > -Robin > > > > > > On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos < > > [email protected]> wrote: > > > > Hi, > > > > I've been using S3Hook for a while, but I've run into some issues for not > > having a defined region. I can't really say why I can't access this > bucket > > without defining the region, but I know that there is no way to define > > region in S3Hook via the connection extra parameter unless a config_file > is > > used, which is not always an available method. > > > > I haven't found any ticket regarding this, is there any alternative ? > > > > Regards, > > Jorge > > > > > > > > > > > > > > > > >
