There is some discussion in this JIRA, https://issues.apache.org/jira/browse/AIRFLOW-115 <https://issues.apache.org/jira/browse/AIRFLOW-115>, about moving to boto3 and restructuring along the lines you discuss.
I totally agree that it would indeed be very useful! -Robin > On 23 Mar 2017, at 16:53, Jorge Alpedrinha Ramos <[email protected]> > wrote: > > Hi Sumit, > > I was actually referring to what Robin (thank you) pointed out which is > reason enough for having region in S3Hook. > > There are even other situations having the region would be beneficial, > that's why the AwsHook has that definition, although I would agree that > their goals are different they typically refer to the same credentials, > just different use cases. > > This makes me feel that it becomes redundant to have both an S3 connection > and an AWS connection with the same credentials just be able to access > region. Wouldn't it be beneficial to implement S3Hook (and S3 connection) > as a special case of AwsHook (and Aws connection) and be able to use both > interchangeably ? > > I'm also aware that one uses boto and the other uses boto3, and it seems > like the project is naturally slowly migrating towards the later. Maybe > there is even some discussion regarding this that I'm not aware. > > If you think this is a desirable move, I may start to design this move, > making sure nothing gets broken in the process. > > WDYT? > > On Thu, Mar 23, 2017 at 2:46 PM Robin Bartholdson < > [email protected]> wrote: > >> I _think_ that’s a different issue (do correct me if I’m wrong) >> >> For a number of regions (see the list on >> http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html), >> you have to specify an extra host-parameter in the boto-S3Connection as >> they only support signature V4 and the host-param is required to have this >> working ( >> http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError >> ) >> >> -Robin >> >> On 23 Mar 2017, at 15:10, Sumit Maheshwari <[email protected]> wrote: >> >> Are we talking about this >> https://github.com/apache/incubator-airflow/pull/2176 ? >> >> >> >> >> On Thu, Mar 23, 2017 at 4:32 PM, Robin Bartholdson < >> [email protected]> wrote: >> >> Hi, >> >> We actually have a patch for this that allows you to specify the host >> parameter in the S3 config in the connections. >> >> Been thinking about submitting this for a while but have not had the time, >> maybe now’s the time to do it :-) >> >> If you want to open a JIRA I can submit the patch we are using in >> production! >> >> -Robin >> >> >> On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos < >> [email protected]> wrote: >> >> Hi, >> >> I've been using S3Hook for a while, but I've run into some issues for not >> having a defined region. I can't really say why I can't access this bucket >> without defining the region, but I know that there is no way to define >> region in S3Hook via the connection extra parameter unless a config_file is >> used, which is not always an available method. >> >> I haven't found any ticket regarding this, is there any alternative ? >> >> Regards, >> Jorge >> >> >> >> >>
smime.p7s
Description: S/MIME cryptographic signature
