Re: Cloud Provider grouping into Plugins

2016-10-16 Thread Arthur Wiedmer
Sounds reasonable to me. +1 We have been meaning to refactor the AWS operators anyway, we could take advantage of this to reorganize the repo a little bit. That makes me think that we have not yet decided of a flow to enable work towards say 2.0 vs 1.8. We might want to start thinking about this.

Re: Cloud Provider grouping into Plugins

2016-10-14 Thread Chris Riccomini
> So I vote for this, but it will have to be done gently to avoid breaking the > existing GCP ones. Same. On Fri, Oct 14, 2016 at 8:51 AM, Jeremiah Lowin wrote: > One reason I do like the idea is that especially in contrib, Operators are > essentially self-documenting and the first clue is just

Re: Cloud Provider grouping into Plugins

2016-10-14 Thread Jeremiah Lowin
One reason I do like the idea is that especially in contrib, Operators are essentially self-documenting and the first clue is just the file name ('my_gcp_operators.py'). Since we no longer greedily import anything, you have to know exactly what file to import to get the functionality you want. Grou

Re: Cloud Provider grouping into Plugins

2016-10-14 Thread Alex Van Boxel
Talking about AWS, it would only make sense if other people would step up to do it for AWS, and even Azure (or don't we have Azure operators?). On Fri, Oct 14, 2016 at 5:25 PM Chris Riccomini wrote: > What do others think? I know Sid is a big AWS user. > > On Fri, Oct 14, 2016 at 8:24 AM, Chris

Re: Cloud Provider grouping into Plugins

2016-10-14 Thread Chris Riccomini
Ya, if we go the deprecation route, and let them float around for a release or two, I'm OK with that (or until we bump major to 2.0). Other than that, it sounds like a good opportunity to clean things up. :) I do notice a lot of AWS/GCP code (e.g. the S3 Redshift operator). On Fri, Oct 14, 2016 a

Re: Cloud Provider grouping into Plugins

2016-10-14 Thread Chris Riccomini
What do others think? I know Sid is a big AWS user. On Fri, Oct 14, 2016 at 8:24 AM, Chris Riccomini wrote: > Ya, if we go the deprecation route, and let them float around for a > release or two, I'm OK with that (or until we bump major to 2.0). > > Other than that, it sounds like a good opportun

Re: Cloud Provider grouping into Plugins

2016-10-14 Thread Alex Van Boxel
Well, I wouldn't touch the on that exist (maybe we could mark them deprecated, but that's all). But I would move (copy) them together and make them consistent (example, let them all use the same default connection_id, ...). For a new user it's quite confusing I think due to different reasons (style

Re: Cloud Provider grouping into Plugins

2016-10-14 Thread Chris Riccomini
Hmm. What advantages would this provide? I'm a little nervous about breaking compatibility. We have a bunch of DAGs which import all kinds of GCP hooks and operators. Wouldn't want those to move. On Fri, Oct 14, 2016 at 7:54 AM, Alex Van Boxel wrote: > Hi all, > > I'm starting to write some very