I think it's very important to discuss this and choose the right approach and follow up quickly. This is very important to introduce some sanity in the structure of 2.0.0. I propose that anyone with a strong opinion voices his or her concerns here and then maybe a week from now we can get to a consensus and vote on the direction we take?
My preferences (but I can be convinced to change some of my choices) are: Case 1: (contrib vs not) *B* (move all that are "well" tested): I think having "incubator" kind of approach for not super tested/documented but still usable code is useful. Though we would have to establish some objective criteria and describe a "growing from incubation" process. Case 2 airflow.operators.foo_operator.FooOperator could become *A:* airflow.operators.foo.FooOperator Case 3: airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator could become *C*: airflow.integration.gcp.operators.bigtable.BigTableOperator Case 4: (separate packages/namesapces) *B:* no namespaces - i think we are very far from modularisation to happen (and how) and it might get never implemented so there are no real benefits I see now. Case 5: "Operator" suffix on class names *B*: I prefer to keep B as this will be much easier when you search for class using your IDE. Case 6: "Isolated" renames *Rename*: We should align names of those operators. Case 7: Zope deprecation or not *B*: I prefer to use Kamil's in-house hack than zope, but I feel we could improve it slightly to remove a bit of boilerplate - it looks like the warning.warn() message should be very similar in all deprecation cases and possibly we can define it in one place (providing that DepracationWarning will continue to work well with IDEs) J, On Tue, Jun 4, 2019 at 2:11 PM Kamil Breguła <[email protected]> wrote: > I created AIP-21 > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths > This is the same document as above, I just deleted my opinions (summary > sections) and work plan. > > > On Fri, Apr 19, 2019 at 2:18 PM Jarek Potiuk <[email protected]> > wrote: > > > Thanks Kamil! Excellent analysis! > > > > My preferences: > > > > Case 1: (contrib vs not) > > B (move all that are "well" tested): I think having "incubator" kind of > > approach for not super tested/documented but still usable code is useful. > > Though we would have to establish some objective criteria and describe a > > "growing from incubation" process. > > > > Case 2 airflow.operators.foo_operator.FooOperator could become > > A: airflow.operators.foo.FooOperator > > > > Case 3: airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator > > could become > > C: airflow.integration.gcp.operators.bigtable.BigTableOperator > > > > Case 4: (separate packages/namesapces) > > B: no namespaces - i think we are very far from modularisation to happen > > (and how) and it might get never implemented so there are no real > benefits > > I see now. > > > > Case 5: "Operator" suffix on class names > > B: I prefer to keep B as this will be much easier when you search for > class > > using your IDE. > > > > Case 7: Zope deprecation or not > > B: I prefer to use Kamil's in-house hack than zope, but I feel we could > > improve it slightly to remove a bit of boilerplate - it looks like the > > warning.warn() message should be very similar in all deprecation cases > and > > possibly we can define it in one place (providing that DepracationWarning > > will continue to work well with IDEs) > > > > > > > > > > On Fri, Apr 19, 2019 at 12:36 AM Kamil Breguła < > [email protected]> > > wrote: > > > > > Inline comment > > > > > > On Fri, Apr 19, 2019 at 12:23 AM Arthur Wiedmer < > > [email protected]> > > > wrote: > > > > > > > > > > > > > > > Case #5 Are we talking about the class name or the file name? > > The > > > > > class > > > > > > > > name is fine to me, but we could remove _operator from the > file > > > > name. > > > > > > > > > > > > > > > > > > > > > > Case #2 describes the change of file names > > > > > > > Case #5 describes the change of class names > > > > > > > > > > > > > > I added examples to two cases to better describe the changes. > > > > > > > > > > > > > > > > > > > Thank you. I guess I see it for file names, but I am wondering > > about > > > > the > > > > > > operators and name collisions. > > > > > > Say I need both the HiveOperator and I inline a custom operator > for > > > > > which I > > > > > > need a hive hook. > > > > > > # Would you recommmend the following? Or does case #5 only apply > to > > > > > > Operators? > > > > > > from airflow.operators.hive import Hive as HiveOperator > > > > > > from airfow.hooks.hive import Hive as HiveHook > > > > > > > > > > > > I was just thinking it might be nice to be able to import what I > > need > > > > > > without renaming and not worry too much about names shadowing > > others. > > > > > > Especially if I am new-ish to Apache Airflow. > > > > > > > > > > > > > > > > > I think that the operator should describe the behavior(verb), hook > > > should > > > > > describe a service(noun). These are other parts of speech. > > > > > In my opinion, HiveOperator should be named HiveExecuteQuery. Hook > > will > > > > be > > > > > called Hive. Then there will be no name conflicts. > > > > > > > > > > > > > But we will still provide the backwards compatibility for a while > with > > > > aliases to the old names, correct? > > > > > > > > > > Yes. It is possible to provide backward compatibility also in this > case. > > > > > > > > > -- > > > > > > Kamil Breguła > > > Polidea <https://www.polidea.com/> | Software Engineer > > > > > > M: +48 505 458 451 <+48505458451> > > > E: [email protected] > > > [image: Polidea] <https://www.polidea.com/> > > > > > > We create human & business stories through technology. > > > Check out our projects! <https://www.polidea.com/our-work> > > > [image: Github] <https://github.com/Polidea> [image: Facebook] > > > <https://www.facebook.com/Polidea.Software> [image: Twitter] > > > <https://twitter.com/polidea> [image: Linkedin] > > > <https://www.linkedin.com/company/polidea> [image: Instagram] > > > <https://instagram.com/polidea> [image: Behance] > > > <https://www.behance.net/polidea> > > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > E: [email protected] > > > > > -- > > Kamil Breguła > Polidea <https://www.polidea.com/> | Software Engineer > > M: +48 505 458 451 <+48505458451> > E: [email protected] > [image: Polidea] <https://www.polidea.com/> > > We create human & business stories through technology. > Check out our projects! <https://www.polidea.com/our-work> > [image: Github] <https://github.com/Polidea> [image: Facebook] > <https://www.facebook.com/Polidea.Software> [image: Twitter] > <https://twitter.com/polidea> [image: Linkedin] > <https://www.linkedin.com/company/polidea> [image: Instagram] > <https://instagram.com/polidea> [image: Behance] > <https://www.behance.net/polidea> > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> E: [email protected]
