Hi Timo, Thanks for the effort and writing up this document. I like the idea to make flink-table scala free, so +1 for the proposal!
It's good to make Java the first-class citizen. For a long time, we have neglected java so that many features in Table are missed in Java Test cases, such as this one[1] I found recently. And I think we may also need to migrate our test cases, i.e, add java tests. This definitely is a big change and will break API compatible. In order to bring a smaller impact on users, I think we should go fast when we migrate APIs targeted to users. It's better to introduce the user sensitive changes within a release. However, it may be not that easy. I can help to contribute. Separation of interface and implementation is a good idea. This may introduce a minimum of dependencies or even no dependencies. I saw your reply in the google doc. Java8 has already supported static method for interfaces, I think we can make use of it? Best, Hequn [1] https://issues.apache.org/jira/browse/FLINK-11001 On Fri, Nov 23, 2018 at 5:36 PM Timo Walther <twal...@apache.org> wrote: > Hi everyone, > > thanks for the great feedback so far. I updated the document with the > input I got so far > > @Fabian: I moved the porting of flink-table-runtime classes up in the list. > > @Xiaowei: Could you elaborate what "interface only" means to you? Do you > mean a module containing pure Java `interface`s? Or is the validation > logic also part of the API module? Are 50+ expression classes part of > the API interface or already too implementation-specific? > > @Xuefu: I extended the document by almost a page to clarify when we > should develop in Scala and when in Java. As Piotr said, every new Scala > line is instant technical debt. > > Thanks, > Timo > > > Am 23.11.18 um 10:29 schrieb Piotr Nowojski: > > Hi Timo, > > > > Thanks for writing this down +1 from my side :) > > > >> I'm wondering that whether we can have rule in the interim when Java > and Scala coexist that dependency can only be one-way. I found that in the > current code base there are cases where a Scala class extends Java and vise > versa. This is quite painful. I'm thinking if we could say that extension > can only be from Java to Scala, which will help the situation. However, I'm > not sure if this is practical. > > Xuefu: I’m also not sure what’s the best approach here, probably we will > have to work it out as we go. One thing to consider is that from now on, > every single new code line written in Scala anywhere in Flink-table (except > of Flink-table-api-scala) is an instant technological debt. From this > perspective I would be in favour of tolerating quite big inchonvieneces > just to avoid any new Scala code. > > > > Piotrek > > > >> On 23 Nov 2018, at 03:25, Zhang, Xuefu <xuef...@alibaba-inc.com> wrote: > >> > >> Hi Timo, > >> > >> Thanks for the effort and the Google writeup. During our external > catalog rework, we found much confusion between Java and Scala, and this > Scala-free roadmap should greatly mitigate that. > >> > >> I'm wondering that whether we can have rule in the interim when Java > and Scala coexist that dependency can only be one-way. I found that in the > current code base there are cases where a Scala class extends Java and vise > versa. This is quite painful. I'm thinking if we could say that extension > can only be from Java to Scala, which will help the situation. However, I'm > not sure if this is practical. > >> > >> Thanks, > >> Xuefu > >> > >> > >> ------------------------------------------------------------------ > >> Sender:jincheng sun <sunjincheng...@gmail.com> > >> Sent at:2018 Nov 23 (Fri) 09:49 > >> Recipient:dev <dev@flink.apache.org> > >> Subject:Re: [DISCUSS] Long-term goal of making flink-table Scala-free > >> > >> Hi Timo, > >> Thanks for initiating this great discussion. > >> > >> Currently when using SQL/TableAPI should include many dependence. In > >> particular, it is not necessary to introduce the specific implementation > >> dependencies which users do not care about. So I am glad to see your > >> proposal, and hope when we consider splitting the API interface into a > >> separate module, so that the user can introduce minimum of dependencies. > >> > >> So, +1 to [separation of interface and implementation; e.g. `Table` & > >> `TableImpl`] which you mentioned in the google doc. > >> Best, > >> Jincheng > >> > >> Xiaowei Jiang <xiaow...@gmail.com> 于2018年11月22日周四 下午10:50写道： > >> > >>> Hi Timo, thanks for driving this! I think that this is a nice thing to > do. > >>> While we are doing this, can we also keep in mind that we want to > >>> eventually have a TableAPI interface only module which users can take > >>> dependency on, but without including any implementation details? > >>> > >>> Xiaowei > >>> > >>> On Thu, Nov 22, 2018 at 6:37 PM Fabian Hueske <fhue...@gmail.com> > wrote: > >>> > >>>> Hi Timo, > >>>> > >>>> Thanks for writing up this document. > >>>> I like the new structure and agree to prioritize the porting of the > >>>> flink-table-common classes. > >>>> Since flink-table-runtime is (or should be) independent of the API and > >>>> planner modules, we could start porting these classes once the code is > >>>> split into the new module structure. > >>>> The benefits of a Scala-free flink-table-runtime would be a Scala-free > >>>> execution Jar. > >>>> > >>>> Best, Fabian > >>>> > >>>> > >>>> Am Do., 22. Nov. 2018 um 10:54 Uhr schrieb Timo Walther < > >>>> twal...@apache.org > >>>>> : > >>>>> Hi everyone, > >>>>> > >>>>> I would like to continue this discussion thread and convert the > outcome > >>>>> into a FLIP such that users and contributors know what to expect in > the > >>>>> upcoming releases. > >>>>> > >>>>> I created a design document [1] that clarifies our motivation why we > >>>>> want to do this, how a Maven module structure could look like, and a > >>>>> suggestion for a migration plan. > >>>>> > >>>>> It would be great to start with the efforts for the 1.8 release such > >>>>> that new features can be developed in Java and major refactorings > such > >>>>> as improvements to the connectors and external catalog support are > not > >>>>> blocked. > >>>>> > >>>>> Please let me know what you think. > >>>>> > >>>>> Regards, > >>>>> Timo > >>>>> > >>>>> [1] > >>>>> > >>>>> > >>> > https://docs.google.com/document/d/1PPo6goW7tOwxmpFuvLSjFnx7BF8IVz0w3dcmPPyqvoY/edit?usp=sharing > >>>>> > >>>>> Am 02.07.18 um 17:08 schrieb Fabian Hueske: > >>>>>> Hi Piotr, > >>>>>> > >>>>>> thanks for bumping this thread and thanks for Xingcan for the > >>> comments. > >>>>>> I think the first step would be to separate the flink-table module > >>> into > >>>>>> multiple sub modules. These could be: > >>>>>> > >>>>>> - flink-table-api: All API facing classes. Can be later divided > >>> further > >>>>>> into Java/Scala Table API/SQL > >>>>>> - flink-table-planning: involves all planning (basically everything > >>> we > >>>> do > >>>>>> with Calcite) > >>>>>> - flink-table-runtime: the runtime code > >>>>>> > >>>>>> IMO, a realistic mid-term goal is to have the runtime module and > >>>> certain > >>>>>> parts of the planning module ported to Java. > >>>>>> The api module will be much harder to port because of several > >>>>> dependencies > >>>>>> to Scala core classes (the parser framework, tree iterations, etc.). > >>>> I'm > >>>>>> not saying we should not port this to Java, but it is not clear to > me > >>>>> (yet) > >>>>>> how to do it. > >>>>>> > >>>>>> I think flink-table-runtime should not be too hard to port. The code > >>>> does > >>>>>> not make use of many Scala features, i.e., it's writing very > >>> Java-like. > >>>>>> Also, there are not many dependencies and operators can be > >>> individually > >>>>>> ported step-by-step. > >>>>>> For flink-table-planning, we can have certain packages that we port > >>> to > >>>>> Java > >>>>>> like planning rules or plan nodes. The related classes mostly extend > >>>>>> Calcite's Java interfaces/classes and would be natural choices for > >>>> being > >>>>>> ported. The code generation classes will require more effort to > port. > >>>>> There > >>>>>> are also some dependencies in planning on the api module that we > >>> would > >>>>> need > >>>>>> to resolve somehow. > >>>>>> > >>>>>> For SQL most work when adding new features is done in the planning > >>> and > >>>>>> runtime modules. So, this separation should already reduce > >>>> "technological > >>>>>> dept" quite a lot. > >>>>>> The Table API depends much more on Scala than SQL. > >>>>>> > >>>>>> Cheers, Fabian > >>>>>> > >>>>>> > >>>>>> > >>>>>> 2018-07-02 16:26 GMT+02:00 Xingcan Cui <xingc...@gmail.com>: > >>>>>> > >>>>>>> Hi all, > >>>>>>> > >>>>>>> I also think about this problem these days and here are my > thoughts. > >>>>>>> > >>>>>>> 1) We must admit that it’s really a tough task to interoperate with > >>>> Java > >>>>>>> and Scala. E.g., they have different collection types (Scala > >>>> collections > >>>>>>> v.s. java.util.*) and in Java, it's hard to implement a method > which > >>>>> takes > >>>>>>> Scala functions as parameters. Considering the major part of the > >>> code > >>>>> base > >>>>>>> is implemented in Java, +1 for this goal from a long-term view. > >>>>>>> > >>>>>>> 2) The ideal solution would be to just expose a Scala API and make > >>> all > >>>>> the > >>>>>>> other parts Scala-free. But I am not sure if it could be achieved > >>> even > >>>>> in a > >>>>>>> long-term. Thus as Timo suggested, keep the Scala codes in > >>>>>>> "flink-table-core" would be a compromise solution. > >>>>>>> > >>>>>>> 3) If the community makes the final decision, maybe any new > features > >>>>>>> should be added in Java (regardless of the modules), in order to > >>>> prevent > >>>>>>> the Scala codes from growing. > >>>>>>> > >>>>>>> Best, > >>>>>>> Xingcan > >>>>>>> > >>>>>>> > >>>>>>>> On Jul 2, 2018, at 9:30 PM, Piotr Nowojski < > >>> pi...@data-artisans.com> > >>>>>>> wrote: > >>>>>>>> Bumping the topic. > >>>>>>>> > >>>>>>>> If we want to do this, the sooner we decide, the less code we will > >>>> have > >>>>>>> to rewrite. I have some objections/counter proposals to Fabian's > >>>>> proposal > >>>>>>> of doing it module wise and one module at a time. > >>>>>>>> First, I do not see a problem of having java/scala code even > within > >>>> one > >>>>>>> module, especially not if there are clean boundaries. Like we could > >>>> have > >>>>>>> API in Scala and optimizer rules/logical nodes written in Java in > >>> the > >>>>> same > >>>>>>> module. However I haven’t previously maintained mixed scala/java > >>> code > >>>>> bases > >>>>>>> before, so I might be missing something here. > >>>>>>>> Secondly this whole migration might and most like will take longer > >>>> then > >>>>>>> expected, so that creates a problem for a new code that we will be > >>>>>>> creating. After making a decision to migrate to Java, almost any > new > >>>>> Scala > >>>>>>> line of code will be immediately a technological debt and we will > >>> have > >>>>> to > >>>>>>> rewrite it to Java later. > >>>>>>>> Thus I would propose first to state our end goal - modules > >>> structure > >>>>> and > >>>>>>> which parts of modules we want to have eventually Scala-free. > >>> Secondly > >>>>>>> taking all steps necessary that will allow us to write new code > >>>>> complaint > >>>>>>> with our end goal. Only after that we should/could focus on > >>>>> incrementally > >>>>>>> rewriting the old code. Otherwise we could be stuck/blocked for > >>> years > >>>>>>> writing new code in Scala (and increasing technological debt), > >>> because > >>>>>>> nobody have found a time to rewrite some non important and not > >>>> actively > >>>>>>> developed part of some module. > >>>>>>>> Piotrek > >>>>>>>> > >>>>>>>>> On 14 Jun 2018, at 15:34, Fabian Hueske <fhue...@gmail.com> > >>> wrote: > >>>>>>>>> Hi, > >>>>>>>>> > >>>>>>>>> In general, I think this is a good effort. However, it won't be > >>> easy > >>>>>>> and I > >>>>>>>>> think we have to plan this well. > >>>>>>>>> I don't like the idea of having the whole code base fragmented > >>> into > >>>>> Java > >>>>>>>>> and Scala code for too long. > >>>>>>>>> > >>>>>>>>> I think we should do this one step at a time and focus on > >>> migrating > >>>>> one > >>>>>>>>> module at a time. > >>>>>>>>> IMO, the easiest start would be to port the runtime to Java. > >>>>>>>>> Extracting the API classes into an own module, porting them to > >>> Java, > >>>>> and > >>>>>>>>> removing the Scala dependency won't be possible without breaking > >>> the > >>>>> API > >>>>>>>>> since a few classes depend on the Scala Table API. > >>>>>>>>> > >>>>>>>>> Best, Fabian > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> 2018-06-14 10:33 GMT+02:00 Till Rohrmann <trohrm...@apache.org>: > >>>>>>>>> > >>>>>>>>>> I think that is a noble and honorable goal and we should strive > >>> for > >>>>> it. > >>>>>>>>>> This, however, must be an iterative process given the sheer size > >>> of > >>>>> the > >>>>>>>>>> code base. I like the approach to define common Java modules > >>> which > >>>>> are > >>>>>>> used > >>>>>>>>>> by more specific Scala modules and slowly moving classes from > >>> Scala > >>>>> to > >>>>>>>>>> Java. Thus +1 for the proposal. > >>>>>>>>>> > >>>>>>>>>> Cheers, > >>>>>>>>>> Till > >>>>>>>>>> > >>>>>>>>>> On Wed, Jun 13, 2018 at 12:01 PM Piotr Nowojski < > >>>>>>> pi...@data-artisans.com> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi, > >>>>>>>>>>> > >>>>>>>>>>> I do not have an experience with how scala and java interacts > >>> with > >>>>>>> each > >>>>>>>>>>> other, so I can not fully validate your proposal, but generally > >>>>>>> speaking > >>>>>>>>>> +1 > >>>>>>>>>>> from me. > >>>>>>>>>>> > >>>>>>>>>>> Does it also mean, that we should slowly migrate > >>>> `flink-table-core` > >>>>> to > >>>>>>>>>>> Java? How would you envision it? It would be nice to be able to > >>>> add > >>>>>>> new > >>>>>>>>>>> classes/features written in Java and so that they can coexist > >>> with > >>>>> old > >>>>>>>>>>> Scala code until we gradually switch from Scala to Java. > >>>>>>>>>>> > >>>>>>>>>>> Piotrek > >>>>>>>>>>> > >>>>>>>>>>>> On 13 Jun 2018, at 11:32, Timo Walther <twal...@apache.org> > >>>> wrote: > >>>>>>>>>>>> Hi everyone, > >>>>>>>>>>>> > >>>>>>>>>>>> as you all know, currently the Table & SQL API is implemented > >>> in > >>>>>>> Scala. > >>>>>>>>>>> This decision was made a long-time ago when the initital code > >>> base > >>>>> was > >>>>>>>>>>> created as part of a master's thesis. The community kept Scala > >>>>>>> because of > >>>>>>>>>>> the nice language features that enable a fluent Table API like > >>>>>>>>>>> table.select('field.trim()) and because Scala allows for quick > >>>>>>>>>> prototyping > >>>>>>>>>>> (e.g. multi-line comments for code generation). The committers > >>>>>>> enforced > >>>>>>>>>> not > >>>>>>>>>>> splitting the code-base into two programming languages. > >>>>>>>>>>>> However, nowadays the flink-table module more and more becomes > >>> an > >>>>>>>>>>> important part in the Flink ecosystem. Connectors, formats, and > >>>> SQL > >>>>>>>>>> client > >>>>>>>>>>> are actually implemented in Java but need to interoperate with > >>>>>>>>>> flink-table > >>>>>>>>>>> which makes these modules dependent on Scala. As mentioned in > an > >>>>>>> earlier > >>>>>>>>>>> mail thread, using Scala for API classes also exposes member > >>>>> variables > >>>>>>>>>> and > >>>>>>>>>>> methods in Java that should not be exposed to users [1]. Java > is > >>>>> still > >>>>>>>>>> the > >>>>>>>>>>> most important API language and right now we treat it as a > >>>>>>> second-class > >>>>>>>>>>> citizen. I just noticed that you even need to add Scala if you > >>>> just > >>>>>>> want > >>>>>>>>>> to > >>>>>>>>>>> implement a ScalarFunction because of method clashes between > >>>> `public > >>>>>>>>>> String > >>>>>>>>>>> toString()` and `public scala.Predef.String toString()`. > >>>>>>>>>>>> Given the size of the current code base, reimplementing the > >>>> entire > >>>>>>>>>>> flink-table code in Java is a goal that we might never reach. > >>>>>>> However, we > >>>>>>>>>>> should at least treat the symptoms and have this as a long-term > >>>> goal > >>>>>>> in > >>>>>>>>>>> mind. My suggestion would be to convert user-facing and runtime > >>>>>>> classes > >>>>>>>>>> and > >>>>>>>>>>> split the code base into multiple modules: > >>>>>>>>>>>>> flink-table-java {depends on flink-table-core} > >>>>>>>>>>>> Implemented in Java. Java users can use this. This would > >>> require > >>>> to > >>>>>>>>>>> convert classes like TableEnvironment, Table. > >>>>>>>>>>>>> flink-table-scala {depends on flink-table-core} > >>>>>>>>>>>> Implemented in Scala. Scala users can use this. > >>>>>>>>>>>> > >>>>>>>>>>>>> flink-table-common > >>>>>>>>>>>> Implemented in Java. Connectors, formats, and UDFs can use > >>> this. > >>>> It > >>>>>>>>>>> contains interface classes such as descriptors, table sink, > >>> table > >>>>>>> source. > >>>>>>>>>>>>> flink-table-core {depends on flink-table-common and > >>>>>>>>>>> flink-table-runtime} > >>>>>>>>>>>> Implemented in Scala. Contains the current main code base. > >>>>>>>>>>>> > >>>>>>>>>>>>> flink-table-runtime > >>>>>>>>>>>> Implemented in Java. This would require to convert classes in > >>>>>>>>>>> o.a.f.table.runtime but would improve the runtime potentially. > >>>>>>>>>>>> What do you think? > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Regards, > >>>>>>>>>>>> > >>>>>>>>>>>> Timo > >>>>>>>>>>>> > >>>>>>>>>>>> [1] > >>>>>>>>>>> http://apache-flink-mailing-list-archive.1008284.n3. > >>>>>>>>>> nabble.com/DISCUSS-Convert-main-Table-API-classes-into- > >>>>>>> traits-tp21335.html > >>>>> > >
