Java APIs are not like Clojure - The current proposal is only to build a few thin wrappers for Inference.
To better represent the two cases and this discussion in particular, here is an example API 1) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out : Option[NDArray] = None) : org.apache.mxnet.NDArrayFuncReturn or 2) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out : NDArray) : org.apache.mxnet.NDArrayFuncReturn The discussion is should we add(generate) 200+ APIs to make it Java compatible, ie., remove the Option class and the None default value which Java does not understand from Option 1) my suggestion was to remove the Option class and create a implicit for backward compatibility and use null instead of None, Andrew and I disagreed on this, so I suggested to raise a discussion on dev@ to get more opinions and one of us will disagree and commit. Thanks for raising it :) | * def Activation (data : org.apache.mxnet.NDArray, act_type : String, out : NDArray = null) : org.apache.mxnet.NDArrayFuncReturn | -- 1) It is not true that Scala users will lose *default/optional* arguments - if we followed the above, they will use null or None, though I do not like using nulls, this is a fine compromise. To keep backward compatibility we can create a implicit to convert Option.None to nulls and Option.Some-> Option.get(), so you are not going to break users who might have been using the APIs that were released in 1.3. The current incompatibility is only this w.r.t. NDArrays. 2) Now about the Scala Macros - they are not simple to read or use, When I and Qing started working on the #Scala Macros to improve the APIs, it took us a good amount of time to get a hang of it. I don't want to add additional code when not necessary. My suggestion and vote is to modify existing Macro(i.e., #1 from the original email with the necessary clarification above) and make it compatible with Java Here are my reasons 1) The NDArray APIs in question are not following functional style of programming, in fact they are just static methods defined on an NDArray object - so Scala users are not losing much by using null in place of None. You can create a implicit to maintain backward compatibility 2) It is adding 220+ APIs(I understand it is generated) for NDArray alone 3) this is adding another 100s of APIs unnecessarily, we are starting with NDArray but we can't stop there, we will have to do this for Symbol, Executor, Iterators, etc., . 3) I don't want to be fixing bugs and maintaining code in 2 places. 4) I want the cryptic code(# scala macros) to a minimum. 5) increased compilation time & bad developer experience - the time to compile has gone up quite a bit since we added the APIs last release on my 3 year old laptop already.. I think adding 400+ APIs unnecessarily would significantly increase build time and bad developer experience 6) I want to keep the core of the framework to be in Scala - because it allows you to write concise code - Yes it has a bit of learning curve, not everyone needs to know. I would rather invest in solidifying the Scala APIs and add more features in Scala(RNN, Support GluonHybridizedBlock...there is quite bit of work ) - do you want to rewrite everything in Scala and Java. 7) Also, the discussion is not creating NDArray class for Java, just generate certain APIs to cater for Java incompatibility. @Andrew: To your response to Qing's comments - you cannot just consider it as just generating NDArray's APIs and instead I suggest to take a wholistic view of all the various implications. @Chris: Yes, Scala has a bit of learning curve - the goal is not having every developer to deal with how these APIs are generated, the problem exists either ways with the above proposal. I might agree if we were to move away completely(with a thorough discussion and valid reasons) and instead use AspectJ or similar to write these APIs, the discussion is about using Scala Macros to generate 2 different types of APIs which are functionally not different and usability wise are very very similar, look at the example. Thanks for your input, I will deposit your 0.02$ in our JIRA bank :) @Carin: It requires more effort to use AspectJ or similar to generate APIs using reflection or at compile time, here we need to generate at compile time so Java users have the API signature on their IDEs. Thanks, Naveen P.S: I am traveling and my responses will be delayed. On Fri, Sep 28, 2018 at 10:25 AM Carin Meier <carinme...@gmail.com> wrote: > Sorry bad paste on the gist - here is the good one > https://gist.github.com/gigasquid/01cd48f563db4739910592dd9ac9db20 > > On Fri, Sep 28, 2018 at 10:24 AM Carin Meier <carinme...@gmail.com> wrote: > > > +1 on option #2 > > > > In the case of minimizing the the overhead for code maintenance, I wanted > > to suggest the option of investigating generating code from the Java > > Reflection for the Java APIs. I did a quick gist from Clojure of what > the > > generated classes look like from the current Scala Symbol.api for > > FullyConnected here > > https://gist.github.com/gigasquid/01cd48f563db4739910592 > > > > I looks like that there is always a base Java class generated will all > the > > arguments. If this is the case, then there is a possibility to generate a > > Java api based on this Java method automatically with just a conversion > for > > the Scala option and it might be reusable for all the packages. > > > > Not sure if it will work for this use case, but thought I would bring it > > up in case it's helpful. > > > > - Carin > > > > On Fri, Sep 28, 2018 at 7:05 AM Davydenko, Denis <d...@amazon.com.invalid > > > > wrote: > > > >> +1 on option #2. Having clear Java interface for NDArray, from my > >> perspective, would be a better experience for Java users as it won't > >> require them to deal with Scala code in any capacity. Overhead of extra > >> code for additional macros is justified, in my mind, as it will be > >> introduced with option #1 either way, just in a different place. > >> > >> -- > >> Thanks, > >> Denis > >> > >> On 9/27/18, 6:14 PM, "YiZhi Liu" <eazhi....@gmail.com> wrote: > >> > >> I vote for "2.) Leave the existing macro in place and add another > >> which generates a Java friendly version" > >> > >> @Qing @Andrew, could you give some examples, so that people can > better > >> understand how it provides "best possible experience" to Java users. > >> > >> I have no strong preference between having JavaShape & JavaContext > or > >> not. > >> On Thu, Sep 27, 2018 at 5:56 PM Andrew Ayres < > >> andrew.f.ay...@gmail.com> wrote: > >> > > >> > That's not really the conversation I'm wanting to have. I want a > >> discussion > >> > about the macros with respect to NDArray so that we can get > >> agreement on > >> > our path forward with respect to implementing the NDArray wrapper. > >> > > >> > The design that was put forth and agreed to was for a a Java > >> wrapper around > >> > the Scala API. Adding a bunch of Java friendly methods inside the > >> Scala > >> > code would create a mess for users. Maintenance would be > >> essentially the > >> > same for both because either way you're going to be updating Java > >> methods > >> > when you make Scala changes. > >> > > >> > Let's please stick with the issue in the original email. > >> > > >> > Thanks, > >> > Andrew > >> > > >> > On Thu, Sep 27, 2018 at 5:22 PM Qing Lan <lanking...@live.com> > >> wrote: > >> > > >> > > I would like to loop this back a layer. Current, there is a > >> discussion in > >> > > the MXNet Scala community on the ways to implement the Java > APIs. > >> Currently > >> > > there are two thoughts: > >> > > > >> > > 1. Make Scala Java Friendly (Create Java compatible methods in > >> the Scala > >> > > Class. such as NDArray with Java compatible constructor) > >> > > 2. Make Java friendly wrappers in Scala (Andrew's explanation > >> below) > >> > > > >> > > The first approach require minimum input from our side to > >> implement > >> > > however bring user a bunch of useless api they may not want to > >> use. It also > >> > > makes Scala package heavier. The good thing is these two > packages > >> require > >> > > minimum maintenance cost. As a tradeoff, if any time in the > >> future we want > >> > > to make Java big (make Java as the primary language supported by > >> MXNet), > >> > > then the migration from Scala to Java will be harmful. Spark > >> consider this > >> > > carefully and decide not to change much on their Scala code base > >> to make it > >> > > more Java. > >> > > > >> > > The second approach will make unique NDArray, Shape, Context and > >> more. The > >> > > good thing about this is we can always holds a version control > on > >> Java. > >> > > Some breaking changes on Scala may not influence much on Java. > It > >> did the > >> > > best way to decouple the module and good for us to build unique > >> pipeline > >> > > for Java. The bad thing with this design is the maintenance cost > >> as we need > >> > > to keep two code bases, but it also make Java side easy to > change > >> to make > >> > > it better compatible with users. > >> > > > >> > > Thanks, > >> > > Qing > >> > > > >> > > On 9/27/18, 3:25 PM, "Andrew Ayres" <andrew.f.ay...@gmail.com> > >> wrote: > >> > > > >> > > Hi, > >> > > > >> > > Currently, we're working to implement a new Java API and > >> would like > >> > > some > >> > > feedback from the community on an implementation detail. In > >> short, the > >> > > new > >> > > Java API will use the existing Scala API (in a manner > similar > >> to how > >> > > the > >> > > current Clojure API works). This basically means that we're > >> making Java > >> > > friendly wrappers to call the existing Scala API. > >> > > > >> > > The feedback we're looking for is on the implementation of > >> NDArray. > >> > > Scala's > >> > > NDArray has a significant amount of code which is generated > >> via macros > >> > > and > >> > > we've got two viable paths to move forward: > >> > > > >> > > 1.) Change the macro to generate Java friendly methods - To > >> do this > >> > > we'll > >> > > modify the macro so that the generated methods won't have > >> > > default/optional > >> > > arguments. There may also have to be some changes to > >> parameter types to > >> > > make them Java friendly. The big advantage here is that > >> ongoing > >> > > maintenance > >> > > will easier. The disadvantages are that we'll be changing > the > >> existing > >> > > Scala NDArray Infer API (it's marked experimental) and Scala > >> users will > >> > > lose the ability to use the default and optional arguments. > >> > > > >> > > 2.) Leave the existing macro in place and add another which > >> generates a > >> > > Java friendly version - The biggest issue here is that we'll > >> be > >> > > doubling > >> > > the number of macros that we've got to maintain. It'll > become > >> even more > >> > > overhead once we start expanding the Java API with more > >> classes that > >> > > use > >> > > generated code like this. The advantages are that the > >> existing Scala > >> > > NDArray Infer API would remain unchanged for Scala users and > >> that the > >> > > new > >> > > macro could be optimized to give the best possible > experience > >> to the > >> > > Java > >> > > API. > >> > > > >> > > Thanks, > >> > > Andrew > >> > > > >> > > > >> > > > >> > >> > >> > >> -- > >> Yizhi Liu > >> DMLC member > >> Amazon Web Services > >> Vancouver, Canada > >> > >> > >> >
