Qing, thanks for a great summary. The discussion(offline and much of it on Slack) was whether to add new APIs for Java that removed *Option* from parameters of the existing Scala APIs and add builders where necessary or refactor Scala APIs to achieve this. My answer and response was based on that ie., for a change like this we do not have different code paths and generate 200+ APIs, we could make existing Scala APIs work for this. My thought is that we cannot just stop with NDArrays and should probably do that for all APIs so users have a consistent experience.
In this thread Yizhi proposed to add Builders(for arguments) to all APIs, Now if we were to do that shouldn't we do it for all APIs or at the least for Symbol so users have a consistent experience? As a reminder Symbol has the same set of APIs as NDArrays. Bob/Marco, i'll send how the APIs look for the 3 different approaches. currently working on something else, Qing if you already have please paste the generated APIs in a gist and link here. On Sun, Sep 30, 2018 at 6:31 PM Qing Lan <lanking...@live.com> wrote: > Here I have done a summary about the opinion most people holds: > > Naveen & Yizhi thoughts in a technical way: > Agreement: -1 for total Java conversion (build all wrappers from > Scala to Java) > Yizhi: A hybrid solution may works the best (Build only necessary > Java class for Java users and make best use on Scala Classes) > Naveen: Stop introducing much of the new things for Java users. > Think of changing the existing APIs to makes them compatible. > Yizhi: +1 on builder solution & Naveen +1 on building builders for > necessary operators > Naveen -1 on 2) Andrew proposed. > > There are some arguments in this thread that happened in our MXNet-Scala > Slack channel before on the topic of changing the existing Scala API to > make them more Java friendly (null problem with removal of optional). > Please ignore these portion and feel free to discuss them in the Slack > channel. > > Java users experience is the thing we care of: > Marco (thing big on this, someday we need a wider support for Java) > Chris (Don't makes Java user to learn Scala and let them say > goodbye) > Bob (please help us understand the core difference of the proposed > design) > Qing (we should think of how to make users use them happily > instead of taking too serious trade off on the design) > > Carin: We can think of generating Java api in Java > > On Andrew's approach 2) > Carin (+1) > Qing (+0.5) > Yizhi (+1) > Denis(+1) > Naveen (-1) > > The optimal goal for any of the design is the experience that Java user > holds: > I know what to use: All of the functions that we use are available > in Java (not something we have to learn Scala) > Easy to use: User write minimum code to create an > easy-to-understand functions > Performance: New Java API should have similar performance across > different language bindings > > Lastly I would like to +1 on Yizhi's recommendation: Makes Java API > hybrid: Implement only the classes that Java users may feel sick and > extends the rest of them to better support Java. It's a good trade of on > Build all wrappers in Java/Extends all Java functionalities on Scala. > > Thanks, > Qing > > > > > > > On 9/30/18, 10:05 AM, "Bob Paulin" <b...@apache.org> wrote: > > +1 to this suggestion. Voting should be seen as a last resort to > resolve differences in the community. Perhaps a small POC of both > approaches (or maybe others as well) might help the community discuss > options in a more concrete way. Sometimes is very difficult to "see" > the differences between approaches in an email thread compared to 2 > branches of code. > > - Bob > > > On 9/30/2018 6:17 AM, Marco de Abreu wrote: > > Maybe we can take a step back and rather look at how we would like > our > > users and mxnet developers to use the API. Imagine like a mock or > pseudo > > code where we write a few examples with the "perfect" Java API. > After that, > > we think about the technical implementation details and how it can > be done > > (I know that we already had a lot of technical discussion, it's just > a > > method to get a second perspective). > > > > Our current discussion is around the technical limitations and > possible > > overhead a Java API might cause, but I think it's most important > that a > > user is actually able to use it. I think we have all been at the > point > > where we tried to use a package and the API was so strange that we > just > > picked an alternative. > > > > We should think a bit long term here. Java is a HUGE market with a > big user > > base, especially in the Enterprise segment. So, theoretically > speaking, > > even if we duplicate everything and decouple it from scala entirely, > we > > might have a good chance that - given a good design - other people > will > > step up and help maintaining and extending the API. If the process to > > change anything in that API is super cryptic, we might block future > efforts > > because the entry barrier is too high. > > > > I think our python API is the best example. We are finally at a > state where > > it's super easy to extend, the usability is good (especially around > gluon) > > and it fits into the languages' idioms. Let's try to take that > experience > > and apply it to the Java API. This might be more work initially and > create > > redundancy, but we have seen that a good API (external as well as > internal) > > design increases contributions and attracts more users. > > > > I won't take any side here because I'm unfamiliar with the scala > side of > > mxnet, just wanted to add an external viewpoint. > > > > -Marco > > > > YiZhi Liu <eazhi....@gmail.com> schrieb am So., 30. Sep. 2018, > 05:15: > > > >> Yes agreement and disagreement stay at technical level only:) > >> > >> Back to the problem, they are unnecessary but good in terms of, > >> 1. Still not good for java users to write 3 nulls in a function > call with 5 > >> or 4 args > >> 2. Every function call with a “tail” null for arg “out”. I would > say, makes > >> it seems not a serious api design to our users > >> 3. Users have uniformed experience, nothing surprising. > >> > >> Given the reasons I listed before, I don’t see things bad for this. > >> > >> I agree we can vote. I suggest to have two votes, one for builder, > one for > >> separating java and scala objects. > >> > >> On Sat, Sep 29, 2018 at 7:43 PM Naveen Swamy <mnnav...@gmail.com> > wrote: > >> > >>> Ah! we agree on something :) lets get more opinions, I am happy to > go > >> with > >>> it. > >>> > >>> On Sat, Sep 29, 2018 at 10:40 PM YiZhi Liu <eazhi....@gmail.com> > wrote: > >>> > >>>> Also sometimes people may not be at the same page when talking > about > >>> option > >>>> #2. What I insist is the builder classes for each operator. > Otherwise I > >>>> actually more support Naveen’s approach - not to totally separate > java > >>> and > >>>> scala objects. > >>>> > >>>> On Sat, Sep 29, 2018 at 7:35 PM YiZhi Liu <eazhi....@gmail.com> > wrote: > >>>> > >>>>> No you haven't answered my question "Since you agree to have 30+ > >>>>> operators have Builder, what prevents from > >>>>> having all of them have Builder?" > >>>>> On Sat, Sep 29, 2018 at 7:30 PM Naveen Swamy <mnnav...@gmail.com > > > >>> wrote: > >>>>>> I think we have had enough of an debate between the two of us > and I > >>>> have > >>>>>> already listed my reasons, I will stop here and see what others > say > >>>>> given > >>>>>> my reasoning. > >>>>>> > >>>>>> -1 to #2) > >>>>>> > >>>>>> Also, by lecture I meant to say "I don't want to list all the > >>> problems > >>>>>> with unnecessary complications and talk about how to design > >> software" > >>>>>> On Sat, Sep 29, 2018 at 10:15 PM YiZhi Liu <eazhi....@gmail.com > > > >>>> wrote: > >>>>>>> And if we find incorrect declaration, we fix it, not simply > >>> assuming > >>>>>>> many of them also has problem and we cannot rely on them - > >>> otherwise > >>>>>>> the type-safe APIs in Scala also does not make sense. > >>>>>>> On Sat, Sep 29, 2018 at 7:10 PM YiZhi Liu <eazhi....@gmail.com > > > >>>> wrote: > >>>>>>>> It also makes sense to me if we have it under namespace > >> NDArray, > >>>> not > >>>>>>>> creating new JavaNDArray. But again, uniform experience is > >>>> important. > >>>>>>>> What I responded is your comment "keep scala macros minimum", > I > >>>> don't > >>>>>>>> think "scala macro" equals "cryptic code". Even though it > does, > >>>> what > >>>>>>>> we need to do is to find an alternative way to do code > >>> generation, > >>>>> not > >>>>>>>> making code generation minimum. > >>>>>>>> > >>>>>>>> Since you agree to have 30+ operators have Builder, what > >> prevents > >>>>> from > >>>>>>>> having all of them have Builder? > >>>>>>>> - They're auto-generated, the auto-generation "cryptic" code > is > >>>>> anyway > >>>>>>>> there. And "two different paths of code" (though I don't > >> totally > >>>>>>>> agree) is anyway there. > >>>>>>>> - What else? 200+ classes is a very tiny increasing in file > >> size > >>>>>>>> (~3MB) compare to current status. And won't have any > >> performance > >>>>> issue > >>>>>>>> on modern JVM. > >>>>>>>> > >>>>>>>> Just remind, technical discussion is not about who gives who a > >>>>> lecture. > >>>>>>>> On Sat, Sep 29, 2018 at 6:41 PM Naveen Swamy < > >> mnnav...@gmail.com > >>>>> wrote: > >>>>>>>>> Well, I am not sure(I don't think) we need Builder for every > >>> API > >>>> in > >>>>>>>>> NDArray. For APIs that take long list of parameters, I agree > >> to > >>>> add > >>>>>>> Builder. > >>>>>>>>> Look at the API distribution based on number of arguments > >> here: > >>> https://gist.github.com/nswamy/2dea72e514cc7bfc675f68aef9fe78bb > >>>>>>>>> about 30 APIs have 7 or more arguments.. I agree to add > >>> Builders > >>>>> for > >>>>>>> these > >>>>>>>>> APIs not separately but to the existing Scala APIs but not > >>>>> separately > >>>>>>> only > >>>>>>>>> for Java. > >>>>>>>>> APIs sorted by number of arguments is here, take a look : > >>>>>>>>> > >>> https://gist.github.com/nswamy/e941cb94658b3960eec40bf00b970ac5 > >>>>>>>>> Many of the arguments i think are actually mandatory but > >>>>> incorrectly > >>>>>>>>> declared optional on the backend, for example look at > >> SwapAxis > >>>>>>>>> "def SwapAxis (data : NDArray, dim1 : Option[Int] = None, > >> dim2 > >>> : > >>>>>>>>> Option[Int] = None, out : Option[NDArray] = None) : > >>>>> NDArrayFuncReturn" > >>>>>>>>> Why is dim1 and dim2 Optional, this is an error in the > >>>> declaration > >>>>> on > >>>>>>> the > >>>>>>>>> backend, I think there might be many of these? > >>>>>>>>> > >>>>>>>>> My answers to your other responses are below inline: > >>>>>>>>> > >>>>>>>>> On Sat, Sep 29, 2018 at 3:37 PM YiZhi Liu < > >> eazhi....@gmail.com > >>>>> wrote: > >>>>>>>>>> Some of my comments inline: > >>>>>>>>>> > >>>>>>>>>>> Why can we not create the builder just for these APIs( > >>> which > >>>> we > >>>>>>>>>> discussed), why is it necessary to add 200 Apis > >>>>>>>>>> It is about unified user-experience. And we get rid of > >>> annoying > >>>>> extra > >>>>>>>>>> "out=null" in every operator. > >>>>>>>>>> > >>>>>>>>>>> Are you suggesting to create builder for each and every > >>> API? > >>>>>>>>>> Only for those are necessary. For NDArray.XXX, yes. > >>>>>>>>>> > >>>>>>>>> I think this is a ridiculous list of Builders, I think we can > >>>> keep > >>>>> the > >>>>>>>>> 'out' parameter > >>>>>>>>> > >>>>>>>>>> 1) The NDArray APIs in question are not following > >> functional > >>>>> style of > >>>>>>>>>> programming, in fact they are just static methods defined > >> on > >>> an > >>>>>>>>>> NDArray object - so Scala users are not losing much by > >> using > >>>>> null in > >>>>>>>>>> place of None. > >>>>>>>>>> You can create a implicit to maintain backward > >> compatibility > >>>>>>>>>> - I doubt implicit can work in such case from None -> null. > >>>>>>>>>> > >>>>>>>>> It is just writing getOrElse in your implicit, so it will > >> work. > >>>>>>>>> scala> implicit def optionStringToString(a: Option[String]): > >>>>> String = { > >>>>>>>>> | a.getOrElse(null) > >>>>>>>>> | } > >>>>>>>>> > >>>>>>>>> 2) It is adding 220+ APIs(I understand it is generated) for > >>>> NDArray > >>>>>>> alone > >>>>>>>>>> - As I explained how it can improve user experiences > >>>>>>>>>> > >>>>>>>>> I don't think we need to write builders for 221 APIs we have, > >>> may > >>>>> be > >>>>>>> for 30 > >>>>>>>>> or so. Uniform experience is good goal but it also has to be > >>>>> practical > >>>>>>> and > >>>>>>>>> make sense. > >>>>>>>>> > >>>>>>>>> 3) this is adding another 100s of APIs unnecessarily, we are > >>>>> starting > >>>>>>> with > >>>>>>>>>> NDArray but we can't stop there, we will have to do this > >> for > >>>>> Symbol, > >>>>>>>>>> Executor, Iterators, etc., . > >>>>>>>>>> - This is a good point, actually I prefer not to make > >>>>> JavaExecutor, > >>>>>>>>>> JavaIterators > >>>>>>>>>> > >>>>>>>>> What I was aiming is also users have the same experience > >> across > >>>>>>> Interfaces > >>>>>>>>> - now you are forgoing uniform experience, so like you said > >> its > >>>> all > >>>>>>>>> trade-off and a good trade-off doesn't cause too much > >> overhead/ > >>>>>>>>> > >>>>>>>>>> 4) I don't want to be fixing bugs and maintaining code in 2 > >>>>> places. > >>>>>>>>>> - Type-safe parsing is shared. I think Qing is more > >> qualified > >>>> to > >>>>>>> comment. > >>>>>>>>> It creates two different paths of code for Scala and Java - > >> how > >>>> is > >>>>> it > >>>>>>> going > >>>>>>>>> to be shared. I am afraid we are going to make it more > >>>> complicated > >>>>> than > >>>>>>>>> necessary by duplicating code. > >>>>>>>>> > >>>>>>>>> 5) I want the cryptic code(# scala macros) to a minimum. > >>>>>>>>>> - MXNet decides to do operator generation in frontend > >>> bindings. > >>>>> It's > >>>>>>>>>> the developers' responsibility to understand the techniques > >>>> they > >>>>> are > >>>>>>>>>> using. Maybe not a so proper analogy - "I don't know RL / > >> RL > >>> is > >>>>> hard > >>>>>>>>>> to tune / ..." is not a reason for "I want to keep RL > >>>>> implementation > >>>>>>>>>> in MXNet as a small part as possible" > >>>>>>>>>> > >>>>>>>>>> Now, this is a response I don't like. I don't know where > >> you > >>>> were > >>>>>>> going > >>>>>>>>> with your analogy but know that it sounds condescending - I > >> am > >>>>> going to > >>>>>>>>> ignore(assuming good intentions) that and explain what I > >> mean. > >>>>>>>>> I here is I the developer/user who deploys MXNet code in > >>>>> production and > >>>>>>>>> have to deal with the aftermath myself not you(MXNet > >>> developers). > >>>>>>>>> From your comment it occurs to me you probably have never > >> been > >>> on > >>>>>>>>> pager-duty. I have been on pager-duty both for the code I > >> wrote > >>>> and > >>>>>>> those > >>>>>>>>> that was written by others and thrown over the fence. > >>>>>>>>> If you get woken up by a beep at the middle of the night, > >> that > >>> is > >>>>> not > >>>>>>> the > >>>>>>>>> time to prove your intelligence. Its time to mitigate the > >> issue > >>>>> asap > >>>>>>> for > >>>>>>>>> that your code needs to be easy to follow, should follow well > >>>>> defined > >>>>>>>>> patterns, etc., -- i don't need to give you a lecture. > >>>>>>>>> IMHO It is extremely important for frameworks like Apache > >> MXNet > >>>>> which > >>>>>>> are > >>>>>>>>> used by others for their production to keep code simple and > >>>>> *cryptic > >>>>>>> code > >>>>>>>>> to a minimum* and yes you(we - MXNet developers) are not > >>>> answering > >>>>> the > >>>>>>>>> beepers when your(MXNet) users deploy their code in their > >>>>> production so > >>>>>>>>> make their life simple. > >>>>>>>>> > >>>>>>>>> 6) increased compilation time & bad developer experience - > >> the > >>>>> time to > >>>>>>>>>> compile has gone up quite a bit since we added the APIs > >> last > >>>>> release > >>>>>>> on my > >>>>>>>>>> 3 year old laptop already.. I think adding 400+ APIs > >>>>> unnecessarily > >>>>>>> would > >>>>>>>>>> significantly increase build time and bad developer > >>> experience > >>>>>>>>>> - I don't think increasing such a bit compilation time is a > >>>>> problem > >>>>>>>>>> compared to bad user experience. > >>>>>>>>> I am not suggesting bad user experience but to take a > >> practical > >>>>>>> approach - > >>>>>>>>> having a bad developer experience is not great either. > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> 7) I want to keep the core of the framework to be in Scala - > >>>>> because it > >>>>>>>>>> allows you to write concise code - Yes it has a bit of > >>> learning > >>>>>>> curve, not > >>>>>>>>>> everyone needs to know. I would rather invest in > >> solidifying > >>>> the > >>>>>>> Scala APIs > >>>>>>>>>> and add more features in Scala(RNN, Support > >>>>>>> GluonHybridizedBlock...there is > >>>>>>>>>> quite bit of work ) - do you want to rewrite everything in > >>>> Scala > >>>>> and > >>>>>>> Java. > >>>>>>>>>> - I agree with "don't rewrite everything in Scala and > >> Java", > >>>> IMO > >>>>>>>>>> JavaNDArray is the only one good to have. JShape, JContext, > >>>> etc. > >>>>> are > >>>>>>>>>> not so necessary. > >>>>>>>>>> > >>>>>>>>>> Either you go all Java or make accommodation in Scala code > >> to > >>>>> work > >>>>>>> for > >>>>>>>>> APIs so your users know what to expect(uniform experience > >>>> across). > >>>>>>>>>> 8) Also, the discussion is not creating NDArray class for > >>> Java, > >>>>> just > >>>>>>>>>> generate certain APIs to cater for Java incompatibility. > >>>>>>>>>> - Yes I agree it's about "generate certain APIs to cater > >> for > >>>> Java > >>>>>>>>>> incompatibility", though I think NDArray.api.XXX does not > >>> meet > >>>>> Java > >>>>>>>>>> users' demands. > >>>>>>>>>> > >>>>>>>>> On Sat, Sep 29, 2018 at 12:05 PM Naveen Swamy < > >>>> mnnav...@gmail.com> > >>>>>>> wrote: > >>>>>>>>>>> I know it is about trade-off. I am suggesting a > >> trade-off > >>> , > >>>>> how > >>>>>>> many > >>>>>>>>>> apis do we have that takes too many parameters ? > >>>>>>>>>>> From what I recall its around 20. Why can we not create > >> the > >>>>>>> builder just > >>>>>>>>>> for these APIs( which we discussed), why is it necessary to > >>> add > >>>>> 200 > >>>>>>> Apis ? > >>>>>>>>>>> Are you suggesting to create builder for each and every > >>> API? > >>>>>>>>>>> I disagree with your opinion that they are not important > >>> and > >>>>> would > >>>>>>> like > >>>>>>>>>> to hear from others. > >>>>>>>>>>> I am curious to see how the #2 looks like compared to #1 > >>>>>>>>>>> Andrew/Qing, can you paste the generated Apis that you > >> have > >>>> for > >>>>>>> both > >>>>>>>>>> Scala and Java in a gist please. > >>>>>>>>>>>> On Sep 29, 2018, at 2:41 PM, YiZhi Liu < > >>>> eazhi....@gmail.com> > >>>>>>> wrote: > >>>>>>>>>>>> Naveen, software designing is all about tradeoff, every > >>>>> feature > >>>>>>> we > >>>>>>>>>>>> introduce causes more compiling time, more efforts to > >>>>> maintain, > >>>>>>> etc. > >>>>>>>>>>>> The main difference is. > >>>>>>>>>>>> > >>>>>>>>>>>> Option #1: Java users do > >>>>>>>>>>>> NDArray.BatchNorm(data, gamma, beta, null, null, null, > >>>> null, > >>>>>>> null, > >>>>>>>>>>>> null, null, null, null, null, null); > >>>>>>>>>>>> (and because every operator has an argument "out", > >> users > >>>>> need to > >>>>>>> add > >>>>>>>>>>>> an extra "null" to the function call almost every > >> time.) > >>>>>>>>>>>> Option #2, Java users do > >>>>>>>>>>>> > >> JavaNDArray.BatchNorm(data).setGamma(gamma).setBeta(beta).invoke(); > >>>>>>>>>>>> I don't think any of the reasons you listed is so > >>> important > >>>>> as > >>>>>>> the > >>>>>>>>>>>> benefit above we got from option #2. > >>>>>>>>>>>>> On Sat, Sep 29, 2018 at 8:24 AM Naveen Swamy < > >>>>>>> mnnav...@gmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>>>>> Java APIs are not like Clojure - The current proposal > >> is > >>>>> only to > >>>>>>>>>> build a > >>>>>>>>>>>>> few thin wrappers for Inference. > >>>>>>>>>>>>> > >>>>>>>>>>>>> To better represent the two cases and this discussion > >> in > >>>>>>> particular, > >>>>>>>>>> here > >>>>>>>>>>>>> is an example API > >>>>>>>>>>>>> > >>>>>>>>>>>>> 1) def Activation (data : org.apache.mxnet.NDArray, > >>>>> act_type : > >>>>>>>>>> String, out > >>>>>>>>>>>>> : Option[NDArray] = None) : > >>>>> org.apache.mxnet.NDArrayFuncReturn > >>>>>>>>>>>>> or > >>>>>>>>>>>>> 2) def Activation (data : org.apache.mxnet.NDArray, > >>>>> act_type : > >>>>>>>>>> String, out > >>>>>>>>>>>>> : NDArray) : org.apache.mxnet.NDArrayFuncReturn > >>>>>>>>>>>>> > >>>>>>>>>>>>> The discussion is should we add(generate) 200+ APIs to > >>>> make > >>>>> it > >>>>>>> Java > >>>>>>>>>>>>> compatible, ie., remove the Option class and the None > >>>>> default > >>>>>>> value > >>>>>>>>>> which > >>>>>>>>>>>>> Java does not understand from Option 1) > >>>>>>>>>>>>> > >>>>>>>>>>>>> my suggestion was to remove the Option class and > >> create > >>> a > >>>>>>> implicit for > >>>>>>>>>>>>> backward compatibility and use null instead of None, > >>>> Andrew > >>>>> and > >>>>>>> I > >>>>>>>>>> disagreed > >>>>>>>>>>>>> on this, so I suggested to raise a discussion on dev@ > >>> to > >>>>> get > >>>>>>> more > >>>>>>>>>> opinions > >>>>>>>>>>>>> and one of us will disagree and commit. Thanks for > >>> raising > >>>>> it :) > >>>>>>>>>>>>> | * def Activation (data : org.apache.mxnet.NDArray, > >>>>> act_type : > >>>>>>>>>> String, out > >>>>>>>>>>>>> : NDArray = null) : > >> org.apache.mxnet.NDArrayFuncReturn | > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> > >>>>>>>>>>>>> 1) It is not true that Scala users will lose > >>>>> *default/optional* > >>>>>>>>>> arguments - > >>>>>>>>>>>>> if we followed the above, they will use null or None, > >>>>> though I > >>>>>>> do not > >>>>>>>>>> like > >>>>>>>>>>>>> using nulls, this is a fine compromise. > >>>>>>>>>>>>> To keep backward compatibility we can create a > >> implicit > >>> to > >>>>>>> convert > >>>>>>>>>>>>> Option.None to nulls and Option.Some-> Option.get(), > >> so > >>>> you > >>>>> are > >>>>>>> not > >>>>>>>>>> going > >>>>>>>>>>>>> to break users who might have been using the APIs that > >>>> were > >>>>>>> released > >>>>>>>>>> in > >>>>>>>>>>>>> 1.3. The current incompatibility is only this w.r.t. > >>>>> NDArrays. > >>>>>>>>>>>>> 2) Now about the Scala Macros - they are not simple to > >>>> read > >>>>> or > >>>>>>> use, > >>>>>>>>>> When I > >>>>>>>>>>>>> and Qing started working on the #Scala Macros to > >> improve > >>>> the > >>>>>>> APIs, it > >>>>>>>>>> took > >>>>>>>>>>>>> us a good amount of time to get a hang of it. I don't > >>> want > >>>>> to > >>>>>>> add > >>>>>>>>>>>>> additional code when not necessary. > >>>>>>>>>>>>> > >>>>>>>>>>>>> My suggestion and vote is to modify existing > >> Macro(i.e., > >>>> #1 > >>>>>>> from the > >>>>>>>>>>>>> original email with the necessary clarification above) > >>> and > >>>>> make > >>>>>>> it > >>>>>>>>>>>>> compatible with Java > >>>>>>>>>>>>> Here are my reasons > >>>>>>>>>>>>> 1) The NDArray APIs in question are not following > >>>> functional > >>>>>>> style of > >>>>>>>>>>>>> programming, in fact they are just static methods > >>> defined > >>>>> on an > >>>>>>>>>> NDArray > >>>>>>>>>>>>> object - so Scala users are not losing much by using > >>> null > >>>> in > >>>>>>> place of > >>>>>>>>>> None. > >>>>>>>>>>>>> You can create a implicit to maintain backward > >>>> compatibility > >>>>>>>>>>>>> 2) It is adding 220+ APIs(I understand it is > >> generated) > >>>> for > >>>>>>> NDArray > >>>>>>>>>> alone > >>>>>>>>>>>>> 3) this is adding another 100s of APIs unnecessarily, > >> we > >>>> are > >>>>>>> starting > >>>>>>>>>> with > >>>>>>>>>>>>> NDArray but we can't stop there, we will have to do > >> this > >>>> for > >>>>>>> Symbol, > >>>>>>>>>>>>> Executor, Iterators, etc., . > >>>>>>>>>>>>> 3) I don't want to be fixing bugs and maintaining code > >>> in > >>>> 2 > >>>>>>> places. > >>>>>>>>>>>>> 4) I want the cryptic code(# scala macros) to a > >> minimum. > >>>>>>>>>>>>> 5) increased compilation time & bad developer > >>> experience - > >>>>> the > >>>>>>> time to > >>>>>>>>>>>>> compile has gone up quite a bit since we added the > >> APIs > >>>> last > >>>>>>> release > >>>>>>>>>> on my > >>>>>>>>>>>>> 3 year old laptop already.. I think adding 400+ APIs > >>>>>>> unnecessarily > >>>>>>>>>> would > >>>>>>>>>>>>> significantly increase build time and bad developer > >>>>> experience > >>>>>>>>>>>>> 6) I want to keep the core of the framework to be in > >>>> Scala - > >>>>>>> because > >>>>>>>>>> it > >>>>>>>>>>>>> allows you to write concise code - Yes it has a bit of > >>>>> learning > >>>>>>>>>> curve, not > >>>>>>>>>>>>> everyone needs to know. I would rather invest in > >>>>> solidifying the > >>>>>>>>>> Scala APIs > >>>>>>>>>>>>> and add more features in Scala(RNN, Support > >>>>>>>>>> GluonHybridizedBlock...there is > >>>>>>>>>>>>> quite bit of work ) - do you want to rewrite > >> everything > >>> in > >>>>>>> Scala and > >>>>>>>>>> Java. > >>>>>>>>>>>>> 7) Also, the discussion is not creating NDArray class > >>> for > >>>>> Java, > >>>>>>> just > >>>>>>>>>>>>> generate certain APIs to cater for Java > >> incompatibility. > >>>>>>>>>>>>> @Andrew: To your response to Qing's comments - you > >>> cannot > >>>>> just > >>>>>>>>>> consider it > >>>>>>>>>>>>> as just generating NDArray's APIs and instead I > >> suggest > >>> to > >>>>> take > >>>>>>> a > >>>>>>>>>> wholistic > >>>>>>>>>>>>> view of all the various implications. > >>>>>>>>>>>>> > >>>>>>>>>>>>> @Chris: Yes, Scala has a bit of learning curve - the > >>> goal > >>>>> is not > >>>>>>>>>> having > >>>>>>>>>>>>> every developer to deal with how these APIs are > >>> generated, > >>>>>>>>>>>>> the problem exists either ways with the above > >> proposal. > >>> I > >>>>> might > >>>>>>> agree > >>>>>>>>>> if we > >>>>>>>>>>>>> were to move away completely(with a thorough > >> discussion > >>>> and > >>>>>>> valid > >>>>>>>>>> reasons) > >>>>>>>>>>>>> and instead use AspectJ or similar to write these > >> APIs, > >>>> the > >>>>>>>>>> discussion is > >>>>>>>>>>>>> about using Scala Macros to generate 2 different types > >>> of > >>>>> APIs > >>>>>>> which > >>>>>>>>>> are > >>>>>>>>>>>>> functionally not different and usability wise are very > >>>> very > >>>>>>> similar, > >>>>>>>>>> look > >>>>>>>>>>>>> at the example. > >>>>>>>>>>>>> Thanks for your input, I will deposit your 0.02$ in > >> our > >>>> JIRA > >>>>>>> bank :) > >>>>>>>>>>>>> @Carin: It requires more effort to use AspectJ or > >>> similar > >>>> to > >>>>>>> generate > >>>>>>>>>> APIs > >>>>>>>>>>>>> using reflection or at compile time, here we need to > >>>>> generate at > >>>>>>>>>> compile > >>>>>>>>>>>>> time so Java users have the API signature on their > >> IDEs. > >>>>>>>>>>>>> Thanks, Naveen > >>>>>>>>>>>>> > >>>>>>>>>>>>> P.S: I am traveling and my responses will be delayed. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> On Fri, Sep 28, 2018 at 10:25 AM Carin Meier < > >>>>>>> carinme...@gmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>> Sorry bad paste on the gist - here is the good one > >>>>>>>>>>>>>> > >> https://gist.github.com/gigasquid/01cd48f563db4739910592dd9ac9db20 > >>>>>>>>>>>>>>> On Fri, Sep 28, 2018 at 10:24 AM Carin Meier < > >>>>>>> carinme...@gmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>>> +1 on option #2 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> In the case of minimizing the the overhead for code > >>>>>>> maintenance, I > >>>>>>>>>> wanted > >>>>>>>>>>>>>>> to suggest the option of investigating generating > >> code > >>>>> from > >>>>>>> the Java > >>>>>>>>>>>>>>> Reflection for the Java APIs. I did a quick gist > >> from > >>>>>>> Clojure of > >>>>>>>>>> what > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>> generated classes look like from the current Scala > >>>>> Symbol.api > >>>>>>> for > >>>>>>>>>>>>>>> FullyConnected here > >>>>>>>>>>>>>>> > >>>> https://gist.github.com/gigasquid/01cd48f563db4739910592 > >>>>>>>>>>>>>>> I looks like that there is always a base Java class > >>>>> generated > >>>>>>> will > >>>>>>>>>> all > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>> arguments. If this is the case, then there is a > >>>>> possibility to > >>>>>>>>>> generate a > >>>>>>>>>>>>>>> Java api based on this Java method automatically > >> with > >>>>> just a > >>>>>>>>>> conversion > >>>>>>>>>>>>>> for > >>>>>>>>>>>>>>> the Scala option and it might be reusable for all > >> the > >>>>>>> packages. > >>>>>>>>>>>>>>> Not sure if it will work for this use case, but > >>> thought > >>>> I > >>>>>>> would > >>>>>>>>>> bring it > >>>>>>>>>>>>>>> up in case it's helpful. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> - Carin > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Fri, Sep 28, 2018 at 7:05 AM Davydenko, Denis > >>>>>>>>>> <d...@amazon.com.invalid > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> +1 on option #2. Having clear Java interface for > >>>> NDArray, > >>>>>>> from my > >>>>>>>>>>>>>>>> perspective, would be a better experience for Java > >>>> users > >>>>> as > >>>>>>> it > >>>>>>>>>> won't > >>>>>>>>>>>>>>>> require them to deal with Scala code in any > >> capacity. > >>>>>>> Overhead of > >>>>>>>>>> extra > >>>>>>>>>>>>>>>> code for additional macros is justified, in my > >> mind, > >>> as > >>>>> it > >>>>>>> will be > >>>>>>>>>>>>>>>> introduced with option #1 either way, just in a > >>>> different > >>>>>>> place. > >>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>> Denis > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 9/27/18, 6:14 PM, "YiZhi Liu" < > >>> eazhi....@gmail.com > >>>>>>> wrote: > >>>>>>>>>>>>>>>> I vote for "2.) Leave the existing macro in place > >>> and > >>>>> add > >>>>>>> another > >>>>>>>>>>>>>>>> which generates a Java friendly version" > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> @Qing @Andrew, could you give some examples, so > >> that > >>>>> people > >>>>>>> can > >>>>>>>>>>>>>> better > >>>>>>>>>>>>>>>> understand how it provides "best possible > >>> experience" > >>>> to > >>>>>>> Java > >>>>>>>>>> users. > >>>>>>>>>>>>>>>> I have no strong preference between having > >>> JavaShape & > >>>>>>> JavaContext > >>>>>>>>>>>>>> or > >>>>>>>>>>>>>>>> not. > >>>>>>>>>>>>>>>> On Thu, Sep 27, 2018 at 5:56 PM Andrew Ayres < > >>>>>>>>>>>>>>>> andrew.f.ay...@gmail.com> wrote: > >>>>>>>>>>>>>>>>> That's not really the conversation I'm wanting to > >>>> have. > >>>>> I > >>>>>>> want a > >>>>>>>>>>>>>>>> discussion > >>>>>>>>>>>>>>>>> about the macros with respect to NDArray so that > >> we > >>>> can > >>>>> get > >>>>>>>>>>>>>>>> agreement on > >>>>>>>>>>>>>>>>> our path forward with respect to implementing the > >>>>> NDArray > >>>>>>> wrapper. > >>>>>>>>>>>>>>>>> The design that was put forth and agreed to was > >> for > >>> a > >>>> a > >>>>> Java > >>>>>>>>>>>>>>>> wrapper around > >>>>>>>>>>>>>>>>> the Scala API. Adding a bunch of Java friendly > >>> methods > >>>>>>> inside the > >>>>>>>>>>>>>>>> Scala > >>>>>>>>>>>>>>>>> code would create a mess for users. Maintenance > >>> would > >>>> be > >>>>>>>>>>>>>>>> essentially the > >>>>>>>>>>>>>>>>> same for both because either way you're going to > >> be > >>>>>>> updating Java > >>>>>>>>>>>>>>>> methods > >>>>>>>>>>>>>>>>> when you make Scala changes. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Let's please stick with the issue in the original > >>>> email. > >>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>> Andrew > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Thu, Sep 27, 2018 at 5:22 PM Qing Lan < > >>>>>>> lanking...@live.com> > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I would like to loop this back a layer. Current, > >>>> there > >>>>> is a > >>>>>>>>>>>>>>>> discussion in > >>>>>>>>>>>>>>>>>> the MXNet Scala community on the ways to > >> implement > >>>> the > >>>>> Java > >>>>>>>>>>>>>> APIs. > >>>>>>>>>>>>>>>> Currently > >>>>>>>>>>>>>>>>>> there are two thoughts: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> 1. Make Scala Java Friendly (Create Java > >> compatible > >>>>>>> methods in > >>>>>>>>>>>>>>>> the Scala > >>>>>>>>>>>>>>>>>> Class. such as NDArray with Java compatible > >>>>> constructor) > >>>>>>>>>>>>>>>>>> 2. Make Java friendly wrappers in Scala (Andrew's > >>>>>>> explanation > >>>>>>>>>>>>>>>> below) > >>>>>>>>>>>>>>>>>> The first approach require minimum input from our > >>>> side > >>>>> to > >>>>>>>>>>>>>>>> implement > >>>>>>>>>>>>>>>>>> however bring user a bunch of useless api they > >> may > >>>> not > >>>>>>> want to > >>>>>>>>>>>>>>>> use. It also > >>>>>>>>>>>>>>>>>> makes Scala package heavier. The good thing is > >>> these > >>>>> two > >>>>>>>>>>>>>> packages > >>>>>>>>>>>>>>>> require > >>>>>>>>>>>>>>>>>> minimum maintenance cost. As a tradeoff, if any > >>> time > >>>>> in the > >>>>>>>>>>>>>>>> future we want > >>>>>>>>>>>>>>>>>> to make Java big (make Java as the primary > >> language > >>>>>>> supported by > >>>>>>>>>>>>>>>> MXNet), > >>>>>>>>>>>>>>>>>> then the migration from Scala to Java will be > >>>> harmful. > >>>>>>> Spark > >>>>>>>>>>>>>>>> consider this > >>>>>>>>>>>>>>>>>> carefully and decide not to change much on their > >>>> Scala > >>>>>>> code base > >>>>>>>>>>>>>>>> to make it > >>>>>>>>>>>>>>>>>> more Java. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> The second approach will make unique NDArray, > >>> Shape, > >>>>>>> Context and > >>>>>>>>>>>>>>>> more. The > >>>>>>>>>>>>>>>>>> good thing about this is we can always holds a > >>>> version > >>>>>>> control > >>>>>>>>>>>>>> on > >>>>>>>>>>>>>>>> Java. > >>>>>>>>>>>>>>>>>> Some breaking changes on Scala may not influence > >>> much > >>>>> on > >>>>>>> Java. > >>>>>>>>>>>>>> It > >>>>>>>>>>>>>>>> did the > >>>>>>>>>>>>>>>>>> best way to decouple the module and good for us > >> to > >>>>> build > >>>>>>> unique > >>>>>>>>>>>>>>>> pipeline > >>>>>>>>>>>>>>>>>> for Java. The bad thing with this design is the > >>>>>>> maintenance cost > >>>>>>>>>>>>>>>> as we need > >>>>>>>>>>>>>>>>>> to keep two code bases, but it also make Java > >> side > >>>>> easy to > >>>>>>>>>>>>>> change > >>>>>>>>>>>>>>>> to make > >>>>>>>>>>>>>>>>>> it better compatible with users. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>> Qing > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 9/27/18, 3:25 PM, "Andrew Ayres" < > >>>>>>> andrew.f.ay...@gmail.com> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Currently, we're working to implement a new Java > >>> API > >>>>> and > >>>>>>>>>>>>>>>> would like > >>>>>>>>>>>>>>>>>> some > >>>>>>>>>>>>>>>>>> feedback from the community on an implementation > >>>>> detail. > >>>>>>> In > >>>>>>>>>>>>>>>> short, the > >>>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>> Java API will use the existing Scala API (in a > >>>> manner > >>>>>>>>>>>>>> similar > >>>>>>>>>>>>>>>> to how > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> current Clojure API works). This basically means > >>>> that > >>>>>>> we're > >>>>>>>>>>>>>>>> making Java > >>>>>>>>>>>>>>>>>> friendly wrappers to call the existing Scala > >> API. > >>>>>>>>>>>>>>>>>> The feedback we're looking for is on the > >>>>> implementation of > >>>>>>>>>>>>>>>> NDArray. > >>>>>>>>>>>>>>>>>> Scala's > >>>>>>>>>>>>>>>>>> NDArray has a significant amount of code which > >> is > >>>>>>> generated > >>>>>>>>>>>>>>>> via macros > >>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>> we've got two viable paths to move forward: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> 1.) Change the macro to generate Java friendly > >>>>> methods - > >>>>>>> To > >>>>>>>>>>>>>>>> do this > >>>>>>>>>>>>>>>>>> we'll > >>>>>>>>>>>>>>>>>> modify the macro so that the generated methods > >>> won't > >>>>> have > >>>>>>>>>>>>>>>>>> default/optional > >>>>>>>>>>>>>>>>>> arguments. There may also have to be some > >> changes > >>> to > >>>>>>>>>>>>>>>> parameter types to > >>>>>>>>>>>>>>>>>> make them Java friendly. The big advantage here > >> is > >>>>> that > >>>>>>>>>>>>>>>> ongoing > >>>>>>>>>>>>>>>>>> maintenance > >>>>>>>>>>>>>>>>>> will easier. The disadvantages are that we'll be > >>>>> changing > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> existing > >>>>>>>>>>>>>>>>>> Scala NDArray Infer API (it's marked > >> experimental) > >>>> and > >>>>>>> Scala > >>>>>>>>>>>>>>>> users will > >>>>>>>>>>>>>>>>>> lose the ability to use the default and optional > >>>>>>> arguments. > >>>>>>>>>>>>>>>>>> 2.) Leave the existing macro in place and add > >>>> another > >>>>>>> which > >>>>>>>>>>>>>>>> generates a > >>>>>>>>>>>>>>>>>> Java friendly version - The biggest issue here > >> is > >>>> that > >>>>>>> we'll > >>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>> doubling > >>>>>>>>>>>>>>>>>> the number of macros that we've got to maintain. > >>>> It'll > >>>>>>>>>>>>>> become > >>>>>>>>>>>>>>>> even more > >>>>>>>>>>>>>>>>>> overhead once we start expanding the Java API > >> with > >>>>> more > >>>>>>>>>>>>>>>> classes that > >>>>>>>>>>>>>>>>>> use > >>>>>>>>>>>>>>>>>> generated code like this. The advantages are > >> that > >>>> the > >>>>>>>>>>>>>>>> existing Scala > >>>>>>>>>>>>>>>>>> NDArray Infer API would remain unchanged for > >> Scala > >>>>> users > >>>>>>> and > >>>>>>>>>>>>>>>> that the > >>>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>> macro could be optimized to give the best > >> possible > >>>>>>>>>>>>>> experience > >>>>>>>>>>>>>>>> to the > >>>>>>>>>>>>>>>>>> Java > >>>>>>>>>>>>>>>>>> API. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>> Andrew > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>> Yizhi Liu > >>>>>>>>>>>>>>>> DMLC member > >>>>>>>>>>>>>>>> Amazon Web Services > >>>>>>>>>>>>>>>> Vancouver, Canada > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> -- > >>>>>>>>>>>> Yizhi Liu > >>>>>>>>>>>> DMLC member > >>>>>>>>>>>> Amazon Web Services > >>>>>>>>>>>> Vancouver, Canada > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Yizhi Liu > >>>>>>>>>> DMLC member > >>>>>>>>>> Amazon Web Services > >>>>>>>>>> Vancouver, Canada > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> Yizhi Liu > >>>>>>>> DMLC member > >>>>>>>> Amazon Web Services > >>>>>>>> Vancouver, Canada > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Yizhi Liu > >>>>>>> DMLC member > >>>>>>> Amazon Web Services > >>>>>>> Vancouver, Canada > >>>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Yizhi Liu > >>>>> DMLC member > >>>>> Amazon Web Services > >>>>> Vancouver, Canada > >>>>> > >>>> -- > >>>> Yizhi Liu > >>>> DMLC member > >>>> Amazon Web Services > >>>> Vancouver, Canada > >>>> > >> -- > >> Yizhi Liu > >> DMLC member > >> Amazon Web Services > >> Vancouver, Canada > >> > > > >
