Re: Feedback request for new Java API
Qing, thanks for a great summary. The discussion(offline and much of it on Slack) was whether to add new APIs for Java that removed *Option* from parameters of the existing Scala APIs and add builders where necessary or refactor Scala APIs to achieve this. My answer and response was based on that ie., for a change like this we do not have different code paths and generate 200+ APIs, we could make existing Scala APIs work for this. My thought is that we cannot just stop with NDArrays and should probably do that for all APIs so users have a consistent experience. In this thread Yizhi proposed to add Builders(for arguments) to all APIs, Now if we were to do that shouldn't we do it for all APIs or at the least for Symbol so users have a consistent experience? As a reminder Symbol has the same set of APIs as NDArrays. Bob/Marco, i'll send how the APIs look for the 3 different approaches. currently working on something else, Qing if you already have please paste the generated APIs in a gist and link here. On Sun, Sep 30, 2018 at 6:31 PM Qing Lan wrote: > Here I have done a summary about the opinion most people holds: > > Naveen & Yizhi thoughts in a technical way: > Agreement: -1 for total Java conversion (build all wrappers from > Scala to Java) > Yizhi: A hybrid solution may works the best (Build only necessary > Java class for Java users and make best use on Scala Classes) > Naveen: Stop introducing much of the new things for Java users. > Think of changing the existing APIs to makes them compatible. > Yizhi: +1 on builder solution & Naveen +1 on building builders for > necessary operators > Naveen -1 on 2) Andrew proposed. > > There are some arguments in this thread that happened in our MXNet-Scala > Slack channel before on the topic of changing the existing Scala API to > make them more Java friendly (null problem with removal of optional). > Please ignore these portion and feel free to discuss them in the Slack > channel. > > Java users experience is the thing we care of: > Marco (thing big on this, someday we need a wider support for Java) > Chris (Don't makes Java user to learn Scala and let them say > goodbye) > Bob (please help us understand the core difference of the proposed > design) > Qing (we should think of how to make users use them happily > instead of taking too serious trade off on the design) > > Carin: We can think of generating Java api in Java > > On Andrew's approach 2) > Carin (+1) > Qing (+0.5) > Yizhi (+1) > Denis(+1) > Naveen (-1) > > The optimal goal for any of the design is the experience that Java user > holds: > I know what to use: All of the functions that we use are available > in Java (not something we have to learn Scala) > Easy to use: User write minimum code to create an > easy-to-understand functions > Performance: New Java API should have similar performance across > different language bindings > > Lastly I would like to +1 on Yizhi's recommendation: Makes Java API > hybrid: Implement only the classes that Java users may feel sick and > extends the rest of them to better support Java. It's a good trade of on > Build all wrappers in Java/Extends all Java functionalities on Scala. > > Thanks, > Qing > > > > > > > On 9/30/18, 10:05 AM, "Bob Paulin" wrote: > > +1 to this suggestion. Voting should be seen as a last resort to > resolve differences in the community. Perhaps a small POC of both > approaches (or maybe others as well) might help the community discuss > options in a more concrete way. Sometimes is very difficult to "see" > the differences between approaches in an email thread compared to 2 > branches of code. > > - Bob > > > On 9/30/2018 6:17 AM, Marco de Abreu wrote: > > Maybe we can take a step back and rather look at how we would like > our > > users and mxnet developers to use the API. Imagine like a mock or > pseudo > > code where we write a few examples with the "perfect" Java API. > After that, > > we think about the technical implementation details and how it can > be done > > (I know that we already had a lot of technical discussion, it's just > a > > method to get a second perspective). > > > > Our current discussion is around the technical limitations and > possible > > overhead a Java API might cause, but I think it's most important > that a > > user is actually able to use it. I think we have all been at the > point > > where we tried to use a package and the API was so strange that we > just > > picked an alternative. > > > > We should think a bit long term here. Java is a HUGE market with a > big user > > base, especially in the Enterprise segment. So, theoretically > speaking, > > even if we duplicate everything and decouple it from scala entirely, > we > > might have a good chance that - given a good
Re: Feedback request for new Java API
Here I have done a summary about the opinion most people holds: Naveen & Yizhi thoughts in a technical way: Agreement: -1 for total Java conversion (build all wrappers from Scala to Java) Yizhi: A hybrid solution may works the best (Build only necessary Java class for Java users and make best use on Scala Classes) Naveen: Stop introducing much of the new things for Java users. Think of changing the existing APIs to makes them compatible. Yizhi: +1 on builder solution & Naveen +1 on building builders for necessary operators Naveen -1 on 2) Andrew proposed. There are some arguments in this thread that happened in our MXNet-Scala Slack channel before on the topic of changing the existing Scala API to make them more Java friendly (null problem with removal of optional). Please ignore these portion and feel free to discuss them in the Slack channel. Java users experience is the thing we care of: Marco (thing big on this, someday we need a wider support for Java) Chris (Don't makes Java user to learn Scala and let them say goodbye) Bob (please help us understand the core difference of the proposed design) Qing (we should think of how to make users use them happily instead of taking too serious trade off on the design) Carin: We can think of generating Java api in Java On Andrew's approach 2) Carin (+1) Qing (+0.5) Yizhi (+1) Denis(+1) Naveen (-1) The optimal goal for any of the design is the experience that Java user holds: I know what to use: All of the functions that we use are available in Java (not something we have to learn Scala) Easy to use: User write minimum code to create an easy-to-understand functions Performance: New Java API should have similar performance across different language bindings Lastly I would like to +1 on Yizhi's recommendation: Makes Java API hybrid: Implement only the classes that Java users may feel sick and extends the rest of them to better support Java. It's a good trade of on Build all wrappers in Java/Extends all Java functionalities on Scala. Thanks, Qing On 9/30/18, 10:05 AM, "Bob Paulin" wrote: +1 to this suggestion. Voting should be seen as a last resort to resolve differences in the community. Perhaps a small POC of both approaches (or maybe others as well) might help the community discuss options in a more concrete way. Sometimes is very difficult to "see" the differences between approaches in an email thread compared to 2 branches of code. - Bob On 9/30/2018 6:17 AM, Marco de Abreu wrote: > Maybe we can take a step back and rather look at how we would like our > users and mxnet developers to use the API. Imagine like a mock or pseudo > code where we write a few examples with the "perfect" Java API. After that, > we think about the technical implementation details and how it can be done > (I know that we already had a lot of technical discussion, it's just a > method to get a second perspective). > > Our current discussion is around the technical limitations and possible > overhead a Java API might cause, but I think it's most important that a > user is actually able to use it. I think we have all been at the point > where we tried to use a package and the API was so strange that we just > picked an alternative. > > We should think a bit long term here. Java is a HUGE market with a big user > base, especially in the Enterprise segment. So, theoretically speaking, > even if we duplicate everything and decouple it from scala entirely, we > might have a good chance that - given a good design - other people will > step up and help maintaining and extending the API. If the process to > change anything in that API is super cryptic, we might block future efforts > because the entry barrier is too high. > > I think our python API is the best example. We are finally at a state where > it's super easy to extend, the usability is good (especially around gluon) > and it fits into the languages' idioms. Let's try to take that experience > and apply it to the Java API. This might be more work initially and create > redundancy, but we have seen that a good API (external as well as internal) > design increases contributions and attracts more users. > > I won't take any side here because I'm unfamiliar with the scala side of > mxnet, just wanted to add an external viewpoint. > > -Marco > > YiZhi Liu schrieb am So., 30. Sep. 2018, 05:15: > >> Yes agreement and disagreement stay at technical level only:) >> >> Back to the problem, they are unnecessary but good in terms of, >> 1. Still not good for java users to write 3 nulls in a function call with 5 >> or 4 args >> 2. Every
Re: Feedback request for new Java API
+1 to this suggestion. Voting should be seen as a last resort to resolve differences in the community. Perhaps a small POC of both approaches (or maybe others as well) might help the community discuss options in a more concrete way. Sometimes is very difficult to "see" the differences between approaches in an email thread compared to 2 branches of code. - Bob On 9/30/2018 6:17 AM, Marco de Abreu wrote: > Maybe we can take a step back and rather look at how we would like our > users and mxnet developers to use the API. Imagine like a mock or pseudo > code where we write a few examples with the "perfect" Java API. After that, > we think about the technical implementation details and how it can be done > (I know that we already had a lot of technical discussion, it's just a > method to get a second perspective). > > Our current discussion is around the technical limitations and possible > overhead a Java API might cause, but I think it's most important that a > user is actually able to use it. I think we have all been at the point > where we tried to use a package and the API was so strange that we just > picked an alternative. > > We should think a bit long term here. Java is a HUGE market with a big user > base, especially in the Enterprise segment. So, theoretically speaking, > even if we duplicate everything and decouple it from scala entirely, we > might have a good chance that - given a good design - other people will > step up and help maintaining and extending the API. If the process to > change anything in that API is super cryptic, we might block future efforts > because the entry barrier is too high. > > I think our python API is the best example. We are finally at a state where > it's super easy to extend, the usability is good (especially around gluon) > and it fits into the languages' idioms. Let's try to take that experience > and apply it to the Java API. This might be more work initially and create > redundancy, but we have seen that a good API (external as well as internal) > design increases contributions and attracts more users. > > I won't take any side here because I'm unfamiliar with the scala side of > mxnet, just wanted to add an external viewpoint. > > -Marco > > YiZhi Liu schrieb am So., 30. Sep. 2018, 05:15: > >> Yes agreement and disagreement stay at technical level only:) >> >> Back to the problem, they are unnecessary but good in terms of, >> 1. Still not good for java users to write 3 nulls in a function call with 5 >> or 4 args >> 2. Every function call with a “tail” null for arg “out”. I would say, makes >> it seems not a serious api design to our users >> 3. Users have uniformed experience, nothing surprising. >> >> Given the reasons I listed before, I don’t see things bad for this. >> >> I agree we can vote. I suggest to have two votes, one for builder, one for >> separating java and scala objects. >> >> On Sat, Sep 29, 2018 at 7:43 PM Naveen Swamy wrote: >> >>> Ah! we agree on something :) lets get more opinions, I am happy to go >> with >>> it. >>> >>> On Sat, Sep 29, 2018 at 10:40 PM YiZhi Liu wrote: >>> Also sometimes people may not be at the same page when talking about >>> option #2. What I insist is the builder classes for each operator. Otherwise I actually more support Naveen’s approach - not to totally separate java >>> and scala objects. On Sat, Sep 29, 2018 at 7:35 PM YiZhi Liu wrote: > No you haven't answered my question "Since you agree to have 30+ > operators have Builder, what prevents from > having all of them have Builder?" > On Sat, Sep 29, 2018 at 7:30 PM Naveen Swamy >>> wrote: >> I think we have had enough of an debate between the two of us and I have >> already listed my reasons, I will stop here and see what others say > given >> my reasoning. >> >> -1 to #2) >> >> Also, by lecture I meant to say "I don't want to list all the >>> problems >> with unnecessary complications and talk about how to design >> software" >> On Sat, Sep 29, 2018 at 10:15 PM YiZhi Liu wrote: >>> And if we find incorrect declaration, we fix it, not simply >>> assuming >>> many of them also has problem and we cannot rely on them - >>> otherwise >>> the type-safe APIs in Scala also does not make sense. >>> On Sat, Sep 29, 2018 at 7:10 PM YiZhi Liu wrote: It also makes sense to me if we have it under namespace >> NDArray, not creating new JavaNDArray. But again, uniform experience is important. What I responded is your comment "keep scala macros minimum", I don't think "scala macro" equals "cryptic code". Even though it does, what we need to do is to find an alternative way to do code >>> generation, > not making code generation minimum. Since you agree to have 30+ operators have Builder, what >> prevents > from having all of them have
Re: Feedback request for new Java API
Maybe we can take a step back and rather look at how we would like our users and mxnet developers to use the API. Imagine like a mock or pseudo code where we write a few examples with the "perfect" Java API. After that, we think about the technical implementation details and how it can be done (I know that we already had a lot of technical discussion, it's just a method to get a second perspective). Our current discussion is around the technical limitations and possible overhead a Java API might cause, but I think it's most important that a user is actually able to use it. I think we have all been at the point where we tried to use a package and the API was so strange that we just picked an alternative. We should think a bit long term here. Java is a HUGE market with a big user base, especially in the Enterprise segment. So, theoretically speaking, even if we duplicate everything and decouple it from scala entirely, we might have a good chance that - given a good design - other people will step up and help maintaining and extending the API. If the process to change anything in that API is super cryptic, we might block future efforts because the entry barrier is too high. I think our python API is the best example. We are finally at a state where it's super easy to extend, the usability is good (especially around gluon) and it fits into the languages' idioms. Let's try to take that experience and apply it to the Java API. This might be more work initially and create redundancy, but we have seen that a good API (external as well as internal) design increases contributions and attracts more users. I won't take any side here because I'm unfamiliar with the scala side of mxnet, just wanted to add an external viewpoint. -Marco YiZhi Liu schrieb am So., 30. Sep. 2018, 05:15: > Yes agreement and disagreement stay at technical level only:) > > Back to the problem, they are unnecessary but good in terms of, > 1. Still not good for java users to write 3 nulls in a function call with 5 > or 4 args > 2. Every function call with a “tail” null for arg “out”. I would say, makes > it seems not a serious api design to our users > 3. Users have uniformed experience, nothing surprising. > > Given the reasons I listed before, I don’t see things bad for this. > > I agree we can vote. I suggest to have two votes, one for builder, one for > separating java and scala objects. > > On Sat, Sep 29, 2018 at 7:43 PM Naveen Swamy wrote: > > > Ah! we agree on something :) lets get more opinions, I am happy to go > with > > it. > > > > On Sat, Sep 29, 2018 at 10:40 PM YiZhi Liu wrote: > > > > > Also sometimes people may not be at the same page when talking about > > option > > > #2. What I insist is the builder classes for each operator. Otherwise I > > > actually more support Naveen’s approach - not to totally separate java > > and > > > scala objects. > > > > > > On Sat, Sep 29, 2018 at 7:35 PM YiZhi Liu wrote: > > > > > > > No you haven't answered my question "Since you agree to have 30+ > > > > operators have Builder, what prevents from > > > > having all of them have Builder?" > > > > On Sat, Sep 29, 2018 at 7:30 PM Naveen Swamy > > wrote: > > > > > > > > > > I think we have had enough of an debate between the two of us and I > > > have > > > > > already listed my reasons, I will stop here and see what others say > > > > given > > > > > my reasoning. > > > > > > > > > > -1 to #2) > > > > > > > > > > Also, by lecture I meant to say "I don't want to list all the > > problems > > > > > with unnecessary complications and talk about how to design > software" > > > > > > > > > > On Sat, Sep 29, 2018 at 10:15 PM YiZhi Liu > > > wrote: > > > > > > > > > > > And if we find incorrect declaration, we fix it, not simply > > assuming > > > > > > many of them also has problem and we cannot rely on them - > > otherwise > > > > > > the type-safe APIs in Scala also does not make sense. > > > > > > On Sat, Sep 29, 2018 at 7:10 PM YiZhi Liu > > > wrote: > > > > > > > > > > > > > > It also makes sense to me if we have it under namespace > NDArray, > > > not > > > > > > > creating new JavaNDArray. But again, uniform experience is > > > important. > > > > > > > > > > > > > > What I responded is your comment "keep scala macros minimum", I > > > don't > > > > > > > think "scala macro" equals "cryptic code". Even though it does, > > > what > > > > > > > we need to do is to find an alternative way to do code > > generation, > > > > not > > > > > > > making code generation minimum. > > > > > > > > > > > > > > Since you agree to have 30+ operators have Builder, what > prevents > > > > from > > > > > > > having all of them have Builder? > > > > > > > - They're auto-generated, the auto-generation "cryptic" code is > > > > anyway > > > > > > > there. And "two different paths of code" (though I don't > totally > > > > > > > agree) is anyway there. > > > > > > > - What else? 200+ classes is a very tiny increasing in file > size > > > > > > > (~3MB)
Re: Feedback request for new Java API
Yes agreement and disagreement stay at technical level only:) Back to the problem, they are unnecessary but good in terms of, 1. Still not good for java users to write 3 nulls in a function call with 5 or 4 args 2. Every function call with a “tail” null for arg “out”. I would say, makes it seems not a serious api design to our users 3. Users have uniformed experience, nothing surprising. Given the reasons I listed before, I don’t see things bad for this. I agree we can vote. I suggest to have two votes, one for builder, one for separating java and scala objects. On Sat, Sep 29, 2018 at 7:43 PM Naveen Swamy wrote: > Ah! we agree on something :) lets get more opinions, I am happy to go with > it. > > On Sat, Sep 29, 2018 at 10:40 PM YiZhi Liu wrote: > > > Also sometimes people may not be at the same page when talking about > option > > #2. What I insist is the builder classes for each operator. Otherwise I > > actually more support Naveen’s approach - not to totally separate java > and > > scala objects. > > > > On Sat, Sep 29, 2018 at 7:35 PM YiZhi Liu wrote: > > > > > No you haven't answered my question "Since you agree to have 30+ > > > operators have Builder, what prevents from > > > having all of them have Builder?" > > > On Sat, Sep 29, 2018 at 7:30 PM Naveen Swamy > wrote: > > > > > > > > I think we have had enough of an debate between the two of us and I > > have > > > > already listed my reasons, I will stop here and see what others say > > > given > > > > my reasoning. > > > > > > > > -1 to #2) > > > > > > > > Also, by lecture I meant to say "I don't want to list all the > problems > > > > with unnecessary complications and talk about how to design software" > > > > > > > > On Sat, Sep 29, 2018 at 10:15 PM YiZhi Liu > > wrote: > > > > > > > > > And if we find incorrect declaration, we fix it, not simply > assuming > > > > > many of them also has problem and we cannot rely on them - > otherwise > > > > > the type-safe APIs in Scala also does not make sense. > > > > > On Sat, Sep 29, 2018 at 7:10 PM YiZhi Liu > > wrote: > > > > > > > > > > > > It also makes sense to me if we have it under namespace NDArray, > > not > > > > > > creating new JavaNDArray. But again, uniform experience is > > important. > > > > > > > > > > > > What I responded is your comment "keep scala macros minimum", I > > don't > > > > > > think "scala macro" equals "cryptic code". Even though it does, > > what > > > > > > we need to do is to find an alternative way to do code > generation, > > > not > > > > > > making code generation minimum. > > > > > > > > > > > > Since you agree to have 30+ operators have Builder, what prevents > > > from > > > > > > having all of them have Builder? > > > > > > - They're auto-generated, the auto-generation "cryptic" code is > > > anyway > > > > > > there. And "two different paths of code" (though I don't totally > > > > > > agree) is anyway there. > > > > > > - What else? 200+ classes is a very tiny increasing in file size > > > > > > (~3MB) compare to current status. And won't have any performance > > > issue > > > > > > on modern JVM. > > > > > > > > > > > > Just remind, technical discussion is not about who gives who a > > > lecture. > > > > > > On Sat, Sep 29, 2018 at 6:41 PM Naveen Swamy > > > > wrote: > > > > > > > > > > > > > > Well, I am not sure(I don't think) we need Builder for every > API > > in > > > > > > > NDArray. For APIs that take long list of parameters, I agree to > > add > > > > > Builder. > > > > > > > Look at the API distribution based on number of arguments here: > > > > > > > > https://gist.github.com/nswamy/2dea72e514cc7bfc675f68aef9fe78bb > > > > > > > about 30 APIs have 7 or more arguments.. I agree to add > Builders > > > for > > > > > these > > > > > > > APIs not separately but to the existing Scala APIs but not > > > separately > > > > > only > > > > > > > for Java. > > > > > > > APIs sorted by number of arguments is here, take a look : > > > > > > > > https://gist.github.com/nswamy/e941cb94658b3960eec40bf00b970ac5 > > > > > > > > > > > > > > Many of the arguments i think are actually mandatory but > > > incorrectly > > > > > > > declared optional on the backend, for example look at SwapAxis > > > > > > > "def SwapAxis (data : NDArray, dim1 : Option[Int] = None, dim2 > : > > > > > > > Option[Int] = None, out : Option[NDArray] = None) : > > > NDArrayFuncReturn" > > > > > > > Why is dim1 and dim2 Optional, this is an error in the > > declaration > > > on > > > > > the > > > > > > > backend, I think there might be many of these? > > > > > > > > > > > > > > My answers to your other responses are below inline: > > > > > > > > > > > > > > On Sat, Sep 29, 2018 at 3:37 PM YiZhi Liu > > > > wrote: > > > > > > > > > > > > > > > Some of my comments inline: > > > > > > > > > > > > > > > > > Why can we not create the builder just for these APIs( > which > > we > > > > > > > > discussed), why is it necessary to add 200 Apis > > > > > > > > It is about unified
Re: Feedback request for new Java API
Ah! we agree on something :) lets get more opinions, I am happy to go with it. On Sat, Sep 29, 2018 at 10:40 PM YiZhi Liu wrote: > Also sometimes people may not be at the same page when talking about option > #2. What I insist is the builder classes for each operator. Otherwise I > actually more support Naveen’s approach - not to totally separate java and > scala objects. > > On Sat, Sep 29, 2018 at 7:35 PM YiZhi Liu wrote: > > > No you haven't answered my question "Since you agree to have 30+ > > operators have Builder, what prevents from > > having all of them have Builder?" > > On Sat, Sep 29, 2018 at 7:30 PM Naveen Swamy wrote: > > > > > > I think we have had enough of an debate between the two of us and I > have > > > already listed my reasons, I will stop here and see what others say > > given > > > my reasoning. > > > > > > -1 to #2) > > > > > > Also, by lecture I meant to say "I don't want to list all the problems > > > with unnecessary complications and talk about how to design software" > > > > > > On Sat, Sep 29, 2018 at 10:15 PM YiZhi Liu > wrote: > > > > > > > And if we find incorrect declaration, we fix it, not simply assuming > > > > many of them also has problem and we cannot rely on them - otherwise > > > > the type-safe APIs in Scala also does not make sense. > > > > On Sat, Sep 29, 2018 at 7:10 PM YiZhi Liu > wrote: > > > > > > > > > > It also makes sense to me if we have it under namespace NDArray, > not > > > > > creating new JavaNDArray. But again, uniform experience is > important. > > > > > > > > > > What I responded is your comment "keep scala macros minimum", I > don't > > > > > think "scala macro" equals "cryptic code". Even though it does, > what > > > > > we need to do is to find an alternative way to do code generation, > > not > > > > > making code generation minimum. > > > > > > > > > > Since you agree to have 30+ operators have Builder, what prevents > > from > > > > > having all of them have Builder? > > > > > - They're auto-generated, the auto-generation "cryptic" code is > > anyway > > > > > there. And "two different paths of code" (though I don't totally > > > > > agree) is anyway there. > > > > > - What else? 200+ classes is a very tiny increasing in file size > > > > > (~3MB) compare to current status. And won't have any performance > > issue > > > > > on modern JVM. > > > > > > > > > > Just remind, technical discussion is not about who gives who a > > lecture. > > > > > On Sat, Sep 29, 2018 at 6:41 PM Naveen Swamy > > wrote: > > > > > > > > > > > > Well, I am not sure(I don't think) we need Builder for every API > in > > > > > > NDArray. For APIs that take long list of parameters, I agree to > add > > > > Builder. > > > > > > Look at the API distribution based on number of arguments here: > > > > > > https://gist.github.com/nswamy/2dea72e514cc7bfc675f68aef9fe78bb > > > > > > about 30 APIs have 7 or more arguments.. I agree to add Builders > > for > > > > these > > > > > > APIs not separately but to the existing Scala APIs but not > > separately > > > > only > > > > > > for Java. > > > > > > APIs sorted by number of arguments is here, take a look : > > > > > > https://gist.github.com/nswamy/e941cb94658b3960eec40bf00b970ac5 > > > > > > > > > > > > Many of the arguments i think are actually mandatory but > > incorrectly > > > > > > declared optional on the backend, for example look at SwapAxis > > > > > > "def SwapAxis (data : NDArray, dim1 : Option[Int] = None, dim2 : > > > > > > Option[Int] = None, out : Option[NDArray] = None) : > > NDArrayFuncReturn" > > > > > > Why is dim1 and dim2 Optional, this is an error in the > declaration > > on > > > > the > > > > > > backend, I think there might be many of these? > > > > > > > > > > > > My answers to your other responses are below inline: > > > > > > > > > > > > On Sat, Sep 29, 2018 at 3:37 PM YiZhi Liu > > wrote: > > > > > > > > > > > > > Some of my comments inline: > > > > > > > > > > > > > > > Why can we not create the builder just for these APIs( which > we > > > > > > > discussed), why is it necessary to add 200 Apis > > > > > > > It is about unified user-experience. And we get rid of annoying > > extra > > > > > > > "out=null" in every operator. > > > > > > > > > > > > > > > Are you suggesting to create builder for each and every API? > > > > > > > Only for those are necessary. For NDArray.XXX, yes. > > > > > > > > > > > > > I think this is a ridiculous list of Builders, I think we can > keep > > the > > > > > > 'out' parameter > > > > > > > > > > > > > 1) The NDArray APIs in question are not following functional > > style of > > > > > > > programming, in fact they are just static methods defined on an > > > > > > > NDArray object - so Scala users are not losing much by using > > null in > > > > > > > place of None. > > > > > > > You can create a implicit to maintain backward compatibility > > > > > > > - I doubt implicit can work in such case from None -> null. > > > > > > > > > > > > > > > > > > > It is
Re: Feedback request for new Java API
IMO, its unnecessary. this is only my opinion and you are free to disagree.
With more opinions one or some of us will have to commit to the majority of
the opinion.
On Sat, Sep 29, 2018 at 10:35 PM YiZhi Liu wrote:
> No you haven't answered my question "Since you agree to have 30+
> operators have Builder, what prevents from
> having all of them have Builder?"
> On Sat, Sep 29, 2018 at 7:30 PM Naveen Swamy wrote:
> >
> > I think we have had enough of an debate between the two of us and I have
> > already listed my reasons, I will stop here and see what others say
> given
> > my reasoning.
> >
> > -1 to #2)
> >
> > Also, by lecture I meant to say "I don't want to list all the problems
> > with unnecessary complications and talk about how to design software"
> >
> > On Sat, Sep 29, 2018 at 10:15 PM YiZhi Liu wrote:
> >
> > > And if we find incorrect declaration, we fix it, not simply assuming
> > > many of them also has problem and we cannot rely on them - otherwise
> > > the type-safe APIs in Scala also does not make sense.
> > > On Sat, Sep 29, 2018 at 7:10 PM YiZhi Liu wrote:
> > > >
> > > > It also makes sense to me if we have it under namespace NDArray, not
> > > > creating new JavaNDArray. But again, uniform experience is important.
> > > >
> > > > What I responded is your comment "keep scala macros minimum", I don't
> > > > think "scala macro" equals "cryptic code". Even though it does, what
> > > > we need to do is to find an alternative way to do code generation,
> not
> > > > making code generation minimum.
> > > >
> > > > Since you agree to have 30+ operators have Builder, what prevents
> from
> > > > having all of them have Builder?
> > > > - They're auto-generated, the auto-generation "cryptic" code is
> anyway
> > > > there. And "two different paths of code" (though I don't totally
> > > > agree) is anyway there.
> > > > - What else? 200+ classes is a very tiny increasing in file size
> > > > (~3MB) compare to current status. And won't have any performance
> issue
> > > > on modern JVM.
> > > >
> > > > Just remind, technical discussion is not about who gives who a
> lecture.
> > > > On Sat, Sep 29, 2018 at 6:41 PM Naveen Swamy
> wrote:
> > > > >
> > > > > Well, I am not sure(I don't think) we need Builder for every API in
> > > > > NDArray. For APIs that take long list of parameters, I agree to add
> > > Builder.
> > > > > Look at the API distribution based on number of arguments here:
> > > > > https://gist.github.com/nswamy/2dea72e514cc7bfc675f68aef9fe78bb
> > > > > about 30 APIs have 7 or more arguments.. I agree to add Builders
> for
> > > these
> > > > > APIs not separately but to the existing Scala APIs but not
> separately
> > > only
> > > > > for Java.
> > > > > APIs sorted by number of arguments is here, take a look :
> > > > > https://gist.github.com/nswamy/e941cb94658b3960eec40bf00b970ac5
> > > > >
> > > > > Many of the arguments i think are actually mandatory but
> incorrectly
> > > > > declared optional on the backend, for example look at SwapAxis
> > > > > "def SwapAxis (data : NDArray, dim1 : Option[Int] = None, dim2 :
> > > > > Option[Int] = None, out : Option[NDArray] = None) :
> NDArrayFuncReturn"
> > > > > Why is dim1 and dim2 Optional, this is an error in the declaration
> on
> > > the
> > > > > backend, I think there might be many of these?
> > > > >
> > > > > My answers to your other responses are below inline:
> > > > >
> > > > > On Sat, Sep 29, 2018 at 3:37 PM YiZhi Liu
> wrote:
> > > > >
> > > > > > Some of my comments inline:
> > > > > >
> > > > > > > Why can we not create the builder just for these APIs( which we
> > > > > > discussed), why is it necessary to add 200 Apis
> > > > > > It is about unified user-experience. And we get rid of annoying
> extra
> > > > > > "out=null" in every operator.
> > > > > >
> > > > > > > Are you suggesting to create builder for each and every API?
> > > > > > Only for those are necessary. For NDArray.XXX, yes.
> > > > > >
> > > > > I think this is a ridiculous list of Builders, I think we can keep
> the
> > > > > 'out' parameter
> > > > >
> > > > > > 1) The NDArray APIs in question are not following functional
> style of
> > > > > > programming, in fact they are just static methods defined on an
> > > > > > NDArray object - so Scala users are not losing much by using
> null in
> > > > > > place of None.
> > > > > > You can create a implicit to maintain backward compatibility
> > > > > > - I doubt implicit can work in such case from None -> null.
> > > > > >
> > > > >
> > > > > It is just writing getOrElse in your implicit, so it will work.
> > > > > scala> implicit def optionStringToString(a: Option[String]):
> String = {
> > > > > | a.getOrElse(null)
> > > > > | }
> > > > >
> > > > > 2) It is adding 220+ APIs(I understand it is generated) for NDArray
> > > alone
> > > > > > - As I explained how it can improve user experiences
> > > > > >
> > > > > I don't think we need to write builders for 221 APIs we have, may
>
Re: Feedback request for new Java API
I am not sure what makes you think that I am suggesting we should not fix
it. I am pointing out that many of those are incorrectly optional so don't
take that into consideration to make a decision whether we need builders
for all.
On Sat, Sep 29, 2018 at 10:15 PM YiZhi Liu wrote:
> And if we find incorrect declaration, we fix it, not simply assuming
> many of them also has problem and we cannot rely on them - otherwise
> the type-safe APIs in Scala also does not make sense.
> On Sat, Sep 29, 2018 at 7:10 PM YiZhi Liu wrote:
> >
> > It also makes sense to me if we have it under namespace NDArray, not
> > creating new JavaNDArray. But again, uniform experience is important.
> >
> > What I responded is your comment "keep scala macros minimum", I don't
> > think "scala macro" equals "cryptic code". Even though it does, what
> > we need to do is to find an alternative way to do code generation, not
> > making code generation minimum.
> >
> > Since you agree to have 30+ operators have Builder, what prevents from
> > having all of them have Builder?
> > - They're auto-generated, the auto-generation "cryptic" code is anyway
> > there. And "two different paths of code" (though I don't totally
> > agree) is anyway there.
> > - What else? 200+ classes is a very tiny increasing in file size
> > (~3MB) compare to current status. And won't have any performance issue
> > on modern JVM.
> >
> > Just remind, technical discussion is not about who gives who a lecture.
> > On Sat, Sep 29, 2018 at 6:41 PM Naveen Swamy wrote:
> > >
> > > Well, I am not sure(I don't think) we need Builder for every API in
> > > NDArray. For APIs that take long list of parameters, I agree to add
> Builder.
> > > Look at the API distribution based on number of arguments here:
> > > https://gist.github.com/nswamy/2dea72e514cc7bfc675f68aef9fe78bb
> > > about 30 APIs have 7 or more arguments.. I agree to add Builders for
> these
> > > APIs not separately but to the existing Scala APIs but not separately
> only
> > > for Java.
> > > APIs sorted by number of arguments is here, take a look :
> > > https://gist.github.com/nswamy/e941cb94658b3960eec40bf00b970ac5
> > >
> > > Many of the arguments i think are actually mandatory but incorrectly
> > > declared optional on the backend, for example look at SwapAxis
> > > "def SwapAxis (data : NDArray, dim1 : Option[Int] = None, dim2 :
> > > Option[Int] = None, out : Option[NDArray] = None) : NDArrayFuncReturn"
> > > Why is dim1 and dim2 Optional, this is an error in the declaration on
> the
> > > backend, I think there might be many of these?
> > >
> > > My answers to your other responses are below inline:
> > >
> > > On Sat, Sep 29, 2018 at 3:37 PM YiZhi Liu wrote:
> > >
> > > > Some of my comments inline:
> > > >
> > > > > Why can we not create the builder just for these APIs( which we
> > > > discussed), why is it necessary to add 200 Apis
> > > > It is about unified user-experience. And we get rid of annoying extra
> > > > "out=null" in every operator.
> > > >
> > > > > Are you suggesting to create builder for each and every API?
> > > > Only for those are necessary. For NDArray.XXX, yes.
> > > >
> > > I think this is a ridiculous list of Builders, I think we can keep the
> > > 'out' parameter
> > >
> > > > 1) The NDArray APIs in question are not following functional style of
> > > > programming, in fact they are just static methods defined on an
> > > > NDArray object - so Scala users are not losing much by using null in
> > > > place of None.
> > > > You can create a implicit to maintain backward compatibility
> > > > - I doubt implicit can work in such case from None -> null.
> > > >
> > >
> > > It is just writing getOrElse in your implicit, so it will work.
> > > scala> implicit def optionStringToString(a: Option[String]): String = {
> > > | a.getOrElse(null)
> > > | }
> > >
> > > 2) It is adding 220+ APIs(I understand it is generated) for NDArray
> alone
> > > > - As I explained how it can improve user experiences
> > > >
> > > I don't think we need to write builders for 221 APIs we have, may be
> for 30
> > > or so. Uniform experience is good goal but it also has to be practical
> and
> > > make sense.
> > >
> > > 3) this is adding another 100s of APIs unnecessarily, we are starting
> with
> > > > NDArray but we can't stop there, we will have to do this for Symbol,
> > > > Executor, Iterators, etc., .
> > > > - This is a good point, actually I prefer not to make JavaExecutor,
> > > > JavaIterators
> > > >
> > > What I was aiming is also users have the same experience across
> Interfaces
> > > - now you are forgoing uniform experience, so like you said its all
> > > trade-off and a good trade-off doesn't cause too much overhead/
> > >
> > >
> > > > 4) I don't want to be fixing bugs and maintaining code in 2 places.
> > > > - Type-safe parsing is shared. I think Qing is more qualified to
> comment.
> > >
> > > It creates two different paths of code for Scala and Java - how is it
> going
Re: Feedback request for new Java API
Also sometimes people may not be at the same page when talking about option
#2. What I insist is the builder classes for each operator. Otherwise I
actually more support Naveen’s approach - not to totally separate java and
scala objects.
On Sat, Sep 29, 2018 at 7:35 PM YiZhi Liu wrote:
> No you haven't answered my question "Since you agree to have 30+
> operators have Builder, what prevents from
> having all of them have Builder?"
> On Sat, Sep 29, 2018 at 7:30 PM Naveen Swamy wrote:
> >
> > I think we have had enough of an debate between the two of us and I have
> > already listed my reasons, I will stop here and see what others say
> given
> > my reasoning.
> >
> > -1 to #2)
> >
> > Also, by lecture I meant to say "I don't want to list all the problems
> > with unnecessary complications and talk about how to design software"
> >
> > On Sat, Sep 29, 2018 at 10:15 PM YiZhi Liu wrote:
> >
> > > And if we find incorrect declaration, we fix it, not simply assuming
> > > many of them also has problem and we cannot rely on them - otherwise
> > > the type-safe APIs in Scala also does not make sense.
> > > On Sat, Sep 29, 2018 at 7:10 PM YiZhi Liu wrote:
> > > >
> > > > It also makes sense to me if we have it under namespace NDArray, not
> > > > creating new JavaNDArray. But again, uniform experience is important.
> > > >
> > > > What I responded is your comment "keep scala macros minimum", I don't
> > > > think "scala macro" equals "cryptic code". Even though it does, what
> > > > we need to do is to find an alternative way to do code generation,
> not
> > > > making code generation minimum.
> > > >
> > > > Since you agree to have 30+ operators have Builder, what prevents
> from
> > > > having all of them have Builder?
> > > > - They're auto-generated, the auto-generation "cryptic" code is
> anyway
> > > > there. And "two different paths of code" (though I don't totally
> > > > agree) is anyway there.
> > > > - What else? 200+ classes is a very tiny increasing in file size
> > > > (~3MB) compare to current status. And won't have any performance
> issue
> > > > on modern JVM.
> > > >
> > > > Just remind, technical discussion is not about who gives who a
> lecture.
> > > > On Sat, Sep 29, 2018 at 6:41 PM Naveen Swamy
> wrote:
> > > > >
> > > > > Well, I am not sure(I don't think) we need Builder for every API in
> > > > > NDArray. For APIs that take long list of parameters, I agree to add
> > > Builder.
> > > > > Look at the API distribution based on number of arguments here:
> > > > > https://gist.github.com/nswamy/2dea72e514cc7bfc675f68aef9fe78bb
> > > > > about 30 APIs have 7 or more arguments.. I agree to add Builders
> for
> > > these
> > > > > APIs not separately but to the existing Scala APIs but not
> separately
> > > only
> > > > > for Java.
> > > > > APIs sorted by number of arguments is here, take a look :
> > > > > https://gist.github.com/nswamy/e941cb94658b3960eec40bf00b970ac5
> > > > >
> > > > > Many of the arguments i think are actually mandatory but
> incorrectly
> > > > > declared optional on the backend, for example look at SwapAxis
> > > > > "def SwapAxis (data : NDArray, dim1 : Option[Int] = None, dim2 :
> > > > > Option[Int] = None, out : Option[NDArray] = None) :
> NDArrayFuncReturn"
> > > > > Why is dim1 and dim2 Optional, this is an error in the declaration
> on
> > > the
> > > > > backend, I think there might be many of these?
> > > > >
> > > > > My answers to your other responses are below inline:
> > > > >
> > > > > On Sat, Sep 29, 2018 at 3:37 PM YiZhi Liu
> wrote:
> > > > >
> > > > > > Some of my comments inline:
> > > > > >
> > > > > > > Why can we not create the builder just for these APIs( which we
> > > > > > discussed), why is it necessary to add 200 Apis
> > > > > > It is about unified user-experience. And we get rid of annoying
> extra
> > > > > > "out=null" in every operator.
> > > > > >
> > > > > > > Are you suggesting to create builder for each and every API?
> > > > > > Only for those are necessary. For NDArray.XXX, yes.
> > > > > >
> > > > > I think this is a ridiculous list of Builders, I think we can keep
> the
> > > > > 'out' parameter
> > > > >
> > > > > > 1) The NDArray APIs in question are not following functional
> style of
> > > > > > programming, in fact they are just static methods defined on an
> > > > > > NDArray object - so Scala users are not losing much by using
> null in
> > > > > > place of None.
> > > > > > You can create a implicit to maintain backward compatibility
> > > > > > - I doubt implicit can work in such case from None -> null.
> > > > > >
> > > > >
> > > > > It is just writing getOrElse in your implicit, so it will work.
> > > > > scala> implicit def optionStringToString(a: Option[String]):
> String = {
> > > > > | a.getOrElse(null)
> > > > > | }
> > > > >
> > > > > 2) It is adding 220+ APIs(I understand it is generated) for NDArray
> > > alone
> > > > > > - As I explained how it can improve user experiences
> > > > > >
> > >
Re: Feedback request for new Java API
I think we have had enough of an debate between the two of us and I have
already listed my reasons, I will stop here and see what others say given
my reasoning.
-1 to #2)
Also, by lecture I meant to say "I don't want to list all the problems
with unnecessary complications and talk about how to design software"
On Sat, Sep 29, 2018 at 10:15 PM YiZhi Liu wrote:
> And if we find incorrect declaration, we fix it, not simply assuming
> many of them also has problem and we cannot rely on them - otherwise
> the type-safe APIs in Scala also does not make sense.
> On Sat, Sep 29, 2018 at 7:10 PM YiZhi Liu wrote:
> >
> > It also makes sense to me if we have it under namespace NDArray, not
> > creating new JavaNDArray. But again, uniform experience is important.
> >
> > What I responded is your comment "keep scala macros minimum", I don't
> > think "scala macro" equals "cryptic code". Even though it does, what
> > we need to do is to find an alternative way to do code generation, not
> > making code generation minimum.
> >
> > Since you agree to have 30+ operators have Builder, what prevents from
> > having all of them have Builder?
> > - They're auto-generated, the auto-generation "cryptic" code is anyway
> > there. And "two different paths of code" (though I don't totally
> > agree) is anyway there.
> > - What else? 200+ classes is a very tiny increasing in file size
> > (~3MB) compare to current status. And won't have any performance issue
> > on modern JVM.
> >
> > Just remind, technical discussion is not about who gives who a lecture.
> > On Sat, Sep 29, 2018 at 6:41 PM Naveen Swamy wrote:
> > >
> > > Well, I am not sure(I don't think) we need Builder for every API in
> > > NDArray. For APIs that take long list of parameters, I agree to add
> Builder.
> > > Look at the API distribution based on number of arguments here:
> > > https://gist.github.com/nswamy/2dea72e514cc7bfc675f68aef9fe78bb
> > > about 30 APIs have 7 or more arguments.. I agree to add Builders for
> these
> > > APIs not separately but to the existing Scala APIs but not separately
> only
> > > for Java.
> > > APIs sorted by number of arguments is here, take a look :
> > > https://gist.github.com/nswamy/e941cb94658b3960eec40bf00b970ac5
> > >
> > > Many of the arguments i think are actually mandatory but incorrectly
> > > declared optional on the backend, for example look at SwapAxis
> > > "def SwapAxis (data : NDArray, dim1 : Option[Int] = None, dim2 :
> > > Option[Int] = None, out : Option[NDArray] = None) : NDArrayFuncReturn"
> > > Why is dim1 and dim2 Optional, this is an error in the declaration on
> the
> > > backend, I think there might be many of these?
> > >
> > > My answers to your other responses are below inline:
> > >
> > > On Sat, Sep 29, 2018 at 3:37 PM YiZhi Liu wrote:
> > >
> > > > Some of my comments inline:
> > > >
> > > > > Why can we not create the builder just for these APIs( which we
> > > > discussed), why is it necessary to add 200 Apis
> > > > It is about unified user-experience. And we get rid of annoying extra
> > > > "out=null" in every operator.
> > > >
> > > > > Are you suggesting to create builder for each and every API?
> > > > Only for those are necessary. For NDArray.XXX, yes.
> > > >
> > > I think this is a ridiculous list of Builders, I think we can keep the
> > > 'out' parameter
> > >
> > > > 1) The NDArray APIs in question are not following functional style of
> > > > programming, in fact they are just static methods defined on an
> > > > NDArray object - so Scala users are not losing much by using null in
> > > > place of None.
> > > > You can create a implicit to maintain backward compatibility
> > > > - I doubt implicit can work in such case from None -> null.
> > > >
> > >
> > > It is just writing getOrElse in your implicit, so it will work.
> > > scala> implicit def optionStringToString(a: Option[String]): String = {
> > > | a.getOrElse(null)
> > > | }
> > >
> > > 2) It is adding 220+ APIs(I understand it is generated) for NDArray
> alone
> > > > - As I explained how it can improve user experiences
> > > >
> > > I don't think we need to write builders for 221 APIs we have, may be
> for 30
> > > or so. Uniform experience is good goal but it also has to be practical
> and
> > > make sense.
> > >
> > > 3) this is adding another 100s of APIs unnecessarily, we are starting
> with
> > > > NDArray but we can't stop there, we will have to do this for Symbol,
> > > > Executor, Iterators, etc., .
> > > > - This is a good point, actually I prefer not to make JavaExecutor,
> > > > JavaIterators
> > > >
> > > What I was aiming is also users have the same experience across
> Interfaces
> > > - now you are forgoing uniform experience, so like you said its all
> > > trade-off and a good trade-off doesn't cause too much overhead/
> > >
> > >
> > > > 4) I don't want to be fixing bugs and maintaining code in 2 places.
> > > > - Type-safe parsing is shared. I think Qing is more qualified to
> comment.
> >
Re: Feedback request for new Java API
And if we find incorrect declaration, we fix it, not simply assuming
many of them also has problem and we cannot rely on them - otherwise
the type-safe APIs in Scala also does not make sense.
On Sat, Sep 29, 2018 at 7:10 PM YiZhi Liu wrote:
>
> It also makes sense to me if we have it under namespace NDArray, not
> creating new JavaNDArray. But again, uniform experience is important.
>
> What I responded is your comment "keep scala macros minimum", I don't
> think "scala macro" equals "cryptic code". Even though it does, what
> we need to do is to find an alternative way to do code generation, not
> making code generation minimum.
>
> Since you agree to have 30+ operators have Builder, what prevents from
> having all of them have Builder?
> - They're auto-generated, the auto-generation "cryptic" code is anyway
> there. And "two different paths of code" (though I don't totally
> agree) is anyway there.
> - What else? 200+ classes is a very tiny increasing in file size
> (~3MB) compare to current status. And won't have any performance issue
> on modern JVM.
>
> Just remind, technical discussion is not about who gives who a lecture.
> On Sat, Sep 29, 2018 at 6:41 PM Naveen Swamy wrote:
> >
> > Well, I am not sure(I don't think) we need Builder for every API in
> > NDArray. For APIs that take long list of parameters, I agree to add Builder.
> > Look at the API distribution based on number of arguments here:
> > https://gist.github.com/nswamy/2dea72e514cc7bfc675f68aef9fe78bb
> > about 30 APIs have 7 or more arguments.. I agree to add Builders for these
> > APIs not separately but to the existing Scala APIs but not separately only
> > for Java.
> > APIs sorted by number of arguments is here, take a look :
> > https://gist.github.com/nswamy/e941cb94658b3960eec40bf00b970ac5
> >
> > Many of the arguments i think are actually mandatory but incorrectly
> > declared optional on the backend, for example look at SwapAxis
> > "def SwapAxis (data : NDArray, dim1 : Option[Int] = None, dim2 :
> > Option[Int] = None, out : Option[NDArray] = None) : NDArrayFuncReturn"
> > Why is dim1 and dim2 Optional, this is an error in the declaration on the
> > backend, I think there might be many of these?
> >
> > My answers to your other responses are below inline:
> >
> > On Sat, Sep 29, 2018 at 3:37 PM YiZhi Liu wrote:
> >
> > > Some of my comments inline:
> > >
> > > > Why can we not create the builder just for these APIs( which we
> > > discussed), why is it necessary to add 200 Apis
> > > It is about unified user-experience. And we get rid of annoying extra
> > > "out=null" in every operator.
> > >
> > > > Are you suggesting to create builder for each and every API?
> > > Only for those are necessary. For NDArray.XXX, yes.
> > >
> > I think this is a ridiculous list of Builders, I think we can keep the
> > 'out' parameter
> >
> > > 1) The NDArray APIs in question are not following functional style of
> > > programming, in fact they are just static methods defined on an
> > > NDArray object - so Scala users are not losing much by using null in
> > > place of None.
> > > You can create a implicit to maintain backward compatibility
> > > - I doubt implicit can work in such case from None -> null.
> > >
> >
> > It is just writing getOrElse in your implicit, so it will work.
> > scala> implicit def optionStringToString(a: Option[String]): String = {
> > | a.getOrElse(null)
> > | }
> >
> > 2) It is adding 220+ APIs(I understand it is generated) for NDArray alone
> > > - As I explained how it can improve user experiences
> > >
> > I don't think we need to write builders for 221 APIs we have, may be for 30
> > or so. Uniform experience is good goal but it also has to be practical and
> > make sense.
> >
> > 3) this is adding another 100s of APIs unnecessarily, we are starting with
> > > NDArray but we can't stop there, we will have to do this for Symbol,
> > > Executor, Iterators, etc., .
> > > - This is a good point, actually I prefer not to make JavaExecutor,
> > > JavaIterators
> > >
> > What I was aiming is also users have the same experience across Interfaces
> > - now you are forgoing uniform experience, so like you said its all
> > trade-off and a good trade-off doesn't cause too much overhead/
> >
> >
> > > 4) I don't want to be fixing bugs and maintaining code in 2 places.
> > > - Type-safe parsing is shared. I think Qing is more qualified to comment.
> >
> > It creates two different paths of code for Scala and Java - how is it going
> > to be shared. I am afraid we are going to make it more complicated than
> > necessary by duplicating code.
> >
> > >
> >
> > 5) I want the cryptic code(# scala macros) to a minimum.
> > > - MXNet decides to do operator generation in frontend bindings. It's
> > > the developers' responsibility to understand the techniques they are
> > > using. Maybe not a so proper analogy - "I don't know RL / RL is hard
> > > to tune / ..." is not a reason for "I want to keep RL implementation
> > >
Re: Feedback request for new Java API
Well, I am not sure(I don't think) we need Builder for every API in
NDArray. For APIs that take long list of parameters, I agree to add Builder.
Look at the API distribution based on number of arguments here:
https://gist.github.com/nswamy/2dea72e514cc7bfc675f68aef9fe78bb
about 30 APIs have 7 or more arguments.. I agree to add Builders for these
APIs not separately but to the existing Scala APIs but not separately only
for Java.
APIs sorted by number of arguments is here, take a look :
https://gist.github.com/nswamy/e941cb94658b3960eec40bf00b970ac5
Many of the arguments i think are actually mandatory but incorrectly
declared optional on the backend, for example look at SwapAxis
"def SwapAxis (data : NDArray, dim1 : Option[Int] = None, dim2 :
Option[Int] = None, out : Option[NDArray] = None) : NDArrayFuncReturn"
Why is dim1 and dim2 Optional, this is an error in the declaration on the
backend, I think there might be many of these?
My answers to your other responses are below inline:
On Sat, Sep 29, 2018 at 3:37 PM YiZhi Liu wrote:
> Some of my comments inline:
>
> > Why can we not create the builder just for these APIs( which we
> discussed), why is it necessary to add 200 Apis
> It is about unified user-experience. And we get rid of annoying extra
> "out=null" in every operator.
>
> > Are you suggesting to create builder for each and every API?
> Only for those are necessary. For NDArray.XXX, yes.
>
I think this is a ridiculous list of Builders, I think we can keep the
'out' parameter
> 1) The NDArray APIs in question are not following functional style of
> programming, in fact they are just static methods defined on an
> NDArray object - so Scala users are not losing much by using null in
> place of None.
> You can create a implicit to maintain backward compatibility
> - I doubt implicit can work in such case from None -> null.
>
It is just writing getOrElse in your implicit, so it will work.
scala> implicit def optionStringToString(a: Option[String]): String = {
| a.getOrElse(null)
| }
2) It is adding 220+ APIs(I understand it is generated) for NDArray alone
> - As I explained how it can improve user experiences
>
I don't think we need to write builders for 221 APIs we have, may be for 30
or so. Uniform experience is good goal but it also has to be practical and
make sense.
3) this is adding another 100s of APIs unnecessarily, we are starting with
> NDArray but we can't stop there, we will have to do this for Symbol,
> Executor, Iterators, etc., .
> - This is a good point, actually I prefer not to make JavaExecutor,
> JavaIterators
>
What I was aiming is also users have the same experience across Interfaces
- now you are forgoing uniform experience, so like you said its all
trade-off and a good trade-off doesn't cause too much overhead/
> 4) I don't want to be fixing bugs and maintaining code in 2 places.
> - Type-safe parsing is shared. I think Qing is more qualified to comment.
It creates two different paths of code for Scala and Java - how is it going
to be shared. I am afraid we are going to make it more complicated than
necessary by duplicating code.
>
5) I want the cryptic code(# scala macros) to a minimum.
> - MXNet decides to do operator generation in frontend bindings. It's
> the developers' responsibility to understand the techniques they are
> using. Maybe not a so proper analogy - "I don't know RL / RL is hard
> to tune / ..." is not a reason for "I want to keep RL implementation
> in MXNet as a small part as possible"
>
> Now, this is a response I don't like. I don't know where you were going
with your analogy but know that it sounds condescending - I am going to
ignore(assuming good intentions) that and explain what I mean.
I here is I the developer/user who deploys MXNet code in production and
have to deal with the aftermath myself not you(MXNet developers).
>From your comment it occurs to me you probably have never been on
pager-duty. I have been on pager-duty both for the code I wrote and those
that was written by others and thrown over the fence.
If you get woken up by a beep at the middle of the night, that is not the
time to prove your intelligence. Its time to mitigate the issue asap for
that your code needs to be easy to follow, should follow well defined
patterns, etc., -- i don't need to give you a lecture.
IMHO It is extremely important for frameworks like Apache MXNet which are
used by others for their production to keep code simple and *cryptic code
to a minimum* and yes you(we - MXNet developers) are not answering the
beepers when your(MXNet) users deploy their code in their production so
make their life simple.
6) increased compilation time & bad developer experience - the time to
> compile has gone up quite a bit since we added the APIs last release on my
> 3 year old laptop already.. I think adding 400+ APIs unnecessarily would
> significantly increase build time and bad developer experience
> - I don't think increasing such a bit compilation time
Re: Feedback request for new Java API
Some of my comments inline: > Why can we not create the builder just for these APIs( which we discussed), > why is it necessary to add 200 Apis It is about unified user-experience. And we get rid of annoying extra "out=null" in every operator. > Are you suggesting to create builder for each and every API? Only for those are necessary. For NDArray.XXX, yes. 1) The NDArray APIs in question are not following functional style of programming, in fact they are just static methods defined on an NDArray object - so Scala users are not losing much by using null in place of None. You can create a implicit to maintain backward compatibility - I doubt implicit can work in such case from None -> null. 2) It is adding 220+ APIs(I understand it is generated) for NDArray alone - As I explained how it can improve user experiences 3) this is adding another 100s of APIs unnecessarily, we are starting with NDArray but we can't stop there, we will have to do this for Symbol, Executor, Iterators, etc., . - This is a good point, actually I prefer not to make JavaExecutor, JavaIterators 4) I don't want to be fixing bugs and maintaining code in 2 places. - Type-safe parsing is shared. I think Qing is more qualified to comment. 5) I want the cryptic code(# scala macros) to a minimum. - MXNet decides to do operator generation in frontend bindings. It's the developers' responsibility to understand the techniques they are using. Maybe not a so proper analogy - "I don't know RL / RL is hard to tune / ..." is not a reason for "I want to keep RL implementation in MXNet as a small part as possible" 6) increased compilation time & bad developer experience - the time to compile has gone up quite a bit since we added the APIs last release on my 3 year old laptop already.. I think adding 400+ APIs unnecessarily would significantly increase build time and bad developer experience - I don't think increasing such a bit compilation time is a problem compared to bad user experience. 7) I want to keep the core of the framework to be in Scala - because it allows you to write concise code - Yes it has a bit of learning curve, not everyone needs to know. I would rather invest in solidifying the Scala APIs and add more features in Scala(RNN, Support GluonHybridizedBlock...there is quite bit of work ) - do you want to rewrite everything in Scala and Java. - I agree with "don't rewrite everything in Scala and Java", IMO JavaNDArray is the only one good to have. JShape, JContext, etc. are not so necessary. 8) Also, the discussion is not creating NDArray class for Java, just generate certain APIs to cater for Java incompatibility. - Yes I agree it's about "generate certain APIs to cater for Java incompatibility", though I think NDArray.api.XXX does not meet Java users' demands. On Sat, Sep 29, 2018 at 12:05 PM Naveen Swamy wrote: > > I know it is about trade-off. I am suggesting a trade-off , how many apis do > we have that takes too many parameters ? > From what I recall its around 20. Why can we not create the builder just for > these APIs( which we discussed), why is it necessary to add 200 Apis ? > Are you suggesting to create builder for each and every API? > > I disagree with your opinion that they are not important and would like to > hear from others. > > I am curious to see how the #2 looks like compared to #1 > Andrew/Qing, can you paste the generated Apis that you have for both Scala > and Java in a gist please. > > > On Sep 29, 2018, at 2:41 PM, YiZhi Liu wrote: > > > > Naveen, software designing is all about tradeoff, every feature we > > introduce causes more compiling time, more efforts to maintain, etc. > > > > The main difference is. > > > > Option #1: Java users do > > NDArray.BatchNorm(data, gamma, beta, null, null, null, null, null, > > null, null, null, null, null, null); > > (and because every operator has an argument "out", users need to add > > an extra "null" to the function call almost every time.) > > > > Option #2, Java users do > > JavaNDArray.BatchNorm(data).setGamma(gamma).setBeta(beta).invoke(); > > > > I don't think any of the reasons you listed is so important as the > > benefit above we got from option #2. > >> On Sat, Sep 29, 2018 at 8:24 AM Naveen Swamy wrote: > >> > >> Java APIs are not like Clojure - The current proposal is only to build a > >> few thin wrappers for Inference. > >> > >> To better represent the two cases and this discussion in particular, here > >> is an example API > >> > >> 1) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out > >> : Option[NDArray] = None) : org.apache.mxnet.NDArrayFuncReturn > >> or > >> 2) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out > >> : NDArray) : org.apache.mxnet.NDArrayFuncReturn > >> > >> The discussion is should we add(generate) 200+ APIs to make it Java > >> compatible, ie., remove the Option class and the None default value which > >> Java does not understand from Option 1) > >> > >> my suggestion was to
Re: Feedback request for new Java API
I know it is about trade-off. I am suggesting a trade-off , how many apis do we have that takes too many parameters ? From what I recall its around 20. Why can we not create the builder just for these APIs( which we discussed), why is it necessary to add 200 Apis ? Are you suggesting to create builder for each and every API? I disagree with your opinion that they are not important and would like to hear from others. I am curious to see how the #2 looks like compared to #1 Andrew/Qing, can you paste the generated Apis that you have for both Scala and Java in a gist please. > On Sep 29, 2018, at 2:41 PM, YiZhi Liu wrote: > > Naveen, software designing is all about tradeoff, every feature we > introduce causes more compiling time, more efforts to maintain, etc. > > The main difference is. > > Option #1: Java users do > NDArray.BatchNorm(data, gamma, beta, null, null, null, null, null, > null, null, null, null, null, null); > (and because every operator has an argument "out", users need to add > an extra "null" to the function call almost every time.) > > Option #2, Java users do > JavaNDArray.BatchNorm(data).setGamma(gamma).setBeta(beta).invoke(); > > I don't think any of the reasons you listed is so important as the > benefit above we got from option #2. >> On Sat, Sep 29, 2018 at 8:24 AM Naveen Swamy wrote: >> >> Java APIs are not like Clojure - The current proposal is only to build a >> few thin wrappers for Inference. >> >> To better represent the two cases and this discussion in particular, here >> is an example API >> >> 1) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out >> : Option[NDArray] = None) : org.apache.mxnet.NDArrayFuncReturn >> or >> 2) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out >> : NDArray) : org.apache.mxnet.NDArrayFuncReturn >> >> The discussion is should we add(generate) 200+ APIs to make it Java >> compatible, ie., remove the Option class and the None default value which >> Java does not understand from Option 1) >> >> my suggestion was to remove the Option class and create a implicit for >> backward compatibility and use null instead of None, Andrew and I disagreed >> on this, so I suggested to raise a discussion on dev@ to get more opinions >> and one of us will disagree and commit. Thanks for raising it :) >> >> | * def Activation (data : org.apache.mxnet.NDArray, act_type : String, out >> : NDArray = null) : org.apache.mxnet.NDArrayFuncReturn | >> -- >> >> 1) It is not true that Scala users will lose *default/optional* arguments - >> if we followed the above, they will use null or None, though I do not like >> using nulls, this is a fine compromise. >> To keep backward compatibility we can create a implicit to convert >> Option.None to nulls and Option.Some-> Option.get(), so you are not going >> to break users who might have been using the APIs that were released in >> 1.3. The current incompatibility is only this w.r.t. NDArrays. >> >> 2) Now about the Scala Macros - they are not simple to read or use, When I >> and Qing started working on the #Scala Macros to improve the APIs, it took >> us a good amount of time to get a hang of it. I don't want to add >> additional code when not necessary. >> >> My suggestion and vote is to modify existing Macro(i.e., #1 from the >> original email with the necessary clarification above) and make it >> compatible with Java >> Here are my reasons >> 1) The NDArray APIs in question are not following functional style of >> programming, in fact they are just static methods defined on an NDArray >> object - so Scala users are not losing much by using null in place of None. >> You can create a implicit to maintain backward compatibility >> 2) It is adding 220+ APIs(I understand it is generated) for NDArray alone >> 3) this is adding another 100s of APIs unnecessarily, we are starting with >> NDArray but we can't stop there, we will have to do this for Symbol, >> Executor, Iterators, etc., . >> 3) I don't want to be fixing bugs and maintaining code in 2 places. >> 4) I want the cryptic code(# scala macros) to a minimum. >> 5) increased compilation time & bad developer experience - the time to >> compile has gone up quite a bit since we added the APIs last release on my >> 3 year old laptop already.. I think adding 400+ APIs unnecessarily would >> significantly increase build time and bad developer experience >> 6) I want to keep the core of the framework to be in Scala - because it >> allows you to write concise code - Yes it has a bit of learning curve, not >> everyone needs to know. I would rather invest in solidifying the Scala APIs >> and add more features in Scala(RNN, Support GluonHybridizedBlock...there is >> quite bit of work ) - do you want to rewrite everything in Scala and Java. >> 7) Also, the discussion is not creating NDArray class for Java, just >> generate certain APIs to cater for Java incompatibility. >> >> @Andrew: To your response to Qing's comments -
Re: Feedback request for new Java API
Moreover, regarding "add 200+ APIs" - this was exactly what we did for type-safe APIs in Scala, we have NDArray/Symbol.BatchNorm and NDArray/Symbol.api.BatchNorm - why did we decide to do that? Because we thought type-safe could provide better user-experience. Given the example I listed above, I think such user-experience benefits are even greater than what we got from type-safe. On Sat, Sep 29, 2018 at 11:41 AM YiZhi Liu wrote: > > Naveen, software designing is all about tradeoff, every feature we > introduce causes more compiling time, more efforts to maintain, etc. > > The main difference is. > > Option #1: Java users do > NDArray.BatchNorm(data, gamma, beta, null, null, null, null, null, > null, null, null, null, null, null); > (and because every operator has an argument "out", users need to add > an extra "null" to the function call almost every time.) > > Option #2, Java users do > JavaNDArray.BatchNorm(data).setGamma(gamma).setBeta(beta).invoke(); > > I don't think any of the reasons you listed is so important as the > benefit above we got from option #2. > On Sat, Sep 29, 2018 at 8:24 AM Naveen Swamy wrote: > > > > Java APIs are not like Clojure - The current proposal is only to build a > > few thin wrappers for Inference. > > > > To better represent the two cases and this discussion in particular, here > > is an example API > > > > 1) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out > > : Option[NDArray] = None) : org.apache.mxnet.NDArrayFuncReturn > > or > > 2) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out > > : NDArray) : org.apache.mxnet.NDArrayFuncReturn > > > > The discussion is should we add(generate) 200+ APIs to make it Java > > compatible, ie., remove the Option class and the None default value which > > Java does not understand from Option 1) > > > > my suggestion was to remove the Option class and create a implicit for > > backward compatibility and use null instead of None, Andrew and I disagreed > > on this, so I suggested to raise a discussion on dev@ to get more opinions > > and one of us will disagree and commit. Thanks for raising it :) > > > > | * def Activation (data : org.apache.mxnet.NDArray, act_type : String, out > > : NDArray = null) : org.apache.mxnet.NDArrayFuncReturn | > > -- > > > > 1) It is not true that Scala users will lose *default/optional* arguments - > > if we followed the above, they will use null or None, though I do not like > > using nulls, this is a fine compromise. > > To keep backward compatibility we can create a implicit to convert > > Option.None to nulls and Option.Some-> Option.get(), so you are not going > > to break users who might have been using the APIs that were released in > > 1.3. The current incompatibility is only this w.r.t. NDArrays. > > > > 2) Now about the Scala Macros - they are not simple to read or use, When I > > and Qing started working on the #Scala Macros to improve the APIs, it took > > us a good amount of time to get a hang of it. I don't want to add > > additional code when not necessary. > > > > My suggestion and vote is to modify existing Macro(i.e., #1 from the > > original email with the necessary clarification above) and make it > > compatible with Java > > Here are my reasons > > 1) The NDArray APIs in question are not following functional style of > > programming, in fact they are just static methods defined on an NDArray > > object - so Scala users are not losing much by using null in place of None. > > You can create a implicit to maintain backward compatibility > > 2) It is adding 220+ APIs(I understand it is generated) for NDArray alone > > 3) this is adding another 100s of APIs unnecessarily, we are starting with > > NDArray but we can't stop there, we will have to do this for Symbol, > > Executor, Iterators, etc., . > > 3) I don't want to be fixing bugs and maintaining code in 2 places. > > 4) I want the cryptic code(# scala macros) to a minimum. > > 5) increased compilation time & bad developer experience - the time to > > compile has gone up quite a bit since we added the APIs last release on my > > 3 year old laptop already.. I think adding 400+ APIs unnecessarily would > > significantly increase build time and bad developer experience > > 6) I want to keep the core of the framework to be in Scala - because it > > allows you to write concise code - Yes it has a bit of learning curve, not > > everyone needs to know. I would rather invest in solidifying the Scala APIs > > and add more features in Scala(RNN, Support GluonHybridizedBlock...there is > > quite bit of work ) - do you want to rewrite everything in Scala and Java. > > 7) Also, the discussion is not creating NDArray class for Java, just > > generate certain APIs to cater for Java incompatibility. > > > > @Andrew: To your response to Qing's comments - you cannot just consider it > > as just generating NDArray's APIs and instead I suggest to take a wholistic > > view of all the various implications. > > >
Re: Feedback request for new Java API
Naveen, software designing is all about tradeoff, every feature we introduce causes more compiling time, more efforts to maintain, etc. The main difference is. Option #1: Java users do NDArray.BatchNorm(data, gamma, beta, null, null, null, null, null, null, null, null, null, null, null); (and because every operator has an argument "out", users need to add an extra "null" to the function call almost every time.) Option #2, Java users do JavaNDArray.BatchNorm(data).setGamma(gamma).setBeta(beta).invoke(); I don't think any of the reasons you listed is so important as the benefit above we got from option #2. On Sat, Sep 29, 2018 at 8:24 AM Naveen Swamy wrote: > > Java APIs are not like Clojure - The current proposal is only to build a > few thin wrappers for Inference. > > To better represent the two cases and this discussion in particular, here > is an example API > > 1) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out > : Option[NDArray] = None) : org.apache.mxnet.NDArrayFuncReturn > or > 2) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out > : NDArray) : org.apache.mxnet.NDArrayFuncReturn > > The discussion is should we add(generate) 200+ APIs to make it Java > compatible, ie., remove the Option class and the None default value which > Java does not understand from Option 1) > > my suggestion was to remove the Option class and create a implicit for > backward compatibility and use null instead of None, Andrew and I disagreed > on this, so I suggested to raise a discussion on dev@ to get more opinions > and one of us will disagree and commit. Thanks for raising it :) > > | * def Activation (data : org.apache.mxnet.NDArray, act_type : String, out > : NDArray = null) : org.apache.mxnet.NDArrayFuncReturn | > -- > > 1) It is not true that Scala users will lose *default/optional* arguments - > if we followed the above, they will use null or None, though I do not like > using nulls, this is a fine compromise. > To keep backward compatibility we can create a implicit to convert > Option.None to nulls and Option.Some-> Option.get(), so you are not going > to break users who might have been using the APIs that were released in > 1.3. The current incompatibility is only this w.r.t. NDArrays. > > 2) Now about the Scala Macros - they are not simple to read or use, When I > and Qing started working on the #Scala Macros to improve the APIs, it took > us a good amount of time to get a hang of it. I don't want to add > additional code when not necessary. > > My suggestion and vote is to modify existing Macro(i.e., #1 from the > original email with the necessary clarification above) and make it > compatible with Java > Here are my reasons > 1) The NDArray APIs in question are not following functional style of > programming, in fact they are just static methods defined on an NDArray > object - so Scala users are not losing much by using null in place of None. > You can create a implicit to maintain backward compatibility > 2) It is adding 220+ APIs(I understand it is generated) for NDArray alone > 3) this is adding another 100s of APIs unnecessarily, we are starting with > NDArray but we can't stop there, we will have to do this for Symbol, > Executor, Iterators, etc., . > 3) I don't want to be fixing bugs and maintaining code in 2 places. > 4) I want the cryptic code(# scala macros) to a minimum. > 5) increased compilation time & bad developer experience - the time to > compile has gone up quite a bit since we added the APIs last release on my > 3 year old laptop already.. I think adding 400+ APIs unnecessarily would > significantly increase build time and bad developer experience > 6) I want to keep the core of the framework to be in Scala - because it > allows you to write concise code - Yes it has a bit of learning curve, not > everyone needs to know. I would rather invest in solidifying the Scala APIs > and add more features in Scala(RNN, Support GluonHybridizedBlock...there is > quite bit of work ) - do you want to rewrite everything in Scala and Java. > 7) Also, the discussion is not creating NDArray class for Java, just > generate certain APIs to cater for Java incompatibility. > > @Andrew: To your response to Qing's comments - you cannot just consider it > as just generating NDArray's APIs and instead I suggest to take a wholistic > view of all the various implications. > > @Chris: Yes, Scala has a bit of learning curve - the goal is not having > every developer to deal with how these APIs are generated, > the problem exists either ways with the above proposal. I might agree if we > were to move away completely(with a thorough discussion and valid reasons) > and instead use AspectJ or similar to write these APIs, the discussion is > about using Scala Macros to generate 2 different types of APIs which are > functionally not different and usability wise are very very similar, look > at the example. > Thanks for your input, I will deposit your 0.02$ in our JIRA bank :)
Re: Feedback request for new Java API
Java APIs are not like Clojure - The current proposal is only to build a few thin wrappers for Inference. To better represent the two cases and this discussion in particular, here is an example API 1) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out : Option[NDArray] = None) : org.apache.mxnet.NDArrayFuncReturn or 2) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out : NDArray) : org.apache.mxnet.NDArrayFuncReturn The discussion is should we add(generate) 200+ APIs to make it Java compatible, ie., remove the Option class and the None default value which Java does not understand from Option 1) my suggestion was to remove the Option class and create a implicit for backward compatibility and use null instead of None, Andrew and I disagreed on this, so I suggested to raise a discussion on dev@ to get more opinions and one of us will disagree and commit. Thanks for raising it :) | * def Activation (data : org.apache.mxnet.NDArray, act_type : String, out : NDArray = null) : org.apache.mxnet.NDArrayFuncReturn | -- 1) It is not true that Scala users will lose *default/optional* arguments - if we followed the above, they will use null or None, though I do not like using nulls, this is a fine compromise. To keep backward compatibility we can create a implicit to convert Option.None to nulls and Option.Some-> Option.get(), so you are not going to break users who might have been using the APIs that were released in 1.3. The current incompatibility is only this w.r.t. NDArrays. 2) Now about the Scala Macros - they are not simple to read or use, When I and Qing started working on the #Scala Macros to improve the APIs, it took us a good amount of time to get a hang of it. I don't want to add additional code when not necessary. My suggestion and vote is to modify existing Macro(i.e., #1 from the original email with the necessary clarification above) and make it compatible with Java Here are my reasons 1) The NDArray APIs in question are not following functional style of programming, in fact they are just static methods defined on an NDArray object - so Scala users are not losing much by using null in place of None. You can create a implicit to maintain backward compatibility 2) It is adding 220+ APIs(I understand it is generated) for NDArray alone 3) this is adding another 100s of APIs unnecessarily, we are starting with NDArray but we can't stop there, we will have to do this for Symbol, Executor, Iterators, etc., . 3) I don't want to be fixing bugs and maintaining code in 2 places. 4) I want the cryptic code(# scala macros) to a minimum. 5) increased compilation time & bad developer experience - the time to compile has gone up quite a bit since we added the APIs last release on my 3 year old laptop already.. I think adding 400+ APIs unnecessarily would significantly increase build time and bad developer experience 6) I want to keep the core of the framework to be in Scala - because it allows you to write concise code - Yes it has a bit of learning curve, not everyone needs to know. I would rather invest in solidifying the Scala APIs and add more features in Scala(RNN, Support GluonHybridizedBlock...there is quite bit of work ) - do you want to rewrite everything in Scala and Java. 7) Also, the discussion is not creating NDArray class for Java, just generate certain APIs to cater for Java incompatibility. @Andrew: To your response to Qing's comments - you cannot just consider it as just generating NDArray's APIs and instead I suggest to take a wholistic view of all the various implications. @Chris: Yes, Scala has a bit of learning curve - the goal is not having every developer to deal with how these APIs are generated, the problem exists either ways with the above proposal. I might agree if we were to move away completely(with a thorough discussion and valid reasons) and instead use AspectJ or similar to write these APIs, the discussion is about using Scala Macros to generate 2 different types of APIs which are functionally not different and usability wise are very very similar, look at the example. Thanks for your input, I will deposit your 0.02$ in our JIRA bank :) @Carin: It requires more effort to use AspectJ or similar to generate APIs using reflection or at compile time, here we need to generate at compile time so Java users have the API signature on their IDEs. Thanks, Naveen P.S: I am traveling and my responses will be delayed. On Fri, Sep 28, 2018 at 10:25 AM Carin Meier wrote: > Sorry bad paste on the gist - here is the good one > https://gist.github.com/gigasquid/01cd48f563db4739910592dd9ac9db20 > > On Fri, Sep 28, 2018 at 10:24 AM Carin Meier wrote: > > > +1 on option #2 > > > > In the case of minimizing the the overhead for code maintenance, I wanted > > to suggest the option of investigating generating code from the Java > > Reflection for the Java APIs. I did a quick gist from Clojure of what > the > > generated classes look like from the
Re: Feedback request for new Java API
Sorry bad paste on the gist - here is the good one https://gist.github.com/gigasquid/01cd48f563db4739910592dd9ac9db20 On Fri, Sep 28, 2018 at 10:24 AM Carin Meier wrote: > +1 on option #2 > > In the case of minimizing the the overhead for code maintenance, I wanted > to suggest the option of investigating generating code from the Java > Reflection for the Java APIs. I did a quick gist from Clojure of what the > generated classes look like from the current Scala Symbol.api for > FullyConnected here > https://gist.github.com/gigasquid/01cd48f563db4739910592 > > I looks like that there is always a base Java class generated will all the > arguments. If this is the case, then there is a possibility to generate a > Java api based on this Java method automatically with just a conversion for > the Scala option and it might be reusable for all the packages. > > Not sure if it will work for this use case, but thought I would bring it > up in case it's helpful. > > - Carin > > On Fri, Sep 28, 2018 at 7:05 AM Davydenko, Denis > wrote: > >> +1 on option #2. Having clear Java interface for NDArray, from my >> perspective, would be a better experience for Java users as it won't >> require them to deal with Scala code in any capacity. Overhead of extra >> code for additional macros is justified, in my mind, as it will be >> introduced with option #1 either way, just in a different place. >> >> -- >> Thanks, >> Denis >> >> On 9/27/18, 6:14 PM, "YiZhi Liu" wrote: >> >> I vote for "2.) Leave the existing macro in place and add another >> which generates a Java friendly version" >> >> @Qing @Andrew, could you give some examples, so that people can better >> understand how it provides "best possible experience" to Java users. >> >> I have no strong preference between having JavaShape & JavaContext or >> not. >> On Thu, Sep 27, 2018 at 5:56 PM Andrew Ayres < >> andrew.f.ay...@gmail.com> wrote: >> > >> > That's not really the conversation I'm wanting to have. I want a >> discussion >> > about the macros with respect to NDArray so that we can get >> agreement on >> > our path forward with respect to implementing the NDArray wrapper. >> > >> > The design that was put forth and agreed to was for a a Java >> wrapper around >> > the Scala API. Adding a bunch of Java friendly methods inside the >> Scala >> > code would create a mess for users. Maintenance would be >> essentially the >> > same for both because either way you're going to be updating Java >> methods >> > when you make Scala changes. >> > >> > Let's please stick with the issue in the original email. >> > >> > Thanks, >> > Andrew >> > >> > On Thu, Sep 27, 2018 at 5:22 PM Qing Lan >> wrote: >> > >> > > I would like to loop this back a layer. Current, there is a >> discussion in >> > > the MXNet Scala community on the ways to implement the Java APIs. >> Currently >> > > there are two thoughts: >> > > >> > > 1. Make Scala Java Friendly (Create Java compatible methods in >> the Scala >> > > Class. such as NDArray with Java compatible constructor) >> > > 2. Make Java friendly wrappers in Scala (Andrew's explanation >> below) >> > > >> > > The first approach require minimum input from our side to >> implement >> > > however bring user a bunch of useless api they may not want to >> use. It also >> > > makes Scala package heavier. The good thing is these two packages >> require >> > > minimum maintenance cost. As a tradeoff, if any time in the >> future we want >> > > to make Java big (make Java as the primary language supported by >> MXNet), >> > > then the migration from Scala to Java will be harmful. Spark >> consider this >> > > carefully and decide not to change much on their Scala code base >> to make it >> > > more Java. >> > > >> > > The second approach will make unique NDArray, Shape, Context and >> more. The >> > > good thing about this is we can always holds a version control on >> Java. >> > > Some breaking changes on Scala may not influence much on Java. It >> did the >> > > best way to decouple the module and good for us to build unique >> pipeline >> > > for Java. The bad thing with this design is the maintenance cost >> as we need >> > > to keep two code bases, but it also make Java side easy to change >> to make >> > > it better compatible with users. >> > > >> > > Thanks, >> > > Qing >> > > >> > > On 9/27/18, 3:25 PM, "Andrew Ayres" >> wrote: >> > > >> > > Hi, >> > > >> > > Currently, we're working to implement a new Java API and >> would like >> > > some >> > > feedback from the community on an implementation detail. In >> short, the >> > > new >> > > Java API will use the existing Scala API (in a manner similar >> to how >> > > the >> > > current Clojure API works). This basically
Re: Feedback request for new Java API
+1 on option #2 In the case of minimizing the the overhead for code maintenance, I wanted to suggest the option of investigating generating code from the Java Reflection for the Java APIs. I did a quick gist from Clojure of what the generated classes look like from the current Scala Symbol.api for FullyConnected here https://gist.github.com/gigasquid/01cd48f563db4739910592 I looks like that there is always a base Java class generated will all the arguments. If this is the case, then there is a possibility to generate a Java api based on this Java method automatically with just a conversion for the Scala option and it might be reusable for all the packages. Not sure if it will work for this use case, but thought I would bring it up in case it's helpful. - Carin On Fri, Sep 28, 2018 at 7:05 AM Davydenko, Denis wrote: > +1 on option #2. Having clear Java interface for NDArray, from my > perspective, would be a better experience for Java users as it won't > require them to deal with Scala code in any capacity. Overhead of extra > code for additional macros is justified, in my mind, as it will be > introduced with option #1 either way, just in a different place. > > -- > Thanks, > Denis > > On 9/27/18, 6:14 PM, "YiZhi Liu" wrote: > > I vote for "2.) Leave the existing macro in place and add another > which generates a Java friendly version" > > @Qing @Andrew, could you give some examples, so that people can better > understand how it provides "best possible experience" to Java users. > > I have no strong preference between having JavaShape & JavaContext or > not. > On Thu, Sep 27, 2018 at 5:56 PM Andrew Ayres > wrote: > > > > That's not really the conversation I'm wanting to have. I want a > discussion > > about the macros with respect to NDArray so that we can get > agreement on > > our path forward with respect to implementing the NDArray wrapper. > > > > The design that was put forth and agreed to was for a a Java wrapper > around > > the Scala API. Adding a bunch of Java friendly methods inside the > Scala > > code would create a mess for users. Maintenance would be essentially > the > > same for both because either way you're going to be updating Java > methods > > when you make Scala changes. > > > > Let's please stick with the issue in the original email. > > > > Thanks, > > Andrew > > > > On Thu, Sep 27, 2018 at 5:22 PM Qing Lan > wrote: > > > > > I would like to loop this back a layer. Current, there is a > discussion in > > > the MXNet Scala community on the ways to implement the Java APIs. > Currently > > > there are two thoughts: > > > > > > 1. Make Scala Java Friendly (Create Java compatible methods in the > Scala > > > Class. such as NDArray with Java compatible constructor) > > > 2. Make Java friendly wrappers in Scala (Andrew's explanation > below) > > > > > > The first approach require minimum input from our side to implement > > > however bring user a bunch of useless api they may not want to > use. It also > > > makes Scala package heavier. The good thing is these two packages > require > > > minimum maintenance cost. As a tradeoff, if any time in the future > we want > > > to make Java big (make Java as the primary language supported by > MXNet), > > > then the migration from Scala to Java will be harmful. Spark > consider this > > > carefully and decide not to change much on their Scala code base > to make it > > > more Java. > > > > > > The second approach will make unique NDArray, Shape, Context and > more. The > > > good thing about this is we can always holds a version control on > Java. > > > Some breaking changes on Scala may not influence much on Java. It > did the > > > best way to decouple the module and good for us to build unique > pipeline > > > for Java. The bad thing with this design is the maintenance cost > as we need > > > to keep two code bases, but it also make Java side easy to change > to make > > > it better compatible with users. > > > > > > Thanks, > > > Qing > > > > > > On 9/27/18, 3:25 PM, "Andrew Ayres" > wrote: > > > > > > Hi, > > > > > > Currently, we're working to implement a new Java API and would > like > > > some > > > feedback from the community on an implementation detail. In > short, the > > > new > > > Java API will use the existing Scala API (in a manner similar > to how > > > the > > > current Clojure API works). This basically means that we're > making Java > > > friendly wrappers to call the existing Scala API. > > > > > > The feedback we're looking for is on the implementation of > NDArray. > > > Scala's > > > NDArray has a significant amount of code which is generated > via macros > > > and > > > we've got two viable paths
Re: Feedback request for new Java API
+1 on option #2. Having clear Java interface for NDArray, from my perspective, would be a better experience for Java users as it won't require them to deal with Scala code in any capacity. Overhead of extra code for additional macros is justified, in my mind, as it will be introduced with option #1 either way, just in a different place. -- Thanks, Denis On 9/27/18, 6:14 PM, "YiZhi Liu" wrote: I vote for "2.) Leave the existing macro in place and add another which generates a Java friendly version" @Qing @Andrew, could you give some examples, so that people can better understand how it provides "best possible experience" to Java users. I have no strong preference between having JavaShape & JavaContext or not. On Thu, Sep 27, 2018 at 5:56 PM Andrew Ayres wrote: > > That's not really the conversation I'm wanting to have. I want a discussion > about the macros with respect to NDArray so that we can get agreement on > our path forward with respect to implementing the NDArray wrapper. > > The design that was put forth and agreed to was for a a Java wrapper around > the Scala API. Adding a bunch of Java friendly methods inside the Scala > code would create a mess for users. Maintenance would be essentially the > same for both because either way you're going to be updating Java methods > when you make Scala changes. > > Let's please stick with the issue in the original email. > > Thanks, > Andrew > > On Thu, Sep 27, 2018 at 5:22 PM Qing Lan wrote: > > > I would like to loop this back a layer. Current, there is a discussion in > > the MXNet Scala community on the ways to implement the Java APIs. Currently > > there are two thoughts: > > > > 1. Make Scala Java Friendly (Create Java compatible methods in the Scala > > Class. such as NDArray with Java compatible constructor) > > 2. Make Java friendly wrappers in Scala (Andrew's explanation below) > > > > The first approach require minimum input from our side to implement > > however bring user a bunch of useless api they may not want to use. It also > > makes Scala package heavier. The good thing is these two packages require > > minimum maintenance cost. As a tradeoff, if any time in the future we want > > to make Java big (make Java as the primary language supported by MXNet), > > then the migration from Scala to Java will be harmful. Spark consider this > > carefully and decide not to change much on their Scala code base to make it > > more Java. > > > > The second approach will make unique NDArray, Shape, Context and more. The > > good thing about this is we can always holds a version control on Java. > > Some breaking changes on Scala may not influence much on Java. It did the > > best way to decouple the module and good for us to build unique pipeline > > for Java. The bad thing with this design is the maintenance cost as we need > > to keep two code bases, but it also make Java side easy to change to make > > it better compatible with users. > > > > Thanks, > > Qing > > > > On 9/27/18, 3:25 PM, "Andrew Ayres" wrote: > > > > Hi, > > > > Currently, we're working to implement a new Java API and would like > > some > > feedback from the community on an implementation detail. In short, the > > new > > Java API will use the existing Scala API (in a manner similar to how > > the > > current Clojure API works). This basically means that we're making Java > > friendly wrappers to call the existing Scala API. > > > > The feedback we're looking for is on the implementation of NDArray. > > Scala's > > NDArray has a significant amount of code which is generated via macros > > and > > we've got two viable paths to move forward: > > > > 1.) Change the macro to generate Java friendly methods - To do this > > we'll > > modify the macro so that the generated methods won't have > > default/optional > > arguments. There may also have to be some changes to parameter types to > > make them Java friendly. The big advantage here is that ongoing > > maintenance > > will easier. The disadvantages are that we'll be changing the existing > > Scala NDArray Infer API (it's marked experimental) and Scala users will > > lose the ability to use the default and optional arguments. > > > > 2.) Leave the existing macro in place and add another which generates a > > Java friendly version - The biggest issue here is that we'll be > > doubling > > the number of macros that we've got to maintain. It'll become even more > > overhead once we start expanding the Java API with more classes that > > use > > generated code like
Re: Feedback request for new Java API
Thanks Chris for your excellent explanation! Personally I like the way we
separate the world that Scala and Java shares so Java developers don't have to
think of how to use "Scala" in order to write Java code. I think usability for
Java users and Scala users is the most important tasks we need to take care of.
A poor usability on these APIs will make them say "good bye" to a language
(like the feedback from "Zhihu"). In that case, the Java wrapper design
significantly hide the "unfriendly" Scala API and create a wall that Scala
users won't have to use them. So I would +0.5 for Andrews Approach. I also vote
for plan 2) since the new secret Java Macros do a bunch of changes that makes
them way different than Scala Macros.
But here I quote some point that Naveen mentioned me offline:
1. Compile time problem: The current build pipeline already takes a long time
to compile the code base with adding more and more to make Java ready. Although
we narrow down the scope of development to just Java Inference API. Future
development will still requires a bunch of Java methods being added. It also
makes the code base redundant a little bit. It will also introduce a "fat jar"
for both Scala and Java users.
2. The development and maintenance cost: These code would cost a lot of times
to develop and maintain than implementing them directly in Scala.
Here are the two biggest shortcomings if we go this way. These need to be
carefully considered and avoided in the implementation.
--- Break line for Code Gen
demo time --
I am also a developer working on Scala Macros a long time. Here I would like to
show the usability improvement we have taken to Java users.
In v1.3, we delivered the Type-safe API which allows Scala user to do something
like:
Symbol.api.FullyConnected(data = Some(data), num_hidden = 128, name = "fc1")
Rather than the old API:
Symbol.FullyConnected("fc1")()(Map("data" -> data, "num_hidden" -> 128))
The scenario for Java user to use this API is:
1. Option is not introduced in Java and the closest Class is Optional however
introduced in 1.8 (we support MXNet Scala from 1.7)
2. Too many args field to pass. The sacrificed point from Scala to Java is we
lose all default field. For some operators, Java user may need to fill in 13
arguments (e.g BatchNorm) even if they are not willing to do so.
In this case, we have carefully re-invented the Java API with
Function+Classbuilder generated by Scala Macros:
The new Java operator API will looks like:
(Java)NDArray.BatchNorm().set().invoke()
In this case, all generated operators for Java will create a operatorBuilder
(BatchNormBuilder in above examples). The builder's setter will direct Java
users what are the optional field they can pass and the Builder constructor
clearly identify the required field for the operators. The only shortcoming for
this api is that user has to call a method "invoke()" to get it run, where the
invoke method would collect all args and call JNI code.
Thanks,
Qing
On 9/27/18, 8:53 PM, "Chris Olivier" wrote:
My $0.02 since I’m working with a lot of java and scala lately, including
the interaction between the two:
Please keep in mind the more complex dependency issues that will be
introduced by requiring Java users to now have to pull in a large scala
dependency base. In addition, a lot is Scala is compiler-dependent with
different jars for 2.10,2.12,etc. and so they’re in different places than
the regular jars and having a Java person suddenly have to deal with this
is a good way to make him say “no thanks”. My day job build and runtime
environment systems, for instance, don’t handle the mixing very well
without having to hack a bunch of build files and even then it causes
problems from time to time. Scala experts won’t have problems but it’s a
steep learning curve for Java (or C++) folks.
On Thu, Sep 27, 2018 at 6:14 PM YiZhi Liu wrote:
> I vote for "2.) Leave the existing macro in place and add another
> which generates a Java friendly version"
>
> @Qing @Andrew, could you give some examples, so that people can better
> understand how it provides "best possible experience" to Java users.
>
> I have no strong preference between having JavaShape & JavaContext or not.
> On Thu, Sep 27, 2018 at 5:56 PM Andrew Ayres
> wrote:
> >
> > That's not really the conversation I'm wanting to have. I want a
> discussion
> > about the macros with respect to NDArray so that we can get agreement on
> > our path forward with respect to implementing the NDArray wrapper.
> >
> > The design that was put forth and agreed to was for a a Java wrapper
> around
> > the Scala API. Adding a bunch of Java friendly methods inside the Scala
> > code would create a mess for users. Maintenance would be essentially the
> > same
Re: Feedback request for new Java API
My $0.02 since I’m working with a lot of java and scala lately, including the interaction between the two: Please keep in mind the more complex dependency issues that will be introduced by requiring Java users to now have to pull in a large scala dependency base. In addition, a lot is Scala is compiler-dependent with different jars for 2.10,2.12,etc. and so they’re in different places than the regular jars and having a Java person suddenly have to deal with this is a good way to make him say “no thanks”. My day job build and runtime environment systems, for instance, don’t handle the mixing very well without having to hack a bunch of build files and even then it causes problems from time to time. Scala experts won’t have problems but it’s a steep learning curve for Java (or C++) folks. On Thu, Sep 27, 2018 at 6:14 PM YiZhi Liu wrote: > I vote for "2.) Leave the existing macro in place and add another > which generates a Java friendly version" > > @Qing @Andrew, could you give some examples, so that people can better > understand how it provides "best possible experience" to Java users. > > I have no strong preference between having JavaShape & JavaContext or not. > On Thu, Sep 27, 2018 at 5:56 PM Andrew Ayres > wrote: > > > > That's not really the conversation I'm wanting to have. I want a > discussion > > about the macros with respect to NDArray so that we can get agreement on > > our path forward with respect to implementing the NDArray wrapper. > > > > The design that was put forth and agreed to was for a a Java wrapper > around > > the Scala API. Adding a bunch of Java friendly methods inside the Scala > > code would create a mess for users. Maintenance would be essentially the > > same for both because either way you're going to be updating Java methods > > when you make Scala changes. > > > > Let's please stick with the issue in the original email. > > > > Thanks, > > Andrew > > > > On Thu, Sep 27, 2018 at 5:22 PM Qing Lan wrote: > > > > > I would like to loop this back a layer. Current, there is a discussion > in > > > the MXNet Scala community on the ways to implement the Java APIs. > Currently > > > there are two thoughts: > > > > > > 1. Make Scala Java Friendly (Create Java compatible methods in the > Scala > > > Class. such as NDArray with Java compatible constructor) > > > 2. Make Java friendly wrappers in Scala (Andrew's explanation below) > > > > > > The first approach require minimum input from our side to implement > > > however bring user a bunch of useless api they may not want to use. It > also > > > makes Scala package heavier. The good thing is these two packages > require > > > minimum maintenance cost. As a tradeoff, if any time in the future we > want > > > to make Java big (make Java as the primary language supported by > MXNet), > > > then the migration from Scala to Java will be harmful. Spark consider > this > > > carefully and decide not to change much on their Scala code base to > make it > > > more Java. > > > > > > The second approach will make unique NDArray, Shape, Context and more. > The > > > good thing about this is we can always holds a version control on Java. > > > Some breaking changes on Scala may not influence much on Java. It did > the > > > best way to decouple the module and good for us to build unique > pipeline > > > for Java. The bad thing with this design is the maintenance cost as we > need > > > to keep two code bases, but it also make Java side easy to change to > make > > > it better compatible with users. > > > > > > Thanks, > > > Qing > > > > > > On 9/27/18, 3:25 PM, "Andrew Ayres" wrote: > > > > > > Hi, > > > > > > Currently, we're working to implement a new Java API and would like > > > some > > > feedback from the community on an implementation detail. In short, > the > > > new > > > Java API will use the existing Scala API (in a manner similar to > how > > > the > > > current Clojure API works). This basically means that we're making > Java > > > friendly wrappers to call the existing Scala API. > > > > > > The feedback we're looking for is on the implementation of NDArray. > > > Scala's > > > NDArray has a significant amount of code which is generated via > macros > > > and > > > we've got two viable paths to move forward: > > > > > > 1.) Change the macro to generate Java friendly methods - To do > this > > > we'll > > > modify the macro so that the generated methods won't have > > > default/optional > > > arguments. There may also have to be some changes to parameter > types to > > > make them Java friendly. The big advantage here is that ongoing > > > maintenance > > > will easier. The disadvantages are that we'll be changing the > existing > > > Scala NDArray Infer API (it's marked experimental) and Scala users > will > > > lose the ability to use the default and optional arguments. > > > > > > 2.) Leave the existing macro in place and add another which >
Re: Feedback request for new Java API
I vote for "2.) Leave the existing macro in place and add another which generates a Java friendly version" @Qing @Andrew, could you give some examples, so that people can better understand how it provides "best possible experience" to Java users. I have no strong preference between having JavaShape & JavaContext or not. On Thu, Sep 27, 2018 at 5:56 PM Andrew Ayres wrote: > > That's not really the conversation I'm wanting to have. I want a discussion > about the macros with respect to NDArray so that we can get agreement on > our path forward with respect to implementing the NDArray wrapper. > > The design that was put forth and agreed to was for a a Java wrapper around > the Scala API. Adding a bunch of Java friendly methods inside the Scala > code would create a mess for users. Maintenance would be essentially the > same for both because either way you're going to be updating Java methods > when you make Scala changes. > > Let's please stick with the issue in the original email. > > Thanks, > Andrew > > On Thu, Sep 27, 2018 at 5:22 PM Qing Lan wrote: > > > I would like to loop this back a layer. Current, there is a discussion in > > the MXNet Scala community on the ways to implement the Java APIs. Currently > > there are two thoughts: > > > > 1. Make Scala Java Friendly (Create Java compatible methods in the Scala > > Class. such as NDArray with Java compatible constructor) > > 2. Make Java friendly wrappers in Scala (Andrew's explanation below) > > > > The first approach require minimum input from our side to implement > > however bring user a bunch of useless api they may not want to use. It also > > makes Scala package heavier. The good thing is these two packages require > > minimum maintenance cost. As a tradeoff, if any time in the future we want > > to make Java big (make Java as the primary language supported by MXNet), > > then the migration from Scala to Java will be harmful. Spark consider this > > carefully and decide not to change much on their Scala code base to make it > > more Java. > > > > The second approach will make unique NDArray, Shape, Context and more. The > > good thing about this is we can always holds a version control on Java. > > Some breaking changes on Scala may not influence much on Java. It did the > > best way to decouple the module and good for us to build unique pipeline > > for Java. The bad thing with this design is the maintenance cost as we need > > to keep two code bases, but it also make Java side easy to change to make > > it better compatible with users. > > > > Thanks, > > Qing > > > > On 9/27/18, 3:25 PM, "Andrew Ayres" wrote: > > > > Hi, > > > > Currently, we're working to implement a new Java API and would like > > some > > feedback from the community on an implementation detail. In short, the > > new > > Java API will use the existing Scala API (in a manner similar to how > > the > > current Clojure API works). This basically means that we're making Java > > friendly wrappers to call the existing Scala API. > > > > The feedback we're looking for is on the implementation of NDArray. > > Scala's > > NDArray has a significant amount of code which is generated via macros > > and > > we've got two viable paths to move forward: > > > > 1.) Change the macro to generate Java friendly methods - To do this > > we'll > > modify the macro so that the generated methods won't have > > default/optional > > arguments. There may also have to be some changes to parameter types to > > make them Java friendly. The big advantage here is that ongoing > > maintenance > > will easier. The disadvantages are that we'll be changing the existing > > Scala NDArray Infer API (it's marked experimental) and Scala users will > > lose the ability to use the default and optional arguments. > > > > 2.) Leave the existing macro in place and add another which generates a > > Java friendly version - The biggest issue here is that we'll be > > doubling > > the number of macros that we've got to maintain. It'll become even more > > overhead once we start expanding the Java API with more classes that > > use > > generated code like this. The advantages are that the existing Scala > > NDArray Infer API would remain unchanged for Scala users and that the > > new > > macro could be optimized to give the best possible experience to the > > Java > > API. > > > > Thanks, > > Andrew > > > > > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Feedback request for new Java API
That's not really the conversation I'm wanting to have. I want a discussion about the macros with respect to NDArray so that we can get agreement on our path forward with respect to implementing the NDArray wrapper. The design that was put forth and agreed to was for a a Java wrapper around the Scala API. Adding a bunch of Java friendly methods inside the Scala code would create a mess for users. Maintenance would be essentially the same for both because either way you're going to be updating Java methods when you make Scala changes. Let's please stick with the issue in the original email. Thanks, Andrew On Thu, Sep 27, 2018 at 5:22 PM Qing Lan wrote: > I would like to loop this back a layer. Current, there is a discussion in > the MXNet Scala community on the ways to implement the Java APIs. Currently > there are two thoughts: > > 1. Make Scala Java Friendly (Create Java compatible methods in the Scala > Class. such as NDArray with Java compatible constructor) > 2. Make Java friendly wrappers in Scala (Andrew's explanation below) > > The first approach require minimum input from our side to implement > however bring user a bunch of useless api they may not want to use. It also > makes Scala package heavier. The good thing is these two packages require > minimum maintenance cost. As a tradeoff, if any time in the future we want > to make Java big (make Java as the primary language supported by MXNet), > then the migration from Scala to Java will be harmful. Spark consider this > carefully and decide not to change much on their Scala code base to make it > more Java. > > The second approach will make unique NDArray, Shape, Context and more. The > good thing about this is we can always holds a version control on Java. > Some breaking changes on Scala may not influence much on Java. It did the > best way to decouple the module and good for us to build unique pipeline > for Java. The bad thing with this design is the maintenance cost as we need > to keep two code bases, but it also make Java side easy to change to make > it better compatible with users. > > Thanks, > Qing > > On 9/27/18, 3:25 PM, "Andrew Ayres" wrote: > > Hi, > > Currently, we're working to implement a new Java API and would like > some > feedback from the community on an implementation detail. In short, the > new > Java API will use the existing Scala API (in a manner similar to how > the > current Clojure API works). This basically means that we're making Java > friendly wrappers to call the existing Scala API. > > The feedback we're looking for is on the implementation of NDArray. > Scala's > NDArray has a significant amount of code which is generated via macros > and > we've got two viable paths to move forward: > > 1.) Change the macro to generate Java friendly methods - To do this > we'll > modify the macro so that the generated methods won't have > default/optional > arguments. There may also have to be some changes to parameter types to > make them Java friendly. The big advantage here is that ongoing > maintenance > will easier. The disadvantages are that we'll be changing the existing > Scala NDArray Infer API (it's marked experimental) and Scala users will > lose the ability to use the default and optional arguments. > > 2.) Leave the existing macro in place and add another which generates a > Java friendly version - The biggest issue here is that we'll be > doubling > the number of macros that we've got to maintain. It'll become even more > overhead once we start expanding the Java API with more classes that > use > generated code like this. The advantages are that the existing Scala > NDArray Infer API would remain unchanged for Scala users and that the > new > macro could be optimized to give the best possible experience to the > Java > API. > > Thanks, > Andrew > > >
Re: Feedback request for new Java API
I would like to loop this back a layer. Current, there is a discussion in the MXNet Scala community on the ways to implement the Java APIs. Currently there are two thoughts: 1. Make Scala Java Friendly (Create Java compatible methods in the Scala Class. such as NDArray with Java compatible constructor) 2. Make Java friendly wrappers in Scala (Andrew's explanation below) The first approach require minimum input from our side to implement however bring user a bunch of useless api they may not want to use. It also makes Scala package heavier. The good thing is these two packages require minimum maintenance cost. As a tradeoff, if any time in the future we want to make Java big (make Java as the primary language supported by MXNet), then the migration from Scala to Java will be harmful. Spark consider this carefully and decide not to change much on their Scala code base to make it more Java. The second approach will make unique NDArray, Shape, Context and more. The good thing about this is we can always holds a version control on Java. Some breaking changes on Scala may not influence much on Java. It did the best way to decouple the module and good for us to build unique pipeline for Java. The bad thing with this design is the maintenance cost as we need to keep two code bases, but it also make Java side easy to change to make it better compatible with users. Thanks, Qing On 9/27/18, 3:25 PM, "Andrew Ayres" wrote: Hi, Currently, we're working to implement a new Java API and would like some feedback from the community on an implementation detail. In short, the new Java API will use the existing Scala API (in a manner similar to how the current Clojure API works). This basically means that we're making Java friendly wrappers to call the existing Scala API. The feedback we're looking for is on the implementation of NDArray. Scala's NDArray has a significant amount of code which is generated via macros and we've got two viable paths to move forward: 1.) Change the macro to generate Java friendly methods - To do this we'll modify the macro so that the generated methods won't have default/optional arguments. There may also have to be some changes to parameter types to make them Java friendly. The big advantage here is that ongoing maintenance will easier. The disadvantages are that we'll be changing the existing Scala NDArray Infer API (it's marked experimental) and Scala users will lose the ability to use the default and optional arguments. 2.) Leave the existing macro in place and add another which generates a Java friendly version - The biggest issue here is that we'll be doubling the number of macros that we've got to maintain. It'll become even more overhead once we start expanding the Java API with more classes that use generated code like this. The advantages are that the existing Scala NDArray Infer API would remain unchanged for Scala users and that the new macro could be optimized to give the best possible experience to the Java API. Thanks, Andrew
Feedback request for new Java API
Hi, Currently, we're working to implement a new Java API and would like some feedback from the community on an implementation detail. In short, the new Java API will use the existing Scala API (in a manner similar to how the current Clojure API works). This basically means that we're making Java friendly wrappers to call the existing Scala API. The feedback we're looking for is on the implementation of NDArray. Scala's NDArray has a significant amount of code which is generated via macros and we've got two viable paths to move forward: 1.) Change the macro to generate Java friendly methods - To do this we'll modify the macro so that the generated methods won't have default/optional arguments. There may also have to be some changes to parameter types to make them Java friendly. The big advantage here is that ongoing maintenance will easier. The disadvantages are that we'll be changing the existing Scala NDArray Infer API (it's marked experimental) and Scala users will lose the ability to use the default and optional arguments. 2.) Leave the existing macro in place and add another which generates a Java friendly version - The biggest issue here is that we'll be doubling the number of macros that we've got to maintain. It'll become even more overhead once we start expanding the Java API with more classes that use generated code like this. The advantages are that the existing Scala NDArray Infer API would remain unchanged for Scala users and that the new macro could be optimized to give the best possible experience to the Java API. Thanks, Andrew
