+1 for using Swig or JavaCPP to generate a low-level API and then high-level Scala and Java API's can converge to use that layer. Scala diverging towards scripting and Java towards some other API (i.e. Deeplearning4j, which I have never used). Using a generated interface layer makes it much more maintainable IMHO.
On Wed, Aug 16, 2017 at 9:19 AM, Nan Zhu <[email protected]> wrote: > Hi, Yizhi, > > You mean users have 2.10 env while we assemble 2.11 in it? > > Best, > > Nan > > On Wed, Aug 16, 2017 at 9:08 AM, YiZhi Liu <[email protected]> wrote: > > > Hi Joern, > > > > The point is that, the front is not a simple wrapper of c_api.h, as > > you mentioned, which can be easily achieved by JavaCPP. > > > > I have noticed the potential conflicts between the assembled scala > > library and the one in users' environment. Can we remove the scala > > library from the assembly jar? @Nan It wouldn't be a problem since the > > scala libraries with same major version are compatible. > > > > 2017-08-16 23:49 GMT+08:00 Joern Kottmann <[email protected]>: > > > Hello, > > > > > > I personally had quite some issues with Scala dependencies in > > > different versions and Spark, where one version is not compatible with > > > the other version. Then you need to debug the dependency tree to find > > > the places where the versions don't match. Every project which would > > > like to use MXnet then has to depend on Scala and might also get > > > conflicts if other dependencies depend on different Scala versions. > > > Probably something which will cause issues for some of your users. > > > Users who want to use Java might not be familiar with Scala dependency > > > problems and have a hard time resolving them by getting strange error > > > messages. > > > > > > The JNI layer could be generated with JavaCPP, then we would not need > > > to write/maintain the C and the jvm side for that our self. > > > A good example of JavaCPP and Scala usage is Apache Mahout [1]. > > > > > > Even if we don't use JavaCPP, the JNI layer should be easy to get into > > > a state where both can share it, the current Scala JNI layers LibInfo > > > classes could be converted to Java classes and would in most cases > > > require only minor changes in the Scala code. > > > > > > Jörn > > > > > > [1] https://github.com/apache/mahout/tree/master/viennacl/src/main > > > > > > On Wed, Aug 16, 2017 at 5:30 PM, Nan Zhu <[email protected]> > wrote: > > >> I agree with Yizhi > > >> > > >> My major concern is the duplicate implementations, which are usually > > one of > > >> the major sources of bugs, especially with two languages which are > > >> naturally interactive (OK, Calling Scala from Java might need some > more > > >> efforts). It is just like we provide C++ & C APIs of MxNet in two > > separated > > >> packages. > > >> > > >> About dependency problem, when you say "As far as I see this has the > > great > > >> disadvantage that the Java API would force Scala as a dependency onto > > the > > >> java users.", would you please give a concrete example causing > critical > > >> issues? > > >> > > >> Best, > > >> > > >> Nan > > >> > > >> > > >> > > >> On Wed, Aug 16, 2017 at 8:19 AM, YiZhi Liu <[email protected]> > wrote: > > >> > > >>> Hi, > > >>> > > >>> If we build the Java API from the very beginning, i.e. the JNI part, > > >>> we have to rewrite the codes for training, predict, inferShape, etc. > > >>> It would be too heavy to maintain a totally new front language. > > >>> > > >>> As far as I see, I don't think Scala library dependency would be a > big > > >>> problem in most cases, unless we are going to use it in embedded > > >>> devices. Could you illustrate some use-cases where you cannot involve > > >>> Scala dependencies? > > >>> > > >>> 2017-08-16 22:13 GMT+08:00 Joern Kottmann <[email protected]>: > > >>> > Hello, > > >>> > > > >>> > the approach which is taken by Spark is described here [1]. > > >>> > > > >>> > As far as I see this has the great disadvantage that the Java API > > >>> > would force Scala as a dependency onto the java users. > > >>> > For a library it is always a great advantage if it doesn't have > many > > >>> > dependencies, or zero dependencies. In our case it could be quite > > >>> > realistic to have a thin wrapper around the C API without needing > any > > >>> > other dependencies (or only dependencies which can't be avoided). > > >>> > > > >>> > The JNI layer could easily be shared between the Java and Scala > API. > > >>> > As far as I understand is the JNI layer in the Scala API anyway > > >>> > private and a change to it wouldn't require that the public part of > > >>> > the Scala API is changed. > > >>> > > > >>> > What do you think? > > >>> > > > >>> > Jörn > > >>> > > > >>> > [1] https://cwiki.apache.org/confluence/display/SPARK/Java+ > > API+Internals > > >>> > > > >>> > On Wed, Aug 16, 2017 at 3:39 PM, YiZhi Liu <[email protected]> > > wrote: > > >>> >> Hi Joern, > > >>> >> > > >>> >> I suggest to build Java API as a wrapper of Scala API, re-use most > > of > > >>> >> the procedures. Referring to the Java API in Apache Spark. > > >>> >> > > >>> >> 2017-08-16 18:21 GMT+08:00 Joern Kottmann <[email protected]>: > > >>> >>> Hello all, > > >>> >>> > > >>> >>> I would like to propose the addition of a Java API to MXNet. > > >>> >>> > > >>> >>> There has been some previous work done for the Scala API, and it > > makes > > >>> >>> sense to at least share the JNI layer between the two. > > >>> >>> > > >>> >>> The Java API probably should be aligned with the Python API (and > > >>> >>> others which exist already) with a few changes to give it a > native > > >>> >>> Java feel. > > >>> >>> > > >>> >>> As far as I understand there are multiple people interested to > > work on > > >>> >>> this and it would be good to maybe come up with a written > proposal > > on > > >>> >>> how things should be. > > >>> >>> > > >>> >>> My motivation is to get a Java API which can be used by Apache > > OpenNLP > > >>> >>> to solve various NLP tasks using Deep Learning based approaches > > and I > > >>> >>> am also interested to work on MXNet. > > >>> >>> > > >>> >>> Jörn > > >>> >> > > >>> >> > > >>> >> > > >>> >> -- > > >>> >> Yizhi Liu > > >>> >> DMLC member > > >>> >> Technical Manager > > >>> >> Qihoo 360 Inc, Shanghai, China > > >>> > > >>> > > >>> > > >>> -- > > >>> Yizhi Liu > > >>> DMLC member > > >>> Technical Manager > > >>> Qihoo 360 Inc, Shanghai, China > > >>> > > > > > > > > -- > > Yizhi Liu > > DMLC member > > Technical Manager > > Qihoo 360 Inc, Shanghai, China > > >
