>
> I'm not sure why users even need to know about the version mapping.  If I'm
> only interested in the Scala package from maven, why do I care which
> version of libmxnet.so I'm using?

If users want to use a specific feature, e.g., Volta GPU support, they need
to first check which MXNet core version brings the feature, then check
which Scala API version works with this MXNet Core version. And vice versa,
as Chris described, if they have problem using the Scala API, they need to
figure out which core version they are using.

They either choose to update their package and then fix potential breaking
> API changes (the likely case), or they stick with the current version.

Then what do we benefit from separating the version?

They will increase major version after RDD is
> removed.  This is basically scenario 1 from above.  It would mean we
> release MXNet 2.0 with the Scala changes.
>
So,
1. Is it crucial enough to change major version for Scala API namespace
changing?
2. If so, can we postpone changing the namespace and wait for MXNet 2.0
release?

2018-03-11 13:56 GMT-07:00 kellen sunderland <kellen.sunderl...@gmail.com>:

> "Did ML lib increase their major version after deprecating RDD?"
>
> Answering my own question.  They will increase major version after RDD is
> removed.  This is basically scenario 1 from above.  It would mean we
> release MXNet 2.0 with the Scala changes.
>
> On Sun, Mar 11, 2018 at 9:54 PM, kellen sunderland <
> kellen.sunderl...@gmail.com> wrote:
>
> > "Why refactoring and deprecating means separating version from mxnet
> > core?  Apache Spark MLLib refactors and deprecates a lot (e.g., they
> > deprecates RDD API), our C API also deprecates things, remember there
> are a
> > bunch of xxxEx in c_api.h?"
> >
> > Did ML lib increase their major version after deprecating RDD?
> >
> > "They will. Scala API runs auto code-generation to extract Symbol method
> > from MXNet core. For example, users can write and compile
> > Symbol.NewOperator with one Scala API version, but they cannot run it
> with
> > an mxnet-core .so which does not have NewOperator / or have NewOperator
> > with different args."
> >
> > Not sure I fully understand the scenario you're describing here.  Is this
> > the case where a user writes a new operator against one version of
> > libmxnet.so and then runs it on an older version?  In this case they'd
> need
> > to set a dependency on the current libmxnet.so ABI that they're running
> > against, and ensure that their jar was using that version or newer.  This
> > is the goal of SemVer per interface.
> >
> > "By doing major version change to Scala API, we remind users 'hey, be
> > careful, we have something incompatible!' But then what?"
> > They either choose to update their package and then fix potential
> breaking
> > API changes (the likely case), or they stick with the current version.
> >
> > "Users get more confused with the version mapping. And it introduces
> > overhead to maintain."
> > I'm not sure why users even need to know about the version mapping.  If
> > I'm only interested in the Scala package from maven, why do I care which
> > version of libmxnet.so I'm using?
> >
> >
> >
> > On Sun, Mar 11, 2018 at 8:06 PM, YiZhi Liu <eazhi....@gmail.com> wrote:
> >
> >> >
> >> > Changing namespaces is one example of a required major version change,
> >> but
> >> > there are more reasons like general refactoring or some deprecated
> APIs
> >> > just being hard to maintain.
> >>
> >> Why refactoring and deprecating means separating version from mxnet
> core?
> >> Apache Spark MLLib refactors and deprecates a lot (e.g., they deprecates
> >> RDD API), our C API also deprecates things, remember there are a bunch
> of
> >> xxxEx in c_api.h?
> >>
> >> They won't get a strange error, assuming we're talking about Scala users
> >> > who are upgrading from a package with the same namespace they will
> rely
> >> on
> >> > the package manager to give them an update which should be painless.
> >>
> >> They will. Scala API runs auto code-generation to extract Symbol method
> >> from MXNet core. For example, users can write and compile
> >> Symbol.NewOperator with one Scala API version, but they cannot run it
> with
> >> an mxnet-core .so which does not have NewOperator / or have NewOperator
> >> with different args.
> >>
> >> By doing major version change to Scala API, we remind users 'hey, be
> >> careful, we have something incompatible!' But then what? Users get more
> >> confused with the version mapping. And it introduces overhead to
> maintain.
> >>
> >> @Chris, I think we can have two separate votes.
> >>
> >>
> >> 2018-03-11 9:19 GMT-07:00 Chris Olivier <cjolivie...@gmail.com>:
> >>
> >> > Ok, so why don’t we have two votes?
> >> >
> >> > 1) change namespace is a separate vote since it’s a code change and
> has
> >> > different voting rules (can be vetoed)
> >> >
> >> > 2) whether to disconnect non-C-API versioning from C-API versioning
> and
> >> > have parallel versioning of all non-C APIs (process rule, so
> majority, I
> >> > think is the rule, right?)
> >> >
> >> > -Chris
> >> >
> >> > On Sun, Mar 11, 2018 at 8:46 AM kellen sunderland <
> >> > kellen.sunderl...@gmail.com> wrote:
> >> >
> >> > > Sorry, the namespace should have been 'org.apache.mxnet' with the
> >> > artifact
> >> > > as 'mxnet-incubating'.
> >> > >
> >> > > On Sun, Mar 11, 2018 at 4:44 PM, kellen sunderland <
> >> > > kellen.sunderl...@gmail.com> wrote:
> >> > >
> >> > > > YiZhi, In general I agree that your points and examples are the
> >> ideal
> >> > > > case, but in the MXNet situation there are some trade-offs we have
> >> to
> >> > > > make.  Let me try to specifically answer your points:
> >> > > >
> >> > > > "Do you mean we have different version for 'ml.dmlc' namespace and
> >> > > > 'org.apache' namespace?"
> >> > > > No I am not trying to saying that. I believe Marco, Naveen and I
> are
> >> > all
> >> > > > proposing we use a single org.apache.incubating.mxnet namespace
> >> moving
> >> > > > forward, which would require a major version change to our product
> >> API
> >> > > > under our current versioning scheme.  Marco and I are proposing we
> >> > apply
> >> > > > this MV change _only_ to the scala package's API.
> >> > > >
> >> > > > "How to tell which Scala API version works with which MXNet core
> >> > version?
> >> > > > By document?"
> >> > > > Yes users will be able to tell via the website, release docs,
> maven
> >> > > > package information, pom file, etc.
> >> > > >
> >> > > > "How many users will read the whole document and carefully pair
> the
> >> > > > version id before they run into a strange error and give up?"
> >> > > > They won't get a strange error, assuming we're talking about Scala
> >> > users
> >> > > > who are upgrading from a package with the same namespace they will
> >> rely
> >> > > on
> >> > > > the package manager to give them an update which should be
> painless.
> >> > > >
> >> > > > Secondly software developers understand that packages, not
> products,
> >> > have
> >> > > > versions.  They know that these versions are used to communicate
> >> when
> >> > > APIs
> >> > > > are broken.  There's examples of Apache packages doing this for
> >> > packages
> >> > > > that include multiple interfaces, for example first-party modules
> >> > > packaged
> >> > > > with the HTTP server, or log4j's language bindings (arguably quite
> >> > > similar
> >> > > > to what Naveen is doing).
> >> > > >
> >> > > > While we can debate the right way to version packages, I think
> >> there's
> >> > a
> >> > > > clear community decision here to get Naveen unblocked:
> >> > > >
> >> > > > (1) We continue semantically versioning across all APIs, meaning
> >> that
> >> > > this
> >> > > > change would get released with MXNet 2.*.
> >> > > > (2) You version package interfaces semantically and have a
> >> compatible
> >> > > > version mapping.
> >> > > > (3) Status quo, we continue to release a Scala package as-is,
> >> breaking
> >> > > > apache guidelines for artifact generation.
> >> > > > (4) We rely on the namespace change itself to communicate a change
> >> in
> >> > the
> >> > > > interface.  We don't consider this a major change.
> >> > > >
> >> > > > My (non-binding) preference would be for option 2.
> >> > > >
> >> > > > -Kellen
> >> > > >
> >> > > > On Sun, Mar 11, 2018 at 12:44 PM, Marco de Abreu <
> >> > > > marco.g.ab...@googlemail.com> wrote:
> >> > > >
> >> > > >> Changing namespaces is one example of a required major version
> >> change,
> >> > > but
> >> > > >> there are more reasons like general refactoring or some
> deprecated
> >> > APIs
> >> > > >> just being hard to maintain. Things like these happen quite
> >> frequently
> >> > > and
> >> > > >> it's a problem every software project has to face and find a
> >> solution
> >> > > for.
> >> > > >>
> >> > > >> Regarding ' How to tell which Scala API version works with which
> >> MXNet
> >> > > >> core
> >> > > >> version?': We could just bundle MXNet with the released API
> >> package as
> >> > > we
> >> > > >> do right now, but we would give each interface it's own version
> and
> >> > > >> publish
> >> > > >> them on their distribution platforms accordingly. Just an
> example:
> >> > > >> >Scala-Package -> MXNet-Version
> >> > > >> >> 1.0 -> 1.0
> >> > > >> >> 1.1 -> 1.1
> >> > > >> >> 2.0 -> 1.2
> >> > > >> >> 2.1 -> 1.3
> >> > > >> >> 3.0 -> 2.0
> >> > > >>
> >> > > >> > R-Package -> MXNet-Version
> >> > > >> >> 1.0 -> 1.0
> >> > > >> >> 2.0 -> 1.1
> >> > > >> >> 2.1 -> 1.2
> >> > > >> >> 2.2 -> 1.3
> >> > > >> >> 3.0 -> 2.0
> >> > > >>
> >> > > >> This is always an N-to-1 mapping, while N being the versions of
> our
> >> > APIs
> >> > > >> and 1 the MXNet Core version. From MXNets versioning perspective,
> >> this
> >> > > >> would then looking the following:
> >> > > >> > MXNet-Version -> APIs
> >> > > >> >> 1.0 -> Scala_1.0; R_1.0
> >> > > >> >> 1.1 -> Scala_1.1; R_2.0
> >> > > >> >> 1.2 -> Scala_2.0; R_2.1
> >> > > >> >> 1.3 -> Scala_2.1; R_2.2
> >> > > >> >> 2.0 -> Scala_3.0; R_3.0
> >> > > >>
> >> > > >> This would give us the liberty to develop MXNet without
> >> restricting us
> >> > > too
> >> > > >> much - of course, major version increments will still have to be
> >> > > >> considered
> >> > > >> carefully. I don't think that this would harm transparency too
> much
> >> > and
> >> > > >> there's no need to write big documentation.
> >> > > >>
> >> > > >> -Marco
> >> > > >>
> >> > > >>
> >> > > >> On Sun, Mar 11, 2018 at 12:16 PM, YiZhi Liu <liuyi...@apache.org
> >
> >> > > wrote:
> >> > > >>
> >> > > >> > I have no idea how separating Scala API version can solve the
> >> > > >> > 'compatibility' problem. Do you mean we have different version
> >> for
> >> > > >> > 'ml.dmlc' namespace and 'org.apache' namespace? Do these two
> >> > versions
> >> > > >> > have same behavior? How to tell which Scala API version works
> >> with
> >> > > >> > which MXNet core version? By document? How many users will read
> >> the
> >> > > >> > whole document and carefully pair the version id before they
> run
> >> > into
> >> > > >> > a strange error and give up?
> >> > > >> >
> >> > > >> > Moreover, changing namespace is an issue that is really rare
> and
> >> > > >> > hardly happens. For other 'compatibility' problem, for example,
> >> the
> >> > > >> > class/function definitions, should handle the compatibility
> >> itself.
> >> > > >> > You'll never expect a project to have a different version for
> >> > changing
> >> > > >> > 'calculate(int)' to 'calculate(float)', it should just add a
> new
> >> > > >> > function 'calculate(float)'.
> >> > > >> >
> >> > > >> > Regarding 'In this case the Scala interface is clearly a
> separate
> >> > > >> > entity from the C API.'. Everything can be seen as a separate
> >> > entity,
> >> > > >> > the mxnet engine, the graph description, operators, python API,
> >> > gluon
> >> > > >> > API, etc. We should think carefully what we want to provide,
> and
> >> > what
> >> > > >> > our users need.
> >> > > >> >
> >> > > >> > As an example, Apache Spark, still has SparkR (R API), PySpark
> >> > (Python
> >> > > >> > API), MLLib, GraphX ... as part of its release, and have the
> same
> >> > > >> > version as Spark core as well as its Scala/Java API.
> >> > > >> >
> >> > > >> > 2018-03-10 23:58 GMT-08:00 kellen sunderland <
> >> > > >> kellen.sunderl...@gmail.com
> >> > > >> > >:
> >> > > >> > > +1 (non-binding) to what Marco is describing.  +1
> >> (non-binding) to
> >> > > >> > getting the Scala bindings with the namespace change into
> Maven.
> >> > > >> > >
> >> > > >> > > The general best practice for SemVer, which is used by most
> >> > projects
> >> > > >> > that employ SemVer, is to apply SemVer to the public APIs of
> >> > packages
> >> > > >> that
> >> > > >> > ship with your project.  If you have several independent APIs
> >> this
> >> > > could
> >> > > >> > mean that they are versioned separately from each other, and
> from
> >> > the
> >> > > >> > overall project versioning mechanism.
> >> > > >> > >
> >> > > >> > > For example, the .NET Core library ships with a number of
> >> > binaries,
> >> > > >> each
> >> > > >> > with their own SemVerioned APIs.  They also have a high-level,
> >> easy
> >> > to
> >> > > >> > understand version for the package as a whole:
> >> > > >> > https://docs.microsoft.com/en-us/dotnet/core/versions/.
> >> > > >> > >
> >> > > >> > > Nodesource has a good description of this:
> >> > > >> http://nodesource.com/blog/
> >> > > >> > semver-a-primer/
> >> > > >> > > “Semver is a scheme for interface versioning for the benefit
> of
> >> > > >> > interface consumers, thus if a tool has multiple interfaces,
> >> e.g. an
> >> > > API
> >> > > >> > and a CLI, these interfaces may evolve independent versioning.”
> >> > > >> > >
> >> > > >> > > SemVer at its core is a communication mechanism to inform
> >> > developers
> >> > > >> of
> >> > > >> > incompatibilities. In this case the Scala interface is clearly
> a
> >> > > >> separate
> >> > > >> > entity from the C API.  I.e. changing the Scala namespace isn’t
> >> > going
> >> > > to
> >> > > >> > break C API users.  It does not communicate anything useful to
> >> these
> >> > > >> users
> >> > > >> > if we up their major version in response to a Scala change, it
> >> > simply
> >> > > >> > breaks compatibility.  If we group all interfaces together, and
> >> > > >> increment
> >> > > >> > whenever any of them has a breaking change we’ll soon be at
> MXNet
> >> > > >> version
> >> > > >> > 587.  We’ll be forcing our users to check compatibility and
> >> update
> >> > > their
> >> > > >> > dependency tracking constantly.  The end result is that our
> users
> >> > will
> >> > > >> stop
> >> > > >> > pulling in new versions of the library.
> >> > > >> > >
> >> > > >> > > What I would propose is that (1) we have a high-level SemVer
> >> > system
> >> > > >> that
> >> > > >> > tracks our C_API.  This is the ‘MXNet’ version that we
> generally
> >> > refer
> >> > > >> to
> >> > > >> > and emphasize for our public releases.  For each API we have an
> >> > > >> independent
> >> > > >> > versioning system that if we can, we fix to the MXNet version.
> >> When
> >> > > it
> >> > > >> > makes sense we version these APIs independently.  So for
> example
> >> we
> >> > > >> could
> >> > > >> > have a MXNet 1.2 release that ships with a 2.0 Scala API / R
> API.
> >> > > >> > >
> >> > > >> > > In terms of Apache process I think shipping artifacts with a
> >> > > >> non-Apache
> >> > > >> > namespace is a bigger issue than whatever versioning
> conventions
> >> we
> >> > > >> decide
> >> > > >> > to use.
> >> > > >> > >
> >> > > >> > > -Kellen
> >> > > >> > >
> >> > > >> > > From: Carin Meier
> >> > > >> > > Sent: Saturday, March 10, 2018 1:41 PM
> >> > > >> > > To: dev@mxnet.incubator.apache.org
> >> > > >> > > Cc: d...@mxnet.apache.org
> >> > > >> > > Subject: Re: Publishing Scala Package/namespace change
> >> > > >> > >
> >> > > >> > > +1 as well. I'm actively developing a Clojure package for
> MXNet
> >> > that
> >> > > >> uses
> >> > > >> > > the jars from the Scala package.
> >> > > >> > >
> >> > > >> > > - Carin
> >> > > >> > >
> >> > > >> > > On Fri, Mar 9, 2018 at 4:44 PM, YiZhi Liu <
> eazhi....@gmail.com
> >> >
> >> > > >> wrote:
> >> > > >> > >
> >> > > >> > >> +1 for changing the namespace asap. for the maven deploy, we
> >> can
> >> > > have
> >> > > >> > >> it build along with pip deployment.
> >> > > >> > >>
> >> > > >> > >>
> >> > > >> > >> 2018-03-09 10:15 GMT-08:00 Naveen Swamy <mnnav...@gmail.com
> >:
> >> > > >> > >> > Hi Guys,
> >> > > >> > >> >
> >> > > >> > >> > I am working on MXNet Scala Inference APIs
> >> > > >> > >> > <https://issues.apache.org/jira/browse/MXNET-50> along
> with
> >> > > >> another
> >> > > >> > >> > contributor Roshani. A while back I noticed that we
> haven't
> >> > been
> >> > > >> > >> publishing
> >> > > >> > >> > the scala package to Maven for a while now(last one being
> >> > > v0.11.1a
> >> > > >> > under
> >> > > >> > >> > the dmlc namespace).
> >> > > >> > >> > Currently users have to build the package manually and
> then
> >> use
> >> > > it,
> >> > > >> > this
> >> > > >> > >> > hinders adoption and also is painful to build everything
> >> from
> >> > > >> source.
> >> > > >> > >> >
> >> > > >> > >> > I also see that we haven't changed the namespace to
> >> org.apache
> >> > > and
> >> > > >> > >> instead
> >> > > >> > >> > are still ml.dmlc namespace.
> >> > > >> > >> >
> >> > > >> > >> > I wanted to seek your opinion about changing the
> MXNet-Scala
> >> > > >> package
> >> > > >> > >> > namespace to org.apache for the Scala package and publish
> to
> >> > > Maven
> >> > > >> in
> >> > > >> > the
> >> > > >> > >> > upcoming release. I understand that this probably breaks
> the
> >> > > Semver
> >> > > >> > >> > semantics that is agreed upon, However I would like to
> point
> >> > out
> >> > > >> that
> >> > > >> > the
> >> > > >> > >> > Scala package has never been published to maven as 1.0
> under
> >> > > >> > org.apache.
> >> > > >> > >> >
> >> > > >> > >> > Open to suggestions.
> >> > > >> > >> >
> >> > > >> > >> > Thanks, Naveen
> >> > > >> > >>
> >> > > >> > >>
> >> > > >> > >>
> >> > > >> > >> --
> >> > > >> > >> Yizhi Liu
> >> > > >> > >> DMLC member
> >> > > >> > >> Amazon Web Services
> >> > > >> > >> Vancouver, Canada
> >> > > >> > >>
> >> > > >> > >
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > --
> >> > > >> > Yizhi Liu
> >> > > >> > DMLC member
> >> > > >> > Amazon Web Services
> >> > > >> > Vancouver, Canada
> >> > > >> >
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Yizhi Liu
> >> DMLC member
> >> Amazon Web Services
> >> Vancouver, Canada
> >>
> >
> >
>



-- 
Yizhi Liu
DMLC member
Amazon Web Services
Vancouver, Canada

Reply via email to