Hi Konstantin,

Thanks for your reply.

> I personally prefer small incremental changes, and starting with some MVP, 
> where M is
> minimal, meaning least possible effort (e.g. number of dependencies) used
> from conan.

Agreed. That said, this seems like a case where adoption decision can't be 
based on MVP, as having one additional dependency to be able to automatically 
download 3 out of some 15 dependencies doesn't seem to be a desirable state to 
be in.
A feature branch is created for you and others for collaboration so that we can 
not only ensure the coverage but also ensure that there's enough people who are 
willing to push this forward.

> may we define an actual scope then

As mentioned in my last reply, I'd recommend solving one of the two main use 
cases of mxnet builds. The efforts should also make it possible to remove these 
scripts here: https://github.com/apache/incubator-mxnet/tree/master/setup-utils.

> but, I am not sure, for me it looks like strange use-case. I can imagine
> you have developers with no internet connection and send them parcels with
> CDs of mxnet source code, but I think it's hard to manage such workflow
> with any tool, no matter submodules, CMake download or conan.

Thanks for the clarificiation. This use case is acutally not that rare. People 
may need to build mxnet to optimize for performance on their custom hardware, 
in a sandboxed environment for security reasons. Submodules can solve it by 
including the source code. Since the offline use case is not what conan is 
designed for, we just need to make sure that conan is not a required build 
tool, and alternatives still work in this use case. I made relevant comment in 
the PR too.

To sum up, I think conan is a tool that can simplify the dependency management 
in mxnet from a concept level. We should make sure that it can provide the 
coverage needed (e.g. have the packages we need with the versions we need and 
fast turnaround for upgrading dependencies), and make sure it doesn't break 
other build needs. For now let's utilize the feature branch to collaborate with 
people who share the interest and are willing to help out with the goal to make 
sure we have a full coverage at the time of adoption.

Best,
-sz

On 2019/05/06 08:10:29, Konstantin Ivlev <tomsks...@gmail.com> wrote: 
> Hi Sheng Zha,
> 
> >  Currently, the linked PR only includes OpenBLAS
> actually, it's OpenBLAS + OpenCV + lapack, three libraries as
> proof-of-concept
> 
> > A proof-of-concept that shows it actually replaces more dependency than
> openblas would be helpful
> may we define an actual scope then, e.g. how much dependencies - 2, 3, 5,
> half of them or all of them?
> 
> > If the value proposition of conan is to simplify the dependency
> management, then it should unify other solutions instead of keeping these
> solutions around.
> personally, I don't think it's easy to do in single shot. I personally
> prefer small incremental changes, and starting with some MVP, where M is
> minimal, meaning least possible effort (e.g. number of dependencies) used
> from conan. then eventually migrate other dependencies to conan one by one,
> until it fully migrates. but that's just my personal vision, if you see
> that this strategy is wrong, it's okay.
> 
> - It's unclear how it impacts people with only an archive of mxnet source
> code but without network access.
> as I have tried, conan doesn't change that much. if you have no network
> access, and only mxnet source code, then "git submodule init" will fail for
> you. if you have also archives of dependencies somewhere on your
> hard-drive, you can manage submodules to clone from the local repository
> rather GitHub. but then you will stuck at the CMake generation step, which
> will fail to download Intel MKL.
> if you use conan instead of submodules and CMake download, there is not
> much difference - it will fail to fetch same dependencies without internet
> connection on clean machine. if you somehow happen to have ab archive of
> conan cache on your hard-drive, you may unpack it and use without internet
> connection.
> but, I am not sure, for me it looks like strange use-case. I can imagine
> you have developers with no internet connection and send them parcels with
> CDs of mxnet source code, but I think it's hard to manage such workflow
> with any tool, no matter submodules, CMake download or conan.
> 
> yours sincerely, Konstantin
> 
> 
> вс, 5 мая 2019 г. в 06:25, Sheng Zha <zhash...@apache.org>:
> 
> > To be clear, my intention is really to prevent a seemingly good solution
> > to exacerbate the problem that it sets out to solve. This tends to happen
> > when there are not enough people to drive it to the end.
> >
> > If there are additional values in this solution that people feel outweighs
> > the problems below, I'd be more than happy to be persuaded to vote
> > otherwise.
> >
> > -sz
> >
> > On 2019/05/04 23:08:43, Sheng Zha <zhash...@apache.org> wrote:
> > > Thank you for the explanation and sorry that I missed the earlier
> > context as it has been a while. While I like the idea of simplifying the
> > dependency management with tools like conan, I have the following concerns
> > on this vote as-is (it's also my take on why I think the PR is stuck):
> > >
> > > - It's unclear how much dependency needs can conan help in mxnet builds.
> > >   Currently, the linked PR only includes OpenBLAS. A proof-of-concept
> > that shows it actually replaces more dependency than openblas would be
> > helpful. On a high-level, there are two types of builds for mxnet:
> > >   * User's custom build-from-source: 1) usually dynamic linking is used.
> > 2) users may not enable all features, and users may want to pull a subset
> > of the dependencies. 3) users may want mxnet build system to pull the
> > dependencies, or they may not. (for conan it's ok to just focus on the
> > former)
> > >   * Binary distribution for pip and maven: 1) static linking is used
> > (requires -fPIC). 2) all features are enabled. 3) dependencies are pulled
> > in with scripts in mxnet.
> > >   Handling one of the above cases would be a good showcase for the value
> > of the new tool.
> > >
> > > - It's unclear how it impacts people with only an archive of mxnet
> > source code but without network access.
> > >   This applies for the dependencies that are captured as submodules that
> > you mentioned as a way that mxnet manages dependency.
> > >
> > > - If the value proposition of conan is to simplify the dependency
> > management, then it should unify other solutions instead of keeping these
> > solutions around.
> > >
> > > Overall, it would be helpful to have a clear message such as what
> > exactly conan can replace, and having a proof of concept that works for
> > this would be helpful. Otherwise, I fear that we may be introducing yet
> > another way to manage dependency that further complicates the existing
> > problem.
> > >
> > > That said, I'm not suggesting that we impose the burden to implement
> > everything on you alone, and it's ok to rally people who are interested in
> > this solution to help out. To facilitate this, I created a feature branch
> > so that it's easier for you and people who are enthusiastic about this to
> > work together [1].
> > >
> > > For now, I'm voting -1 to this proposal and I hope you understand.
> > >
> > > -sz
> > >
> > > [1] https://github.com/apache/incubator-mxnet/tree/conan
> > >
> > > On 2019/05/03 07:51:34, Konstantin Ivlev <tomsks...@gmail.com> wrote:
> > > > hi Sheng Zha,
> > > >
> > > > on pull request review I was told by Anirudh anirudhacharya and Roshani
> > > > Nagmote to start discussion/vote on the mxnet dev list. it seems to be
> > a
> > > > vicious circle now - on GitHub I am told to use vote, and on vote I am
> > told
> > > > to use GitHub, this doesn't help much.
> > > > FYI GitHub review stuck, it's already opened since November 2018, and
> > it's
> > > > still not approved (however, there were no objections during the
> > review).
> > > > Previous discussion in e-mail thread also didn't encounter any
> > objections,
> > > > and all questions were answered.
> > > > JIRA ticket has no discussion at all (except it has duplicates of
> > comments
> > > > made on GitHub).
> > > > so let's process with 3-day vote for now, as other communication
> > channels
> > > > were already tried with no success.
> > > >
> > > > yours sincerely, Konstantin
> > > >
> > > > пт, 3 мая 2019 г. в 14:17, Sheng Zha <zhash...@apache.org>:
> > > >
> > > > > Hi Konstantin,
> > > > >
> > > > > While conan looks like an option that's worth exploring, given that
> > your
> > > > > request is to merge the pull request, I'd suggest that the request
> > should
> > > > > go through the regular pull request review and it doesn't really
> > need a
> > > > > vote (as it doesn't substitute reviews anyway)
> > > > >
> > > > > If you would like to gather more attention to it, feel free to ping
> > in a
> > > > > discussion thread.
> > > > >
> > > > > -sz
> > > > >
> > > > > On 2019/05/03 06:29:55, Konstantin Ivlev <tomsks...@gmail.com>
> > wrote:
> > > > > > Dear MXNet community,
> > > > > >
> > > > > > This is the 3-day vote to add conan support for Apache MXNet
> > (incubating)
> > > > > > version v1.4.1.
> > > > > > The voting on dev@ list will start May 03 23:59:59 (PST) and
> > close on
> > > > > May
> > > > > > 06 23:59:59.
> > > > > >
> > > > > > Background: conan is open-source, freeware, cross-platform package
> > > > > manager
> > > > > > for C and C++ projects, written in python. it provides integration
> > with
> > > > > > various build systems, include CMake. conan may use bintray as a
> > server
> > > > > to
> > > > > > store and download pre-built packages, or packages might be always
> > built
> > > > > > from sources.
> > > > > >
> > > > > > Problem: currently (as for v1.4.1), Apache MXNet (incubating) is
> > using
> > > > > > several ways to fetch 3rd-party dependencies simultaneously, for
> > > > > instance:
> > > > > > 1. download GitHub archives during the build
> > > > > > - OpenBLAS
> > > > > > - OpenCV
> > > > > > 2. conda (alternative way to GitHub archives)
> > > > > > 3. download from CMake
> > > > > > - Intel Math Kernel Library (MKL)
> > > > > > 4. Git submodules
> > > > > > - cub
> > > > > > - dlpack
> > > > > > - dmlc-core
> > > > > > - googletest
> > > > > > - mkldnn
> > > > > > - mshadow
> > > > > > - onnx-tensorrt
> > > > > > - openmp
> > > > > > - ps-lite
> > > > > > - tvm
> > > > > > therefore, there are multiple places to look for 3rd parties, and
> > its
> > > > > hard
> > > > > > to update them, as you need to remember or figure it out how to
> > update a
> > > > > > particular dependency to newer version, for instance.
> > > > > > current Apache MXNet (incubating) build instructions differ very
> > much per
> > > > > > platform, require to download and unzip some archives manually,
> > > > > specifying
> > > > > > variables with paths to this archives, in conjunction of updating
> > git
> > > > > > submodules,
> > > > > >
> > > > > > Action: merge pull request providing an initial conan support for
> > Apache
> > > > > > MXNet (incubating). support conan as an alternate approach to fetch
> > > > > various
> > > > > > 3rd-party dependencies. old approaches will be still available,
> > supported
> > > > > > and left intact.
> > > > > >
> > > > > > Below are links to
> > > > > > 1) conan web-site:  https://conan.io/
> > > > > > 2) conan GitHub repository: https://github.com/conan-io/conan
> > > > > > 3) conan documentation: https://docs.conan.io/en/latest/
> > > > > > 4) bintray: https://bintray.com
> > > > > > 5) pull request adding conan support to Apache MXNet (incubating):
> > > > > > https://github.com/apache/incubator-mxnet/pull/13400
> > > > > > 6) JIRA issue: https://issues.apache.org/jira/browse/MXNET-1229
> > > > > > 7) previous email discussion:
> > > > > >
> > > > >
> > https://lists.apache.org/thread.html/301a46a637f7e3c249c475713f701bef7530c32bc92d8834c0882897@%3Cdev.mxnet.apache.org%3E
> > > > > > 8) MXNet build instructions:
> > > > > > https://mxnet-tqchen.readthedocs.io/en/latest/how_to/build.html
> > > > > > 9) MXNet build instructions (Windows):
> > > > > >
> > > > >
> > https://mxnet.incubator.apache.org/versions/master/install/windows_setup.html
> > > > > > 10) MXNet build instructions (OSX):
> > > > > >
> > http://mxnet.incubator.apache.org/versions/master/install/osx_setup.html
> > > > > > 11) MXNet build instructions (Linux):
> > > > > >
> > > > >
> > http://mxnet.incubator.apache.org/versions/master/install/ubuntu_setup.html
> > > > > > 12) MXNet development setup (OSX):
> > > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+Developer+Setup+on+Mac
> > > > > >
> > > > > > Please remember to TEST first before voting accordingly:
> > > > > > +1 = approve
> > > > > > +0 = no opinion
> > > > > > -1 = disapprove (provide reason)
> > > > > >
> > > > > > Best regards,
> > > > > > Konstantin Ivlev
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Reply via email to