Re: [VOTE] add conan support for Apache MXNet (incubating)
Hi Konstantin, Thanks for your reply. > I personally prefer small incremental changes, and starting with some MVP, > where M is > minimal, meaning least possible effort (e.g. number of dependencies) used > from conan. Agreed. That said, this seems like a case where adoption decision can't be based on MVP, as having one additional dependency to be able to automatically download 3 out of some 15 dependencies doesn't seem to be a desirable state to be in. A feature branch is created for you and others for collaboration so that we can not only ensure the coverage but also ensure that there's enough people who are willing to push this forward. > may we define an actual scope then As mentioned in my last reply, I'd recommend solving one of the two main use cases of mxnet builds. The efforts should also make it possible to remove these scripts here: https://github.com/apache/incubator-mxnet/tree/master/setup-utils. > but, I am not sure, for me it looks like strange use-case. I can imagine > you have developers with no internet connection and send them parcels with > CDs of mxnet source code, but I think it's hard to manage such workflow > with any tool, no matter submodules, CMake download or conan. Thanks for the clarificiation. This use case is acutally not that rare. People may need to build mxnet to optimize for performance on their custom hardware, in a sandboxed environment for security reasons. Submodules can solve it by including the source code. Since the offline use case is not what conan is designed for, we just need to make sure that conan is not a required build tool, and alternatives still work in this use case. I made relevant comment in the PR too. To sum up, I think conan is a tool that can simplify the dependency management in mxnet from a concept level. We should make sure that it can provide the coverage needed (e.g. have the packages we need with the versions we need and fast turnaround for upgrading dependencies), and make sure it doesn't break other build needs. For now let's utilize the feature branch to collaborate with people who share the interest and are willing to help out with the goal to make sure we have a full coverage at the time of adoption. Best, -sz On 2019/05/06 08:10:29, Konstantin Ivlev wrote: > Hi Sheng Zha, > > > Currently, the linked PR only includes OpenBLAS > actually, it's OpenBLAS + OpenCV + lapack, three libraries as > proof-of-concept > > > A proof-of-concept that shows it actually replaces more dependency than > openblas would be helpful > may we define an actual scope then, e.g. how much dependencies - 2, 3, 5, > half of them or all of them? > > > If the value proposition of conan is to simplify the dependency > management, then it should unify other solutions instead of keeping these > solutions around. > personally, I don't think it's easy to do in single shot. I personally > prefer small incremental changes, and starting with some MVP, where M is > minimal, meaning least possible effort (e.g. number of dependencies) used > from conan. then eventually migrate other dependencies to conan one by one, > until it fully migrates. but that's just my personal vision, if you see > that this strategy is wrong, it's okay. > > - It's unclear how it impacts people with only an archive of mxnet source > code but without network access. > as I have tried, conan doesn't change that much. if you have no network > access, and only mxnet source code, then "git submodule init" will fail for > you. if you have also archives of dependencies somewhere on your > hard-drive, you can manage submodules to clone from the local repository > rather GitHub. but then you will stuck at the CMake generation step, which > will fail to download Intel MKL. > if you use conan instead of submodules and CMake download, there is not > much difference - it will fail to fetch same dependencies without internet > connection on clean machine. if you somehow happen to have ab archive of > conan cache on your hard-drive, you may unpack it and use without internet > connection. > but, I am not sure, for me it looks like strange use-case. I can imagine > you have developers with no internet connection and send them parcels with > CDs of mxnet source code, but I think it's hard to manage such workflow > with any tool, no matter submodules, CMake download or conan. > > yours sincerely, Konstantin > > > вс, 5 мая 2019 г. в 06:25, Sheng Zha : > > > To be clear, my intention is really to prevent a seemingly good solution > > to exacerbate the problem that it sets out to solve. This tends to happen > > when there are not enough people to drive it to the end. > > > > If there are additional values in this solution that people feel outweighs > > the problems below, I'd be more than happy to be persuaded to vote > > otherwise. > > > > -sz > > > > On 2019/05/04 23:08:43, Sheng Zha wrote: > > > Thank you for the explanation and sorry that I missed the earlier > > context as it
Re: [VOTE] add conan support for Apache MXNet (incubating)
Hi Sheng Zha, > Currently, the linked PR only includes OpenBLAS actually, it's OpenBLAS + OpenCV + lapack, three libraries as proof-of-concept > A proof-of-concept that shows it actually replaces more dependency than openblas would be helpful may we define an actual scope then, e.g. how much dependencies - 2, 3, 5, half of them or all of them? > If the value proposition of conan is to simplify the dependency management, then it should unify other solutions instead of keeping these solutions around. personally, I don't think it's easy to do in single shot. I personally prefer small incremental changes, and starting with some MVP, where M is minimal, meaning least possible effort (e.g. number of dependencies) used from conan. then eventually migrate other dependencies to conan one by one, until it fully migrates. but that's just my personal vision, if you see that this strategy is wrong, it's okay. - It's unclear how it impacts people with only an archive of mxnet source code but without network access. as I have tried, conan doesn't change that much. if you have no network access, and only mxnet source code, then "git submodule init" will fail for you. if you have also archives of dependencies somewhere on your hard-drive, you can manage submodules to clone from the local repository rather GitHub. but then you will stuck at the CMake generation step, which will fail to download Intel MKL. if you use conan instead of submodules and CMake download, there is not much difference - it will fail to fetch same dependencies without internet connection on clean machine. if you somehow happen to have ab archive of conan cache on your hard-drive, you may unpack it and use without internet connection. but, I am not sure, for me it looks like strange use-case. I can imagine you have developers with no internet connection and send them parcels with CDs of mxnet source code, but I think it's hard to manage such workflow with any tool, no matter submodules, CMake download or conan. yours sincerely, Konstantin вс, 5 мая 2019 г. в 06:25, Sheng Zha : > To be clear, my intention is really to prevent a seemingly good solution > to exacerbate the problem that it sets out to solve. This tends to happen > when there are not enough people to drive it to the end. > > If there are additional values in this solution that people feel outweighs > the problems below, I'd be more than happy to be persuaded to vote > otherwise. > > -sz > > On 2019/05/04 23:08:43, Sheng Zha wrote: > > Thank you for the explanation and sorry that I missed the earlier > context as it has been a while. While I like the idea of simplifying the > dependency management with tools like conan, I have the following concerns > on this vote as-is (it's also my take on why I think the PR is stuck): > > > > - It's unclear how much dependency needs can conan help in mxnet builds. > > Currently, the linked PR only includes OpenBLAS. A proof-of-concept > that shows it actually replaces more dependency than openblas would be > helpful. On a high-level, there are two types of builds for mxnet: > > * User's custom build-from-source: 1) usually dynamic linking is used. > 2) users may not enable all features, and users may want to pull a subset > of the dependencies. 3) users may want mxnet build system to pull the > dependencies, or they may not. (for conan it's ok to just focus on the > former) > > * Binary distribution for pip and maven: 1) static linking is used > (requires -fPIC). 2) all features are enabled. 3) dependencies are pulled > in with scripts in mxnet. > > Handling one of the above cases would be a good showcase for the value > of the new tool. > > > > - It's unclear how it impacts people with only an archive of mxnet > source code but without network access. > > This applies for the dependencies that are captured as submodules that > you mentioned as a way that mxnet manages dependency. > > > > - If the value proposition of conan is to simplify the dependency > management, then it should unify other solutions instead of keeping these > solutions around. > > > > Overall, it would be helpful to have a clear message such as what > exactly conan can replace, and having a proof of concept that works for > this would be helpful. Otherwise, I fear that we may be introducing yet > another way to manage dependency that further complicates the existing > problem. > > > > That said, I'm not suggesting that we impose the burden to implement > everything on you alone, and it's ok to rally people who are interested in > this solution to help out. To facilitate this, I created a feature branch > so that it's easier for you and people who are enthusiastic about this to > work together [1]. > > > > For now, I'm voting -1 to this proposal and I hope you understand. > > > > -sz > > > > [1] https://github.com/apache/incubator-mxnet/tree/conan > > > > On 2019/05/03 07:51:34, Konstantin Ivlev wrote: > > > hi Sheng Zha, > > > > > > on pull request review I was to
Re: [VOTE] add conan support for Apache MXNet (incubating)
To be clear, my intention is really to prevent a seemingly good solution to exacerbate the problem that it sets out to solve. This tends to happen when there are not enough people to drive it to the end. If there are additional values in this solution that people feel outweighs the problems below, I'd be more than happy to be persuaded to vote otherwise. -sz On 2019/05/04 23:08:43, Sheng Zha wrote: > Thank you for the explanation and sorry that I missed the earlier context as > it has been a while. While I like the idea of simplifying the dependency > management with tools like conan, I have the following concerns on this vote > as-is (it's also my take on why I think the PR is stuck): > > - It's unclear how much dependency needs can conan help in mxnet builds. > Currently, the linked PR only includes OpenBLAS. A proof-of-concept that > shows it actually replaces more dependency than openblas would be helpful. On > a high-level, there are two types of builds for mxnet: > * User's custom build-from-source: 1) usually dynamic linking is used. 2) > users may not enable all features, and users may want to pull a subset of the > dependencies. 3) users may want mxnet build system to pull the dependencies, > or they may not. (for conan it's ok to just focus on the former) > * Binary distribution for pip and maven: 1) static linking is used > (requires -fPIC). 2) all features are enabled. 3) dependencies are pulled in > with scripts in mxnet. > Handling one of the above cases would be a good showcase for the value of > the new tool. > > - It's unclear how it impacts people with only an archive of mxnet source > code but without network access. > This applies for the dependencies that are captured as submodules that you > mentioned as a way that mxnet manages dependency. > > - If the value proposition of conan is to simplify the dependency management, > then it should unify other solutions instead of keeping these solutions > around. > > Overall, it would be helpful to have a clear message such as what exactly > conan can replace, and having a proof of concept that works for this would be > helpful. Otherwise, I fear that we may be introducing yet another way to > manage dependency that further complicates the existing problem. > > That said, I'm not suggesting that we impose the burden to implement > everything on you alone, and it's ok to rally people who are interested in > this solution to help out. To facilitate this, I created a feature branch so > that it's easier for you and people who are enthusiastic about this to work > together [1]. > > For now, I'm voting -1 to this proposal and I hope you understand. > > -sz > > [1] https://github.com/apache/incubator-mxnet/tree/conan > > On 2019/05/03 07:51:34, Konstantin Ivlev wrote: > > hi Sheng Zha, > > > > on pull request review I was told by Anirudh anirudhacharya and Roshani > > Nagmote to start discussion/vote on the mxnet dev list. it seems to be a > > vicious circle now - on GitHub I am told to use vote, and on vote I am told > > to use GitHub, this doesn't help much. > > FYI GitHub review stuck, it's already opened since November 2018, and it's > > still not approved (however, there were no objections during the review). > > Previous discussion in e-mail thread also didn't encounter any objections, > > and all questions were answered. > > JIRA ticket has no discussion at all (except it has duplicates of comments > > made on GitHub). > > so let's process with 3-day vote for now, as other communication channels > > were already tried with no success. > > > > yours sincerely, Konstantin > > > > пт, 3 мая 2019 г. в 14:17, Sheng Zha : > > > > > Hi Konstantin, > > > > > > While conan looks like an option that's worth exploring, given that your > > > request is to merge the pull request, I'd suggest that the request should > > > go through the regular pull request review and it doesn't really need a > > > vote (as it doesn't substitute reviews anyway) > > > > > > If you would like to gather more attention to it, feel free to ping in a > > > discussion thread. > > > > > > -sz > > > > > > On 2019/05/03 06:29:55, Konstantin Ivlev wrote: > > > > Dear MXNet community, > > > > > > > > This is the 3-day vote to add conan support for Apache MXNet > > > > (incubating) > > > > version v1.4.1. > > > > The voting on dev@ list will start May 03 23:59:59 (PST) and close on > > > May > > > > 06 23:59:59. > > > > > > > > Background: conan is open-source, freeware, cross-platform package > > > manager > > > > for C and C++ projects, written in python. it provides integration with > > > > various build systems, include CMake. conan may use bintray as a server > > > to > > > > store and download pre-built packages, or packages might be always built > > > > from sources. > > > > > > > > Problem: currently (as for v1.4.1), Apache MXNet (incubating) is using > > > > several ways to fetch 3rd-party dependencies simultaneously, for
Re: [VOTE] add conan support for Apache MXNet (incubating)
Thank you for the explanation and sorry that I missed the earlier context as it has been a while. While I like the idea of simplifying the dependency management with tools like conan, I have the following concerns on this vote as-is (it's also my take on why I think the PR is stuck): - It's unclear how much dependency needs can conan help in mxnet builds. Currently, the linked PR only includes OpenBLAS. A proof-of-concept that shows it actually replaces more dependency than openblas would be helpful. On a high-level, there are two types of builds for mxnet: * User's custom build-from-source: 1) usually dynamic linking is used. 2) users may not enable all features, and users may want to pull a subset of the dependencies. 3) users may want mxnet build system to pull the dependencies, or they may not. (for conan it's ok to just focus on the former) * Binary distribution for pip and maven: 1) static linking is used (requires -fPIC). 2) all features are enabled. 3) dependencies are pulled in with scripts in mxnet. Handling one of the above cases would be a good showcase for the value of the new tool. - It's unclear how it impacts people with only an archive of mxnet source code but without network access. This applies for the dependencies that are captured as submodules that you mentioned as a way that mxnet manages dependency. - If the value proposition of conan is to simplify the dependency management, then it should unify other solutions instead of keeping these solutions around. Overall, it would be helpful to have a clear message such as what exactly conan can replace, and having a proof of concept that works for this would be helpful. Otherwise, I fear that we may be introducing yet another way to manage dependency that further complicates the existing problem. That said, I'm not suggesting that we impose the burden to implement everything on you alone, and it's ok to rally people who are interested in this solution to help out. To facilitate this, I created a feature branch so that it's easier for you and people who are enthusiastic about this to work together [1]. For now, I'm voting -1 to this proposal and I hope you understand. -sz [1] https://github.com/apache/incubator-mxnet/tree/conan On 2019/05/03 07:51:34, Konstantin Ivlev wrote: > hi Sheng Zha, > > on pull request review I was told by Anirudh anirudhacharya and Roshani > Nagmote to start discussion/vote on the mxnet dev list. it seems to be a > vicious circle now - on GitHub I am told to use vote, and on vote I am told > to use GitHub, this doesn't help much. > FYI GitHub review stuck, it's already opened since November 2018, and it's > still not approved (however, there were no objections during the review). > Previous discussion in e-mail thread also didn't encounter any objections, > and all questions were answered. > JIRA ticket has no discussion at all (except it has duplicates of comments > made on GitHub). > so let's process with 3-day vote for now, as other communication channels > were already tried with no success. > > yours sincerely, Konstantin > > пт, 3 мая 2019 г. в 14:17, Sheng Zha : > > > Hi Konstantin, > > > > While conan looks like an option that's worth exploring, given that your > > request is to merge the pull request, I'd suggest that the request should > > go through the regular pull request review and it doesn't really need a > > vote (as it doesn't substitute reviews anyway) > > > > If you would like to gather more attention to it, feel free to ping in a > > discussion thread. > > > > -sz > > > > On 2019/05/03 06:29:55, Konstantin Ivlev wrote: > > > Dear MXNet community, > > > > > > This is the 3-day vote to add conan support for Apache MXNet (incubating) > > > version v1.4.1. > > > The voting on dev@ list will start May 03 23:59:59 (PST) and close on > > May > > > 06 23:59:59. > > > > > > Background: conan is open-source, freeware, cross-platform package > > manager > > > for C and C++ projects, written in python. it provides integration with > > > various build systems, include CMake. conan may use bintray as a server > > to > > > store and download pre-built packages, or packages might be always built > > > from sources. > > > > > > Problem: currently (as for v1.4.1), Apache MXNet (incubating) is using > > > several ways to fetch 3rd-party dependencies simultaneously, for > > instance: > > > 1. download GitHub archives during the build > > > - OpenBLAS > > > - OpenCV > > > 2. conda (alternative way to GitHub archives) > > > 3. download from CMake > > > - Intel Math Kernel Library (MKL) > > > 4. Git submodules > > > - cub > > > - dlpack > > > - dmlc-core > > > - googletest > > > - mkldnn > > > - mshadow > > > - onnx-tensorrt > > > - openmp > > > - ps-lite > > > - tvm > > > therefore, there are multiple places to look for 3rd parties, and its > > hard > > > to update them, as you need to remember or figure it out how to update a > > > particular dependency to newer version,
Re: [VOTE] add conan support for Apache MXNet (incubating)
Hi Konstantin, Kellen, Thank you guys for the very detailed explanation! I was lacking some relevant contexts and previous discussions, which got me confused previously. My understanding that C++ is short of a perfect package manager. I totally agree that for any C++ project, it would be great to try possible opportunities to make our building experience smoother. This will largely help improve the usability of any project in the C++ community. TensorFlow uses Bazel officially, CMake supported unofficially. PyTorch seems to use CMake as wellMy. My understanding is that it helps users from the community to quickly pick up the building process. If we could help customers (industrial or academic, internal or external) build stuff more smoothly, it would be definitely a big plus. It will be a good idea if we have something to try out or play with. For example, instructions for building MXNet with Conan, customizing MXNet build with Conan, etc. Considering mshadow is going to be merged into mxnet, I believe it is a good opportunity to illustrate the power and convenience of Conan. Again, thank you guys for the valuable and patient explanation! Thanks, Junru On Fri, May 3, 2019 at 10:14 AM Konstantin Ivlev wrote: > Hi Junru, > > > I am actually a bit concerned about the security issues. We are asked to > download binaries from third-party websites, which are not controlled or > validated by Apache > it's possible to run conan server inside apache network and download > binaries and sources only from this remote > > > CMake does support download artifacts from a pre-defined website very > well > same for conan, it also supports downloading of artifacts from pre-defined > web-sites > > > rather than broken links to nonsense > please let me know which links are broken. I've checked them all, and they > are all available for me. may be your ISP is blocking something? > > > most of the dependencies you mentioned adopts CMake, rather than Conan > I don't think it's relevant to compare CMake (meta-build-system) and Conan > (package manager), they have different purposes, and they might be used > simultaneously, or only one of them might be used, they aren't mutually > exclusive. > > > Would you mind at least mentioning the benefits somewhere in this thread > okay, let me list some benefits (not all of them, but something I have on > mind) > - cross-platform, runs on Windows, Linux, OSX, FreeBSD, etc., also supports > cross-compiling for various platforms, like iOS, Android, Emscripten, > Windows Phone, etc. > - open-source (MIT licensed) and freeware > - supports both build from sources and pre-built artifacts (for instance, > if compare to usage of submodules, you probably wouldn't use pre-built > binaries in that case, as repo size will grow) > - supports multiple versions of libraries (some package managers provide > only "latest") > - supports options of libraries (whatever you need is configurable per > library, e.g. shared vs static, with or without assembler, multi-threaded > vs single-threaded, etc) > - supports installation of build tools as well (something like bison, flex, > nasm, yasm, etc.) > - has concept of profiles and their management (include options, tools and > environment variables to be used for build) > - supports various build systems (CMake, Meson, MSBuild, boost build, > etc.), pretty much build system agnostic > - decentralized, you may use in-house server (e.g. conan server or > artifactory), or bintray > - extensible via hooks > there are some comparisons of package managers on reddit (e.g. > https://www.reddit.com/r/cpp/comments/9m4l0p/conan_vcpkg_or_build2/ and > > https://www.reddit.com/r/cpp/comments/8ldhu0/opinions_on_the_conan_package_manager/ > ). > > > and eagerly asking for avote? I believe that reasonable discussion would > keep us within a *healthy* discussion. > I am asking for a vote because I was told to do this on GitHub discussion > by MXNet developers. I've already started discussions using all suggested > channels (GitHub, JIRA and this mail list). all questions were answered, > and no further questions appeared until today. I was thinking we already > have passed the discussion stage, as all discussion have stalled with no > objections (nobody clearly said something like "we're going to adopt conan" > or "we will not use conan definetely"). so my impression was discussion > stage was already passed, and now it's time for the decision. sorry if I > had wrong impression, I am not really very familiar with your processes. > > > why not I simply specify versions in git submidule, which everyone > understands how it behaves > I haven't said usage of submodules itself is something bad, if you're fine > with submodules, it's okay, but it seems like for MXNet they don't fit all > use-cases, as some dependencies are downloaded via other ways (as mentioned > above, via conda, cmake or just downloaded from GitHub archives in CI > scripts). this causes fragmentation and incl
Re: [VOTE] add conan support for Apache MXNet (incubating)
Hi Junru, > I am actually a bit concerned about the security issues. We are asked to download binaries from third-party websites, which are not controlled or validated by Apache it's possible to run conan server inside apache network and download binaries and sources only from this remote > CMake does support download artifacts from a pre-defined website very well same for conan, it also supports downloading of artifacts from pre-defined web-sites > rather than broken links to nonsense please let me know which links are broken. I've checked them all, and they are all available for me. may be your ISP is blocking something? > most of the dependencies you mentioned adopts CMake, rather than Conan I don't think it's relevant to compare CMake (meta-build-system) and Conan (package manager), they have different purposes, and they might be used simultaneously, or only one of them might be used, they aren't mutually exclusive. > Would you mind at least mentioning the benefits somewhere in this thread okay, let me list some benefits (not all of them, but something I have on mind) - cross-platform, runs on Windows, Linux, OSX, FreeBSD, etc., also supports cross-compiling for various platforms, like iOS, Android, Emscripten, Windows Phone, etc. - open-source (MIT licensed) and freeware - supports both build from sources and pre-built artifacts (for instance, if compare to usage of submodules, you probably wouldn't use pre-built binaries in that case, as repo size will grow) - supports multiple versions of libraries (some package managers provide only "latest") - supports options of libraries (whatever you need is configurable per library, e.g. shared vs static, with or without assembler, multi-threaded vs single-threaded, etc) - supports installation of build tools as well (something like bison, flex, nasm, yasm, etc.) - has concept of profiles and their management (include options, tools and environment variables to be used for build) - supports various build systems (CMake, Meson, MSBuild, boost build, etc.), pretty much build system agnostic - decentralized, you may use in-house server (e.g. conan server or artifactory), or bintray - extensible via hooks there are some comparisons of package managers on reddit (e.g. https://www.reddit.com/r/cpp/comments/9m4l0p/conan_vcpkg_or_build2/ and https://www.reddit.com/r/cpp/comments/8ldhu0/opinions_on_the_conan_package_manager/ ). > and eagerly asking for avote? I believe that reasonable discussion would keep us within a *healthy* discussion. I am asking for a vote because I was told to do this on GitHub discussion by MXNet developers. I've already started discussions using all suggested channels (GitHub, JIRA and this mail list). all questions were answered, and no further questions appeared until today. I was thinking we already have passed the discussion stage, as all discussion have stalled with no objections (nobody clearly said something like "we're going to adopt conan" or "we will not use conan definetely"). so my impression was discussion stage was already passed, and now it's time for the decision. sorry if I had wrong impression, I am not really very familiar with your processes. > why not I simply specify versions in git submidule, which everyone understands how it behaves I haven't said usage of submodules itself is something bad, if you're fine with submodules, it's okay, but it seems like for MXNet they don't fit all use-cases, as some dependencies are downloaded via other ways (as mentioned above, via conda, cmake or just downloaded from GitHub archives in CI scripts). this causes fragmentation and includes complexity. so, it's clear that submodules somehow didn't satisfy all MXNet needs, as they aren't used for everything. > Everyone know how to include a sub-directory in cmake in one line probably, because not all of your dependencies are using CMake to build, so you can't simply include them into cmake in one line. yours sincerely, Konstantin пт, 3 мая 2019 г. в 23:10, Junru Shao : > I am actually a bit concerned about the security issues. We are asked to > download binaries from third-party websites, which are not controlled or > validated by Apache. Although it is claimed to be “decentralized”, I am > really not convinced where the security comes from. > > In the meantime, sacrificing security doesn’t really bring us tangible > benefits. CMake does support download artifacts from a pre-defined website > very well. We may also have pre-built binaries stored in our CI docker > without having to download it them from the internet. > > Another point is that I am not convinced of any advantage of Conan over > other package managers for C++. Would you mind at least mentioning the > benefits somewhere in this thread, rather than carelessly includes tons of > irrelevant links (some of which are even wrong) and eagerly asking for a > vote? I believe that reasonable discussion would keep us within a *healthy* > discussion. > > Last but not least, as
Re: [VOTE] add conan support for Apache MXNet (incubating)
So firstly let's try to keep our responses empathetic and avoid ad-hom comments. It might be beneficial to take some time to review the Apache Code of Conduct [1]. Konstantin has taken a lot of time to think about dependency management in MXNet on a volunteer basis which is commendable. Second, Junru, I share many of your security concerns, but my understanding is that Conan.io allows you to pull dependencies as binaries, or as source using the -build option, so we're not limited to strictly pulling binaries from remote servers. Some benefits I see: * A uniform method of pulling dependencies is much easier to maintain and reason about. Need to update a package because of a security vulnerability? Go to the single place we configure dependencies and update it. * We have many dependencies that do not need to be checked out depending on the build options a user desires (so called build conditional dependencies). There's not much point in downloading / checking out these dependencies if you're not going to use them. * Subrepo sources have to have certified license reviews every release. Using a package manager would solve this issue. * We have an extra user base (Conan.io users) who get exposure to MXNet, growing our user base. Many of these benefits we'd get with other package management systems. One option I had previously proposed was Hunter, which is basically a wrapper around CMake's ExternalProject functionality. The tradeoff I see between the two is that Hunter (or ExternalProject via CMake) wold be lighter weight but would have less support, a smaller community and would be hard to use consistently across the project with the variety of collaborators it has. 1: https://www.apache.org/foundation/policies/conduct.html. 2: https://github.com/ruslo/hunter On Fri, May 3, 2019 at 9:10 AM Junru Shao wrote: > I am actually a bit concerned about the security issues. We are asked to > download binaries from third-party websites, which are not controlled or > validated by Apache. Although it is claimed to be “decentralized”, I am > really not convinced where the security comes from. > > In the meantime, sacrificing security doesn’t really bring us tangible > benefits. CMake does support download artifacts from a pre-defined website > very well. We may also have pre-built binaries stored in our CI docker > without having to download it them from the internet. > > Another point is that I am not convinced of any advantage of Conan over > other package managers for C++. Would you mind at least mentioning the > benefits somewhere in this thread, rather than carelessly includes tons of > irrelevant links (some of which are even wrong) and eagerly asking for a > vote? I believe that reasonable discussion would keep us within a *healthy* > discussion. > > Last but not least, as we all know, most of the dependencies you mentioned > adopts CMake, rather than Conan, the meta-generator which generates CMake. > I didn’t see your logic stands like “oh you have tons of dependencies so > you must use Conan”, why not I simply specify versions in git submidule, > which everyone understands how it behaves? Everyone know how to include a > sub-directory in cmake in one line, so why we have to write python to make > it less understandable and more complicated? > > In conclusion, we need to be reasonable in healthy discussion. I don’t > particularly want to rudely +1 or -1 for a thing that is unclear to me, but > I really want to see pros and cons, discussion about issues and concerns, > rather than broken links to nonsense. > > Looking forward to your reply! > > Thanks, > Junru > > On Fri, May 3, 2019 at 08:05 kellen sunderland < > kellen.sunderl...@gmail.com> > wrote: > > > Hey Konstantin. Thanks for starting an email thread and sorry for the > > confusion. I think the ides is that we should discuss and agree on > > Conan.io adoption first on the dev list, then start merging PRs. Release > > 1.4.1 is already in testing and the 1.5 code freeze deadline is also near > > so I think it could be difficult to make such a large change on one of > > those releases. I've looked into package management solutions for the > > project before. I was in favour of hunter, but I think Conan's adoption > > rate now makes it the best option. It's simple to use and is becoming > > industry standard, with a minor downside of requiring Python (which has > > meanwhile become the most popular dev language). I'd personally be -1 > for > > 1.4.1 or 1.5, +1 for using Conan in 1.6 or 2.0. > > > > -Kellen > > > > On Fri, May 3, 2019 at 12:59 AM Konstantin Ivlev > > wrote: > > > > > hi Sheng Zha, > > > > > > on pull request review I was told by Anirudh anirudhacharya and Roshani > > > Nagmote to start discussion/vote on the mxnet dev list. it seems to be > a > > > vicious circle now - on GitHub I am told to use vote, and on vote I am > > told > > > to use GitHub, this doesn't help much. > > > FYI GitHub review stuck, it's already opened sin
Re: [VOTE] add conan support for Apache MXNet (incubating)
I am actually a bit concerned about the security issues. We are asked to download binaries from third-party websites, which are not controlled or validated by Apache. Although it is claimed to be “decentralized”, I am really not convinced where the security comes from. In the meantime, sacrificing security doesn’t really bring us tangible benefits. CMake does support download artifacts from a pre-defined website very well. We may also have pre-built binaries stored in our CI docker without having to download it them from the internet. Another point is that I am not convinced of any advantage of Conan over other package managers for C++. Would you mind at least mentioning the benefits somewhere in this thread, rather than carelessly includes tons of irrelevant links (some of which are even wrong) and eagerly asking for a vote? I believe that reasonable discussion would keep us within a *healthy* discussion. Last but not least, as we all know, most of the dependencies you mentioned adopts CMake, rather than Conan, the meta-generator which generates CMake. I didn’t see your logic stands like “oh you have tons of dependencies so you must use Conan”, why not I simply specify versions in git submidule, which everyone understands how it behaves? Everyone know how to include a sub-directory in cmake in one line, so why we have to write python to make it less understandable and more complicated? In conclusion, we need to be reasonable in healthy discussion. I don’t particularly want to rudely +1 or -1 for a thing that is unclear to me, but I really want to see pros and cons, discussion about issues and concerns, rather than broken links to nonsense. Looking forward to your reply! Thanks, Junru On Fri, May 3, 2019 at 08:05 kellen sunderland wrote: > Hey Konstantin. Thanks for starting an email thread and sorry for the > confusion. I think the ides is that we should discuss and agree on > Conan.io adoption first on the dev list, then start merging PRs. Release > 1.4.1 is already in testing and the 1.5 code freeze deadline is also near > so I think it could be difficult to make such a large change on one of > those releases. I've looked into package management solutions for the > project before. I was in favour of hunter, but I think Conan's adoption > rate now makes it the best option. It's simple to use and is becoming > industry standard, with a minor downside of requiring Python (which has > meanwhile become the most popular dev language). I'd personally be -1 for > 1.4.1 or 1.5, +1 for using Conan in 1.6 or 2.0. > > -Kellen > > On Fri, May 3, 2019 at 12:59 AM Konstantin Ivlev > wrote: > > > hi Sheng Zha, > > > > on pull request review I was told by Anirudh anirudhacharya and Roshani > > Nagmote to start discussion/vote on the mxnet dev list. it seems to be a > > vicious circle now - on GitHub I am told to use vote, and on vote I am > told > > to use GitHub, this doesn't help much. > > FYI GitHub review stuck, it's already opened since November 2018, and > it's > > still not approved (however, there were no objections during the review). > > Previous discussion in e-mail thread also didn't encounter any > objections, > > and all questions were answered. > > JIRA ticket has no discussion at all (except it has duplicates of > comments > > made on GitHub). > > so let's process with 3-day vote for now, as other communication channels > > were already tried with no success. > > > > yours sincerely, Konstantin > > > > пт, 3 мая 2019 г. в 14:17, Sheng Zha : > > > > > Hi Konstantin, > > > > > > While conan looks like an option that's worth exploring, given that > your > > > request is to merge the pull request, I'd suggest that the request > should > > > go through the regular pull request review and it doesn't really need a > > > vote (as it doesn't substitute reviews anyway) > > > > > > If you would like to gather more attention to it, feel free to ping in > a > > > discussion thread. > > > > > > -sz > > > > > > On 2019/05/03 06:29:55, Konstantin Ivlev wrote: > > > > Dear MXNet community, > > > > > > > > This is the 3-day vote to add conan support for Apache MXNet > > (incubating) > > > > version v1.4.1. > > > > The voting on dev@ list will start May 03 23:59:59 (PST) and close > on > > > May > > > > 06 23:59:59. > > > > > > > > Background: conan is open-source, freeware, cross-platform package > > > manager > > > > for C and C++ projects, written in python. it provides integration > with > > > > various build systems, include CMake. conan may use bintray as a > server > > > to > > > > store and download pre-built packages, or packages might be always > > built > > > > from sources. > > > > > > > > Problem: currently (as for v1.4.1), Apache MXNet (incubating) is > using > > > > several ways to fetch 3rd-party dependencies simultaneously, for > > > instance: > > > > 1. download GitHub archives during the build > > > > - OpenBLAS > > > > - OpenCV > > > > 2. conda (alternative way to GitHub archives) > >
Re: [VOTE] add conan support for Apache MXNet (incubating)
Hey Konstantin. Thanks for starting an email thread and sorry for the confusion. I think the ides is that we should discuss and agree on Conan.io adoption first on the dev list, then start merging PRs. Release 1.4.1 is already in testing and the 1.5 code freeze deadline is also near so I think it could be difficult to make such a large change on one of those releases. I've looked into package management solutions for the project before. I was in favour of hunter, but I think Conan's adoption rate now makes it the best option. It's simple to use and is becoming industry standard, with a minor downside of requiring Python (which has meanwhile become the most popular dev language). I'd personally be -1 for 1.4.1 or 1.5, +1 for using Conan in 1.6 or 2.0. -Kellen On Fri, May 3, 2019 at 12:59 AM Konstantin Ivlev wrote: > hi Sheng Zha, > > on pull request review I was told by Anirudh anirudhacharya and Roshani > Nagmote to start discussion/vote on the mxnet dev list. it seems to be a > vicious circle now - on GitHub I am told to use vote, and on vote I am told > to use GitHub, this doesn't help much. > FYI GitHub review stuck, it's already opened since November 2018, and it's > still not approved (however, there were no objections during the review). > Previous discussion in e-mail thread also didn't encounter any objections, > and all questions were answered. > JIRA ticket has no discussion at all (except it has duplicates of comments > made on GitHub). > so let's process with 3-day vote for now, as other communication channels > were already tried with no success. > > yours sincerely, Konstantin > > пт, 3 мая 2019 г. в 14:17, Sheng Zha : > > > Hi Konstantin, > > > > While conan looks like an option that's worth exploring, given that your > > request is to merge the pull request, I'd suggest that the request should > > go through the regular pull request review and it doesn't really need a > > vote (as it doesn't substitute reviews anyway) > > > > If you would like to gather more attention to it, feel free to ping in a > > discussion thread. > > > > -sz > > > > On 2019/05/03 06:29:55, Konstantin Ivlev wrote: > > > Dear MXNet community, > > > > > > This is the 3-day vote to add conan support for Apache MXNet > (incubating) > > > version v1.4.1. > > > The voting on dev@ list will start May 03 23:59:59 (PST) and close on > > May > > > 06 23:59:59. > > > > > > Background: conan is open-source, freeware, cross-platform package > > manager > > > for C and C++ projects, written in python. it provides integration with > > > various build systems, include CMake. conan may use bintray as a server > > to > > > store and download pre-built packages, or packages might be always > built > > > from sources. > > > > > > Problem: currently (as for v1.4.1), Apache MXNet (incubating) is using > > > several ways to fetch 3rd-party dependencies simultaneously, for > > instance: > > > 1. download GitHub archives during the build > > > - OpenBLAS > > > - OpenCV > > > 2. conda (alternative way to GitHub archives) > > > 3. download from CMake > > > - Intel Math Kernel Library (MKL) > > > 4. Git submodules > > > - cub > > > - dlpack > > > - dmlc-core > > > - googletest > > > - mkldnn > > > - mshadow > > > - onnx-tensorrt > > > - openmp > > > - ps-lite > > > - tvm > > > therefore, there are multiple places to look for 3rd parties, and its > > hard > > > to update them, as you need to remember or figure it out how to update > a > > > particular dependency to newer version, for instance. > > > current Apache MXNet (incubating) build instructions differ very much > per > > > platform, require to download and unzip some archives manually, > > specifying > > > variables with paths to this archives, in conjunction of updating git > > > submodules, > > > > > > Action: merge pull request providing an initial conan support for > Apache > > > MXNet (incubating). support conan as an alternate approach to fetch > > various > > > 3rd-party dependencies. old approaches will be still available, > supported > > > and left intact. > > > > > > Below are links to > > > 1) conan web-site: https://conan.io/ > > > 2) conan GitHub repository: https://github.com/conan-io/conan > > > 3) conan documentation: https://docs.conan.io/en/latest/ > > > 4) bintray: https://bintray.com > > > 5) pull request adding conan support to Apache MXNet (incubating): > > > https://github.com/apache/incubator-mxnet/pull/13400 > > > 6) JIRA issue: https://issues.apache.org/jira/browse/MXNET-1229 > > > 7) previous email discussion: > > > > > > https://lists.apache.org/thread.html/301a46a637f7e3c249c475713f701bef7530c32bc92d8834c0882897@%3Cdev.mxnet.apache.org%3E > > > 8) MXNet build instructions: > > > https://mxnet-tqchen.readthedocs.io/en/latest/how_to/build.html > > > 9) MXNet build instructions (Windows): > > > > > > https://mxnet.incubator.apache.org/versions/master/install/windows_setup.html > > > 10) MXNet build instructions (OSX): > > > > http://mxnet.incubator.ap
Re: [VOTE] add conan support for Apache MXNet (incubating)
hi Sheng Zha, on pull request review I was told by Anirudh anirudhacharya and Roshani Nagmote to start discussion/vote on the mxnet dev list. it seems to be a vicious circle now - on GitHub I am told to use vote, and on vote I am told to use GitHub, this doesn't help much. FYI GitHub review stuck, it's already opened since November 2018, and it's still not approved (however, there were no objections during the review). Previous discussion in e-mail thread also didn't encounter any objections, and all questions were answered. JIRA ticket has no discussion at all (except it has duplicates of comments made on GitHub). so let's process with 3-day vote for now, as other communication channels were already tried with no success. yours sincerely, Konstantin пт, 3 мая 2019 г. в 14:17, Sheng Zha : > Hi Konstantin, > > While conan looks like an option that's worth exploring, given that your > request is to merge the pull request, I'd suggest that the request should > go through the regular pull request review and it doesn't really need a > vote (as it doesn't substitute reviews anyway) > > If you would like to gather more attention to it, feel free to ping in a > discussion thread. > > -sz > > On 2019/05/03 06:29:55, Konstantin Ivlev wrote: > > Dear MXNet community, > > > > This is the 3-day vote to add conan support for Apache MXNet (incubating) > > version v1.4.1. > > The voting on dev@ list will start May 03 23:59:59 (PST) and close on > May > > 06 23:59:59. > > > > Background: conan is open-source, freeware, cross-platform package > manager > > for C and C++ projects, written in python. it provides integration with > > various build systems, include CMake. conan may use bintray as a server > to > > store and download pre-built packages, or packages might be always built > > from sources. > > > > Problem: currently (as for v1.4.1), Apache MXNet (incubating) is using > > several ways to fetch 3rd-party dependencies simultaneously, for > instance: > > 1. download GitHub archives during the build > > - OpenBLAS > > - OpenCV > > 2. conda (alternative way to GitHub archives) > > 3. download from CMake > > - Intel Math Kernel Library (MKL) > > 4. Git submodules > > - cub > > - dlpack > > - dmlc-core > > - googletest > > - mkldnn > > - mshadow > > - onnx-tensorrt > > - openmp > > - ps-lite > > - tvm > > therefore, there are multiple places to look for 3rd parties, and its > hard > > to update them, as you need to remember or figure it out how to update a > > particular dependency to newer version, for instance. > > current Apache MXNet (incubating) build instructions differ very much per > > platform, require to download and unzip some archives manually, > specifying > > variables with paths to this archives, in conjunction of updating git > > submodules, > > > > Action: merge pull request providing an initial conan support for Apache > > MXNet (incubating). support conan as an alternate approach to fetch > various > > 3rd-party dependencies. old approaches will be still available, supported > > and left intact. > > > > Below are links to > > 1) conan web-site: https://conan.io/ > > 2) conan GitHub repository: https://github.com/conan-io/conan > > 3) conan documentation: https://docs.conan.io/en/latest/ > > 4) bintray: https://bintray.com > > 5) pull request adding conan support to Apache MXNet (incubating): > > https://github.com/apache/incubator-mxnet/pull/13400 > > 6) JIRA issue: https://issues.apache.org/jira/browse/MXNET-1229 > > 7) previous email discussion: > > > https://lists.apache.org/thread.html/301a46a637f7e3c249c475713f701bef7530c32bc92d8834c0882897@%3Cdev.mxnet.apache.org%3E > > 8) MXNet build instructions: > > https://mxnet-tqchen.readthedocs.io/en/latest/how_to/build.html > > 9) MXNet build instructions (Windows): > > > https://mxnet.incubator.apache.org/versions/master/install/windows_setup.html > > 10) MXNet build instructions (OSX): > > http://mxnet.incubator.apache.org/versions/master/install/osx_setup.html > > 11) MXNet build instructions (Linux): > > > http://mxnet.incubator.apache.org/versions/master/install/ubuntu_setup.html > > 12) MXNet development setup (OSX): > > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+Developer+Setup+on+Mac > > > > Please remember to TEST first before voting accordingly: > > +1 = approve > > +0 = no opinion > > -1 = disapprove (provide reason) > > > > Best regards, > > Konstantin Ivlev > > >
Re: [VOTE] add conan support for Apache MXNet (incubating)
hi Sheng Zha, on pull request review I was told by Anirudh anirudhacharya and Roshani Nagmote to start discussion/vote on the mxnet dev list. it seems to be a vicious circle now - on GitHub I am told to use vote, and on vote I am told to use GitHub, this doesn't help much. FYI GitHub review stuck, it's already opened since November 2018, and it's still not approved (however, there were no objections during the review). Previous discussion in e-mail thread also didn't encounter any objections, and all questions were answered. JIRA ticket has no discussion at all (except it has duplicates of comments made on GitHub). so let's process with 3-day vote for now, as other communication channels were already tried with no success. yours sincerely, Konstantin пт, 3 мая 2019 г. в 14:17, Sheng Zha : > Hi Konstantin, > > While conan looks like an option that's worth exploring, given that your > request is to merge the pull request, I'd suggest that the request should > go through the regular pull request review and it doesn't really need a > vote (as it doesn't substitute reviews anyway) > > If you would like to gather more attention to it, feel free to ping in a > discussion thread. > > -sz > > On 2019/05/03 06:29:55, Konstantin Ivlev wrote: > > Dear MXNet community, > > > > This is the 3-day vote to add conan support for Apache MXNet (incubating) > > version v1.4.1. > > The voting on dev@ list will start May 03 23:59:59 (PST) and close on > May > > 06 23:59:59. > > > > Background: conan is open-source, freeware, cross-platform package > manager > > for C and C++ projects, written in python. it provides integration with > > various build systems, include CMake. conan may use bintray as a server > to > > store and download pre-built packages, or packages might be always built > > from sources. > > > > Problem: currently (as for v1.4.1), Apache MXNet (incubating) is using > > several ways to fetch 3rd-party dependencies simultaneously, for > instance: > > 1. download GitHub archives during the build > > - OpenBLAS > > - OpenCV > > 2. conda (alternative way to GitHub archives) > > 3. download from CMake > > - Intel Math Kernel Library (MKL) > > 4. Git submodules > > - cub > > - dlpack > > - dmlc-core > > - googletest > > - mkldnn > > - mshadow > > - onnx-tensorrt > > - openmp > > - ps-lite > > - tvm > > therefore, there are multiple places to look for 3rd parties, and its > hard > > to update them, as you need to remember or figure it out how to update a > > particular dependency to newer version, for instance. > > current Apache MXNet (incubating) build instructions differ very much per > > platform, require to download and unzip some archives manually, > specifying > > variables with paths to this archives, in conjunction of updating git > > submodules, > > > > Action: merge pull request providing an initial conan support for Apache > > MXNet (incubating). support conan as an alternate approach to fetch > various > > 3rd-party dependencies. old approaches will be still available, supported > > and left intact. > > > > Below are links to > > 1) conan web-site: https://conan.io/ > > 2) conan GitHub repository: https://github.com/conan-io/conan > > 3) conan documentation: https://docs.conan.io/en/latest/ > > 4) bintray: https://bintray.com > > 5) pull request adding conan support to Apache MXNet (incubating): > > https://github.com/apache/incubator-mxnet/pull/13400 > > 6) JIRA issue: https://issues.apache.org/jira/browse/MXNET-1229 > > 7) previous email discussion: > > > https://lists.apache.org/thread.html/301a46a637f7e3c249c475713f701bef7530c32bc92d8834c0882897@%3Cdev.mxnet.apache.org%3E > > 8) MXNet build instructions: > > https://mxnet-tqchen.readthedocs.io/en/latest/how_to/build.html > > 9) MXNet build instructions (Windows): > > > https://mxnet.incubator.apache.org/versions/master/install/windows_setup.html > > 10) MXNet build instructions (OSX): > > http://mxnet.incubator.apache.org/versions/master/install/osx_setup.html > > 11) MXNet build instructions (Linux): > > > http://mxnet.incubator.apache.org/versions/master/install/ubuntu_setup.html > > 12) MXNet development setup (OSX): > > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+Developer+Setup+on+Mac > > > > Please remember to TEST first before voting accordingly: > > +1 = approve > > +0 = no opinion > > -1 = disapprove (provide reason) > > > > Best regards, > > Konstantin Ivlev > > >
Re: [VOTE] add conan support for Apache MXNet (incubating)
Hi Konstantin, While conan looks like an option that's worth exploring, given that your request is to merge the pull request, I'd suggest that the request should go through the regular pull request review and it doesn't really need a vote (as it doesn't substitute reviews anyway) If you would like to gather more attention to it, feel free to ping in a discussion thread. -sz On 2019/05/03 06:29:55, Konstantin Ivlev wrote: > Dear MXNet community, > > This is the 3-day vote to add conan support for Apache MXNet (incubating) > version v1.4.1. > The voting on dev@ list will start May 03 23:59:59 (PST) and close on May > 06 23:59:59. > > Background: conan is open-source, freeware, cross-platform package manager > for C and C++ projects, written in python. it provides integration with > various build systems, include CMake. conan may use bintray as a server to > store and download pre-built packages, or packages might be always built > from sources. > > Problem: currently (as for v1.4.1), Apache MXNet (incubating) is using > several ways to fetch 3rd-party dependencies simultaneously, for instance: > 1. download GitHub archives during the build > - OpenBLAS > - OpenCV > 2. conda (alternative way to GitHub archives) > 3. download from CMake > - Intel Math Kernel Library (MKL) > 4. Git submodules > - cub > - dlpack > - dmlc-core > - googletest > - mkldnn > - mshadow > - onnx-tensorrt > - openmp > - ps-lite > - tvm > therefore, there are multiple places to look for 3rd parties, and its hard > to update them, as you need to remember or figure it out how to update a > particular dependency to newer version, for instance. > current Apache MXNet (incubating) build instructions differ very much per > platform, require to download and unzip some archives manually, specifying > variables with paths to this archives, in conjunction of updating git > submodules, > > Action: merge pull request providing an initial conan support for Apache > MXNet (incubating). support conan as an alternate approach to fetch various > 3rd-party dependencies. old approaches will be still available, supported > and left intact. > > Below are links to > 1) conan web-site: https://conan.io/ > 2) conan GitHub repository: https://github.com/conan-io/conan > 3) conan documentation: https://docs.conan.io/en/latest/ > 4) bintray: https://bintray.com > 5) pull request adding conan support to Apache MXNet (incubating): > https://github.com/apache/incubator-mxnet/pull/13400 > 6) JIRA issue: https://issues.apache.org/jira/browse/MXNET-1229 > 7) previous email discussion: > https://lists.apache.org/thread.html/301a46a637f7e3c249c475713f701bef7530c32bc92d8834c0882897@%3Cdev.mxnet.apache.org%3E > 8) MXNet build instructions: > https://mxnet-tqchen.readthedocs.io/en/latest/how_to/build.html > 9) MXNet build instructions (Windows): > https://mxnet.incubator.apache.org/versions/master/install/windows_setup.html > 10) MXNet build instructions (OSX): > http://mxnet.incubator.apache.org/versions/master/install/osx_setup.html > 11) MXNet build instructions (Linux): > http://mxnet.incubator.apache.org/versions/master/install/ubuntu_setup.html > 12) MXNet development setup (OSX): > https://cwiki.apache.org/confluence/display/MXNET/MXNet+Developer+Setup+on+Mac > > Please remember to TEST first before voting accordingly: > +1 = approve > +0 = no opinion > -1 = disapprove (provide reason) > > Best regards, > Konstantin Ivlev >
