Re: assimilation of mshadow into the MXNet codebase

2019-04-05 Thread Tianqi Chen
Technically, mshadow is sufficient for MXNet. Adopting other libraries (
eigen or xtensor) will unnecessarily increase the codebase complexity
without any additional gains.

Given that mshadow is only used by mxnet. I do support donating it into
mxnet codebase.
To respect the original mshadow community. I would recommend starting a
community RFC In the mshadow github issue for a week, before we start the
migrating process.
Also, I would recommend a rebase merge just like the case of MXNet.jl code
base to preserve the contribution history.

Tianqi


On Fri, Apr 5, 2019 at 9:25 PM Alfredo Luque
 wrote:

> Do you have a link to both of these proposals?
>
> On Fri, Apr 5, 2019 at 20:14 Anirudh Acharya 
> wrote:
>
> > Hi Pedro,
> >
> > mshadow is mostly used for tensor arithmetic. There have been discussions
> > about including it within mxnet. I think it is a good idea.
> >
> > As a more long term solution using libraries like eigen to perform linear
> > algebra operations was also suggested by anirudh2290@. I think xtensor(
> > https://github.com/QuantStack/xtensor ) can also be a candidate here.
> >
> > -
> > Anirudh
> >
> >
> > On Fri, Apr 5, 2019 at 7:03 PM Pedro Larroy <
> pedro.larroy.li...@gmail.com>
> > wrote:
> >
> > > Hi
> > >
> > > Some developers have noticed that working in mshadow is cumbersome as
> > > it's a 3rdparty subrepo.
> > >
> > > Since mshadow is a bunch of headers which don't have much of
> > > independent tests / library functionality, me and other developers
> > > believe that it would be good to assimilate this code in the
> > > repository for ease of contribution and changes without having to go
> > > trough contortions to test PRs that modify mshadow.
> > >
> > > Would anybody oppose this change?
> > >
> > > Thanks and have a nice weekend.
> > >
> > > Pedro.
> > >
> >
>


Re: assimilation of mshadow into the MXNet codebase

2019-04-05 Thread Alfredo Luque
Do you have a link to both of these proposals?

On Fri, Apr 5, 2019 at 20:14 Anirudh Acharya  wrote:

> Hi Pedro,
>
> mshadow is mostly used for tensor arithmetic. There have been discussions
> about including it within mxnet. I think it is a good idea.
>
> As a more long term solution using libraries like eigen to perform linear
> algebra operations was also suggested by anirudh2290@. I think xtensor(
> https://github.com/QuantStack/xtensor ) can also be a candidate here.
>
> -
> Anirudh
>
>
> On Fri, Apr 5, 2019 at 7:03 PM Pedro Larroy 
> wrote:
>
> > Hi
> >
> > Some developers have noticed that working in mshadow is cumbersome as
> > it's a 3rdparty subrepo.
> >
> > Since mshadow is a bunch of headers which don't have much of
> > independent tests / library functionality, me and other developers
> > believe that it would be good to assimilate this code in the
> > repository for ease of contribution and changes without having to go
> > trough contortions to test PRs that modify mshadow.
> >
> > Would anybody oppose this change?
> >
> > Thanks and have a nice weekend.
> >
> > Pedro.
> >
>


Re: assimilation of mshadow into the MXNet codebase

2019-04-05 Thread Anirudh Acharya
Hi Pedro,

mshadow is mostly used for tensor arithmetic. There have been discussions
about including it within mxnet. I think it is a good idea.

As a more long term solution using libraries like eigen to perform linear
algebra operations was also suggested by anirudh2290@. I think xtensor(
https://github.com/QuantStack/xtensor ) can also be a candidate here.

-
Anirudh


On Fri, Apr 5, 2019 at 7:03 PM Pedro Larroy 
wrote:

> Hi
>
> Some developers have noticed that working in mshadow is cumbersome as
> it's a 3rdparty subrepo.
>
> Since mshadow is a bunch of headers which don't have much of
> independent tests / library functionality, me and other developers
> believe that it would be good to assimilate this code in the
> repository for ease of contribution and changes without having to go
> trough contortions to test PRs that modify mshadow.
>
> Would anybody oppose this change?
>
> Thanks and have a nice weekend.
>
> Pedro.
>


assimilation of mshadow into the MXNet codebase

2019-04-05 Thread Pedro Larroy
Hi

Some developers have noticed that working in mshadow is cumbersome as
it's a 3rdparty subrepo.

Since mshadow is a bunch of headers which don't have much of
independent tests / library functionality, me and other developers
believe that it would be good to assimilate this code in the
repository for ease of contribution and changes without having to go
trough contortions to test PRs that modify mshadow.

Would anybody oppose this change?

Thanks and have a nice weekend.

Pedro.


Re: [MXNET 2.0 Wishlist] [DISCUSS] Single build system

2019-04-05 Thread Junru Shao
I agree with Tianqi and Marco. Probably what should happen is to let cmake
be the default in some minor release, and completely deprecate makefiles in
2.0.

On Fri, Apr 5, 2019 at 10:23 AM Marco de Abreu 
wrote:

> I think this is rather about the depreciation of the make based build
> system. We currently have make and cmake in parallel but with diverging
> feature support.
>
> -Marco
>
> Tianqi Chen  schrieb am Fr., 5. Apr. 2019,
> 11:42:
>
> > I am in favor of using CMake. And I personally think CMake is not
> something
> > that has to be introduced in a 2.0. It can simply be part of a minor
> > release.
> >
> > Tianqi
> >
> > On Thu, Apr 4, 2019 at 10:31 AM Kellen Sunderland 
> > wrote:
> >
> > > Hello MXNet devs,
> > >
> > > I'd like to start a thread discussing what our build system should look
> > > like in MXNet 2.0.  I'd propose that although the current make system
> has
> > > served us well in the past, we remove it along with the bump to 2.0.
> The
> > > end goal I'd like to see is that we have a clean build system, without
> a
> > > bunch of conditional logic that makes contributing and testing MXNet a
> > > simpler process.  Additionally I'd propose we target a minimum cmake
> > > version of 3.7 for reasons described below.
> > >
> > > First I'd like to give some context on why I'd propose we don't just
> > switch
> > > to cmake, but we also target a relatively new version (version 3.7 from
> > > Nov, 2016) of cmake.  The largest benefits in making this change would
> > > apply to CUDA builds where cmake itself has quite inconsistent
> > > functionality between versions.  One persistent annoyance I've had with
> > > cmake is that we've had conditional logic for the FindCUDA command
> which
> > at
> > > one point targeted some modern cmake features, but then in subsequent
> > > versions of cmake the way these features works was tweaked, and now I
> > find
> > > these cmake features are consistently broken to the point that I
> require
> > a
> > > bunch of -D defines to compile properly or to use an IDE.  An
> additional
> > > CUDA related issue is that every time there's a new SM added to NVCC we
> > > have to make a few source changes to support it.  I could see this
> being
> > > problematic for users who may suddenly realize that due to their
> > > compilation settings, they may not actually be enabling the features
> they
> > > think they are with their shiny new GPUs.
> > >
> > > As an alternative if we, for example, target cmake 3.7 at a minimum,
> and
> > we
> > > want to find cuda and then build a list of reasonable PTX/BINS we could
> > use
> > > the following command[1]:
> > >
> > > 
> > > FindCUDA(...)
> > > ...
> > > CUDA_SELECT_NVCC_ARCH_FLAGS(ARCH_FLAGS 3.0 3.5+PTX 5.2(5.0) Maxwell)
> > >   LIST(APPEND CUDA_NVCC_FLAGS ${ARCH_FLAGS})
> > > 
> > >
> > > Simple, concise, and it would help to make the building experience more
> > > consistent across platforms, build environments and IDEs (looking at
> you
> > > CLion).  We'd of course need to do a little experimentation work to
> make
> > > sure that this does indeed work as intended, and can replace the
> > currently
> > > complex findCuda logic we have in our build systems, but for the sake
> of
> > > the proposal let's assume these cmake commands do indeed work
> > consistently
> > > as documented from cmake 3.7 onwards.
> > >
> > > To give users a chance to update their tooling I'd also suggest we
> begin
> > > warning users at least a release in advance that make based builds will
> > be
> > > deprecated in MXNet 2.0 so they can begin migrating to cmake.  I'd also
> > > want to display deprecation messages for unused cmake flags (such as
> the
> > > profiler flag) for a release before the 2.0 release, and then remove
> them
> > > in 2.0.
> > >
> > > Of course not all users have cmake 3.7 on their systems, some of our
> > > employers force use to use ridiculously outdated linux distributions.
> > The
> > > good news for these users is that if we can offer Docker compilation
> with
> > > an image that has a supported version of cmake and we should be able to
> > > build a portable binary that work even with very old distributions of
> > > Linux.  Additionally installing cmake from source is also fairly
> > > straightforward [2] and works quite well on older distros in my
> > experience.
> > >
> > > Looking forward to hearing what others think.  Any preferred build
> > systems
> > > that you all would want to use?  Is cmake the right system to
> centralize
> > > on?  If so, is version 3.7 a reasonable minimum version to target?  Is
> > the
> > > 2.0 release a good point at which we can think about simplifying build
> > > logic?
> > >
> > > 1: https://cmake.org/cmake/help/v3.7/module/FindCUDA.html
> > > 2: https://github.com/Kitware/CMake
> > >
> >
>


Re: [MXNET 2.0 Wishlist] [DISCUSS] Single build system

2019-04-05 Thread Marco de Abreu
I think this is rather about the depreciation of the make based build
system. We currently have make and cmake in parallel but with diverging
feature support.

-Marco

Tianqi Chen  schrieb am Fr., 5. Apr. 2019, 11:42:

> I am in favor of using CMake. And I personally think CMake is not something
> that has to be introduced in a 2.0. It can simply be part of a minor
> release.
>
> Tianqi
>
> On Thu, Apr 4, 2019 at 10:31 AM Kellen Sunderland 
> wrote:
>
> > Hello MXNet devs,
> >
> > I'd like to start a thread discussing what our build system should look
> > like in MXNet 2.0.  I'd propose that although the current make system has
> > served us well in the past, we remove it along with the bump to 2.0.  The
> > end goal I'd like to see is that we have a clean build system, without a
> > bunch of conditional logic that makes contributing and testing MXNet a
> > simpler process.  Additionally I'd propose we target a minimum cmake
> > version of 3.7 for reasons described below.
> >
> > First I'd like to give some context on why I'd propose we don't just
> switch
> > to cmake, but we also target a relatively new version (version 3.7 from
> > Nov, 2016) of cmake.  The largest benefits in making this change would
> > apply to CUDA builds where cmake itself has quite inconsistent
> > functionality between versions.  One persistent annoyance I've had with
> > cmake is that we've had conditional logic for the FindCUDA command which
> at
> > one point targeted some modern cmake features, but then in subsequent
> > versions of cmake the way these features works was tweaked, and now I
> find
> > these cmake features are consistently broken to the point that I require
> a
> > bunch of -D defines to compile properly or to use an IDE.  An additional
> > CUDA related issue is that every time there's a new SM added to NVCC we
> > have to make a few source changes to support it.  I could see this being
> > problematic for users who may suddenly realize that due to their
> > compilation settings, they may not actually be enabling the features they
> > think they are with their shiny new GPUs.
> >
> > As an alternative if we, for example, target cmake 3.7 at a minimum, and
> we
> > want to find cuda and then build a list of reasonable PTX/BINS we could
> use
> > the following command[1]:
> >
> > 
> > FindCUDA(...)
> > ...
> > CUDA_SELECT_NVCC_ARCH_FLAGS(ARCH_FLAGS 3.0 3.5+PTX 5.2(5.0) Maxwell)
> >   LIST(APPEND CUDA_NVCC_FLAGS ${ARCH_FLAGS})
> > 
> >
> > Simple, concise, and it would help to make the building experience more
> > consistent across platforms, build environments and IDEs (looking at you
> > CLion).  We'd of course need to do a little experimentation work to make
> > sure that this does indeed work as intended, and can replace the
> currently
> > complex findCuda logic we have in our build systems, but for the sake of
> > the proposal let's assume these cmake commands do indeed work
> consistently
> > as documented from cmake 3.7 onwards.
> >
> > To give users a chance to update their tooling I'd also suggest we begin
> > warning users at least a release in advance that make based builds will
> be
> > deprecated in MXNet 2.0 so they can begin migrating to cmake.  I'd also
> > want to display deprecation messages for unused cmake flags (such as the
> > profiler flag) for a release before the 2.0 release, and then remove them
> > in 2.0.
> >
> > Of course not all users have cmake 3.7 on their systems, some of our
> > employers force use to use ridiculously outdated linux distributions.
> The
> > good news for these users is that if we can offer Docker compilation with
> > an image that has a supported version of cmake and we should be able to
> > build a portable binary that work even with very old distributions of
> > Linux.  Additionally installing cmake from source is also fairly
> > straightforward [2] and works quite well on older distros in my
> experience.
> >
> > Looking forward to hearing what others think.  Any preferred build
> systems
> > that you all would want to use?  Is cmake the right system to centralize
> > on?  If so, is version 3.7 a reasonable minimum version to target?  Is
> the
> > 2.0 release a good point at which we can think about simplifying build
> > logic?
> >
> > 1: https://cmake.org/cmake/help/v3.7/module/FindCUDA.html
> > 2: https://github.com/Kitware/CMake
> >
>


Re: [MXNET 2.0 Wishlist] [DISCUSS] Single build system

2019-04-05 Thread Tianqi Chen
I am in favor of using CMake. And I personally think CMake is not something
that has to be introduced in a 2.0. It can simply be part of a minor
release.

Tianqi

On Thu, Apr 4, 2019 at 10:31 AM Kellen Sunderland  wrote:

> Hello MXNet devs,
>
> I'd like to start a thread discussing what our build system should look
> like in MXNet 2.0.  I'd propose that although the current make system has
> served us well in the past, we remove it along with the bump to 2.0.  The
> end goal I'd like to see is that we have a clean build system, without a
> bunch of conditional logic that makes contributing and testing MXNet a
> simpler process.  Additionally I'd propose we target a minimum cmake
> version of 3.7 for reasons described below.
>
> First I'd like to give some context on why I'd propose we don't just switch
> to cmake, but we also target a relatively new version (version 3.7 from
> Nov, 2016) of cmake.  The largest benefits in making this change would
> apply to CUDA builds where cmake itself has quite inconsistent
> functionality between versions.  One persistent annoyance I've had with
> cmake is that we've had conditional logic for the FindCUDA command which at
> one point targeted some modern cmake features, but then in subsequent
> versions of cmake the way these features works was tweaked, and now I find
> these cmake features are consistently broken to the point that I require a
> bunch of -D defines to compile properly or to use an IDE.  An additional
> CUDA related issue is that every time there's a new SM added to NVCC we
> have to make a few source changes to support it.  I could see this being
> problematic for users who may suddenly realize that due to their
> compilation settings, they may not actually be enabling the features they
> think they are with their shiny new GPUs.
>
> As an alternative if we, for example, target cmake 3.7 at a minimum, and we
> want to find cuda and then build a list of reasonable PTX/BINS we could use
> the following command[1]:
>
> 
> FindCUDA(...)
> ...
> CUDA_SELECT_NVCC_ARCH_FLAGS(ARCH_FLAGS 3.0 3.5+PTX 5.2(5.0) Maxwell)
>   LIST(APPEND CUDA_NVCC_FLAGS ${ARCH_FLAGS})
> 
>
> Simple, concise, and it would help to make the building experience more
> consistent across platforms, build environments and IDEs (looking at you
> CLion).  We'd of course need to do a little experimentation work to make
> sure that this does indeed work as intended, and can replace the currently
> complex findCuda logic we have in our build systems, but for the sake of
> the proposal let's assume these cmake commands do indeed work consistently
> as documented from cmake 3.7 onwards.
>
> To give users a chance to update their tooling I'd also suggest we begin
> warning users at least a release in advance that make based builds will be
> deprecated in MXNet 2.0 so they can begin migrating to cmake.  I'd also
> want to display deprecation messages for unused cmake flags (such as the
> profiler flag) for a release before the 2.0 release, and then remove them
> in 2.0.
>
> Of course not all users have cmake 3.7 on their systems, some of our
> employers force use to use ridiculously outdated linux distributions.  The
> good news for these users is that if we can offer Docker compilation with
> an image that has a supported version of cmake and we should be able to
> build a portable binary that work even with very old distributions of
> Linux.  Additionally installing cmake from source is also fairly
> straightforward [2] and works quite well on older distros in my experience.
>
> Looking forward to hearing what others think.  Any preferred build systems
> that you all would want to use?  Is cmake the right system to centralize
> on?  If so, is version 3.7 a reasonable minimum version to target?  Is the
> 2.0 release a good point at which we can think about simplifying build
> logic?
>
> 1: https://cmake.org/cmake/help/v3.7/module/FindCUDA.html
> 2: https://github.com/Kitware/CMake
>


RE: [MXNET 2.0 Wishlist] [DISCUSS] Single build system

2019-04-05 Thread Zhao, Patric
+1 single build system.



> -Original Message-
> From: Qing Lan [mailto:lanking...@live.com]
> Sent: Friday, April 5, 2019 5:27 AM
> To: dev@mxnet.incubator.apache.org
> Subject: Re: [MXNET 2.0 Wishlist] [DISCUSS] Single build system
> 
> +1 to have a single build system
> 
> Currently the way of publish and the way of doing CI test is very different.
> The instruction shown on the website should match the way we deliver it to
> the users.
> Having a single build process would simplify the maintainance cost and reach
> to the best performance.
> 
> Thanks,
> Qing
> 
> 
> From: Marco de Abreu 
> Sent: Thursday, April 4, 2019 15:01
> To: dev@mxnet.incubator.apache.org
> Subject: Re: [MXNET 2.0 Wishlist] [DISCUSS] Single build system
> 
> +1 towards having a single build system
> 
> I'd like to add the benefit of this approach allowing us to have the same 
> build
> logic across all operating systems. It would be great if we could make
> x86/Unix, x86/windows, x86/mac and ARM/Unix first class citizens from the
> beginning.
> 
> -Marco
> 
> Kellen Sunderland  schrieb am Do., 4. Apr. 2019, 12:31:
> 
> > Hello MXNet devs,
> >
> > I'd like to start a thread discussing what our build system should
> > look like in MXNet 2.0.  I'd propose that although the current make
> > system has served us well in the past, we remove it along with the
> > bump to 2.0.  The end goal I'd like to see is that we have a clean
> > build system, without a bunch of conditional logic that makes
> > contributing and testing MXNet a simpler process.  Additionally I'd
> > propose we target a minimum cmake version of 3.7 for reasons described
> below.
> >
> > First I'd like to give some context on why I'd propose we don't just
> > switch to cmake, but we also target a relatively new version (version
> > 3.7 from Nov, 2016) of cmake.  The largest benefits in making this
> > change would apply to CUDA builds where cmake itself has quite
> > inconsistent functionality between versions.  One persistent annoyance
> > I've had with cmake is that we've had conditional logic for the
> > FindCUDA command which at one point targeted some modern cmake
> > features, but then in subsequent versions of cmake the way these
> > features works was tweaked, and now I find these cmake features are
> > consistently broken to the point that I require a bunch of -D defines
> > to compile properly or to use an IDE.  An additional CUDA related
> > issue is that every time there's a new SM added to NVCC we have to
> > make a few source changes to support it.  I could see this being
> > problematic for users who may suddenly realize that due to their
> > compilation settings, they may not actually be enabling the features they
> think they are with their shiny new GPUs.
> >
> > As an alternative if we, for example, target cmake 3.7 at a minimum,
> > and we want to find cuda and then build a list of reasonable PTX/BINS
> > we could use the following command[1]:
> >
> > 
> > FindCUDA(...)
> > ...
> > CUDA_SELECT_NVCC_ARCH_FLAGS(ARCH_FLAGS 3.0 3.5+PTX 5.2(5.0)
> Maxwell)
> >   LIST(APPEND CUDA_NVCC_FLAGS ${ARCH_FLAGS})
> > 
> >
> > Simple, concise, and it would help to make the building experience
> > more consistent across platforms, build environments and IDEs (looking
> > at you CLion).  We'd of course need to do a little experimentation
> > work to make sure that this does indeed work as intended, and can
> > replace the currently complex findCuda logic we have in our build
> > systems, but for the sake of the proposal let's assume these cmake
> > commands do indeed work consistently as documented from cmake 3.7
> onwards.
> >
> > To give users a chance to update their tooling I'd also suggest we
> > begin warning users at least a release in advance that make based
> > builds will be deprecated in MXNet 2.0 so they can begin migrating to
> > cmake.  I'd also want to display deprecation messages for unused cmake
> > flags (such as the profiler flag) for a release before the 2.0
> > release, and then remove them in 2.0.
> >
> > Of course not all users have cmake 3.7 on their systems, some of our
> > employers force use to use ridiculously outdated linux distributions.
> > The good news for these users is that if we can offer Docker
> > compilation with an image that has a supported version of cmake and we
> > should be able to build a portable binary that work even with very old
> > distributions of Linux.  Additionally installing cmake from source is
> > also fairly straightforward [2] and works quite well on older distros in my
> experience.
> >
> > Looking forward to hearing what others think.  Any preferred build
> > systems that you all would want to use?  Is cmake the right system to
> > centralize on?  If so, is version 3.7 a reasonable minimum version to
> > target?  Is the
> > 2.0 release a good point at which we can think about simplifying build
> > logic?
> >
> > 1: