Re: [VOTE] Release Apache MXNet (incubating) version 1.7.0.rc0

[Qing Lan]

My +1 (binding) on 1.7.0

Tested:

  *   Build from Source with static build instruction
  *   Tested Scala pacakge and passed all tests

Thanks,
Qing


From: Tao Lv 
Sent: Friday, July 10, 2020 0:03
To: dev@mxnet.incubator.apache.org 
Cc: d...@mxnet.apache.org ; Bob Paulin 
; Henri Yandell ; Jason Dai 
; Markus Weimer ; Michael Wall 

Subject: Re: [VOTE] Release Apache MXNet (incubating) version 1.7.0.rc0

+1 (binding)

I did:
- Verify the key and signature;
- Untar the source code package;
- Build from source code with makefile, USE_BLAS=mkl, USE_MKLDNN=1;
- Check mx.__version__;
- Run benchmark_score.py under examples/image-classification.

-tao

On Fri, Jul 10, 2020 at 2:18 PM Chen, Ciyong  wrote:

> Hi Community,
>
> I would like to call for action to test/validate/vote for the release
> candidate (1.7.0.rc0)
> As there's not any voting result during the scheduled time window, I would
> like to extend the time windows to July 13, 23:59:59 PST.
> Please prepare your time and provide feedback if you've tried with the
> pre-release code bases, thanks!
>
> Best regards,
> Ciyong
>
> -Original Message-
> From: Chen, Ciyong 
> Sent: Monday, July 6, 2020 10:48 PM
> To: d...@mxnet.apache.org
> Cc: Bob Paulin ; Henri Yandell ; Jason
> Dai ; Markus Weimer ; Michael
> Wall 
> Subject: RE: [VOTE] Release Apache MXNet (incubating) version 1.7.0.rc0
>
> For the language bindings and windows platform, may I have your support to
> help verify these features? Thanks!
>
> @lanking520 to help verify the Scala/Java @gigasquid to help verify the
> Clojure
> @hetong007 to help verify the R
> @yajiedesign to help verify the windows platform
>
> Best regards,
> Ciyong Chen
>
> -Original Message-
> From: Chen, Ciyong 
> Sent: Monday, July 6, 2020 10:39 PM
> To: d...@mxnet.apache.org
> Cc: Bob Paulin ; Henri Yandell ; Jason
> Dai ; Markus Weimer ; Michael
> Wall 
> Subject: [VOTE] Release Apache MXNet (incubating) version 1.7.0.rc0
>
> Dear MXNet community,
>
> This is the vote to release Apache MXNet (incubating) version 1.7.0.
> Voting will start July 6, 23:59:59 PST and close on July 9, 23:59:59 PST.
>
> Link to release notes:
> https://cwiki.apache.org/confluence/display/MXNET/1.7.0+Release+notes
>
> Link to release candidate:
> https://github.com/apache/incubator-mxnet/releases/tag/1.7.0.rc0
>
> Link to source and signatures on apache dist server:
> https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.7.0.rc0<
> https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.7.0.rc0/>
>
> Please remember to TEST first before voting accordingly:
> +1 = approve
> +0 = no opinion
> -1 = disapprove (provide reason)
>
> Additional notes:
>
>   *   There was an issue and discussion[1] regarding on a few numpy
> operators failed due to numpy 1.19.0 released on Jun 20, 2020, which exists
> in all branches (works with numpy <= 1.18.5). As numpy operator is still an
> experimental feature in 1.7.0 release and mainly targeting in MXNet 2.0
> release, so I decided to not block the voting and instead let the Community
> decide whether this is a blocker for the release.
>
> [1] https://github.com/apache/incubator-mxnet/issues/18600
>
> Best regards,
> Ciyong Chen
>
>

Re: [VOTE] Release Apache MXNet (incubating) version 1.6.0.rc1

[Qing Lan]

+1 (binding)

Built 1.6.0.rc1 on my Mac with MKLDNN

Scala build/test passed.

Thanks,
Qing


From: Chaitanya Bapat 
Sent: Friday, January 10, 2020 12:21
To: dev@mxnet.incubator.apache.org 
Cc: d...@mxnet.apache.org 
Subject: Re: [VOTE] Release Apache MXNet (incubating) version 1.6.0.rc1

+1

Built from the dist [1] on Ubuntu 16.04 DL AMI for CPU + MKLDNN
Tested
1. OpPerf (benchmark utility) - Promising results (faster forward times for
certain ops compared to 1.4.0 and 1.5.1)
2. Large tensor support (used the USE_INT64_TENSOR_SIZE = ON flag while
building) : Tests pass

Thanks Przemyslaw for leading 1.6.0! It's taken long. But we're close to
the finish line! Awesome work!

Thanks
Chai

[1] https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.6.0.rc1/

On Thu, 9 Jan 2020 at 22:05, Chen, Ciyong  wrote:

> +1
>
> Build from source on CentOS 7.6 with GCC 4.8.5 with MKLDNN.
> Unit tests passed and imagenet examples (with MKL-DNN subgraph backend)
> looked good on performance and accuracy in both FP32 and INT8 mode, RNN
> training worked.
>
> Thanks,
> -Ciyong
>
> -Original Message-
> From: Lai Wei 
> Sent: Wednesday, January 8, 2020 8:56 AM
> To: dev@mxnet.incubator.apache.org
> Cc: d...@mxnet.apache.org
> Subject: Re: [VOTE] Release Apache MXNet (incubating) version 1.6.0.rc1
>
> +1
> Build from source on Ubuntu with CUDA/CUDNN/MKLDNN and tested with
> keras-mxnet.
> Unit tests passed and example works on CPU/GPU.
>
>
> Best Regards
>
> Lai
>
>
> On Tue, Jan 7, 2020 at 11:49 AM Lin Yuan  wrote:
>
> > Correction: it was built from source on Ubuntu 16.04
> >
> > On Tue, Jan 7, 2020 at 11:42 AM Lin Yuan  wrote:
> >
> > > +1
> > >
> > > Build from source on Ubuntu 18 with CUDA/CUDNN/NCCL on and verified
> > > it works with Horovod 0.18.2
> > >
> > > On Tue, Jan 7, 2020 at 9:55 AM Przemysław Trędak
> > > 
> > > wrote:
> > >
> > >> Dear MXNet community,
> > >>
> > >> This is the vote to release Apache MXNet (incubating) version 1.6.0.
> > >> Voting starts today and will close on Friday 1/10/2020 23:59 PST.
> > >>
> > >> Link to release notes:
> > >> https://cwiki.apache.org/confluence/display/MXNET/1.6.0+Release+not
> > >> es
> > >>
> > >> Link to release candidate:
> > >> https://github.com/apache/incubator-mxnet/releases/tag/1.6.0.rc1
> > >>
> > >> Link to source and signatures on apache dist server:
> > >> https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.6.0.rc1/
> > >>
> > >> The differences comparing to previous release candidate 1.6.0.rc0:
> > >> * Fix for RNN gradient calculation for MKLDNN ([v1.6.x] Cherry-pick
> > >> MKL-DNN Rnn operator enhancements to v1.6.x (#17225))
> > >> * Fix for Windows CMake build (Backport #16980 #17031 #17018 #17019
> > >> to
> > >> 1.6 branch (#17213))
> > >> * CPU counterpart to contrib multihead attention operators
> > >> (Interleaved MHA for CPU path (#17138) (#17211))
> > >> * Fix for #16060 (fix norm sparse fallback (#17149))
> > >> * Fix for inconsistent names in estimator API (fix parameter names
> > >> in
> > the
> > >> estimator api (#17051) (#17162))
> > >> * Fixes for OpenMP (Backport 3rdparty/openmp fixes (#17193))
> > >> * Fix for pointwise fusion speed for large networks (which was the
> > reason
> > >> of -1 in the vote for rc0) as well as fixes for nondeterminism in
> > >> sum of squares operator and trainer parameter order (Backport
> > >> #17002, #17068
> > and
> > >> #17114 to 1.6 branch (#17137))
> > >>
> > >>
> > >> Please remember to TEST first before voting accordingly:
> > >> +1 = approve
> > >> +0 = no opinion
> > >> -1 = disapprove (provide reason)
> > >>
> > >>
> > >> Best regards,
> > >> Przemyslaw Tredak
> > >>
> > >
> >
>


--
*Chaitanya Prakash Bapat*
*+1 (973) 953-6299*

[image: https://www.linkedin.com//in/chaibapat25]
[image: https://www.facebook.com/chaibapat]
[image:
https://twitter.com/ChaiBapchya] [image:
https://www.linkedin.com//in/chaibapat25]

Re: [VOTE] Release Apache MXNet (incubating) 1.5.1.rc0

[Qing Lan]

+1 for Scala/Java test. Passed all tests for CPU/GPU build.
Also tested build from source with static build.

Thanks,
Qing

From: Tao Lv 
Sent: Tuesday, September 17, 2019 14:14
To: dev@mxnet.incubator.apache.org 
Subject: [VOTE] Release Apache MXNet (incubating) 1.5.1.rc0

Dear MXNet community,



This is the 3-day vote to release Apache MXNet (incubating) version 1.5.1.

Voting on dev@ will start September 17, 12:00pm (PST)  and close on
September 20, 12:00pm (PST).



1) Link to release notes:

https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Notes



2) Link to release candidate:

https://github.com/apache/incubator-mxnet/releases/tag/1.5.1.rc0



3) Link to source and signatures on Apache dist server:

https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.5.1.rc0/



Please remember to TEST first before voting accordingly:

+1 = approve

+0 = no opinion

-1 = disapprove (provide reason)



Thanks,

-tao

Re: [DISCUSS] Apache MXNet: Path to graduation

[Qing Lan]

Hi Lieven,

Thanks for your comments. After the discussion with several committers and 
contributors offline, we agreed that there are space for improvement.


  1.  About the Gluon naming

As we know, Gluon is born with the unique API design pattern. It gradually 
became the dominant Python front end for MXNet. I would suggest to discuss more 
with the Brand owner and see if there could be a further integration with 
MXNet. To MXNet itself, it becomes more popular with this frontend. We lean on 
the strong community and improve our product better by consuming the feedback 
from it.

 2. Diversity of the PMC
Currently, we have 40 PMC numbers from different companies, like Amazon, Uber, 
NVIDIA, ByteDance and a lot more. We are trying to grow the number and invite 
indivials from different companies as well as research institute.

3. Release rotation
In the history, most of the releases were done by the Amazon side. Currently, 
we are moving on to rotate this responsibility with contributors/committers not 
from Amazon to start working on them.

4. Committers from different firm/institution should have real work on MXNet
I can tell from the issues/PRs/rfcs they submitted and indeed and indeed we 
should encourage the committers who is less active to be involved into MXNet 
contribution.

Thanks,
Qing


From: Lieven Govaerts 
Sent: Saturday, August 10, 2019 5:59
To: dev@mxnet.incubator.apache.org 
Cc: d...@mxnet.apache.org 
Subject: Re: [DISCUSS] Apache MXNet: Path to graduation

Hi Qing,

as a user and ASF member observing this project:

On Sat, 10 Aug 2019 at 01:44, Qing Lan  wrote:

> Hi All,
>
> I would like to start a thread to discuss about the graduation for Apache
> MXNet. From my time working in the community, I saw a great improvement in
> most of the area that we do to make MXNet a better place. We keep tracking
> on all of the issues user raised and reviewing PRs. We follow the Apache
> Way to release the package in official repository.
>
>
in terms of code, documentation, visibility this project is certainly in a
healthy state, I see a lot of interest of companies and people, the
community is growing... As a user that gives me confidence my time invested
in this product is well spent.


> In 2017, Apache MXNet joined the Apache incubation project. I think now is
> a good time to review the path to graduate MXNet and move forward to it.
> Please feel free to share your thoughts on graduation and space for
> improvement.
>
>
If I may share one observation: I don't see the community working a lot on
non-code topics. One example that I personally find important is the
discussion of the Gluon brand. People have expressed confusion about how
the name is used by multiple non-ASF projects, the MXNet team finds the
Gluon name very valuable yet the discussion on how to protect the name and
decide on acceptable use by other projects has stalled [1]. I suggest you
make a decision on this topic before you go for graduation.

regards,

Lieven

[1]
https://mail-archives.apache.org/mod_mbox/mxnet-dev/201903.mbox/%3ccac_cu1gi+3s6ob48kt0x5wta4oxdum8uq9tmnyku2ujyaya...@mail.gmail.com%3e




> You can find more about graduation policy in here:
> https://incubator.apache.org/guides/graduation.html
>
> Thanks,
> Qing
>

[DISCUSS] Apache MXNet: Path to graduation

[Qing Lan]

Hi All,

I would like to start a thread to discuss about the graduation for Apache 
MXNet. From my time working in the community, I saw a great improvement in most 
of the area that we do to make MXNet a better place. We keep tracking on all of 
the issues user raised and reviewing PRs. We follow the Apache Way to release 
the package in official repository.

In 2017, Apache MXNet joined the Apache incubation project. I think now is a 
good time to review the path to graduate MXNet and move forward to it. Please 
feel free to share your thoughts on graduation and space for improvement.

You can find more about graduation policy in here:
https://incubator.apache.org/guides/graduation.html

Thanks,
Qing

Re: [Announcement] New Committer - Lai Wei

[Qing Lan]

Congrats Lai!


From: Kshitij Kalambarkar 
Sent: Sunday, August 4, 2019 0:19
To: dev@mxnet.incubator.apache.org 
Subject: Re: [Announcement] New Committer - Lai Wei

Congratulations Lai!

On Sun, Aug 4, 2019, 09:29 Zhao, Patric  wrote:

> Congratulation, Lai.
>
> Well done for the very challenge release 1.5 and you make the progress
> going smoothly 
>
>
> > -Original Message-
> > From: kellen sunderland 
> > Sent: Sunday, August 4, 2019 9:32 AM
> > To: dev@mxnet.incubator.apache.org
> > Subject: Re: [Announcement] New Committer - Lai Wei
> >
> > Congrats Lai.  Well deserved.
> >
> > On Sat, Aug 3, 2019, 6:18 PM Jake Lee  wrote:
> >
> > > Congratulations! Lai
> > >
> > > On Sat, Aug 3, 2019 at 6:13 PM Sheng Zha  wrote:
> > >
> > > > Hi all,
> > > >
> > > > Please join me in welcoming Lai (Roy) Wei as a new committer of
> > > > Apache MXNet (incubating)!
> > > >
> > > > Lai was one of the main contributor to the MXNet Keras frontend.
> > > > More recently, he contributed the Gluon estimator API, which enables
> > > > easy usage and better modularization for the training scripts of
> > > > Gluon. He also persisted and helped driving the long-running 1.5.0
> > > > release process.
> > > >
> > > > Welcome, Lai!
> > > >
> > > > -sz
> > > >
> > > >
> > >
>

Re: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc2

[Qing Lan]

Have successfully fixed the issue on OSX.

Scala/Java build is fine:

osx-cpupassed (Qing)
linux-cpu  passed (Zach)
linux-gpu  passed (Zach)

+1 for the release.

Thanks,
Qing



From: Qing Lan 
Sent: Monday, July 8, 2019 12:47
To: d...@mxnet.apache.org; dev@mxnet.incubator.apache.org
Subject: Re: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc2

Hi All,

I found the problem when I tried to build from source with my Mac:

clang: error: unsupported option '-fopenmp'
clang: error: unsupported option '-fopenmp'
make: *** [build/src/operator/nn/mkldnn/mkldnn_act.o] Error 1
make: *** [build/src/operator/nn/cudnn/cudnn_batch_norm.o] Error 1

I use "make -j4" with tar.gz package

Thanks,
Qing




From: Sheng Zha 
Sent: Friday, July 5, 2019 17:42
To: d...@mxnet.apache.org
Subject: Re: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc2

+1

On 2019/06/27 17:05:40, Lai Wei  wrote:
> Dear MXNet community,
>
> This is the 3-day vote to release Apache MXNet (incubating) version 1.5.0.
> Voting on dev@ will start June 26, 23:59:59(PST)  and close on June 29,
> 23:59:59.
>
> 1) Link to release notes:
> https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Notes
>
>
> 2) Link to release candidate:
>
> https://github.com/apache/incubator-mxnet/releases/tag/1.5.0.rc2
>
>
>
> 3) Link to source and signatures on apache dist server:
>
> https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.5.0.rc2/
>
>
>
> Please remember to TEST first before voting accordingly:
>
> +1 = approve
> +0 = no opinion
> -1 = disapprove (provide reason)
> --
> Best Regards
>
> Lai
>

Re: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc2

[Qing Lan]

Hi All,

I found the problem when I tried to build from source with my Mac:

clang: error: unsupported option '-fopenmp'
clang: error: unsupported option '-fopenmp'
make: *** [build/src/operator/nn/mkldnn/mkldnn_act.o] Error 1
make: *** [build/src/operator/nn/cudnn/cudnn_batch_norm.o] Error 1

I use "make -j4" with tar.gz package

Thanks,
Qing




From: Sheng Zha 
Sent: Friday, July 5, 2019 17:42
To: d...@mxnet.apache.org
Subject: Re: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc2

+1

On 2019/06/27 17:05:40, Lai Wei  wrote:
> Dear MXNet community,
>
> This is the 3-day vote to release Apache MXNet (incubating) version 1.5.0.
> Voting on dev@ will start June 26, 23:59:59(PST)  and close on June 29,
> 23:59:59.
>
> 1) Link to release notes:
> https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Notes
>
>
> 2) Link to release candidate:
>
> https://github.com/apache/incubator-mxnet/releases/tag/1.5.0.rc2
>
>
>
> 3) Link to source and signatures on apache dist server:
>
> https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.5.0.rc2/
>
>
>
> Please remember to TEST first before voting accordingly:
>
> +1 = approve
> +0 = no opinion
> -1 = disapprove (provide reason)
> --
> Best Regards
>
> Lai
>

Re: [DISCUSS] Make MXNet deploy it's own distribution

[Qing Lan]

In that case, the answer is yes. The Scala package will be published in one 
version with a variaty of backend package choices. User can easily attach and 
detach different MXNet versions. However, the Scala package cannot run without 
a backend.

Another key advantage of this design will be a broader support on different 
implementations such as Java Cpp. User will be able to implement their 
customized MXNet frontend to use the native library.

Thanks,
Qing


From: Sheng Zha 
Sent: Tuesday, July 2, 2019 22:14
To: dev@mxnet.incubator.apache.org
Subject: Re: [DISCUSS] Make MXNet deploy it's own distribution

Does it mean that the scala binding of mxnet will be an independent package 
that doesn’t directly depend on the native package, and user projects need to 
declare dependency on both the scala binding and one of the native packages?

-sz

> On Jul 2, 2019, at 5:50 PM, Frank Liu  wrote:
>
> Currently, MXNet were built along with different language bindings such as
> Scala.
>
> The libmxnet.so files are bundled within scala jar package.
>
> It would be nice to distribute libmxnet.so library independently in maven,
> and scala package can choose which native library to use.
>
> Here is the design document on cwiki:
> https://cwiki.apache.org/confluence/display/MXNET/Make+MXNet+deploy+it%27s+own+distribution
>
> Thanks,
>
> Frank

Re: Dependency Update

[Qing Lan]

Great work Jake! The content on CPU/GPU build instruction is really helpful.

Thanks,
Qing


From: Jake Lee 
Sent: Wednesday, May 22, 2019 17:26
To: dev@mxnet.incubator.apache.org
Subject: Dependency Update

Dear Community,

I have been working on dependency udpate of MXNet. The goal is to upgrade
the dependencies that have security vulnerability issues and make MXNet
great again by being benefit from latest CUDA, cuDNN and NCCL software. I
documented the process on PR<
https://github.com/apache/incubator-mxnet/pull/15045>. Big thanks to Sheng
Zha(szha), Dick Carter(DickJC123), Anirudh Subramanian(anirudh2290), Qing
Lan(lanking520), Per Goncalves da Silva(perdasilva) for supporting me!

Thanks,
Jake

[Announcement] New Committer - Zach Kimberg

[Qing Lan]

Hi All,

Please join me in welcoming Zach Kimberg (https://github.com/zachgk) as a new 
committer.

He has been solving some important bugs in MXNet JVM with respect to usage 
improvement, build issues and a lot more. He also created the Jenkins based 
publish pipeline for us to have standard way to build and test static-linked 
package conveniently for everyone in the community. Moreover, he solved a bunch 
of License problems we have in MXNet and brought several fixes to let us get 
1.4.0 release on time.

Thanks,
Qing

Re: [VOTE] Release Apache MXNet (incubating) version 1.4.1.rc0

[Qing Lan]

+1 (binding)

build from source works for OSX and Ubuntu CPU
Scala build/test successfully with Dynamic link and static link.

Thanks,
Qing


From: Sheng Zha 
Sent: Wednesday, May 1, 2019 13:14
To: d...@mxnet.apache.org
Subject: Re: [VOTE] Release Apache MXNet (incubating) version 1.4.1.rc0

Hi all,

Reminder that the vote for 1.4.1 release is still ongoing. If you can, please 
help out. Thank you.

-sz

On 2019/04/30 06:51:45, Junru Shao  wrote:
> Dear MXNet community,
>
> This is the 3-day vote to release Apache MXNet (incubating) version v1.4.1.
> The voting on dev@ list will start Apr 29 23:59:59 (PST) and close on May
> 02 23:59:59.
>
> Below are links to
> 1) Release notes:
> https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.4.1+Release+Notes
> .
> 2) Release Candidate:
> https://github.com/apache/incubator-mxnet/releases/tag/1.4.1.rc0.
> 3) Source and signatures on Apache dist server:
> https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.4.1.rc0/.
>
> Please remember to TEST first before voting accordingly:
> +1 = approve
> +0 = no opinion
> -1 = disapprove (provide reason)
>
> Best regards,
> Junru Shao
>

Re: [MXNET 2.0 Wishlist] [DISCUSS] Single build system

[Qing Lan]

+1 to have a single build system

Currently the way of publish and the way of doing CI test is very different. 
The instruction shown on the website should match the way we deliver it to the 
users.
Having a single build process would simplify the maintainance cost and reach to 
the best performance.

Thanks,
Qing


From: Marco de Abreu 
Sent: Thursday, April 4, 2019 15:01
To: dev@mxnet.incubator.apache.org
Subject: Re: [MXNET 2.0 Wishlist] [DISCUSS] Single build system

+1 towards having a single build system

I'd like to add the benefit of this approach allowing us to have the same
build logic across all operating systems. It would be great if we could
make x86/Unix, x86/windows, x86/mac and ARM/Unix first class citizens from
the beginning.

-Marco

Kellen Sunderland  schrieb am Do., 4. Apr. 2019, 12:31:

> Hello MXNet devs,
>
> I'd like to start a thread discussing what our build system should look
> like in MXNet 2.0.  I'd propose that although the current make system has
> served us well in the past, we remove it along with the bump to 2.0.  The
> end goal I'd like to see is that we have a clean build system, without a
> bunch of conditional logic that makes contributing and testing MXNet a
> simpler process.  Additionally I'd propose we target a minimum cmake
> version of 3.7 for reasons described below.
>
> First I'd like to give some context on why I'd propose we don't just switch
> to cmake, but we also target a relatively new version (version 3.7 from
> Nov, 2016) of cmake.  The largest benefits in making this change would
> apply to CUDA builds where cmake itself has quite inconsistent
> functionality between versions.  One persistent annoyance I've had with
> cmake is that we've had conditional logic for the FindCUDA command which at
> one point targeted some modern cmake features, but then in subsequent
> versions of cmake the way these features works was tweaked, and now I find
> these cmake features are consistently broken to the point that I require a
> bunch of -D defines to compile properly or to use an IDE.  An additional
> CUDA related issue is that every time there's a new SM added to NVCC we
> have to make a few source changes to support it.  I could see this being
> problematic for users who may suddenly realize that due to their
> compilation settings, they may not actually be enabling the features they
> think they are with their shiny new GPUs.
>
> As an alternative if we, for example, target cmake 3.7 at a minimum, and we
> want to find cuda and then build a list of reasonable PTX/BINS we could use
> the following command[1]:
>
> 
> FindCUDA(...)
> ...
> CUDA_SELECT_NVCC_ARCH_FLAGS(ARCH_FLAGS 3.0 3.5+PTX 5.2(5.0) Maxwell)
>   LIST(APPEND CUDA_NVCC_FLAGS ${ARCH_FLAGS})
> 
>
> Simple, concise, and it would help to make the building experience more
> consistent across platforms, build environments and IDEs (looking at you
> CLion).  We'd of course need to do a little experimentation work to make
> sure that this does indeed work as intended, and can replace the currently
> complex findCuda logic we have in our build systems, but for the sake of
> the proposal let's assume these cmake commands do indeed work consistently
> as documented from cmake 3.7 onwards.
>
> To give users a chance to update their tooling I'd also suggest we begin
> warning users at least a release in advance that make based builds will be
> deprecated in MXNet 2.0 so they can begin migrating to cmake.  I'd also
> want to display deprecation messages for unused cmake flags (such as the
> profiler flag) for a release before the 2.0 release, and then remove them
> in 2.0.
>
> Of course not all users have cmake 3.7 on their systems, some of our
> employers force use to use ridiculously outdated linux distributions.  The
> good news for these users is that if we can offer Docker compilation with
> an image that has a supported version of cmake and we should be able to
> build a portable binary that work even with very old distributions of
> Linux.  Additionally installing cmake from source is also fairly
> straightforward [2] and works quite well on older distros in my experience.
>
> Looking forward to hearing what others think.  Any preferred build systems
> that you all would want to use?  Is cmake the right system to centralize
> on?  If so, is version 3.7 a reasonable minimum version to target?  Is the
> 2.0 release a good point at which we can think about simplifying build
> logic?
>
> 1: https://cmake.org/cmake/help/v3.7/module/FindCUDA.html
> 2: https://github.com/Kitware/CMake
>

Re: R help

[Qing Lan]

Change to 
http://data.mxnet.io/models/imagenet/inception-bn/Inception-BN-0126.params
 would work. Please keep track on this PR:

https://github.com/apache/incubator-mxnet/pull/14504
[https://avatars2.githubusercontent.com/u/1753787?s=400=4]

Fixes for CI downloads by lebeg · Pull Request #14504 · 
apache/incubator-mxnet
Description Fixed for CI verification builds. Changes Removed the silent curl 
option for explicit errors Added curl download retries Replaced model download 
links from dmlc to mxnet
github.com



Thanks,
Qing


From: Anirudh Acharya 
Sent: Monday, March 25, 2019 13:47
To: dev@mxnet.incubator.apache.org
Subject: Re: R help

Yes, that is the error, need to dig deeper why that URL is not working.


Thanks
Anirudh


On Mon, Mar 25, 2019 at 10:40 AM kellen sunderland <
kellen.sunderl...@gmail.com> wrote:

> Is this the error?
> "test_model.R:129: error: Fine-tune
>
> cannot open URL
> 'http://data.dmlc.ml/models/imagenet/inception-bn/Inception-BN-0126.params
> '
> 1: GetInception() at R-package/tests/testthat/test_model.R:129
> 2: download.file("
> http://data.dmlc.ml/models/imagenet/inception-bn/Inception-BN-0126.params
> ",
>destfile = "model/Inception-BN-0126.params")"
>
> Looks like the
> http://data.dmlc.ml/models/imagenet/inception-bn/Inception-BN-0126.params
> is failing for me as well.
>
>
> On Mon, Mar 25, 2019 at 10:37 AM Anirudh Acharya 
> wrote:
>
> > Hi Per da Silva,
> >
> > Let me know if I can help, we can chat offline.
> >
> > From first glance it would seem
> >
> >- R:MKLDNN CPU is passing whereas R:CPU is failing
> >- R:GPU might have failed due to this "cannot open URL '
> >
> >
> http://data.dmlc.ml/models/imagenet/inception-bn/Inception-BN-0126.params
> >'"
> >
> >
> > Thanks
> > Anirudh
> >
> >
> > On Mon, Mar 25, 2019 at 7:34 AM Per da Silva 
> wrote:
> >
> > > Dear community,
> > >
> > > I'm working on a PR <
> > https://github.com/apache/incubator-mxnet/pull/14513>
> > > to update CI GPU jobs to be based on CUDA v10. However, for some
> reason,
> > > amongst other things, the R tests are failing
> > > <
> > >
> >
> http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-14513/4/pipeline
> > > >.
> > > I would really appreciate some help from the R experts to get it sorted
> > =D
> > >
> > > Thanks in advance,
> > >
> > > Per
> > >
> >
>

Re: [VOTE] Release Apache MXNet (incubating) version 1.4.0.rc3

[Qing Lan]

+1 (binding) on the release. Checked Mac + Linux (Ubuntu 16.04) build from 
source successfully. Checked Scala build with no errors.

On 2/15/19, 6:08 PM, "Piyush Ghai"  wrote:

Dear MXNet community,

I would like to propose a vote to release Apache MXNet (incubating) version 
v1.4.0.
Voting will start today, Friday February 15th 6pm PST and will close on 
Monday,
February 18th 6pm PST.

Link to release notes:


https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.4.0+Release+Notes
 


Link to release candidate 1.4.0.rc3:
 
https://github.com/apache/incubator-mxnet/releases/tag/1.4.0.rc3
 /

Link to source and signatures on apache dist server:
https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.4.0.rc3/ 
 


Please remember to TEST first before voting accordingly:
+1 = approve
+0 = no opinion
-1 = disapprove (provide reason)


Best regards,
Piyush

[DISCUSS] upgrade openblas

[Qing Lan]

Dear Community,

Recently I observed a performance regression on a MXNet after switching Atlas 
to openblas 0.3.3. OpenBLAS community address this issue in their 0.3.5 
release. So I plan to upgrade openblas in our publish system to gain the 
performance improvement, here to discuss if anyone had concerns with this.

Currently this upgrade seemed to bring a negative impact to Julia: 
https://github.com/xianyi/OpenBLAS/issues/1955

Reference:
https://github.com/xianyi/OpenBLAS/issues/1897

Thanks,
Qing

Re: [RESTARTING][VOTE] Release Apache MXNet (incubating) version 1.4.0.rc2

[Qing Lan]

Hi Michael, 

Could you please guide how to proceed with this? Given that we have a 
possibility of announcing MXNet support in Horovod with their next release and 
this would help MXNet increase our visibility.

Thanks,
Qing

On 2/12/19, 2:16 PM, "Michael Wall"  wrote:

Team,

Here is my read on the situation.  The vote has been canceled.  Justin's
point was that a -1 doesn't mean you must cancel a vote for the reasons he
outlined.  But here the vote needs to be restarted and the issue Luciano
found needs to be addressed.

That issue is that there are files in MXNet source tree that do not have
the required licenses headers,
http://www.apache.org/legal/release-policy.html#license-headers.  For
example, the top level README.md is missing the header
https://raw.githubusercontent.com/apache/incubator-mxnet/master/README.md.
Excluding 3rd party files from the RAT check is fine, but not files
originating from the MXNet repo.

It would be good to know exactly how Luciano ran the RAT check, cc'd.  Here
is a link to the thread

https://lists.apache.org/thread.html/51e9ab05edae2089c74a253000a92d5aa5c6406f54e5bd0a0b3c3879@%3Cgeneral.incubator.apache.org%3E
.

Justin's other point, aIso cc'd, was that the vote with the podling doesn't
have to take 72 hours before going to the incubator list.

I realize this is not what everyone is pushing for, so interested in
other's thoughts.  Especially other mentors.

Mike

On Tue, Feb 12, 2019 at 4:47 PM Aaron Markham 
wrote:

> +1
> I disagree about 3rd party considerations. This is an ecosystem after all.
> The distributed training story is quite nice with Horovod. Given my
> interaction with tensorflow with  Horovod and dynamic training with MXNet
> and the kvstore, this new route is, IMO, easier to setup and manage.
> I see the benefit for getting it out there sooner than later, and market
> timings are important to the project and adoption. If Uber's announcement
> drives traffic to MXNet, but then people can't set it up with a stable
> release package, there's a lost opportunity for growing the community. Why
> miss the opportunity for a RAT license?
>
> On Tue, Feb 12, 2019, 13:14 Dave Fisher  wrote:
>
> > Hi -
> >
> > Third party vendor considerations do not matter. Are you voting +1 with
> > your Apache hat on or your Amazon hat?
> >
> > Regards,
> > Dave
> >
> > > On Feb 11, 2019, at 10:16 PM, Lin Yuan  wrote:
> > >
> > > +1 binding
> > > Horovod is going to release it's 0.16.0 in the coming week with MXNet
> > > integration. We need to release 1.4.0 which includes all the
> dependencies
> > > for Horovod integration.
> > >
> > > Best,
> > >
> > > Lin
> > >
> > > On Mon, Feb 11, 2019 at 9:30 PM Steffen Rochel <
> steffenroc...@gmail.com>
> > > wrote:
> > >
> > >> Dear community -
> > >> based on Justin's and community feedback I'm suggesting to restart 
the
> > >> vote.
> > >> Current status:
> > >> binding votes:
> > >> +1: 2 votes (Henri, Jason)
> > >> -1:  1 vote (Luciano)
> > >>
> > >> non-binding:
> > >> +1: 1 vote (Kellen)
> > >>
> > >> The community is investigating feedback from Luciano that the
> exclusion
> > >> file is to broad and potentially missing files which can and must 
have
> > >> apache license headers not to be checked.
> > >>
> > >> Regards,
> > >> Steffen
> > >>
> > >>
> > >>
> > >>
> > >> On Mon, Feb 11, 2019 at 10:08 AM Hagay Lupesko 
> > wrote:
> > >>
> > >>> Based on Justin's feedback, can we resume the vote instead of
> > cancelling
> > >>> it?
> > >>>
> > >>> On Mon, Feb 11, 2019 at 12:02 AM Justin Mclean <
> > jus...@classsoftware.com
> > >>>
> > >>> wrote:
> > >>>
> >  Hi,
> > 
> >  In future don’t be so hasty to cancel a release vote, people mind
> can
> > >> be
> >  changed and a -1 is not a veto on a release.
> > 
> >  Thanks,
> >  Justin
> > 
> > 
> > 
> -
> >  To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> >  For additional commands, e-mail: general-h...@incubator.apache.org
> > 
> > 
> > >>>
> > >>
> >
> >
>

Re: [CANCELLED][VOTE] Release Apache MXNet (incubating) version 1.4.0.rc2

[Qing Lan]

Hi All,

Can we move the VOTE forward since the RAT license should not be a problem that 
block the release. We can always add that one in our future releases (e.g 1.4.1 
or 1.5.0).

As you may aware, 1.4.0 release started very early this year and delayed a 
couple of times until now. From the Apache Release process: 
http://www.apache.org/legal/release-policy.html, it should be fine for us to 
move forward if majority provide +1 than -1. 

Again, move forward does not mean we should not fix it or ignore problems that 
exist in the code. We will address and fix them in the next release.

Thanks,
Qing

On 2/10/19, 10:28 PM, "Steffen Rochel"  wrote:

Dear community -
I'm cancelling the vote due to -1 feedback from Luciano due to RAT
failures.
For details see

https://lists.apache.org/thread.html/51e9ab05edae2089c74a253000a92d5aa5c6406f54e5bd0a0b3c3879@%3Cgeneral.incubator.apache.org%3E

The MXNet community will discuss next steps.

Regards,
Steffen

Re: [Announcement] New Committer -- Lin Yuan

[Qing Lan]

Congrats Lin!
> 
> 
> Congratulations Lin
> 
>> On Sat, Feb 2, 2019, 3:27 PM Tianqi Chen > 
>> Dear Community:
>> 
>> Please join me to welcome Lin Yuan(@apeforest) as a new committer of
>> Apache(incubating) MXNet!
>> 
>> He has contributed to various improvements, including better compatibility
>> of larger arrays across the codebase.
>> 
>> Commits:
>> https://github.com/apache/incubator-mxnet/commits?author=apeforest
>> 
>> https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+author%3Aapeforest
>> 
>> 
>> Reviews:
>> https://github.com/apache/incubator-mxnet/pulls?utf8=%
>> E2%9C%93=reviewed-by%3Aapeforest
>> 
>> dev@ activitivity
>> https://lists.apache.org/list.html?*@mxnet.apache.org:lte=6M:Lin%20Yuan
>> 
>> Tianqi
>> 

[DISCUSS] Current Publish problems

[Qing Lan]

Hi all,

Recently Zach announced the availability for MXNet Maven publishing pipeline 
and general static-build instructions. In order to make it better, I drafted a 
document that includes the problems we have for this pipeline: 
https://cwiki.apache.org/confluence/display/MXNET/Outstanding+problems+with+publishing.
 Some of them may need to be addressed very soon.

Please kindly review and leave any comments you may have in this thread or in 
the document.

thanks,
Qing

Re: Question about Gluon Api for Scala package & JVM langs

[Qing Lan]

Hi Carin,

Thanks for your question. I would like to know which part(s) of Gluon you would 
like to see for JVM in general?
Since itself if big, we can start working on the components users needed the 
most and then cover the rest of the part. 
Please feel free to draft a proposal somewhere and we can discuss about it.

Thanks,
Qing

On 1/21/19, 4:53 PM, "Carin Meier"  wrote:

Currently the Scala package supports the Module and FeedForward APIs. Since
quite a bit of the documentation is focusing now on the Gluon API, I
wondered if there were thoughts of bringing the Gluon interface to the
Scala package in the future for the JVM langs.

- Carin

Re: [ANNOUNCE] Jenkins Nightly Release Pipeline with MXNet Scala

[Qing Lan]

Great work Zach! 
Please reach out to Scala community if you are interested to learn more about 
the nightly build for MXNet Scala.

Thanks,
Qing

On 1/19/19, 8:54 AM, "Carin Meier"  wrote:

Thanks for everyone involved in this effort. This not only is a huge win
for the Scala community but also for the Clojure community which depends on
the Scala jar artifact.

Well done.

On Fri, Jan 18, 2019 at 9:58 PM Zach Kimberg 
wrote:

> Hi,
>
> A little over a month ago, we announced the nightly build of the Scala
> package on Nexus [1]. It featured the same statically linked binary build
> logic used by the Python pip to make the adoption as easy for our JVM 
users
> as for our python users. However, that release occurred on a private
> pipeline using code that was not publicly available.
>
> First, I would like to thank Sheng for contributing the pip binary build
> scripts to the community and making them accessible as part of the MXNet
> repository [2]. Now, everyone can produce similar published artifacts for
> their own needs and we can better verify the release production code as
> part of the Jenkins CI.
>
> Using his contribution, we have created a new job on the MXNet Jenkins for
> publishing artifacts on a nightly basis [3]. In order to ensure the 
highest
> quality for our releases regardless of user system, it will automatically
> test the artifacts across other distributions including Ubuntu 16.04,
> Ubuntu 18.04, and CentOS 7 as part of the deployment.
>
> You can find the code for the nightly publish pipeline on the repo [4]. We
> hope that others can work off of this pipeline to help expand the same
> static building and thorough testing to additional MXNet packages and
> language bindings in the future.
>
> Special thanks to Qing for his help throughout the project, Sheng for the
> binary build logic, Marco for his reviews and support working with 
Jenkins,
> Anton for setting up the development Jenkins for us, and Frank and Naveen
> for work on the Scala maven build and deployment.
>
> Zach
>
> [1] - https://repository.apache.org/#nexus-search;quick~org.apache.mxnet
> [2] -
> https://github.com/apache/incubator-mxnet/tree/master/tools/staticbuild
> [3] -
>
> 
http://jenkins.mxnet-ci.amazon-ml.com/job/restricted-publish-artifacts/job/master/
> [4] - https://github.com/apache/incubator-mxnet/tree/master/ci/publish
>

Re: Taxonomy on our cwiki

[Qing Lan]

Agreed +1. 
Could we draft a plan on CWIKI and let's sign up our name to migrate the pages 
to the right location?

Thanks,
Qing

On 1/21/19, 6:18 AM, "Anton Chernov"  wrote:

A quick tip about links to the wiki pages, note the difference in links:

* https://cwiki.apache.org/confluence/display/MXNET/Release+Process (1)
* https://cwiki.apache.org/confluence/x/BINjB (2)

If sharing was done via the 'Share' menu the link (2) would persist after
any structual movements.

Best
Anton


сб, 19 янв. 2019 г. в 16:49, Pedro Larroy :

> +1
>
> On Sat, Jan 19, 2019 at 2:51 PM Zhao, Patric 
> wrote:
> >
> > +1, Good idea.
> >
> > It's not very easy to find out the related contents since lots of
> folders in the website.
> >
> >
> > > -Original Message-
> > > From: Sheng Zha [mailto:zhash...@apache.org]
> > > Sent: Saturday, January 19, 2019 3:28 AM
> > > To: dev@mxnet.incubator.apache.org
> > > Subject: Taxonomy on our cwiki
> > >
> > > Hi MXNet,
> > >
> > > Given that currently cwiki is the only place other than mxnet website
> for
> > > mxnet-related documentation, I'd like to request your attention to the
> > > (slightly disorganized) cwiki page of MXNet. The top level folders
> (and their
> > > contents) currently looks like this:
> > > - Design Proposals* (bag of proposals, not in order)
> > > - Development* (mixture of guides, roadmaps, processes)
> > > - Release Process (release notes)
> > > - Website (guides and proposals)
> > > - MXNet Clojure (call for contribution, guides)
> > > - MXNet Keras Integration (design)
> > > - MXNet-ONNX Integration (design, dev status)
> > > - MXNet R Package (guide, backlog)
> > > - MXNet-Scala (design, dev status, guide)
> > > - Content Formatting Templates (not a folder but link to two docs)
> > > - How-to articles (1 guide)
> > > - Community (guide on apache-related processes)
> > > - Data IO (designs)
> > > - Continuous Integration (guides, designs)
> > > - Meetups and Hangouts (events)
> > >
> > > And here are two good examples from successful Apache projects:
> > > - Apache Flink: an **audience-oriented** structure [1]
> > >   Users (Presentations and How-to)
> > >   Contributors (Dev processes and How-to)
> > >   Committers (Infra, Dev processes, Release processes, Releases)
> > >   Roadmaps and Feature Designs (archive)
> > > - Apache OpenNLP: a **content-oriented** structure [2]
> > >   Guides
> > >   External Resources
> > >   Proposals
> > >   Releasing
> > >
> > > Clean organization helps content discovery and saves time on locating
> useful
> > > content. Given that we have good amount of content on the wiki page, I
> > > suggest that we decide on a cleaner taxonomy, re-organize contents
> > > accordingly, and add future contents accordingly. To provide a
> starting point
> > > for the discussion, I suggest:
> > > - Given the state we are in, start with content-oriented organization,
> use
> > > these top-level categories: Guides (including processes and how-tos),
> > > Development (including designs, proposals, notes, roadmaps), Community
> > > (including events, activities, external resources and contents)
> > > - If people strongly prefer audience-oriented structure, later we can
> adopt a
> > > structure similar to Flink's.
> > >
> > > Feel free to share your thoughts and preferences here. Thanks.
> > >
> > > -sz
> > >
> > > [1]
> > >
> https://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+Homehttp
> > > s://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+Home
> > > [2] https://cwiki.apache.org/confluence/display/OPENNLP/Index
>

Re: [Article] Object Detection with Clojure MXNet

[Qing Lan]

That looks cool! So glad to see the active community of Clojure.
BTW, Piyush (piyushghai) plans to add the bounding box support for Scala this 
week.

Thanks,
Qing

On 1/20/19, 7:29 AM, "Carin Meier"  wrote:

Thanks - props for the images belongs to the original Scala article

https://medium.com/apache-mxnet/object-detection-with-mxnet-scala-inference-api-9049230c77fd
and Kedar - the contributor that ported the object detection feature to
Clojure. They are pretty awesome though.



On Sat, Jan 19, 2019 at 4:57 PM Marco de Abreu 
wrote:

> Great article! I have to admit I always love your picture choices :D
>
> -Marco
>
> Am Sa., 19. Jan. 2019, 21:57 hat Carin Meier 
> geschrieben:
>
> > I just blogged about the new Object Detection feature that was just
> ported
> > from the Scala package into the Clojure package.
> >
> >
> 
http://gigasquidsoftware.com/blog/2019/01/19/object-detection-with-clojure-mxnet/
> >
> > I posted it in Slack but in an effort to direct more communication
> towards
> > this mailing list, I'm putting it out here too :)
> >
> > - Carin
> >
>

[DISCUSS] Make MKLDNN as a default on Maven nightly build

[Qing Lan]

Hi all,

I would like to raise a discussion on whether to make MKLDNN as a default in 
nightly build (1.5.0-SNAPSHOT) for MXNet Scala/Java binding. Currently Scala 
build with MKLDNN is supported since 
https://github.com/apache/incubator-mxnet/pull/13819 with CI. I do see the 
performance increase when dealing with the inference and it is also necessary 
to get it in nightly for beta-testing in order to make it official in 1.5.0.

Thanks,
Qing

Re: joining slack channel

[Qing Lan]

Invite sent.

-Qing

On 1/9/19, 2:18 PM, "louisfeng.list@"  
wrote:

Could I please get an invite to the Slack channel too?

Thanks,
Louis

On 2019/01/04 16:09:03, "Lv, Tao A"  wrote: 
> Invitation is sent to mue...@amazon.com. You can find mxnet in ASF 
channels.
> 
> -Original Message-
> From: Muenz, Edison [mailto:mue...@amazon.com.INVALID] 
> Sent: Friday, January 4, 2019 11:46 PM
> To: dev@mxnet.incubator.apache.org
> Subject: Re: joining slack channel
> 
> Hi, can I get an invite to the Slack channel as well?
> 
> From: Aaron Markham 
> Sent: Friday, January 4, 2019 3:42 PM
> To: dev@mxnet.incubator.apache.org
> Subject: Re: joining slack channel
> 
> I sent you an invite.
> Cheers,
> Aaron
> 
> On Fri, Jan 4, 2019 at 1:12 PM István Fehérvári  wrote:
> 
> > Please add me to the slack channel.
> >
> > Thank you,
> > Istvan
> >
> 
> 
> 
> Amazon Development Center Germany GmbH
> Krausenstr. 38
> 10117 Berlin
> Geschaeftsfuehrer: Christian Schlaeger, Ralf Herbrich
> Ust-ID: DE 289 237 879
> Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
> 
> 
> 

Re: Apache MXNet v1.4.0 release status

[Qing Lan]

Hi all,

I added a section F in the document that explained the current static-linked 
dependencies we used for official release. As there are a few licenses are 
under BSD3 and GPL, we need to handle them in our next release. Please take a 
look and leave any concerns you may have.

Thanks,
Qing

On 1/7/19, 8:33 PM, "kellen sunderland"  wrote:

So I see two quick options that should cut down on the dependency licenses
required for TRT in the source release.

1: We can simply remove in the release package the submodules for onnx in
folder incubator-mxnet/3rdparty/onnx-tensorrt/third_party/onnx/third_party.
None of those dependencies are used in the build (I've just verified
locally on my machine).
2: We can make a cmake based checkout system and ensure we only checkout
the required files when TRT builds are enabled (similar to the current
mkl-ml setup).

I'd suggest option 1 for this release, and that we support option 2 for the
1.5 release.

On Mon, Jan 7, 2019 at 8:19 PM Lv, Tao A  wrote:

> What should I do for the double headers in 3rdparty/mkldnn/src/cpu/xbyak/?
>
> -tao
>
> -Original Message-
> From: Steffen Rochel [mailto:steffenroc...@gmail.com]
> Sent: Tuesday, January 8, 2019 10:51 AM
> To: dev@mxnet.incubator.apache.org
> Subject: Re: Apache MXNet v1.4.0 release status
>
> Kellen and Tao -
> yes, the understanding is that dependencies need to be considered and all
> licences referenced to include in top level LICENSE file.
> Appreciate your help with it.
> Steffen
>
> On Mon, Jan 7, 2019 at 6:39 PM kellen sunderland <
> kellen.sunderl...@gmail.com> wrote:
>
> > Sorry to hear about the licensing issues.  I was following the general
> > vote but I'm still lacking some clarity around what licenses in the
> > onnx-trt repo need to be surfaced.  I believe onnx-trt is MIT
> > licensed, but it includes Onnx as a third party repo which then brings
> > in dependencies with a variety of licenses.  The proposal is that we
> > look at these on an individual basis and then add them to our top level
> LICENSE file right?
> >
> > An alternative is that we may be able to checkout a smaller source
> > code dependency tree if we remove a few unneeded ONNX's dependencies
> > (pybind and google-bench).  My hope is that this wouldn't affect our
> > compilation process and would get us down to two licenses to report
> > (just Onnx and Onnx-TRT, both MIT).
> >
> > On Mon, Jan 7, 2019 at 6:07 PM Meghna Baijal
> > 
> > wrote:
> >
> > > Hi All,
> > > For some more context, these were the last emails I sent on the dev
> > > and legal lists requesting help on the open questions  –
> > >
> > > 1. Question on legal about the CC-By-2.5 <
> > >
> > http://mail-archives.apache.org/mod_mbox/www-legal-discuss/201805.mbox
> > /%3CCAK1xzDe6ECToKt_2cTR_7txQQCwHeYfvxXDfmuGgfA3jaTs=j...@mail.gmail.com
> > %3E
> > > >
> > > 2. Question on dev about googletest file <
> > >
> > http://mail-archives.apache.org/mod_mbox/mxnet-dev/201804.mbox/%3CCAMG
> > gKDC8szdfFqQhhSNpwwT_3zi4LBS7A=u4v7kj4ule44u...@mail.gmail.com%3E
> > > >
> > > 3. General Request for review of the licenses wiki <
> > >
> > https://mail-archives.apache.org/mod_mbox/mxnet-dev/201801.mbox/%3CCAM
> > GgKDCi=s933zcVWwei15i5uBC1h88VUogt3Br=Vq28=vi...@mail.gmail.com%3E
> > > >
> > >
> > >  (Note: You can click on the the “>>” next to the thread on the top
> > > right to view the next responses in the email threads in the apache
> > > archive. )
> > >
> > > Thanks,
> > > Meghna Baijal
> > >
> > > On Mon, Jan 7, 2019 at 4:30 PM Steffen Rochel
> > > 
> > > wrote:
> > >
> > > > Dear MXNet community -
> > > > as you should have seen in previous email, voting for v1.4.0.rc0
> > > > has
> > been
> > > > cancelled. We received a -1 vote due to outstanding license issues.
> > > > Please help to update
> > > >
> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+Source+License
> > s
> > > > and
> > > > resolve outstanding issues.
> > > >
> > > > I would like to ask specifically for help from contributors to
> > > > mkldnn, opemmp and onnx-tensorrt to address the feedback from
> > > > Justin - see
> > > >
> > > >
> > >
> > https://lists.apache.org/thread.html/ebb8c4c00fb66dd98da13621c7dcb8753
> > fee57562a861d61379d31e9@%3Cgeneral.incubator.apache.org%3E
> > > > .
> > > >
> > > > I suggest to fix the issues first on master, then cherry-pick and
> > > > merge
> > > to
> > > > 1.4.x branch.
> > > >
> > > > I'm suggesting to exclude Julia from 1.4.0 release as integration
> > > > into MXNet repo and upgrade to 0.7+ is WIP.
   

Re: [Announcement] New Committer - Roshani Nagmote

[Qing Lan]

Congrats Roshani! Great to have you here!

-Qing

> 
> 
> Congrats Roshani.  Well deserved.
> 
>> On Tue, Jan 8, 2019, 8:29 AM Marco de Abreu > 
>> Great to have you on board, Roshani!
>> 
>> -Marco
>> 
>> Am Di., 8. Jan. 2019, 15:18 hat Carin Meier 
>> geschrieben:
>> 
>>> Please join me in welcoming Roshani Nagmote as a new committer.
>>> 
>>> She has been active in the project for quite some time. She has managed
>> the
>>> 1.3.0 release as well as being involved various features including the
>> Java
>>> API and ONNX operators.
>>> 
>>> We are excited to have her onboard as a committer.
>>> 
>>> Github Activity
>>> 
>>> 
>> https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+involves%3ARoshrini
>>> +
>>> 
>>> Confluence
>>> 
>>> 
>> https://cwiki.apache.org/confluence/users/viewuserprofile.action?username=roshrini
>>> 
>> 

Malformed package uploaded to Maven central

[Qing Lan]

Dear Community,

Recently I tried to improve the Maven automated publish procedure and tested 
publish the package. However, I accidently used maven to upload a package to a 
close release branch: 
https://repository.apache.org/#nexus-search;gav~org.apache.mxnet~~1.5.0~~/.
 However, it seemed that I didn’t have the access to remove this package since 
it is controlled by Maven central. In this case, I regretfully request a 
PMC/PPMC member to file an Apache Infra ticket to remove this package from 
there so it won’t influence the current maven users to download them. The 
influence is limited to OSX users who are using official releases of MXNet 
Scala/Java packages.

I apologize for this act and won’t do any more risky experiment until I am 
fully aware of the consequence of it.
Qing

Re: [DISCUSS] About the usage of CUDA/CUDNN

[Qing Lan]

Hi Kellen,

Firstly the restricted node is completely isolated to the PR-checking CI system 
(physically) which is explained in here: 
https://cwiki.apache.org/confluence/display/MXNET/Restricted+jobs+and+nodes.
What you are mentioning: the Public CIs are all having troubles if they are 
public accessible. I am not sure how secure the restricted node is. However, 
the only way I can think of from your end is to downloading all deps in a 
single machine and run everything there (disconnected from internet). It would 
bring us the best security we have. 

Thanks,
Qing

On 12/17/18, 2:06 PM, "kellen sunderland"  wrote:

I'm not in favour of publishing artifacts from any Jenkins based systems.
There are many ways to bundle artifacts and publish them from an automated
system.  Why we would use a CI system like Jenkins for this task?  Jenkins
frequently has security vulnerabilities and is designed to run arbitrary
code from the internet.  It is a real possibility that an attacker could
pivot from any Jenkins based CI system to infect artifacts which would then
potentially be pushed to repositories our users would consume.  I would
consider any system using Jenkins as insecure-by-design, and encourage us
to air-gapped any artifact generation (websites, jars, PyPi packages)
completely from a system like that.

An alternative I could see is a simple Dockerfile (no Jenkins) that builds
all artifacts end-to-end and can be run in an automated account well
outside our CI account.

On Mon, Dec 17, 2018 at 1:53 PM Qing Lan  wrote:

> Dear community,
>
> Currently me and Zach are working on the Automated-publish pipeline on
> Jenkins which is a pipeline used to publish Maven packages and pip 
packages
> nightly build. We are trying to use NVIDIA deb which could help us to 
build
> different CUDA/CUDNN versions in the publish system. Sheng has provided a
> script here: https://github.com/apache/incubator-mxnet/pull/13646. This
> provide a very concrete and automatic solution from downloading to
> installing on the system. The only scenario we are facing is: It seemed
> NVIDIA has a restriction on distributing CUDA. We are not sure if it is
> legally-safe for us to use this in public.
>
> We would be grateful if somebody has a better context on it and help us
> out!
>
> Thanks,
> Qing
>

[DISCUSS] About the usage of CUDA/CUDNN

[Qing Lan]

Dear community,

Currently me and Zach are working on the Automated-publish pipeline on Jenkins 
which is a pipeline used to publish Maven packages and pip packages nightly 
build. We are trying to use NVIDIA deb which could help us to build different 
CUDA/CUDNN versions in the publish system. Sheng has provided a script here: 
https://github.com/apache/incubator-mxnet/pull/13646. This provide a very 
concrete and automatic solution from downloading to installing on the system. 
The only scenario we are facing is: It seemed NVIDIA has a restriction on 
distributing CUDA. We are not sure if it is legally-safe for us to use this in 
public.

We would be grateful if somebody has a better context on it and help us out!

Thanks,
Qing

[DISCUSS] About the PR merging policy

[Qing Lan]

Hi all,

Recently I self-merged my PR without getting approvals from other committers 
https://github.com/apache/incubator-mxnet/pull/13617 and only contributors 
approval. I apologize to the community and thank Marco for pointing out the 
problem. I took a lesson that we should at least have one committer’s approval 
to merge the code. However, I just found this section is missing in the CWiki 
https://cwiki.apache.org/confluence/display/MXNET/Become+an+Apache+MXNet+%28incubating%29+Committer+and+PPMC+Member.
 So I would like to discuss in here:

How to conduct the PR reviewing/merging. How many approvals (Committers and 
Contributors) we should get in order to merge?

How to deal with disagreement in the discussion (e.g a contributor/committer 
request a change)?

Please don’t hesitate to share your thoughts!

Thanks,
Qing

Re: [Announce] Upcoming Apache MXNet (incubating) 1.4.0 release

[Qing Lan]

Hi all,
I have a critical bug-fix PR 
https://github.com/apache/incubator-mxnet/pull/13330 that essentially fix the 
problems for supporting inference with different shape in Scala/Java 
(introduced in v1.1). I would like to request to cherry-pick this one in 1.4.

Thanks,
Qing

On 11/29/18, 3:00 PM, "Pedro Larroy"  wrote:

PR is ready from my side and passes the tests, unless somebody raises
any concerns it's good to go.
On Thu, Nov 29, 2018 at 9:50 PM Steffen Rochel  
wrote:
>
> Pedro - added  to 1.4.0 tracking list
> 

>
> Do you have already ETA?
> Steffen
>
> On Thu, Nov 29, 2018 at 6:13 AM Pedro Larroy 

> wrote:
>
> > Hi all.
> >
> > There are two important issues / fixes that should go in the next
> > release in my radar:
> >
> > 1) https://github.com/apache/incubator-mxnet/pull/13409/files
> > There is a bug in shape inference on CPU when not using MKL, also we
> > are running activation on CPU via MKL when we compile CUDNN+MKLDNN.
> > I'm finishing a fix for these issues in the above PR.
> >
> > 2) https://github.com/apache/incubator-mxnet/issues/13438
> > We are seeing crashes due to unsafe setenv in multithreaded code.
> > Setenv / getenv from multiple threads is not safe and is causing
> > segfaults. This piece of code (the handlers in pthread_atfork) already
> > caused a very difficult to diagnose hang in a previous release, where
> > a fork inside cudnn would deadlock the engine.
> >
> > I would remove setenv from 2) as a mitigation, but we would need to
> > check for regressions as we could be creating additional threads
> > inside the engine.
> >
> > I would suggest that we address these two major issues before the next
> > release.
> >
> > Pedro
> >
> >
> >
> > On Sun, Nov 25, 2018 at 11:41 PM Steffen Rochel 

> > wrote:
> > >
> > > Dear MXNet community,
> > >
> > > I will be the release manager for the upcoming Apache MXNet 1.4.0
> > release.
> > > Sergey Kolychev will be co-managing the release and providing help 
from
> > the
> > > committers side.
> > > A release candidate will be cut on November 29, 2018 and voting will
> > start
> > > December 7, 2018. Release notes have been drafted here [1]. If you 
have
> > any
> > > additional features in progress and would like to include it in this
> > > release, please assure they have been merged by November 27, 2018.
> > Release
> > > schedule is available here [2].
> > >
> > > Feel free to add any other comments/suggestions. Please help to review
> > and
> > > merge outstanding PR's and resolve issues impacting the quality of the
> > > 1.4.0 release.
> > >
> > > Regards,
> > >
> > > Steffen
> > >
> > > [1]
> > >
> > 
https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.4.0+Release+Notes
> > >
> > > [2]
> > 
https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.4.0+Release+Plan+and+Status
> > >
> > >
> > >
> > >
> > > On Tue, Nov 20, 2018 at 7:15 PM kellen sunderland <
> > > kellen.sunderl...@gmail.com> wrote:
> > >
> > > > Spoke too soon[1], looks like others have been adding Turing 
support as
> > > > well (thanks to those helping with this).  I believe there's still a
> > few
> > > > changes we'd have to make to claim support though (mshadow CMake
> > changes,
> > > > PyPi package creation tweaks).
> > > >
> > > > 1:
> > > >
> > > >
> > 
https://github.com/apache/incubator-mxnet/commit/2c3357443ec3d49a11e93c89f278264ce10c2f08
> > > >
> > > > On Tue, Nov 20, 2018 at 7:00 PM kellen sunderland <
> > > > kellen.sunderl...@gmail.com> wrote:
> > > >
> > > > > Hey Steffen, I'd like to be able to merge this PR for version 1.4:
> > > > > https://github.com/apache/incubator-mxnet/pull/13310 . It fixes a
> > > > > regression in master which causes incorrect feature vectors to be
> > output
> > > > > when using the TensorRT feature.  (Thanks to Nathalie for helping 
me
> > > > track
> > > > > down the root cause of the issue).   I'm currently blocked on a CI
> > issue
> > > > I
> > > > > haven't seen before, but hope to have it resolved by EOW.
> > > > >
> > > > > One call-out I would make is that we currently don't support 
Turing
> > > > > architecture (sm_75).  I've been slowly trying to add support, 
but I
> > > > don't
> > > > > think I'd have capacity to do this done by EOW.  Does anyone feel
> > > > strongly
> > > > > we need this in the 1.4 release?  From my 

Re: Scala Symbol API Question

[Qing Lan]

Hi Sam, 

please join the Slack MXNet. We also have a #MXNet-Scala channel there where 
you can ask more questions with Scala/Java.

Apart from that I have placed an issue for you:
https://github.com/apache/incubator-mxnet/issues/13403

Thanks,
Qing

On 11/23/18, 11:22 PM, "Sam Bean"  wrote:

Hello, I have some questions about the Scala API for the Symbol library.

I'm trying to figure out how to do something like this
https://github.com/ufoym/mxnet/blob/master/example/vae/VAE.py#L83, however
it seems the Scala Symbol API does not allow the mixing of symbols and
constants like the python library does.

It seems like if I want to use constants in my loss functions I'm going to
have to have a very large argsDict when I go to train since every variable
in the loss function definition will have to be symbolic. Is there a better
way to do this?

-- 
Sam Bean
*StockX*
*Tech Lead, Machine Learning and Personalization*
*––*
samb...@stockx.com
stockx.com

Re: CI impaired

[Qing Lan]

Appreciated for your effort and help to make CI a better place!

Qing 

On 11/21/18, 4:38 PM, "Lin Yuan"  wrote:

Thanks for your efforts, Marco!

On Wed, Nov 21, 2018 at 4:02 PM Anirudh Subramanian 
wrote:

> Thanks for the quick response and mitigation!
>
> On Wed, Nov 21, 2018 at 3:55 PM Marco de Abreu
>  wrote:
>
> > Hello,
> >
> > today, CI had some issues and I had to cancel all jobs a few minutes 
ago.
> > This was basically caused by the high load that is currently being put 
on
> > our CI system due to the pre-release efforts for this Friday.
> >
> > It's really unfortunate that we just had outages of three core 
components
> > within the last two days - sorry about that!. To recap, we had the
> > following outages (which are unrelated to the parallel refactor of the
> > Jenkins pipeline):
> > - (yesterday evening) The Jenkins master ran out of disk space and thus
> > processed requests at reduced capacity
> > - (this morning) The Jenkins master got updated which broke our
> > autoscalings upscaling capabilities.
> > - (new, this evening) Jenkins API was irresponsive: Due to the high
> number
> > of jobs and a bad API design in the Jenkins REST API, the 
time-complexity
> > of a simple create or delete request was quadratic which resulted in all
> > requests timing out (that was the current outage). This resulted in our
> > auto scaling to be unable to interface with the Jenkins master.
> >
> > I have now made improvements to our REST API calls which reduced the
> > complexity from O(N^2) to O(1). The reason was an underlying redirect
> loop
> > in the Jenkins createNode and deleteNode REST API in combination with
> > unrolling the entire slave and job graph (which got quite huge during
> > extensive load) upon every single request. Since we had about 150
> > registered slaves and 1000 jobs in the queue, the duration for a single
> > REST API call rose to up to 45 seconds (we execute up to a few hundred
> > queries per auto scaling loop). This lead to our auto scaling timing 
out.
> >
> > Everything should be back to normal now. I'm closely observing the
> > situation and I'll let you know if I encounter any additional issues.
> >
> > Again, sorry for any caused inconveniences.
> >
> > Best regards,
> > Marco
> >
> > On Wed, Nov 21, 2018 at 5:10 PM Gavin M Bell 
> > wrote:
> >
> > > Yes, let me add to the kudos, very nice work Marco.
> > >
> > >
> > > "I'm trying real hard to be the shepherd." -Jules Winnfield
> > >
> > >
> > > > On Nov 21, 2018, at 5:04 PM, Sunderland, Kellen
> > >  wrote:
> > > >
> > > > Appreciate the big effort in bring the CI back so quickly.  Thanks
> > Marco.
> > > >
> > > > On Nov 21, 2018 5:52 AM, Marco de Abreu <
> marco.g.ab...@googlemail.com
> > .INVALID>
> > > wrote:
> > > > Thanks Aaron! Just for the record, the new Jenkins jobs were
> unrelated
> > to
> > > > that incident.
> > > >
> > > > If somebody is interested in the details around the outage:
> > > >
> > > > Due to a required maintenance (disk running full), we had to upgrade
> > our
> > > > Jenkins master because it was running on Ubuntu 17.04 (for an 
unknown
> > > > reason, it used to be 16.04) and we needed to install some packages.
> > > Since
> > > > the support for Ubuntu 17.04 was stopped, this resulted in all
> package
> > > > updates and installations to fail because the repositories were 
taken
> > > > offline. Due to the unavailable maintenance package and other issues
> > with
> > > > the installed OpenJDK8 version, we made the decision to upgrade the
> > > Jenkins
> > > > master to Ubuntu 18.04 LTS in order to get back to a supported
> version
> > > with
> > > > maintenance tools. During this upgrade, Jenkins was automatically
> > updated
> > > > by APT as part of the dist-upgrade process.
> > > >
> > > > In the latest version of Jenkins, some labels have been changed 
which
> > we
> > > > depend on for our auto scaling. To be more specific:
> > > >> Waiting for next available executor on mxnetlinux-gpu
> > > > has been changed to
> > > >> Waiting for next available executor on ‘mxnetlinux-gpu’
> > > > Notice the quote characters.
> > > >
> > > > Jenkins does not offer a better way than to parse these messages
> > > > unfortunately - there's no standardized way to express queue items.
> > Since
> > > > our parser expected the above message without quote signs, this
> message
> > > was
> > > > discarded.
> > > >
> > > > We support various queue reasons (5 of them to be exact) that
> indicate
> > > > resource starvation. If we run super low on capacity, 

Re: [ANNOUNCEMENT] New Committer: Qing Lan

[Qing Lan]

Thanks everyone! Looking forward to contribute more to the community!

Thanks,
Qing

On 11/20/18, 6:31 PM, "Steffen Rochel"  wrote:

Congratulation Qing!

On Tue, Nov 20, 2018 at 6:29 PM Hagay Lupesko  wrote:

> Congrats Qing! Awesome to see you become a committer!
>
> On Tue, Nov 20, 2018 at 4:26 PM Marco de Abreu
>  wrote:
>
> > Great to have your on board, Qing!
> >
> > Am Mi., 21. Nov. 2018, 01:24 hat Naveen Swamy 
> > geschrieben:
> >
> > > The Project Podling Management Committee (PPMC) for Apache MXNet has
> > > invited Qing Lan based on his contribution to MXNet Scala to become a
> > > committer and we are pleased to announce that he has accepted.
> > >
> > > Qing, thanks a lot for your contribution and continued effort to
> support
> > > MXNet community.
> > >
> > > Please join me in welcoming Qing to the project!
> > >
> > > Thanks, Naveen
> > > (on behalf of Apache MXNet PPMC)
> > >
> >
>

Re: [VOTE] Separating PMC and Committership

[Qing Lan]

+1 (non-binding)

On 11/8/18, 2:41 PM, "kellen sunderland"  wrote:

+1 (non-binding)

On Thu, Nov 8, 2018 at 10:37 AM Thomas DELTEIL 
wrote:

> +1 (non-binding)
>
> Le jeu. 8 nov. 2018 à 10:04, Carin Meier  a écrit :
>
> > Reminder - Vote ends tomorrow-  Friday Nov 9th at 6:00 am EST
> >
> > On Mon, Nov 5, 2018 at 11:29 AM Carin Meier 
> wrote:
> >
> > > This is a procedural vote on whether to separate the committer and 
PPMC
> > > levels in the project. The current state is that a user is considered
> as
> > > both a committer and a PPMC member at the same time. This vote is to
> > change
> > > that to be able to invite a person in as a committer separately from a
> > PPMC
> > > member.
> > >
> > > Document reference:
> > >
> >
> 
https://cwiki.apache.org/confluence/display/MXNET/Become+an+Apache+MXNet+%28incubating%29+Committer+and+PPMC+Member
> > >
> > > Discussion thread:
> > >
> >
> 
https://lists.apache.org/thread.html/9c6ecda02e081aa6b689c92badc9dcf05ced6fb3691fd370471773d1@%3Cdev.mxnet.apache.org%3E
> > >
> > > The vote will be a procedural issue vote as defined
> > > https://www.apache.org/foundation/voting.html
> > >
> > > Votes on procedural issues follow the common format of majority rule
> > unless
> > > otherwise stated. That is, if there are more favourable votes than
> > > unfavourable ones, the issue is considered to have passed -- 
regardless
> > of
> > > the number of votes in each category. (If the number of votes seems 
too
> > > small to be representative of a community consensus, the issue is
> > typically
> > > not pursued. However, see the description of lazy consensus
> > >  for a
> > > modifying factor.)
> > >
> > > The vote will run until Friday Nov 9th at 6:00 am EST
> > >
> > > Thanks,
> > > Carin
> > >
> > >
> > >
> >
>

Re: [Discussion] Recognise Reviewers, Besides Committers and PMC

[Qing Lan]

+1
Let's have a reviewer list somewhere with a certain format: such as C++, Gluon, 
Scala/Java based on language or some other category. etc. In the future, label 
bot would automatically assign reviewers based on this kind of documentation.

Thanks,
Qing

On 10/21/18, 11:44 PM, "YiZhi Liu"  wrote:

+1
I also suggest add reviewer list link to the PR template, so that
developers can easily request review from those reviewers.
On Sun, Oct 21, 2018 at 8:30 PM Tianqi Chen  wrote:
>
> I was suggesting something more concrete:
>
> - Add a Reviewers section to
> https://github.com/apache/incubator-mxnet/blob/master/CONTRIBUTORS.md to
> list a list of Reviewers.
> - This is a "pesudo role", but holds weight as committers should 
highly
> value their reviews during the PR process.
> - The committers/PMC could actively look for good contributors and 
nominate
> them as Reviewer.
> - Contributors are encouraged to seek reviews from the list of reviewers.
> - The committers should actively solicit code reviews from the reviewers
> when reviewing PRs and take their reviews into serious consideration.
>
> - PMCs should actively look for new committers in the current Reviewers
>- Notably, the history reviews plus contribution likely will provide a
> good indication on whether the person can uphold the quality standard of
> the codebase, and provide helpful feedbacks(which is the trait that needed
> from committer to merge code)
>
> Tianqi
>
>
> On Sun, Oct 21, 2018 at 5:13 PM Steffen Rochel 
> wrote:
>
> > +1
> > With the release announcement for MXNet 1.3 all contributors incl. code
> > reviewers have been recognized. I suggest all future release 
announcements
> > should include such recognition. Are you suggesting to highlight most
> > active reviewers in release announcement or regularly (e.g. monthly),
> > specifically from non-committers?
> >
> > On Sun, Oct 21, 2018 at 10:11 AM Tianqi Chen  wrote:
> >
> > > Also re another email-thread(I sent out one with my institutional 
email
> > > which get blocked initially, so this one was a bit duplication of 
that).
> > I
> > > think it should really be the job of committers to recognize potential
> > > reviewers, github also makes it easier to do so, e.g.
> > >
> > >
> > 
https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=reviewed-by%3Apiiswrong
> > >
> > > Tianqi
> > >
> > > On Fri, Oct 19, 2018 at 12:05 PM Carin Meier 
> > wrote:
> > >
> > > > +1 Great idea. Adding a name to the contributor list is a good idea.
> > > Also,
> > > > I've found that thanking the person for the review on the PR is 
another
> > > way
> > > > to express gratitude for their time and effort.
> > > >
> > > > On Fri, Oct 19, 2018 at 2:51 PM Tianqi Chen  
wrote:
> > > >
> > > > > Dear MXNet Community:
> > > > >
> > > > > There is a great discussion going on in terms of lowering the 
barrier
> > > of
> > > > > entries and encourage more contribution to the project.  One of 
the
> > > > general
> > > > > goals is to encourage a broader pool of contributions. I want to 
make
> > > the
> > > > > following proposal:
> > > > >
> > > > > Besides Committers and PMC, let us also recognize Reviewers in the
> > > > > community.  This is a "pseudo role" as there is no such official 
role
> > > in
> > > > > Apache. But I want to explore the possibility of recognizing 
active
> > > > > reviewers for example, by adding a list of names in the 
contributor
> > > list.
> > > > > In general, I find it is really helpful to have more code reviews.
> > > > > Recognizing good reviewers early enables us to find committer
> > > candidates,
> > > > > and encourage them to contribute and understand what is the bar of
> > code
> > > > > quality that is required to merge the code.
> > > > >
> > > > > This can provide the community with more evidence when recruiting 
new
> > > > > committers. After all the write access of committership is about 
to
> > the
> > > > > code and understand the consequence of the responsibility -- 
which is
> > > > > usually can be found in high-quality review history.
> > > > >
> > > > > Please let me know what you think.
> > > > >
> > > > > Tianqi
> > > > >
> > > >
> > >
> >



-- 
Yizhi Liu
DMLC member
Amazon Web Services
Vancouver, Canada

Re: MXNet - Label Bot functionality

[Qing Lan]

Hi Harsh,

This new label bot design looks great! I would like to encourage people to 
review it and move forward to benefit the MXNet community.
Since this new design needs webhook support from Apache, let's go through the 
following steps to get this done:

1. Demo and contributors review stage: all contributors are encouraged to 
review the PR here:
https://github.com/MXNetEdge/mxnet-infrastructure/pull/15 and leave your 
thoughts so Harsh can apply them in his design.
2. Committers review stage: Once all contributors think the design is good to 
go, let's get committers involved to get a review.
3. Committers send request to Apache Infra to get the webhook setup.
4. Harsh finally deploy the model and all of us can use it in incubator-mxnet 
repo!

Some fun fact I would like to share:
1. This new bot can recommend labels and reply to people who file it!
2. It response time from 5mins -> less than 20 seconds

Thanks,
Qing

On 10/15/18, 11:11 AM, "Harsh Patel"  wrote:

Hey,
I have a demo available that users and developers can play around with --
this is in regards to the post I had made regarding the updated label bot
functionality. This is available on my fork (
https://github.com/harshp8l/incubator-mxnet) if the developers would be
able to provide feedback that would be great.
The updated usage of this label bot:
To add labels: @mxnet-label-bot, add ['label1', 'label2']
To remove labels: @mxnet-label-bot, remove ['label1', 'label2']
To update labels: @mxnet-label-bot, update ['label3', 'label4']
(warning: with update - this will remove all other labels for a specific
issue and update only with the labels the user specifies). Thanks.

My PR for reference:
https://github.com/MXNetEdge/mxnet-infrastructure/pull/15

My Design:

https://cwiki.apache.org/confluence/display/MXNET/Machine+Learning+Based+GitHub+Bot

Best,
-Harsh

On Mon, Oct 15, 2018 at 12:54 AM Hagay Lupesko  wrote:

> +1
> Thanks for the contribution!
>
> On Fri, Oct 12, 2018 at 1:41 AM kellen sunderland <
> kellen.sunderl...@gmail.com> wrote:
>
> > Awesome work!  Many thanks.
> >
> > On Fri, Oct 12, 2018, 1:19 AM Harsh Patel 
> > wrote:
> >
> > > Hey,
> > > I am looking to contribute to MXNet. I have a working implementation
> > based
> > > on my proposed design structure according to this wiki page (
> > >
> > >
> >
> 
https://cwiki.apache.org/confluence/display/MXNET/Machine+Learning+Based+GitHub+Bot
> > > )
> > > - under 7.
> > > I have provided users with functionality allowing for adding, 
updating,
> > and
> > > deleting labels for our issues. The response time with the bot to
> provide
> > > the aforementioned functionality has been reduced as well. Users 
should
> > > expect speedy updates to labels based on requests made to the label
> bot.
> > > The total number of GitHub API calls have been further reduced as well
> > > preventing potential bottlenecks that could result from GitHub.
> > > I would like to have a webhook for this repo:
> > > https://github.com/apache/incubator-mxnet so that this functionality
> > will
> > > be used and tested by the developers here. Thanks.
> > >
> > > Best,
> > > -Harsh Patel
> > >
> >
>

Re: Time out for Travis CI

[Qing Lan]

From the link it looks like "Travis CI offers a free account" instead of Apache 
buy it. It may just be a free user account with extension on the numbers of 
nodes it can runs on. I think we may need to reach out to Travis or Apache to 
clarify that we currently have the service that paid version have instead of an 
extension of "free user account".

Thanks,
Qing

On 10/1/18, 6:15 PM, "kellen sunderland"  wrote:

I actually thought we were already using a paid plan through Apache
https://blogs.apache.org/infra/entry/apache_gains_additional_travis_ci

    On Tue, Oct 2, 2018, 3:11 AM Qing Lan  wrote:

> Are we currently on a free plan? If we are, probably the unlimited build
> minutes would help
>
> Thanks,
> Qing
>
> On 10/1/18, 6:08 PM, "kellen sunderland" 
> wrote:
>
> Does the global time out change for paid plans?  I looked into it
> briefly
> but didn't see anything that would indicate it does.
>
> On Tue, Oct 2, 2018, 2:25 AM Pedro Larroy <
> pedro.larroy.li...@gmail.com>
> wrote:
>
> > I think there's two approaches that we can take to mitigate the
> build &
> > test time problem, in one hand use a paid travis CI plan, in other
> improve
> > the unit tests in suites and only run a core set of tests, as we
> should do
> > on devices, but on this case we reduce coverage.
> >
> > https://travis-ci.com/plans
> >
> > Pedro.
> >
> > On Sat, Sep 29, 2018 at 6:53 PM YiZhi Liu 
> wrote:
> >
> > > This makes sense. Thanks
> > >
> > > On Sat, Sep 29, 2018 at 6:36 PM kellen sunderland <
> > > kellen.sunderl...@gmail.com> wrote:
> > >
> > > > Hey Zhennan, yes this is the exact problem, and I agree with 
your
> > points
> > > > completely.  This is why when we first added Travis we attempted
> to
> > > > communicate that it would be informational only, and that we'd
> need to
> > > > iterate on the config before it would be a test that people
> should
> > > consider
> > > > 'required'.  Apologies, we should have been more straightforward
> about
> > > > those tradeoffs.  The strong point in favour of adding Travis in
> > > > informational mode was that we had a serious MacOS specific bug
> that we
> > > > wanted to verify was fixed.
> > > >
> > > > The good news is I've opened a PR which I hope will speed up
> these
> > builds
> > > > to the point that they won't rely on caching.  Once it is merged
> it
> > would
> > > > be very helpful if you could rebase on this PR and test to
> ensure that
> > > > large changes no longer hit the global timeout without cache.
> > > > https://github.com/apache/incubator-mxnet/pull/12706
> > > >
> > > > On Sun, Sep 30, 2018 at 2:48 AM Qin, Zhennan <
> zhennan@intel.com>
> > > > wrote:
> > > >
> > > > > Hi YiZhi and Kellen,
> > > > >
> > > > > From my point of view, travis should be able to get passed
> from a
> > > scratch
> > > > > build. Pending result on ccache hit/miss is not a good idea.
> For this
> > > PR,
> > > > > as it changed many header file, lots of files need be
> recompiled,
> > just
> > > > like
> > > > > a scratch build. I think that's the reason that travis
> timeout. This
> > > > should
> > > > > be fixed before enabling travis, as it will block any change
> to those
> > > > base
> > > > > header file. Again, it's not a special case with this PR only,
> you
> > can
> > > > find
> > > > > same problem on other PRs:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> 
https://travis-ci.org/apache/incubator-mxnet/builds/433172088?utm_source=github_status_medium=notification
> > > >

Re: Time out for Travis CI

[Qing Lan]

Are we currently on a free plan? If we are, probably the unlimited build 
minutes would help

Thanks,
Qing

On 10/1/18, 6:08 PM, "kellen sunderland"  wrote:

Does the global time out change for paid plans?  I looked into it briefly
but didn't see anything that would indicate it does.

On Tue, Oct 2, 2018, 2:25 AM Pedro Larroy 
wrote:

> I think there's two approaches that we can take to mitigate the build &
> test time problem, in one hand use a paid travis CI plan, in other improve
> the unit tests in suites and only run a core set of tests, as we should do
> on devices, but on this case we reduce coverage.
>
> https://travis-ci.com/plans
>
> Pedro.
>
> On Sat, Sep 29, 2018 at 6:53 PM YiZhi Liu  wrote:
>
> > This makes sense. Thanks
> >
> > On Sat, Sep 29, 2018 at 6:36 PM kellen sunderland <
> > kellen.sunderl...@gmail.com> wrote:
> >
> > > Hey Zhennan, yes this is the exact problem, and I agree with your
> points
> > > completely.  This is why when we first added Travis we attempted to
> > > communicate that it would be informational only, and that we'd need to
> > > iterate on the config before it would be a test that people should
> > consider
> > > 'required'.  Apologies, we should have been more straightforward about
> > > those tradeoffs.  The strong point in favour of adding Travis in
> > > informational mode was that we had a serious MacOS specific bug that 
we
> > > wanted to verify was fixed.
> > >
> > > The good news is I've opened a PR which I hope will speed up these
> builds
> > > to the point that they won't rely on caching.  Once it is merged it
> would
> > > be very helpful if you could rebase on this PR and test to ensure that
> > > large changes no longer hit the global timeout without cache.
> > > https://github.com/apache/incubator-mxnet/pull/12706
> > >
> > > On Sun, Sep 30, 2018 at 2:48 AM Qin, Zhennan 
> > > wrote:
> > >
> > > > Hi YiZhi and Kellen,
> > > >
> > > > From my point of view, travis should be able to get passed from a
> > scratch
> > > > build. Pending result on ccache hit/miss is not a good idea. For 
this
> > PR,
> > > > as it changed many header file, lots of files need be recompiled,
> just
> > > like
> > > > a scratch build. I think that's the reason that travis timeout. This
> > > should
> > > > be fixed before enabling travis, as it will block any change to 
those
> > > base
> > > > header file. Again, it's not a special case with this PR only, you
> can
> > > find
> > > > same problem on other PRs:
> > > >
> > > >
> > > >
> > >
> >
> 
https://travis-ci.org/apache/incubator-mxnet/builds/433172088?utm_source=github_status_medium=notification
> > > >
> > > >
> > >
> >
> 
https://travis-ci.org/apache/incubator-mxnet/builds/434404305?utm_source=github_status_medium=notification
> > > >
> > > >
> > > > Thanks,
> > > > Zhennan
> > > >
> > > > -Original Message-
> > > > From: YiZhi Liu [mailto:eazhi@gmail.com]
> > > > Sent: Sunday, September 30, 2018 5:15 AM
> > > > To: eazhi@gmail.com
> > > > Cc: dev@mxnet.incubator.apache.org
> > > > Subject: Re: Time out for Travis CI
> > > >
> > > > while other PRs are all good.
> > > > On Sat, Sep 29, 2018 at 2:13 PM YiZhi Liu 
> wrote:
> > > > >
> > > > > Honestly I don't know yet. I can help to investigate. Just given
> the
> > > > > evidence that, travis timeout every time it gets re-triggered - 2
> > > > > times at least. Correct me if I'm wrong @ Zhennan On Sat, Sep 29,
> > 2018
> > > > > at 1:54 PM kellen sunderland  wrote:
> > > > > >
> > > > > > Reading over the PR I don't see what aspects would cause extra
> > > > > > runtime YiZhi, could you point them out?
> > > > > >
> > > > > > On Sat, Sep 29, 2018 at 8:46 PM YiZhi Liu 
> > > wrote:
> > > > > >
> > > > > > > Kellen, I think this PR introduces extra runtime in CI, thus
> > > > > > > causes the timeout. Which means, once merged, every PR later
> will
> > > > > > > see same timeout in travis.
> > > > > > >
> > > > > > > So shall we modify the changes to decrease the test running
> time?
> > > > > > > or just disable the Travis CI?
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Sep 28, 2018 at 9:17 PM Qin, Zhennan
> > > > > > > 
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Kellen,
> > > > > > > >
> > > > > > > > Thanks for your explanation. Do you have a time plan to 
solve
> > > > > > > > the
> > > > > > > timeout issue? Rebasing can't work for my case. Or shall we 
run
> > it
> > > > > > > silently to disallow it voting X for overall CI result? 
Because
> > 

Re: Feedback request for new Java API

[Qing Lan]

t;>> implicit for
>>>>>>>>>>>>> backward compatibility and use null instead of None,
>>>> Andrew
>>>>> and
>>>>>>> I
>>>>>>>>>> disagreed
>>>>>>>>>>>>> on this, so I suggested to raise a discussion on dev@
>>> to
>>>>> get
>>>>>>> more
>>>>>>>>>> opinions
>>>>>>>>>>>>> and one of us will disagree and commit. Thanks for
>>> raising
>>>>> it :)
>>>>>>>>>>>>> | * def Activation (data : org.apache.mxnet.NDArray,
>>>>> act_type :
>>>>>>>>>> String, out
>>>>>>>>>>>>> : NDArray = null) :
>> org.apache.mxnet.NDArrayFuncReturn |
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) It is not true that Scala users will lose
>>>>> *default/optional*
>>>>>>>>>> arguments -
>>>>>>>>>>>>> if we followed the above, they will use null or None,
>>>>> though I
>>>>>>> do not
>>>>>>>>>> like
>>>>>>>>>>>>> using nulls, this is a fine compromise.
>>>>>>>>>>>>> To keep backward compatibility we can create a
>> implicit
>>> to
>>>>>>> convert
>>>>>>>>>>>>> Option.None to nulls and Option.Some-> Option.get(),
>> so
>>>> you
>>>>> are
>>>>>>> not
>>>>>>>>>> going
>>>>>>>>>>>>> to break users who might have been using the APIs that
>>>> were
>>>>>>> released
>>>>>>>>>> in
>>>>>>>>>>>>> 1.3. The current incompatibility is only this w.r.t.
>>>>> NDArrays.
>>>>>>>>>>>>> 2) Now about the Scala Macros - they are not simple to
>>>> read
>>>>> or
>>>>>>> use,
>>>>>>>>>> When I
>>>>>>>>>>>>> and Qing started working on the #Scala Macros to
>> improve
>>>> the
>>>>>>> APIs, it
>>>>>>>>>> took
>>>>>>>>>>>>> us a good amount of time to get a hang of it. I don't
>>> want
>>>>> to
>>>>>>> add
>>>>>>>>>>>>> additional code when not necessary.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My suggestion and vote is to modify existing
>> Macro(i.e.,
>>>> #1
>>>>>>> from the
>>>>>>>>>>>>> original email with the necessary clarification above)
>>> and
>>>>> make
>>>>>>> it
>>>>>>>>>>>>> compatible with Java
>>>>>>>>>>>>> Here are my reasons
>>>>>>>>>>>>> 1) The NDArray APIs in question are not following
>>>> functional
>>>>>>> style of
>>>>>>>>>>>>> programming, in fact they are just static methods
>>> defined
>>>>> on an
>>>>>>>>>> NDArray
>>>>>>>>>>>>> object - so Scala users are not losing much by using
>>> null
>>>> in
>>>>>>> place of
>>>>>>>>>> None.
>>>>>>>>>>>>> You can create a implicit to maintain backward
>>>> compatibility
>>>>>>>>>>>>> 2) It is adding 220+ APIs(I understand it is
>> generated)
>>>> for
>>>>>>> NDArray
>>>>>>>>>> alone
>>>>>>

Re: Feedback request for new Java API

[Qing Lan]

or a a Java wrapper
> around
> > the Scala API. Adding a bunch of Java friendly methods inside the Scala
> > code would create a mess for users. Maintenance would be essentially the
> > same for both because either way you're going to be updating Java 
methods
> > when you make Scala changes.
> >
> > Let's please stick with the issue in the original email.
> >
> > Thanks,
> > Andrew
> >
> > On Thu, Sep 27, 2018 at 5:22 PM Qing Lan  wrote:
> >
> > > I would like to loop this back a layer. Current, there is a discussion
> in
> > > the MXNet Scala community on the ways to implement the Java APIs.
> Currently
> > > there are two thoughts:
> > >
> > > 1. Make Scala Java Friendly (Create Java compatible methods in the
> Scala
> > > Class. such as NDArray with Java compatible constructor)
> > > 2. Make Java friendly wrappers in Scala (Andrew's explanation below)
> > >
> > > The first approach require minimum input from our side to implement
> > > however bring user a bunch of useless api they may not want to use. It
> also
> > > makes Scala package heavier. The good thing is these two packages
> require
> > > minimum maintenance cost. As a tradeoff, if any time in the future we
> want
> > > to make Java big (make Java as the primary language supported by
> MXNet),
> > > then the migration from Scala to Java will be harmful. Spark consider
> this
> > > carefully and decide not to change much on their Scala code base to
> make it
> > > more Java.
> > >
> > > The second approach will make unique NDArray, Shape, Context and more.
> The
> > > good thing about this is we can always holds a version control on 
Java.
> > > Some breaking changes on Scala may not influence much on Java. It did
> the
> > > best way to decouple the module and good for us to build unique
> pipeline
> > > for Java. The bad thing with this design is the maintenance cost as we
> need
> > > to keep two code bases, but it also make Java side easy to change to
> make
> > > it better compatible with users.
> > >
> > > Thanks,
> > > Qing
> > >
> > > On 9/27/18, 3:25 PM, "Andrew Ayres"  wrote:
> > >
> > > Hi,
> > >
> > > Currently, we're working to implement a new Java API and would 
like
> > > some
> > > feedback from the community on an implementation detail. In short,
> the
> > > new
> > > Java API will use the existing Scala API (in a manner similar to
> how
> > > the
> > > current Clojure API works). This basically means that we're making
> Java
> > > friendly wrappers to call the existing Scala API.
> > >
> > > The feedback we're looking for is on the implementation of 
NDArray.
> > > Scala's
> > > NDArray has a significant amount of code which is generated via
> macros
> > > and
> > > we've got two viable paths to move forward:
> > >
> > > 1.) Change the macro to generate Java friendly methods  - To do
> this
> > > we'll
> > > modify the macro so that the generated methods won't have
> > > default/optional
> > > arguments. There may also have to be some changes to parameter
> types to
> > > make them Java friendly. The big advantage here is that ongoing
> > > maintenance
> > > will easier. The disadvantages are that we'll be changing the
> existing
> > > Scala NDArray Infer API (it's marked experimental) and Scala users
> will
> > > lose the ability to use the default and optional arguments.
> > >
> > > 2.) Leave the existing macro in place and add another which
> generates a
> > > Java friendly version - The biggest issue here is that we'll be
> > > doubling
> > > the number of macros that we've got to maintain. It'll become even
> more
> > > overhead once we start expanding the Java API with more classes
> that
> > > use
> > > generated code like this. The advantages are that the existing
> Scala
> > > NDArray Infer API would remain unchanged for Scala users and that
> the
> > > new
> > > macro could be optimized to give the best possible experience to
> the
> > > Java
> > > API.
> > >
> > > Thanks,
> > > Andrew
> > >
> > >
> > >
>
>
>
> --
> Yizhi Liu
> DMLC member
> Amazon Web Services
> Vancouver, Canada
>

Re: Reformulating to a more efficient design of Mxnet-label-Bot

[Qing Lan]

Great work Harsh! I like your webhook design. This would allow us to do a great 
more for the label bot and speed up the response time.

-Marco: I think Anton means the "Assignees" field in issues and PRs

Thanks,
Qing
On 9/27/18, 5:06 PM, "Marco de Abreu"  
wrote:

You mean like a replacement for the codeowners feature?

Anton Chernov  schrieb am Fr., 28. Sep. 2018, 01:39:

> As a feature request: Could we include detection and proposal of reviewers
> to the bot as well?
>
> Anton
>
> чт, 27 сент. 2018 г. в 15:27, Harsh Patel :
>
> > Hey,
> > I'm Harsh Patel, and I am looking to contribute to MXNet. I wanted to 
get
> > some feedback to improvements that could be made with the current
> structure
> > that we have for automatically labelling issues and pull requests. I 
have
> > linked my proposed design structure on the bottom of this wiki page (
> >
> >
> 
https://cwiki.apache.org/confluence/display/MXNET/Machine+Learning+Based+GitHub+Bot
> > )
> > - it should be under 7. Overall, users will benefit from this design
> since
> > it will allow adding, updating, and deleting of labels freely. Label
> > creation will be faster since this model focuses on labelling an issue 
as
> > soon as it is made. Another key benefit is that we minimize the number 
of
> > total GitHub API calls that need to be made. Feedback would be much
> > appreciated - I would like to hear what the developers have to say!
> Thanks.
> >
>

Re: Feedback request for new Java API

[Qing Lan]

I would like to loop this back a layer. Current, there is a discussion in the 
MXNet Scala community on the ways to implement the Java APIs. Currently there 
are two thoughts:

1. Make Scala Java Friendly (Create Java compatible methods in the Scala Class. 
such as NDArray with Java compatible constructor)
2. Make Java friendly wrappers in Scala (Andrew's explanation below)

The first approach require minimum input from our side to implement however 
bring user a bunch of useless api they may not want to use. It also makes Scala 
package heavier. The good thing is these two packages require minimum 
maintenance cost. As a tradeoff, if any time in the future we want to make Java 
big (make Java as the primary language supported by MXNet), then the migration 
from Scala to Java will be harmful. Spark consider this carefully and decide 
not to change much on their Scala code base to make it more Java.

The second approach will make unique NDArray, Shape, Context and more. The good 
thing about this is we can always holds a version control on Java. Some 
breaking changes on Scala may not influence much on Java. It did the best way 
to decouple the module and good for us to build unique pipeline for Java. The 
bad thing with this design is the maintenance cost as we need to keep two code 
bases, but it also make Java side easy to change to make it better compatible 
with users.

Thanks,
Qing

On 9/27/18, 3:25 PM, "Andrew Ayres"  wrote:

Hi,

Currently, we're working to implement a new Java API and would like some
feedback from the community on an implementation detail. In short, the new
Java API will use the existing Scala API (in a manner similar to how the
current Clojure API works). This basically means that we're making Java
friendly wrappers to call the existing Scala API.

The feedback we're looking for is on the implementation of NDArray. Scala's
NDArray has a significant amount of code which is generated via macros and
we've got two viable paths to move forward:

1.) Change the macro to generate Java friendly methods  - To do this we'll
modify the macro so that the generated methods won't have default/optional
arguments. There may also have to be some changes to parameter types to
make them Java friendly. The big advantage here is that ongoing maintenance
will easier. The disadvantages are that we'll be changing the existing
Scala NDArray Infer API (it's marked experimental) and Scala users will
lose the ability to use the default and optional arguments.

2.) Leave the existing macro in place and add another which generates a
Java friendly version - The biggest issue here is that we'll be doubling
the number of macros that we've got to maintain. It'll become even more
overhead once we start expanding the Java API with more classes that use
generated code like this. The advantages are that the existing Scala
NDArray Infer API would remain unchanged for Scala users and that the new
macro could be optimized to give the best possible experience to the Java
API.

Thanks,
Andrew

Some feedback from MXNet Zhihu topic

[Qing Lan]

Hi all,

There was a trend topic in Zhihu (a 
famous Chinese Stackoverflow+Quora) asking about the status of MXNet in 2018 
recently. Mu replied the thread and obtained more than 300+ `like`.
However there are a few concerns addressed in the comments of this thread, I 
have done some simple translation from Chinese to English:

1. Documentations! Until now, the online doc still contains:
1. Depreciated but not updated doc
2. Wrong documentation with poor description
3. Document in Alpha stage such as you must install `pip –pre` 
in order to run.

2. Examples! For Gluon specifically, many examples are still mixing Gluon/MXNet 
apis. The mixure of mx.sym, mx.nd mx.gluon confused the users of what is the 
right one to choose in order to get their model to work. As an example, 
Although Gluon made data encapsulation possible, still there are examples using 
mxn.io.ImageRecordIter with tens of params (feels like gluon examples are 
simply the copy from old Python examples).

3. Examples again! Comparing to PyTorch, there are a few examples I don't like 
in Gluon:
1. Available to run however the code structure is still very 
complicated. Such as example/image-classification/cifar10.py. It seemed like a 
consecutive code concatenation. In fact, these are just a series of layers 
mixed with model.fit. It makes user very hard to modify/extend the model.
2. Only available to run with certain settings. If users try to 
change a little bit in the model, crashes will happen. For example, the 
multi-gpu example in Gluon website, MXNet hide the logic that using batch size 
to change learning rate in a optimizer. A lot of newbies didn't know this fact 
and they would only find that the model stopped converging when batch size 
changed.
3. The worst scenario is the model itself just simply didn't 
work. Maintainers in the MXNet community didn't run the model (even no 
integration test) and merge the code directly. It makes the script not able run 
till somebody raise the issues and fix it.

4. The Community problem. The core advantage for MXNet is it's scalability and 
efficiency. However, the documentation of some tools are confusing. Here are 
two examples:

1. im2rec contains 2 versions, C++ (binary) and python. But 
nobody would thought that the argparse in these tools are different (in the 
meantime, there is no appropriate examples to compare with, users could only 
use them by guessing the usage).

2. How to combine MXNet distributed platform with 
supercomputing tool such as Slurm? How do we do profiling and how to debug. A 
couples of companies I knew thought of using MXNet for distributed training. 
Due to lack of examples and poor support from the community, they have to 
change their models into TensorFlow and Horovod.

5. The heavy code base. Most of the MXNet examples/source 
code/documentation/language binding are in a single repo. A git clone operation 
will cost tens of Mb. The New feature PR would takes longer time than expected. 
The poor reviewing response / rules keeps new contributors away from the 
community. I remember there was a call for document-improvement last year. The 
total timeline cost a user 3 months of time to merge into master. It almost 
equals to a release interval of Pytorch.

6. To Developers. There are very few people in the community discussed the 
improvement we can take to make MXNet more user-friendly. It's been so easy to 
trigger tens of stack issues during coding. Again, is that a requirement for 
MXNet users to be familiar with C++? The connection between Python and C lacks 
a IDE lint (maybe MXNet assume every developers as a VIM master). 
API/underlying implementation chaged frequently. People have to release their 
code with an achieved version of MXNet (such as TuSimple and MSRA). Let's take 
a look at PyTorch, an API used move tensor to device would raise a thorough 
discussion.

There will be more comments translated to English and I will keep this thread 
updated…
Thanks,
Qing

Re: Off-Heap Memory Management in MXNet Scala

[Qing Lan]

Nice document! Way better than current .dispose() in Scala!

Thanks,
Qing

On 9/11/18, 6:04 PM, "Chris Olivier"  wrote:

wow, incredible document!

On Tue, Sep 11, 2018 at 2:37 PM Naveen Swamy  wrote:

> Hi All,
>
> I am working on managing Off-Heap Memory Management and have written a
> proposal here based on my prototype and research I did.
>
> Please review the doc and provide your feedback ?
>
> https://cwiki.apache.org/confluence/display/MXNET/JVM+Memory+Management
>
> I had offline discussion with a few people I work with and added their
> feedback to the doc as well.
>
> Thanks, Naveen
>

New Java Inference API

[Qing Lan]

Hi All,

Here is an update for the Java Inference API design doc on CWIKI: 
https://cwiki.apache.org/confluence/display/MXNET/MXNet+Java+Inference+API. 
Currently, MXNet Java bindings is an extension of MXNet Scala API that allow 
users to use Java to do inference on MXNet. Users will be able to import 
pre-trained MXNet model and do single/batch inference on it.

Please take a look the design document again and feel free to leave any 
thoughts you have.

Thanks,
Qing

On 5/10/18, 11:08 AM, "Andrew Ayres"  wrote:

Hi Kellen,

Thanks for the feedback. You bring up an interesting idea about the
dependencies. I'll add that to the list of things to look into.

As for the threading, my current thinking is that we implement a dispatcher
thread like suggested in the Scala threading discussion
https://discuss.mxnet.io/t/fixing-thread-safety-issues-in-scala-library/236.
I would definitely like to hide such complexities from the user.

Andrew


On Thu, May 10, 2018 at 3:22 AM, kellen sunderland <
kellen.sunderl...@gmail.com> wrote:

> Hey Andrew, thanks for the write-up.  I think having a Java binding will 
be
> very useful for enterprise users.  Doc looks good but two things I'm
> curious about:
>
> How are you planning to handle thread safe inference?   It'll be great if
> you can hide the complexity of dealing with dispatch threading from users.
>
> The other thing I think a solid Java API could provide is a limited number
> of dependencies.  There's some simple things we can do to make this happen
> (create a statically linked, portable so) but there's also some complexity
> around minimizing dependencies MXNet.  For example we'll likely want to
> release MKL flavoured binaries, we should have a few versions of CUDA
> supported.  We could try and have one version that has an absolute minimum
> of dependencies (maybe statically linking with openblas).  It might be 
good
> to document exactly the packages you're planning to release, and give some
> more details about what the dependencies for the packages would be.
>
> Many thanks for looking into this, I think it'll be a big improvement for
> many of our users.
>
> -Kellen
>
> On Thu, May 10, 2018, 12:57 AM Andrew Ayres 
> wrote:
>
> > Hi all,
> >
> > There has been a lot of interest expressed in having a Java API for 
doing
> > inference. The general idea is that after training a model using python,
> > users would like to be able to load the model for inference inside their
> > existing production eco-system.
> >
> > We've begun exploring a few options for the implementation at <
> > https://cwiki.apache.org/confluence/display/MXNET/
> MXNet+Java+Inference+API
> > >
> > and would appreciate any insights/feedback.
> >
> > Thanks,
> > Andrew
> >
>

Re: [DISCUSS] improve MXNet Scala release process

[Qing Lan]

Hi all,

I have created a design document on Automated Scala publish in here:
https://cwiki.apache.org/confluence/display/MXNET/Automated+MXNet+Scala+release+design

Please kindly review it and leave any thoughts you may have.

Thanks,
Qing

On 8/3/18, 10:26 AM, "Naveen Swamy"  wrote:

Hi Carin,

The thinking right now is to publish nightly to the Apache Snapshot
repository, building and validating with integration tests. We(Qing, me and
Andrew Ayres) will work on a elaborate document detailing the process and
we'll loop you in as well.

Thanks, Naveen

On Fri, Aug 3, 2018 at 8:35 AM, Carin Meier  wrote:

> Hi,
>
> I was thinking about the process for publishing the Clojure jar as well. I
> think, since it will be published to Nexus/Maven and the building of it
> depends on the Scala jar artifact, it might make sense to combine the
> publishing of the Clojure jar at the same time as the Scala Jar. I haven't
> worked out all the details yet, but I'm thinking with that it would be a
> couple command lines in the same script that handles the Scala deployment.
>
> Or if it is a separate process, it might be able to share common parts.
>
> Could you please keep me in the loop of the deployment process so we can
> figure out the best place to work in Clojure as well?
>
> Thanks,
    > Carin
>
> On Tue, Jul 31, 2018 at 3:49 PM, Qing Lan  wrote:
>
> > Upon offline discussion with Marco,
> >
> > He proposed a plan that can actually help us conduct 3):
> > 1. This job will not be trigger when PR runs and strictly limit
> > that only committer can run the restricted job.
> > 2. The code being run in there will only covers the code from 
the
> > branch you choose to go, it will be committers responsibilities not to
> > merge any trivial credential grabber code.
> > 3. Test this is simple. The restricted job uses a similar
> > architecture with current CI. You can send a PR with dockerfiles, 
scripts
> > and configurations on Jenkins to give it a test to run the job with a
> mock
> > credential. Finally please contact people working on CI to give it a 
test
> > run and they will do the last step to merge your change to CI.
> > 4. Marco also mentioned the security level of the credentials.
> The
> > credential being used in the AWS Credential services will be assigned
> with
> > an individual IAM role, which only allows to access to the credentials
> that
> > role being assigned to, and used in the restricted job you have set up.
> >
> > I would also like to encourage people in this list  to join the
> > https://cwiki.apache.org/confluence/display/MXNET/
    > > MXNet+Berlin+Office+Hours as the people who is working on improving the
> > CI are there ready to help.
> >
> > Thanks,
> > Qing
> >
> >
> > On 7/28/18, 11:44 PM, "Qing Lan"  wrote:
> >
> > Thanks Marco, Naveen and Sheng's feedback.
> >
> > About the 1): Scala side will only pack the mxnet binary only and 
use
> > dynamic links to all the rest dependencies. So indeed it will require
> users
> > to install all deps as the same as the builder platforms version and 
this
> > will make them hard to use. Let's please collaborate and create a (set
> of)
> > general CI script(s) to install the deps and bring static links to the
> > package.
> >
> > About 3): it is indeed a general problems for both Scala and Python
> > publish. If there is a good way we can safely store the credentials, we
> can
> > definitely give automated publish a go. And thanks again for Marco's
> option
> > provided below, I think we can make use of the restricted slaves and 
give
> > it a test run. And to Marco:
> > 1. Will this restricted jobs being triggered in every PR runs or
> > it just depends on where you put it (like I put in nightly it will never
> >be trigger in PR)? Will there be a potential risk like a PR attack
> > (create a PR to grab credentials)
> >
> > 2. How do we make sure the coding being run there is under
> control
> > and not be changed by anyone?
> >
> > 3. If I want to test this functionality, where is the best place
> > to create the job and make a test run?
> >
> > Thanks,
> > Qing
> &

Re: Significant windows build improvements from source

[Qing Lan]

Just an update from previous thread. 

I tried Windows build using 
http://mxnet.incubator.apache.org/install/windows_setup.html instruction. The 
OpenCV links there only contains prebuilt vs12 (VS 2013). Apart from that, user 
need to create a new C++ project after VS2015 installation because it did not 
trigger the c compiler installation. I copied the command from Pedro's Python 
script and it improves a little. However, even with these changes, I am still 
facing build failure. I think developers may need some script to automate these 
process to build backend dependencies on Windows. Do we have them already?

Thanks,
Qing

On 8/2/18, 10:10 AM, "Qing Lan"  wrote:

Hi Pedro,

Great works! Thanks for supporting Windows build in here.

Thanks,
Qing

On 8/2/18, 10:08 AM, "Marco de Abreu" 
 wrote:

Hello Pedro,

These are great efforts! It's great to see that we are improving our
situation around Windows.

Looking forward to using it.

Best regards,
Marco

Pedro Larroy  schrieb am Do., 2. Aug. 
2018,
19:04:

> Hi
>
> I have taken some effort to cleanup the Windows build process and 
condense
> it in a single python script, provided that the required dependencies 
are
> installed (opencv, openblas and cuda/cudnn in gpu).
>
> Now there's a single command necessary to get a build:
>
> python ci\build_windows.py
>
> The build flavour can be given with the '-f' argument.
>
> This produces a windows_package.7z file with MXNet libraries, python
> bindings and includes.
>
> For testing, just running a powershell script like:
>
>ci/windows/test_py3_cpu.ps1
>
> Will execute the unit tests.
>
> Let me know what you think.  I pretty much would like to see these 
changes
> in the release branch which will help with any windows issues that we 
might
> find.
>
> https://github.com/apache/incubator-mxnet/pull/11947
>
> Simple installation of dependencies and update to the documentation 
about
> windows builds from source will come in the future.
>
> Pedro.
>

Re: Significant windows build improvements from source

[Qing Lan]

Hi Pedro,

Great works! Thanks for supporting Windows build in here.

Thanks,
Qing

On 8/2/18, 10:08 AM, "Marco de Abreu"  
wrote:

Hello Pedro,

These are great efforts! It's great to see that we are improving our
situation around Windows.

Looking forward to using it.

Best regards,
Marco

Pedro Larroy  schrieb am Do., 2. Aug. 2018,
19:04:

> Hi
>
> I have taken some effort to cleanup the Windows build process and condense
> it in a single python script, provided that the required dependencies are
> installed (opencv, openblas and cuda/cudnn in gpu).
>
> Now there's a single command necessary to get a build:
>
> python ci\build_windows.py
>
> The build flavour can be given with the '-f' argument.
>
> This produces a windows_package.7z file with MXNet libraries, python
> bindings and includes.
>
> For testing, just running a powershell script like:
>
>ci/windows/test_py3_cpu.ps1
>
> Will execute the unit tests.
>
> Let me know what you think.  I pretty much would like to see these changes
> in the release branch which will help with any windows issues that we 
might
> find.
>
> https://github.com/apache/incubator-mxnet/pull/11947
>
> Simple installation of dependencies and update to the documentation about
> windows builds from source will come in the future.
>
> Pedro.
>

Re: [DISCUSS] improve MXNet Scala release process

[Qing Lan]

Upon offline discussion with Marco,

He proposed a plan that can actually help us conduct 3):
1. This job will not be trigger when PR runs and strictly limit that 
only committer can run the restricted job.
2. The code being run in there will only covers the code from the 
branch you choose to go, it will be committers responsibilities not to merge 
any trivial credential grabber code.
3. Test this is simple. The restricted job uses a similar architecture 
with current CI. You can send a PR with dockerfiles, scripts and configurations 
on Jenkins to give it a test to run the job with a mock credential. Finally 
please contact people working on CI to give it a test run and they will do the 
last step to merge your change to CI. 
4. Marco also mentioned the security level of the credentials. The 
credential being used in the AWS Credential services will be assigned with an 
individual IAM role, which only allows to access to the credentials that role 
being assigned to, and used in the restricted job you have set up.

I would also like to encourage people in this list  to join the 
https://cwiki.apache.org/confluence/display/MXNET/MXNet+Berlin+Office+Hours as 
the people who is working on improving the CI are there ready to help.

Thanks,
Qing


On 7/28/18, 11:44 PM, "Qing Lan"  wrote:

Thanks Marco, Naveen and Sheng's feedback.

About the 1): Scala side will only pack the mxnet binary only and use 
dynamic links to all the rest dependencies. So indeed it will require users to 
install all deps as the same as the builder platforms version and this will 
make them hard to use. Let's please collaborate and create a (set of) general 
CI script(s) to install the deps and bring static links to the package.

About 3): it is indeed a general problems for both Scala and Python 
publish. If there is a good way we can safely store the credentials, we can 
definitely give automated publish a go. And thanks again for Marco's option 
provided below, I think we can make use of the restricted slaves and give it a 
test run. And to Marco: 
1. Will this restricted jobs being triggered in every PR runs or it 
just depends on where you put it (like I put in nightly it will never   be 
trigger in PR)? Will there be a potential risk like a PR attack (create a PR to 
grab credentials)

2. How do we make sure the coding being run there is under control and 
not be changed by anyone?

3. If I want to test this functionality, where is the best place to 
create the job and make a test run?

Thanks,
Qing



On 7/27/18, 5:44 PM, "Marco de Abreu" 
 wrote:

Hi all,

about the credential management: We already have a solution based on
restricted slaves [1] and AWS secrets manager [2] that is generally
classified to generate binaries and handle credentials. It was designed
with continuous deployment in mind, but we haven't tested it in that 
field
yet.

To properly assess the requirements, it would be great if we have this
security critical part outlined for each release pipeline. We could then
check and see if our existing solution matches all requirements or if
further work is necessary.

Best regards,
Marco

[1]

https://cwiki.apache.org/confluence/display/MXNET/Restricted+jobs+and+nodes
[2] 
https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html

On 7/27/18, 5:43 PM, "Sheng Zha"  wrote:

Thanks, Naveen. Once we have clarity on 3), it should be no problem for
scala to reuse the same solution. For 1), if this is indeed an issue, it
seems that we may have rushed a bit on the scala releases. Are there any
user reports?

-sz

On Fri, Jul 27, 2018 at 5:26 PM, Naveen Swamy  
wrote:

> I collaborate with Qing as a part of my day time job, to give you a 
little
> more perspective on the proposed work
>
> For 1)
> What we found is that users often run into conflicts when they use a
> different version of the dependency(CUDA, CUDNN, OpenBLAS, OpenCV, 
etc,.)
> and the one we build with MXNet backend and use in the MXNet Scala 
package.
> Also it makes its not very straight-forward for users to install these
> dependencies themselves in order to lower the entry barrier and to 
make
> everything work out of the box we are thinking to build MXNet all 
these
> dependencies with MXNet (as a static library) and embed them in the 
MXNet
> Scala package. This is also inspired by the work you have done for 
Apache
> MXNet pip packages, Ideally I would like to reuse some of that work.
>
&

Re: [DISCUSS] improve MXNet Scala release process

[Qing Lan]

st to link to them,
> and
> > let maven manage these dependencies.
> >
> > For 2, since the current CI is based on Jenkins, I imagine it would be
> some
> > sort of Jenkins pipeline? Other people who're better versed in Jenkins
> can
> > chime in.
> >
> > Personally, I'm interested in 3 as well. I have a pipeline for building
> pip
> > packages that's currently not utilizing the CI, and the item 3 is the
> > blocker too. Once you finish, it would be great to refer to the same
> > solution, so that I can move it into the same CI.
> >
> > -sz
> >
> > On Fri, Jul 27, 2018 at 4:37 PM, Qing Lan  wrote:
> >
> > > Hi all,
> > >
> > > Recently contributors on Scala Language development worked together 
and
> > > finally able to publish Scala package on Maven. Now I would like to
> > raise a
> > > discussion to automate Scala release process and also discover a
> standard
> > > way to release different packages for MXNet so we won’t ask any
> > individuals
> > > to spend a long time to publish the package.
> > >
> > > There are three blocks that stop this automated process:
> > >
> > >   1.  How to build general hardware-compatible backend dependencies 
for
> > > MXNet (Linux CPU/GPU Mac OSX)
> > >   2.  How to automate the frontend release process and CI integration
> > >   3.  How to keep credentials for the release in the pipeline
> > >
> > > Scala Release process created by Naveen: https://cwiki.apache.org/
> > > confluence/display/MXNET/MXNet-Scala+Release+Process
> > >
> > > Thanks,
> > > Qing
> > >
> > >
> >
>

[DISCUSS] improve MXNet Scala release process

[Qing Lan]

Hi all,

Recently contributors on Scala Language development worked together and finally 
able to publish Scala package on Maven. Now I would like to raise a discussion 
to automate Scala release process and also discover a standard way to release 
different packages for MXNet so we won’t ask any individuals to spend a long 
time to publish the package.

There are three blocks that stop this automated process:

  1.  How to build general hardware-compatible backend dependencies for MXNet 
(Linux CPU/GPU Mac OSX)
  2.  How to automate the frontend release process and CI integration
  3.  How to keep credentials for the release in the pipeline

Scala Release process created by Naveen: 
https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process

Thanks,
Qing

Data Type improvement in MXNet Scala

[Qing Lan]

Hi all,

I would like to propose a change on the MXNet Scala data_type:

Depreciate the `provideData` and `provideLabel` functions to provide `DataDesc` 
type instead of `ListMap[String, Shape]` type.

There were several issues regarding to this since v0.12 on Scala. Several users 
reported that the `implicit def ListMap2Descs` helps to do some downcasting job 
however missing information during this process (layout property).
MXNet Python language has already change this part into `DataDesc`. In that 
case, I plan to add two more methods that will take `DataDesc` and depreciate 
the functions that mentioned above. The downcasting method `ListMap2Descs` will 
also be depreciated and not to be used in current MXNet Scala code base.

Thanks,
Qing

References:
Code position:
https://github.com/apache/incubator-mxnet/blob/master/scala-package/core/src/main/scala/org/apache/mxnet/IO.scala#L312-L315
https://github.com/apache/incubator-mxnet/blob/master/scala-package/core/src/main/scala/org/apache/mxnet/IO.scala#L358-L363

Issue links:
https://github.com/apache/incubator-mxnet/issues/11775
https://github.com/apache/incubator-mxnet/issues/10753
https://github.com/apache/incubator-mxnet/issues/11571

Re: [VOTE] Subscribe dev@ to Github Activities

[Qing Lan]

-1, 
unless we can keep this under control. It's not all of the PRs or Issues 
worthwhile to be involved into discussion. 
I hope we can put this under control such as @subscribe_dev as a bot to spread 
the information to dev@.

Thanks,
Qing

On 7/17/18, 9:26 AM, "Lin Yuan"  wrote:

+1, I think they are very relevant to dev and as Aaron said we can always
set up personalized filter.

On Tue, Jul 17, 2018 at 9:21 AM Aaron Markham 
wrote:

> +1, I don't read your emails anyways. Just kidding. I think it would be
> good to see the action, even if I eventually have to setup filters if it
> gets overwhelming.
>
> On Tue, Jul 17, 2018 at 9:15 AM, Tianqi Chen 
> wrote:
>
> > +1, most of issue and PR activities are about development, and they
> belong
> > to dev. It also helps us to recognizes contributors who are actively
> > contributing but less vocal via emails -- there are many of them.
> >
> > Tianqi
> >
> > On Tue, Jul 17, 2018 at 8:47 AM, Anirudh  wrote:
> >
> > > -1
> > >
> > > The low signal to noise ratio would mean that we may miss important
> > emails.
> > > Even with the different filters that we may setup for dev@, the emails
> > > would be too many to not miss the important ones. We would see more 
and
> > > more people starting a design discussion on an issue or PR. Because of
> > the
> > > low signal to noise ratio on the dev@ list, many may miss these
> > > discussions.
> > >
> > > Slowly, this would erode the purpose of the dev@ list as this means
> that
> > > you don't really have to do anything explicitly on the dev@ list.
> > > You can start a design discussion on a github issue. You can start a
> > > vote/discussion on a github issue.
> > >
> > > Anirudh
> > >
> > > On Mon, Jul 16, 2018 at 4:35 AM, Timur Shenkao 
> > wrote:
> > >
> > > > +1 if my vote can be taken into account
> > > >
> > > > On Mon, Jul 16, 2018 at 4:32 AM, Sheng Zha 
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'm starting a vote on subscribing dev@ to Github activities. See
> > > > previous
> > > > > discussion thread here
> > > > > <
> https://lists.apache.org/thread.html/3d883f6a3cbc8e81e810962e0c0fe7
> > > > > bfd01f0b78d3cb44034f566442@%3Cdev.mxnet.apache.org%3E>
> > > > > .
> > > > >
> > > > > The vote lasts for three days and ends on 7/18/2018 at 9pm pst.
> > > > >
> > > > > -sz
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Subscribe dev@ to Github Activities?

[Qing Lan]

+0.5
Is there a @dev-apache option we can have? 
So we can decide whether to share the discussion on this mailing list.

Thanks,
Qing

On 7/12/18, 3:08 PM, "Pedro Larroy"  wrote:

-1   It's a lot of traffic, whomever wants to subscribe can do it in
github. I'm afraid it will decrease signal to noise ratio in the list.

On Thu, Jul 12, 2018 at 11:32 PM Lin Yuan  wrote:

> +1
>
> On Thu, Jul 12, 2018 at 12:26 PM Anirudh Acharya 
> wrote:
>
> > +1
> >
> > On Thu, Jul 12, 2018 at 11:51 AM Piyush Ghai 
> > wrote:
> >
> > > +1
> > > > On Jul 12, 2018, at 11:50 AM, Tianqi Chen 
> > > wrote:
> > > >
> > > > +1
> > > >
> > > > On Thu, Jul 12, 2018 at 11:10 AM, Sheng Zha 
> > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> Should we subscribe dev list to github updates on mxnet repo? Both
> > > github
> > > >> issues/PRs and the dev list are intended for technical discussions
> and
> > > in
> > > >> that aspect largely share the same goal. Since MXNet has most
> activity
> > > >> github, this could help dev@ to become more active. Some pros and
> > cons:
> > > >>
> > > >> Pros:
> > > >> - There have been many high quality discussions that happen on
> github
> > to
> > > >> which the dev list can benefit.
> > > >> - Replies on update emails are reflected on the specific issue/PR.
> > > >> - Users can also choose to click on the link and go to github to
> > > >> participate in discussion.
> > > >> - We still have the ability to carry out dev@ only conversation.
> > > >>
> > > >> Cons:
> > > >> - Higher volume on dev list.
> > > >> - Some discussions might not be suitable for dev@. (though I can't
> > > think
> > > >> of
> > > >> why such conversation should happen on github either)
> > > >>
> > > >> -sz
> > > >>
> > >
> > >
> >
>

Re: Access Permission of MXNet label bot

[Qing Lan]

I think putting in the Infra can be a really good solution. 
We do not expose the credential to the outside and we can make sure it can be 
run in a timely manner.

Thanks,
Qing

On 7/12/18, 11:11 AM, "Marco de Abreu"  
wrote:

Hello Cathy,

unfortunately, we're not allowed to use bot accounts at Apache.

An option we have is that we run your bot in our infrastructure with the
credentials of a committer with the permission you have mentioned. The only
restriction would be that you would not be able to access that server
because the credentials are confidential user data of a committer. Would
this work for you?

Best regards,
Marco

On Thu, Jul 12, 2018 at 8:57 PM Yuelin Zhang 
wrote:

> Hi,
>
> I am working to improve the GitHub issue triage process by creating a 
label
> bot(more info here
> <
> 
https://cwiki.apache.org/confluence/display/MXNET/Deep+Learning+Based+GitHub+Label+Bot
> >
> on
> the cwiki), I have initial version of label bot ready. I would like to get
> some opinions about access permission of MXNet label bot.
>
> Right now, all issues in MXNet repo are manually labeled. The process 
looks
> like below:
> First, contributors/committers go through the issues to triage them and
> suggest labels and add comment on the issue requesting @committer to add
> labels.
>
> This process will cause notification spam to both committers and users. 
The
> long gap between user creating an issue and we labelling them will cause
> the process time consuming and not very smooth.
>
> We want to simplify/automate this issue labeling process. Right now an
> initial version of the label bot which can:
>
>1.  Send issue report daily. This report will show how many issue
>open/closed, list uncommented/unlabeled issues and show an pie chart of
>labels added in a week. Sample report here
><
> 
https://cwiki.apache.org/confluence/display/MXNET/Deep+Learning+Based+GitHub+Label+Bot#DeepLearningBasedGitHubLabelBot-SampleIssueReport
> >
>.
>2.  Generate a spread sheet of unlabeled issues with recommended 
labels.
>A contributor will open the sheet and fill in labels with reference of
>bot's recommendations. In this case, contributor can deal with all
>unlabeled issues at a time. Sample sheet here
><
> 
https://cwiki.apache.org/confluence/display/MXNET/Deep+Learning+Based+GitHub+Label+Bot#DeepLearningBasedGitHubLabelBot-SampleSpreadSheet
> >
>.
>3.  Read labels filled in that sheet and apply labels to GitHub issues.
>(tested on my personal Github repo)
>
>
> This bot can be triggered daily so that all issues will be labeled in one
> day without notification spam.
>
> *However,  this bot doesn't have access to add labels. We have two
> options:*
>
> - Use a committer's Oauth token with limited scope. So far according to my
> research, the most limited scope is "public_repo", this contains access to
> code. Except this one, Github doesn't have smaller scope available to add
> labels. Available scopes here
> <
> 
https://developer.github.com/apps/building-oauth-apps/understanding-scopes-for-oauth-apps/
> >
> .
>
> - Create a bot account having minimum permissions. For this, we will need
> an account to be created from Apache Infrastructure with proper access and
> they can control the access for the account through secret manager
>  .
> Having a bot account is beneficial for future work, not only for labelling
> but also other automatic processes.
>
> Please let me know if you have any other ideas to do this.
>
> Thanks,
> Cathy
>

Re: Proposal to release the Examples for Scala in V1.3

[Qing Lan]

Hi Marco,

Currently all Scala examples I have changed contains test coverage on CI (in 
the example/src/test/). It's been temporarily disabled because of the memory 
leaks.

I would also like to raise a discussion on a general standard for testing 
examples, this is what Scala side doing:

1. For small training examples we can finish them and check the dev accuracy
2. For big training examples, we do 5-10 Epoch to check it is runnable
3. For trivial examples (like neural style), we check it is runnable
4. For inference examples and classification specifically, we run with several 
test cases to measure if it can get the result correctly.

Thanks,
Qing

On 7/12/18, 11:00 AM, "Marco de Abreu"  
wrote:

Hi Qing,

thanks a lot for your efforts around the Scala examples and to assist us
getting the Scala API to a broader user base! This is a great user-facing
approach!

The following might not be in the current course of how we handle examples
in Python, but let me elaborate: I have seen a lot of problems in the last
months related to examples in Python in the last months that could have
been avoided by testing them in integration tests. These errors included
syntax errors, typos, wrong logic, outdated APIs or other more
context-related errors. Thus, I'd like to strive for full test coverage of
our examples in the long term. I know that we can't get this done within a
single night, but since you are now reworking the way our examples work in
Scala, I'd like to kindly request to assess whether it's possible to
include testing of Scala examples in your plan right from the start. This
will ensure that we have a good first impression with our users, increases
our overall test coverage and also reduces the maintenance overhead we are
having with examples breaking without us knowing.

I think the memory leaks are adjacent problems and that examples are a good
way to make them visible. Since they are already there, we shouldn't block
ourselves from writing and publishing examples. Our users would encounter
them anyways, but by making the examples, we can properly document and
track them ahead of time. The big advantage right now would be that we
could mark these things as known problems in our patch notes and thus our
users would know that we are aware of them and that we are going to fix
them by (hopefully) the following release.

Best regards,
Marco

On Thu, Jul 12, 2018 at 8:41 PM YiZhi Liu  wrote:

> Agree to make incremental improvements. As long as the changes do not
> change the current states of memory leak, but improve the documents
> and demonstrate the way to use type-safe apis, I think it is fair to
> merge into v1.3.
> On Wed, Jul 11, 2018 at 4:43 PM Naveen Swamy  wrote:
> >
> > Qing, its great that you are working on improving the Scala Examples 
with
> > documentation and tests.
> > Ideally, we shouldn't have any memory leaks..it erodes trust with our
> > users, however I understand it can take significant time and debugging
> > effort to resolve them. Given that these leaks already exist, may be
> should
> > call out on the documentation that this is a known issue and we will 
work
> > on it.
> > IMO for this case making incremental improvements to examples makes
> sense.
> >
> > I would like to hear what others think.
> > -Naveen
> >
> > On Wed, Jul 11, 2018 at 3:47 PM, Qing Lan  wrote:
> >
> > > Hi all,
> > >
> > > I would like to propose the release plan for MXNet Scala examples in
> V1.3.
> > > Currently, the examples in the main repository are not maintained for 
a
> > > long time. I have spent some time to use the improved Scala API (known
> as
> > > the .api) to improve the examples with documentation to run and add CI
> > > tests. However, I am facing the difficulties to resolve memory leaks 
as
> > > most of the examples did not contains memory leak fixes. It crashes
> the CI
> > > because of that. We can temporarily disable the CI test for some of 
the
> > > examples and release them with 1.3 and enable them once we have a
> better GC
> > > in MXNet Scala. Here is a brief Pros and Cons for this release:
> > >
> > > Pros:
> > >
> > >   *   A good set of demos on how to use improved Training API
> > >   *   Examples are fixed with full documentation to help to run them
> > >
> > > Cons:
> > >
> > >   *   Examples contains the memory leak issues may cause problems
> > >
> > > Please kindly share your ideas in this thread and I am really
> appreciate
> > > for your help.
> > >
> > > Thanks,
> > > Qing
> > >
>
>
>
> --
> Yizhi Liu
> DMLC member
> Amazon Web Services
> Vancouver, Canada
>

Proposal to release the Examples for Scala in V1.3

[Qing Lan]

Hi all,

I would like to propose the release plan for MXNet Scala examples in V1.3. 
Currently, the examples in the main repository are not maintained for a long 
time. I have spent some time to use the improved Scala API (known as the .api) 
to improve the examples with documentation to run and add CI tests. However, I 
am facing the difficulties to resolve memory leaks as most of the examples did 
not contains memory leak fixes. It crashes the CI because of that. We can 
temporarily disable the CI test for some of the examples and release them with 
1.3 and enable them once we have a better GC in MXNet Scala. Here is a brief 
Pros and Cons for this release:

Pros:

  *   A good set of demos on how to use improved Training API
  *   Examples are fixed with full documentation to help to run them

Cons:

  *   Examples contains the memory leak issues may cause problems

Please kindly share your ideas in this thread and I am really appreciate for 
your help.

Thanks,
Qing

Re: The operator check for Scala Package

[Qing Lan]

Thanks Jun for your clarification. I think one of the advantages for MXNet is 
it's language binding. Although the number of customers using Scala is great 
less than Python, it is still one important language we need to support. 

About the first question, I would say we all in. At least all breaking changes 
should let all language binding maintainers notified. Then we make the changes 
to make sure we have stable support to the customers. The reason for Scala 
build failure could be many, so we need to diagnose the problem and see if we 
need to collaborate and solve the problems together. I don't think the other 
language maintainers and backend must take the responsibilities to diagnose a 
problems of a language they may not familiar with.

2. It depends differently case by case, we cannot bring a unique time for 
different failures we have. If you mean the operator check, it could cause you 
(time to build scalapkg) + (30 seconds to change the code) + (time to run the 
CI to make sure it worked) + (send a PR to make that change on master branch). 
This update should be done by the Scala developers for sure.

3. If there is a nightly build failure, maybe we can think of checking that on 
a weekly basis. Spend 15 mins every week to review the operator changes so we 
don't miss any important points.

Thanks,
Qing



On 6/20/18, 9:57 PM, "Jun Wu"  wrote:

We should reach an agreement on the responsibility/ownership before moving
forward.

1. Who will take the ownership of fixing Scala build failure if an operator
is changed/added in C++?
2. What is the expected turn around time of fixing the scala build failure.
3. What if we are short of resources of fixing the scala build failure? How
do we deal with the failing nightly CI every night?

With all being said, there is at least one thing that must be clear, we
certainly don't want to put the extra burden on the backend developers.

On Wed, Jun 20, 2018 at 9:39 PM Qing Lan  wrote:

> Hi All,
>
> Thank you all for your feedback. This changes do influence a lot on the
> PRs related to operator changes. I will take what Marco proposed to place
> that in the nightly build rather than every CI we runs.
>
> Thanks,
> Qing
>
> On 6/20/18, 8:40 PM, "Hao Jin"  wrote:
>
> I don't think we should keep this hash check thing. As Qing stated
> before
> on this thread, if there's any change in documentation the hash value
> is
> also changed and then flagged as a problem, how will that break any
> user's
> code? I would go for something like Marco's proposal: moving this to 
an
> asynchronous check.
> At this very moment, I've got 4 PRs that are all related to "operator
> changes", adopting this kind of method is adding extra work for me and
> every other contributor whose work involves changes on operator code,
> and I
> don't think it's a reasonable kind of extra work like tracking PRs on
> JIRA.
> Hao
>
> On Wed, Jun 20, 2018 at 8:10 PM, Naveen Swamy 
> wrote:
>
> > Agreed, for new APIs we can turn into a nightly job and report on
> dev@.
> > The
> > goal here is not to burden anyone but to catch changes on the 
backend
> > before it propagates and break user's code and co-ordinate changes
> across
> > language bindings which IMO is essential, so I would like to keep 
for
> > changes on the existing API.
> >
> > On Wed, Jun 20, 2018 at 7:58 PM, Marco de Abreu <
> > marco.g.ab...@googlemail.com.invalid> wrote:
> >
> > > I think we should go with an asychronous approach using a nightly
> job
> > that
> > > detects the changes and reports them accordingly. We could then
> send out
> > a
> > > report if there is a mismatch.
> > >
> > > I agree with the already stated points that we should not put the
> burden
> > of
> > > adding frontend API bindings to somebody who adds a Backend
> function.
> > This
> > > is not scalable and rather poses a risk in reduced quality or
> increased
> > > review overhead.
> > >
> > > The sparse support is the perfect example. I don't know if haibin
> knows
> > > Scala, but he would have had to make the entire mapping for Scala
> without
> > > being very familiar with it (sorry if I'm wrong on this haibin,
> it's just
> > >

Re: The operator check for Scala Package

[Qing Lan]

oposed is little bit
> too
> > > > heavy to developers, and not quite friendly to new contributors.
> > > > On Wed, Jun 20, 2018 at 10:22 AM Haibin Lin <
> haibin.lin@gmail.com>
> > > > wrote:
> > > > >
> > > > > I appreciate the effort and understand the motivation. However, 
I'm
> > > > > concerned that it basically means merging operator PRs becomes
> > > > sequential.
> > > > > Developers who work on operators have to update their PR every
> time a
> > > new
> > > > > operator is merged to master, the burden becomes significant if
> > > there're
> > > > 20
> > > > > ONNX/sparse operators to add and many PRs are submitted/reviewed 
in
> > > > > parallel.
> > > > >
> > > > > On Wed, Jun 20, 2018 at 10:13 AM, Qing Lan 
> > > wrote:
> > > > >
> > > > > > Hi Haibin,
> > > > > >
> > > > > > The operator change is any changes on the operator on C++ side.
> > > Trigger
> > > > > > the check failed?
> > > > > >- change the documentation of operator in C
> > > >   Yes
> > > > > >- change the documentation such as README.md
>  No
> > > > > >- add/remove/modify operator
>  Yes
> > > > > >- add/remove/modify operator parameter
> > > >  Yes
> > > > > >
> > > > > > Thanks,
> > > > > > Qing
> > > > > >
> > > > > > On 6/20/18, 10:01 AM, "Haibin Lin" 
> > > wrote:
> > > > > >
> > > > > > Could you elaborate what you mean by operator change? Does 
it
> > > > check the
> > > > > > operator interface? Would updated operator documentation 
fail
> > the
> > > > > > check?
> > > > > > Would adding a new operator fail this check?
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Jun 20, 2018 at 9:48 AM, Qing Lan <
> lanking...@live.com
> > >
> > > > wrote:
> > > > > >
> > > > > > > Hi Macro,
> > > > > > >
> > > > > > > Thanks for your feedback! I believe this should not be a
> > block
> > > > for
> > > > > > > contributor in most of the cases.
> > > > > > > Firstly, this would only be triggered if there is an
> operator
> > > > changes
> > > > > > > (Only general operators).
> > > > > > > Secondly, it is simple to go through. Just need to change 
1
> > > line
> > > > of
> > > > > > code
> > > > > > > would make the PR passed. However it do requires
> contributor
> > to
> > > > do
> > > > > > this or
> > > > > > > the Scalatest will fail. I have made the error message
> > > > instructive
> > > > > > that
> > > > > > > would help the contributor to dive in and make the 
changes.
> > > > > > >
> > > > > > > I also updated the design document to explain in detail.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Qing
> > > > > > >
> > > > > > >
> > > > > > > On 6/19/18, 12:09 PM, "Marco de Abreu" <
> > > > marco.g.ab...@googlemail.com
> > > > > > .INVALID>
> > > > > > > wrote:
> > > > > > >
> > > > > > > Okay, thanks for elaborating. I definitely see your
> point
> > > > there
> > > > > > and we
> > > > > > > definitely don't want these changes to pile up.
> > > > > > >
> > > > > > > I don't feel strongly about this and won't stand in 
the
   

Re: The operator check for Scala Package

[Qing Lan]

Hi Macro,

Thanks for your feedback! I believe this should not be a block for contributor 
in most of the cases. 
Firstly, this would only be triggered if there is an operator changes (Only 
general operators). 
Secondly, it is simple to go through. Just need to change 1 line of code would 
make the PR passed. However it do requires contributor to do this or the 
Scalatest will fail. I have made the error message instructive that would help 
the contributor to dive in and make the changes.

I also updated the design document to explain in detail.

Thanks,
Qing


On 6/19/18, 12:09 PM, "Marco de Abreu"  
wrote:

Okay, thanks for elaborating. I definitely see your point there and we
definitely don't want these changes to pile up.

I don't feel strongly about this and won't stand in the way, I just want to
express my concern that this could lead to people having to touch all
language interfaces although they might not familiar with them at all. On
the other hand we got enough contributors who could help them then before
the PR can get merged. So either way works, but I just wanted to highlight
that this could make it harder to make changes in the backend for people
who are not familiar with our frontend API languages. If we got enough
people who could actively support our contributors in such a case, we
should be totally fine with blocking a PR until the APIs have been adapted.

-Marco

On Tue, Jun 19, 2018 at 11:58 AM Naveen Swamy  wrote:

> Marco,
>
> Qing and I are working together on this. The idea is that we fail the 
build
> if there is a operator change on the backend and have not synced to the
> Scala API. We want to catch this before breaking the user's code which 
will
> be a pretty bad experience.
>
>
>
> On Tue, Jun 19, 2018 at 11:54 AM, Marco de Abreu <
> marco.g.ab...@googlemail.com.invalid> wrote:
>
> > Hi Qing,
> >
> > thank you for working on improving the compatibility of our APIs!
> >
> > Your linked proposal does not describe the mentioned FILEHASH. Could you
> > elaborate a bit? Would this be a hash of the entire file, some hash
> created
> > based on the signature of the underlying C++ methods or maybe a 
different
> > approach?
> >
> > Also, at which step would developers be notified of the change? I'd
> propose
> > that we make this check a nightly job to prevent it from blocking a PR
> and
> > forcing contributors who are not familiar with Scala having to make a
> > change to the Scala package. This would allow us to follow up
> > asynchronously but still provide actionable events that one can be
> > subscribed to.
> >
> > Best regards,
> > Marco
> >
> > On Tue, Jun 19, 2018 at 11:00 AM Qing Lan  wrote:
> >
> > > Hi all,
> > >
> > > I am one of the maintainer for MXNet Scala package. Currently I am
> > > building up a hash-check system on the generated API through C. The PR
> > is
> > > in this URL:
> > > https://github.com/apache/incubator-mxnet/pull/11239
> > > A file named FILEHASH will be added to the Scala that created the MD5
> > > string of the generated API document. It will check the signature of
> all
> > > the operators during Scala CI testing. The reason I am doing this is 
to
> > > make sure Scala developers will always be reminded if there is an
> > operator
> > > name/argument changes happened in different PRs. More detailed info
> > > explained in here:
> > >
> > > https://cwiki.apache.org/confluence/display/MXNET/
> > MXNet+Scala+API+Usability+Improvement
> > >
> > > Pros:
> > > This method will always help us keep backwards compatibility of
> operators
> > > for Scala
> > > Cons:
> > > Require update on the FILEHASH with new contents when there is an
> > operator
> > > change. Developers who changed the operator should `make scalapkg` to
> > > update that file.
> > >
> > > Please leave any thoughts you may have for this design and feel free 
to
> > > review the code.
> > >
> > > Thanks,
> > > Qing
> > >
> > >
> >
>

The operator check for Scala Package

[Qing Lan]

Hi all,

I am one of the maintainer for MXNet Scala package. Currently I am building up 
a hash-check system on the generated API through C. The PR  is in this URL:
https://github.com/apache/incubator-mxnet/pull/11239
A file named FILEHASH will be added to the Scala that created the MD5 string of 
the generated API document. It will check the signature of all the operators 
during Scala CI testing. The reason I am doing this is to make sure Scala 
developers will always be reminded if there is an operator name/argument 
changes happened in different PRs. More detailed info explained in here:
https://cwiki.apache.org/confluence/display/MXNET/MXNet+Scala+API+Usability+Improvement

Pros:
This method will always help us keep backwards compatibility of operators for 
Scala
Cons:
Require update on the FILEHASH with new contents when there is an operator 
change. Developers who changed the operator should `make scalapkg` to update 
that file.

Please leave any thoughts you may have for this design and feel free to review 
the code.

Thanks,
Qing

Re: MXNet issues labeling

[Qing Lan]

Hi All,
I would like to quote this:
```
Cathy:
I am working on this label bot to automate/simplify this labeling issue
process and send weekly report to maintainers. Design proposal is on cwiki:
https://cwiki.apache.org/confluence/display/MXNET/Deep+Learning+Based+GitHub+Label+Bot

Please feel free to let me know if you have
suggestions/requirements/expectations.

Thanks,
Cathy
```
Currently Cathy has done the design for labelling the issues and I believe that 
will help our community to track more issues quickly. 
Since we received a lot of +1s from this list last time, I would like to ask 
for a Label adding/removing permission for the bot that Cathy designed. 
Can we do something like this for a GitHub account?

Thanks,
Qing

On 5/22/18, 9:56 AM, "Roshani Nagmote"  wrote:

Sorry for incomplete email. Sent it too fast.
Thanks for all the comments. Aaron guessed it right. We can treat it as a
multi-label classification problem. :)
We are currently working on the design document, will share it on dev list
once it is more concrete.

@Marco, seems like a good idea too but as you said, it will still involve
manual labeling. We can do this as a temporary solution but need a
long-term solution.
@Hao, will explore that option too! Thanks!

Thanks,
Roshani


On Tue, May 22, 2018 at 9:52 AM, Roshani Nagmote 
wrote:

> Thanks for all the comments. Aaron guessed it right. We can treat it as a
> multi-label classification problem. :)
> We are currently working on the design document, will share it on dev list
> once it is more concrete.
>
> @Marco, seems like a good idea too but as you said, it will still involve
> manual labeling. We can do this as a temporary solution but need a more
> @Hao, will explore that option too! Thanks!
>
> Thanks,
> Roshani
>
> On Mon, May 21, 2018 at 5:42 PM, Jin, Hao  wrote:
>
>> Definitely a good idea. I think maybe we can also try out the new gluon
>> NLP toolkit on this task?
>>
>> On 5/21/18, 5:24 PM, "Anirudh"  wrote:
>>
>> Yes, I guessed that :). Was looking for more details.
>>
>> On Mon, May 21, 2018 at 5:03 PM, Aaron Markham <
>> aaron.s.mark...@gmail.com>
>> wrote:
>>
>> > AI obviously.
>> >
>> > On Mon, May 21, 2018 at 5:01 PM, Anirudh 
>> wrote:
>> >
>> > > Hi Roshani,
>> > >
>> > > Good suggestion! How will the bot decide what labels to add ?
>> > >
>> > > Anirudh
>> > >
>> > > On Mon, May 21, 2018 at 4:39 PM, Naveen Swamy > >
>> > wrote:
>> > >
>> > > > +1 for the proposal to triage issues, I think committers should
>> also do
>> > > > this exercise to understand the customer pain.
>> > > >
>> > > > I am also inclined to use a bot account like how Tensorflow and
>> other
>> > > repos
>> > > > do it, https://github.com/googlebot.
>> > > > https://github.com/tensorflow/tensorflow/pull/19445#event-16
>> 38027271
>> > -->
>> > > > This is auto-tagged by the bot, it would be cool if we could do
>> that as
>> > > > well.
>> > > >
>> > > >
>> > > >
>> > > > On Mon, May 21, 2018 at 4:25 PM, sandeep krishnamurthy <
>> > > > sandeep.krishn...@gmail.com> wrote:
>> > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Roshani for starting this thread.
>> > > > >
>> > > > > Yes, I think labeling the issues will help a lot in driving
>> the
>> > > attention
>> > > > > of contributors to specific areas and make it easy for new
>> > contributors
>> > > > to
>> > > > > search and pick their contribution.
>> > > > >
>> > > > > I agree manually doing it all the time is not scalable and
>> efficient.
>> > > > Your
>> > > > > proposal on bot script to auto-label, similar to the working
>> of
>> > Jenkins
>> > > > bot
>> > > > > to re-test, re-build actions, will be very useful and
>> effective.
>> > > Hence, I
>> > > > > am more inclined to your *option 1* to have a bot account to
>> add
>> > > labels.
>> > > > >
>> > > > > Best,
>> > > > > Sandeep
>> > > > >
>> > > > > On Mon, May 21, 2018 at 4:16 PM, Roshani Nagmote <
>> > > > > roshaninagmo...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Some of us here at Amazon as a part of our day job, are
>> triaging
>> > > Github
>> > > > > > issues to find where MXNet users are experiencing
>> difficulty and
>> > help
>> > > > the
>> > > > > > community focus on those areas. This 

Re: Vote to stop using JIRA

[Qing Lan]

+ 0
Can we keep this optional? Not totally abandon or support. 
Pros:
I used JIRA to track most of my PRs and can place them together at one place. 
It also being helpful if we find some issues and group them together as one 
case. 
Cons:
Currently JIRA does not allow someone who is not contributor to file an issue 
which makes first-time contributor hard to submit a PR.

Thanks,
Qing


On 6/8/18, 2:27 PM, "Hao Jin"  wrote:

+0.5
I'm an SDE working for MXNet Engine team at AWS and I've been using JIRA
for nearly every single PR (28 out of 29 PR at this moment) I created since
I joined the team 3 months ago. I wouldn't say it's a really bad
experience, but definitely not good.
I do agree that we need something like JIRA and extra effort other than
writing code to manage the project, but the current user experience with
JIRA really has room for improvement.
The more important question for the community is that, is JIRA good for
attracting and retaining open source contributors? Such a user experience
of JIRA could drive contributors away if we're really enforcing it.
In conclusion, I think the usage of JIRA should be of one's or team's own
choice, if you do have the need to manage some development process (like
our team), just continue to use it, but we should no longer enforce or
persuade anyone to use it.
I'm also attaching a typical workflow of mine using JIRA with sprint
management and story point tracking for a single task, I think everyone who
has used JIRA so far would share part of my experience.
Thanks,
Hao

Appendix: what I need to do when I'm working on a task with JIRA:
1. Before I start working on something:
1) create a JIRA ticket for that, choose the correct type and define a
proper name for it
2) go to backlog page to add it to a sprint if you want to use the
sprint management.
3) go to sprint management page to assign story point value if you need
the functionality (we recently started doing that)
2. When I start working on the task:
1) dig my ticket up (on the sprint page or backlog page or search for
it)
2) click "open and start progress" to move it to "IN PROGRESS" phase.
3. When I finish the code, for every new PR I'll have to:
1) dig my ticket up
2) copy the ticket number so that I can add it to the PR title
3) an extra click to move the ticket to REVIEW phase
4) create a PR on github, paste the "MXNET-xxx" I just copied inside an
extra pair of square brackets "[]"
5) still need to refer the related github issue in the PR if I'm
solving one of them
4. For every code review or comment I receive on the new PR:
1) the JIRA bot logs 10 mins on the ticket no matter what kind of
comment it is
2) JIRA sends me an email for every single one of such logs (so if you
receive 10 code review comments in a single code review you get 10 emails,
my highest record was 17, while github would bundle all of them in only 1
mail)
5. After the PR is merged I get an email from JIRA saying this is merged
(for the bot, merge is like another comment on the PR) but I still have to:
1) dig my ticket up
2) manually move it to DONE phase (bot does not do that automatically).
6. The task is considered finished at this point, but any new comments on
the PR will trigger the bot once again to log 10 mins on your ticket
together with another email coming to your mailbox, while github is already
sending an email for that.

On Fri, Jun 8, 2018 at 2:23 PM, Naveen Swamy  wrote:

> Eric/Mu/Tianqi,
>
> A couple of contributors  have used JIRA for the Scala project -- this is
> the first time, so there is some learning for us, We started off with a
> design proposal, followed by a call for contribution. We kept our progress
> open on the dev list and we found one contributor to help us during
> debugging/testing/maven package creation and also one of them is working 
on
> the Memory allocation issue in Scala not as a part of their day job; This
> is where I find value in managing the project features on JIRA, designs on
> the public wiki, etc.,. I am not claiming that this is perfect, this is a
> WIP and as a community we should give it a chance and try it out.
>
> I don't think most of us here have even looked at JIRA.
>
> P.S: I am traveling, my response will be delayed.
>
> -Naveen
>
>
> On Fri, Jun 8, 2018 at 12:34 PM, Anirudh  wrote:
>
> > Hi,
> >
> > I am not a big fan of JIRA either. But I would like to raise this issue
> of
> > committing to a decision after a VOTE has passed. I saw in PRs that 
there
> > was a disregard for JIRA and ignoring requests on PRs to link to JIRA.
> > Because of this, JIRA was not given a fair 

Re: Make scalapkg fails if USE_BLAS is set to openblas/mkl/apple

[Qing Lan]

Hi Pedro,
It came from this line:
https://github.com/apache/incubator-mxnet/blob/master/src/initialize.cc#L44

It brings us no information other than a "segmentation fault : 11". This came 
from internal to send an issue if there is a build failure. 

Thanks,
Qing


On 6/6/18, 11:17 AM, "Pedro Larroy"  wrote:

Naveen, do you have more information about why the signal handler was
causing issues for the scala package? This was also failing on my PR
validation which included backtraces for crashes in CI.

On Wed, Jun 6, 2018 at 6:38 PM, Anton Chernov  wrote:

> The problem still persists with make build, so no correlation with the
> build system.
>
> 2018-06-06 15:49 GMT+02:00 Naveen Swamy :
>
> > I am using make to build
> >
> > On Wed, Jun 6, 2018 at 6:47 AM, Anton Chernov 
> wrote:
> >
> > > Yes, we have checked that as well, but did not help in our case. I've
> > > checked out a commit close to 1.1 where it was still working and built
> it
> > > with the new ci scripts (using cmake build for armv7) and it failed
> very
> > > similar. It seems that the problem might be in the way we are building
> > the
> > > library.
> > >
> > > Are you using cmake or make for builds?
> > >
> > > 2018-06-06 14:47 GMT+02:00 Naveen Swamy :
> > >
> > > > By the way it turned out the problem was when we used
> > USE_SIGNAL_HANDLER
> > > > along with a combination of flags, try removing signal handler and
> see
> > if
> > > > it works
> > > >
> > > > > On Jun 6, 2018, at 12:28 AM, Anton Chernov 
> > > wrote:
> > > > >
> > > > > Unfortunately, I think this is the same behaviour that we're
> > observing
> > > on
> > > > > Raspberry Pi's. Currently we are bisecting the release to find the
> > > > breaking
> > > > > commit to have an idea what exactly is broken.
> > > > >
> > > > > What I can say for now that this failure is not deterministic (on
> > > RPi's)
> > > > > and the library import to python passes in 1 of 4 times. The
> creation
> > > of
> > > > > NDArray's in this case fails though in all cases with similar
> message
> > > > that
> > > > > the stack is corrupted.
> > > > >
> > > > > Will update on findings.
> > > > >
> > > > > -- Anton
> > > > >
> > > > >
> > > > > 2018-06-05 16:19 GMT+02:00 Pedro Larroy <
> > pedro.larroy.li...@gmail.com>
> > > :
> > > > >
> > > > >> Could you compile with debug symbols or get a core file? From 
this
> > > > output
> > > > >> is not clear why the crash is happening.
> > > > >>
> > > > >>> On Sun, May 27, 2018 at 10:04 AM, Naveen Swamy <
> mnnav...@gmail.com
> > >
> > > > wrote:
> > > > >>>
> > > > >>> Hi,
> > > > >>> I am working to publish MXNet-Scala package to maven and
> > encountering
> > > > an
> > > > >>> issue when trying to build with openblas/mkl/apple. This is on
> both
> > > the
> > > > >>> master and the 1.2.0 branch? Can some one help with this.
> > > > >>> make scalapkg fails when it calls the MXNet backend to get all
> the
> > > > APIs ?
> > > > >>> can someone help here? should I publish with blas disabled? I
> have
> > > > >> already
> > > > >>> quite a bit of time on this/
> > > > >>>
> > > > >>> [INFO]
> > > > >>> [INFO] Segmentation fault: 11
> > > > >>> [INFO]
> > > > >>> [INFO] Stack trace returned 10 entries:
> > > > >>> [INFO] [bt] (0)
> > > > >>> /home/ubuntu/mxnet-master/scala-package/init-native/
> > > > >>> linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(
> > > > >>> dmlc::StackTrace[abi:cxx11]()+0x1bc)
> > > > >>> [0x7f2f04ca58ec]
> > > > >>> [INFO] [bt] (1)
> > > > >>> /home/ubuntu/mxnet-master/scala-package/init-native/
> > > > >>> linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+
> > 0x31d7a4f)
> > > > >>> [0x7f2f07971a4f]
> > > > >>> [INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0)
> > > > >> [0x7f3096cd24b0]
> > > > >>> [INFO] [bt] (3)
> > > > >>> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/
> > > > >>> libjvm.so(+0x3e4afc)
> > > > >>> [0x7f3093e0aafc]
> > > > >>> [INFO] [bt] (4)
> > > > >>> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/
> > > > >>> libjvm.so(+0xa239d6)
> > > > >>> [0x7f30944499d6]
> > > > >>> [INFO] [bt] (5)
> > > > >>> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/
> > > > >>> libjvm.so(+0xa24cdc)
> > > > >>> [0x7f309444acdc]
> > > > >>> [INFO] [bt] (6)
> > > > >>> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/
> > > > >>> libjvm.so(+0xa24e4c)
> > > > >>> [0x7f309444ae4c]
> > > > >>> [INFO] [bt] (7)
> > > > >>> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/
> > > > >>> 

Show deprecation warning message for deprecated API

[Qing Lan]

Hi all,

I would like to quote an 
issue sent from Sandeep:
```

We should always show Deprecated warning message for deprecated API.
For example, I used mx.sym.SoftmaxActivation() API, but, it was deprecated in 
favor of mx.sym.softmax(). However, I never saw a warning message.

Similarly, we should identify and warn for all deprecated API usage.
```
I think we should add something to the API to label them deprecated and this 
process should be done automatically. Currently, these pieces of information 
appear in the Description of the function, which can be extracted using this C 
API:
“mxSymbolGetAtomicSymbolInfo”.
“.. note:: `Concat` is deprecated. Use `concat` instead.”
Regex can be used to determine whether the API is deprecated or not.

Thanks,
Qing

Re: MXNet issues labeling

[Qing Lan]

Great topic! Vote for Option 1
Labelling and problem triaging process should be a lot simpler for all 
developers and bring less load to committers.

Thanks,
Qing

On 5/21/18, 4:26 PM, "sandeep krishnamurthy"  
wrote:

Thanks,

Roshani for starting this thread.

Yes, I think labeling the issues will help a lot in driving the attention
of contributors to specific areas and make it easy for new contributors to
search and pick their contribution.

I agree manually doing it all the time is not scalable and efficient. Your
proposal on bot script to auto-label, similar to the working of Jenkins bot
to re-test, re-build actions, will be very useful and effective. Hence, I
am more inclined to your *option 1* to have a bot account to add labels.

Best,
Sandeep

On Mon, May 21, 2018 at 4:16 PM, Roshani Nagmote 
wrote:

> Hi,
>
> Some of us here at Amazon as a part of our day job, are triaging Github
> issues to find where MXNet users are experiencing difficulty and help the
> community focus on those areas. This is done by assigning labels to the
> Github issues. We do know that only labeling won’t solve the real problem
> but we will expand our scope to also attempt to resolve the issues.
> Categorizing issues could also help contributors and maintainers who know 
a
> particular area to pick up the issue and help the user.
>
> Right now, we just manually go through the issues. If they are questions,
> we redirect users to start a discussion on discuss forum, find the
> appropriate labels and then ask one of the committers to add those labels.
> This process is not very smooth as its completely manual and every time we
> need to ask committers to add labels.
>
> We want to be able to automate/simplify this issue labeling process.
> Right now, as far as I know, there's no way for non-committers to add
> labels. So, I want to propose two options:
>
> - Using a separate account having minimum permissions to run the bot 
script
> which will do the labeling. For this, we will need an account to be 
created
> from Apache infrastructure with proper access and they can control the
> access for the account through
> https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html
>
> - Using one of the committers auth token to run the script.
>
> Please let me know if you have any other ideas to do this.
>
> Thanks,
> Roshani
>



-- 
Sandeep Krishnamurthy

Re: The New Scala API

[Qing Lan]

t; > first generate a trait for all the APIs and have the the instance in
> the
> > > existing symbol. The usage would be something like below:
> > >
> > > trait SymbolAPIBase {
> > > /**
> > > api doc
> > > */
> > > def Activation()
> > > }
> > >
> > > object SymbolAPI extends SymbolAPIBase {
> > > def Activation()
> > > }
> > >
> > > object Symbol {
> > >  val api: SymbolBase = new SymbolAPI()
> > > }
> > >
> > > Now users would use it as
> > > Symbol.api.Activation()
> > >
> > > Creating a trait/Interface as several advantages:
> > > 1) User do not have to know which concrete class to use, can be 
decided
> > > later(Dependency Injection or Configuration driven application)
> > > 2) Easier to change the implementation later without breaking the user
> > code
> > > -> You can insert different implementations through configuration 
using
> > > frameworks such as Spring, etc., -> This is standard in most JVM 
driven
> > > projects
> > > 3) Easier to Unit test -> You can create Mocks easily using Mockito
> > > 4) helps with multiple inheritance…you cannot inherit multiple classes
> > >
> > > Let me know what do you guys think.
> > >
> > > Thanks, Naveen
> > >
> > > On Sun, Apr 22, 2018 at 9:09 PM, YiZhi Liu <liuyi...@apache.org>
> wrote:
> > >
> > > > I personally like the design here. Since we have seen technical
> > > > difficulties of compatibility, I would like to ask people pay
> > > > attention to the 'How to combine with existing APIs' section:
> > > > https://cwiki.apache.org/confluence/display/MXNET/
> > > > MXNet+Scala+API+Usability+Improvement#MXNetScalaAPIUsabilityImprovem
> > ent-
> > > > HowtocombinewithexistingAPIs
> > > >
> > > > Qing proposed three options,
> > > >
> > > > 1. Add a new Class/Object called "NewSymbol/NDArray" with full
> > > > implementation.
> > > > 2. Create a new Class and change the name space for all of the
> > > > functions (e.g Activation -> NewActivation) and let Symbol/NDArray
> > > > extends that.
> > > > 3. Create a new Class and override the Same functions with different
> > > > implementations.
> > > >
> > > > If we have to choose from option 1 and 2, I would like to +0.5 for
> > > > option 2, with which users can quickly aware of the new easy-to-use
> > > > API: they type 'Symbol.' in IDE as usual and these functions pop up.
> > > >
> > > > 2018-04-19 10:58 GMT-07:00 Qing Lan <lanking...@live.com>:
> > > > > Hi All,
> > > > >
> > > > > I am Qing, one of the Scala API maintainer for MXNet. I would like
> to
> > > > propose a new design on Scala APIs, it will be really helpful for
> user
> > to
> > > > use MXNet Symbol/NDArray. This is a follow-up from Naveen’s 
proposal.
> > > > >
> > > > > Background:
> > > > > The current design on Scala would take arguments as key-value pair
> > and
> > > > didn’t provide the type information for different arguments. There
> are
> > > > document missing for different functions which makes it even hard to
> > use.
> > > > >
> > > > > Our approach:
> > > > > We will provide a better designed Scala API for user to use with
> full
> > > > documentation and arguments definition. All arguments will be
> > > specifically
> > > > targeted to different functions. Please see one example that we show
> in
> > > the
> > > > Wiki<https://cwiki.apache.org/confluence/display/MXNET/
> > > > MXNet+Scala+API+Usability+Improvement> and leave any thoughts you
> may
> > > > have. This wiki includes examples, targets and scenarios we have so
> > far.
> > > > >
> > > > > Thanks,
> > > > > Qing
> > > >
> > > >
> > > >
> > > > --
> > > > Yizhi Liu
> > > > DMLC member
> > > > Amazon Web Services
> > > > Vancouver, Canada
> > > >
> > >
> >
>

The New Scala API

[Qing Lan]

Hi All,

I am Qing, one of the Scala API maintainer for MXNet. I would like to propose a 
new design on Scala APIs, it will be really helpful for user to use MXNet 
Symbol/NDArray. This is a follow-up from Naveen’s proposal.

Background:
The current design on Scala would take arguments as key-value pair and didn’t 
provide the type information for different arguments. There are document 
missing for different functions which makes it even hard to use.

Our approach:
We will provide a better designed Scala API for user to use with full 
documentation and arguments definition. All arguments will be specifically 
targeted to different functions. Please see one example that we show in the 
Wiki
 and leave any thoughts you may have. This wiki includes examples, targets and 
scenarios we have so far.

Thanks,
Qing