Re: CUDA recommendation

2019-05-24 Thread Jake Lee
Hi Aaron,
The performance regression that Sam mentioned above is this one [1]. Since
CUDA 10.1 doesn’t have  the regression, Nvidia suggested us to move to CUDA
10.1. Regarding documentation. I will raised a PR to update it. Thanks.
Jake
[1]
https://github.com/apache/incubator-mxnet/issues/14725

Aaron Markham 於 2019年5月24日 週五,17:52寫道:

> Sounds like we need an * near the CUDA 10.1 recommendation if there
> are known performance issues. Is there a particular issue # tracking
> the performance issues?
> I'm seeing a CUDA 10 Windows issue here that seems unresolved:
> https://github.com/apache/incubator-mxnet/issues/14479
>
> On Fri, May 24, 2019 at 5:00 PM Sheng Zha  wrote:
> >
> > 10.1 is recommended. The oldest CUDA version that we release is 8.0.
> >
> > -sz
> >
> > On 2019/05/24 23:29:38, Marco de Abreu  wrote:
> > > While we are at the topic, did we actually agree on dropping support
> for
> > > some versions? So far we are releasing all the way been to cuda 7.5 I
> think
> > >
> > > -Marco
> > >
> > > Skalicky, Sam  schrieb am Fr., 24. Mai
> 2019,
> > > 23:43:
> > >
> > > > Hi Aaron
> > > >
> > > > Right now, the most stable version is CUDA 9.2. CUDA 10 is supported
> and
> > > > some pip wheels are available, but there are known performance
> issues. And
> > > > we are quickly moving to CUDA 10.1. So things are still in flux now.
> I
> > > > think the best approach would be to wait a couple more weeks before
> > > > updating this part of the docs.
> > > >
> > > > Sam
> > > >
> > > >
> > > > > On May 24, 2019, at 2:09 PM, Aaron Markham <
> aaron.s.mark...@gmail.com>
> > > > wrote:
> > > > >
> > > > > What version of CUDA is currently recommended?
> > > > > Now that there are packages for CUDA 10, shouldn't the build from
> > > > > source (and other) documentation reflect the latest greatest
> > > > > combinations?
> > > > >
> > > > > Specifically the Ubuntu guide [1] states: "CUDA 9.2 is
> recommended."
> > > > >
> > > > > [1]
> > > >
> https://mxnet.incubator.apache.org/versions/master/install/ubuntu_setup.html#cuda-dependencies
> > > > >
> > > > > Cheers,
> > > > > Aaron
> > > >
> > > >
> > >
>


Re: Dependency Update

2019-05-22 Thread Jake Lee
Thanks Aaron that's a great suggestion.
The reason why I put it under tools/dependencies is that the doc is
intended for developers who want to contribe to update the dependencies of
our PyPI package.
Regarding the CI, I'm also working on upgrading CUDA/cuDNN version that CI
use -PR[1][2].

Thanks,
Jake

[1] https://github.com/apache/incubator-mxnet/pull/14986
[2] https://github.com/apache/incubator-mxnet/pull/14950

On Wed, May 22, 2019 at 3:15 PM Aaron Markham 
wrote:

> Thanks for doing a thorough look at the version ranges. I have this PR
> [1] waiting for review that tries to pin graphviz and opencv, and it
> updates CI as well as the docs that go on the website.
> I think your updates would be beneficial in the docs that go on the
> website and should also update CI. Is there a benefit to having them
> as a readme in /tools? Doesn't this create extra maintenance with
> these version numbers being in three places (website install
> instructions, /tools folder, /ci folder)?
>
> [1] https://github.com/apache/incubator-mxnet/pull/14987
>
> On Wed, May 22, 2019 at 2:31 PM Qing Lan  wrote:
> >
> >
> > Great work Jake! The content on CPU/GPU build instruction is really
> helpful.
> >
> > Thanks,
> > Qing
> >
> > 
> > From: Jake Lee 
> > Sent: Wednesday, May 22, 2019 17:26
> > To: dev@mxnet.incubator.apache.org
> > Subject: Dependency Update
> >
> > Dear Community,
> >
> > I have been working on dependency udpate of MXNet. The goal is to upgrade
> > the dependencies that have security vulnerability issues and make MXNet
> > great again by being benefit from latest CUDA, cuDNN and NCCL software. I
> > documented the process on PR<
> > https://github.com/apache/incubator-mxnet/pull/15045>. Big thanks to
> Sheng
> > Zha(szha), Dick Carter(DickJC123), Anirudh Subramanian(anirudh2290), Qing
> > Lan(lanking520), Per Goncalves da Silva(perdasilva) for supporting me!
> >
> > Thanks,
> > Jake
>


Dependency Update

2019-05-22 Thread Jake Lee
Dear Community,

I have been working on dependency udpate of MXNet. The goal is to upgrade
the dependencies that have security vulnerability issues and make MXNet
great again by being benefit from latest CUDA, cuDNN and NCCL software. I
documented the process on PR<
https://github.com/apache/incubator-mxnet/pull/15045>. Big thanks to Sheng
Zha(szha), Dick Carter(DickJC123), Anirudh Subramanian(anirudh2290), Qing
Lan(lanking520), Per Goncalves da Silva(perdasilva) for supporting me!

Thanks,
Jake


Re: Python2 End of Life

2019-05-13 Thread Jake Lee
+1 Recently I upgraded the Numpy version and found out that Pylint had
false alarm on it. The Pylint fix is only available on Python3. So I
changed the default python version of 'make pylint' command to python3 (PR
haven't been merged). It's time to drop support for Python2.

On Mon, May 13, 2019 at 1:37 PM Junru Shao  wrote:

> +1
>
> On Mon, May 13, 2019 at 1:34 PM Aaron Markham 
> wrote:
>
> > +1 for the pledge and to start moving things to Python 3.
> > I think our installation instructions and tutorials can be updated to
> > default to Python3 and we should update Python2-only tutorials. I know
> > we have a handful of those, and when I spot them, I'll create an
> > issue.
> > I can also look at migrating the docs build to Python 3.
> > Should we add a new label for issues relating to migrating to Python3?
> > Cheers,
> > Aaron
> >
> > On Mon, May 13, 2019 at 12:04 PM Zach Kimberg  >
> > wrote:
> > >
> > > Right now, the official date for ending support for Python 2.7 (and all
> > of
> > > python2) is set to January 1 [1]. As part of it, a number of projects
> > have
> > > pledged to drop support for Python2 in or before 2020 including
> > Tensorflow,
> > > requests, pandas, ipython, numpy, pillow, and Cython [2]. I believe we
> > > should also join in this pledge on python3statement.org [2] because it
> > > would help clean up our project and it would be difficult to continue
> > > supporting Python2 anyway when some of our dependencies are dropping
> > > support.
> > >
> > > As a concrete step, we should decide on a date to remove all usages of
> > > Python2 from our CI and consider that officially dropping support.
> > > Following that, we can expect PRs will end up breaking support for
> > Python2.
> > > I suggest just using the same date that Python is dropping support of
> > > January 1. We may also need to update some examples or scripts that
> were
> > > written only for python2 that are around the project. Any thoughts?
> > >
> > > Zach
> > >
> > >
> > > [1] - https://www.python.org/dev/peps/pep-0373/
> > > [2] - https://python3statement.org/
> >
>


Re: [Announcement] New Committer - Zach Kimberg

2019-05-09 Thread Jake Lee
Congrat!

On Thu, May 9, 2019 at 10:37 AM Yuan Tang  wrote:

> Welcome!
>
> On Thu, May 9, 2019 at 1:36 PM Marco de Abreu 
> wrote:
>
> > Welcome!
> >
> > Hagay Lupesko  schrieb am Do., 9. Mai 2019, 19:33:
> >
> > > Congratulations Zach - well deserved!
> > >
> > > On Thu, May 9, 2019, 13:26 Qing Lan  wrote:
> > >
> > > > Hi All,
> > > >
> > > > Please join me in welcoming Zach Kimberg (https://github.com/zachgk)
> > as
> > > a
> > > > new committer.
> > > >
> > > > He has been solving some important bugs in MXNet JVM with respect to
> > > usage
> > > > improvement, build issues and a lot more. He also created the Jenkins
> > > based
> > > > publish pipeline for us to have standard way to build and test
> > > > static-linked package conveniently for everyone in the community.
> > > Moreover,
> > > > he solved a bunch of License problems we have in MXNet and brought
> > > several
> > > > fixes to let us get 1.4.0 release on time.
> > > >
> > > > Thanks,
> > > > Qing
> > > >
> > >
> >
>


Re: [Announcement] New Committer - Patric Zhao

2019-03-21 Thread Jake Lee
Congrats, Patric!

Jake

On Thu, Mar 21, 2019 at 12:03 PM Lin Yuan  wrote:

> Congrats, Patric!
>
> On Thu, Mar 21, 2019 at 10:32 AM Yuxi Hu  wrote:
>
> > Congrats, Patric! Well deserved!
> >
> > On Wed, Mar 20, 2019 at 1:08 PM kellen sunderland <
> > kellen.sunderl...@gmail.com> wrote:
> >
> > > Congrats Patric!
> > >
> > > On Sun, Mar 17, 2019 at 10:34 PM Hagay Lupesko 
> > wrote:
> > >
> > > > Congrats Patric!
> > > >
> > > > On Fri, Mar 15, 2019 at 7:49 AM Joshua Z. Zhang <
> cheungc...@gmail.com>
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > >
> > > > >  Congrats Patrick!
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >  Zhi
> > > > >
> > > > > >
> > > > > > On Mar 15, 2019 at 10:46 PM,   > > > > marco.g.ab...@gmail.com)>  wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > >  Congratulations, great to have you on board!
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > > Lv, Tao Aschrieb am Fr., 15. März 2019,
> > > 15:38:
> > > > > >
> > > > > > >  Wow, congratulation Patric!
> > > > > > >
> > > > > > >  -Original Message-
> > > > > > >  From: Steffen Rochel [mailto:steffenroc...@gmail.com]
> > > > > > >  Sent: Friday, March 15, 2019 10:25 PM
> > > > > > >  To: dev@mxnet.incubator.apache.org
> > > > > > >  Cc: patric zhao  
> > > > > > >  Subject: Re: [Announcement] New Committer - Patric Zhao
> > > > > > >
> > > > > > >  Congratulation Patrick!
> > > > > > >  Steffen
> > > > > > >
> > > > > > >  On Fri, Mar 15, 2019 at 5:38 AM Zhao, Patric  <
> > > > patric.z...@intel.com>
> > > > >
> > > > > > >  wrote:
> > > > > > >
> > > > > > >   >  I am very glad to have this opportunity to contribute to
> the
> > > > > > >   >  Apache/MXNet community :)
> > > > > > >   >
> > > > > > >   >  Thanks all of the supports from the community and Intel.
> > > > > > >   >
> > > > > > >   >  BR,
> > > > > > >   >
> > > > > > >   >  --Patric
> > > > > > >   >
> > > > > > >   >
> > > > > > >   >   >  -Original Message-
> > > > > > >   >   >  From: MiraiWK WKCN [mailto:w...@live.cn]
> > > > > > >   >   >  Sent: Friday, March 15, 2019 12:52 AM
> > > > > > >   >   >  To: dev@mxnet.incubator.apache.org; patric zhao
> > > > > > >   >   >   
> > > > > > >   >   >  Subject: Re: [Announcement] New Committer - Patric
> Zhao
> > > > > > >   >   >
> > > > > > >   >   >  Welcome Peng Zhao!
> > > > > > >   >   >  Peng is the AI Tech Leader in Intel Corporation. We
> have
> > > > > good
> > > > > > >   >   >  cooperation before. He is very professional and
> > > contribute a
> > > > > lot to
> > > > > > >   >   >  MXNet,
> > > > > > >   >  especially deep
> > > > > > >   >   >  learning boost on CPU.
> > > > > > >   >   >
> > > > > > >   >   >  
> > > > > > >   >   >  From: Anirudh Subramanian  
> > > > > > >   >   >  Sent: Thursday, March 14, 2019 3:54:50 PM
> > > > > > >   >   >  To: dev@mxnet.incubator.apache.org; patric zhao
> > > > > > >   >   >  Subject: [Announcement] New Committer - Patric Zhao
> > > > > > >   >   >
> > > > > > >   >   >  Hi all,
> > > > > > >   >   >
> > > > > > >   >   >  Please join me to welcome Patric Zhao as a new
> committer
> > > of
> > > > > Apache
> > > > > > >   >   >  (incubating) MXNet!
> > > > > > >   >   >
> > > > > > >   >   >  Patric has put in great effort around MKLDNN
> integration
> > > > into
> > > > > MXNet
> > > > > > >   >   >  and
> > > > > > >   >  has
> > > > > > >   >   >  been involved in features like quantization, graph
> > fusion
> > > > and
> > > > > fused
> > > > > > >   >   >  RNN operators for CPU.
> > > > > > >   >   >
> > > > > > >   >   >  Dev List activity:
> > > > > > >   >   >
> > > > > > >   >
> > > > >
> > https://lists.apache.org/list.html?d...@mxnet.apache.org:lte=3y:patric.
> > > > > > >   >  zhao
> > > > > > >   >   >
> > > > > > >   >   >  Issues:
> > > > > > >   >   >  https://github.com/apache/incubator-
> > > > > > >   >   >
> > > > > mxnet/issues?utf8=%E2%9C%93=is%3Aissue+involves%3Apengzhao-intel+
> > > > > > >   >   >
> > > > > > >   >   >  PR Reviews:
> > > > > > >   >   >  https://github.com/apache/incubator-
> > > > > > >   >   >
> > > > > mxnet/pulls?utf8=%E2%9C%93=is%3Apr+reviewed-by%3Apengzhao-intel
> > > > > > >   >   >
> > > > > > >   >   >  Proposals involved in:
> > > > > > >   >   >
> > > > >
> https://cwiki.apache.org/confluence/display/MXNET/MXNet+Graph+Optimi
> > > > > > >   >   >  z
> > > > > > >   >   >  ation+and+Quantization+based+on+subgraph+and+MKL-DNN
> > > > > > >   >   >
> > > > >
> https://cwiki.apache.org/confluence/display/MXNET/Fused+RNN+Operator
> > > > > > >   >   >  s
> > > > > > >   >   >  +for+CPU
> > > > > > >   >   >   <
> > > > >
> https://cwiki.apache.org/confluence/display/MXNET/MXNet+Graph+Optim
> > > > > > >   >   >  i
> > > > > > >   >   >  zation+and+Quantization+based+on+subgraph+and+MKL-DNN>
> > > > > > >   >   >
> > > > > > >   >   >
> > > > > > >   >   >  Thanks,
> > > > > > >   >   >  Anirudh
> > > > > > >   >
> > > > > > >
> > > > > >
> 

Re: [Announcement] New Committer -- Lin Yuan

2019-02-03 Thread Jake Lee
Congrats Lin!

Tianqi Chen  於 2019年2月2日 週六 下午3:27寫道:

> Dear Community:
>
> Please join me to welcome Lin Yuan(@apeforest) as a new committer of
> Apache(incubating) MXNet!
>
> He has contributed to various improvements, including better compatibility
> of larger arrays across the codebase.
>
> Commits:
> https://github.com/apache/incubator-mxnet/commits?author=apeforest
>
> https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+author%3Aapeforest
>
>
> Reviews:
> https://github.com/apache/incubator-mxnet/pulls?utf8=%
> E2%9C%93=reviewed-by%3Aapeforest
>
> dev@ activitivity
> https://lists.apache.org/list.html?*@mxnet.apache.org:lte=6M:Lin%20Yuan
>
> Tianqi
>


Re: [Announce] Upcoming Apache MXNet (incubating) 1.4.0 release

2018-12-06 Thread Jake Lee
Hi Steffen,

It would be nice to add this PR to 1.4 release.
https://github.com/apache/incubator-mxnet/pull/13550

It fixes the imagedetiter issue for mxnet 1.3
https://github.com/apache/incubator-mxnet/issues/13037.

Thanks,

Jake Lee

Lin Yuan  於 2018年11月29日 週四 下午7:27寫道:

> Hi Steffen,
>
> Can we add the following PR to 1.4.0 release:
>
> https://github.com/apache/incubator-mxnet/pull/13452
>
> It's just a Python API returning header path so it should not cause any
> regression issues. But it is required for Horovod to integrate MXNet. It's
> better to have this in a minor release than patch release.
>
> Thanks,
>
> Lin
>
> On Thu, Nov 29, 2018 at 6:46 PM Steffen Rochel 
> wrote:
>
> > Hi Zhi - thanks for the improvement, which we should consider for 1.4.0.
> > However, I don't see any tests with the PR and think it is too risky to
> add
> > changes without tests. I will add your PR to the tracking list, but would
> > like to ask you to add functional tests before completing the PR to
> master
> > and v1.4.x branch.
> >
> > Steffen
> >
> > On Thu, Nov 29, 2018 at 5:01 PM Joshua Z. Zhang 
> > wrote:
> >
> > > Hi, I would like to bring a critical performance and stability patch of
> > > existing gluon dataloader to 1.4.0:
> > > https://github.com/apache/incubator-mxnet/pull/13447 <
> > > https://github.com/apache/incubator-mxnet/pull/13447>.
> > >
> > > This PR is finished, waiting for CI to pass.
> > >
> > > Steffen, could you help me add that to the tracked list?
> > >
> > > Best,
> > > Zhi
> > >
> > > > On Nov 29, 2018, at 4:25 PM, Naveen Swamy 
> wrote:
> > > >
> > > > the tests are randomly failing in different stages
> > > >
> > >
> >
> http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-13105/
> > > > This PR has failed 8 times so far
> > > >
> > > > On Thu, Nov 29, 2018 at 3:43 PM Steffen Rochel <
> > steffenroc...@gmail.com>
> > > > wrote:
> > > >
> > > >> Pedro - ok. Please add PR to v1.4.x branch after merge to master and
> > > please
> > > >> update tracking page
> > > >> <
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.4.0+Release+Plan+and+Status#ApacheMXNet(incubating)1.4.0ReleasePlanandStatus-OpenPRstotrack
> > > >>>
> > > >> .
> > > >> Steffen
> > > >>
> > > >> On Thu, Nov 29, 2018 at 3:00 PM Pedro Larroy <
> > > pedro.larroy.li...@gmail.com
> > > >>>
> > > >> wrote:
> > > >>
> > > >>> PR is ready from my side and passes the tests, unless somebody
> raises
> > > >>> any concerns it's good to go.
> > > >>> On Thu, Nov 29, 2018 at 9:50 PM Steffen Rochel <
> > > steffenroc...@gmail.com>
> > > >>> wrote:
> > > >>>>
> > > >>>> Pedro - added  to 1.4.0 tracking list
> > > >>>> <
> > > >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.4.0+Release+Plan+and+Status#ApacheMXNet(incubating)1.4.0ReleasePlanandStatus-OpenPRstotrack
> > > >>>>
> > > >>>>
> > > >>>> Do you have already ETA?
> > > >>>> Steffen
> > > >>>>
> > > >>>> On Thu, Nov 29, 2018 at 6:13 AM Pedro Larroy <
> > > >>> pedro.larroy.li...@gmail.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi all.
> > > >>>>>
> > > >>>>> There are two important issues / fixes that should go in the next
> > > >>>>> release in my radar:
> > > >>>>>
> > > >>>>> 1) https://github.com/apache/incubator-mxnet/pull/13409/files
> > > >>>>> There is a bug in shape inference on CPU when not using MKL, also
> > we
> > > >>>>> are running activation on CPU via MKL when we compile
> CUDNN+MKLDNN.
> > > >>>>> I'm finishing a fix for these issues in the above PR.
> > > >>>>>
> > > >>>>> 2) https://github.com/apache/incubator-mxnet/issues/13438
> 

Re: Regressions in NDArrayIter

2018-09-11 Thread Jake Lee
I am looking into it.

Could you provide the shape of NDArrayIter and minimal reproducible code?
Thank you so much!

Alfredo Luque  於 2018年9月11日 週二 下午6:05寫道:

> Looks like https://github.com/apache/incubator-mxnet/pull/12285 broke a
> ton
> of our test cases iterating over 3D NDArray instances (eg; MNIST) by
> creating an index out of range.
>
> Stacktrace:
>
> .com/airbnb/bighead/python/bighead/ml_frameworks/mxnet/gluon.py", line
> 434, in transform
> for batch in data_iter:
>   File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py",
> line 228, in __next__
> return self.next()
>   File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py",
> line 680, in next
> label = self.getlabel()
>   File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py",
> line 750, in getlabel
> return self._batchify(self.label)
>   File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py",
> line 732, in _batchify
> first_data = self._getdata(data_source, start=self.cursor)
>   File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py",
> line 694, in _getdata
> end = end if end is not None else data_source[0][1].shape[0]
> IndexError: list index out of range
>
> I’ve created an issue at
> https://github.com/apache/incubator-mxnet/issues/12526
>
>
> We’ll be pinning to the previous build until it’s reverted/patched, but let
> us know if we can help provide more regression tests here.
>
> —
> Alfredo Luque
> Software Engineer
> Machine Learning Infrastructure
> Airbnb
> San Francisco, CA
>